Easy-to-understand network protocols (TCP/IP overview)

My recent work is related to network protocols, which gives me the opportunity to study network protocols more deeply. For a long time before, my understanding of network protocols remained at a relatively shallow level.

For example: TCP is connection-oriented and reliable transmission, while UDP is connectionless and unreliable transmission. TCP requires three handshakes to establish a connection, which will cause delay, while UDP is faster.

For example: socket programming, server socket create, bind, listen, accept, read/write, shutdown/close, client socket create, connect, read/write, shutdown/close, plus epoll/select.

For example: I know that network programming should ignore SIGPIPE signals or it will hang, read returns 0 means that the other end actively closed, non-blocking read should be placed in a loop and the return value should be considered, multiplexing and the difference between blocking and non-blocking.

This is how I understand the difference between TCP and UDP: from Beijing to Hangzhou, TCP is equivalent to building a high-speed rail line (building a connection) and then opening it to traffic for shipment (transmitting data), while UDP is equivalent to sending a courier and not caring if it is lost (directly transmitting data).

Is the above understanding correct? It can be said that it is correct or not. For application programmers, is it enough to have the above understanding and be familiar with the socket programming interface? Is it not enough?

The great physicist Feynman proposed an efficient Feynman learning method, which is to start with the problem, try to explain all the problems, and replace learning with teaching. Once you can explain all the problems clearly, you have learned it. So I want to try to explain TCP/IP clearly, so that I can learn it myself and help readers by the way.

Although "TCP/IP Detailed Explanation Volume 1" is a very rigorous and detailed book about the Internet protocol family, in my opinion, it is a little obscure and may require several readings before you can understand it. Although I am not able to explain this issue better, because I have experienced the process of going from confusion to a little understanding, which may be incomparable to the masters, I will try my best to explain the knowledge related to TCP/IP in plain language.

What is TCP/IP

The TCP/IP protocol suite is a collection of protocols, also called the Internet protocol suite, which is used to implement mutual communication between hosts on the Internet. TCP and IP are just two of the protocols, and they are also two very important protocols, so TCP/IP is used to name this Internet protocol suite. In fact, it also includes other protocols, such as UDP, ICMP, IGMP, ARP/RARP, etc.

Network Layering

The classic ISO seven-layer network model is found in the university's "Computer Networks" textbook, but the seven-layer division is too detailed, a bit cumbersome, and not easy to remember.

The Internet protocol family TCP/IP is divided into four coarse-grained layers. The comparison chart of the two divisions makes the relationship between them clear at a glance.

Layering is a common technique in the computer field. For example, the three-layer architecture of the Internet backend, "access-logic-storage", is a typical application of the layered idea.

The purpose of layering is isolation. Functions are divided by layering, problems are broken down, interfaces are agreed upon between layers, and implementation details are shielded.

TCP/IP is divided into link layer, network layer, transport layer and application layer from bottom to top. The lower layers provide capabilities to the upper layers, and the upper layers use the capabilities of the lower layers to provide higher abstraction.

• The link layer, also known as the network interface layer, includes the operating system's device drivers and network cards, which together handle the details of the physical interface with the transmission medium (optical fiber, etc.).
• The network layer, also known as the IP layer, is responsible for handling the transmission of IP datagrams in the network. The IP layer transmits IP datagrams and, with the help of routing tables, transmits IP datagrams from one end of the network to the other. In short: IP implements packet routing transmission, and IP protocols and routers work at the network layer.
• The transport layer provides end-to-end communication, including TCP, which is connection-oriented and highly reliable, and UDP, which is connectionless and unreliable. TCP may seem better, but it is not. UDP is faster and widely used because it does not require connection establishment overhead. For example, the new generation of Internet protocol HTTP3 has switched from TCP to UDP. The transport layer protocol should be selected according to the application scenario.
• The application layer is related to applications. Different applications solve different problems and require different application layer protocols.

The link layer handles the transmission of data on the media, as well as the details of the host's interaction with the network card, optical fiber, etc. Because it is related to hardware, it requires the help of the system driver. The link layer protocol defines these details, such as how to send data from the network card to the optical fiber, what format to use for encoding, etc. It solves the problem of data representation and flow on the media.

It is definitely not enough to have link layer functions alone. There are thousands of machines on the network. Host A communicates with Host B, but you cannot send data to Host C. Therefore, in imitation of reality, network addresses must be assigned to hosts. IP addresses are used to identify a host in the network. When a data packet is sent, it needs to be correctly routed to the destination. This is like the path you need to take from home to work. A map is needed, and the routing table is similar to this map. IP solves the problem of data packet transmission routing in the network.

Having the transmission and routing capabilities of the network layer is not enough, because IP packets may be lost during transmission. For example, packets may be lost if the buffer of the router is full. This is unreliable. If reliable transmission is required, the transport layer needs to provide more capabilities based on the IP layer. TCP solves the reliability problem. Specifically, if a packet is lost, the TCP layer will be responsible for timeout retransmission. It ensures reliable transmission through reception confirmation and retransmission mechanisms. In addition, because IP packets are independently routed, a piece of data is split into two IP packets, x and y, and sent successively from host A to host B. These two packets may choose different transmission paths. In this way, it is possible that packet y arrives before packet x, but we hope to restore the information of this data at the receiving end (host B), but we cannot control the order of arrival of IP packets. Therefore, we need to restore the data at the receiving end. I only need to record which part of the data block it belongs to in packets x and y, and then reassemble the data. This is exactly what TCP does. It will reassemble IP packets to ensure sequence and submit them to the application layer.

Sometimes there is no need to ensure reliability and order, which is what UDP can provide. It simply encapsulates the data into an IP packet and then sends it to the destination through IP layer routing.

Going further up, there are application layer protocols, such as http, and game server custom protocols. Application layer protocols are usually based on TCP or UDP for transmission.

Layering

What is a protocol? I am too lazy to look up various authoritative definitions of protocols. I think protocol is an agreement, which is similar to the meaning of the word protocol in real life. Network protocol is an agreement that both parties in communication abide by. More specifically, it defines the format, rules and process of data transmission on the network.

Because the network is a layered model and different layers have different functions, rules and corresponding protocols are defined for each layer.

As mentioned earlier, the TCP/IP protocol family includes many protocols, which belong to different layers and play different roles.

• TCP and UDP are the two main transport layer protocols.
• IP is the main protocol of the network layer. Both TCP and UDP need to use the IP protocol for data transmission.
• ICMP is the Internet Control Message Protocol, which is an auxiliary protocol of IP. The IP layer uses it to exchange error messages and other important information with other hosts or routers. For example, when a packet passes through a router node, it exceeds the network's length limit for the packet, and it is not fragmented. Then an ICMP packet will be sent to the sender to report the error information. ICMP is used to assist IP in completing data packet transmission.
• IGMP is the Internet Group Management Protocol, which is used to multicast a packet to multiple hosts.
• ARP (Address Resolution Protocol) and RARP (Reverse Address Resolution Protocol) are used to convert the addresses of the IP layer and the link layer. The IP layer uses the IP address and the link layer uses the Mac address.
• The application layer and transport layer use end-to-end protocols, and the network layer provides a hop-by-hop protocol.

Encapsulation

A sends a piece of data to B through the network. It can be imagined that simply transmitting the original data is not enough, because during the network transmission process, the network packet reaches a router and needs to be forwarded, and forwarding must rely on some additional information of the data packet, such as the target machine.

When the sender sends data, it adds some control information to the original data according to the protocol format and packages it into a data packet that can be correctly transmitted on the network. This process is called encapsulation.

The TCP/IP protocol family is encapsulated layer by layer, from the application layer to the link layer, and some additional information (header and tail) is added at each layer.

• User data is added to the application header by the application and transferred to the TCP layer for processing
• After the TCP layer adds the TCP header, a TCP segment is generated.
• The TCP segment passes through the IP layer and adds the IP header to generate an IP datagram.
• IP datagram passes through the link layer, is processed by the Ethernet driver, and then adds the Ethernet header + trailer to generate an Ethernet frame. The length of the Ethernet frame is between 46 and 1500.

To be more precise, the data unit transmitted at the IP and link layers is called a packet. A packet can be either an IP datagram or a fragment of an IP datagram.

UDP encapsulation is slightly different from TCP, mainly in that an 8-byte UDP header is added after passing through the transport layer (UDP) to generate a UDP datagram.

During the encapsulation process, when passing through the TCP/UDP layer, the port number will be added to the TCP/UDP header; when passing through the IP layer, the protocol type (TCP or UDP or ICMP or IGMP) will be added to the IP header; when passing through the link layer, the frame type (IP or ARP or RARP) will be added to the Ethernet header. This information will be used for processing at the receiving end.

After receiving the data, the receiving end needs to perform the opposite decapsulation operation of the sending end. We can compare the data encapsulation at the sending end to putting on clothes layer by layer after taking a shower, while the operation at the receiving end is similar to taking off clothes layer by layer before taking a shower, peeling off the beginning and the end to obtain the original data transmitted.

Because the hosts on the network have different byte orders, now in order to transmit through the network, it is necessary to agree on a unified network byte order (big endian). Hosts that use little endian must convert to big endian when transmitting data over the network.

address

Each interface on the Internet has a unique network address, also called an IP address. There are two versions of IP addresses, IPv4 and IPv6. IPv4 is a 32-bit 4-byte integer. The value range of each byte (8bit) is 0~255, so the 4-byte IPv4 can be represented by a byte value separated by four dots, such as 140.252.13.88. Each decimal value corresponds to each byte in the 32-bit integer. This representation is called dotted decimal representation. Obviously, the dotted decimal representation and int32 representation are easy to convert between each other.

IPv4 addresses are divided into five categories: A, B, C, and D. The numerical space represented by a 32-bit address is limited, and it is difficult to assign independent IP addresses to all networked devices on the Internet. Therefore, there are problems such as dynamic allocation, sharing, and public network + intranet address conversion (NAT). In essence, these are to solve the problem of insufficient IP addresses.

IPv6 uses 128 bits, and 2 to the power of 128 is very large, and it is said that it can assign an IP address to every grain of sand on the earth.

IP datagrams (network layer) use IP addresses, while Ethernet frames (link layer) use hardware (48-bit Mac) addresses. ARP and RARP are used to map (convert) IP addresses to hardware addresses.

port

TCP/UDP uses 16-bit port numbers to identify (distinguish) applications. For example, host A sends an IP packet to host B. After the kernel of host B receives the IP packet, which application should it hand it over to for processing? The port number is used for this purpose. The kernel will maintain the correspondence between the port number and the application.

The commonly used application layer protocols have agreed port numbers, which are also known as well-known port numbers. Port numbers between 1024 and 5000 are temporarily allocated to TCP/IP, while those greater than 5000 are used for other purposes. In other words, when you use TCP to connect to a network server, the port number locally allocated to the socket will be between 1024 and 5000, depending on the port allocation strategy of the operating system.

Domain Name System

The Domain Name System (DNS) provides conversion between host names and IP addresses. For example, www.baidu.com is a domain name. An application can obtain the IP address of a host with a given name through a standard library function (gethostbyname). The standard library function (gethostbyaddr) implements the inverse operation.

The IP address is a string of numbers, which has unclear meaning and is not easy to remember. The host name has a clearer meaning, and it is easy for you to remember www.baidu.com. This is why the host name is needed in addition to the IP address.

Use

After the receiving end receives the Ethernet data frame, it needs to peel the protocol stack from the bottom to the top, like peeling an onion, that is, follow the order of link layer -> network layer -> transport layer -> application layer, remove the header and tail added by each layer protocol, take out the data, and hand it over to the top layer application. This process is called demultiplexing, which is called demultiplexing according to the book.

Looking back at the encapsulation description above, at the transport layer, network layer, and link layer, the port number is stored in the TCP/IP header, the protocol type is stored in the IP header, and the frame type is stored in the Ethernet frame header. So at the receiving end, the headers are removed layer by layer, the corresponding information is taken out, and then it is dispatched and thrown to different modules for processing. The figure above shows the entire processing process.

summary

This article discusses concepts such as addresses, domain names, ports, TCP/IP layered model, encapsulation, and sharing.

You can best remember the four-layer division of TCP/IP link layer -> network layer -> transport layer -> application layer.

It is best for you to clearly distinguish the concepts of TCP segment, UDP datagram, IP datagram, IP fragment, Ethernet frame, and packet, the data unit transmitted between the IP layer and the link layer. This way you will appear more professional and not so unsophisticated when talking.

Data encapsulation, read it several times and you will remember it.

• TCP encapsulation format: Ethernet header (14) + IP header (20) + TCP header (20) + application data + Ethernet trailer (4)
• UDP encapsulation format: Ethernet header (14) + IP header (20) + UDP header (8) + application data + Ethernet trailer (4)

The application layer protocol is implemented at the application layer, while the transport layer, network layer, and link layer are all implemented in the kernel. Therefore, it is difficult to modify or optimize the underlying protocol because you can hardly change the kernel, because you cannot change the OS of a large number of devices on the network at the same time. This is the so-called network device rigidity problem. HTTP3 uses UDP instead of TCP, which is to achieve reliable transmission at the application layer itself.

Each Ethernet frame has a length limit (48~1500), and each device on the network also has a limit on the packet length. If the IP packet is too large, it must be fragmented. Fragmentation may occur at the sender or in the intermediate device, but fragmentation should be avoided as much as possible. The IP packet will carry information to allow it to be reassembled after fragmentation. You can learn about the concept of MTU.

ICMP and IGMP logically belong to the network layer because they are subsidiary protocols of the IP protocol, but in fact, both ICMP and IGMP messages are encapsulated as IP datagrams for transmission, so they can be regarded as protocols above the IP layer.

Similarly, ARP and RARP are used to convert IP addresses and hardware MAC addresses. Logically, they belong to the link layer, but in fact, arp and rarp messages are encapsulated into Ethernet Frame transmission like IP datagrams.

After the receiving end receives the Ethernet frame, it will go through the de-multiplexing process and finally hand over the original data to the application.

TCP/IP protocol applications often use the socket programming interface.

There are many network-related tools, such as ping, ifconfig, netstat, arp, tcpdump, wireshark, etc.