Meituan second interview: TCP's four waves, can it be reduced to three?

Hello everyone, I am Xiaolin. I have posted this article before, but I forgot to mark it as original. Some readers reported that my article was reposted by others, but the original author information was not indicated, so I reposted it and marked it as original. ---------- Main Text ----------

Last week, a reader was asked during an interview with Meituan: Among the four TCP handshakes, can the second ACK message be sent together with the third FIN message?

Although we learned that TCP handshaking requires four times to complete TCP handshaking, in some cases, TCP four times handshaking can be turned into TCP three times handshaking.

And when using the Wireshark tool to capture packets, we often see that the TCP handshake process is three times instead of four times, as shown in the following figure:

First, let's answer why the RFC document defines the TCP handshake process as four times? Then, let's answer under what circumstances will three handshakes occur?

Why does TCP need to wave four times?

The process of TCP's four handshakes is as follows:

Specific process:

• The client actively calls the function to close the connection, and then sends a FIN message. This FIN message means that the client will not send any more data and enters the FIN_WAIT_1 state;
• The server receives the FIN message and immediately replies with an ACK confirmation message. At this time, the server enters the CLOSE_WAIT state. When receiving the FIN message, the TCP protocol stack will insert an end-of-file character EOF into the receiving buffer for the FIN packet. The server application can perceive this FIN packet through the read call. This EOF will be placed after other received data that has been queued, so it is necessary to continue to read the received data in the receiving buffer;
• Next, when the server is reading data, it will naturally read EOF at the end, and then read() will return 0. At this time, if the server application has data to send, it will call the function to close the connection after sending the data. If the server application has no data to send, it can directly call the function to close the connection. At this time, the server will send a FIN packet. This FIN message means that the server will not send any more data, and then it will be in the LAST_ACK state;
• The client receives the FIN packet from the server and sends an ACK confirmation packet to the server. At this time, the client will enter the TIME_WAIT state;
• After the server receives the ACK confirmation packet, it enters the final CLOSE state;
• After 2MSL time, the client also enters the CLOSE state;

You can see that a FIN and an ACK are required in each direction, so it is often called four waves.

Why does TCP need to wave four times?

When the server receives the FIN message from the client, the kernel will immediately return an ACK message. However, the server application may still have data to send, so it cannot send the FIN message immediately. Instead, it gives the control of sending the FIN message to the server application:

• If the server application has data to send, it will call the function to close the connection after sending the data;
• If the server application has no data to send, it can directly call the function to close the connection.

From the above process, we can see that the control of whether to send the third wave is not in the kernel, but in the application of the passive closing party (the server in the above figure). Because the application may still have data to send, it is up to the application to decide when to call the function to close the connection. When the function to close the connection is called, the kernel will send a FIN message, so the ACK and FIN of the server are generally sent separately.

Does the FIN message have to call the function that closes the connection before it can be sent?

Not necessarily. If a process exits, whether it is a normal exit or an abnormal exit (such as a process crash), the kernel will send a FIN message and complete four handshakes with the other party.

Brutal Closure vs Graceful Closure

When introducing TCP's four handshakes, the function of closing the connection was not introduced in detail. In fact, there are two functions for closing the connection:

• The close function closes the sending and reading directions of the socket at the same time, that is, the socket no longer has the ability to send and receive data. If multiple processes/threads share the same socket, if one process calls close, it will only make the socket reference count -1, which will not make the socket unavailable, and will not send a FIN message. Other processes can still read and write the socket normally until the reference count becomes 0, and then a FIN message will be sent.
• The shutdown function can specify that the socket only closes the sending direction but not the reading direction, that is, the socket no longer has the ability to send data, but still has the ability to receive data. If multiple processes/threads share the same socket, shutdown will make the socket unavailable regardless of the reference count, and then send a FIN message. If other processes attempt to use the socket, they will be affected.

If the client uses the close function to close the connection, then during the TCP four-wave process, if the client receives data from the server, the client's kernel will return a RST message to the server because the client no longer has the ability to send and receive data. Then the kernel will release the connection, and the TCP four-wave process will not be completed. Therefore, we often say that calling close is a rough shutdown.

When the server receives the RST, the kernel will release the connection. When the server application initiates a read or write operation again, it will sense that the connection has been released:

• If it is a read operation, an RST error will be returned, which is the common Connection reset by peer.
• If it is a write operation, the program will generate a SIGPIPE signal. The application layer code can capture and process the signal. If it is not processed, the process will terminate and exit abnormally by default.

In contrast, the shutdown function can specify to close only the sending direction but not the reading direction. Therefore, even during the TCP four-wave process, if the client receives data sent by the server, it can read the data normally, and then it will go through the complete TCP four-wave process. Therefore, we often say that calling shutdown is an elegant shutdown.

But please note that the shutdown function can also specify "only close the reading direction, not the sending direction", but the kernel will not send a FIN message at this time, because sending a FIN message means that we will no longer send any data. If shutdown specifies "do not close the sending direction", it means that the socket still has the ability to send data, so the kernel will not send FIN.

Under what circumstances will three waves occur?

When the passive closing party (the server in the figure above) has "no data to send" and "the TCP delayed confirmation mechanism is turned on" during the TCP wave process, the second and third waves will be transmitted together, resulting in three waves.

Then, because the TCP delayed acknowledgment mechanism is enabled by default, when we capture packets, we see more triple waves than quadruple waves.

What is TCP delayed acknowledgment mechanism?

When sending an ACK without data, its network efficiency is also very low, because it also has a 40-byte IP header and TCP header, but does not carry data packets. In order to solve the problem of low ACK transmission efficiency, TCP delayed acknowledgment is derived. TCP delayed acknowledgment strategy:

• When there is response data to be sent, ACK will be sent to the other party immediately along with the response data.
• When there is no response data to send, ACK will be delayed for a while to wait for whether there is response data to send together

If the other party's second data message arrives during the delay waiting for sending ACK, ACK will be sent immediately.

The delay waiting time is defined in the Linux kernel, as shown below:

The key is the HZ value. HZ is related to the system clock frequency. Each operating system is different. In my Linux system, the HZ value is 1000, as shown below:

Knowing the size of HZ, we can calculate:

• The maximum delay confirmation time is 200 ms (1000/5)
• The shortest delay confirmation time is 40 ms (1000/25)

How to turn off TCP delayed acknowledgment mechanism?

If you want to turn off the TCP delayed confirmation mechanism, you can enable TCP_QUICKACK in the Socket settings. Enabling TCP_QUICKACK is equivalent to turning off the TCP delayed confirmation mechanism.

 // 1 means turning on TCP_QUICKACK, that is, turning off the TCP delayed confirmation mechanism
 int value = 1 ;
 setsockopt ( socketfd , IPPROTO_TCP , TCP_QUICKACK , ( char * ) & value , sizeof ( int ) ) ;

Experimental verification

Experiment 1

Next, let's do an experiment to verify this conclusion:

When the passive closing party (the server in the figure above) has "no data to send" and "the TCP delayed confirmation mechanism is turned on" during the TCP wave process, the second and third waves will be transmitted together, resulting in three waves.

The server code is as follows. It does a simple job: it reads data, and then immediately calls close to close the connection when read returns 0. Because the TCP delayed acknowledgment mechanism is enabled by default, no special settings are required.

 #include < stdlib .h >
 #include < stdio .h >
 #include < errno .h >
 #include < string .h >
 #include < netdb .h >
 #include < sys / types.h >
 #include < netinet / in .h >
 #include < sys / socket.h >
 #include < netinet / tcp.h >

 #define MAXLINE 1024

 int main ( int argc , char * argv [ ] )
 {

 // 1. Create a listening socket
 int listenfd = socket ( AF_INET , SOCK_STREAM , 0 ) ;
 if ( listenfd < 0 )
 {
 fprintf ( stderr , "socket error : %s\n" , strerror ( errno ) ) ;
 return - 1 ;
 }

 // 2. Initialize server address and port
 struct sockaddr_in server_addr ;
 bzero ( & server_addr , sizeof ( struct sockaddr_in ) ) ;
 server_addr .sin_family = AF_INET ;
 server_addr .sin_addr .s_addr = htonl ( INADDR_ANY ) ;
 server_addr .sin_port = htons ( 8888 ) ;

 // 3. Bind address + port
 if ( bind ( listenfd , ( struct sockaddr * ) ( & server_addr ) , sizeof ( struct sockaddr ) ) < 0 )
 {
 fprintf ( stderr , "bind error:%s\n" , strerror ( errno ) ) ;
 return - 1 ;
 }

 printf ( "begin listen....\n" ) ;

 // 4. Start monitoring
 if ( listen ( listenfd , 128 ) )
 {
 fprintf ( stderr , "listen error:%s\n\a" , strerror ( errno ) ) ;
 exit ( 1 ) ;
 }


 // 5. Get the connected socket
 struct sockaddr_in client_addr ;
 socklen_t client_addrlen = sizeof ( client_addr ) ;
 int clientfd = accept ( listenfd , ( struct sockaddr * ) & client_addr , & client_addrlen ) ;
 if ( clientfd < 0 ) {
 fprintf ( stderr , "accept error:%s\n\a" , strerror ( errno ) ) ;
 exit ( 1 ) ;
 }

 printf ( "accept success\n" ) ;

 char message [ MAXLINE ] = { 0 } ;

 while ( 1 ) {
 // 6. Read the data sent by the client
 int n = read ( clientfd , message , MAXLINE ) ;
 if ( n < 0 ) { // read error
 fprintf ( stderr , "read error:%s\n\a" , strerror ( errno ) ) ;
 break ;
 } else if ( n == 0 ) { // Return 0 , indicating that the FIN message is read
 fprintf ( stderr , "client closed \n" ) ;
 close ( clientfd ) ; // No data to send, close the connection immediately
 break ;
 }

 message [ n ] = 0 ;
 printf ( "received %d bytes: %s\n" , n , message ) ;
 }

 close ( listenfd ) ;
 return 0 ;
 }

The client code is as follows. It is also very simple. After successfully connecting to the server, it sends data to the server, and then after sleeping for one second, it calls close to close the connection. Therefore, the client is the active closing party:

 #include < stdlib .h >
 #include < stdio .h >
 #include < errno .h >
 #include < string .h >
 #include < netdb .h >
 #include < sys / types.h >
 #include < netinet / in .h >
 #include < sys / socket.h >

 int main ( int argc , char * argv [ ] )
 {

 // 1. Create a listening socket
 int connectfd = socket ( AF_INET , SOCK_STREAM , 0 ) ;
 if ( connectfd < 0 )
 {
 fprintf ( stderr , "socket error : %s\n" , strerror ( errno ) ) ;
 return - 1 ;
 }

 // 2. Initialize server address and port
 struct sockaddr_in server_addr ;
 bzero ( & server_addr , sizeof ( struct sockaddr_in ) ) ;
 server_addr .sin_family = AF_INET ;
 server_addr .sin_addr .s_addr = inet_addr ( "127.0.0.1" ) ;
 server_addr .sin_port = htons ( 8888 ) ;

 // 3. Connect to the server
 if ( connect ( connectfd , ( struct sockaddr * ) ( & server_addr ) , sizeof ( server_addr ) ) < 0 )
 {
 fprintf ( stderr , "connect error:%s\n" , strerror ( errno ) ) ;
 return - 1 ;
 }

 printf ( "connect success\n" ) ;


 char sendline [ 64 ] = "hello, i am xiaolin" ;

 // 4. Send data
 int ret = send ( connectfd , sendline , strlen ( sendline ) , 0 ) ;
 if ( ret != strlen ( sendline ) ) {
 fprintf ( stderr , "send data error:%s\n" , strerror ( errno ) ) ;
 return - 1 ;
 }

 printf ( "already send %d bytes\n" , ret ) ;

 sleep ( 1 ) ;

 // 5. Close the connection
 close ( connectfd ) ;
 return 0 ;
 }

Compile the server and client code:

First enable the server:

Then use the tcpdump tool to start capturing packets. The command is as follows:

 tcpdump - i lo tcp and port 8888 - s0 - w / home / tcp_close .pcap

Then enable the client. You can see that after successfully connecting to the server, it exits after sending the data.

At this time, the server output:

Next, let's take a look at the results of the packet capture.

It can be seen that TCP waved 3 times. Therefore, the following conclusion is correct.

Conclusion: When the passive closing party (the server in the above figure) has "no data to send" and "the TCP delayed confirmation mechanism is turned on (enabled by default)" during the TCP wave process, the second and third waves will be combined for transmission, resulting in three waves.

Experiment 2

Let's do another experiment to see if the TCP delayed acknowledgment mechanism is turned off. Will there be four handshakes? The client code remains unchanged, and the server code needs to add something. In the above server code, the code to turn on the TCP_QUICKACK mechanism is added as follows:

After compiling the server code, we started running the server and client codes and used tcpdump to capture packets. The result of the packet capture is as follows, and we can see that there are four hand waves.

Therefore, when the passive closing party (the server in the above figure) has "no data to send" during the TCP wave process and "turns off the TCP delayed confirmation mechanism", there will be four waves.

Why should the code for setting TCP_QUICKACK be placed after read returns 0?

I also found out after many experiments that setting TCP_QUICKACK before bind is not effective. Only when read returns 0, setting TCP_QUICKACK will cause four waves. I checked the information online and found that setting TCP_QUICKACK is not permanent, so if you want to reply ACK immediately every time you read data, you have to reset TCP_QUICKACK after each data read. The purpose of my experiment here is to reply ACK message immediately after receiving the FIN message (first wave) from the client, so TCP_QUICKACK is set when read returns 0. Of course, in actual applications, no one will set TCP_QUICKACK at this position, because the operating system helps us optimize the four waves to three waves through the TCP delayed confirmation mechanism, which is a good thing.

Summarize

When the passive closing party has no data to send during the TCP handshake process, and TCP_QUICKACK is not enabled (the default is not enabled, which means that the TCP delayed confirmation mechanism is in use), then the second and third handshakes will be combined for transmission, resulting in three handshakes. Therefore, the three handshakes are caused by the TCP delayed confirmation mechanism.