I found a mistake in the book!

I discussed some TCP issues with my friends over the weekend. When I was looking up the book "Linux Server High Performance Programming", I found this sentence in the book:

The book says that a connection in the TIME_WAIT state will return a RST message after receiving a SYN message of the same four-tuple, and the other party will disconnect after receiving it.

The author of the book only mentioned this sentence without providing any source code or packet capture evidence.

At first, I thought this logic made sense, but when I looked at the TCP source code myself, I found that it was not the case.

So, today we will discuss this question, "During the normal TCP handshake process, what happens when a connection in the TIME_WAIT state receives a SYN of the same four-tuple?"

The problem phenomenon is as follows, the left side is the server, and the right side is the client:

Conclusion first

Before analyzing the TCP source code with you, let me first tell you the conclusion.

The key to this problem is to see whether the "sequence number and timestamp" of SYN is legal, because after the connection in TIME_WAIT state receives SYN, it will determine whether the "sequence number and timestamp" of SYN is legal, and then perform different processing based on different judgment results.

Let me first explain to you what is a "legal" SYN?

• Legal SYN: The "sequence number" of the client's SYN is larger than the "next sequence number expected to be received" on the server, and the "timestamp" of the SYN is larger than the "timestamp of the last message received" on the server.
• Illegal SYN: The "sequence number" of the client's SYN is smaller than the "next sequence number expected to be received" of the server, or the "timestamp" of the SYN is smaller than the "timestamp of the last message received" of the server.

The above SYN legal judgment is based on the scenario where both parties have enabled the TCP timestamp mechanism. If neither party has enabled the TCP timestamp mechanism, the SYN legal judgment is as follows:

• Legitimate SYN: The "sequence number" of the client's SYN is larger than the "next sequence number expected to be received" on the server.
• Illegal SYN: The "sequence number" of the client's SYN is smaller than the "next sequence number expected to be received" on the server.

Received a legitimate SYN

If a connection in the TIME_WAIT state receives a "legitimate SYN", it will reuse this four-tuple connection, skip 2MSL and change to the SYN_RECV state, and then proceed with the connection establishment process.

Take the following figure as an example. Both parties have enabled the TCP timestamp mechanism. TSval is the timestamp when the message is sent:

In the figure above, when the third FIN message is received, the TSval (21) of the message is recorded and saved in the ts_recent variable. Then the next expected sequence number is calculated. In this example, the next expected sequence number is 301, which is saved in the rcv_nxt variable.

When a connection in the TIME_WAIT state receives a SYN, because the SYN's seq (400) is greater than rcv_nxt (301), and the SYN's TSval (30) is greater than ts_recent (21), it is a "legal SYN", so the four-tuple connection will be reused, skipping 2MSL and transitioning to the SYN_RECV state, and then the connection establishment process can be carried out.

Received an illegal SYN

If a connection in the TIME_WAIT state receives an "illegal SYN", it will reply with a fourth ACK message. After receiving it, the client finds that it is not the confirmation number (ack num) it expected, so it returns a RST message to the server.

Take the following figure as an example. Both parties have enabled the TCP timestamp mechanism. TSval is the timestamp when the message is sent:

In the figure above, when the third FIN message is received, the TSval (21) of the message is recorded and saved in the ts_recent variable. Then the next expected sequence number is calculated. In this example, the next expected sequence number is 301, which is saved in the rcv_nxt variable.

After the connection in TIME_WAIT state receives SYN, because the seq (200) of SYN is less than rcv_nxt (301), it is an "illegal SYN", and it will reply with an ACK message that is the same as the fourth wave. After receiving it, the client finds that it is not the confirmation number it expected, so it returns a RST message to the server.

If the client does not receive SYN + ACK after waiting for a while, it will time out and retransmit the SYN message. When the number of retransmissions reaches the maximum value, the connection will be disconnected.

PS: Here is a question, will the connection in TIME_WAIT state be disconnected if RST is received?

Source code analysis

The following source code analysis is based on the Linux 4.2 kernel code.

After receiving a TCP message, the Linux kernel executes the tcp_v4_rcv function. The main code related to the TIME_WAIT state in this function is as follows:

 int tcp_v4_rcv ( struct sk_buff * skb )
 {
 struct sock * sk ;
 ...
 //After receiving the message, this function will be called to find the corresponding sock
 sk = __inet_lookup_skb ( & tcp_hashinfo , skb , __tcp_hdrlen ( th ), th - > source ,
 th - > dest , sdif , & refcounted );
 if ( ! sk )
 goto no_tcp_socket ;

 process :
 //If the connection status is time_wait, it will jump to do_time_wait
 if ( sk - > sk_state == TCP_TIME_WAIT )
 goto do_time_wait ;

 ...

 do_time_wait :
 ...
 //The tcp_timewait_state_process function processes the message received in the time_wait state
 switch ( tcp_timewait_state_process ( inet_twsk ( sk ), skb , th )) {
 // If it is TCP_TW_SYN, then allow this SYN to reestablish the connection
 // Allow TIM_WAIT state to transition to SYN_RECV
 case TCP_TW_SYN : {
 struct sock * sk2 = inet_lookup_listener ( ... .);
 if ( sk2 ) {
 ... .
 goto process ;
 }
 }
 // If it is TCP_TW_ACK, then return the ACK in memory
 case TCP_TW_ACK :
 tcp_v4_timewait_ack ( sk , skb );
 break ;
 // If it is TCP_TW_RST, send RESET packet directly
 case TCP_TW_RST :
 tcp_v4_send_reset ( sk , skb );
 inet_twsk_deschedule_put ( inet_twsk ( sk ));
 goto discard_it ;
 // If it is TCP_TW_SUCCESS, discard the packet directly without any response
 case TCP_TW_SUCCESS :;
 }
 goto discard_it ;
 }

The code process:

• After receiving the message, the __inet_lookup_skb() function will be called to find the corresponding sock structure;
• If the connection status is TIME_WAIT, it will jump to do_time_wait processing;
• The tcp_timewait_state_process() function processes the received message and performs corresponding processing based on the return value.

First of all, if the received SYN is legal, the tcp_timewait_state_process() function will return TCP_TW_SYN and reuse the connection. If the received SYN is illegal, the tcp_timewait_state_process() function will return TCP_TW_ACK and send the last sent ACK.

Next, let’s see how the tcp_timewait_state_process() function determines the SYN packet.

 enum tcp_tw_status
 tcp_timewait_state_process ( struct inet_timewait_sock * tw , struct sk_buff * skb ,
 const struct tcphdr * th )
 {
 ...
 //paws_reject is false, indicating that no timestamp wrapping occurs
 //paws_reject is true, indicating that timestamp wrapping has occurred
 bool paws_reject = false ;

 tmp_opt . saw_tstamp = 0 ;
 //The TCP header has options and the old connection has the timestamp option enabled
 if ( th - > doff > ( sizeof ( * th ) >> 2 ) && tcptw - > tw_ts_recent_stamp ) {
 // Parsing options
 tcp_parse_options ( twsk_net ( tw ), skb , & tmp_opt , 0 , NULL );

 if ( tmp_opt . saw_tstamp ) {
 ...
 // Check whether the timestamp of the received message has been wrapped
 paws_reject = tcp_paws_reject ( & tmp_opt , th - > rst );
 }
 }

 ... .

 //It is a SYN packet, no RST, no ACK, the timestamp does not wrap around, and the sequence number does not wrap around,
 if ( th - > syn && ! th - > rst && ! th - > ack && ! paws_reject &&
 ( after ( TCP_SKB_CB ( skb ) - > seq , tcptw - > tw_rcv_nxt ) ||
 ( tmp_opt.saw_tstamp && // New connection starts with timestamp
 ( s32 )( tcptw - > tw_ts_recent - tmp_opt . rcv_tsval ) < 0 ))) { // Timestamp does not wrap around
 // Initialize the serial number
 u32 isn = tcptw - > tw_snd_nxt + 65535 + 2 ;
 if ( isn == 0 )
 isn ++ ;
 TCP_SKB_CB ( skb ) - > tcp_tw_isn = isn ;
 return TCP_TW_SYN ; //Allow reuse of TIME_WAIT quad to reestablish connection
 }


 if ( ! th - > rst ) {
 // If the timestamp wraps around, or the message contains ack, extend the duration of the TIMEWAIT state again
 if ( paws_reject || th - > ack )
 inet_twsk_schedule ( tw , & tcp_death_row , TCP_TIMEWAIT_LEN ,
 TCP_TIMEWAIT_LEN );

 // Return TCP_TW_ACK, send the last ACK
 return TCP_TW_ACK ;
 }
 inet_twsk_put ( tw );
 return TCP_TW_SUCCESS ;
 }

If both parties have enabled the TCP timestamp mechanism, the tcp_paws_reject() function will be used to determine whether the timestamp has wrapped, that is, whether the "timestamp of the currently received message" is greater than the "timestamp of the last received message":

• If it is greater, it means that no timestamp wrapping has occurred and the function returns false.
• If it is less than, it means that the timestamp wraps around and the function returns true.

From the source code, we can see that after receiving a SYN packet, if the timestamp of the SYN packet does not wrap around, that is, the timestamp is incremented, and the sequence number of the SYN packet does not wrap around, that is, the sequence number of the SYN is "greater than" the sequence number expected to be received next time, a sequence number will be initialized, and then TCP_TW_SYN will be returned, and then the connection will be reused, which means that 2MSL will be skipped and the state will be changed to SYN_RECV, and then the connection establishment process can be carried out.

If neither party has enabled the TCP timestamp mechanism, it is only necessary to determine whether the sequence number of the SYN packet has been wrapped. If the sequence number of the SYN packet is greater than the sequence number expected to be received next time, 2MSL can be skipped and the connection can be reused.

If the SYN packet is illegal, TCP_TW_ACK will be returned, and then the same ACK as the last time will be sent to the other party.

In the TIME_WAIT state, will the connection be terminated if a RST is received?

I left a question earlier. Will a connection in TIME_WAIT state be disconnected if it receives RST?

Whether the connection will be disconnected or not depends on the kernel parameter net.ipv4.tcp_rfc1337 (the default value is 0):

• If this parameter is set to 0, receiving a RST message will end the TIME_WAIT state early and release the connection.
• If this parameter is set to 1, RST packets will be discarded.

The source code is processed as follows:

 enum tcp_tw_status
 tcp_timewait_state_process ( struct inet_timewait_sock * tw , struct sk_buff * skb ,
 const struct tcphdr * th )
 {
 ... .
 //The timestamp of the rst message does not wrap around
 if ( ! paws_reject &&
 ( TCP_SKB_CB ( skb ) - > seq == tcptw - > tw_rcv_nxt &&
 ( TCP_SKB_CB ( skb ) - > seq == TCP_SKB_CB ( skb ) - > end_seq || th - > rst ))) {

 //Process rst message
 if ( th - > rst ) {
 //If this option is not enabled, tw will be recycled immediately when RST is received, but this is risky.
 if ( twsk_net ( tw ) - > ipv4 . sysctl_tcp_rfc1337 == 0 ) {
 kill :
 //Delete the tw timer and release tw
 inet_twsk_deschedule_put ( tw );
 return TCP_TW_SUCCESS ;
 }
 } else {
 //Re-extend the duration of the TIMEWAIT state
 inet_twsk_reschedule ( tw , TCP_TIMEWAIT_LEN );
 }

 ...
 return TCP_TW_SUCCESS ;
 }
 }

The connection is released when a RST message is received in the TIME_WAIT state, which is equivalent to skipping the 2MSL time. However, this is still risky.

The parameter sysctl_tcp_rfc1337 is proposed in the rfc 1337 document. Its purpose is to avoid skipping the 2MSL time due to receiving a RST message in the TIME_WAIT state. The document also gives the potential problems of skipping the 2MSL time.

The TIME_WAIT state lasts for 2MSL for two main purposes:

• Prevent data in historical connections from being incorrectly received by subsequent connections with the same four-tuple;
• Ensure that the party that "passively closes the connection" can be closed correctly;

Although the TIME_WAIT state lasts a bit long and seems unfriendly, it is designed to avoid messy things.

However, the book "UNIX Network Programming" says: TIME_WAIT is our friend, it is helpful to us, don't try to avoid this state, but should figure it out.

Therefore, I personally think it is safer to set net.ipv4.tcp_rfc1337 to 1.

Summarize

During the normal TCP handshake process, what happens when a connection in the TIME_WAIT state receives a SYN of the same four-tuple?

If both parties have enabled the timestamp mechanism:

• If the client's SYN "sequence number" is larger than the server's "expected next received sequence number", and the SYN "timestamp" is larger than the server's "timestamp of the last received message", then the four-tuple connection will be reused, skipping 2MSL and transitioning to the SYN_RECV state, and then the connection establishment process can proceed.
• If the client's SYN "sequence number" is smaller than the server's "expected next sequence number", or the SYN "timestamp" is smaller than the server's "timestamp of the last received message", then the server will reply with a fourth ACK message. After receiving it, the client finds that it is not the confirmation number it expected, so it returns a RST message to the server.

In the TIME_WAIT state, will the connection be terminated if a RST is received?

• If the net.ipv4.tcp_rfc1337 parameter is 0, the TIME_WAIT state ends early and the connection is released.
• If the net.ipv4.tcp_rfc1337 parameter is 1, the RST packet will be discarded.