It feels so good to be validated!

Hello everyone, I am Xiaolin.

Previously, I answered a reader’s interview question for Tencent on the official account. The question was as follows:

Regarding this issue, I had no way to verify my conclusion using experiments, so the conclusion was based on studying the TCP source code.

But, just yesterday!

A reader captured a packet capture diagram similar to this interview scenario at work. I took a look and found that the phenomenon was consistent with the conclusion I drew from my previous study of the TCP source code.

It’s such a great feeling to be validated!

I think this case is quite interesting because it illustrates well that the TCP transport protocol is received in order.

So, let’s first review this problem on Tencent’s side, and then look at the packet capture diagram that is similar to this problem.

Review Questions

This Tencent online question may be due to a problem in the wording of the reader who asked the question.

Because if the FIN message arrives at the client before the data packet, the FIN message is actually an out-of-order message, and the client's TCP connection will not switch from the FIN_WAIT_2 state to the TIME_WAIT state.

Therefore, we need to pay attention to "how to handle the out-of-order FIN messages received in the FIN_WAIT_2 state, and when does the TCP connection enter the TIME_WAIT state?"

Let me state the conclusion first:

In the FIN_WAIT_2 state, if an out-of-order FIN message is received, it will be added to the "out-of-order queue" in the kernel and will not enter the TIME_WAIT state.

When the data packet delayed by the network is received again, it will determine whether there is data in the out-of-order queue, and then check whether there is available data in the out-of-order queue. If a message with the same sequence number as the current message can be found in the out-of-order queue, it will check whether the message has a FIN flag. If a FIN flag is found, it will enter the TIME_WAIT state.

I also drew a picture, you can understand it with the help of the picture.

God-like packet capture diagram

The following is a packet capture image sent to me by a reader yesterday. The abnormal situation in the image is somewhat similar to the phenomenon of the previous question:

You may wonder why the seq of both parties starts with 0 during TCP handshake? This is because the packet capture graph has been optimized to show relative values ​​instead of real values. Showing relative values ​​is convenient for analysis.

For the convenience of text description, I number the abnormal messages as follows:

In the figure, port number 11710 is the client, and port number 8080 is the server. In addition, number 4 is the http request sent by the client. The packet capture diagram does not show TCP information, so here is the text supplement: number 4 data message seq = 1, ack = 1, len = 27.

Number 6 is the FIN message, that is, the FIN message sent by the server to the client (the first wave), but it is an out-of-order FIN message, because from the ack = 1 in the number 4 message, we know that the client expects the sequence number of the next message to be received to be 1, and the seq = 177 of the currently received FIN message, which is not the message the client expects to receive next time, so it is out of order.

After receiving the out-of-order FIN message, the client does not switch from establish to close_wait state, but puts the out-of-order FIN message into the out-of-order queue in the kernel. Because if it enters the close_wait state at this time, it will send the FIN message immediately (the third wave), and there will be no message number 8 and 9 sent by the client later.

Number 8 is the response message, which is the client's response message to the out-of-order FIN message number 6. You can see that in this response message, seq = 28, ack = 1. Because it is not the next message I expect, the ack in the response message is still 1.

Number 7 is a data message, which is the data message sent by the server to the client. The message has seq = 1, ack = 28, and len = 176. Since seq is 1, it is the message the client expects to receive. After receiving the message, the client returns a response message with number 9. This response message has seq = 28 and ack = 178, where ack = 178 tells the server: "I have received all the messages you sent before seq = 178 (excluding seq = 178). The seq of the message I expect to receive next time is 178".

When the client receives data packet No. 7, it will do one more thing. It will check whether there is available data in the "out-of-order queue". If it can find a message in the out-of-order queue that "maintains the order" of the sequence number of the currently received message, it will move the message in the out-of-order queue to the data queue that can be processed normally. For example, in this case, the data range of message No. 7 seq = 1, len = 176 is 1~176, and the seq of the FIN message in the out-of-order queue is 177. The seq of these two messages happens to maintain the order, so the FIN message will be taken out of the out-of-order queue and processed together, and then the FIN flag is found, so the state will be changed.

Therefore, after the client responds to data message No. 7, it immediately sends a FIN message (the third wave), and then the server responds to the message, and the four waves are completed.

This packet capture diagram is slightly different from the previous Tencent interview question. The difference is:

• Tencent's interview question was about receiving out-of-order FIN messages in FIN_WAIT_2 state (third wave);
• The packet capture diagram shows that an out-of-order FIN message (the first wave) is received in the establish state;

In the above two TCP states, when receiving an out-of-order FIN message, the state will not be changed immediately, but will only be placed in an out-of-order queue by the kernel. When a data packet with a sequence number that matches the sequence number expected by the "receiver" is received, it will check whether there is available data in the "out-of-order queue". If a message with the sequence number of the currently received message can be found in the out-of-order queue, it will check whether the message has a FIN flag. If a FIN flag is found, the state will be changed.

Therefore, we can see from here that the TCP transmission protocol is received in order. If an out-of-order message is received and it is within the receive window range (messages with sequence numbers outside the receive window range will be discarded), it will be cached in the out-of-order queue in the kernel without any other processing. It will be processed together with the messages that can keep the sequence number of the messages in the out-of-order queue.

The TCP layer must ensure that the received byte data is complete and in order, so if a TCP message with a lower sequence number is lost during network transmission, even if a TCP message with a higher sequence number has been received, the application layer cannot read this data from the kernel.

For example, as shown below:

In the figure, the sender sends many packets, each of which has its own sequence number, which you can think of as the TCP sequence number. Packet 3 is lost in the network. Even if packets 4-6 are received by the receiver, the TCP data in the kernel is not continuous, so the receiver's application layer cannot read it from the kernel. Only after packet 3 is retransmitted can the receiver's application layer read the data from the kernel.