Manually sending packets only handshakes twice, I discovered the secret of TCP

[[399328]]

Planet Questions

There are at least 9,000 articles written on the topic of TCP three-way handshake.

I wrote this article today because a friend asked this question in my Knowledge Planet:

To summarize, three small questions:

After the client sends the third handshake, does it send the data directly regardless of whether the server receives it or not?

Can the third handshake of TCP carry data?

If for various reasons, the server does not receive the third handshake packet sent by the client, how does the server handle the subsequent data sent by the client?

My Answer

Here are my answers:

First, let's answer the player's initial question: After the client sends the third handshake, does it send the data directly regardless of whether the server has received it?

You can guess theoretically that if the answer to the above question is no, it means that the client still needs to confirm that the server has received its third handshake packet before sending data. How to confirm it? Does the server have to reply to itself: I have received your third handshake packet, you can send data.

But if this is the case, does it become a four-way handshake instead of a three-way handshake?

So thinking about it the other way around, the answer to this question is yes, that is: after the client sends the third handshake packet, it no longer needs confirmation from the server and can send data immediately.

The following is a connection establishment diagram from "TCP/IP Protocol Detailed Explanation" (Volume 1). You can see that on the client side, after sending the third handshake packet, the state changes to ESTABLISH state, and data transmission begins without waiting for server confirmation.

Theory alone is not enough, let's capture the packets and take a look. Below is a handshake sequence diagram of TCP connection establishment captured by the packet capture software. You can also see that after the third handshake packet is sent, the client on the left immediately sends out a formal data transmission: an HTTP request packet.

So the answer to this question is clear.

Let's look at the second question: When the client sends the third handshake packet, will it transmit the data together?

In fact, you can see from the two figures above that the TCP three-way handshake does not carry valid application layer data, and data transmission only starts after the handshake is completed. But if we have to ask: when the client sends the third handshake data packet, can it carry some data along with it?

Regarding this question, the most authoritative answer is to look at the RFC standard document. The specification of the TCP standard protocol is recorded in RFC793, numbered 793. The link is as follows:

https://www.rfc-editor.org/rfc/rfc793.html

The document is a bit long and is in English, so it might be a bit difficult to read.

In the event handling section, you will find the following text:

The general idea is: if our synchronization packet SYN has been confirmed, the connection state is changed to ESTABLISHED, and then the third handshake packet sent may contain data (if there is already data waiting in line for transmission)

This makes it very clear: in the TCP standard protocol specification, the third handshake packet allows data transmission!

The last question: If for various reasons, the server does not receive the third handshake packet sent by the client, how does the server handle the subsequent data sent by the client?

I'll keep you in suspense here and then read on.

The following is the essence of this article:

Experimental demonstration

The three-way handshake of TCP to establish a connection is automatically completed by the operating system kernel protocol stack. As an underlying service, this process is transparent to the application. When we develop applications, we only need to use the application layer programming interface, such as the socket interface.

Therefore, most people's concept of TCP three-way handshake is still based on books, blogs, and public account articles. Today, we will send TCP data packets ourselves to implement the three-way handshake!

Send the package yourself to verify our conclusions above!

The tool used is the artifact mentioned in a previous article: scapy.

In order to view the data conveniently, I found a website without HTTPS, pinged its domain name, got the IP address, shook hands with it and sent a GET request packet.

 from scapy.all import *
 def tcp_test(ip, port, data): 
 
 # First handshake, send SYN packet
 # Request port and initial sequence number are randomly generated
 # Use sr1 instead of send, because sr1 will receive the returned content
    ans = sr1(IP(dst=ip) / TCP(dport=port, sport=RandShort(), seq=RandInt(), flags= 'S' ), verbose= False ) 
 
 # Assume that the other party has sent the second handshake packet: ACK+SYN 
 
 # The target port that the other party replies to is the request port used by our side (the one randomly generated above)
 sport = ans[TCP].dport
 s_seq = ans[TCP].ack
 d_seq = ans[TCP].seq + 1 
 
 # The third handshake, sending ACK confirmation packet
    send(IP(dst=ip) / TCP(dport=port, sport=sport, ack=d_seq, seq=s_seq, flags= 'A' ), verbose= False ) 
 
 # Make a GET request
    send(IP(dst=ip)/TCP(dport=port, sport=sport, seq=s_seq, ack=d_seq, flags=24)/data, verbose= False ) 
   
   
 if __name__ == '__main__' :
 data = 'GET / HTTP/1.1\n'  
    data += 'Host: www.chengtu.com\n'  
    data += 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\n'  
    data += 'Accept: text/html'  
 data += '\r\n\r\n'   
 
    tcp_test( "150.138.151.65" , 80, data)

Execute the above code to capture the packet:

As you can see, the three-way handshake was successfully completed and the server returned data, proving that manual programming for handshake is feasible.

The following is a question raised by a fellow player in the demonstration planet: Can the third handshake packet carry data?

Let's try it and see:

 from scapy.all import *
 def tcp_test_2(ip, port, data): 
 
 # First handshake, send SYN packet
 # Request port and initial sequence number are randomly generated
 # Use sr1 instead of send, because sr1 will receive the returned content
    ans = sr1(IP(dst=ip) / TCP(dport=port, sport=RandShort(), seq=RandInt(), flags= 'S' ), verbose= False ) 
 
 # Assume that the other party has sent the second handshake packet: ACK+SYN 
 
 # The target port that the other party replies to is the request port used by our side (the one randomly generated above)
 sport = ans[TCP].dport
 s_seq = ans[TCP].ack
 d_seq = ans[TCP].seq + 1 
 
 # The third handshake, send an ACK confirmation packet, and bring the data along
    send(IP(dst=ip) / TCP(dport=port, sport=sport, ack=d_seq, seq=s_seq, flags= 'A' )/data, verbose= False ) 
 
 
 if __name__ == '__main__' : 
 
 data = 'GET / HTTP/1.1\n'  
    data += 'Host: www.chengtu.com\n'  
    data += 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\n'  
    data += 'Accept: text/html'  
 data += '\r\n\r\n'   
 
    tcp_test_2( "150.138.151.65" , 80, data)

You see, in the third handshake, my GET request was passed, and the TCP protocol can still work normally!

This is the situation of Linux. I tried it on our university's website. Because the school website does not use HTTPS (which is outrageous) and is made with ASP.NET technology stack (don't ask me how I know), the server is Windows and it can still work normally, which means that the Windows protocol stack also supports this operation.

Next, let's verify another question: What will happen if the server does not receive the third handshake packet and sends data directly?

How to verify? It's very simple. Just comment out the line that sends the third handshake. Instead of sending the third handshake, just send a GET request directly:

 from scapy.all import *
 def tcp_test(ip, port, data): 
 
 # First handshake, send SYN packet
 # Request port and initial sequence number are randomly generated
 # Use sr1 instead of send, because sr1 will receive the returned content
    ans = sr1(IP(dst=ip) / TCP(dport=port, sport=RandShort(), seq=RandInt(), flags= 'S' ), verbose= False ) 
 
 # Assume that the other party has sent the second handshake packet: ACK+SYN 
 
 # The target port that the other party replies to is the request port used by our side (the one randomly generated above)
 sport = ans[TCP].dport
 s_seq = ans[TCP].ack
 d_seq = ans[TCP].seq + 1 
 
 # The third handshake, sending ACK confirmation packet
    # send(IP(dst=ip) / TCP(dport=port, sport=sport, ack=d_seq, seq=s_seq, flags= 'A' ), verbose= False ) 
 
 # Make a GET request
    send(IP(dst=ip)/TCP(dport=port, sport=sport, seq=s_seq, ack=d_seq, flags= 'A' )/data, verbose= False ) 
   
   
 if __name__ == '__main__' :
 data = 'GET / HTTP/1.1\n'  
    data += 'Host: www.chengtu.com\n'  
    data += 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\n'  
    data += 'Accept: text/html'  
 data += '\r\n\r\n'   
 
    tcp_test( "150.138.151.65" , 80, data)

It turns out that it still works normally! After some analysis, I found that this method is actually equivalent to the above situation: the data is directly included in the third handshake packet.

Although the line about the third handshake is commented out here, the ACK flag is set in the GET request packet sent directly, so the server regards this GET packet as the third handshake.

So the conclusion is: if the server does not receive the third handshake packet, it will send the data directly, and the server will treat the packet carrying the application data as the third handshake (provided that this packet carries an ACK mark).

In addition to my answer above, this golfer commented and added a question:

In fact, seeing this, the answer to this question must have been clear in mind, but let's still simulate it experimentally: first send a request packet with data, and then send a third handshake packet, and see what happens?

As can be seen from the figure, the request packet with data sent directly is regarded as the third handshake packet, while the nominal third handshake packet sent later, that is, the black line in the figure, is regarded as a repeated invalid packet and is ignored, without affecting the communication.

The above is my complete answer to this golfer’s question.

This article is reprinted from the WeChat public account "Programming Technology Universe", which can be followed through the following QR code. To reprint this article, please contact the Programming Technology Universe public account.