TCP SYN Queue and Accept Queue

First we must understand that a TCP socket in the "LISTENING" state has two independent queues:

• SYN Queue
• Accept Queue

These two terms are sometimes also called "reqsk_queue", "ACK backlog", "listen backlog", or even "TCP backlog", but we will use the above two terms in this article to avoid confusion.

SYN Queue

The SYN queue stores the connections that received SYN packets (corresponding to the kernel code structure: struct inet_request_sock). Its responsibility is to reply to SYN+ACK packets and retransmit when no ACK packet is received until the timeout. Under Linux, the number of retransmissions is:

 $ sysctl net.ipv4.tcp_synack_retries
 net.ipv4.tcp_synack_retries = 5  

The documentation describes tcp_synack_retries as follows:

 tcp_synack_retries - int 
 
 The number of times to retransmit SYNACKs for a passive TCP connection. This value cannot exceed 255.
 The default value is 5. If the initial RTO is 1 second, the corresponding last retransmission is 31 seconds.
 The corresponding last timeout is 63 seconds later.

After sending SYN+ACK, the SYN queue waits for the ACK packet sent from the client (that is, the last packet of the three-way handshake). When the ACK packet is received, the corresponding SYN queue is first found, and then the relevant data in the corresponding SYN queue is checked to see if it matches. If it matches, the kernel removes the data related to the connection from the SYN queue, creates a complete connection (corresponding to the structure of the kernel code: struct inet_sock), and adds this connection to the Accept queue.

Accept Queue

The Accept queue stores established connections, that is, connections waiting to be taken away by the upper-layer application. When the process calls accept(), the socket is taken out of the queue and passed to the upper-layer application.

This is a simple description of how Linux handles SYN packets. By the way, when the socket has TCP_DEFER_ACCEPT and TCP_FASTOPEN enabled, the working method will be slightly different, which will not be introduced in this article.

Queue size limit

The application sets the maximum size of the SYN queue and the Accept queue by calling the listen(2) system call and passing in the backlog parameter. For example, the following example sets the maximum size of both the SYN queue and the Accept queue to 1024:

 listen(sfd, 1024)

Note that in kernels prior to 4.3, the size of the SYN queue is calculated in another way.

The maximum size of the SYN queue was previously configured using net.ipv4.tcp_max_syn_backlog, but this is no longer used. Now net.core.somaxconn is used to represent the maximum size of both the SYN queue and the Accept queue. On our servers, we set it to 16k:

 $ sysctl net.core.somaxconn
 net.core.somaxconn = 16384  

What is the appropriate queue size?

After knowing the above information, you may ask, what is the appropriate queue size?

The answer is: it depends. For most TCP services, this is not too important. For example, before Go version 1.11, there was no method to set the queue size.

However, there are some legitimate reasons to increase the queue size:

• When the rate of connection establishment requests is really high, the SYN queue may need to be set larger even for a high-performance service.
• The size of the SYN queue, in other words, is the number of connections waiting for ACK packets. That is, the longer the average round-trip time with the client, the more connections will accumulate in the SYN queue. For scenarios where most clients are far away from the server, for example, the round-trip time is more than a few hundred milliseconds, the queue size can be set larger.
• If the TCP_DEFER_ACCEPT option is turned on, it will cause the socket to stay in the SYN-RECV state for a longer time, which increases the time it stays in the SYN queue.

However, setting the backlog too large can also have adverse effects:

• Each slot in the SYN queue takes up some memory. When encountering a SYN Flood attack, there is no need to waste resources for these attacking packets. In the 4.14 kernel, each inet_request_sock structure in the SYN queue will take up 256 bytes of memory.

In Linux, if you want to check the current status of the SYN queue, you can use the ss command to query the socket in the SYN-RECV state. For example, the following execution result indicates that there are currently 119 elements in the SYN queue of port 80 and 78 in the SYN queue of port 443.

 $ ss -n state syn-recv sport = :80 | wc -l
 119
 $ ss -n state syn-recv sport = :443 | wc -l
 78

You can also observe this data through our SystemTap script:

 resq.stp

What if the program doesn't call accept() fast enough?

What happens if the program doesn't call accept() fast enough?

 TcpExtListenOverflows/LINUX_MIB_LISTENOVERFLOWS
 TcpExtListenDrops/LINUX_MIB_LISTENDROPS

When this happens, we can only hope that the program's processing performance will return to normal later and the client will resend the packets that were discarded by the server.

This behavior of the kernel is acceptable for most services. By the way, this behavior can be modified by adjusting the global parameter net.ipv4.tcp_abort_on_overflow, but it is best not to change this parameter.

You can observe the status of the Accept queue overflow by viewing the count of nstat:

 $ nstat -az TcpExtListenDrops
 TcpExtListenDrops 49199 0.0

But this is a global count. It is not intuitive to observe. For example, sometimes we observe that it is increasing, but all service programs seem to be normal. In this case, we can use the ss command to observe the Accept queue size of a single listening port:

 $ ss -plnt sport = :6443|cat
 State Recv-Q Send-Q Local Address:Port Peer Address:Port
 LISTEN 0 1024 *:6443 *:*

The Recv-Q column shows the number of sockets in the Accept queue, and the Send-Q column shows the maximum size of the queue. In the above example, we found that there were no sockets that were not accepted by the program, but we still found that the ListenDrops count was increasing.

This is because our program is only temporarily stuck and does not process new connections, rather than permanently not processing them. After a while, the program returns to normal. In this case, it is difficult to observe this phenomenon using the ss command, so we wrote a SystemTap script that hooks into the kernel and prints out the discarded SYN packets:

 $ sudo stap -v acceptq.stp
 time (us) acceptq qmax local addr remote_addr
 1495634198449075 1025 1024 0.0.0.0:6443 10.0.1.92:28585
 1495634198449253 1025 1024 0.0.0.0:6443 10.0.1.92:50500
 1495634198450062 1025 1024 0.0.0.0:6443 10.0.1.92:65434
 ...

Through the above operations, we can observe which SYN packets are affected by ListenDrops. Thus, we can also know which programs are losing connections.