What exactly is a socket? Do you want to know?

I believe that everyone was like me when they first started learning socket.

I am confused and have a vague idea of ​​socket.

In this article, I plan to start from a beginner's perspective, so that everyone can understand what a socket is in my eyes , as well as the principles and kernel implementation of a socket.

The concept of socket

The story begins with a plug.

Plugs and sockets

When I plug the plug into the socket, it looks like it connects the two.

Fans are "connected" to the power system

The English word for socket is also called socket.

Coincidentally, we programmers also use something called socket when doing network programming.

In fact, the two are very similar. Through the socket, we can establish a "connection" with a machine. The process of establishing a "connection" is like inserting a socket into a slot.

You may have understood the general concept, but I believe you are still vague about sockets.

Let’s start with the usage scenarios that everyone is most familiar with.

Socket usage scenarios

We want to send data from a process on computer A to a process on computer B.

At this time, we need to choose a way to send the data. If we need to ensure that the data can be sent to the other party, then choose the reliable TCP protocol. If the data is lost, it doesn’t matter. It depends on God’s will. Then choose the unreliable UDP protocol.

For beginners, TCP is undoubtedly the first choice.

What is TCP

At this time, you need to use socket programming.

So the first step is to create a TCP socket, as shown below.

 sock_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

This method will return socket_fd, which is the handle of the socket file, a number, equivalent to the socket's ID number.

After obtaining the socket_fd, the server can execute the bind(), listen(), and accept() methods in sequence, and then wait for the client's connection request.

For the client, after getting the socket_fd, you can execute the connect() method to initiate a request to establish a connection to the server, and the TCP three-way handshake will occur.

Handshake connection establishment process

After the connection is established, the client can execute the send() method to send messages, and the server can execute the recv() method to receive messages. Conversely, the server can also execute the send() method and the client can execute the recv() method.

So far, this is the usage scenario that most of our programmers are most familiar with.

Socket design

Now, we have seen and used sockets, but for most programmers, it is a black box.

Since it is a black box, let's just assume that we forget about sockets and redesign a kernel network transmission function.

From an operational point of view, network transmission is nothing more than sending and receiving data between the sender and the remote end, which corresponds to writing data and reading data.

Read, write, send and receive

But obviously, things are not that simple.

There are two more problems here.

The first is that there may be more than one receiving end and sending end, so we need some information to distinguish them. Everyone must be familiar with this. We can use IP and port. IP is used to locate which computer it is, and port is used to locate which process on this computer.

The second is that there are many differences in the transmission methods between the sender and the receiver. It can be a reliable TCP protocol or an unreliable UDP protocol, and even needs to support the ping command based on the ICMP protocol.

What is sock

Anyone who has written code knows that in order to support these functions, we need to define a data structure to support these functions.

This data structure is called sock.

To solve the first problem above, we can add IP and port fields to sock.

sock adds IP and port fields

As for the second question, we will find that although these protocols are different, they still have some similar functions. For example, some logic when sending and receiving data can be fully reused. According to the idea of ​​object-oriented programming, we can regard different protocols as different object classes (or structures), extract the common parts, and reuse the functions through "inheritance".

Realize network transmission function based on various socks

So, we re-divided the functions and defined some data structures.

Various socks that inherit sock

Sock is the most basic structure, maintaining some send and receive data buffers that may be used by any protocol.

inet_sock refers to sock that uses network transmission function. On the basis of sock, it also adds TTL, port, IP address and other fields related to network transmission. At this point, everyone is confused. Is there anything that is not transmitted through the network? Yes, for example, Unix domain socket is used for communication between local processes, directly reading and writing files without going through the network protocol stack. This is a very useful thing, I will definitely talk about it later (painting cake).

inet_connection_sock is a connection-oriented sock. It adds fields related to connection-oriented protocols based on inet_sock, such as accept queue, packet fragment size, number of retries after handshake failure, etc. Although the connection-oriented protocol we are referring to now refers to TCP, Linux needs to support the expansion of other new connection-oriented protocols in terms of design.

tcp_sock is a sock structure dedicated to the TCP protocol. It adds TCP-specific sliding window and congestion avoidance functions based on inet_connection_sock. Similarly, the UDP protocol also has a dedicated data structure called udp_sock.

OK, now that we have this set of data structures, we connect them to the hardware network card to realize the function of network transmission.

Provides socket layer

As you can imagine, the code here must be very complex. It also operates the network card hardware and requires relatively high operating system permissions. Considering performance and security, it was decided to put it in the operating system kernel.

Since the network transmission function is built into the kernel, what should user-space applications do if they want to use this function?

This is easy to do. In line with the principle of not reinventing the wheel, we abstract this part of the function into simple interfaces. In the future, others only need to call these interfaces to drive the large number of complex data structures we have written to send data.

So the question is, how to expose this function to other programmers so that they can use it more conveniently?

Since sending and receiving data with the remote server process can be abstracted as "reading and writing", operating files can also be abstracted as "reading and writing", and there is a saying that "everything in Linux is a file", then we simply encapsulate the kernel sock into a file. When creating a sock, a file is also created. The file has a handle fd, which is simply an ID number in the file system, through which you can uniquely identify which sock it is.

This file handle fd is actually the sock_fd in sock_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP).

The handle is exposed to the user, and then the user can operate the sock handle like a file handle. When operating the handle in user space, the file system will direct the operation to the kernel sock structure.

Yes, operating this special file is equivalent to operating the corresponding sock in the kernel.

Find sock through file

After we have the sock_fd handle, we need to provide some interface methods to make it easier for users to implement specific network programming functions. We listed these interfaces and found that we need send(), recv(), bind(), listen(), and connect(). At this point, our kernel network transmission function is designed.

Do they look familiar now? The above interface methods are actually the interfaces provided by socket.

So, socket is actually a code library or interface layer, which is between the kernel and the application, providing some highly encapsulated interfaces for us to use the kernel network transmission function.

Realize network transmission function based on sock

At this point, we should understand that although the application code we usually write uses sockets to implement the function of sending and receiving data packets, it is actually not the application that actually performs the network communication function, but the Linux kernel. It is equivalent to the application outsourcing this part of the network transmission work to the Linux kernel through the interface provided by the socket.

Does this sound like the service architecture with front-end and back-end separation that we are most familiar with? Although this is not very rigorous, it seems that Linux is divided into two services: application and kernel. The kernel is like the back-end, exposing many API interfaces, one of which is the socket's ​send()​ and ​recv()​ methods. The application is like the front-end, responsible for calling the interface provided by the kernel to implement the desired function.

The process calls the kernel function through the socket

Seeing this, I’m worried that everyone may be a little confused, so let me make a small summary.

In the kernel space of the operating system, the structure that implements the network transmission function is sock. Based on different protocols and application scenarios, it will be generalized into various types of xx_sock, which, combined with hardware, jointly implement the network transmission function. In order to expose this part of the function to the application in the user space, the socket layer was introduced, and the sock was embedded in the framework of the file system. The sock became a special file, and the user can use the file handle in the user space, that is, socket_fd, to operate the network transmission capability of the kernel sock.

This socket_fd is an int type number. Now looking back at the Chinese translation of socket, I understand it as a set of numbers used for connection, which seems very reasonable.

Network layering and network transmission based on sock

How socket realizes network communication

The above briefly mentions how to implement network communication functions.

Now let's talk.

The structure of this sock is actually very complex. Let's take the most commonly used TCP protocol as an example to briefly understand how it implements network transmission functions.

I divide it into two stages: establishing connection and data transmission.

Establishing a connection

For TCP, to transmit data, you must first establish a connection between the client and the server.

On the client side, when the code executes the connect(sockfd, "ip:port") method provided by the socket, it will find the corresponding file through the sockfd handle, and then point to the kernel's sock structure based on the information in the file. The three-way handshake is actively initiated through this sock structure.

TCP three-way handshake

A connection that has not completed three handshakes on the server side is called a semi-connection, and a connection that has completed three handshakes is called a full connection. They are stored in a semi-connection queue and a full-connection queue, respectively. These two queues are created when you execute the listen() method. When the server executes the accept() method, a full connection is taken out of the full-connection queue.

Semi-connected queues and fully connected queues

At this point, the connection is ready, and you can start transferring data.

Although they are both called queues, the semi-connected queue is actually a hash table, while the fully-connected queue is actually a linked list.

So the question is, why is the semi-connected queue designed as a hash table while the fully connected queue is a linked list? This has been mentioned in my previous article "Can a TCP connection be established without accept?", so I will not repeat it here.

Data Transfer

In order to realize the functions of sending and receiving data, the sock structure has a send buffer and a receive buffer. Although it is called a buffer, it is actually a linked list with data ready to be sent or received hanging on it.

When the application executes the send() method to send data, it will also find the corresponding file through the sock_fd handle, find the send buffer in the sock structure according to the sock structure pointed to by the file, put the data into the send buffer, and then end the process. The kernel decides when to send the data based on its mood.

The process of receiving data is similar. When data is sent to the Linux kernel, it is not immediately given to the application, but is first placed in the receiving buffer. The data lies quietly, humbly waiting for the application to execute the recv() method to get it. Just like my article, it lies in your tweet list, humbly waiting for a like, follow, and retweet. Understand?

sock send and receive buffers

The IP and port are not actually under sock, but under inet_sock. The above drawing is just for simplification. . .

So the question is, sending data is initiated by the application, and everyone has no problem with this.

What about receiving data? When data is sent from a remote end, how do we notify and give it to the application?

This requires the use of a waiting queue.

Waiting queue in sock

When your application process executes the recv() method to try to obtain (in a blocking scenario) the data in the receive buffer.

• If there is data, that's great, just take it. There is no doubt about that.
• But if there is no data, it will register its own process information in the waiting queue used by this sock, and then the process will sleep. If data is sent from the remote end at this time, when the data enters the receiving buffer, the kernel will take out the process in the waiting queue of the sock and wake up the process to get the data.

When there is no data in recv, the process enters the waiting queue

Sometimes, you will see multiple processes listening to the same socket_fd through fork. In the kernel, they are all the same sock. After multiple processes execute listen(), they are all waiting for a connection to come in, so they will register their own process information in the waiting queue of the kernel sock corresponding to this socket_fd. If a connection really comes at this time, which process in the waiting queue should be awakened to receive the connection? The answer to this question is quite interesting.

• Before Linux 2.6, all processes in the waiting queue will be awakened. But in the end, only one process will handle the connection request, and the other processes will go back to sleep. These awakened processes have nothing to do and can only go back to sleep, which will consume certain resources. It's like you are on the street in Guangdong, and you want to ask for directions. You call out "handsome guy", and dozens of people turn around at the same time, but you actually only need one of them to tell you how to go. In the computer field, this kind of scene where you accidentally startle these handsome guys is called the shocking herd effect.
• After Linux 2.6, only one of the processes in the waiting queue will be woken up. Yes, the thundering herd problem of socket monitoring has been fixed.

Thundering Herd Effect

Seeing this, the problem arises again.

When the server listens, how to distinguish multiple clients when so much data is sent to one socket?

Taking TCP as an example, after the server executes the listen method, it will wait for the client to send data. The data packet sent by the client will have the source IP address and port, as well as the destination IP address and port. These four elements form a four-tuple, which can be used to uniquely identify a client.

Actually, it is not rigorous to say it is a quaternary, because there is a lot of other information in the process, it can also be said to be a quintuple. . . But it is enough to understand it roughly, so let's leave it at that. . .

Quadruple

The server creates a new kernel sock and generates a hash key using the four-tuple, putting it into a hash table.

Quadruple maps to hash keys

Next time a message comes in, the server can generate a hash key using the four-tuple in the message and retrieve the corresponding sock from the hash table. So the server distinguishes multiple clients through the four-tuple.

Multiple hash_keys correspond to multiple clients

How does sock implement "inheritance"

Finally, one question remains.

Everyone knows that the Linux kernel is implemented in C language, and C language has no classes and no inheritance features. How does it achieve the effect of "inheritance"?

In C language, the memory in the structure is continuous. Put the "parent class" to be inherited at the first position of the structure, as shown below.

 struct tcp_sock {
 /* inet_connection_sock has to be the first member of tcp_sock */
 struct inet_connection_sock inet_conn ;
 // Other fields
 }

 struct inet_connection_sock {
 /* inet_sock has to be the first member! */
 struct inet_sock icsk_inet ;
 // Other fields
 }

Then we can forcibly intercept the memory by the length of the structure name, so that we can convert the structure and achieve an effect similar to "inheritance".

 // sock converted to tcp_sock
 static inline struct tcp_sock * tcp_sk ( const struct sock * sk )
 {
 return ( struct tcp_sock * ) sk ;
 } 

Memory layout

Summarize

• Socket is a Chinese socket. I understand it as a set of numbers used for connection . It may not be accurate. Comments are welcome.
• sock is in the kernel, socket_fd is in user space, and the socket layer is between the kernel and user space.
• In the kernel space of the operating system, the structure that implements the network transmission function is sock. Based on different protocols and application scenarios, it will be generalized into various types of xx_sock, which, combined with hardware, jointly implement the network transmission function. In order to expose this part of the function to the application in the user space, the socket layer was introduced, and the sock was embedded in the framework of the file system. The sock became a special file, and the user can use the file handle in the user space, that is, socket_fd, to operate the network transmission capability of the kernel sock.
• The server can distinguish multiple clients through the four-tuple.
• The kernel achieves a similar inheritance effect through the C language feature that "the memory in the structure is continuous".

at last

This is the 27th article in the Illustrated Network series.

Most articles start with a question, then start with the basics and finally answer the question.

What I hope to do is to discuss a topic in depth so that novices can understand it and people with some work experience can also gain something from reading it.

Therefore, the time cost from topic selection to writing and drawing is extremely high.