Single machine million TCP connection test source code

[[382120]]

This article is reprinted from the WeChat public account "Kaida Neigong Xiuxian", written by Zhang Yanfei allen. To reprint this article, please contact the WeChat public account "Kaida Neigong Xiuxian".

After reading the two articles on achieving one million TCP connections on a single machine, many students have reported that they want to actually do the experiment and feel it. To make it easier for everyone, I have compiled the source code I used in the experiment today.

There are two methods I use to test millions of connections:

• The first is to open only one process on the server side, and then use many client IPs to connect
• The second is that the server starts multiple processes, so that the client can use only one IP

Let's look at the first method today, and the other method will be posted next time. The source code address is as follows, you need to copy it manually~

Source code address: https://github.com/yanfeizhang/coder-kung-fu/tree/main/tests/network/test01

Since the whole experiment is a bit complicated, I decided to write a separate article with the source code to make it easier for everyone to do it.

1. Server Preparation

1.1 Adjustment of the maximum number of open file handles

I won't go into details, but will give you the best adjustment method. If you get stuck or want to know more details, please see below.

"Get to the bottom of it. See how I deal with the Too many open files error!"

First, increase the system-level maximum handle number fs.file-max to more than 1 million, and make sure to add a little margin, so just set it to 1.1 million. Also, modify the system process-level parameter fs.nr_open, and also change it to 1.1 million.

 #vi /etc/sysctl.conf
 fs.file- max = 1100000
 fs.nr_open = 1100000

Use sysctl -p to make the setting effective. Use sysctl -a to verify whether it is effective.

 #sysctl -p
 #sysctl -a
 fs.file- max = 1100000
 fs.nr_open = 1100000

Next, increase the maximum number of open files for a user process (nofile), which is configured in /etc/security/limits.conf. These two are user process level, and can be configured differently for different users. For simplicity, we will directly configure it to all users*.

 # vi /etc/security/limits.conf
 * soft nofile 1010000
 * hard nofile 1010000

Note that hard nofile must be smaller than fs.nr_open, otherwise users may not be able to log in.

After configuration, open a new console. Use the ulimit command to verify that it is effective.

 #ulimit -n
 1010000

1.2 Start the server and start listening

Start the server

 php server.php 0.0.0.0 8090

Use the netstat command to ensure that the server is listening successfully.

 netstat -nlt | grep 8090
 tcp 0 0.0.0.0:8090 0.0.0.0:* LISTEN

2. Client Preparation

2.1 Adjust the available port range

By default, Linux only opens more than 30,000 available ports. But in our experiment today, a process needs to reach 50,000 concurrent connections. Therefore, the kernel parameters of the port range also need to be modified.

 #vi /etc/sysctl.conf
 net.ipv4.ip_local_port_range = 5000 65000

Execute sysctl -p to make it effective.

2.2 Increase the maximum number of open files

Corresponding to the server, the client's fs.file-max also needs to be increased to 1.1 million. However, setting the process-level parameter fs.nr_open to 60000 is sufficient.

 #vi /etc/sysctl.conf
 fs.file- max = 1100000
 fs.nr_open=60000

Use sysctl -p to make the setting effective. Use sysctl -a to check whether it is actually effective.

 #sysctl -p
 #sysctl -a
 fs.file- max = 1100000
 fs.nr_open = 60000

Next, increase the maximum number of open files for a user process (nofile), the same as the configuration method in the server. However, since we want to test the client with 20 processes, each process can open up to 50,000 files. Again, leave some room, so set it to 55,000.

 # vi /etc/security/limits.conf
 * soft nofile 55000
 * hard nofile 55000

After configuration, open a new console. Use the ulimit command to verify that it is effective.

 #ulimit -n
 55000

2.3 Select a new IP address available to the client

Because we are going to use a single client to connect to the same port of the same server. The maximum number of connections that a single client IP can achieve is limited by the number of ports. The maximum number is 65535 - 1024 = 64k.

One solution is to use 20 clients, each of which initiates 50,000 connections to connect to the server at the same time. This allows the server to reach 1 million TCP concurrency. However, this method is too difficult to implement in practice, so it is not feasible.

Another way is to configure multiple IPs for one machine. You can use the ifconfig command on Linux to configure multiple IPs for one machine. For example, if your network card device on Linux is eth0, and you want to configure an alias eth0:1, with an IP of CIP1 and a mask of 255.255.248.0. The command to configure the IP is as follows:

 ifconfig eth0:1 CIP1 netmask 255.255.248.0 up

So the next thing is to find 20 available IPs in your network environment. Assume that the available IPs are CIP1, CIP2, ..., CIP20.

Note: The configured IP must not conflict with other machines in the LAN, otherwise it will affect the normal sending and receiving of network packets of these machines.

2.4 Configure a new IP for the client

After determining the IP address, start modifying the $ips array and subnet mask in the test source code clientd.php.

In order to ensure that there are no such IPs in the LAN, you need to first execute a small tool provided in the code to verify it.

 php clientd.php ping

When the ping results of all IPs are false, you can proceed to the next step of the experiment.

Configure all IP addresses and start the network card.

 php clientd.php ifup

Use the ifconfig command to check whether the IP configuration is successful.

 #ifconfig
 eth0
 eth0:0
 eth0:1
 ...
 eth:19

After the IP configuration is completed, you can proceed to the next experiment.

3. Start the connection experiment

Modify the server IP and port in clientd.php. Then start the connection

 php clientd.php start

At the same time, start another console and use the watch command to observe the number of ESTABLISH state connections in real time.

Of course, the experiment process will not be smooth sailing, and various unexpected situations may occur. Don't panic if you encounter problems. Look for what is wrong according to the error message. Then adjust it and try again.

When you redo the process, you need to restart the client and server. For the client, the way to kill all client processes is

 php clientd.php stop

For the server, it is even simpler. Just terminate the server process with ctrl + c and restart it. If you find that the port is occupied, it is because the operating system has not recovered it. Just wait a while and then restart the server.

When you find that the number of connections exceeds 1 million, your experiment is successful.

 watch "ss -ant | grep ESTABLISH"  
 1000013

Don't forget to check the memory usage of your server and client at this time.

First use cat proc/meminfo to check, focusing on slab memory overhead.

 $ cat /proc/meminfo
 MemTotal: 3922956 kB
 MemFree: 96652 kB
 MemAvailable: 6448 kB
 Buffers: 44396 kB
 ......
 Slab: 3241244KB kB

Then use slabtop to see which kernel objects are allocated by the kernel and what the size of each of them is.

When you succeed in this experiment, I believe you will gain a lot!

4. End of the experiment

When the experiment is over, the server process can be canceled by pressing ctrl + c. The client code may need to be closed manually.

 php clientd.php stop

Finally, remember to cancel the new IP temporarily configured for the experiment

 php clientd.php ifdown

Repeat the test source code address used in this article

Address: https://github.com/yanfeizhang/coder-kung-fu/tree/main/tests/network/test01