HTTP, TCP, IP, and Ethernet in one article

This article is reprinted from the WeChat public account "Front-end Log", written by Meng Sixing. Please contact the front-end log public account for reprinting this article.

Recently, the department organized a front-end performance optimization exchange meeting, and everyone proposed many optimization points from entering the page URL to the final page display content. But at the same time, it was found that many students could not connect the knowledge of the HTTP protocol layer, so I compiled this article, hoping to bring some inspiration to everyone.

When we initiate an AJAX request on a page, what happens at the network protocol level?

 // Make a request
 fetch ( 'https://baidu.com' )
 // Protocol layer 1...
 // Protocol layer 2...
 // Protocol layer 3...
 . then (res=>
 // Get the result
 console.log(res)
 })

As shown in the above code, we initiated a network request to baidu.com and finally got the specific response content in the then method.

The results of packet capture using Wireshark are as follows:

As can be seen in the figure, when requesting baidu.com, the connection is first established through the TCP 3-way handshake, then the content is transmitted through HTTP, and finally the connection is disconnected through TCP 4-way handshake.

The actual process is more complicated. We mainly analyze the following points:

• Connection establishment phase
  • Find the target server through IP addressing (network layer)
  • Find the server hardware interface (data link layer) through Mac addressing
  • Transmit bit information to the server hardware interface through the network cable (physical layer)
  • DNS domain name resolution (application layer)
  • Establishing a TCP connection (transport layer)
• Sending data phase
  • Establishing SSL secure connection (application layer)
  • Send HTTP request (application layer)

Connection establishment phase

To obtain the web content of baidu.com, you need to establish a connection with the baidu server. How do you establish this connection?

1. Get Baidu's IP address through DNS.
2. Establish a TCP connection.

DNS domain name resolution

Through DNS resolution, we can find the IP address corresponding to the Baidu server.

As shown in the figure:

After DNS resolution, we can get the IP addresses of baidu.com: 39.156.69.79 and 220.181.38.148. Usually the client will randomly select an IP address for communication.

Domain name resolution steps

In fact, the IP address does not necessarily need to be obtained through DNS resolution. It is usually cached by the client, and the DNS server is requested only when there is no hit in the DNS cache.

The judgment steps are as follows:

1. Determine whether the browser has cached IP addresses.
2. Determine whether the local machine has cached the IP address, such as by checking the Host file.
3. Determine whether the local domain name resolution server has a cached IP address, such as: Telecom, Unicom and other operators.
4. Resolve the domain name IP address to the DNS root domain name resolution server.
5. Submit the DNS secondary domain name resolution server to resolve the domain name IP address.
6. And so on, finally get the IP address.

Establishing a TCP connection

With the IP address, the client and server can establish a connection, starting with a TCP connection.

TCP is a connection-oriented, reliable, byte stream-based transport layer communication protocol.

At this layer, the data we transmit is packed into messages one byte at a time, and when the length of the message reaches the maximum segment size (MSS), the message is sent. If the message to be transmitted is very long, it may be split into multiple TCP messages for transmission.

The TCP message header is as follows:

We mainly look at the following points:

• Source port, destination port.
• Sequence number: seq, the unique identifier of the message.
• Confirmation number: ack, the confirmation mark of the message, which is used to confirm whether seq has been received.
• TCP Flags:
  • SYN is 1, indicating that this is a connection request or a connection acceptance request. It is used to create a connection and synchronize sequence numbers.
  • ACK is 1, indicating that the confirmation number field is valid. Note that the capitalized ACK here is just a mark and is not the same as the confirmation number ack.
  • FIN is 1 to request to release the connection.
• Window: Indicates the number of bytes that the sender can receive, that is, the receive window size, which is used for flow control.

Next, let's take a look at how TCP establishes a connection.

As shown in the figure, establishing a TCP connection requires three steps, commonly known as a three-way handshake.

• First handshake: The client sends a sequence number seq=x to the server, indicating that a connection has begun to be established.
• Second handshake: The server sends back an ack=x+1 identifier to confirm receipt of the first handshake, and sends its own identifier seq=y.
  • The client confirms that the data it sends can be received by the server.
• Third handshake: The client sends an ack=y+1 flag, which confirms receipt of the second handshake.
  • The server confirms that the data it sends can be received by the client.

After three handshakes, it is guaranteed that both the client and the server can send and receive data normally, and the TCP connection is successfully established.

TCP reliable transmission principle

As mentioned above, TCP is a reliable transmission. Why is that?

This is because TCP uses the stop-and-wait protocol ARQ internally, which achieves reliable transmission of information through confirmation and retransmission mechanisms.

For example:

• The client sends data M1
• The server confirms that data M1 has been received
• The client sends data M2
• The server confirms that data M2 has been received
• And so on...

During this period, if a piece of data has not been confirmed for a long time, the client will retransmit the data. In this way, the server will get confirmation for each data sent, which ensures the reliability of the data.

Although ARQ can meet data reliability requirements, it can only send and confirm one request at a time, which is too inefficient. Therefore, the continuous ARQ protocol was created.

The continuous ARQ protocol sends a group of data continuously, and then waits for confirmation information of this group of data in batches, which is like turning a single-threaded ARQ into a multi-threaded one, greatly improving the efficiency of resource utilization.

like:

• The client sends data M1, M2, M3, and M4.
• The server confirms that data M4 has been received, indicating that data M4 and before have been received.
• The client sends data M5, M6, M7, and M8.
• The server confirms that data M8 has been received, indicating that data M8 and before have been received.

In this process, the server does not need to return confirmation information for each data, but confirms multiple data together when it receives them. This method is called cumulative confirmation.

Here is a question, how does TCP find the destination server for each handshake?

A: Through IP protocol.

Find the target server based on IP protocol

The purpose of the IP protocol is to achieve data forwarding at the network layer. It continuously jumps through routers and eventually successfully delivers the data to the destination.

Each TCP handshake and data interaction mentioned above is transmitted through the IP protocol.

The IP header is as follows:

We just need to focus on the following two points:

• Source IP address
• Destination IP address

The execution process of initiating an IP request is as follows:

1. Construct IP request header (source IP, destination IP).
2. The IP protocol uses an algorithm to calculate a path to the server.
3. The sender queries the routing table, finds the IP address of the next hop (usually a router), and sends the data.
4. The router queries the routing table, finds the next hop IP address, and sends the data.
5. Repeat step 4 until the destination LAN is found.
6. Send data.

The routing table exists in the computer or router, and consists of four parts: destination IP address, subnet mask, next hop address, and sending interface. Through the destination IP address, the next hop address can be found for forwarding.

For example: A wants to send IP data to G.

The specific process is as follows:

• A generates IP header (source IP: A, destination IP: G)

A queries the routing table and finds that the next hop is B, so it passes the data to B.

• B generates IP header (source IP: A, destination IP: G)

B queries the routing table and finds that the next hop is E, so it passes the data to E.

• E Generate IP header (source IP: A, destination IP: G)

E queries the routing table and finds that the next hop is G, so it passes the data to G.

• Arrive at destination G.

Are you wondering why IP transmits data along this path to G?

In fact, there is more than one path in the above figure. We can reach the destination G through ABEG, and we can also reach G through ABCFHG. Both paths can complete the task. Why doesn't IP choose the path ABCFHG?

This involves the IP addressing algorithm.

IP addressing algorithm

We can think of all computers in the network as points, and the connections between computers as lines, and these points and lines form a graph.

For example:

Through the above diagram, we have transformed the complex network into a mathematical problem. The IP addressing algorithm is actually the shortest path algorithm in graph theory.

There are two implementations of the shortest path algorithm in the IP protocol:

• RIP protocol
  • Each node stores the location information of other nodes (number of hops and next hop IP).
  • By exchanging data with neighboring nodes and updating the shortest distance to the destination, repeating this process continuously, you can get the shortest path from the starting point to the end point.
  • It is simple to implement, has low overhead, and is suitable for small networks.
  • Use the distance vector algorithm to ensure that the number of IP routing hops is minimized.
  • principle
• OSPF Protocol
  • Starting from the starting point, the greedy algorithm strategy is adopted to traverse the adjacent nodes of the vertex that is closest to the starting point and has not been visited each time until it reaches the end point.
  • Suitable for large networks.
  • Use Dijkstra's algorithm to ensure the fastest IP routing jump.
  • principle

Through the above two protocols, we can find the path to the destination.

Here comes a question: How does IP data jump from one router to another?

A: Through Ethernet protocol.

Find the server hardware interface through Mac addressing

The IP protocol is mainly used to find the optimal path, and the specific transmission is done by the Ethernet protocol.

Ethernet belongs to the data link layer, which is mainly responsible for the communication between adjacent devices. The principle is to find the physical interface of the communicating parties by querying the switch Mac table, and then start communication.

The Ethernet message header is as follows:

We only need to care about the following three points:

• Source Mac Address
• Destination Mac Address
• Check code CRC: Check whether the current frame is valid.

As you can see, the Ethernet layer communicates through Mac addresses. Where do the Mac addresses come from?

Answer: Through ARP protocol.

ARP protocol is a protocol that finds Mac address by resolving IP address. After IP address is converted into Mac address, Ethernet data transmission can be carried out.

For example:

When machine A sends data to machine C:

• A constructs an Ethernet message (source address: A, destination address: C) and sends the data frame through the network card.
• The data frame arrives at switch B, and the switch extracts the Mac address of destination address C.
• B queries the Mac table and matches C's hardware interface based on the destination Mac address.
  • If the hardware interface of C is found, send the data.
  • If the hardware interface of C is not found, a broadcast message is sent to all machines directly connected to B to find C. Once found, C is recorded in the Mac table.

After the above process, we found the hardware interface of the target machine.

Through the Ethernet protocol, we have found the hardware interface of the target machine. How do we send information next?

A: Through the physical layer.

Transmit bit information to the server hardware interface through the network cable

In the era without WiFi, we could only access the Internet by plugging in a network cable, which is actually one of the devices at the physical layer.

Network cables can be made of a variety of materials, the most common of which are optical fiber and electrical cables.

The transmission principles of optical fiber and cable are similar, both use two signals to simulate binary data, one signal is one bit.

• In the cable: high potential represents 1, low potential represents 0.
• In optical fiber: light on represents 1, and light off represents 0.

For example, in optical fiber, we can know the transmitted binary data by observing the flashes of light.

With these physical devices, we can convert complex data into optical or electrical signals for transmission.

Sending data phase

Sending data can be divided into two steps:

• Establishing SSL security layer
• Sending HTTP Requests

Establishing SSL security layer

The case in this article is to send an HTTPS request, so before sending the data, an SSL security layer will be created for data encryption.

There are two common encryption methods:

• Asymmetric encryption
  • A has the key, B does not, and they both have a common lock. When B sends data to A, it will lock the data before sending it.
  • When receiving data, A uses the key to unlock the lock and get the data. Except for A, others do not have the key and cannot get the data.
  • One-way communication encryption is achieved.
• Symmetric encryption
  • Both A and B have the same key and a common lock. Every time they send data, they put the data in the lock and send it.
  • When receiving data, A and B use their own keys to unlock it. Others who do not have the key cannot obtain the data.
  • Two-way communication encryption is achieved.

Internet communication is bidirectional, so we need to use symmetric encryption. However, how can we ensure that both parties have the same key?

Current solution:

• First use asymmetric encryption to negotiate the secret key so that both communicating parties get the same key.
• Then use symmetric encryption for encrypted transmission.

The key negotiation process is shown in the figure:

Key points in the figure:

1. The client sends the encryption algorithms it supports.
2. The server selects an encryption algorithm and returns a digital certificate.
3. The client confirms that the certificate is valid.
4. The client generates a random number, encrypts it using the server's public key in the certificate, and then sends it to the server.
5. The server uses the private key to decrypt and obtain a random number.
6. Both parties use the encryption algorithm determined in step 2 to encrypt the random number and obtain the same symmetric encryption key.

Ok, after the key negotiation, our SSL security layer is established.

There is a problem during key negotiation:

When negotiating a key, how can we ensure that we are negotiating with the real server and not a middleman?

Answer: Digital certificate.

Digital certificates focus on 2 parts:

• Server Public Key
• Digital Signature

The digital signature is generated by encrypting the server public key and the certificate private key to prevent the server public key from being tampered with.

With a digital certificate, the client can determine whether the server is the real server by verifying the certificate.

The verification logic is as follows:

It can be seen that the digital certificate is decrypted using the same algorithm. If the same information digest is obtained, the data is guaranteed to be valid. If they are inconsistent, the verification will fail and subsequent requests will be rejected.

So far, all preparations are ready, and the next step is to send the HTTP request.

Sending HTTP Requests

The HTTP protocol actually establishes a communication rule and specifies the communication format between the client and the server.

Take requesting the Baidu homepage as an example:

As shown in the figure above, when making an HTTP request, the following rules must be followed:

• Request method (required) GET
• Request address (required) /
• HTTP protocol version (required) 1.1
• Other HTTP header fields (optional) Host, User-Agent, Accept
• Request parameters, placed after a blank line (optional)

When the server responds to the request, it also follows the HTTP response rules:

• HTTP protocol version (required) 1.1
• Response status code (required) 200
• Status code description (required) OK
• Other HTTP header fields (optional): Date, Server, ETag, Last-Modified, etc.
• Request parameters, placed after a blank line (optional)

As long as we follow this rule, we can perform HTTP communication.

So far, we have analyzed all the processes of data request. Do you understand them all?

Thoughts and Conclusions

This article uses a network request to conduct a process analysis of the entire HTTP, TCP, IP, Ethernet and other protocols, and finally sorts them out:

1. Request baidu.com.
2. DNS resolves baidu.com and obtains the IP address.
3. Establish a TCP connection.
4. The IP protocol uses an algorithm to calculate the optimal path to the server.
5. When the IP jumps along the path, the IP address is converted into a Mac address through the ARP protocol.
6. Ethernet finds the hardware interfaces of both communicating parties through Mac addresses.
7. The physical layer uses the network cable as a carrier to transmit bit signals between two hardware interfaces.
8. The TCP connection is established.
9. Establish SSL security layer.
10. Send an HTTP request.