Really "understand" the gameplay of HTTP protocol body

We have discussed the characteristics of HTTP and the start line, and focused on the request method and status code. These two things are very important because they are often used in conjunction with the header field. I have repeatedly emphasized that the subsequent content will involve related content. From this chapter until HTTP/2, I will take you to learn and practice the core header field part of HTTP/1 through Node. Most of the capabilities of HTTP are actually extended through the header field.

In this chapter, we will learn about the header fields related to the body.

Let's recall what we know about the body so far. In the 0.9 era, there was only the body of the response, but no request body. It was not until 1.0 that the request body and response body were added, that is, both the request and the response had a body. 1.1 expanded some of the body fields. Let's take a look at the content of the body header field and how to negotiate it.

1. MIME

I have briefly talked about this before, and I believe everyone has some impression of it. MIME plays a very important role in the HTTP body system. We need to have a deeper understanding. I have said before that when we need to transmit some data or content, how can I transmit the data to the other party and correctly translate the data? We will not go into too much detail about the transmission of TCP, and the translation work is done by the communicating parties, or the client and the server.

The client sends a picture to the server. How does the server know that it is a picture? Or vice versa, how does the client know that it is a picture? Theoretically, no matter what method is used, it will not work. Unless I tell you "this is a picture". Doesn't it feel a bit simple? To put it bluntly, it is negotiation. Even if the server receives the message from the client and knows that it is a picture, I just don't parse it according to the picture and directly report an error to you, and you can't do anything about it.

But the meaning of "standard" is that we have to follow the standard, so... although the server can not follow the rules, we have to learn according to the rules.

Let's continue. Hahaha, what is MIME? MIME stands for Multipurpose Internet Mail Extensions. It was originally used in email systems to allow emails to send multiple types of data. Here is a more detailed introduction. If you are interested, you can read it yourself. When HTTP also needed this thing, it was found that MIME was good, I could use it directly, saving me from having to develop my own system. So HTTP took some of it and used it.

MIME divides data into eight categories, and the format is roughly like this: type/subtype. Its eight types are roughly as follows:

• Text: used to represent text messages in a standardized manner. Text messages can be in multiple character sets and/or multiple formats.
• Multipart: It is used to connect multiple parts of the message body to form a message. These parts can be different types of data.
• Application: used to transfer application data or binary data;
• Message: used to package an E-mail message;
• Image: used to transmit static image data;
• Audio: used to transmit audio or voice data;
• Video: used to transmit dynamic image data, which can be a video data format edited together with audio;
• Font: used to transfer font files;
• Model: used to transfer 3D model files.

Are you familiar with some of these types, such as Text, Multipart, Application, Image, Video, etc.? We must have come into contact with them more or less in actual work. Then, let's take a look at the subtypes:

• text/plain
• text/html (HTML files)
• application/xhtml+xml (XHTML files)
• image/gif (GIF image)
• image/jpeg (JPEG image)
• image/png (PNG image)
• audio/mpeg (MP3 audio)
• audio/aac (AAC audio)
• video/mpeg (MPEG video)
• video/mp4 (MPEG-4 video)
• application/octet-stream (arbitrary binary data)
• application/json (JSON file)
• application/pdf (PDF file)
• application/msword (Microsoft Word files)
• application/vnd.openxmlformats-officedocument.wordprocessingml.document (Microsoft Word 2007 file)
• application/vnd.wap.xhtml+xml (wap1.0+)
• application/xhtml+xml (wap2.0+)
• message/rfc822 (RFC 822 format)
• multipart/alternative (HTML form and plain text form of HTML email, the same content is represented in different forms)
• application/x-www-form-urlencoded (form sent using HTTP POST method)
• multipart/form-data (same as above, but mainly used when a form is submitted with a file upload)

I have listed most of the data types and their subtypes. Of course, you don't necessarily need to fully understand these things, but you can understand their meanings by name. The parts I put in bold are actually the most common data types in our daily work.

2. Data Types

In HTTP, we can use the Accept field to tell the server what type of data we want to receive, and the server uses the Content header field to tell the client what data is actually sent. Note that Accept and Content are a category, which is also the core content of this chapter. It contains a lot of header fields, which we will also talk about slowly.

Let's go back to the header field that indicates the data type. The Accept field indicates the MIME type that the client can understand. You can use "," to separate and list multiple types to give the server more options, for example:

 Accept : application / json , text / html , application / xml

This tells the server that the data types I can parse are json, html, and xml, and that I can get data within these types.

Accordingly, the server uses the Content-Type header field to inform the client of the actual type of the entity data:

 Content - Type : application / json

In this way, the browser will know that it is a json file when reading Content-Type, and then parse it through the engine, and that's it.

It's simple, right?

Then... I still want to emphasize that if the server receives the data type that the client wants, but I don't give it to you according to what you want, then there is actually no problem at all. Therefore, in the early RFC1945, Accept was attached to other functions. It was not until 1.1 that it was officially added to the standard.

3. Data Compression

Normally, when we transmit data, in order to save bandwidth, we will compress the data before transmitting it. The same is true in HTTP. There are many ways to compress data. Of course, there are much fewer ways than MIME, only three:

gzip: Familiar, that is, GNU zip compression format, which is also the most popular compression format on the Internet;

deflate: zlib (deflate) compression format, second only to gzip in popularity;

br: A new compression algorithm (Brotli) optimized specifically for HTTP.

The client can then use the Accept-Encoding field to mark the supported compression formats, or use "," to separate multiple supported formats. The server will then put the compression format actually used in the Content-Encoding field.

 Accept - Encoding : gzip , deflate , br

 Content - Encoding :

In actual use, these two fields can be omitted. Omission by the client means that compression is not supported, and omission by the server tells the client that the transmitted data is not compressed.

4. Language Type

With the data type and compression type, the machine can identify what the transmitted data is and how to decompress it. However, there are so many countries and regions in the world, and different countries and regions use different languages. Even people in the same country and region may use different languages. So how can the browser display the language that everyone can understand? In other words, how can I correctly encode this data according to different situations? In other words, it is actually an internationalization issue.

I guess you who have learned this already know how to solve it, negotiation, fields. Hahaha, it feels a bit boring. There is no suspense at all.

The field used for the request header is Accept-Language, and the entity header field in the response message is Content-Language. Please note that the Accept header field is a request header field, while Content is an entity header field, not a response header field. Please pay attention to this.

Here are some examples:

 Accept - Language : zh - CN , zh , en

 Content - Language : zh - CN

It's very simple and not complicated, but it's not over yet. What are the values ​​corresponding to these header fields? Well... these things are called language types, which are natural languages ​​used by humans, such as English, Chinese, French, etc., and these natural languages ​​also have their own dialects, so they are similar to data types and are also in the form of type-subtype. However, unlike language types, data types use "/" to separate parent and child classes, while language types use "-" to separate them.

For example, en means English, en-US means American English, en-GB means British English, and of course there are more language types. You can learn about them yourself. There is no point in saying more here.

At this point, the server knows what type of language to use, but you have to know that the underlying nature of computers is 0 and 1. How can I translate 0 and 1 into the corresponding language? This requires the use of character sets. In the early days of computer development, it was very chaotic. People in various countries and regions defined their own systems and invented many encodings to process their own texts, such as ASCII for English and GBK for Chinese. This results in the same text being displayed completely differently using different encodings.

So later Unicode and UTF-8 came into being, accommodating all the languages ​​in the world in one solution.

In the HTTP request header, Accept-Charset can be used to express the encoding type that the client can accept. However, there is no corresponding field in the response header. Instead, it is indicated by "charset=xxx" after the data type of the Content-Type field. You should pay special attention to this.

 Accept - Charset : gbk , utf - 8

 Content - Type : text / html ; charset = utf - 8

However, modern browsers support multiple character sets, so usually, the Accept-Charset request header is not sent, and the server does not return Content-Language, because the language used can be completely inferred from the character set. Therefore, there is usually only Accent-Language in the request header and only Content-Type in the response header.

5. Quality Value

The English name of the quality value is called quality factory, which literally means weight. It uses q as a parameter in HTTP, in the form of "q=value", and this value can be between 0 and 1, including two decimal places of 0 and 1. The value in the field is separated by ";".

One thing to emphasize here is that in most other languages, such as JavaScript, the semicolon ";" is stronger than the comma "," in sentence punctuation, but in HTTP it is the opposite. Let's look at an example:

 Accept : text / html , application / xml ; q = 0.9 , */* ; q = 0.8

What does this mean? The browser most wants the server to send an HTML file. If it is not specified, the default weight is 1, followed by an XML file with a weight of 0.9, and finally any file type with a weight of 0.8. After receiving the request, the server will prioritize returning HTML based on this content.

6. Vary

This is a bit strange, let's learn it. It means that the response message I return to you refers to which header fields. In other words, the process of the client and the browser negotiating how to return the response message is not transparent. You don't know how the negotiation is done, or the server may not care whether you negotiate or not.

However, friendlier servers will add a Vary field in the response header to record the request header field that the server refers to during content negotiation, providing some information.

 Vary : Accept - Encoding , User - Agent , Accept

The above example shows that the server returns a response message after referring to the three fields of Accept-Encoding, User-Agent, and Accept.

The Vary field can be considered as a special "version tag" of the response message. Whenever the Accept and other request headers change, the Vary field will also change with the response message. In other words, the same URI may have multiple different "versions", which are mainly used by proxy servers in the middle of the transmission link to implement cache services. This will be mentioned again when talking about "HTTP cache" later.

7. Block transmission

In the first six sections, we talked about how data is negotiated in HTTP so that both the client and the server know how to process the data. And if the data volume is too large, we can also compress the transmitted data by negotiating compression. It seems that everything is fine, but what if the file I want to transmit is very large? For example, a video... a small one is hundreds of megabytes, and a large one is several G, and the compression efficiency for videos is very low, so how do you transmit it?

Well... the title is the answer. We can't reduce a large volume of data as a whole, so we can only split this particularly large data into small pieces. The server transmits these small pieces of data to the browser, and the browser reassembles and restores them according to certain rules after receiving them.

This idea is called chunked in HTTP, which means chunked transfer encoding. It can be represented in the response message by "Transfer-Encoding: chunked", which means that the body in the response message is not sent all at once, but is divided into many chunks and sent one by one.

One thing you should note is that the length of a response message is either known or unknown. It is impossible to know and not know at the same time. What does this mean? Transfer-Encoding: chunked and Content-length are mutually exclusive and cannot appear in the response header at the same time.

8. Range Request

With block transmission, we can send a large amount of data one by one, solving the problem of large files getting stuck during transmission. Let's take video as an example. You are watching a TV series on Tencent Video or iQiyi, and suddenly an ad pops up in the video. Or you don't want to watch the beginning and end of the TV series, so you will skip this part by dragging the progress bar. In this case, range request is needed.

In other words, we hope to obtain a certain fragment of a large file, but block transmission cannot do this. Block transmission can only divide the blocks and send them to you at the beginning of the transmission, and it is impossible to determine the data in a certain range that I need.

To solve this problem, you need to use range requests. Range requests allow the client to use a dedicated field in the request header to indicate that only a part of the entire file is obtained.

Range requests are not a must-have feature for Web servers. They can be implemented or not, so the server must use the field "Accept-Ranges: bytes" in the response header to explicitly inform the client that it supports range requests. If it does not support range requests, you can use "Accept-Ranges: none" to inform the client, or simply do not send the Accept-Ranges field.

The client uses "Range" as the request header format for range requests, which is "bytes=xy". x and y are the range data in bytes. x must start from 0, for example, 0-9 refers to the first 10 bytes, and so on.

When the server receives the Range field, it does four things:

First, the server will check whether the range you passed is legal. If it is illegal, the server will directly give you a 416, telling you that the requested data range is illegal.

Secondly, if the range is legal, the server will read the file fragment according to your range and return a 206 status code, which is Partial Content, indicating that part of the original data is returned.

When returning partial data, the server adds a Content-Range response header to tell the client the actual offset and total size of the resource in the format of "bytes xy/length".

The last step is to send the data.

Range requests can be used not only to watch the video drag progress, but also for multi-terminal downloading and breakpoint resumption during downloading. In fact, they are also implemented based on it. We will talk about this in practice in the next chapter.

9. Multi-segment data

Based on the range request, we can also request more than one range of fragments, that is, request multiple pieces of data at one time. In this case, a special MIME type is required: multipart/byterange, which means that the body of the message is composed of multiple byte sequences, and a boundary=xxx is also required to give the segmentation marker between the segments. Like this:

 Content - Type : multipart / byteranges ; boundary = 00000000001

The boundary=00000000001 is the split mark, starting with --00000000001 and ending with --00000000001--.

Well... the theory is definitely a bit vague, but we will see clearly what it looks like in the next article when we put it into practice.

Summarize

In this article, we mainly talked about two things, one is files, and the other is file transfer. Regarding files, we talked about file types, language types, simple compression, etc., and file transfer mainly has two parts, one is block transfer, and the other is segment transfer. That's all, it's all theory.

In addition, I would like to emphasize the issues of language type and internationalization discussed in the fourth part. In fact, internationalization in HTTP refers to the data language in the file you transmit, not the internationalization plug-in we use in the front-end single-page application. There is a difference between the two.