Build an HTTP experimental environment by yourself

This is the last lecture of the "Ice Breaking" section. I will first briefly review the previous content, and then actually operate on the Windows system, using several application software to build a "minimized" HTTP experimental environment to facilitate the subsequent learning of the "Basics", "Advanced" and "Security" sections.

Review of “Ice Breaking”

The HTTP protocol was born 30 years ago and was originally designed to transmit plain text data. However, due to its flexible form and the ability to connect all resources on the Internet with technologies such as URI and HTML to form a complex hypertext system that allows people to freely access information, it has developed rapidly.

HTTP has multiple versions. The most widely used one is HTTP/1.1, which can almost be said to be the cornerstone of the entire Internet. However, the performance of HTTP/1.1 is difficult to meet the needs of today's high-traffic websites, so HTTP/2 and HTTP/3 appeared. However, these two new versions of the protocol have not yet been fully promoted. In the foreseeable future, HTTP/1.1 will continue to exist.

HTTP is translated into Chinese as "Hypertext Transfer Protocol". It is an application layer protocol, usually based on TCP/IP, which can transmit text, pictures, audio, video and other data between any two points in the network.

The two endpoints in the HTTP protocol are called the requester and the responder. The requester is usually a web browser, also called a user agent, and the responder is a web server, which stores most of the static or dynamic resources on the network.

There are some "middlemen" between the browser and the server, such as CDN, gateway, proxy, etc. They also comply with the HTTP protocol and can help users obtain resources more quickly and securely.

HTTP is not an isolated protocol, but requires the cooperation of many other protocols at the lower level. The most basic one is TCP/IP, which implements addressing, routing and reliable data transmission, and the DNS protocol, which implements the location search of hosts on the Internet.

A more accurate name for HTTP is "HTTP over TCP/IP", and another "HTTP over SSL/TLS" is HTTPS with added security features.

Software Introduction

As the saying goes, "Practice makes perfect", and there is also a saying that "Talking without practice is just a lie". To study the HTTP protocol, it is best to have a practical, operational and verifiable environment. Learning through actual data and phenomena is definitely much better than simply "talking".

Of course there are ready-made environments. As long as you can use a browser to access the Internet, there will be HTTP protocol, and you can conduct experiments. But the real network environment is too complicated, with many irrelevant interference factors, and these "noises" will "drowned" the truly useful information.

So, my suggestion to you is: build a "minimal" environment, in which there are only two endpoints of the HTTP protocol: the requester and the responder, and remove all unnecessary links so that you can grasp the key points and quickly grasp the essence of HTTP.

Let me briefly talk about the application software used in this "minimized" environment:

• Wireshark
• Chrome/Firefox
• Telnet
• OpenResty

Wireshark is a well-known network packet capture tool that can intercept all traffic transmitted in the TCP/IP protocol stack and filter it arbitrarily by protocol type, address, port, etc. It is very powerful and is an essential tool for learning network protocols.

It is like a "high-speed camera" in the online world, which "films" the network transmission process that happens in just a moment, and then "slows down" it afterwards, allowing us to calm down and carefully analyze what happened at that moment.

Chrome is a browser developed by Google and is one of the mainstream browsers. It is not only convenient for surfing the Internet, but also a good debugger. It supports HTTP/1.1, HTTPS, HTTP/2, QUIC and other protocols very well. You can use F12 to open the "Developer Tools" to observe various data in the entire HTTP transmission process in great detail.

If you are more accustomed to using Firefox, that's fine. In fact, its functions are not much different from Chrome. Just choose the one you like.

Unlike Wireshark, Chrome and Firefox are “experts after the fact” and cannot observe the HTTP transmission process, but only the results.

Telnet is a classic virtual terminal that logs into the host remotely based on the TCP protocol. We can use it to simulate the behavior of the browser, manually send HTTP requests after connecting to the server, and completely eliminate the interference of the browser, so that we can study the HTTP protocol from the most primitive level.

You may be unfamiliar with OpenResty. It is an "enhanced package" based on Nginx. In addition to Nginx, it also contains a lot of useful functional modules. It not only supports HTTP/HTTPS, but also integrates the scripting language Lua to simplify Nginx secondary development, facilitate and quickly build dynamic gateways, and can be used as an application container to write business logic.

The reason for choosing OpenResty instead of using Nginx directly is that it is equivalent to a "superset" of Nginx, with richer functions and easier installation and deployment. I will also use Lua to write some server-side scripts to implement simple Web server response logic for easy experimentation.

Installation Process

The installation process of this "minimized" environment is also relatively simple, and it only takes you less than half an hour to complete it.

I have created a project for this column on GitHub: http_study. You can download it directly using "git clone" or go to the Release page to download the compressed package.

The operating environment I use is Windows 10. If you are using Mac or Linux, you can use virtual machine software such as VirtualBox to install a Windows virtual machine and then operate in it.

First, you need to get the latest http_study project source code. Assuming the cloned or unzipped directory is "D:\http_study", the completed operation should look like the following figure.

The installation of Chrome and WireShark is relatively simple, just click "Next" all the way. Use the latest version. My version may not be the latest, Chrome is 73, WireShark is 3.0.0.

Windows 10 comes with Telnet, which does not need to be installed, but it is not enabled by default, so you need to configure it a little bit.

Open the Windows settings window, search for "Telnet", and you will find "Turn Windows features on or off". In this window, find "Telnet Client" and check the box. You can refer to the screenshot.

Next, we need to install OpenResty. Go to its official website, click "Download" in the left column, enter the download page, and download the version suitable for your system (here I downloaded the 64-bit 1.15.8.1, the package name is "openresty-1.15.8.1-win64.zip").

Then please note that you must unzip the OpenResty compressed package into the "D:\http_study" directory just now and rename it to "openresty".

The installation work is about to be completed. In order to enable the browser to use the DNS domain name to access our experimental environment, we also need to modify the local hosts file, located in "C:\WINDOWS\system32\drivers\etc", and add three lines of mapping from the local IP address to the test domain name. You can also refer to the hosts file in the GitHub project. This is equivalent to "hosting" three virtual hosts on a physical machine.

 127.0.0.1 www.chrono.com127.0.0.1 www.metroid.net127.0.0.1 origin.ioCopy code

Note that modifying the hosts file requires administrator privileges. Editing it directly with Notepad will not work. You can switch to administrator status, or use other advanced editors, such as Notepad++. It is also best to make a backup before making any changes.

At this point, our installation is complete! After that, you can use Wireshark, Chrome, and Telnet to "tamper" with this environment at will. It doesn't matter if you break it. Just delete the directory and perform the operation again to restore it.

Test verification

The experimental environment has been set up, but it still needs to be run and a simple test verification is performed to see if it is running normally.

First we have to start the web server, which is OpenResty.

There are four batch files in the "www" directory of http_study, namely:

• start: Start the OpenResty server;
• stop: stop the OpenResty server;
• reload: restart the OpenResty server;
• list: List the started OpenResty server processes.

Double-click the "start" batch file to start the OpenResty server and run it in the background. During this process, there may be a warning from the Windows Firewall. Just select "Allow".

After running, double-click "list" to check whether OpenResty has started normally. There should be two nginx.exe background processes, as shown in the figure below.

Now that we have a web server, we need to run Wireshark and start capturing packets.

Because our experimental environment runs on the local 127.0.0.1, which is the loopback address. Therefore, in Wireshark, we need to select "Npcap loopback Adapter" and select "HTTP TCP port (80)" as the filter, that is, only capture HTTP-related data packets. Double-click "Npcap loopback Adapter" in the start interface to start capturing network data on the local machine.

Then we open Chrome, enter "http://localhost/" in the address bar, access the OpenResty server just started, and you will see a simple welcome interface as shown below.

At this time, if you look back at Wireshark, it should show that some data has been captured. You can use the mouse to click the "Stop Capture" button in the toolbar to tell Wireshark to "stop here" and stop capturing packets.

As for what these data are and what they mean, I will introduce them in detail in the next lecture.

If you can get to this step on your computer, it means that the "minimized" experimental environment has been successfully built. Don't forget to run the batch "stop" to stop the OpenResty server after the experiment.

summary

This time we learned how to build an HTTP experimental environment on our own computer. Here is a brief summary of today's content.

1. The real network environment is too complex and there are many interference factors. Building a "minimized" environment can help you quickly grasp the key points and master the essence of HTTP.
2. We choose Wireshark as the packet capture tool to capture all traffic transmitted in the TCP/IP protocol stack;
3. We choose Chrome or Firefox browser as the user agent in the HTTP protocol;
4. We chose OpenResty as the web server, which is an "enhanced package" of Nginx with rich functions;
5. Telnet is a command line tool that can be used to log in to the host to simulate browser operations;
6. You can download the source code of the dedicated project for this column on GitHub. Just unzip OpenResty into it to complete the construction of the experimental environment.

Homework

1. Based on what you have learned today, build this HTTP experimental environment on your own computer and test and verify it.
2. Due to space limitations, I cannot introduce Wireshark in detail. If you have time, you can search for Wireshark-related information on the Internet to learn more about its usage.