A pitfall when using HTTP Client that you must avoid

Preface

As software developers, we know that seemingly normal systems may contain many pitfalls. Today, I would like to share with you a pitfall that I encountered during actual combat due to improper use of HTTP Client.

The author traced the problem all the way through the symptoms and finally found the root cause of the problem: HTTP Client. The conclusion is very simple, but the analysis process is worth learning from.

Problem phenomenon

Scenario: Almost every system has the function of asynchronously calling third-party services. The system in charge implements a message queue based on a blocking queue to call third-party services. In order to ensure idempotence, the queue is consumed sequentially. This leads to a problem. Once one of the messages is blocked, the subsequent messages cannot be consumed. When the queue is full, messages cannot be added to the queue.

Seemingly: In extremely rare scenarios, the message queue is blocked for more than ten minutes. What the hell is this?

The following is the step-by-step troubleshooting of the problem.

Troubleshooting

The first thing that comes to mind is whether there is a problem with the underlying mechanism of the message queue. For example, the message queue uses while(true) polling + sleep to achieve continuous access to data by producers and consumers.

Since the message is blocked, is it because the thread sleeps too long? Later, I thought that even if the CPU is sliced, it will not sleep and will not be awakened. At the same time, it is unlikely that the thread is interrupted. Because if it is interrupted, the entire queue will be suspended and will not be restored to normal after a delay.

I was confused here for a long time, and read the source code of the message queue implementation for a long time, but there was no breakthrough. So, I discussed it with a friend, and one sentence reminded me: it may not be a problem of sleep, but a problem of producers or consumers.

So I carefully checked the logs and found that it was true: the producer dropped data into the queue once, and then did not drop data again for a long time; the consumer still had consumption logs a few minutes after the producer dropped data into the queue. Obviously, the producer was blocked by the consumer.

Preliminary conclusion: The consumer consumes for too long, causing the queue to be full and the producer to be blocked when adding data to the queue.

Empirical guess: There are HTTP requests in the consumer, and the HTTP requests may hold the connection for a long time without releasing it.

Root of the problem

Once the cause was located, it was easy to solve. First, the log was analyzed and it was found that there was indeed an HTTP request. The request parameters were printed before the request, but the return result was never printed. After searching the log, it was finally found that the return result log was printed after 15 minutes, and the log content was that the other party's service was abnormal.

Looking at the code again, I found that the HTTP request is implemented based on HTTP Client, and the person who wrote this code did not set a timeout. In order to keep the business desensitized, I found a similar code:

 public static String doPostWithJSON ( String url , String json ) throws Exception {
 CloseableHttpClient client = HttpClients .createDefault ( ) ;
 HttpPost httpPost = new HttpPost ( url ) ;
 httpPost .setHeader ( "Content-Type" , "application/json;charset=UTF-8" ) ;
 StringEntity se = new StringEntity ( json , Charset .forName ( "UTF-8" ) ) ;
 se .setContentType ( "application/json" ) ;
 httpPost .setEntity ( se ) ;
 CloseableHttpResponse response = client .execute ( httpPost ) ;
 HttpEntity entity = response .getEntity ( ) ;
 String result = EntityUtils .toString ( entity , "UTF-8" ) ;
 return result ;
 }

Like the above code, the request parameters are set, but the HttpPost timeout is not specified. The seemingly normal code hides a huge pit, resulting in the HTTP request waiting all the time.

The situation I encountered was not so bad. The other party set the timeout time to 15 minutes and returned the result. If the other party did not set the timeout time, you might have to wait forever. When the business volume is large, it will lead to disasters!

HTTP Client timeout settings

Finding the problem is often the most difficult thing. Once you find the problem, it is much easier to solve it. Different versions of HTTP Client have different ways of setting the timeout. This is another major drawback of HTTP Client. The API version changes too much.

Configuration of version 4.3:

 httpClient .getParams ( ) .setParameter ( CoreConnectionPNames .CONNECTION_TIMEOUT , 10000 ) ;
 httpClient .getParams ( ) .setParameter ( CoreConnectionPNames .SO_TIMEOUT , 10000 ) ;

Configuration for versions later than 4.3:

 RequestConfig requestConfig = RequestConfig .custom ( ) .setSocketTimeout ( 10000 ) .setConnectTimeout ( 10000 ) .build ( ) ;
 httpGet .setConfig ( requestConfig ) ;

Among them, setConnectTimeout is the connection timeout period, in milliseconds.

setSocketTimeout is the timeout for requesting data, in milliseconds. If an interface is accessed and no data can be returned within the specified time, the call is abandoned directly.

For the use of other versions, it is recommended to refer to the relevant API instructions.

summary

In the process of practice, we often encounter some inexplicable problems, and the reasons for these problems may be insufficient practical experience or lack of familiarity with the use of APIs. Where does the experience come from? It comes from the investigation, summary, and accumulation of problems like this one.

For example, after reading this article, you already know that you must set a timeout when using HTTP Client. At the same time, you can also draw inferences from this that you need to consider the timeout and corresponding exception handling when making HTTP calls.