Remember who was to blame for a thread pool-induced fault?

This article is reproduced from the WeChat public account "Third Prince Ao Bing". Please contact the Third Prince Ao Bing public account for reprinting this article.

[[329971]]

background

Ao Bing encountered a problem at work before. I defined a thread pool to execute tasks, but the tasks were not completed after the program was executed. I almost collapsed at that time.

The business scenario is as follows: Due to statistical business needs, order information needs to be written from the main database to the statistical database through the statistical business code (logical processing is required in the middle, so binlog cannot be used).

Due to code quality and historical reasons, the current re-statistics interface is single-threaded. A rough calculation shows that there are 1 million order information in total, and it takes about 10 seconds to process every 100 items. So in theory, it takes 28 hours to process all the information, not including the later query time caused by limit paging in MySQL and the possible memory overflow that causes the statistics to be terminated.

Based on the above reasons, and the most important point: the statistical business is performed according to the center to which the order belongs, and simultaneous statistics by various centers will not lead to dirty data.

Therefore, I plan to use a thread pool to assign a thread to each center to perform statistical operations.

Business realization

 //Thread factory, used to name each thread in the thread pool
 ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat( "stats-pool-%d" ).build(); 
 
 // Create a thread pool and use a bounded blocking queue to prevent memory overflow
 ExecutorService statsThreadPool = new ThreadPoolExecutor(5, 10,
                0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<>(100), namedThreadFactory);
 // Traverse all centers and submit a task to the thread pool for each centerId
 statsThreadPool.submit(new StatsJob(centerId));

After creating the thread pool, I submitted a task to the thread pool for each centerId. In my expectation, since the number of core threads in the thread pool is 5, up to 5 centers can perform statistical business at the same time, which will greatly shorten the total statistical time of 1 million data. So I started to execute the re-statistics business with great excitement.

question

After running for a long time, when I checked the statistical progress, I found a very strange problem (as shown below).

The thread marked by the blue box is in WAIT state, indicating that the thread is idle, but from the log I can see that the thread has not completed its task, because there are 100,000 data in this center, but the log shows that it has only run half of it, and there are no more logs about this center.

What is the reason for this?

I immediately thought of Sanwai. It must be that Sanwai stepped into the company with his left foot first this morning, causing the code to not adapt to the local environment. This must be the case. I'll go find him.

Debugging and reasons

Kekesanwai is just a joke, we still need to find the real reason.

It is conceivable that this thread is blocked for some reason and does not continue, but there is no abnormal information in the log...

Experienced engineers may already know the reason...

Due to my personal level of thread, I have not found the reason yet, so I can only give up using the thread pool and run it with a single thread...

Fortunately, the task running in a single thread actually threw an error (why say lucky?), so I immediately thought that the previous thread in the WAIT state might have been interrupted because of the same error, causing the task to not continue.

Why do I say lucky? Because if the single-threaded task had not thrown an error, I might not have thought of this reason for a long time.

Deep dive into thread pool exception handling

The cause of the problem at work has been found here, and the subsequent solution process is also very simple, so I won’t mention it here.

But the question arises again, why when using the thread pool, the thread is interrupted due to an exception but no information is thrown? Also, if the exception is in the main function, it will be thrown out instead of being swallowed like the thread pool.

If a child thread throws an exception, how will the thread pool handle it?

The way I submit tasks to the thread pool is: threadPoolExecutor.submit(Runnbale task);. Later I learned that submitting tasks using the execute() method will print out exception logs. Here I will study why when submitting tasks using submit, exceptions in the task will be "swallowed".

For tasks submitted in the submit() form, we look directly at the source code:

 public Future<?> submit(Runnable task) {
    if (task == null ) throw new NullPointerException();
 // Wrapped into a RunnableFuture object and ready to be added to the work queue
    RunnableFuture<Void> ftask = newTaskFor(task, null );
 execute (ftask);
 return ftask;
 }

It will be wrapped into a RunnableFuture object by the thread pool, and ultimately it is actually a FutureTask object. After being added to the work queue of the thread pool and then calling the start() method, the run() method of the FutureTask object starts running, that is, the task starts executing.

 public void run() {
    if (state != NEW || !UNSAFE.compareAndSwapObject(this,runnerOffset, null , Thread.currentThread()))
 return ;
 try {
        Callable<V> c = callable;
        if (c != null && state == NEW) {
 V result;
 boolean ran;
 try {
                result = c.call();
 ran = true ;
            } catch (Throwable ex) {
 //Catch exceptions in subtasks
                result = null ;
 ran = false ;
                setException(ex);
 }
 if (ran)
 set (result);
 }
 finally
 runner = null ;
 int s = state;
        if (s >= INTERRUPTING)
            handlePossibleCancellationInterrupt(s);
 }
 }

In the run() method of the FutureTask object, the exception thrown by the task is caught, and then in the setException(ex); method, the thrown exception is placed in the outcome object, which is the result of executing the get() method of the FutureTask object returned by the submit() method.

However, in the thread pool, the result of executing the child thread is not obtained, so the exception is not thrown, that is, it is "swallowed".

This is why no exception is thrown when the thread pool's submit() method submits a task.

Thread pool custom exception handling method

Called when defining ThreadFactory

setUncaughtExceptionHandler method, customize the exception handling method. For example:

 ThreadFactory namedThreadFactory = new ThreadFactoryBuilder()
                .setNameFormat( "judge-pool-%d" )
                .setUncaughtExceptionHandler((thread, throwable)-> logger.error( "ThreadPool {} got exception" , thread,throwable))
 .build();

In this way, an error log will be written for the exception thrown by each thread in the thread pool, and it will not be invisible.

Follow-up

After fixing the exception of the single thread task, I continued to use the thread pool to re-count the business, and finally ran it to the end, and finally completed the task.

Afterwards, I also told Sanwai to take the right step when entering the company in the future, otherwise it would have a great impact on the feng shui of writing code.

Summary: This accident also serves as a warning to everyone. When using the thread pool, you need to pay attention to the fact that the exception of the child thread will be lost if it is not captured, which may make it impossible to find the cause when debugging according to the log later.