How to use gdb to accurately locate deadlock problems in multithreading

[[337631]]

This article is reprinted from the WeChat public account "Program Cat Adult", the author is Program Cat Adult. Please contact Program Cat Adult's public account to reprint this article.

Many people will encounter deadlock problems during multi-threaded development. Deadlock problems are also frequently asked questions during interviews. Here we introduce how to use gdb+python scripts in C++ to debug deadlock problems, and how to detect deadlocks during program execution.

First, let's introduce what deadlock is. Let's look at the definition in Wikipedia:

Deadlock (English: Deadlock), also translated as deadlock, is a term in computer science. When two or more computing units are waiting for each other to stop running in order to obtain system resources, but neither of them exits early, it is called deadlock. In a multi-tasking operating system, the operating system must solve this problem in order to coordinate different processes and whether they can obtain system resources in order to make the system operate.

Wikipedia introduces process deadlock. Deadlock can also occur in multi-threading. The same principle applies, so I will not introduce it in detail here.

Four conditions for deadlock

• No preemption: System resources cannot be forced to exit from a process (thread), and already acquired resources cannot be preempted before they are used up.
• Hold and wait: When a process (thread) is blocked for requesting resources, it holds on to the resources it has obtained.
• Mutual exclusion: Resources can only be allocated to one process (thread) at a time and cannot be shared by multiple processes (threads).
• Circular waiting: A series of processes (threads) hold resources required by other processes (threads).

Deadlock will only occur if the above four conditions are met at the same time. To eliminate deadlock, you only need to destroy any one of the conditions.

How to debug multithreaded deadlock issues

Most of the reasons for multi-threaded deadlocks are due to inconsistent locking orders in multiple threads. See the following code that will cause deadlock:

 using std::cout; 
 
 std::mutex mutex1;
 std::mutex mutex2;
 std::mutex mutex3; 
 
 void FuncA() {
   std::lock_guard<std::mutex> guard1(mutex1);
   std::this_thread::sleep_for(std::chrono::seconds(1));
   std::lock_guard<std::mutex> guard2(mutex2);
   std::this_thread::sleep_for(std::chrono::seconds(1));
 } 
 
 void FuncB() {
   std::lock_guard<std::mutex> guard2(mutex2);
   std::this_thread::sleep_for(std::chrono::seconds(1));
   std::lock_guard<std::mutex> guard3(mutex3);
   std::this_thread::sleep_for(std::chrono::seconds(1));
 } 
 
 void FuncC() {
   std::lock_guard<std::mutex> guard3(mutex3);
   std::this_thread::sleep_for(std::chrono::seconds(1));
   std::lock_guard<std::mutex> guard1(mutex1);
   std::this_thread::sleep_for(std::chrono::seconds(1));
 } 
 
 int main() {
 std::thread A(FuncA);
 std::thread B(FuncB);
 std::thread C(FuncC); 
 
   std::this_thread::sleep_for(std::chrono::seconds(5)); 
 
 if (A.joinable()) {
 A. join ();
 }
 if (B.joinable()) {
 B. join ();
 }
 if (C.joinable()) {
 C. join ();
 }
 cout << "hello\n" ;
 return 0;
 }

As shown in the figure:

• Thread A already holds mutex1 and wants to apply for mutex2. It can release mutex1 and mutex2 only after obtaining mutex2. At this time, mutex2 is occupied by thread B.
• Thread B already holds mutex2 and wants to apply for mutex3. It can release mutex2 and mutex3 only after obtaining mutex3. At this time, mutex3 is occupied by thread C.
• Thread C already holds mutex3 and wants to apply for mutex1. It can release mutex3 and mutex1 only after obtaining mutex1. At this time, mutex1 is occupied by thread A.

The three threads refused to give in to each other, resulting in a deadlock.

Traditional gdb debugging multi-threaded deadlock method

(1) The attach id is associated with the process id where the deadlock occurs

 (gdb) attach 109
 Attaching to process 109
 [New LWP 110]
 [New LWP 111]
 [New LWP 112]
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1" .
 0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (threadid=140339109693184, thread_return=0x0, abstime=0x0,
 block=<optimized out >) at pthread_join_common.c:89
 89 pthread_join_common.c: No such file or directory.

(2)info threads View the information of all threads in the current process, and also view some stack information

 (gdb) info threads
 Id Target Id Frame
 * 1 Thread 0x7fa33ff10740 (LWP 109) "out" 0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (
 threadid=140339109693184, thread_return=0x0, abstime=0x0, block=<optimized out >) at pthread_join_common.c:89
 2 Thread 0x7fa33ec80700 (LWP 110) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 3 Thread 0x7fa33e470700 (LWP 111) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 4 Thread 0x7fa33dc60700 (LWP 112) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

Here you can see that threads 2, 3, and 4 are all in the lock_wait state. Basically, you can see whether there is a problem, but it is not certain. Here you need to use info threads multiple times to see if there are any changes in these threads. If there is no change for multiple times, then a deadlock has basically occurred.

(3)Thread id enters the specific thread

 (gdb) thread 2
 [Switching to thread 2 (Thread 0x7fa33ec80700 (LWP 110))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 135 ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.

(4) bt View the current thread stack information

 (gdb) bt
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 #1 0x00007fa33f9ea023 in __GI___pthread_mutex_lock (mutex=0x7fa340204180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78
 #2 0x00007fa340000fff in __gthread_mutex_lock(pthread_mutex_t*) ()
 #3 0x00007fa3400015b2 in std::mutex::lock() ()
 #4 0x00007fa3400016d8 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()
 #5 0x00007fa34000109b in FuncA() ()
 #6 0x00007fa340001c07 in void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) ()

The debugging is basically done here. For pthread_mutex_t, it can print which thread holds it. Then, by repeating steps 3 and 4, you can determine which threads and locks are deadlocked. For std::mutex, gdb cannot print specific mutex information, and you cannot see which thread holds the mutex. You can only enter the threads one by one to view the stack information.

However, is there any good way to locate deadlock for std::mutex in c++11?

have.

This can be considered as the fifth step, continue:

(5) Source loads the deadlock.py script

 (gdb) source -v deadlock.py
 Type "deadlock"   to detect deadlocks.

(6) Enter deadlock to detect deadlock

 (gdb) deadlock
 [Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S
 [Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S
 [Switching to thread 2 (Thread 0x7f5585e80700 (LWP 122))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S
 #1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78
 [Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 #1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f55874041c0 <mutex3>) at ../nptl/pthread_mutex_lock.c:78
 [Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]
 #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
 #1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404140 <mutex1>) at ../nptl/pthread_mutex_lock.c:78
 Found deadlock!
 Thread 2 (LWP 122) is waiting on pthread_mutex_t (0x00007f5587404180) held by Thread 3 (LWP 123)
 Thread 3 (LWP 123) is waiting on pthread_mutex_t (0x00007f55874041c0) held by Thread 4 (LWP 124)
 Thread 4 (LWP 124) is waiting on pthread_mutex_t (0x00007f5587404140) held by Thread 2 (LWP 122)

Looking directly at the results, the script detected the deadlock and pointed out which specific threads caused the deadlock. According to the output information, it can be clearly seen that the loop formed by the thread locks caused the deadlock. After finding out which specific threads formed the deadlock loop, you can check the stack information of the corresponding threads to see which locks are waiting.

The principle of deadlock detection script:

Let’s take the above picture as an example:

• Thread A already holds mutex1 and wants to apply for mutex2. It can release mutex1 and mutex2 only after obtaining mutex2. At this time, mutex2 is occupied by thread B.
• Thread B already holds mutex2 and wants to apply for mutex3. It can release mutex2 and mutex3 only after obtaining mutex3. At this time, mutex3 is occupied by thread C.
• Thread C already holds mutex3 and wants to apply for mutex1. It can release mutex3 and mutex1 only after obtaining mutex1. At this time, mutex1 is occupied by thread A.

As shown in the figure, three threads form a loop. Deadlock detection is to check whether there is a loop between threads. It is relatively easy to check the deadlock loop alone. Here, the concept of simple loop is also involved, because the normally detected loop may be a large loop, not the loop with the least number of weight vertices. If the number of vertices in the detected loop is large, the cost of positioning will increase. The script is a simple loop for detection. It involves the strongly connected component algorithm and the simple loop algorithm. It is relatively complicated and I will not introduce it in detail. The script comes from Facebook's folly library (I recommend you to check Google's abseil and Facebook's folly, both of which are good things). The code is too long to be listed in the article. If you need it, you can download it yourself or follow me and add me as a friend to send it to you.

How to detect deadlock in your code

The principle is the same as described above. During the thread locking process, a graph is always maintained to record the relationship between threads.

A->B, B->C, C->A