by default, a pthread mutex is not recursive. When a thread trying to lock the same mutex that itself is already holding, the thread will stuck at second locking forever, waiting for the owner to release it ( the owner is itself and itself is busy trying to lock it again, and will never get a chance to unlock it ).
above is a typical scenario of so called "self deadlock", "recursive deadlock", “self lockup".
General rules of writing "recursive deadlock avoidance" code are:
1. make the mutex internal data structure, never expose it to applications.
2. make sure no recursive call within your own code when the call chain involves a lock. For example, we can easily avoid API "a" calling API "a", but it is also important to avoid API "a" calling API "b", which calling API "a". That is, we need to avoid the following call chain if there is a lock involved in either "api_a" or "api_b".
"api_a" ---> ”api_b" ----> "api_a"
3. It is easy to avoid recursive call chain via a "call chain graph" within your own code, however, it is not so easy when two libraries written by different parties have cross reference. E.g., what if "api_a" is from liba.so, but "api_b" is from libb.so? It is hard to avoid such recursion. The best way is just to avoid cross-dependency between two ( or more than two ) libraries.
I recently hit the scenario #3 above with three libraries involved.
My thread had corrupted memory and when it tried to malloc() again after the corruption, it self locked up wit the following call chain:
#0 0x00007f6711224d2b in pthread_once () from /lib64/libpthread.so.0
#1 0x00007f6711536b84 in backtrace () from /lib64/libc.so.6
#2 0x00007f67114a884b in __libc_message () from /lib64/libc.so.6
#3 0x00007f67114ae1c3 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007f67114b214f in _int_malloc () from /lib64/libc.so.6
#5 0x00007f67114b2ce1 in malloc () from /lib64/libc.so.6
#6 0x00007f6713274c92 in local_strdup () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f6713278654 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f6713282a44 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f671327e1b6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f67132824fa in _dl_open () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f671155ed30 in do_dlopen () from /lib64/libc.so.6
#12 0x00007f671327e1b6 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#13 0x00007f671155ee87 in __libc_dlopen_mode () from /lib64/libc.so.6
#14 0x00007f6711536a55 in init () from /lib64/libc.so.6
#15 0x00007f6711224d33 in pthread_once () from /lib64/libpthread.so.0
#16 0x00007f6711536b84 in backtrace () from /lib64/libc.so.6
#17 0x00007f67114a884b in __libc_message () from /lib64/libc.so.6
#18 0x00007f67114ae1c3 in malloc_printerr () from /lib64/libc.so.6
#19 0x00007f67114b214f in _int_malloc () from /lib64/libc.so.6
#20 0x00007f67114b2ce1 in malloc () from /lib64/libc.so.6
To understand the above call chain, let' start with frame #17:
#17 0x00007f67114a884b in __libc_message () from /lib64/libc.so.6
because of the memory corruption, libc "__libc_message" decides to print out a message to the user, the source code is here:
sysdeps/unix/sysv/linux/libc_fatal.c
51 __libc_message (int do_abort, const char *fmt, ...)
52 {
53 va_list ap;
54 va_list ap_copy;
55 int fd = -1;
56
57 va_start (ap, fmt);
58 va_copy (ap_copy, ap);
170 if (do_abort)
171 {
172 if (do_abort > 1 && written)
173 {
174 void *addrs[64];
175 #define naddrs (sizeof (addrs) / sizeof (addrs[0]))
176 int n = __backtrace (addrs, naddrs);
177 if (n > 2)
178 {
179 #define strnsize(str) str, strlen (str)
180 #define writestr(str) write_not_cancel (fd, str)
181 writestr (strnsize ("======= Backtrace: =========\n"));
182 __backtrace_symbols_fd (addrs + 1, n - 1, fd); ================> trying to do a backtrace()
183
184 writestr (strnsize ("======= Memory map: ========\n"));
185 int fd2 = open_not_cancel_2 ("/proc/self/maps", O_RDONLY);
186 char buf[1024];
187 ssize_t n2;
188 while ((n2 = read_not_cancel (fd2, buf, sizeof (buf))) > 0)
189 if (write_not_cancel (fd, buf, n2) != n2)
190 break;
191 close_not_cancel_no_status (fd2);
192 }
193 }
194
195 /* Terminate the process. */
196 abort ();
197 }
198 }
at stack frame #16, backtrace() tries to call pthread_once(), why? well, it calls pthread_once with a "init" function, which is doing something special:
#16 0x00007f6711536b84 in backtrace () from /lib64/libc.so.6
sysdeps/x86_64/backtrace.c
94 int
95 __backtrace (array, size)
96 void **array;
97 int size;
98 {
99 struct trace_arg arg = { .array = array, .cfa = 0, .size = size, .cnt = -1 };
100 #ifdef SHARED
101 __libc_once_define (static, once);
102
103 __libc_once (once, init); =====> this "init()" is a special function for backtrace(), only backtrace() needs it.
104 if (unwind_backtrace == NULL)
105 return 0;
106 #endif
107
108 if (size >= 1)
109 unwind_backtrace (backtrace_helper, &arg);
110
111 /* _Unwind_Backtrace seems to put NULL address above
112 _start. Fix it up here. */
113 if (arg.cnt > 1 && arg.array[arg.cnt - 1] == NULL)
114 --arg.cnt;
115 return arg.cnt != -1 ? arg.cnt : 0;
116 }
The special "init()” for backtrace() has its own purpose, “__libc_once()" is ”pthread_once".
let's see what is so special about this "init()":
sysdeps/x86_64/backtrace.c
49 static void
50 init (void)
51 {
52 libgcc_handle = __libc_dlopen ("libgcc_s.so.1");
53
54 if (libgcc_handle == NULL)
55 return;
56
57 unwind_backtrace = __libc_dlsym (libgcc_handle, "_Unwind_Backtrace");
58 unwind_getip = __libc_dlsym (libgcc_handle, "_Unwind_GetIP");===> trying to using libgcc to get call chain.
59 if (unwind_getip == NULL)
60 unwind_backtrace = NULL;
61 unwind_getcfa = (__libc_dlsym (libgcc_handle, "_Unwind_GetCFA")
62 ?: dummy_getcfa);
63 }
It trying to load "libgcc" to get callchain IP (instruction pointer)!
Of course, we want to load "ligbcc" only once, hence using pthread_once().
The problem with the above "init()" is that the "__libc_dlopen" will need malloc(), and the whole call chain started with calling "malloc()", hence the recursion.
let's take a simplified view of the original call chain:
#5 0x00007f67114b2ce1 in malloc () from /lib64/libc.so.6
#10 0x00007f67132824fa in _dl_open () from /lib64/ld-linux-x86-64.so.2
#14 0x00007f6711536a55 in init () from /lib64/libc.so.6
#15 0x00007f6711224d33 in pthread_once () from /lib64/libpthread.so.0
#16 0x00007f6711536b84 in backtrace () from /lib64/libc.so.6
#20 0x00007f67114b2ce1 in malloc () from /lib64/libc.so.6
Above is a simplified view of the original call chain. we can see it starts with "malloc()", inside "malloc()", we are calling "malloc()" again later on, hence the "recursive deadlock".
The reason why a lock is involved is that pthread_once always needs a lock:
nptl/pthread_once.c
28 __pthread_once (once_control, init_routine)
29 pthread_once_t *once_control;
30 void (*init_routine) (void);
31 {
32 /* XXX Depending on whether the LOCK_IN_ONCE_T is defined use a
33 global lock variable or one which is part of the pthread_once_t
34 object. */
35 if (*once_control == PTHREAD_ONCE_INIT)
36 {
37 lll_lock (once_lock, LLL_PRIVATE); ====> pthread_once() needs this lock, interesting this is a lock for all pthread_once caller. this is arguable.
38
39 /* XXX This implementation is not complete. It doesn't take
40 cancelation and fork into account. */
41 if (*once_control == PTHREAD_ONCE_INIT)
42 {
43 init_routine ();
44
45 *once_control = !PTHREAD_ONCE_INIT;
46 }
47
48 lll_unlock (once_lock, LLL_PRIVATE);
49 }
50
51 return 0;
52 }
It is very interesting to notice that the above recursive lockup is arguably a bug inside glibc code, and it was reported as early as 2005, and was fixed inside malloc() ( part of glibc ) in 2015:
https://sourceware.org/bugzilla/show_bug.cgi?id=956