I'm running into a deadlock using libevent. I've been trying to figure out why or where it is happening for the last 3 days but have failed to do so. The deadlock occurs with the event base responsible for sending network packets. Below is a gdb output showing the deadlocked threads.
Thread 32 (Thread 27389):
#0 0xb7713424 in __kernel_vsyscall ()
#1 0xb76ae169 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:142
#2 0xb76a95d9 in _L_lock_839 () from /lib/libpthread.so.0
#3 0xb76a946b in __pthread_mutex_lock (mutex=0x8749930) at pthread_mutex_lock.c:82
#4 0x080c10ed in event_add (ev=0x87622f8, tv=0x0) at event.c:1886
#5 0x08088831 in server_send_outgoing_packets (arguments=0xbfa7a79c) at protocol.c:2450
#6 0xb76a6cc9 in start_thread (arg=0xb4e65b70) at pthread_create.c:304
#7 0xb75f669e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Thread 29 (Thread 27392):
#0 0xb7713424 in __kernel_vsyscall ()
#1 0xb76ae169 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:142
#2 0xb76a95d9 in _L_lock_839 () from /lib/libpthread.so.0
#3 0xb76a946b in __pthread_mutex_lock (mutex=0x8749930) at pthread_mutex_lock.c:82
#4 0x080c1757 in event_del (ev=0x898f678) at event.c:2108
#5 0x080c1b3e in event_free (ev=0x898f678) at event.c:1739
#6 0x0807467b in destruct_client (arguments=0xb2dc2890) at destruct.c:62
#7 0x0805ef50 in iterator_remove (iterator=0xb2853038) at linked-list.c:351
#8 0x0807f404 in server_manager_worker (arguments=0xbfa7a6e4) at protocol.c:271
#9 0xb76a6cc9 in start_thread (arg=0xb4b62b70) at pthread_create.c:304
#10 0xb75f669e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Thread 24 (Thread 27382):
#0 0xb7713424 in __kernel_vsyscall ()
#1 0xb76ae169 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:142
#2 0xb76a95d9 in _L_lock_839 () from /lib/libpthread.so.0
#3 0xb76a946b in __pthread_mutex_lock (mutex=0x8749930) at pthread_mutex_lock.c:82
#4 0x080c1757 in event_del (ev=0xb27fd068) at event.c:2108
#5 0x080c1b3e in event_free (ev=0xb27fd068) at event.c:1739
#6 0x08074b0d in destruct_server (arguments=0xb2732558) at destruct.c:181
#7 0x0805ef50 in iterator_remove (iterator=0xade4f080) at linked-list.c:351
#8 0x08098118 in client_connector_connection_manager_worker (arguments=0xbfa7a614) at protocol.c:6064
#9 0xb76a6cc9 in start_thread (arg=0xb556cb70) at pthread_create.c:304
#10 0xb75f669e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Thread 21 (Thread 27350):
#0 evmap_io_active (base=0x872b890, fd=247, events=36) at evmap.c:399
#1 0x080c84ae in epoll_dispatch (base=0x872b890, tv=0xb6e1c334) at epoll.c:436
#2 0x080c2914 in event_base_loop (base=0x872b890, flags=<value optimized out>) at event.c:1537
#3 0x080a02a9 in send_outgoing_packets (arguments=0xbfa7a80c) at worker.c:36
#4 0xb76a6cc9 in start_thread (arg=0xb6e1cb70) at pthread_create.c:304
#5 0xb75f669e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Thread 7 (Thread 27383):
#0 0xb7713424 in __kernel_vsyscall ()
#1 0xb76ae169 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:142
#2 0xb76a95d9 in _L_lock_839 () from /lib/libpthread.so.0
#3 0xb76a946b in __pthread_mutex_lock (mutex=0x8749930) at pthread_mutex_lock.c:82
#4 0x080c10ed in event_add (ev=0xb27fd068, tv=0x0) at event.c:1886
#5 0x08095eb8 in client_connector_send_outgoing_packets (arguments=0xbfa7a7b8) at protocol.c:5471
#6 0xb76a6cc9 in start_thread (arg=0xb546bb70) at pthread_create.c:304
#7 0xb75f669e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Thread 21 calls event_base_loop(send_event_base, EVLOOP_NONBLOCK). The other threads are either trying to add or delete events from the event base. For some reason it seems after some random time Thread 21 never relinquishes the send_event_base lock, therefore all other threads get stuck trying to acquire the lock!!!!
I first thought that my callbacks could be getting deadlocked, but I've ensured this is not the case by having every callback print a message on entry, and a message on exit.
I'm calling evthread_use_pthreads() before calling any libevent function.
Any help or insight would be much appreciated.