|
Running on Linux with pthreads. One thread (CBTcpProxyListenerThread below) adds
bufferevents (with option BEV_OPT_THREADSAFE) to an event_base. A second thread (CBTcpProxySenderThread) dispatches on the
event_base. bufferevents are removed from the event_base either by a
third thread or by the CBTcpProxySenderThread by calling bufferevent_free (without
calling bufferevent_disable first – is this a misuse?). The deadlock happens on pretty high load: ~6000 bufferevents
are added and removed per second. Each one is triggered for write ~10 times per
seconds (which gives ~60,000 triggeres per-second). Here’s the deadlock stack: Thread 1 (CBTcpProxyListenerThread) #0 0x00000030c1c0c758 in __lll_mutex_lock_wait () from
/opt/breach-proxy/lib64/libpthread.so.0 #1 0x00000030c1c087fa in _L_mutex_lock_908 () from
/opt/breach-proxy/lib64/libpthread.so.0 #2 0x00000030c1c08682 in pthread_mutex_lock () from
/opt/breach-proxy/lib64/libpthread.so.0 #3 0x00002aaab13039c8 in evthread_posix_lock (mode=0,
_lock=0x15f27b20) at evthread_pthread.c:71 #4 0x00002aaab10e0f05 in event_add (ev=0x15f27d58,
tv=0x0) at event.c:1815 #5 0x00002aaab10ed356 in _bufferevent_add_event
(ev=0x15f27d58, tv=0x15f27e40) at bufferevent.c:824 #6 0x00002aaab10edf3e in be_socket_enable
(bufev=0x15f27cc0, event=4) at bufferevent_sock.c:548 #7 0x00002aaab10ec296 in bufferevent_enable
(bufev=0x15f27cc0, event=4) at bufferevent.c:418 #8 0x00002aaaaaac398d in CBTcpProxySenderThread::AddEvents
(this=0x15f27490, pxcn=@0x15e649d0) at sender_thread.cpp:56 #9 0x00002aaaaaac4a83 in CBTcpProxy::AddEvents
(this=0x7fff061d3a70, pxcn=@0x15e649d0) at tcpproxy.cpp:465 #10 0x00002aaaaaabc920 in complete_pxcn_after_server_connect
(fd=236, what=4, arg=0x15e649d0) at listener_thread.cpp:234 #11 0x00002aaab10dedca in event_process_active_single_queue
(base=0x15c04650, activeq=0x15c066f0) at event.c:1232 #12 0x00002aaab10df35d in event_process_active
(base=0x15c04650) at event.c:1290 #13 0x00002aaab10df98c in event_base_loop (base=0x15c04650,
flags=0) at event.c:1483 #14 0x00002aaab10df3c6 in event_base_dispatch
(event_base=0x15c04650) at event.c:1317 #15 0x00002aaaaaabdae3 in CBTcpProxyListenerThread::run
(this=0x15c04510) at listener_thread.cpp:41 … Thread 2 (CBTcpProxySenderThread) #0 0x00000030c1c0c758 in __lll_mutex_lock_wait () from
/opt/breach-proxy/lib64/libpthread.so.0 #1 0x00000030c1c087fa in _L_mutex_lock_908 () from
/opt/breach-proxy/lib64/libpthread.so.0 #2 0x00000030c1c08682 in pthread_mutex_lock () from
/opt/breach-proxy/lib64/libpthread.so.0 #3 0x00002aaab13039c8 in evthread_posix_lock (mode=0,
_lock=0x15f27b20) at evthread_pthread.c:71 #4 0x00002aaab10e188e in event_del (ev=0x2aaabc0678a0)
at event.c:2015 #5 0x00002aaab10ee095 in be_socket_destruct
(bufev=0x2aaabc067890) at bufferevent_sock.c:581 #6 0x00002aaab10ec956 in
_bufferevent_decref_and_unlock (bufev=0x2aaabc067890) at bufferevent.c:600 #7 0x00002aaab10ed85f in bufferevent_writecb (fd=321,
event=4, arg=0x2aaabc067890) at bufferevent_sock.c:306 #8 0x00002aaab10df1a7 in event_persist_closure
(base=0x15f275e0, ev=0x2aaabc067928) at event.c:1184
#9 0x00002aaab10ded6d in event_process_active_single_queue
(base=0x15f275e0, activeq=0x15f27b00) at event.c:1227 #10 0x00002aaab10df35d in event_process_active
(base=0x15f275e0) at event.c:1290 #11 0x00002aaab10df98c in event_base_loop (base=0x15f275e0,
flags=0) at event.c:1483 #12 0x00002aaab10df3c6 in event_base_dispatch
(event_base=0x15f275e0) at event.c:1317 #13 0x00002aaaaaac3ab2 in CBTcpProxySenderThread::run
(this=0x15f27490) at sender_thread.cpp:35 … Thanks, Avi |