[Libevent-users] Tricky bug when using libevhtp/libevent-openssl

[JiaQiang Xu <xjqkilling@xxxxxxxxx>]

Hi,

I'm writing a http server with libevhtp and libevent-openssl.
I met an assert failure after responsing to some GET requests.

The trace back looks like

(gdb) bt
#0  0x00007f5185578425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f518557bb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f518868bbdc in event_exit (errcode=24558) at log.c:79
#3  0x00007f518868bc8c in event_errx (eval=-559030611, fmt=<optimized
out>) at log.c:136
#4  0x00007f5188681b35 in _evbuffer_decref_and_unlock
(buffer=0x7f51700127f0) at buffer.c:544
#5  0x00007f5188685ae9 in _bufferevent_decref_and_unlock
(bufev=0x7f51700051f0) at bufferevent.c:629
#6  0x00007f518867eb86 in event_process_active (base=<optimized out>)
at event.c:1391
#7  event_base_loop (base=0x7f51700008f0, flags=0) at event.c:1621
#8  0x0000000000428bce in _evthr_loop ()
#9  0x00007f5187309e9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f5185635cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()
(gdb) f 4
#4  0x00007f5188681b35 in _evbuffer_decref_and_unlock
(buffer=0x7f51700127f0) at buffer.c:544
544 buffer.c: No such file or directory.
(gdb) p *buffer
$1 = {first = 0x7f51700068b0, last = 0x7f5170005440, last_with_datap =
0x7f51700127f0, total_len = 0, n_add_for_cb = 0, n_del_for_cb = 0,
  lock = 0x0, own_lock = 0, freeze_start = 0, freeze_end = 0,
deferred_cbs = 1, flags = 0, cb_queue = 0x7f5170000a00, refcnt = 0,
  deferred = {cb_next = {tqe_next = 0x7f5170012840, tqe_prev =
0x7f5170012840}, queued = 0,
    cb = 0x7f5188681e50 <evbuffer_deferred_callback>, arg =
0x7f51700127f0}, callbacks = {tqh_first = 0x0, tqh_last =
0x7f5170012868},
  parent = 0x7f51700051f0}
(gdb) f 5
#5  0x00007f5188685ae9 in _bufferevent_decref_and_unlock
(bufev=0x7f51700051f0) at bufferevent.c:629
629 bufferevent.c: No such file or directory.
(gdb) p *bufev
$2 = {ev_base = 0x7f51700008f0, be_ops = 0x7f518918b580, ev_read =
{ev_active_next = {tqe_next = 0x7f5170005288,
      tqe_prev = 0x7f5170000d30}, ev_next = {tqe_next = 0x0, tqe_prev
= 0x7f5170002380}, ev_timeout_pos = {ev_next_with_common_timeout = {
        tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1},
ev_fd = 77, ev_base = 0x7f51700008f0, _ev = {ev_io = {ev_io_next = {
          tqe_next = 0x0, tqe_prev = 0x7f5170012e40}, ev_timeout =
{tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {
          tqe_next = 0x0, tqe_prev = 0x7f5170012e40}, ev_ncalls = 0,
ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 128,
    ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec =
0, tv_usec = 0},
    ev_callback = 0x7f5188f8a390 <be_openssl_readeventcb>, ev_arg =
0x7f51700051f0}, ev_write = {ev_active_next = {tqe_next = 0x0,
      tqe_prev = 0x7f5170000d30}, ev_next = {tqe_next =
0x7f5170005200, tqe_prev = 0x7f5170002380}, ev_timeout_pos = {
      ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev =
0x0}, min_heap_idx = -1}, ev_fd = 77, ev_base = 0x7f51700008f0,
    _ev = {ev_io = {ev_io_next = {tqe_next = 0x7f5170005200, tqe_prev
= 0x7f5170012e40}, ev_timeout = {tv_sec = 0, tv_usec = 0}},
      ev_signal = {ev_signal_next = {tqe_next = 0x7f5170005200,
tqe_prev = 0x7f5170012e40}, ev_ncalls = 0, ev_pncalls = 0x0}},
    ev_events = 20, ev_res = 4, ev_flags = 128, ev_pri = 0 '\000',
ev_closure = 2 '\002', ev_timeout = {tv_sec = 0, tv_usec = 0},
    ev_callback = 0x7f5188f8a0e0 <be_openssl_writeeventcb>, ev_arg =
0x7f51700051f0}, input = 0x7f5170000e00, output = 0x7f51700127f0,
  wm_read = {low = 0, high = 0}, wm_write = {low = 0, high = 0},
readcb = 0x424b60 <_evhtp_connection_readcb>,
  writecb = 0x424d60 <_evhtp_connection_writecb>, errorcb = 0x424ae0
<_evhtp_connection_eventcb>, cbarg = 0x22e7420, timeout_read = {
    tv_sec = 0, tv_usec = 0}, timeout_write = {tv_sec = 0, tv_usec =
0}, enabled = 6}

The libevent I linked to is optimized, so the line numbers may not be accurate.

If I understand correctly, it's caused by incorrectly unrefing the
evbuffer of the bufferevent. But I checked my code and the code of
libevhtp, they only call bufferevent_free() to release a buffer event
and  don't operate on the underlying evbuffer directly. And the
evbuffer struct looks sound except for incorrect refcnt. So I don't
think it's memory corruption error.

This may be caused by some bug inside libevent-openssl or I'm not
using it correctly. One special thing I do is I use deferred callbacks
on the evbuffer so that my write callback is not called recursively by
libevent-openssl.

Any clue on what happened?

Thanks in advance.

Jiaqiang
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.