[Libevent-users] Deep recursive call of bufferevent write callback when openssl is in use

[JiaQiang Xu <xjqkilling@xxxxxxxxx>]

Hi,

I'm using libevent to write an http server for transferring files. I
use openssl for https.

I use bufferevent_openssl_socket_new() to create a bufferevent. Since
the files may be large,
I set up a write callback for the bufferevent to send the file in
small pieces. When the write callback
is called, I use bufferevent_write() to send 64KB data.

The code is basically as what follows

static void
write_data_cb (struct bufferevent *bev, void *ctx)
{
    SendData *data = ctx;
    char buf[64 * 1024];
    int n;

    n = readn (data->fd, buf, sizeof(buf));
    if (n < 0) {
        free_senddir_data (data);
    } else if (n > 0) {
        bufferevent_write (bev, buf, n);
        data->remain -= n;
    }
}

However, sometimes the server crashes. Part of the back trace:

#0 write_data_cb (bev=0x7f27f8034720, ctx=0x7f27f803b1a0) at access-file.c:105
#1 0x00007f2800aeb5ca in do_write (bev_ssl=0x7f27f8034720, atmost=) at
bufferevent_openssl.c:708
#2 0x00007f2800aebfc3 in consider_writing (bev_ssl=0x7f27f8034720) at
bufferevent_openssl.c:874
#3 0x00007f28001e3d11 in evbuffer_run_callbacks
(buffer=0x7f27f80108b0, running_deferred=) at buffer.c:486
#4 0x00007f28001e63c7 in evbuffer_add (buf=0x7f27f80108b0,
data_in=0x7f27fc70c1e0, datlen=) at buffer.c:1612
#5 0x00007f28001e8f80 in bufferevent_write (bufev=,
data=0x7f27f803b1a0, size=139809641416480) at bufferevent.c:379
#6 0x0000000000410da9 in write_data_cb (bev=0x7f27f8034720, ctx=) at
access-file.c:227
#7 0x00007f2800aeb5ca in do_write (bev_ssl=0x7f27f8034720, atmost=) at
bufferevent_openssl.c:708
#8 0x00007f2800aebfc3 in consider_writing (bev_ssl=0x7f27f8034720) at
bufferevent_openssl.c:874
#9 0x00007f28001e3d11 in evbuffer_run_callbacks
(buffer=0x7f27f80108b0, running_deferred=) at buffer.c:486
#10 0x00007f28001e63c7 in evbuffer_add (buf=0x7f27f80108b0,
data_in=0x7f27fc71c400, datlen=) at buffer.c:1612
#11 0x00007f28001e8f80 in bufferevent_write (bufev=,
data=0x7f27f803b1a0, size=139809641416480) at bufferevent.c:379
#12 0x0000000000410da9 in write_data_cb (bev=0x7f27f8034720, ctx=) at
access-file.c:227
......
#755 0x00007f28001e8f80 in bufferevent_write (bufev=,
data=0x7f27f803b1a0, size=139809641416480) at bufferevent.c:379
#756 0x0000000000410da9 in write_data_cb (bev=0x7f27f8034720, ctx=) at
access-file.c:227
#757 0x00007f2800aeb5ca in do_write (bev_ssl=0x7f27f8034720, atmost=)
at bufferevent_openssl.c:708
#758 0x00007f2800aebfc3 in consider_writing (bev_ssl=0x7f27f8034720)
at bufferevent_openssl.c:874
#759 0x00007f2800aec109 in be_openssl_writeeventcb (fd=, what=,
ptr=0x7f27f8034720)
at bufferevent_openssl.c:952
#760 0x00007f28001e1a9d in event_process_active_single_queue
(activeq=0x7f27f80013b0, base=) at event.c:1350
#761 event_process_active (base=) at event.c:1420
#762 event_base_loop (base=0x7f27f8000f70, flags=0) at event.c:1621

It's obvious that my write_data_cb() is called recursively. When the
stack is too deep, the server crashes.

After reading the code of libevent openssl, I found that a evbuffer
callback (be_openssl_outbuf_cb())
gets called whenever bufferevent_write() is called. This callback
function in turn writes out data to
ssl and call write_data_cb() again.

Is this a bug? I don't understand why be_openssl_outbuf_cb() is
necessary. Is it really necessary to send
the data immediately when it's added to the evbuffer?

Thanks.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.