Re: [Libevent-users] async writes for bufferevent_write/bufferevent_write_buffer?

[Mark Ellzey <mthomas@xxxxxxxxxx>]

On Thu, Jul 18, 2013 at 09:14:14AM -0600, J. Scott Dorr wrote:
> Thanks for your response!
> 
> I have an overly simplified, single threaded SSL test server (based on examples found in the libevent documentation) that has the following callback funcs:
> 
> static void
> ssl_readcb(struct bufferevent * bev, void * arg)
> {
>     struct evbuffer *in = bufferevent_get_input(bev);
> 
>     printf("Received %zu bytes\n", evbuffer_get_length(in));
>     printf("----- data ----\n");
>     printf("%.*s\n", (int)evbuffer_get_length(in), evbuffer_pullup(in, -1));
> 
>     bufferevent_write_buffer(bev, in);
>     printf("::: wrote to the buffer\n");
>     sleep(2);
>     printf("::: done sleeping\n");
> }
> 
> static void
> ssl_writecb(struct bufferevent * bev, void * arg)
> {
>     printf("writecb: in\n");
> }
> 
> 
> I connect to the server with the following:
> 
> openssl s_client -connect 0:9999
> 
> 
> With the code as written above, because of the 'sleep(2)', I would expect the following in a fully async model:
> 
> type 'hello' on the client
> see the server receive 'hello' by printing the '---- data ----', etc lines
> see the server print '::: wrote to the buffer'
> 2 seconds to elapse
> see the server print '::: done sleeping'
> see the client receive the echo 'hello'
> server prints 'writecb: in'
> 
> Instead, this happens:
> 
> type 'hello' on the client
> see the server receive 'hello' by printing the '---- data ----', etc lines
> see the server print '::: wrote to the buffer'
> see the client receive the echo 'hello'
> 2 seconds elapse
> see the server print '::: done sleeping'
> server prints 'writecb: in'
> 
> 
> The write is being sent to the client immediately from within the readcb.   Please let me know if I'm doing something wrong, but I'm not sure how to interpret this behavior other than there's at least one synchronous write happening from within the read handler?
> 

A little about deferred callbacks:

When you do something like bufferevent_write(), the actual underlying
protocol write is not executed immediately; it is queued up to be run in
the next iteration of the event loop. So if you do a sleep() in your
callback, you will not drop back to the main loop until you return.
Though even without the sleep, the write will not be done until the next
loop.

Another thing you must take into account is OpenSSL itself, and the way
it deals with non-blocking IO. A write to a non-blocking ssl socket can
return one of several different statuses. For example, SSL_write() can
return SSL_ERROR_WANT_READ. This means you have to stop writing, and
start reading. In order to reduce potential resource exhaustion, the
read is not done until the next iteration of the loop. Once that read
has succeeded, SSL_write() is attempted again.

A final thing you have to look at is how your operating system deals
with network IO. Say, for example, your operating system has TCP nagle
enabled, sending a small payload of "hello" may have a bit of latency
associated with it since nagle is going to wait for a set amount of time
for more data to buffer up into one packet.

In your test code above, I suggest turning off deferred. I would also
suggest using strace / dtruss if available to watch for read / write / recv /send.

***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.