Re: [Libevent-users] Bufferevent client read callback sporadically not called

[Nick Mathewson <nickm@xxxxxxxxxxxxx>]

On Fri, May 11, 2012 at 5:07 AM, Alexander Klauer
<alexander.klauer@xxxxxxxxxxxxxx> wrote:
> Hi,
>
> I'm new to libevent and have the following problem with bufferevents.
> I have a server talking to many clients. Very occasionally (when the
> number of clients is around 1000 or larger), the server writes a
> message to the client, which is read by the client (strace shows that
> readv() is successfully called) but the read callback of the client is
> not invoked. Subsequently, both server and client side of the
> connection wait for input in epoll_wait() until timeout.
>
> The problem exhibits itself with libevent 2.0.16, 2.0.19 and the 2.1.1
> alpha. I'm using a default event base (level-triggered I/O) with
> default watermarks. Surely, I must be making a simple mistake
> somewhere, like introducing a race condition, but I can't find it.
>
> I made a simple program which demonstrates the behaviour:
>
> http://pastebin.com/FaCuCwU5
>
> Compile with gcc -std=gnu99 -Wall -pedantic test.c -levent and run
> with ./a.out [number of clients].
> For me, with 500 clients, everything is fine, while with 1000 clients,
> a few connections stall. Make sure you set your file descriptor limit
> sufficiently high.
>
> Any help is appreciated.

So, there are a couple of issues in the code to look at, and a couple
of points in the Libevent code where you could start debugging.

Issues in your example:

* In server_read_cb, you do an evbuffer_add() to queue data to be sent
followed immediately by a bufferevent_free() on the same buffer.
That's not a safe pattern: bufferevent_free() does not guarantee how
much data (if any) is sent before the bufferevent gets closed.

* In server_read_cb, you don't handle partial writes.  There is no
guarantee that the entire message from the client, including the
newline, will arrive in a single readv call.  (When it's all on
localhost, it *should* work ok with a few small messages, but it's not
a habit to get into)

* You're using LEV_OPT_DEFERRED_ACCEPT, but your protocol doesn't
entitle you to do that.  LEV_OPT_DEFERRED_ACCEPT tells the OS kernel
that it does not have to not treat the connection as having been
accepted until the client actually writes something.  Your clients
don't write anything until they *get* a message, so Linux doesn't need
to tell Libevent "hey, you could accept some more sockets" until they
do.

When I fixed the first and third issue above, your example started
working for me.  I tested it with 2048 pairs of sockets.

yrs,
-- 
Nick
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.