Re: [Libevent-users] multithreading problem

On Jan 8, 2013, at 12:55 AM, Björn K. wrote:

That piece of code should distribute the incoming connections over the
threads (I want to change that code later to select the thread with
the lowest number of handled connections). If I don't use this code
and use instead
bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx, ...
the problem remains.

Furthermore I don't think it's a problem of running out of memory. The
segmentation fault even occurs when there are just a few connections
to the server (once the third connection produced the crash).
I tried to get a segmentation fault during a run in valgrind. One time
I succeeded and got this:

==310== Invalid read of size 8
==310==    at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
==310==    by 0x41403F: bufferevent_enable (bufferevent.c:448)
==310==    by 0x4044A9: acceptcb (in /home/login/test/simple)
==310==    by 0x424325: listener_read_cb (listener.c:412)
==310==    by 0x41D889: event_process_active_single_queue (event.c:1471)
==310==    by 0x41E0E6: event_base_loop (event.c:1538)
==310==    by 0x4049E0: main (in /home/login/test/simple)
==310== Address 0x62531a0 is 464 bytes inside a block of size 560 free'd
==310==    at 0x4C240FD: free (vg_replace_malloc.c:366)
==310==    by 0x413E2A: bufferevent_decref_and_unlock_ (bufferevent.c:703)
==310==    by 0x41DAD4: event_process_active_single_queue (event.c:1476)
==310==    by 0x41E0E6: event_base_loop (event.c:1538)
==310==    by 0x404693: worker_function (in /home/login/test/simple)
==310==    by 0x54258C9: start_thread (pthread_create.c:300)
==310==

(And many similar messages. What they have in common is that some
memory is freed in the worker thread and the main thread tries to
access it.)

But I don't know why memory of the bufferevent can be freed in the
worker thread before the buffer event is enabled.

I think the key to solve these issues is change the design - do not use libevent apis from multiple threads except event_active().

The simplest solution would be to use libevhtp <https://github.com/ellzey/libevhtp/> which uses smart event based thread pool and looks very clean.

If you want more control, the simplest solution is to get a single process working, then fork few worker processes in the main process, all sharing the same listener file descriptor, bound in the main process. Then let the kernel do the load balancing for you. Successful projects like nginx and unicorn <https://github.com/blog/517-unicorn> work like this.

When I try the code you submitted (after some cleanup to make it compile and run), it does not respond when accessing with a browser. How are you stress testing this code?

2013/1/7 Mark Ellzey <mthomas@xxxxxxxxxx>:
On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:

at a second glance:

next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
working as intended?
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.