[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [Libevent-users] multithreading problem
Hey BjÃrn,
multithreading and libevent is a bit tricky (at least it was for me). I
would also suggest to change your design...
I developed a multithreaded websocket server based on libevent a year ago:
My approach was to create a single event base per thread. Each event base
in each thread listens to the same port. Each event base processes its
established websocket-connections. For inter-thread-communication i used
socketpairs between the event bases in the threads.
With this design i was able to achieve ~350.000 connections and a
throughput of about 90.000 websocket-messages per second with a ~85-90%
CPU load on a quadcore machine with 4 gb ram.
Filipe
> The posted code compiles on my machine without modifications. Thus I
> wonder why you have to do some cleanup.
> But the example code fails silently. Perhaps you don't run it with
> enough rights to bind to port 443?
>
> Most of the time I access the server with a browser and keep the
> reload button pressed. Sometimes I use the apache ab tool.
>
> I tried to understand how libevhtp works. If I get it right, all
> connections are handled in the main thread and only the payload is
> processed in a worker thread. For non-ssl connections this is fine but
> for example for a ssl-only static web page server I think I do not
> really profit from multithreading this way because all of the ssl
> stuff is done in the main thread.
>
> 2013/1/8 Nir Soffer <nirsof@xxxxxxxxx>:
>>
>> On Jan 8, 2013, at 12:55 AM, BjÃrn K. wrote:
>>
>> That piece of code should distribute the incoming connections over the
>> threads (I want to change that code later to select the thread with
>> the lowest number of handled connections). If I don't use this code
>> and use instead
>> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx,
>> ...
>> the problem remains.
>>
>> Furthermore I don't think it's a problem of running out of memory. The
>> segmentation fault even occurs when there are just a few connections
>> to the server (once the third connection produced the crash).
>> I tried to get a segmentation fault during a run in valgrind. One time
>> I succeeded and got this:
>>
>> ==310== Invalid read of size 8
>> ==310== at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
>> ==310== by 0x41403F: bufferevent_enable (bufferevent.c:448)
>> ==310== by 0x4044A9: acceptcb (in /home/login/test/simple)
>> ==310== by 0x424325: listener_read_cb (listener.c:412)
>> ==310== by 0x41D889: event_process_active_single_queue (event.c:1471)
>> ==310== by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310== by 0x4049E0: main (in /home/login/test/simple)
>> ==310== Address 0x62531a0 is 464 bytes inside a block of size 560
>> free'd
>> ==310== at 0x4C240FD: free (vg_replace_malloc.c:366)
>> ==310== by 0x413E2A: bufferevent_decref_and_unlock_
>> (bufferevent.c:703)
>> ==310== by 0x41DAD4: event_process_active_single_queue (event.c:1476)
>> ==310== by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310== by 0x404693: worker_function (in /home/login/test/simple)
>> ==310== by 0x54258C9: start_thread (pthread_create.c:300)
>> ==310==
>>
>> (And many similar messages. What they have in common is that some
>> memory is freed in the worker thread and the main thread tries to
>> access it.)
>>
>> But I don't know why memory of the bufferevent can be freed in the
>> worker thread before the buffer event is enabled.
>>
>>
>> I think the key to solve these issues is change the design - do not use
>> libevent apis from multiple threads except event_active().
>>
>> The simplest solution would be to use libevhtp
>> <https://github.com/ellzey/libevhtp/> which uses smart event based
>> thread
>> pool and looks very clean.
>>
>> If you want more control, the simplest solution is to get a single
>> process
>> working, then fork few worker processes in the main process, all sharing
>> the
>> same listener file descriptor, bound in the main process. Then let the
>> kernel do the load balancing for you. Successful projects like nginx and
>> unicorn <https://github.com/blog/517-unicorn> work like this.
>>
>> When I try the code you submitted (after some cleanup to make it compile
>> and
>> run), it does not respond when accessing with a browser. How are you
>> stress
>> testing this code?
>>
>>
>> 2013/1/7 Mark Ellzey <mthomas@xxxxxxxxxx>:
>>
>> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
>>
>>
>>
>> at a second glance:
>>
>>
>> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
>>
>> working as intended?
>>
>> ***********************************************************************
>>
>> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
>>
>> unsubscribe libevent-users in the body.
>>
>> ***********************************************************************
>> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
>> unsubscribe libevent-users in the body.
>>
>>
> ***********************************************************************
> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> unsubscribe libevent-users in the body.
>
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.