Re: [Libevent-users] libevent performance problem for detect the socket is readable

[Azat Khuzhin <a3at.mail@xxxxxxxxx>]

On Tue, Aug 18, 2015 at 01:35:05PM +0800, xiaoxin gao wrote:
> ïi all,
> 
>  I use libevent in my program, but I meet some performance problem.
> 
> I create one class named SocketEventBase. SocketEventBase is one thread and
> use event_base_dispatch() to check socket.
> 
> There is one list<socket*> in the SocketEventBase. Use SocketEventBase to
> listen all the socket in the list.
> 
> When the socket is ready to read, libevent framework will remove socket
> from the list and I will use another thread(named Handler) to deal the
> request by using
> the callback function.
> 
> After Handler complete the request, i will add the socket to event base
> again.
> 
> So during the run-time, there are lots of event will be remove from and add
> to
> the event base.
> 
> I try to use more event base to get better performance. I use
> handler-thread-pool-num to control the num of SocketEventBase. Assume
> handler-thread-pool-num is four, all requests will be dispatch to the four
> event base.

How do you schedule events between threads?

> When i don't use libevent, one thread will service one socket.

I.e. for every accept() you have thread?

> After i use libevent, i wish less threads can service same num of socket
> and will get better performance.
> 
> But i test my program as below.
> 
> I use 300 client threads to test my program.
> (It is a sysbench oltp test, and libevent is used inside a proxy, which is
> located between the sysbench and backend mysql)
> 
> When i don't use libevent, i get 10000~12000 tps.
> 
> After i use libevent, different handler-thread-pool-num get different
> result.
> 
> handler-thread-pool-num          tps
> 1                               1101.17
> 2                               2065.75
> 4                               3354.30
> 8                               3995.18
> 16                              4308.92
> 
> 
> Then i monitor my program like below:
> +----------------------+-----------+--------+
> | Monitor_point        | Cost_time | %      |
> +----------------------+-----------+--------+
> | session_other        | 37462     | 78.36  |
> | session_svc          | 47805     | 100.00 |
> | session_switch       | 484       | 1.01   |
> | session_work         | 9670      | 20.23  |
> +----------------------+-----------+-------+
> 
> session_svc : all the time of the requests be done
> session_switch : the time is when socket is ready to read, move request
> from SocketEventBase to Handler.
> session_work : the time is Handler deal the requests
> session_other : the time is from adding socket to SocketEventBase's list to
> find the socket is ready to read. (The time is also include waiting for
> sysbench client to send the request, but it still too much, cause the
> session_work is only 20%, which also
> includes the time for backend mysql handle the sql request and send back
> result )

Maybe your callbacks take a long time to run? (In this case event loop
will be idle, and it will handle new events only when you will return
from you callback).

> 
> I don't know why session_other needs so much time. Is libevent need so much
> time to find the socket is ready?

It don't need any time for this, since it uses polling, IOW kernel must
tell when the fd is ready to read, but yes it can have some overhead,
but it must be very small.

> some other test result as below:
> 
> client thread = 30
> handler-thread-pool-num=10
> tps 2696.23
> session_other 14.56%
> 
> client thread = 60
> handler-thread-pool-num=10
> tps 4323.21
> session_other 27.18%
> 
> client thread = 90
> handler-thread-pool-num=10
> tps 5019.76
> session_other 41.99%
> 
> client thread = 120
> handler-thread-pool-num=10
> tps 4893.41
> session_other 54.31%

Sorry for such a long delay, but can you provide some samples for you
benchmarks (maybe including some scripts to run with libevent and
without)?

Thanks,
Azat.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.