Re: Re: Re: [Libevent-users] Cross-linked socket and openssl bufferevent throughput issue

["Thompson, Adam" <athompson@xxxxxxxxxxxxxx>]

On Fri, Mar 01, 2019 at 08:22:24PM +0300, Azat Khuzhin wrote:
> > The exact throughput is somewhat difficult to measure but it's around 300
> > to 500 kbps through the system.  There are other factors here but the
> > overall architecture is easily capable of getting into many mbps in testing.
> 
> This is *very slow* and this is not read(4k) problem, it is something
> way more than this. Maybe you can share minimal client/server/proxy
> that I can take a look at? (the best case scenario if you will provide
> server/client that I can connect to each other and see the throughput,
> for example if both of them will print received payload)

Unfortunately such a thing would be commercially sensitive information
as it would mean sharing details of our internal software architecture so I
regret that I can't do so.  I'm sorry for being somewhat unhelpful in this
regard.  However I can say that we are using a pair of pairs of bufferevents
so it is quite probable that small reads are amplified due to the overhead
of passing multiple smaller chunks of data through the system.

In addition there may well be other issues; I notice that there have been some
unrelated bug fixes around openssl bufferevents.  The reason for starting with
the 4K reads is that this is the most obvious source of throughput issues from
the strace output.  I'm also happy to conduct any other debugging you'd find
helpful to track down what's happening.  As for server and client we're
using Spirent to performance test the system however, as I said, a previous
version of this component demonstrates a much higher throughput figure.

> 
> > CPU seems fine (32 core server is being used for testing and it's no where
> > near 100% usage on any core).
> 
> I wrote about CPU just to underline that this will introduce some
> latency in compare to the plain bufferevent (of course I didn't mean
> that the problem is that huge that you will see 100% CPU consumption)

Agreed and appologies if it appears I was implying that that was what you
meant.

> 
> > > You can add an API for evbuffer to change the default size and create
> > > a pull request with your changes here [2]. Are you interested in this?
> > > Or should I?
> >
> > I'm interested however I suspect you're best placed to make the patch.  I'm
> > certainly happy to help where possible though.
> 
> I can help you with patches *if* you will dig into aspects that I just
> wrote about.
> 
> > Although, as I said, you're probably best placed to patch this one, are
> > there any guides to contributing to the project (i.e.  testing and coding requirements etc)
> > that I can look at?
> 
> Yes, but it does not contain a lot of details for now:
>   https://github.com/libevent/libevent/blob/master/CONTRIBUTING.md
> 
> And it will be shown to you (by github) on issue/pull request creation time

Ok, thanks, will have a look at the script it mentions in the future
however, for various reasons it'll be a while before I can put together a
patch.  As such, if you have an idea and can make the changes it'd probably
be far quicker for you to make the changes and I'll be more than happy to
test them in our environment to see what difference they make.  Again, sorry
for not being more helpful.

Regards,
Adam.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.