Re: [Libevent-users] Zero-copy http client?

[Thor Lancelot Simon <tls@xxxxxxxxx>]

On Thu, Dec 08, 2016 at 09:39:25AM -0500, Ryan Daum wrote:
> Hi there,
> 
> I have a feature request for the http client facility, or at least a
> request for advice if there's already a way to do what I want.
> 
> Profiling my application which uses libevent's http client piece to make a
> large number of GET requests for fairly large amounts of data I see the
> majority of the process's time is spent in memcpy.

I wrote an SSL-intercepting HTTP proxy for a network equipment vendor
using an early version of libevent -- but not so early that it lacked
HTTP support or the beginnings of bufferevent.

It didn't have any SSL or HTTPS support, so we considered adding that
and built a few prototypes.  Our conclusion was that the amount of
copying caused by bufferevent made using anything built on it a nonstarter
for our purpose -- we paid at least two copies in each direction (in the
front out of the proxy and out the back) which wasn't sustainable given
the memory bandwidth of our platform at the time and the need to handle
multiple Gbit/sec of traffic.  Modern platforms have vastly more memory
bandwidth, but modern networks are multiple *tens* of Gbit/sec of traffic
so your mileage may vary.

I ended up building a very thin, though not pretty, layer that had a single
buffer used for both Tx and Rx.  If we used hardware, we could receive and
decrypt directly into the buffer from which we'd transmit (zero copies);
with software SSL, we paid one copy to get the cleartext.  This
meant building datastructures, and utility code, that handled both the
protocol recordkeeping and the buffer recordkeeping in one shot, avoiding
the extra copies involved in all the abstraction.  It also meant getting
annoyingly intimate with OpenSSL in a few cases.  And no, I didn't get it
right the first time -- it took a couple of months of beta testing until
it was entirely stable.  Using bufferevent would have been much, much
less development time and certainly less debugging.

But copies are slow, while zero-copy code is hard to write.

Thor
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.