Re: [Libevent-users] Mixing ssl bufferevents with normal events

[Nick Mathewson <nickm@xxxxxxxxxxxxx>]

On Tue, Aug 30, 2011 at 2:09 PM, Jack Scott Dorr <j.scott.dorr@xxxxxxxxx> wrote:
> I have a libevent based server that used to use a custom ssl wrapper to handle ssl traffic. This server takes ssl traffic on one end and speaks non-crypto on the other end.
>
> For a variety of reasons, the switch was made to use the built in ssl bufferevents to handle the ssl portion. The non-crypto traffic continues to use 'regular' events. They use the same event base.
>
> However, the server is behaving as if the bufferevents are acting at a higher priority than the normal events. Under sustained load, the normal events are just not being pulled off the queue, or being pulled off so slowly that the inbound network buffer to the server gets filled (errno 11/resource temporarily unavailable seen from clients connecting from the localhost via UDS).
>
> Has anyone seen anything like this before?  Is it considered bad form to mix bufferevents and non-bufferevents on the same event base?
>

Hi!  I haven't seen the situation you're describing, but it does sound
unpleasant.  It is supposed to be okay to mix bufferevent code with
non-bufferevent code.

What OS are you using?  Are the SSL bufferevents running as a filter
around socket bufferevents, or directly?  What flags are you creating
them with, if any?

I don't know what's going on in your case, but here are some ideas
that might help:

- Try using priorities on the event_base, and setting your regular
events to have more priority than the bufferevents.  This might fix
stuff, or it might make the situation worse.

- Look at the code that send out non-SSL data.  Is it doing so
aggressively?  The bufferevent code pretty much tries to read and
write as much as it can whenever it can, so if your non-bufferevent
code is only doing small reads and writes, it might get

- There's support for rate-limiting bufferevents.  Perhaps assigning
all the ssl bufferevents to a rate-limiting group with a reasonable
(but high) limit will make stuff better for now.  You can also control
how much data gets read and written by limiting how much you queue for
output, and assigning a high-watermark for reading.

- There have in the past been some bugs where you could repeatedly
trigger callbacks for the same thing in a loop.  I think we've fixed
all of those, but it might be worthwhile trying the "defer callbacks"
flag on the bufferevents flag to see whether it helps.  (If it does
help here, that's a problem in libevent, but at least then we'd know
where the problem was.)

- If you can trace down why this is happening, there might be a better
way to fix this in libevent.  I'd start by looking at profiles, to see
where the program is spending its time and resources.  Then I'd try to
see where the resources are going exactly -- are your callbacks
getting called infrequently?  Not at all?  Frequently, but they can't
write much?  etc.

hope this helps,
-- 
Nick
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.