Re: [Libevent-users] Callback after all events from the current batch have been cleared

[Zack Weinberg <zackw@xxxxxxxxx>]

On Tue, Jun 26, 2012 at 4:07 PM, Nick Mathewson <nickm@xxxxxxxxxxxxx> wrote:
> On Jun 26, 2012 3:08 PM, "Zack Weinberg" <zackw@xxxxxxxxx> wrote:
>> By "deferred callbacks" do you mean the implementation of
>> BEV_OPT_DEFER_CALLBACKS, or something else? ÂThis program doesn't use
>> that, although maybe it should (I'm not at all clear on what that's
>> supposed to be for).
>
> Yes, that. They were a feature introduced in 2.0 to deal with crazy
> priority inversions and callback loops in some of the code paths. For
> example, without them it was pretty easy for a bufferevent_pair to blow the
> stack by having one of the pair's callbacks cause another one to get
> invoked.
>
> I believe at least one of the bufferevent types (pair) will automatically
> run with deferred callbacks; you should check that you're not using a type
> with that property.

The program uses only socket bufferevents.

>> By code inspection, in 2.0 (didn't check 2.1), I can see a way for it
>> to happen if BEV_OPT_DEFER_CALLBACKS *is* used: both variants of
>> bufferevent_run_deferred_callbacks potentially call several callbacks
>> on the same bufferevent without checking in between to see if one of
>> the callbacks has deallocated the bufferevent. Â(You've got some sort
>> of reference count in there, so that may be safe as far as the 'struct
>> bufferevent' itself, but there is no such protection for the callback
>> data.)
>
> Hm. Now *that's* a bug. Gotta think about the right fix there: a minimal fix
> would probably be better for 2.0, though my preferred strategy for 2.1 would
> probably be more invasive. Any thoughts or patches would be welcome.

Well, the first thing I'd try is to check the bufferevent reference
count after each callback and see if it has dropped to 1 (since the
function holds one reference itself) and bail out if so.  But I don't
feel I understand the reference counting / locking scheme well enough
to write code here.

> To narrow this down, what bufferevent types and other pieces of libevent are
> you using? And have you seen these issues only with bufferevent callbacks,
> or others too?

The only bufferevent type we use is socket bufferevents, created
variously with bufferevent_socket_connect(_hostname) and
bufferevent_socket_new; the latter case uses the fd provided to the
callback of an evconnlistener.

We also use evconnlisteners, timer events, evdns, bits of evutil, and
a small number of signal events (these monitor for conditions that
should cause orderly shutdown).

The crash seems to only happen with bufferevent callbacks, and in the
stack traces I have to hand, it is _bufferevent_run_readcb on the
stack immediately below the callback that was invoked with a
deallocated 'arg' pointer.  I think I might have also seen it with
_bufferevent_run_eventcb too but I don't seem to have preserved any of
those.  I'm not sure how much I _trust_ those stack traces - the
compiler seems to have been doing tail-call optimization, possibly on
both the library and the program.

zw
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.