Re: [Libevent-users] Clarify new behavior?

[Ralph Castain <rhc@xxxxxxxxxxxx>]

On Oct 23, 2010, at 12:40 PM, Ralph Castain wrote:

> 
> On Oct 23, 2010, at 9:42 AM, Nick Mathewson wrote:
> 
>> On Sat, Oct 23, 2010 at 5:14 AM, Ralph Castain <rhc@xxxxxxxxxxxx> wrote:
>>> Hi folks
>>> 
>>> I successfully updated our libevent integration in Open MPI, but have encountered a problem with one use-case that used to work and now doesn't. Before proceeding to devise a fix, I just wanted to confirm that I accurately understand the issue.
>>> 
>>> The problem arises from this scenario:
>>> 
>>> 1. we receive a command via a message that we receive in a file descriptor event. We "push" the command message into a timer event (duration zero time) to help break a threading issue, and then return from the file descriptor event.
>>> 
>>> 2. the event library is called with LOOP_ONCE, causing the timer event to fire.
>>> 
>>> 3. from within the timer event, the command causes us to execute a procedure that results in us having to wait for another event to occur. We "block" in that position, running a loop that includes a call to progress the event library (i.e., a call to event_loop(LOOP_ONCE)).
>>> 
>>> This last step used to work just fine, yet now yields the following warning:
>>> 
>>> warn] event_base_loop: reentrant invocation.  Only one event_base_loop can run on each event_base at once.: Resource temporarily unavailable
>>> 
>>> and no progression of the event library occurs.
>>> 
>>> I can see a few ways around this problem, but all involve considerable code rewrite. Before I do so, I just wanted to confirm my understanding that this behavior changed (i.e., recursion is no longer allowed), and see if any alternative approaches to revising such codes are recommended.
>> 
>> Hi, Ralph!  I'll try to answer in more detail later on -- I can't
>> write too much now, but I wanted to drop you a line before you started
>> rewriting your code.
>> 
>> I think that the warning you hit may be overzealous.  It was meant to
>> detect errors that would previously pass silently and produce
>> hard-to-diagnose crashes, such as running event_base_dispatch on the
>> same event base from multiple threads at once.  But I think your use
>> case may either a) work right, or b) be trivial to make work right,
>> and if so, we should fix Libevent not to warn about it.
>> 
>> I'll look harder at the code here and try to see what's going on here,
>> but calling the loop from a callback really was working fine with
>> Libevent 1.4, then I think 2.0 should support it.
> 
> FWIW: I decided to just try commenting out the "return" following the warning in event.c, and everything works fine.
> 
> Not recommending that as a permanent fix, of course - it may just work due to our peculiar usage. After all, we are running single-threaded, and so you can't call the dispatch from multiple threads. But thought you might like to know.

Been thinking about this problem a bit. I wonder if the solution isn't simply to allow recursive calls to event_loop, but for the event library to ignore any event that is currently active? In other words, have the library go ahead and run thru all its events, setting a flag in the event struct when that event handler is called. Subsequent loops check all events -except- those whose flag is set.

When an event handler returns, you reset the flag, thus making it active again, or remove the event if it is non-persistent (might be risky in this scenario as popping up the recursion tree could yield stale data - can be handled, but complicated).

I haven't thought thru all the thread locking issues around setting/resetting that flag, but I think you'd be okay so long as you refuse to allow an event handler to block while waiting to re-trigger the same event it resides inside (either programmatically somehow or by fiat - i.e., "you will thread lock if you do something this stupid") - perhaps add a lock/condition to the event struct so you can lock only that event when checking/changing the flag. Some performance considerations here, of course.

Not sure you can be more re-entrant than that...

HTH
Ralph


> 
>> 
>> Thanks for the heads-up, and thanks to everybody else who's hunting
>> for regressions and other bugs in 2.0.8-rc.
>> 
>> yrs,
>> -- 
>> Nick
>> ***********************************************************************
>> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
>> unsubscribe libevent-users    in the body.
> 

***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.