Re: [Libevent-users] Help with progress thread

[Nick Mathewson <nickm@xxxxxxxxxxxxx>]

On Tue, Nov 9, 2010 at 3:28 PM, Ralph Castain <rhc@xxxxxxxxxxxx> wrote:
> Hi folks
>
> I'm running into a problem that probably results from my ignorance. So I figured I would ask if someone can tell me what I'm doing wrong.
>
> I have a thread that loops the event library with the following call:
>
>        events += event_loop(mca_oob_tcp_component.event_base, EVLOOP_ONCE);
>
> In another thread, I create and add an event that is supposed to occur when a file descriptor is ready for "write":
>
>         event_set(mca_oob_tcp_component.event_base,
>                       &peer->peer_send_event,
>                       peer->peer_sd,
>                       EV_WRITE|EV_PERSIST,
>                       mca_oob_tcp_peer_send_handler,
>                       peer);
>         event_add(&peer->peer_send_event, 0);
>
> Note that the event_set and event_add can (and in this case, probably do) occur while I am in the event_loop function.
>
> What I find is that this event never gets fired. However, events that are set and added -before- going into the loop do fire.
>
> So my question is: given that this is a dynamic system, how do I get the event_loop to "see" new events? I want to block in the loop, so I don't really want to set NONBLOCK if I can avoid it (otherwise, I would have to add another blocking call somewhere to keep the thread from endlessly cycling).

So I'm assuming that you've got all the threading callbacks set up
(probably via evthread_use_pthreads()) before you created the event
base, so that evthread_make_base_notifiable() was called on the
event_base when you created it.  If that's not the case, that's
probably the problem there.

Otherwise, I think this one is an honest-to-goodness bug.

Here's what happens when you add an event from another thread.

The event_add locks the event_base's structures, makes the changes
necessary to add the event, and then notices that it isn't running in
the main thread, so it needs to "alert" the main thread to exit and
re-enter the dispatch function[*].  It does this by calling
evthread_notify_base(), which sends a byte down a pipe (on most Unix)
or a socketpair (on Windows), or by using an eventfd (on Linux) [**].
There's an (EVLIST_INTERNAL) event handler for this
pipe/socketpair/eventfd that notices that we've been "notified" so we
can drain the pipe/socketpair/eventfd.

Now here's where I think the bug is: that event counts as an active
event, and processing it gets counted as processing an event, so if
you're running the loop with EVLOOP_ONCE, the loop will exit right
after it notices that it should wake up, even though no user callbacks
were actually entered.

The logic is in event_base_loop():

    		if (N_ACTIVE_CALLBACKS(base)) {
			event_process_active(base);
			if (!base->event_count_active && (flags & EVLOOP_ONCE))
				done = 1;
                } ...

I think that the second "if" might be wrong.  We maybe want to
continue looping not only if there are active callbacks, but also if
the only active events that we processed were internal events.

With the current behavior, with EVLOOP_ONCE, I think adding the event
from another thread will exit the loop right away.  Is that consistent
with the behavior you're seeing?  If not, then the bug is possibly in
the notification stuff, though that part actually is fairly well
tested.

It's also possible that I'm writing this too late at night for me, and
I've deeply misanalyzed the code here.

(As an aside, we should probably also continue looping if
N_ACTIVE_CALLBACKS(base) is nonzero.)

[*] For backends like poll and select, we need to re-enter the
dispatch function after getting event changes so that we can call
select/poll with their new arguments.  For stuff like epoll and
kqueue, we need to process the changelist and re-enter the wait
function.
[**] Yes, I do indeed think that kqueue should use EVFILT_USER, but
that's a 2.1 thing.

-- 
Nick
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.