Re: [Libevent-users] libevent2 and delete-after-close with kqueue

[Adrian Chadd <adrian@xxxxxxxxxxxxxxx>]

On 8 June 2014 21:04, Nick Mathewson <nickm@xxxxxxxxxxxxx> wrote:
> On Sat, Jun 7, 2014 at 3:41 PM, Adrian Chadd <adrian@xxxxxxxxxxxxxxx> wrote:
>> Hi all,
>>
>> I'm trying to chase down a bug with libevent2-trunk.
>>
>> The TL;DR is that I have ktrace traces from FreeBSD showing that an FD
>> delete event is being kqueued to a kqueue FD after the FD has been
>> closed in that thread. It's responsible for ENOTCAPABLE showing up.
>> Something like EBADF just drops through; ENOTCAPABLE currently causes
>> the IO loop to terminate. This is with a multi-threaded server, but
>> there's one event base per thread and one accept FD per event base.
>>
>> It _looks_ like the evhttp / bufferevent code is doing the right thing
>> - ie, it's deleting the events (which should delete them from the
>> kqueue list) before calling close(). But there's a bunch of places
>> that I have to actually go and check. Unfortunately "EV_DELETE"
>> entries are being left on the kqueue list and not removed. The only
>> way to know that these need to be removed is to actually call the
>> event deletion function from evutil.c when the socket is closed.
>> Teaching evutil_closesocket() would do it, but it would require an
>> eventbase. It also implies that a socket is only in one event base -
>> it's quite possible servers have an FD in multiple event bases. Ugh.
>>
>> What do people think?
>
> Ow, that's annoying.
>
> One thing to consider is that there's a parallel issue with closing
> fds with an epoll-based backend.  (Look at the documentation about the
> difference between the epoll and the epoll-with-changelist backends.)
> Possibilities that I can see include:
>
>   * Have a kqueue-without-changelist backend that flushes all
> EV_DELETEs immediately.
>
>   * Have an API like event_finalize that removes any pending
> EV_DELETEs as needed.
>
>   * Have a "delete event and close fd" API.
>
> Obviously, these possibilities are maybe not so good.  Let's see if we
> can think of others.

I think the "clean" thing to do is to have a "delete events for this
FD from the changelist in this event base" API.

If people run one FD per event base then it'll translate easily.

If people run a handful of shared FDs across all event bases (eg
accept sockets) then they'll have to add some serialisation to handle
that case.

We'd then have to liberally sprinkle it around the codebase - ie,
wherever evutil_closesocket() is called - but are those all guaranteed
to be used as "one FD per event base" ?


-a
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.