Re: [Libevent-users] Multi-threading question

[David Goulet <dgoulet@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
NotDashEscaped: You need GnuPG to verify this message



Programmist Setevik:
> I have a  "multiplexed multi-threaded" network IO program  that
> been working just fine for a long while. It is a typical hand-coded
> epoll-driven,  as opposed to libevent-driven, model that I am 
> thinking about converting to libevent. Just a literal conversion
> is what I have in mind.
> 
> I am really pleased with how my logic been working in this very
> flexible non-blocking and scale-able manner and I want to see if I
> can replicate it with libevent2, while making my life a bit easier
> by delegating this net IO minutiae to libevent.
> 
> Here's my question. Can I have:
> 
> 1) "Main" thread  that does something like this, (pseudo code):
> 
> for ( all configured listen ports)  { FD = socket() bind() set
> non-block() listen add FD to a list
> 
> }
> 
> for (configurable # of worker threads) {
> 
> ctx = malloc() ctx->ev_base = setup event_base add each FD ,
> register it with ctx->ev_base  // mark the FD as "listener" FD so
> we know to acept() on EV_READ start worker thread, passing it the
> ctx // so that each thread has its own ev_base
> 
> } ... do whatever, w/o ever touching any of the ev_bases and/or
> accesing any of the sockets/FD that are registered with ev_bases 
> ...
> 
> 2) Worker threads, in their run logic
> 
> do the regular loop over *their own* ev_base, accepting new conns,
> registering them as needed with * their own ev_base*,  for
> read/write/error callbacks and doing the regular net IO stuff
> 
> 
> #####################
> 
> Each thread only accesses the ev_base that was passed to it, never 
> seeing/accessing the ev_bases of its peers (other threads). Will
> this work with libevent on Linux ? Don't care about supporting 
> Win/Irix/Solaris/BSD etc. I understand, and my legacy code is a
> proof of, that in Linux you can have *same FD* added to multiple
> epoll sets, in different threads and OS will inherently lock it
> appropriately so that different thread don't step on ea-other - at
> least when using them for EV_READY leading to accept().
> 
> In other words, an FD that is registered with a number of epoll 
> selectors in a number of threads, when an IO is ready, will only
> be reported to _one_ selector/thread. It is a race and only one
> worker thread will win it, but it still works fine.

I work a lot with epoll() and this behavior is not correct. If you add
the same FD to two or more epoll sets, all epoll_wait() will return.

From epoll(7) man page:
----
Q2  Can two epoll instances wait for the same file descriptor? If so,
are events reported to both epoll file descriptors?

A2  Yes, and events would be reported to both. However, careful
programming may be needed to do this correctly.
----

I'm curious, were you able to get this behavior with some magic
fcntl/prctl or it's what you are looking to do ?

Thanks!
David

> 
> Any advice ?  Will it work as intended (I don't see how it
> shouldn't because behind the scenes, on Linux, libevent will use
> the very same epoll. Any gotchas ? Again, Linux is all I care
> about.
> 
> 
> I don't want to have a dedicated accept()or thread that then passes
> the accept()ed FDs to the workers (using the usual things, like a 
> synchronized queue, or passing FD over a socket etc).
> 
> 
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBCgAGBQJQdZl2AAoJEELoaioR9I02qOUH/1qtDOUXcUDydw9tlMpdqAvm
3u9T6sYBPZMdCStmqlO96onQcRY4Ibk8N0Ojp9ZPi6IqdL4XysydySr7WMEXNom0
YKUq+g2vK+NgMnPz+ya+WcpiBZ9dc5vGgIiJyGUQaETPqasPHTBbF7NaWlRbopu9
3lBdn7CVLH0wwYMhh4wnpqpU0dNVd2uAOo+iJHcw/sBaiWXWCizaP/t92+hsFkia
cChYoUAnebL2e7uFPwglBGtVfn1uImDe2xHYYJDh9YRNGEp7tE4JLItm1FHk5q3P
35oN8tUseSvs2QRoyHd9QCnvVK7aNf2n8SYpBIM4/0zeVrspv1t4t4qZwQ0VWJM=
=dqzV
-----END PGP SIGNATURE-----
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.