Re: [Libevent-users] Different behavior of my code

[William Ahern <william@xxxxxxxxxxxxxxxxxx>]

On Thu, Sep 08, 2011 at 09:40:02AM -0700, Scott Lamb wrote:
<snip>
> Going back to Bernd Schoeller's original question: if "in.dat" and
> "out.dat" are regular files, there's nothing useful libevent can do in
> this program. If you just want to try out libevent, I'd suggest
> playing with sockets instead. If you want to write your own cp that
> overlaps reads and writes, I'd suggest (1) using threads, (2) aio, or
> (3) mmap()ing the source file and doing a write() directly from this
> memory to the destination. (The mmap() only works if your platform's
> virtual memory size is sufficient for the file in question. I'd avoid
> it on 32-bit platforms.)

Option number #4: *BSD sendfile() or Linux splice(). Or maybe it's #1(b). 

Spool up a thread in the background which just moves data between a file
descriptor and a domain socket or pipe (or a collection of these). Just have
the thread keep the socket/pipe full, and close the write end when the file
returns EOF.

Technically a thread could just read+write the data between the descriptors,
but that's not as cool. sendfile() and splice() should be as or more
efficient than mmap.

Option number #5: use a multi-process or direct multi-thread model. Even if
one event loop context is blocking on file I/O, the other N-1 contexts are
likely chugging along. This doesn't require a fully multi-threaded event
loop. Unless you're reading a file over NFS or CIFS, even blocking disk I/O
will be on the order of tens or hundreds of milliseconds, not seconds.
Unless you have extremely tight latency requirements, just spreading the
load across multiple processes should be fine. Disk I/O is usually faster
than the pipe to the client (this usually still holds even if the aggregate
bandwidth available to the host is huge), so as long as the socket buffers
are kept full and file I/O completed in discrete chunks (quickly yielding
back to the event loop) the clients won't notice.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.