Re: [Libevent-users] http server and infinite streams

[Mark Ellzey <mthomas@xxxxxxxxxx>]

On Tue, May 10, 2011 at 04:04:27PM -0700, Scott Lamb wrote:
> I don't see the point in libevent implementing spooling to a file
> (descriptor). If an application wants that, it can implement it on top
> of the existing evhttp_request_set_chunked_cb + the fairly small
> patches being proposed (call the chunked cb more often, offer some
> sort of flow control, make it possible for the server to set a chunked
> callback for POST bodies). It's not possible to do the reverse in a
> satisfactory way:

The problem - and the main reason I never actually posted the patch -
was because there is a flaw in the set_chunked_cb logic. It only makes
sense when the chunks are of reasonable size. You are only getting
data after that one chunk has been fully processed. 

> * fork()+exec() in the request path is slow (think CGI). Maybe this
> isn't significant for long requests but if you also get any small
> requests it'd be problematic. Also even ignoring performance, it's
> surprisingly hard to fork() correctly in threaded programs.
> 

I can't figure out whether you are working through some other problems by
typing them out, or if this is a response to something I said. 


> * Likewise, directly spooling to a file on disk is not satisfying for
> several reasons:
> 
> ** It doesn't solve the problem of failing on an infinite stream; it
> makes some larger requests possible but for infinite ones just makes
> the failure mode running out of disk instead of running out of memory.
> 

Lets put this issue to rest.. *NOTHING* solves the problem of infinite inputs.  

> ** It could as much as double the response time for proxy servers:
> they must receive the entire request then send the entire request
> instead of doing them simultaneously. And it'd break the downstream
> client's progress indicator.
> 

Then keep it in memory.

> ** It's too slow to be done synchronously in the network thread (a
> single seek is ~10 ms under even ideal conditions, and I was just
> reading about cases at work where it could be more like a second) and
> there's no satisfactory async API for plain files, so it would require
> threading which complicates things. Currently I believe libevent
> doesn't manage any thread pools; the application is required to do all
> that. And libevent can even be compiled without threading support so
> you'd need to this support to be conditional on threading.
> 

A majority of developers understand the issues that coincide with disk
IO. Again, I must reiterate - this would be an optional flag. Having a
cross compatible mechanism for transparently spooling data for large
streams (note: not chunks) of data isn't bad, it's an alternative. 

> ** As previously mentioned, there are embedded cases where it's
> impossible because there is no disk.
> 

Then keep it in memory.

> I think libevent should provide the minimal API that satisfies all use
> cases, and that's chunked callback + flow control.
> 

Option A: use the partial data flag method I previously posted (which
solves both chunking and large data issues).

Option B: depend only on the chunk callback mechanism, which does not
solve the issue with large data streams. 

Option C: Use a spooling method.

Option D: Implement all of the above options, allow the developer to
choose their own doom.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.