Re: [Libevent-users] http server and infinite streams

[Nick Mathewson <nickm@xxxxxxxxxxxxx>]

On Tue, Apr 26, 2011 at 1:49 PM, Cliff Frey <cliff@xxxxxxxxxx> wrote:
> (apologies if this is a duplicate post)
>
> It might be clearer to just look at my patches and commit messages
>    https://github.com/clifffrey/Libevent/commits/http_oom_prevention
> But I've summarized my problems/fixes below.  Please let me know if
> you would rather I submit a github pull request, or attach patches to
> the email, or submit bugs/features on the website.

Pull requests are just fine.  My first suggestion here would be that
I'd be a lot more comfortable here with some unit tests.

> I want to send a very large (potentially infinite) file to a slow HTTP
> client, and I don't want the server to run out of memory while doing
> it (imagine I want someone to use one GET request to receive a very
> busy stream-of-events, and if they cannot keep up, I want to just skip
> some events rather than infinitely-buffering them).
>
> To do this, I need to be able to get a callback when
> evhttp_connection->bufev's write buffer is empty (or close to empty).
> I could imagine an implementation where you can explicitly check the
> length and explicitly get a callback when the length falls below some
> threshold.  Another option is to change write_chunk to make a callback
> when the write buffer is completely empty.  This is what I
> implemented, but I am open to other suggestions.

If you're interested on the write buffer getting close to empty, the
evhttp connection can have a write callback and set a  write low-water
mark on the bufev so that the callback will get invoked when it falls
below a threshold.  I'm not too picky about the implementation here as
a first cut so long as the API remains sane and supportable in the
future.

> Also, a completely different bug:  If you want to support potentially
> infinite POST streams from clients (imagine that you wanted to
> implement word-count as an http server, where they POST a document,
> and you return the word count) then clients can run your server out of
> memory by sending one very large chunk.  I think that the
> evhttp_request_set_chunked_cb callback should be called on every read,
> not just when a complete HTTP chunk has been read.  I have made a
> patch that does this, but I worry that maybe some user out there
> depends on the only-read-complete-http-chunk behavior.

Perhaps a flag of some kind could be set so that anybody depending on
the old behavior can get it?

-- 
Nick
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.