[Libevent-users] http server and infinite streams

[Cliff Frey <cliff@xxxxxxxxxx>]

(apologies if this is a duplicate post)

It might be clearer to just look at my patches and commit messages
    https://github.com/clifffrey/Libevent/commits/http_oom_prevention
But I've summarized my problems/fixes below.  Please let me know if
you would rather I submit a github pull request, or attach patches to
the email, or submit bugs/features on the website.

I want to send a very large (potentially infinite) file to a slow HTTP
client, and I don't want the server to run out of memory while doing
it (imagine I want someone to use one GET request to receive a very
busy stream-of-events, and if they cannot keep up, I want to just skip
some events rather than infinitely-buffering them).

To do this, I need to be able to get a callback when
evhttp_connection->bufev's write buffer is empty (or close to empty).
I could imagine an implementation where you can explicitly check the
length and explicitly get a callback when the length falls below some
threshold.  Another option is to change write_chunk to make a callback
when the write buffer is completely empty.  This is what I
implemented, but I am open to other suggestions.

Also, a completely different bug:  If you want to support potentially
infinite POST streams from clients (imagine that you wanted to
implement word-count as an http server, where they POST a document,
and you return the word count) then clients can run your server out of
memory by sending one very large chunk.  I think that the
evhttp_request_set_chunked_cb callback should be called on every read,
not just when a complete HTTP chunk has been read.  I have made a
patch that does this, but I worry that maybe some user out there
depends on the only-read-complete-http-chunk behavior.

Thanks, and libevent + http is awesome,

Cliff
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.