Re: [Libevent-users] http server and infinite streams

[Nick Mathewson <nickm@xxxxxxxxxxxxx>]

On Sat, Apr 30, 2011 at 9:55 PM, Cliff Frey <cliff@xxxxxxxxxx> wrote:
> I added tests, handled the infinite-receive case as well as the
> infinite-send case, and slightly renamed a few things.
>
> https://github.com/clifffrey/Libevent/tree/http-transfer-throttling
>
> I believe that the changes there are enough to prevent out-of-memory
> conditions when sending or receiving huge HTTP requests.
>
> Part of me wonders if this is the wrong way to go with the fixes, and
> if it would somehow be better/natural to make the HTTP layer act a
> little bit more like a filter, and expose a struct bufferevent
> interface to the incoming/outgoing data stream.  However, that seems
> like it would end up being a bigger rewrite, and this is a concrete
> fix to a specific problem, so this change still feels worthwhile to
> me.

Hi!  I was going to try to think more and post on Monday with more
cogent review, but I hear that a review before the weekend gets
underway would be of more use to you, so I'm going to substitute speed
for cogency: I hope that the result is still useful.

So I like the idea of getting read-by-read notification of data as it
arrives.  I'm not thrilled with the idea of changing default behavior
wrt what counts as a read, though.  Either a flag or another kind of
callback would make me much more comfortable about the change here.

I think that the evhttp_send_reply_chunk_with_cb behavior might be a
little screwy.  Suppose that you send one chunk with a cb, then send
another chunk with cb == NULL.  The second time you call
evhttp_send_reply_chunk_with_cb, it won't clear out the cb, so the
first cb will get called for the second chunk.  That can't be
as-intended, right?

I *think* that some part of this could be done by exposing the
bufferevent more, but like you I'm a little unclear on the details
here.  Pausing and unpausing could (I think) be done with
bufferevent_enable and disable on the user's part; throttling could be
done with the rate-limiting functionality and the high-water/low-water
functionality; and did you know that you can have more than one
callback on an evbuffer?  You can get into lots of fun with that.

But on consideration, this code is pretty darn nice.  I'm hoping other
evhttp users will have a look at it too, but on the whole I am pretty
happy about merging it into 2.1.

See also my next mail to this list. :)

-- 
Nick
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.