Re: [Libevent-users] http server and infinite streams

[Cliff Frey <cliff@xxxxxxxxxx>]

I spent some more time thinking about this, and I have a few use cases that it might be worth keeping in mind.

It would be nice if the API made it easy to hook up a received HTTP request (where we are the server) to an outgoing HTTP request (where we are the client) in order to make building an HTTP proxy very simple.  Not a huge deal if this isn't super simple though.

It should be easy to hook up a zlib compressor/decompressor to the HTTP library for supporting compressed transfers (with Accept-Encoding).

I'm not sure who should be responsible for allocating the bufferevent objects.  If there was an easy way to connect two existing bufferevent objects to one another (like bufferevent_pair_new, but for existing ones), then it would be fine for the http library to provide the bufferevent objects, but it would probably be more flexible if the caller provided the bufferevent objects (but then I am unsure who should be responsible for cleanup of them).

So, in any case, it boils down to something like

Option 1 (from my previous email)

// used instead of evhttp_set_gen_cb, cb is called as soon as the request

// is received, before the content of the request has been received.

void evhttp_set_gen_req_startcb(evhttp *, void (*cb)(struct evhttp_request *, struct bufferevent *bev, void *), void *arg);

// this returns a bufferevent that can be used to send a reply to the client

struct bufferevent *evhttp_send_reply_bev(struct evhttp_request *req, int code, const char *reason);

Option 2

// used instead of evhttp_set_gen_cb, cb is called as soon as the request

// is received, before the content of the request has been received.

// To received the contents of the message, a bufferevent * must be returned,

// which the evhttp library will fill with the request data, respecting the

// highwater mark.  Once BEV_EVENT_EOF has been received, the evhttp library

// will not access the bufferevent * again, and it is up to the user to free

// the bufferevent.

struct bufferevent *evhttp_set_gen_req_startcb(evhttp *, void (*cb)(struct evhttp_request *, void *), void *arg);

// evhttp will free bev after receiving BEV_EVENT_EOF

void evhttp_send_reply_bev(struct evhttp_request *req, int code, const char *reason, struct bufferevent *bev);

I think that I like option 2 better.

Cliff

On Tue, May 10, 2011 at 11:36 PM, Cliff Frey <cliff@xxxxxxxxxx> wrote:

So I like the idea of getting read-by-read notification of data as it
arrives.  I'm not thrilled with the idea of changing default behavior
wrt what counts as a read, though.  Either a flag or another kind of
callback would make me much more comfortable about the change here.

I will try and make a new version of the read-breakup change that adds a flag of some sort.

I think that the evhttp_send_reply_chunk_with_cb behavior might be a
little screwy.  Suppose that you send one chunk with a cb, then send
another chunk with cb == NULL.  The second time you call
evhttp_send_reply_chunk_with_cb, it won't clear out the cb, so the
first cb will get called for the second chunk.  That can't be
as-intended, right?

I *think* that some part of this could be done by exposing the
bufferevent more, but like you I'm a little unclear on the details
here.  Pausing and unpausing could (I think) be done with
bufferevent_enable and disable on the user's part; throttling could be
done with the rate-limiting functionality and the high-water/low-water
functionality; and did you know that you can have more than one
callback on an evbuffer?  You can get into lots of fun with that.

But on consideration, this code is pretty darn nice.  I'm hoping other
evhttp users will have a look at it too, but on the whole I am pretty
happy about merging it into 2.1.

On the whole I agree with you that my proposed API is a little bit screwy.  It does get the job done in a fairly simple manner though... which is why I did it.  In any case, here is an attempt at a cleaner API, but I wanted to get feedback from the list on it.  This API very-much just builds on the current API (although I imagine that in the actual change to http.c, much of the current API would be implemented in terms of the new API...)

Receiving HTTP requests as an HTTP server:

Current interface
  Useful for simple servers, impossible to receive infinite/streaming requests
    set a maximum incoming request size (to avoid OOM)
    set a callback that gets a complete request:

      evhttp_set_gencb(evhttp *, void (*cb)(struct evhttp_request *, void *), void *arg);

Proposed interface

Leave the current interface for simple servers, but add another callback that supports receiving streaming requests.  If this other callback is registered, then it takes precedence over the old one.

This is called as soon as a request is received, before the body of the request has been received (if any)

evhttp_set_gen_req_startcb(evhttp *, void (*cb)(struct evhttp_request *, struct bufferevent *bev, void *), void *arg);

If the request has content-length 0, then bev is NULL.  If 'cb' sets the highwater mark of bev, then the http library will attempt to respect that, and will stop reading from the underlying TCP connection if bev is full.  Is is up to 'cb' to register callbacks on bev that will receive future events (including EOF for the end of the request)

Sending an HTTP response as an HTTP server:

Current interface: provides two decent options, does not support flow control when sending possibly-infinite streams to slow clients (i.e. very frequent stock ticker symbol updates over a persistent connection to a very very slow client)

either

  evhttp_send_reply(struct evhttp_request *req, int code, const char *reason, struct evbuffer *databuf);

or

  evhttp_send_reply_start(struct evhttp_request *req, int code, const char *reason);

  evhttp_send_reply_chunk(struct evhttp_request *req, struct evbuffer *databuf);

  evhttp_send_reply_chunk_with_cb(struct evhttp_request *req, struct evbuffer *databuf, void (*cb)(struct evhttp_request *, void *), void *arg);

Proposed interface: add a third option that allows for flow control

  struct bufferevent *evhttp_send_reply_bev(struct evhttp_request *req, int code, const char *reason);

It is the caller's responsibility to fill the returned bufferevent with the data to be sent to the client, and to eventually bufferevent_flush(BEV_FINISHED) on it.  At this point, the caller should never access the bufferevent or evhttp_request again (i.e. the http library will free it when finished).  The evhttp library would only empty this bufferevent when the underlying TCP connection's bufferevent was not too full (probably with a configurable highwater mark defining full, i.e. evhttp_connection_set_xmit_highwater or something)

I believe that a similar set of additions could be made with the http-client API, but I wanted to get feedback on this potential server API first.  It would likely involve incompatible changes to the evhttp_request structure, but people are not supposed to depend on that anyways.

Cliff