Re: [Libevent-users] http server and infinite streams

[Mark Ellzey <mthomas@xxxxxxxxxx>]

On Tue, Apr 26, 2011 at 10:49:46AM -0700, Cliff Frey wrote:
> Also, a completely different bug: ?If you want to support potentially
> infinite POST streams from clients (imagine that you wanted to
> implement word-count as an http server, where they POST a document,
> and you return the word count) then clients can run your server out of
> memory by sending one very large chunk. ?I think that the
> evhttp_request_set_chunked_cb callback should be called on every read,
> not just when a complete HTTP chunk has been read. ?I have made a
> patch that does this, but I worry that maybe some user out there
> depends on the only-read-complete-http-chunk behavior.
> 

A while back I went and did some looking for a proper way to handle
such situations. The first method I came up with would allow a callback
to be executed when a full set of headers (before any post data) is recvd.

This would allow you to set the chunk callback function which normally
can only be used by a client.

https://github.com/ellzey/Libevent/commit/36218640b3b19fc94a75e2e691fc41ccda81224c

Another approach I tried was to extend the request handling to allow for
your normal http callback to get partial chunks of data. When you
twiddle the streaming flag, you would need to check evhttp_request's
flags for EVHTTP_REQ_PARTIAL_STREAM. If it is set, you can write the
output wherever, and the buffer will be drained transparently
afterwards. 

https://github.com/ellzey/Libevent/commit/4ead4de1da726d801c7506c7af648809cd7284f9

These both work in theory, but if I remember correctly have bugs.

Picking the "right way" to deal with this issue is hard. The current
handler seems clunky to work with, and hacky to modify. This is primarily 
due to feature creep..

I think the right balance of API transparency, good resource management 
and abstractions would be introducing more and more user defined callbacks. 
Such a method would allow a developer to "hook" into all parts of the request 
cycle.  

While I support these types of changes, I often wonder if the better
solution to these problems is to gut the httpd API and create a more robust
and abstracted replacement. *prepares for flogging* 

Libevent's http API was created for JIT services, not a apache/nginx/iis 
replacement. But the reality is, developers have shown that the system should 
be just as robust as the real ones. 

A project to keep an eye on (one I have started using after dealing with
the many faults of the libevent http engine) is https://github.com/ry/http-parser 
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.