[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [Libevent-users] Help with Libevent multi-threaded HTTP server
On Sat, Jan 03, 2015 at 08:22:52PM +0000, Gerry Sweeney wrote:
> Hi Azat,
>
> Ok thanks. You are right, there is not much different in the two patterns, they are almost the same apart from the first pattern the request is served as soon as there is a free thread to serve it and the second is essentially an open connection that can have data sent down it at any time by another thread.
>
first pattern -> libevhtp
As for the second pattern, am I understand correct, that the main goal
of it is is nhave separate thread for non-idle connection?
IOW we can't reuse that separate thread to do some job for other
connections.
If so, I don't think that this is AIO's task, *but*
all you need is just move handling of this request into separate
thread, and a brief look at libevhtp shows me that it is pretty easy to
add.
> If libevent already works like this how do I control how many worker threads it will use? Are they created on demand? or in a pool? This is the sort of information thats really unclear to me. I have have a look in the source code as you suggest to see if I can get a better understanding.
>
This is not libevent, but libevhtp:
https://github.com/ellzey/libevhtp/blob/master/examples/thread_design.c
In short:
evhtp_use_threads()
> Gerry
>
>
>
> Gerry Sweeney
> http://gerrysweeney.com/
>
>
> > On 3 Jan 2015, at 17:43, Azat Khuzhin <a3at.mail@xxxxxxxxx> wrote:
> >
> > On Sat, Jan 03, 2015 at 08:41:31PM +0300, Azat Khuzhin wrote:
> >> On Sat, Jan 03, 2015 at 05:02:36PM +0000, Gerry Sweeney wrote:
> >>> Hi Azat,
> >>>
> >>> Thank you for the detailed response and sorry for the late reply, I do really appreciate your time to respond - thank you. May I also appologize for the incompleteness of the example, I lifted the code out of a bigger project and failed to compile it stand-alone before I sent it - lesson learned :)
> >>>
> >>> Yes that fixed the problem, it also highlighted another problem, I was linking against an earlier version of the lib event library so the evthread_use_pthreads() could not be resolved at link time, I have also corrected that problem and also used âevent2/event.hâ instead of âevent.hâ. It is not that obvious from the various examples that there are different headers and library versions to contend with (thats my excuse anyway :)
> >>
> >> event.h is just a deprecated wrapper for event2/*, so it must works with
> >> event.h too (But *yes, it is deprecated, so please don't use it*).
> >>
> >>>
> >>> It now works as expected so thank you for your help. I did try to use libevhtp but I could not get it to compile, I think this was most likely down to the fact I was including version 1x of the lib event, I will try it again.
> >>
> >> AFAIR libevhtp needs libevent 2.
> >>
> >>>
> >>> I am trying to create two different HTTP server patterns: -
> >>>
> >>> First one is a simple multi-threaded web server, where I receive requests asynchronously complete HTTP requests, event driven into a buffer (so presumably a buffered event) - this is the async part. Once I receive a valid HTTP request from the client, I put the request (and its connection) into a queue to be serviced synchronously by a thread from a pool of worker threads, ones the request has been served, the connection is once again put into the async receive queue ready for the next request.
> >>>
> >>> The second pattern, I want to create a server to be able to accept a large number of long-term and generally idle connections. Like the first example, the event driven receive will handle the request part, and once a valid request has been received the connection will be added to a connection list. At any point, other threads in the server will be able to send data to any clients open connection by looking up the open connection in the connection list. To visualise in HTTP terms the response data would most likely look like a never-ending chunked response, each chunk would be sent as times dictated by the server system.
> >>
> >> I don't see *big* difference between this two patterns (but maybe I
> >> missed something?):
> >> 1) accept() -> do job in a thread -> wait for other requests from client
> >> 2) accept() -> do job in a thread (send chunked responses forever)
> >>
> >> And libevhtp works just like this, it accept() in the main thread, and
> >> after this schedule callbacks into separate thread pool, by using pipes
> >> (pipe(2)).
> >
> > Sorry, not pipes, but sockets (i.e. socketpair(2))
> >
> >>
> >> Here is brief explanation how this part works in libevhtp:
> >> - evhtp.c:
> >> _evhtp_accept_cb() -> evthr_pool_defer()
> >> - evthr.c:
> >> evthr_pool_defer() -> evthr_defer() -> send() -> ... -> _evthr_read_cmd()
> >>
> >> And _evthr_read_cmd() works in a separate thread, while send() in main.
> >>
> >> And please could you avoid top-posting?
> >>
> >> Cheers,
> >> Azat.
> >>
> >>>
> >>> I am sure there are more formal names/descriptions for these patterns but thats as I understand them. These patterns are both terribly important for web-scale application design so they are worthy of some good documentation I think.
> >>>
> >>> For me personally, I thought it would be good to learn how to do this using libevent just as an exercise to learn how the library and event model implemented by libevent works, but so far its not been terribly obvious how I should approach either of these. I tried the first pattern but failed to make it work, evthread_use_pthreads() might solve this now so I will need to test.
> >>>
> >>> It would be terribly helpful if someone could point me in the right general direction with some pseudo code, which functions I would call to achieve those patterns, I know that may be asking a bit much but I thought I would ask anyway.
> >>>
> >>> For what its worth, I will make an effort do document my efforts on these two patterns on my blog http://gerrysweeney.com/ in order to hopefully feedback to the community.
> >>>
> >>> Gerry
> >>>
> >>>>
> >>>
> >>>
> >>> Gerry Sweeney
> >>> http://gerrysweeney.com/
> >>>
> >>>
> >>>> On 30 Dec 2014, at 16:00, Azat Khuzhin <a3at.mail@xxxxxxxxx> wrote:
> >>>>
> >>>> On Sun, Dec 28, 2014 at 06:49:28PM +0000, Gerry Sweeney wrote:
> >>>>> Hi Azat,
> >>>>>
> >>>>> See below, this is the program, see the comments in main() for an explanation of the problem. This needs to be a c++ compile as I am using an STL <vector>. I have made it work by freeing the event_base object by calling free_event_base() but that feels like the wrong thing to do.
> >>>>>
> >>>>> Thanks for the response, I appreciate itâany help would be much appreciated
> >>>>
> >>>> Please see comments for code, in short all you need in
> >>>> evthread_use_pthreads(), but you code have some design issues and also
> >>>> errors that don't allow me to compile this simple program.
> >>>>
> >>>>>
> >>>>> Gerry
> >>>>>
> >>>>> ----------------------------------
> >>>>> #include <sys/types.h>
> >>>>> #include <sys/time.h>
> >>>>> #include <sys/queue.h>
> >>>>> #include <stdlib.h>
> >>>>> #include <unistd.h>
> >>>>> #include <err.h>
> >>>>> #include <event.h>
> >>>>> #include <evhttp.h>
> >>>>> #include <fcntl.h>
> >>>>> #include <sys/socket.h>
> >>>>> #include <netinet/in.h>
> >>>>> #include <iostream>
> >>>>> #include <vector>
> >>>>>
> >>>>> void httpserver_ProcessRequest(struct evhttp_request *req) {
> >>>>> struct evbuffer *buf = evbuffer_new();
> >>>>> if (buf == NULL)
> >>>>> return;
> >>>>>
> >>>>> evbuffer_add_printf(buf, "Requested: %s", evhttp_request_uri(req));
> >>>>>
> >>>>> evhttp_send_reply(req, HTTP_OK, "OK", buf);
> >>>>>
> >>>>> evbuffer_free(buf);
> >>>>> }
> >>>>>
> >>>>> void* httpserver_Dispatch(void *arg) {
> >>>>>
> >>>>> event_base_dispatch((struct event_base*)arg);
> >>>>>
> >>>>> std::cout << "Event dispatch thread ended..." << std::endl;
> >>>>>
> >>>>> return 0;
> >>>>> }
> >>>>>
> >>>>> int httpserver_bindsocket(int port, int backlog)
> >>>>> {
> >>>>> int r;
> >>>>> int nfd;
> >>>>> nfd = socket(AF_INET, SOCK_STREAM, 0);
> >>>>> if (nfd < 0) return -1;
> >>>>>
> >>>>> int one = 1;
> >>>>> r = setsockopt(nfd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, sizeof(int));
> >>>>>
> >>>>> struct sockaddr_in addr;
> >>>>> memset(&addr, 0, sizeof(addr));
> >>>>> addr.sin_family = AF_INET;
> >>>>> addr.sin_addr.s_addr = INADDR_ANY;
> >>>>> addr.sin_port = htons(port);
> >>>>>
> >>>>> r = bind(nfd, (struct sockaddr*)&addr, sizeof(addr));
> >>>>> if (r < 0) return -1;
> >>>>> r = listen(nfd, backlog);
> >>>>> if (r < 0) return -1;
> >>>>>
> >>>>> int flags;
> >>>>> if ((flags = fcntl(nfd, F_GETFL, 0)) < 0
> >>>>> || fcntl(nfd, F_SETFL, flags | O_NONBLOCK) < 0)
> >>>>> return -1;
> >>>>>
> >>>>> return nfd;
> >>>>> }
> >>>>>
> >>>>> int httpserver_start(int port, int nthreads, int backlog)
> >>>>> {
> >>>>> int r, i;
> >>>>> int nfd = httpserver_bindsocket(port, backlog);
> >>>>> if (nfd < 0)
> >>>>> return -1;
> >>>>>
> >>>>> struct workder_t
> >>>>> {
> >>>>> struct event_base* base;
> >>>>> struct evhttp* httpd;
> >>>>> struct evhttp_bound_socket* soc;
> >>>>> pthread_t thr;
> >>>>> };
> >>>>>
> >>>>> std::vector<workder_t> _workers;
> >>>>>
> >>>>> for (i = 0; i < nthreads; i++)
> >>>>> {
> >>>>> workder_t worker;
> >>>>>
> >>>>> worker.base = event_base_new();
> >>>>> if (worker.base == NULL)
> >>>>> return -1;
> >>>>>
> >>>>> worker.httpd = evhttp_new(worker.base);
> >>>>> if (worker.httpd == NULL)
> >>>>> return -1;
> >>>>>
> >>>>> worker.soc = evhttp_accept_socket_with_handle(worker.httpd, nfd);
> >>>>>
> >>>>> evhttp_set_gencb(worker.httpd, httpserver_GenericHandler, NULL);
> >>>>
> >>>> No such function -- httpserver_GenericHandler.
> >>>>
> >>>>>
> >>>>> r = pthread_create(&worker.thr, NULL, httpserver_Dispatch, worker.base);
> >>>>>
> >>>>> if (r != 0)
> >>>>> return -1;
> >>>>>
> >>>>> _workers.push_back(worker);
> >>>>> }
> >>>>>
> >>>>> std::cout << "Running - press return to stop..." << std::endl;
> >>>>> getchar();
> >>>>> std::cout << "Stopping..." << std::endl;
> >>>>>
> >>>>> close(nfd);
> >>>>> std::cout << "Closed listening socket..." << std::endl;
> >>>>>
> >>>>> for(auto worker = _workers.begin(); worker != _workers.end(); ++worker)
> >>>>> {
> >>>>> // I would expect that this would cause the thread blocked in event_base_dispatch() function called in httpserver_Dispatch function to end
> >>>>> event_base_loopbreak(worker->base);
> >>>>>
> >>>>> // seems like I should do this, but not sureâworks either way
> >>>>> evhttp_free(worker->httpd);
> >>>>
> >>>> This will close accept socket, because of LEV_OPT_CLOSE_ON_FREE in
> >>>> evhttp_accept_socket_with_handle(), you must use evhttp_bind_listener()
> >>>> here to avoid this.
> >>>>
> >>>> You could look into libevhtp, it has thread support out-of-the-box, and
> >>>> it is simpler since it is library that just implements http server.
> >>>>
> >>>>>
> >>>>> // If I do this the the event_base_dispatch() does return so I get the expected result, but this feels wrong to
> >>>>> // destroy the base before the thread ends
> >>>>> // comment the next line to see the problem
> >>>>> event_base_free(worker->base);
> >>>>
> >>>> Here you first freeing base, and then in separate thread
> >>>> event_base_dispatch() will use some internal fields in it after free --
> >>>> IOW you must free all resources after pthread_join().
> >>>>
> >>>>>
> >>>>> // We are in the main thread so we join with each worker thread here
> >>>>> pthread_join(worker->thr, NULL);
> >>>>> }
> >>>>> std::cout << âClosed..." << std::endl;
> >>>>
> >>>> Bad quotes.
> >>>>
> >>>>>
> >>>>> return 0;
> >>>>> }
> >>>>>
> >>>>> int main(int argc, char* argv[])
> >>>>> {
> >>>>
> >>>> In short all you need is add the next line here:
> >>>> evthread_use_pthreads();
> >>>>
> >>>> This will enable inter-thread notification for loop break.
> >>>>
> >>>> [ I do have some other changes to you program, so if just adding
> >>>> evthread_use_pthreads() didn't work for you -- let me know, I will send
> >>>> you patch with my local changes ].
> >>>>
> >>>>> httpserver_start(8080, NUM_THREADS, 100);
> >>>>
> >>>> NUM_THREADS undefined.
> >>>> ***********************************************************************
> >>>> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> >>>> unsubscribe libevent-users in the body.
> >>>
> >>> ***********************************************************************
> >>> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> >>> unsubscribe libevent-users in the body.
> >>
> >> --
> >> Respectfully
> >> Azat Khuzhin
> >
> > --
> > Respectfully
> > Azat Khuzhin
> > ***********************************************************************
> > To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> > unsubscribe libevent-users in the body.
>
> ***********************************************************************
> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> unsubscribe libevent-users in the body.
--
Respectfully
Azat Khuzhin
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.