Hi Azat,
Thank you for the detailed response and sorry for the late reply, I do really appreciate your time to respond - thank you. May I also appologize for the incompleteness of the example, I lifted the code out of a bigger project and failed to compile it stand-alone before I sent it - lesson learned :)
Yes that fixed the problem, it also highlighted another problem, I was linking against an earlier version of the lib event library so the evthread_use_pthreads() could not be resolved at link time, I have also corrected that problem and also used âevent2/event.hâ instead of âevent.hâ. It is not that obvious from the various examples that there are different headers and library versions to contend with (thats my excuse anyway :)
It now works as expected so thank you for your help. I did try to use libevhtp but I could not get it to compile, I think this was most likely down to the fact I was including version 1x of the lib event, I will try it again.
I am trying to create two different HTTP server patterns: -
First one is a simple multi-threaded web server, where I receive requests asynchronously complete HTTP requests, event driven into a buffer (so presumably a buffered event) - this is the async part. Once I receive a valid HTTP request from the client, I put the request (and its connection) into a queue to be serviced synchronously by a thread from a pool of worker threads, ones the request has been served, the connection is once again put into the async receive queue ready for the next request.
The second pattern, I want to create a server to be able to accept a large number of long-term and generally idle connections. Like the first example, the event driven receive will handle the request part, and once a valid request has been received the connection will be added to a connection list. At any point, other threads in the server will be able to send data to any clients open connection by looking up the open connection in the connection list. To visualise in HTTP terms the response data would most likely look like a never-ending chunked response, each chunk would be sent as times dictated by the server system.
I am sure there are more formal names/descriptions for these patterns but thats as I understand them. These patterns are both terribly important for web-scale application design so they are worthy of some good documentation I think.
For me personally, I thought it would be good to learn how to do this using libevent just as an exercise to learn how the library and event model implemented by libevent works, but so far its not been terribly obvious how I should approach either of these. I tried the first pattern but failed to make it work, evthread_use_pthreads() might solve this now so I will need to test.
It would be terribly helpful if someone could point me in the right general direction with some pseudo code, which functions I would call to achieve those patterns, I know that may be asking a bit much but I thought I would ask anyway.
For what its worth, I will make an effort do document my efforts on these two patterns on my blog http://gerrysweeney.com/ in order to hopefully feedback to the community.
Gerry
>
Gerry Sweeney
http://gerrysweeney.com/
> On 30 Dec 2014, at 16:00, Azat Khuzhin <a3at.mail@xxxxxxxxx> wrote:
>
> On Sun, Dec 28, 2014 at 06:49:28PM +0000, Gerry Sweeney wrote:
>> Hi Azat,
>>
>> See below, this is the program, see the comments in main() for an explanation of the problem. This needs to be a c++ compile as I am using an STL <vector>. I have made it work by freeing the event_base object by calling free_event_base() but that feels like the wrong thing to do.
>>
>> Thanks for the response, I appreciate itâany help would be much appreciated
>
> Please see comments for code, in short all you need in
> evthread_use_pthreads(), but you code have some design issues and also
> errors that don't allow me to compile this simple program.
>
>>
>> Gerry
>>
>> ----------------------------------
>> #include <sys/types.h>
>> #include <sys/time.h>
>> #include <sys/queue.h>
>> #include <stdlib.h>
>> #include <unistd.h>
>> #include <err.h>
>> #include <event.h>
>> #include <evhttp.h>
>> #include <fcntl.h>
>> #include <sys/socket.h>
>> #include <netinet/in.h>
>> #include <iostream>
>> #include <vector>
>>
>> void httpserver_ProcessRequest(struct evhttp_request *req) {
>>Â Â struct evbuffer *buf = evbuffer_new();
>>Â Â if (buf == NULL)
>>Â Â Â Â return;
>>
>>Â Â evbuffer_add_printf(buf, "Requested: %s", evhttp_request_uri(req));
>>
>>Â Â evhttp_send_reply(req, HTTP_OK, "OK", buf);
>>
>>Â Â evbuffer_free(buf);
>> }
>>
>> void* httpserver_Dispatch(void *arg) {
>>
>>Â Â event_base_dispatch((struct event_base*)arg);
>>
>>Â Â std::cout << "Event dispatch thread ended..." << std::endl;
>>
>>Â Â return 0;
>> }
>>
>> int httpserver_bindsocket(int port, int backlog)
>> {
>>Â Â int r;
>>Â Â int nfd;
>>Â Â nfd = socket(AF_INET, SOCK_STREAM, 0);
>>Â Â if (nfd < 0) return -1;
>>
>>Â Â int > >>Â Â r = setsockopt(nfd, SOL_SOCKET, SO_REUSEADDR, (char *)&one, sizeof(int));
>>
>>Â Â struct sockaddr_in addr;
>>Â Â memset(&addr, 0, sizeof(addr));
>>Â Â addr.sin_family = AF_INET;
>>Â Â addr.sin_addr.s_addr = INADDR_ANY;
>>Â Â addr.sin_port = htons(port);
>>
>>Â Â r = bind(nfd, (struct sockaddr*)&addr, sizeof(addr));
>>Â Â if (r < 0) return -1;
>>Â Â r = listen(nfd, backlog);
>>Â Â if (r < 0) return -1;
>>
>>Â Â int flags;
>>Â Â if ((flags = fcntl(nfd, F_GETFL, 0)) < 0
>>Â Â Â Â || fcntl(nfd, F_SETFL, flags | O_NONBLOCK) < 0)
>>Â Â Â Â return -1;
>>
>>Â Â return nfd;
>> }
>>
>> int httpserver_start(int port, int nthreads, int backlog)
>> {
>>Â Â int r, i;
>>Â Â int nfd = httpserver_bindsocket(port, backlog);
>>Â Â if (nfd < 0)
>>Â Â Â Â return -1;
>>
>>Â Â struct workder_t
>>Â Â {
>>Â Â Â Â struct event_base* base;
>>Â Â Â Â struct evhttp* httpd;
>>Â Â Â Â struct evhttp_bound_socket* soc;
>>Â Â Â Â pthread_t thr;
>>Â Â };
>>
>>Â Â std::vector<workder_t> _workers;
>>
>>Â Â for (i = 0; i < nthreads; i++)
>>Â Â {
>>Â Â Â Â workder_t worker;
>>
>>Â Â Â Â worker.base = event_base_new();
>>Â Â Â Â if (worker.base == NULL)
>>Â Â Â Â Â Â return -1;
>>
>>Â Â Â Â worker.httpd = evhttp_new(worker.base);
>>Â Â Â Â if (worker.httpd == NULL)
>>Â Â Â Â Â Â return -1;
>>
>>Â Â Â Â worker.soc = evhttp_accept_socket_with_handle(worker.httpd, nfd);
>>
>>Â Â Â Â evhttp_set_gencb(worker.httpd, httpserver_GenericHandler, NULL);
>
> No such function -- httpserver_GenericHandler.
>
>>
>>Â Â Â Â r = pthread_create(&worker.thr, NULL, httpserver_Dispatch, worker.base);
>>
>>Â Â Â Â if (r != 0)
>>Â Â Â Â Â Â return -1;
>>
>>Â Â Â Â _workers.push_back(worker);
>>Â Â }
>>
>>Â Â std::cout << "Running - press return to stop..." << std::endl;
>>Â Â getchar();
>>Â Â std::cout << "Stopping..." << std::endl;
>>
>>Â Â close(nfd);
>>Â Â std::cout << "Closed listening socket..." << std::endl;
>>
>>Â Â for(auto worker = _workers.begin(); worker != _workers.end(); ++worker)
>>Â Â {
>>   // I would expect that this would cause the thread blocked in event_base_dispatch() function called in httpserver_Dispatch function to end
>>Â Â Â Â event_base_loopbreak(worker->base);
>>
>>Â Â Â Â Â // seems like I should do this, but not sureâworks either way
>>Â Â Â Â evhttp_free(worker->httpd);
>
> This will close accept socket, because of LEV_OPT_CLOSE_ON_FREE in
> evhttp_accept_socket_with_handle(), you must use evhttp_bind_listener()
> here to avoid this.
>
> You could look into libevhtp, it has thread support out-of-the-box, and
> it is simpler since it is library that just implements http server.
>
>>
>>Â Â Â // If I do this the the event_base_dispatch() does return so I get the expected result, but this feels wrong to
>>Â Â Â Â Â // destroy the base before the thread ends
>>Â Â Â Â Â Â // comment the next line to see the problem
>>Â Â Â Â event_base_free(worker->base);
>
> Here you first freeing base, and then in separate thread
> event_base_dispatch() will use some internal fields in it after free --
> IOW you must free all resources after pthread_join().
>
>>
>>Â Â Â // We are in the main thread so we join with each worker thread here
>>Â Â Â Â pthread_join(worker->thr, NULL);
>>Â Â }
>>Â Â std::cout << âClosed..." << std::endl;
>
> Bad quotes.
>
>>
>>Â Â return 0;
>> }
>>
>> int main(int argc, char* argv[])
>> {
>
> In short all you need is add the next line here:
> evthread_use_pthreads();
>
> This will enable inter-thread notification for loop break.
>
> [ I do have some other changes to you program, so if just adding
> evthread_use_pthreads() didn't work for you -- let me know, I will send
> you patch with my local changes ].
>
>>Â Â httpserver_start(8080, NUM_THREADS, 100);
>
> NUM_THREADS undefined.
> ***********************************************************************
> To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
> unsubscribe libevent-users  in the body.
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users  in the body.