[Libevent-users] evhttp_send_reply segfaults and aborts randomly

[David Titarenco <david.titarenco@xxxxxxxxx>]

I wrote a high performance HTTP event server in C++ using libevent 2.0.1-alpha. The server isn't that complex so I'll try to explain what it does and whether or not this is a bug in libevent or a problem with my code.

A "bead" is basically a client connection. It looks like so:
struct bead {
    long int id;
    int comet;
    struct event *timeout_timer;

    int    lastMessage;
    int    delta;
    string whoAmI;
    string currCircle;
    string userHash;
    string cookieHash;
    string stickyHash;
    string appHash;
    bead() : whoAmI(""), currCircle(""), userHash(""), cookieHash(""), appHash("")   {}

};

It's mapped against the appropriate request pointer at it's initialization:
map<evhttp_request *,bead> rMap;


In main():
evhttp_set_gencb(http_server, generic_request_handler, NULL);

In generic_request_handler():
void generic_request_handler(struct evhttp_request *req, void *arg) {
    const char *ruri = evhttp_request_get_uri(req);

    if (strncmp(ruri, "/admin.eve", 10) == 0) {
        struct evbuffer *evb = evbuffer_new();
        evbuffer_add_printf(evb, "<h2>Hello. I am Eve.</h2><br> \
                Connections since start: %i <br> \
                Current active connections in STL container: %i <br>", (int) num_con, (int)rMap.size());
        evhttp_send_reply(req, HTTP_OK, "OK", evb);
        evbuffer_free(evb);
    }
    else 
        long_polling_handler(req, NULL);
}

and lets look at the long_polling_handler():
void long_polling_handler(struct evhttp_request *req, void *arg) {
    // tons of code that handles POST variables pertinent to my app, creates new_bead, sets a timeout timer dependent on delta, etc, etc.
    rMap.insert(std::pair<evhttp_request *,bead>(req, new_bead));
}

The program loops over all the beads like so:
void handle_beads() {
    for( map<evhttp_request *,bead>::iterator it=rMap.begin(); it != rMap.end(); ++it)
    {
        long_polling_bead(it);
    }
}

static void iterate_cb(int fd, short event, void *arg) {
    handle_beads();
    event_free(evtq_timer);
    evtq_timer = evtimer_new(base, iterate_cb, NULL);
    evutil_timerclear(&tv);
    tv.tv_usec = 100000;
    event_add(evtq_timer, &tv);
    return;
}

The iteration callback is called every 100000 microseconds.

And finally, long_polling_bead(). This is where the program breaks...
void long_polling_bead(map<evhttp_request *,bead>::iterator it) {
// lots of code that connects to an SQL database, compares the user state (from POST) to the server state (from SQL)
// here, we either return 503 or  504
// 503 means NOT_UPDATED
// 504 tells nginx it should hit PHP (my server acts as an HTTP event proxy)
// but this is how I finish the requests:

            struct evbuffer *evb = evbuffer_new();
            evbuffer_add_printf(evb, "{\"error\":\"Some JSON stuff here\"}");
            evhttp_send_reply((*it).first,503,"ERROR",evb);
            evbuffer_free(evb);
            event_free((*it).second.timeout_timer);
            pLog->Write("Some log message");
            rMap.erase(it);
            return;

}

Here are the gdb backtraces:

#0  0xb7c739b4 in free () from /lib/libc.so.6
#1  0xb7f44b18 in mm_free (ptr=0x1) at event.c:2065
#2  0xb7f579b6 in evhttp_response_code (req=0xfffffff9, code=503, reason=0x813d2c8 "ERROR") at http.c:2124
#3  0xb7f5cd23 in evhttp_send_reply (req=0x98f9ba0, code=503, reason=0x813d2c8 "ERROR", databuf=0x991efb0) at http.c:2053
#4  0x0811911b in long_polling_bead (it={_M_node = 0xb7d4d170}) at eve.new.cpp:386
#5  0x08119cb1 in handle_beads () at eve.new.cpp:477
#6  0x08119d02 in iterate_cb (fd=-1, event=1, arg=0x0) at eve.new.cpp:482
#7  0xb7f47f18 in event_base_loop (base=0x98ce9c0, flags=0) at event.c:925
#8  0xb7f48c85 in event_base_dispatch (event_base=0x98ce9c0) at event.c:1005
#9  0x08118cbf in main (argc=1, argv=0xbf84b514) at eve.new.cpp:622

#0  0xb7fdb7f2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0xb7c93df0 in raise () from /lib/libc.so.6
#2  0xb7c95701 in abort () from /lib/libc.so.6
#3  0xb7fb5447 in event_exit (errcode=0) at log.c:77
#4  0xb7fb5476 in event_errx (eval=-559030611, fmt=0xb7fcb170 "%s:%d: Assertion %s failed in %s") at log.c:134
#5  0xb7fbddbc in evhttp_send_reply (req=0x9e4f5a0, code=503, reason=0x813d2c8 "ERROR", databuf=0x9e04648) at http.c:2037
#6  0x0811911b in long_polling_bead (it={_M_node = 0xb7dae170}) at eve.new.cpp:386
#7  0x08119cb1 in handle_beads () at eve.new.cpp:477
#8  0x08119d02 in iterate_cb (fd=-1, event=1, arg=0x0) at eve.new.cpp:482
#9  0xb7fa8f18 in event_base_loop (base=0x9dec9c0, flags=0) at event.c:925
#10 0xb7fa9c85 in event_base_dispatch (event_base=0x9dec9c0) at event.c:1005
#11 0x08118cbf in main (argc=1, argv=0xbfd8d4b4) at eve.new.cpp:622

I hope I was thorough enough, thanks for any help.

 - David Titarenco