[Libevent-users] Timeouts, request_cancel'ing, and null requests on connections.

[Nick Grabenstein <ng1213@xxxxxxxxx>]

I've never used a mailing list before so bear with me if I'm doing something horribly wrong.  Also my C/C++ skillz are weak at best.  Apologies in advance.

I'm trying to write a super simple HTTP Client that can handle arbitrary timeouts.  The code is attached, but I'm going to try and walk you guys through my logic.

The current implementation is to wrap the normal cb, the cb_arg, the request, and timer related objs in a struct which is the new cb_arg and then have two different callbacks for the two scenarios (timeout expires, request handled before the timeout).  The 'success' timeout gets rid of the old timer from the eventloop and then runs the old cb and cb_arg as it would normally.  The 'timeout' callback runs evhttp_cancel_request on the request object and then runs the normal callback with NULL as the first arg (as if the request had failed).  

So far I'm pretty sure everything I'm doing makes sense....

except when I set up a timeout of 200ms to www.google.com and I attempt to fetch this url as fast as possible I end up with:

[err] http.c:704: Assertion req != NULL failed in evhttp_connection_fail

I did a TOONNNNN of digging (printing out pointers, states of evconn's, evconn fd's, etc.) and found a few things.

The connection that has req == NULL (req != NULL is failing) has the same pointer as a connection that has timed out previously but is not currently in use.  When the conn timed out it was in the EVCONN_CONNECTING (or similar enum) state.  There are connections that have timed out and been canceled who were also in the EVCONN_CONNECTING state BEFORE the req == NULL conn failed.  When the connection fails the assert it correctly has it's FD set to -1 and it's state is correctly set to 0 (EVCONN_DISCONNECTD).  The traceback indicates that the error is happening entirely within libevent (as far as I can tell).

/lib/x86_64-linux-gnu/libc.so.6(+0x364c0)
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)
/usr/local/lib/libevent-2.0.so.5(+0x1f7ba)
/usr/local/lib/libevent-2.0.so.5(+0x1fb2b)

/usr/local/lib/libevent-2.0.so.5(+0x29e7e)
/usr/local/lib/libevent-2.0.so.5(+0x2a016)
/usr/local/lib/libevent-2.0.so.5(bufferevent_socket_connect+0x173)
/usr/local/lib/libevent-2.0.so.5(+0x1abff)
/usr/local/lib/libevent-2.0.so.5(+0x3171e)

/usr/local/lib/libevent-2.0.so.5(+0x2bf94)
/usr/local/lib/libevent-2.0.so.5(event_base_loop+0x7af)

So what I *think* is happening is the connection is trying to start up a request, trying to connect, then before it can even connect I'm timing it out and killing the request.  Then somehow the connection (even though it has been properly nil'd out and everything) is completing and trying to do the socket_connect stuff.  I have no idea how this is happening and that's why I'm now coming to you guys.

Any and all help is appreciated.

Aside:  I feel like this shoudln't be that hard? I feel like I must be doing something COMPLETELY wrong....

-nic


#include "http_manager.h"
#include "../request_handlers/base_request_handler.h"

#include <boost/lexical_cast.hpp>

#include <sstream>

#include <event2/event.h>
#include <event2/buffer.h>
#include <event2/http.h>
#include <event2/http_struct.h>

#include "../utils/timer.h"

event_base *HttpManager::base = NULL;
evdns_base *HttpManager::dns_base = NULL;
map<const char *, connection_pool> HttpManager::connection_map;

struct TimedRequest {
    ~TimedRequest(){if (tv) delete(tv); if(timer) delete(timer); if (stop_watch) delete(stop_watch);}

    Timer *stop_watch;
    evhttp_request *req;
    event *timer;
    timeval *tv;

    void (*cb)(struct evhttp_request *, void *);
    void *cb_arg;
};

struct PooledCallback {
    void (*cb)(struct evhttp_request *, void *);
    void *cb_arg;
    evhttp_connection *conn;
    const char *host;
};

void HttpManager::init(event_base *b)
{
    base = b;
    dns_base = evdns_base_new(base, 0);
    evdns_base_resolv_conf_parse(dns_base, DNS_OPTION_NAMESERVERS, "/etc/resolv.conf");
}

void HttpManager::make_request(const char *host, int port,
                               const char *uri,
                               enum evhttp_cmd_type method,
                               const string *body,
                               const int milliseconds,
                               void (*cb)(struct evhttp_request *, void *),
                               void *cb_arg)
{
    evhttp_connection* conn = get_connection(host, port);
    PooledCallback *pooled_cb = new PooledCallback();
    pooled_cb->cb = cb ? cb : HttpManager::generic_callback;
    pooled_cb->cb_arg = cb_arg ? cb_arg : NULL;
    pooled_cb->conn = conn;
    pooled_cb->host = host;

    TimedRequest *success_timed_req = new TimedRequest();
    Timer *stop_watch = new Timer();
    success_timed_req->cb = HttpManager::pooled_calback;
    success_timed_req->cb_arg = (void*)pooled_cb;
    success_timed_req->stop_watch = stop_watch;

    evhttp_request* req = evhttp_request_new(HttpManager::request_cb,
                                             success_timed_req);

    if (milliseconds > 0) {
        timeval *tv = new timeval();
        tv->tv_sec = 0;
        tv->tv_usec = 1000 * milliseconds;

        // deleted in callback
        TimedRequest *timed_req = new TimedRequest();
        timed_req->req = req;
        timed_req->cb = HttpManager::pooled_calback;
        timed_req->cb_arg = pooled_cb;
        timed_req->tv = tv;


        event *timer = evtimer_new(base, HttpManager::timer_cb, (void *)timed_req);

        timed_req->timer = timer;
        timed_req->stop_watch = stop_watch;


        evtimer_add(timer, tv);


        success_timed_req->timer = timer;
    }


    evhttp_add_header(req->output_headers, "Host", "127.0.0.1");
    string size = "0";
    if (body) {
        stringstream stream;
        stream << body->size();
        size = stream.str();
        evbuffer_add(evhttp_request_get_output_buffer(req), body->c_str(), body->size());
    }
    evhttp_add_header(req->output_headers, "Content-Length", size.c_str());
    evhttp_make_request(conn, req, method, uri);
}

void HttpManager::pooled_calback(struct evhttp_request *req, void *state) {
     PooledCallback *wrapper = static_cast<PooledCallback *>(state);
     evhttp_connection *connection = wrapper->conn;
     const char *host = wrapper->host;
     connection_map[host].push_back(connection);
     wrapper->cb(req, wrapper->cb_arg);
     delete(wrapper);
}

void HttpManager::request_cb(struct evhttp_request *req, void *cb_arg)
{
    TimedRequest *wrapper = static_cast<TimedRequest *>(cb_arg);
    event *timer = wrapper->timer;
    if (timer){
        evtimer_del(timer);
    }

    cout << "calling real callback after " << wrapper->stop_watch->stop() << "ms" << endl;
    wrapper->cb(req, wrapper->cb_arg);

    delete(wrapper);
}

void HttpManager::timer_cb(evutil_socket_t socket, short flag, void *cb_arg)
{
    TimedRequest *wrapper = static_cast<TimedRequest *>(cb_arg);
    timeval *tv = wrapper->tv;
    evhttp_request *req = wrapper->req;
    cout << "cancelling request because " << tv->tv_usec/1000 << "ms timeout actually timed out after " << wrapper->stop_watch->stop() << "ms" << endl;
    evhttp_cancel_request(req);

    wrapper->cb(NULL, wrapper->cb_arg);

    delete(wrapper);
}

evhttp_connection* HttpManager::get_connection(const char *host, int port)
{
    connection_pool &pool = connection_map[host];
    evhttp_connection* connection;
    if (pool.empty()) {
        connection = evhttp_connection_base_new(base, dns_base, host, port);
    } else {
        // pop off the front
        connection = pool.front();
        pool.pop_front();
    }
    return connection;
}

void HttpManager::generic_callback(struct evhttp_request *req, void *state)
{

    printf("beginning of cb\n");
    if (req == NULL) {
        printf("timed out!\n");
    } else if (req->response_code == 0) {
        printf("connection refused!\n");
    } else if (req->response_code != 200) {
        printf("error: %u %s\n", req->response_code, req->response_code_line);
    } else {
        struct evbuffer* response_buffer = evhttp_request_get_input_buffer(req);
        size_t len = evbuffer_get_length(response_buffer);
        len = evbuffer_get_length(response_buffer);
        char response[len + 1];
        response[len] = '\0';
        evbuffer_remove(response_buffer, response, len);
        printf("\n\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!\nsuccess: %u %s\nBODY: \n%s\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n\n\n", req->response_code, req->response_code_line, response);
    }
    printf("end of cb function\n");
}