[Libevent-users] Problems with deferred HTTP handlers over SSL

[Amarin Phaosawasdi <amarinph@xxxxxxxxxxxx>]

Hi All,

We're working on fixing an issue with one of our servers, which serves HTTPS requests on one end, and makes asynchronous RPCs on the other end.

The server uses libevhtp to serve HTTPS requests. But since it is also based on libevent and we suspect it might have something to do with bufferevent_openssl, we thought it is appropriate to post it here.

When a HTTPS request comes in, a the server invokes an async RPC to an internal server. When the RPC finishes, we then use the RPC results to provide an HTTPS response to the browser.

However, almost all the time the browser gives an error.

- "SSL received a record with an incorrect Message Authentication Code. (Error code: ssl_error_bad_mac_read)." (in Firefox)

- "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error." (in Chrome)

- "Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data. (in Chrome)

Rarely, when the browser doesn't give an error, it either succeeds or blocks forever.

Repeated fails normally end with a segmentation fault in the server.

When a synchronous RPC is used instead (everything is finished on the callback stack), the error is gone and every request is successful.

Also if the server is configured to serve HTTP requests instead of HTTPS, it also always succeeds regardless or the RPC calling mode.

Does anybody have an idea what might be going on?

The attached code can be used to reproduce this issue. When the server is run, use a browser to connect to https://127.0.0.1:1025.

Tested with:

gcc/g++ 4.4.5

Both openssl 0.9.8 and 1.0.0f

libevhtp development branch

libevent master branch

Debian 2.6.39-bpo.2-amd64 x86_64

Thank you,

Amarin

Attachment: libevhtp.key
Description: Binary data

Attachment: libevhtp.pem
Description: application/x509-ca-cert

#include <unistd.h>
#include <pthread.h>
#include "evhtp.h"

void *DeferredHandler(void *arg) {
  sleep(1);
  evhtp_request_t *req = (evhtp_request_t*)arg;
  evhtp_headers_add_header(
      req->headers_out,
      evhtp_header_new("content-type", "text/plain; charset=utf-8", 0, 0));
  evbuffer_add(req->buffer_out, "hello", 5);
  evhtp_send_reply(req, EVHTP_RES_OK);
  pthread_exit(NULL);
}

void Handler(evhtp_request_t *req, void *) {
  pthread_t t;
  pthread_create(&t, NULL, DeferredHandler, (void*)req);
  pthread_detach(t);  // this fails
  // pthread_join(t, NULL);  // but this works
}

int main() {
  evbase_t *base = event_base_new();
  if (!base) return 1;

  evhtp_t *http = evhtp_new(base, NULL);
  if (!http) return 1;

  evhtp_set_gencb(http, Handler, NULL);
  evhtp_use_threads(http, NULL, 4, NULL);

  evhtp_ssl_cfg_t scfg;
  char ciphers[] = "RC4+RSA:HIGH:+MEDIUM:+LOW";
  char libevhtp_pemfile[] = "libevhtp.pem";
  scfg.pemfile = libevhtp_pemfile;
  char libevhtp_privfile[] = "libevhtp.key";
  scfg.privfile = libevhtp_privfile;
  scfg.cafile = NULL;
  scfg.capath = NULL;
  scfg.ciphers = ciphers;
  scfg.ssl_opts = SSL_OP_NO_SSLv2;
  scfg.verify_peer = SSL_VERIFY_NONE;
  scfg.verify_depth = 0;
  scfg.x509_verify_cb = NULL;
  scfg.x509_chk_issued_cb = NULL;
  scfg.store_flags = 0;
  scfg.scache_type = evhtp_ssl_scache_type_internal;
  scfg.scache_timeout = 1024;
  scfg.scache_size = 1024;
  scfg.scache_init = NULL;
  scfg.scache_add = NULL;
  scfg.scache_get = NULL;
  scfg.scache_del = NULL;
  scfg.args = NULL;
  evhtp_ssl_init(http, &scfg);

  // bind and serve
  if (evhtp_bind_socket(http, "127.0.0.1", 1025, 128) == -1) return 1;
  event_base_loop(base, 0);
  event_base_free(base);

  return 0;
}