[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[Libevent-users] evbuffer_add_file() file descriptor exhaustion experiment
Hi,
I am struggling with exhaustion of file descriptors when using
evbuffer_add_file(). My application responds to every line of input with
contents of a file. I don't know how to efficiently control open
descriptors number so when input rate is high I hit the open file
descriptors limit very quickly. I would like to ask how to resolve the
problem correctly. I have seen that long time ago someone has asked a
similar question[1], but there was no answer.
I attach a short source code to present my problem.
Ideas made up so far (excerpt from the attached source):
Bad ideas:
1. Depending on bytes count in the output evbuffer would be
troublesome if the sent file changes between
evbuffer_add_file() calls -- it would require a list of last
N file sizes to check if evbuffer size is below the sum of N-K
sizes (N - high water, N-K - low water).
2. Checking fd number returned from open() wouldn't be enough
because after sending contents associated with the oldest call
to evbuffer_add_file() the fd number is freed and the next
call to open() returns a low number even though descriptors
with higher numbers are still in use.
3. Iterating through file descriptors to check their states seems
inefficient.
4. Listing files from /proc/self/fd seems inefficient.
5. One could implement a rate limit on requests but it's a quirk.
There is no guarantee of the output rate.
6. ???
Possibly useful ideas:
1. extend evbuffer_cb_info to have open file descriptors count
2. implement evbuffer descriptor closed callback
3. implement a function to check evbuffer open descriptors count
4. ???
[1] http://archives.seul.org/libevent/users/Nov-2011/msg00019.html
--
Marcin Szewczyk http://wodny.org
mailto:Marcin.Szewczyk@xxxxxxxxxx <- remove b / usuÅ b
xmpp:wodny@xxxxxxxxx xmpp:wodny@xxxxxxxxxx
/* evbuffer_add_file() file descriptor exhaustion experiment
*
* it connects to CONN_ADDRESS and for every received line it responds with
* contents of a file specified by argv[1]
*
* feeding command:
* (while true; do echo hello; done) | nc -q0 -l -p 5000
*
* version: 0.1
*
* gcc evbuffer_add_file.c -o evbuffer_add_file -levent -Wall -pedantic -std=gnu99 -g
*
* Marcin Szewczyk <libevent at wodny.org>
*
*/
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <error.h>
#include <errno.h>
#include <signal.h>
#include <event2/event.h>
#include <event2/bufferevent.h>
#include <event2/buffer.h>
#define CONN_ADDRESS "127.0.0.1:5000"
void logmsg (const char* message, ...) {
va_list args;
va_start(args, message);
vfprintf(stderr, message, args);
fprintf(stderr, "\n");
va_end(args);
}
struct context {
struct sockaddr_in maddr;
char *fpath;
struct evbuffer* evbuff_in;
struct evbuffer* evbuff_out;
};
void event_cb (struct bufferevent *bev, short events, void *ptr) {
struct context *ctx = ptr;
struct sockaddr_in addr;
socklen_t size = sizeof (addr);
int retval;
if (events & BEV_EVENT_CONNECTED) {
retval = getpeername (bufferevent_getfd (bev), (struct sockaddr*) &addr, &size);
if (retval) {
logmsg ("rejected connection");
} else {
logmsg ("connected");
ctx->evbuff_in = bufferevent_get_input(bev);
ctx->evbuff_out = bufferevent_get_output(bev);
}
} else if (events & BEV_EVENT_TIMEOUT) {
logmsg ("timeout");
bufferevent_enable(bev, EV_READ|EV_WRITE);
} else if (events & (BEV_EVENT_ERROR|BEV_EVENT_EOF)) {
logmsg ("connection closed");
exit (0);
}
}
void read_cb (struct bufferevent *bev, void *ptr) {
struct context *ctx = ptr;
char *line;
size_t len;
struct stat stat_;
int fd;
// How to implement a limit?
//
// Bad ideas:
// 1. Depending on bytes count in the output evbuffer would be
// troublesome if the sent file changes between
// evbuffer_add_file() calls -- it would require a list of last
// N file sizes to check if evbuffer size is below the sum of N-K
// sizes (N - high water, N-K - low water).
// 2. Checking fd number returned from open() wouldn't be enough
// because after sending contents associated with the oldest call
// to evbuffer_add_file() the fd number is freed and the next
// call to open() returns a low number even though descriptors
// with higher numbers are still in use.
// 3. Iterating through file descriptors to check their states seems
// inefficient.
// 4. Listing files from /proc/self/fd seems inefficient.
// 5. One could implement a rate limit on requests but it's a quirk.
// There is no guarantee of the output rate.
// 6. ???
//
// Possibly useful ideas:
// 1. extend evbuffer_cb_info to have open file descriptors count
// 2. implement evbuffer descriptor closed callback
// 3. implement a function to check evbuffer open descriptors count
// 4. ???
logmsg ("entered read_cb");
while ((line = evbuffer_readln (ctx->evbuff_in, &len, EVBUFFER_EOL_CRLF))) {
logmsg ("got line (output evbuffer length: %ld", evbuffer_get_length (ctx->evbuff_out));
fd = open (ctx->fpath, O_RDONLY);
if (fd < 0)
error (1, errno, "open");
fstat (fd, &stat_);
evbuffer_add_file (ctx->evbuff_out, fd, 0, stat_.st_size);
}
}
struct bufferevent* connection_new (struct event_base *base, struct context *ctx) {
int conn;
struct bufferevent *bev;
conn = socket (AF_INET, SOCK_STREAM, IPPROTO_TCP);
evutil_make_socket_nonblocking (conn);
// removing BEV_OPT_DEFER_CALLBACKS doesn't help
bev = bufferevent_socket_new (base, conn, BEV_OPT_CLOSE_ON_FREE | BEV_OPT_DEFER_CALLBACKS);
if (bev == NULL)
return NULL;
// not effective
bufferevent_setwatermark (bev, EV_READ, 0, 100);
bufferevent_setcb (bev, read_cb, NULL, event_cb, ctx);
bufferevent_enable (bev, EV_READ|EV_WRITE);
bufferevent_socket_connect (bev, (struct sockaddr*)&ctx->maddr, sizeof (ctx->maddr));
return bev;
}
int main (int argc, char **argv) {
struct event_base *base;
struct bufferevent *bev;
struct context ctx;
struct sigaction newaction;
int len = sizeof (struct sockaddr_in);
int retval;
if (argc != 2)
error (1, 0, "filename required");
memset (&ctx, 0, sizeof (struct context));
ctx.fpath = argv[1];
retval = evutil_parse_sockaddr_port (CONN_ADDRESS, (struct sockaddr*)&ctx.maddr, &len);
if (retval)
error (1, 0, "failed to parse address");
memset (&newaction, 0, sizeof(newaction));
newaction.sa_handler = SIG_IGN;
sigaction (SIGPIPE, &newaction, NULL);
base = event_base_new ();
if (base == NULL)
error (1, 0, "failed to create libevent base");
bev = connection_new (base, &ctx);
if (bev == NULL)
error (1, 0, "failed to create bufferevent");
event_base_dispatch (base);
return 0;
}