|
Hi,
I've been using libevent (2.0.21) for some time now. I have encountered a very rare and strange issue on one of our production environments. The machine was at 100% cpu, and had been caught on some kind of infinite loop. Using GDB I traced it to libevent.
Just to clarify, I verified with tcpdump that the machine was not under a DDOS attack.
strace showed a constant call to "gettimeofday" (or one of it's equivalent), and a failure due to too many open file descriptors.
I was able to obtain the following from gdb #0 0x00007fd495cbf2d3 in vfprintf () from /lib64/libc.so.6 #1 0x00007fd495d7aaa0 in __vsnprintf_chk () from /lib64/libc.so.6 #2 0x00007fd4969d71c2 in vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at /usr/include/bits/stdio2.h:78 #3 evutil_vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at evutil.c:1577 #4 0x00007fd4969d7263 in evutil_snprintf (buf=<value optimized out>, buflen=<value optimized out>, format=<value optimized out>) at evutil.c:1554 #5 0x00007fd4969d623d in _warn_helper (severity=2, errstr=0x7fd495dd0d70 "Too many open files", fmt=<value optimized out>, ap=<value optimized out>) at log.c:183 #6 0x00007fd4969d6634 in event_sock_warn (sock=<value optimized out>, fmt=0x7fd4969ee50b "Error from accept() call") at log.c:124 #7 0x00007fd4969d365c in listener_read_cb (fd=41, what=<value optimized out>, p=0x7fd460004dc0) at listener.c:441 #8 0x00007fd4969c8f8c in event_process_active_single_queue (base=0x1046480, flags=0) at event.c:1350 #9 event_process_active (base=0x1046480, flags=0) at event.c:1420
#10 event_base_loop (base=0x1046480, flags=0) at event.c:1621
Since it was a production environment there was little more I could in do. In the end GDB crashed the process (or the OS?).
Any ideas?
10x,
Tomer.
|