On Fri, Jun 05, 2015 at 08:46:35PM +0000, Tomer Heber wrote:
> Hi,
>
>
> I've been using libevent (2.0.21) for some time now. I have encountered a very rare and strange issue on one of our production environments.
>
> The machine was at 100% cpu, and had been caught on some kind of infinite loop. Using GDB I traced it to libevent.
>
>
> Just to clarify, I verified with tcpdump that the machine was not under a DDOS attack.
>
>
> strace showed a constant call to "gettimeofday" (or one of it's equivalent), and a failure due to too many open file descriptors.
>
>
> I was able to obtain the following from gdb
>
> #0 0x00007fd495cbf2d3 in vfprintf () from /lib64/libc.so.6
>
> #1 0x00007fd495d7aaa0 in __vsnprintf_chk () from /lib64/libc.so.6
>
> #2 0x00007fd4969d71c2 in vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at /usr/include/bits/stdio2.h:78
>
> #3 evutil_vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at evutil.c:1577
>
> #4 0x00007fd4969d7263 in evutil_snprintf (buf=<value optimized out>, buflen=<value optimized out>, format=<value optimized out>) at evutil.c:1554
>
> #5 0x00007fd4969d623d in _warn_helper (severity=2, errstr=0x7fd495dd0d70 "Too many open files", fmt=<value optimized out>, ap=<value optimized out>) at log.c:183
>
> #6 0x00007fd4969d6634 in event_sock_warn (sock=<value optimized out>, fmt=0x7fd4969ee50b "Error from accept() call") at log.c:124
>
> #7 0x00007fd4969d365c in listener_read_cb (fd=41, what=<value optimized out>, p=0x7fd460004dc0) at listener.c:441
>
> #8 0x00007fd4969c8f8c in event_process_active_single_queue (base=0x1046480, flags=0) at event.c:1350
>
> #9 event_process_active (base=0x1046480, flags=0) at event.c:1420
>
> #10 event_base_loop (base=0x1046480, flags=0) at event.c:1621
>
>
> Since it was a production environment there was little more I could in do. In the end GDB crashed the process (or the OS?).
>
>
> Any ideas?
>
>
> 10x,
>
> Tomer.
>
"#5 0x00007fd4969d623d in _warn_helper (severity=2,
errstr=0x7fd495dd0d70 "Too
many open files", fmt=<value optimized out>, ap=<value optimized out>)
at
log.c:183
"
You either you're not closing up sockets when you're done with them, or
you need to increase your max open fd limit. (ulimit -n).
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.