Re: [Libevent-users] Infinite Loop in libevent.

[Mark Ellzey <mthomas@xxxxxxxxxx>]

On Fri, Jun 05, 2015 at 08:46:35PM +0000, Tomer Heber wrote:
> Hi,
> 
> 
> I've been using libevent (2.0.21) for some time now. I have encountered a very rare and strange issue on one of our production environments.
> 
> The machine was at 100% cpu, and had been caught on some kind of infinite loop. Using GDB I traced it to libevent.
> 
> 
> Just to clarify, I verified with tcpdump that the machine was not under a DDOS attack.
> 
> 
> strace showed a constant call to "gettimeofday" (or one of it's equivalent), and a failure due to too many open file descriptors.
> 
> 
> I was able to obtain the following from gdb
> 
> #0  0x00007fd495cbf2d3 in vfprintf () from /lib64/libc.so.6
> 
> #1  0x00007fd495d7aaa0 in __vsnprintf_chk () from /lib64/libc.so.6
> 
> #2  0x00007fd4969d71c2 in vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at /usr/include/bits/stdio2.h:78
> 
> #3  evutil_vsnprintf (buf=0x7fd48abfc468 ": Too many open files", buflen=1000, format=<value optimized out>, ap=<value optimized out>) at evutil.c:1577
> 
> #4  0x00007fd4969d7263 in evutil_snprintf (buf=<value optimized out>, buflen=<value optimized out>, format=<value optimized out>) at evutil.c:1554
> 
> #5  0x00007fd4969d623d in _warn_helper (severity=2, errstr=0x7fd495dd0d70 "Too many open files", fmt=<value optimized out>, ap=<value optimized out>) at log.c:183
> 
> #6  0x00007fd4969d6634 in event_sock_warn (sock=<value optimized out>, fmt=0x7fd4969ee50b "Error from accept() call") at log.c:124
> 
> #7  0x00007fd4969d365c in listener_read_cb (fd=41, what=<value optimized out>, p=0x7fd460004dc0) at listener.c:441
> 
> #8  0x00007fd4969c8f8c in event_process_active_single_queue (base=0x1046480, flags=0) at event.c:1350
> 
> #9  event_process_active (base=0x1046480, flags=0) at event.c:1420
> 
> #10 event_base_loop (base=0x1046480, flags=0) at event.c:1621
> 
> 
> Since it was a production environment there was little more I could in do. In the end GDB crashed the process (or the OS?).
> 
> 
> Any ideas?
> 
> 
> 10x,
> 
> Tomer.
> 

"#5  0x00007fd4969d623d in _warn_helper (severity=2,
errstr=0x7fd495dd0d70 "Too
many open files", fmt=<value optimized out>, ap=<value optimized out>)
at
log.c:183
"

You either you're not closing up sockets when you're done with them, or
you need to increase your max open fd limit. (ulimit -n). 
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.