[Libevent-users] pthread_cond_timedwait realtime clock race

[David Goulet <dgoulet@xxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi everyone,

I'm David from the LTTng project (http://lttng.org). I'm sharing here
an issue we ran into. The solution could benefit the libevent project.

A couple days ago we've hit a clock race using pthread timedwait
conditions (although documented in the man page). We've found a
solution and I wanted to share it with the libevent community since
I've checked and it seems that it can happen in libevent.

Here goes. The pthread_cond_timedwait() needs an absolute timeout
value, generally taken from the realtime clock of the system (using
gettimeofday or clock_gettime(CLOCK_REALTIME,...)). One point to
mention is that this clock can go backward or forward at any time with
for instance a NTP adjustment (no new knowledge here).

So, the race occurred between the clock time acquisition and the
pthread_cond_timedwait() call where it immediately returned with a
ETIMEOUT (normal behavior) but breaking a good part of the boot
process of our daemon. By the time the pthread cond call was
reschedule, the system had a NTP adjustment thus making the timeout
period invalid.

We've hit that race at a massive cluster reboot where only a couple of
nodes were affected by it since it's highly unlikely to encounter but
still possible.

After some research, we discovered a fantastic call,
pthread_condattr_setclock() which can be used to set the condition to
use the CLOCK_MONOTONIC hence having the guarantee to never jump in
time between calls.

Now, I know this is not new stuff but the fact that we've hit that
recently on large clusters of machines makes me think that it could be
very interesting that libevent cond wait call could avoid this race.
(evthread_posix_cond_wait).

Thoughts?

Thanks everyone! Cheers
David
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQWNYtAAoJEELoaioR9I027a0H/2wPAKVRH63zsIX2QVpwL76p
VfJGHk8b2S9i1av55JTyt8YmN+iXE1U7Qf7bNQA0VUy59Ykd5V+cqSLDbRCnfPog
fSeZyMFacWKHQ36JsbSb77xKYkMAC0d9n+Na+JRXjc4jdv5eTiI9vlCQDhyXFR9t
i4XHqDB97PJh/2xcecYlgtbOM5QeLcglDVQpcRIwhHzC9yx0eLKiPgU6+uMMequO
JK6e+N8pElwf8bywE/NnmLM2/NpWqwr7i3IbC+Zi8vgcI2yG+7rp/En+/B4SPA0m
qVGTV4NHAWTY4cmMxvJY9PXPc5l3Sp/sjziBZTlCgsMpT5Cnd/ygOaZGHnjq3T8=
=zUj4
-----END PGP SIGNATURE-----
***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.