[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [Libevent-users] Test failures on Mac OS X 10.5.8




On Jun 11, 2011, at 12:28 AM, Nick Mathewson wrote:

On Wed, Jun 8, 2011 at 9:28 PM, Nir Soffer <nirsof@xxxxxxxxx> wrote:
Hi,

I'm getting some random dns failures with versions from the last week:

The failures are random - sometimes all the tests pass, sometimes some of the backends fail in the same place, and sometimes all of them fail in the
same place.

I did not see these errors in 2.0.11-stable, but maybe I did not run the
tests enough. I did not test the latest release yet (2.0.12-stable).

The most common error is in resolve_reverse.

It looks here like your nameserver isn't answering requests about
1.0.0.127.in-addr.arpa. , or it's answering them but Libevent isn't
getting the answers.  This is likelier to be an issue with your
nameserver than with Libevent, though it does show a flaw in the unit
tests.  A few of the dns unit tests in test/regress_dns.c require you
to have a working nameserver, which arguably is a dumb thing to
assume.

I'm not sure of the best fix here.

A) We could have the tests use Libevent's own dns server
functionality, but that isn't such a great way to find bugs in
Libevent, since any misunderstanding of DNS by the client code is
likely to be matched by a misunderstanding in the server code.

B) We could have the tests set up a dummy DNS server to look at the
format of requests, compare them to the expected format, and send back
canned replies, but that would be a fair bit of work.

C) We could have the tests respond to failure by explaining what's
going on and asking the user to freak out only if their nameserver
ought to be answering requests as expected.

D) We could disable the affected tests by default.

Thoughts?

I usually test using B - use fake server/client and send both good and bad messages and see how the tested code behave. These tests can always run and do not depend on the development environment. Of course this will not detect issue with certain real servers/clients. When you find such issue in the field, you add more test with data collected from these.

Testing against real client/server is also important, but should not be part of the main test suite, since it may not work on some environment because while the code is correct, or the tests takes too much time.

So lets separate the dns tests that depends on a dns server/client from "make check" and run them with something like "make check-dns".

***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users    in the body.