[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-dev] Unit testing blog posts



Hey Damian!

Wasn't sure if follow up privately or to the list, sending this to the list for now.

On Thu, 16 Feb 2012 22:09:44 +0100, Damian Johnson <atagar1@xxxxxxxxx> wrote:

Nice posts and thanks for improving the tor tests! I'm not entirely in
agreement with the last point (about tests covering all cases). If the
test space is decently small then exercising everything can better
ensure that you don't violate a set of invariants. For example, one of
stem's unit tests attempt every combination of authentication methods
against every set of failures they can encounter, making sure that we
properly report success/failure and never raise an unexpected type of
exception...
https://gitweb.torproject.org/stem.git/blob/HEAD:/test/unit/connection/authentication.py#l69

But that said, in general you're probably right. -Damian

Thanks for the feedback. You're right, covering all cases is not bad in itself. The last point was the weakest, both because I had thought the least about it, and because I didn't have an example to back it up.

The main thing I had in mind when I wrote that point was some test cases I had recently reviewed at work. They make some statistical computations, and for the tests it *creates* hundreds of test cases (and calculates the expected values!) in a loop, then checks the actual values are the same as the (calculated) expected. That makes the test code as complex as the production code, making it very hard to know if the tests are correct, and once something fails, they're pretty hard to debug.

As I had seen it before, and my impression was that the reason for this happening was developers not realising test code is different than "production" code (among other things, because it doesn't have to be "generic" as in "covering all cases"). In hindsight, after you comment, maybe the main/only bad thing about that test was *calculating* the expected value.

--
Esteban
_______________________________________________
tor-dev mailing list
tor-dev@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev