[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-commits] [tor/master] finish up doc/WritingTests.txt
commit f3804a4d6f488e2cd35001ec856ebc79ed769bbb
Author: Nick Mathewson <nickm@xxxxxxxxxxxxxx>
Date: Thu Oct 8 10:55:31 2015 -0400
finish up doc/WritingTests.txt
---
doc/WritingTests.txt | 144 +++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 136 insertions(+), 8 deletions(-)
diff --git a/doc/WritingTests.txt b/doc/WritingTests.txt
index 62a17e3..977b836 100644
--- a/doc/WritingTests.txt
+++ b/doc/WritingTests.txt
@@ -66,6 +66,8 @@ and stuff like that.
=== Finding test coverage
+Test coverage is a measurement of which lines your tests actually visit.
+
When you configure Tor with the --enable-coverage option, it should
build with support for coverage in the unit tests, and in a special
"tor-cov" binary.
@@ -245,29 +247,155 @@ And later, you can restore the original function with:
For more information, see the definitions of this mocking logic in
testsupport.h.
+=== Okay but what should my tests actually do?
+
+We talk above about "test coverage" -- making sure that your tests visit
+every line of code, or every branch of code. But visiting the code isn't
+enough: we want to verify that it's correct.
+
+So when writing tests, try to make tests that should pass with any correct
+implementation of the code, and that should fail if the code doesn't do what
+it's supposed to do.
+
+You can write "black-box" tests or "glass-box" tests. A black-box test is
+one that you write without looking at the structure of the function. A
+glass-box one is one you implement while looking at how the function is
+implemented.
+
+In either case, make sure to consider common cases *and* edge cases; success
+cases and failure csaes.
+
+For example, consider testing this function:
+
+ /** Remove all elements E from sl such that E==element. Preserve
+ * the order of any elements before E, but elements after E can be
+ * rearranged.
+ */
+ void smartlist_remove(smartlist_t *sl, const void *element);
+
+In order to test it well, you should write tests for at least all of the
+following cases. (These would be black-box tests, since we're only looking
+at the declared behavior for the function:
+
+ * Remove an element that is in the smartlist.
+ * Remove an element that is not in the smartlist.
+ * Remove an element that appears in the smartlist more than once.
+
+And your tests should verify that it behaves correct. At minimum, you should
+test:
+
+ * That other elements before E are in the same order after you call the
+ functions.
+ * That the target element is really removed.
+ * That _only_ the target element is removed.
+
+When you consider edge cases, you might try:
+
+ * Remove an element from an empty list.
+ * Remove an element from a singleton list containing that element.
+ * Remove an element for a list containing several instances of that
+ element, and nothing else.
+
+Now let's look at the implementation:
+
+ void
+ smartlist_remove(smartlist_t *sl, const void *element)
+ {
+ int i;
+ if (element == NULL)
+ return;
+ for (i=0; i < sl->num_used; i++)
+ if (sl->list[i] == element) {
+ sl->list[i] = sl->list[--sl->num_used]; /* swap with the end */
+ i--; /* so we process the new i'th element */
+ sl->list[sl->num_used] = NULL;
+ }
+ }
+
+Based on the implementation, we now see three more edge cases to test:
+
+ * Removing NULL from the list.
+ * Removing an element from the end of the list
+ * Removing an element from a position other than the end of the list.
+
+
+=== What should my tests NOT do?
+
+Tests shouldn't require a network connection.
+
+Whenever possible, tests shouldn't take more than a second. Put the test
+into test/slow if it genuinely needs to be run.
+
+Tests should not alter global state unless they run with TT_FORK: Tests
+should not require other tests to be run before or after them.
+
+Tests should not leak memory or other resources.
+
+When possible, tests should not be over-fit to the implementation. That is,
+the test should verify that the documented behavior is implemented, but
+should not break if other permissible behavior is later implemented.
+
=== Advanced techniques: Namespaces
-XXXX write this. danah boyd made us some really awesome stuff here.
+Sometimes, when you're doing a lot of mocking at once, it's convenient to
+isolate your identifiers within a single namespace. If this were C++, we'd
+already have namespaces, but for C, we do the best we can with macros and
+token-pasting.
+
+We have some macros defined for this purpose in src/test/test.h. To use
+them, you define NS_MODULE to a prefix to be used for your identifiers, and
+then use other macros in place of identifier names. See src/test/test.h for
+more documentation.
Integration tests: Calling Tor from the outside
-----------------------------------------------
-XXXX WRITEME
+Some tests need to invoke Tor from the outside, and shouldn't run from the
+same process as the Tor test program. Reasons for doing this might include:
+
+ * Testing the actual behavior of Tor when run from the command line
+ * Testing that a crash-handler correctly logs a stack trace
+ * Verifying that a violating a sandbox or capability requirement will
+ actually crash the program.
+ * Needing to run as root in order to test capability inheritance or
+ user switching.
+
+To add one of these, you generally want a new C program in src/test. Add it
+to TESTS and noinst_PROGRAMS if it can run on its own and return success or
+failure. If it needs to be invoked multiple times, or it needs to be
+wrapped, add a new shell script to TESTS, and the new program to
+noinst_PROGRAMS. If you need access to any environment variable from the
+makefile (eg ${PYTHON} for a python interpreter), then make sure that the
+makefile exports them.
Writing integration tests with Stem
-----------------------------------
-XXXX WRITEME
+The 'stem' library includes extensive unit tests for the Tor controller
+protocol.
+
+For more information on writing new tests for stem, have a look around
+the tst/* directory in stem, and find a good example to emulate. You
+might want to start with
+https://gitweb.torproject.org/stem.git/tree/test/integ/control/controller.py
+to improve Tor's test coverage.
+
+You can run stem tests from tor with "make test-stem", or see
+https://stem.torproject.org/faq.html#how-do-i-run-the-tests .
System testing with Chutney
---------------------------
-XXXX WRITEME
-
-Who knows what evil lurks in the timings of networks? The Shadow knows!
------------------------------------------------------------------------
+The 'chutney' program configures and launches a set of Tor relays,
+authorities, and clients on your local host. It has a 'test network'
+functionality to send traffic through them and verify that the traffic
+arrives correctly.
-XXXX WRITEME
+You can write new test networks by adding them to 'networks'. To add
+them to Tor's tests, add them to the test-network or test-network-all
+targets in Makefile.am.
+(Adding new kinds of program to chutney will still require hacking the
+code.)
_______________________________________________
tor-commits mailing list
tor-commits@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits