[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-commits] [tor/master] Write a bunch more test for doc/WritingTests
commit 9338847bf427b59d6dd5634fc2f8998ce0e269c1
Author: Nick Mathewson <nickm@xxxxxxxxxxxxxx>
Date: Wed Aug 5 11:47:38 2015 -0400
Write a bunch more test for doc/WritingTests
---
doc/WritingTests.txt | 128 +++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 117 insertions(+), 11 deletions(-)
diff --git a/doc/WritingTests.txt b/doc/WritingTests.txt
index b0f8722..62a17e3 100644
--- a/doc/WritingTests.txt
+++ b/doc/WritingTests.txt
@@ -32,25 +32,97 @@ set STEM_SOURCE_DIR to the checkout, and run "make test-stem".
To run the Chutney tests as well, fetch chutney from the git repository,
set CHUTNEY_PATH to the checkout, and run "make test-network".
+To run all of the above, run "make test-full".
+
+To run all of the above, plus tests that require a working connection to the
+internet, run "make test-full-online".
+
=== Running particular subtests
-XXXX WRITEME
+The Tor unit tests are divided into separate programs and a couple of
+bundled unit test programs.
+
+Separate programs are easy. For example, to run the memwipe tests in
+isolation, you just run ./src/test/test-memwipe .
+
+To run tests within the unit test programs, you can specify the name
+of the test. The string ".." can be used as a wildcard at the end of the
+test name. For example, to run all the cell format tests, enter
+"./src/test/test cellfmt/..". To run
+
+Many tests that need to mess with global state run in forked subprocesses in
+order to keep from contaminating one another. But when debugging a failing test,
+you might want to run it without forking a subprocess. To do so, use the
+"--no-fork" option with a single test. (If you specify it along with
+multiple tests, they might interfere.)
+
+You can turn on logging in the unit tests by passing one of "--debug",
+"--info", "--notice", or "--warn". By default only errors are displayed.
+
+Unit tests are divided into "./src/test/test" and "./src/test/test-slow".
+The former are those that should finish in a few seconds; the latter tend to
+take more time, and may include CPU-intensive operations, deliberate delays,
+and stuff like that.
=== Finding test coverage
When you configure Tor with the --enable-coverage option, it should
build with support for coverage in the unit tests, and in a special
-"tor-cov" binary. If you launch
+"tor-cov" binary.
+
+Then, run the tests you'd like to see coverage from. If you have old
+coverage output, you may need to run "reset-gcov" first.
+
+Now you've got a bunch of files scattered around your build directories
+called "*.gcda". In order to extract the coverage output from them, make a
+temporary directory for them and run "./scripts/test/coverage ${TMPDIR}",
+where ${TMPDIR} is the temporary directory you made. This will create a
+".gcov" file for each source file under tests, containing that file's source
+annotated with the number of times the tests hit each line. (You'll need to
+have gcov installed.)
+
+You can get a summary of the test coverage for each file by running
+"./scripts/test/cov-display ${TMPDIR}/*" . Each line lists the file's name,
+the number of uncovered lines, the number of uncovered lines, and the
+coverage percentage.
+
+For a summary of the test coverage for each _function_, run
+"./scripts/test/cov-display -f ${TMPDIR}/*" .
+
+=== Comparing test coverage
+
+Sometimes it's useful to compare test coverage for a branch you're writing to
+coverage from another branch (such as git master, for example). But you
+can't run "diff" on the two coverage outputs directly, since the actual
+number of times each line is executed aren't so important, and aren't wholly
+deterministic.
+
+Instead, follow the instructions above for each branch, creating a separate
+temporary directory for each. Then, run "./scripts/test/cov-diff ${D1}
+${D2}", where D1 and D2 are the directories you want to compare. This will
+produce a diff of the two directories, with all lines normalized to be either
+covered or uncovered.
+
+To count new or modified uncovered lines in D2, you can run:
+
+ "./scripts/test/cov-diff ${D1} ${D2}" | grep '^+ *\#' |wc -l
-XXXX "make test-network" doesn't know about "tor-cov"; you don't get
-XXXX coverage from that yet, unless you do "cp src/or/tor-cov
-XXXX src/or/tor" before you run it.
What kinds of test should I write?
----------------------------------
-XXXX writeme.
+Integration testing and unit testing are complementary: it's probably a
+good idea to make sure that your code is hit by both if you can.
+
+If your code is very-low level, and its behavior is easily described in
+terms of a relation between inputs and outputs, or a set of state
+transitions, then it's a natural fit for unit tests. (If not, please
+consider refactoring it until most of it _is_ a good fit for unit
+tests!)
+If your code adds new externally visible functionality to Tor, it would
+be great to have a test for that functionality. That's where
+integration tests more usually come in.
Unit and regression tests: Does this function do what it's supposed to?
-----------------------------------------------------------------------
@@ -117,7 +189,7 @@ Sometimes you need to test a function, but you don't want to expose
it outside its usual module.
To support this, Tor's build system compiles a testing version of
-teach module, with extra identifiers exposed. If you want to
+each module, with extra identifiers exposed. If you want to
declare a function as static but available for testing, use the
macro "STATIC" instead of "static". Then, make sure there's a
macro-protected declaration of the function in the module's header.
@@ -134,13 +206,47 @@ CRYPTO_CURVE25519_PRIVATE, so they can see this declaration.
=== Mock functions for testing in isolation
-Often we want to test that a function works right, but the function depends
-on other functions whose behavior is hard to observe, or whose
+Often we want to test that a function works right, but the function to
+be tested depends on other functions whose behavior is hard to observe,
+or which require a working Tor network, or something like that.
-XXXX WRITEME
+To write tests for this case, you can replace the underlying functions
+with testing stubs while your unit test is running. You need to declare
+the underlying function as 'mockable', as follows:
-=== Advanced techniques: Namespaces
+ MOCK_DECL(returntype, functionname, (argument list));
+
+and then later implement it as:
+
+ MOCK_IMPL(returntype, functionname, (argument list))
+ {
+ /* implementation here */
+ }
+
+For example, if you had a 'connect to remote server' function, you could
+declare it as:
+
+
+ MOCK_DECL(int, connect_to_remote, (const char *name, status_t *status));
+When you declare a function this way, it will be declared as normal in
+regular builds, but when the module is built for testing, it is declared
+as a function pointer initialized to the actual implementation.
+
+In your tests, if you want to override the function with a temporary
+replacement, you say:
+
+ MOCK(functionname, replacement_function_name);
+
+And later, you can restore the original function with:
+
+ UNMOCK(functionname);
+
+For more information, see the definitions of this mocking logic in
+testsupport.h.
+
+
+=== Advanced techniques: Namespaces
XXXX write this. danah boyd made us some really awesome stuff here.
_______________________________________________
tor-commits mailing list
tor-commits@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits