[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-dev] Integration testing plans, draft, v1
Hi! This is the outcome of some discussion I've had with dgoulet, and
some work I've done at identifying current problem-points in our use
of integration testing tools.
I'm posting it here for initial feedback, and so I have a URL to link
to in my monthly report. :)
INTEGRATION TEST PLANS FOR TOR
November 2014
1. Goals and nongoals and scope
This is not a list of all the tests we need; this is just a list of
the kind of tests we can and should run with Chutney.
These tests need to be the kind that a random developer can run on
their own and reasonably hope to do. Longer/more expensive tests
may be okay too, but if it needs anything spiffer than a linux
desktop, consider using Shadow instead.
Setting up an environment to run the tests needs to be so easy that
nobody who writes C for Tor is likely to be dissuaded from running
them.
Writing new tests needs to be pretty simple too.
Most tests need to be runnable on all the platforms we support.
We should support load tests. Though doing so is not likely to give
an accurate picture of how the network behaves under load, it's
probably good enough to identify bottlenecks in the code. (David
Goulet has had some success here already for identifying HS
performance issues.)
We should specify our design and interfaces to keep components
loosely coupled and easily replaceable. With that done, we should
avoid over-designing components at first: experience teaches that
only experience can teach what facilities need which features.
2. Architecture
Here are the components. I'm treating them as conceptually
separate, though in practice several of them may get folded into
Chutney.
A. Usage simulator
One or more programs that emulate users and servers on the
internet. It reports what succeeded and what failed and how long
everything took. Right now we're using curl and nc for this.
B. Network manager
This is what Chutney does today. It launches a set of Tor nodes
according to a provided configuration.
C. Testcase scripts
We do this in shell today: launch the network, wait for it to
bootstrap, send some traffic through it, and report success or
failure.
D. Test driver
This part is responsible for determining which testcases to run,
in what order, on what network.
There is no current analogue to this step; we've only got the one
test-network.sh script, and it assumes a single type of network.
One thing to notice here is that testcase scripts need to work with
multiple kinds of network manager configurations. For example, we'd
like to be able to run HTTP-style connectivity tests on small
networks, large networks, heterogenous networks, dispersed networks,
and so on. We therefore need to make sure that each kind of network
can work with as many tests as possible, so that the work needed to
write the tests doesn't grow quadratically.
The coupling for the components will go as possible:
A. Usage simulations will need to expose their status to test
scripts.
B. The network manager will need to expose information about
available networks and network configurations to the test scripts,
so that the test scripts know how to configure usage simulations to
use them. It will need to expose commands like "wait until
bootstrapped", "check server logs for trouble", etc.
C. Each testcase needs to be able to identify which features it
needs from a network, invoke the network commands it needs, and
invoke usage simulations. It needs to export information about its
running status, and whether it's making progress.
D. The test driver needs to be able to enumerate networks and
testcases and figure out which are compatible with each other, and
which can run locally, and which ones meet the user's requirements.
2.1. Minimal versions of the above:
A. The minimal user tools are an http server and client. Use
appropriate tooling to support generating and receiving hundreds to
thousands of simultaneous requests.
B. The minimal network manager is probably chutney. It needs the
ability to export information about networks, to "wait until
bootstrapped", to export information about servers, and so on. Its
network list needs to turn into a database of named networks.
C. Testcases need to be independent, and ideally abstracted. They
shouldn't run in-process with chutney. For starters, they can
duplicate the current functionality of test-network and of dgoulet's
hidden service tests. Probing for features can be keyword-based.
reporting results can use some logging framework.
D. The test driver can do its initial matching by keyword tags
exported by the other objects. It should treat testcases and
networks as abitrary subprocesses that it can launch, so that they
can be written in any language.
3. A short inventory of use-cases
- Voting
- Controllers
- HS
* IP+RP up/down
* HSDir up/down
* Authenticated HS
- Bad clients
* Wrong handshake
* Wrong desc.
- Bad relays
* dropping cells/circ/traffic
* BAD TLS
- Pathing
* Does it behaves the way we think?
* 4 hops for HS
- Relay
* Up/Down
* Multiple IPs for a single key
* OOM handling
* Scheduling
- Client
* Does traffic goes through?
- For HS and Exit
* DNS testing
- Caches at the exit
* Stream isolation using n SocksPort
* AddressMap
4. Longer-term
We might look into using "The Internet" (no, not that one. The one
at https://github.com/nsec/the-internet) on Linux to simulate latency
between nodes.
When designing tools and systems, we should do so with an eye to
migrating them into shadow.
We should refactor or adapt chutney to support using stem, to take
advantage of stem's improved templating and control features, so
we can better inspect running servers. We should probably retain the
original hands-off non-controller chutney design too, to better
detect heisenbugs.
5. Immediate steps
- Turn the above into a work plan.
- Specify initial interfaces, with plan for migration to better ones
once we have more experience.
- Identify current chutney and test-network issues standing in the
way of reliably getting current tests to work.
- Refactor our current integration tests to the above framework.
_______________________________________________
tor-dev mailing list
tor-dev@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev