TLS, threads, and workers

[Roger Dingledine <arma@mit.edu>]

Nick and I built a plan today. We're going to switch to TLS for links.
This has a number of related design fallouts.

Because we want to avoid half-second hiccups for the PK crypto ops
(non-blocking for SSL refers to network, not cpu), we must do the SSL
handshakes in a separate thread. Because we can't pass sockets between
processes, we're going to get rid of the current fork()ing model, and move
to pthreads. This means we'll be at the mercy of all the funny threading
implementations out there, but hopefully that won't be so bad. Besides,
forking sucked too.

We'll still have the async model. There will be one main thread, and
some workers. Workers only talk to the main thread (not to each other),
and only through a single shared work queue in memory (work units are:
function to call, data to give it, a callback to hand back along with
the answer). We might use a socketpair along with the work queue (shared
by all workers), to wake up the main thread while it's polling. We'll
leave threading details until after TLS is in.

Our read/write pollarray abstraction gets a bit more exciting, because
SSL can sometimes return "I want to read" when you try to write, and
vice versa. Not so bad.

We need to start using full-blown certificates rather than just PEM
encodings of public keys. Also not so bad.

TLS supports link key rotation very easily. This is good. Link key
rotation is generally fast (eg hashing the old key, some nonces, the
stuff that's gone over the wire), so doesn't need to use a separate
thread. (We still need to resolve rotating the encryption key, but we
needed to do that anyway.)

TLS has 24 bytes overhead on every 'record'. A record could be between
1 and 63 cells, depending on how many cells we ask it to send at once.
It's interesting to note that if the record is many cells, then SSL won't
admit to having any data until the entire record has arrived (the hash
is at the end of the record). Under normal circumstances this should
be fine. But it could turn ugly: for example, how well does it handle
bandwidth throttling and TCP pushback? This becomes especially important
when limited bandwidth is spread over hundreds of connections. We will
have to throttle in terms of reading from and writing to the SSL objects;
I'm not about to start messing with the TLS stuff on the wire (and OpenSSL
doesn't let us, anyway). Hopefully this will all have a happy ending;
we'll just have to see how it performs.

Our dream of moving to a trivial little crypto library is now gone:
we are married to OpenSSL (or something equally complex).

--Roger