Re: TLS, threads, and workers

[Nick Mathewson <nickm@alum.mit.edu>]

On Thu, 2003-09-04 at 12:20, Paul Syverson wrote:
> On Thu, Sep 04, 2003 at 03:23:35AM -0400, Roger Dingledine wrote:
> > Nick and I built a plan today. We're going to switch to TLS for links.
> > This has a number of related design fallouts.
> 
> I hope this is a plan rather than a done decision at this point.  I am
> concerned that this has not been adequately discussed. I should also
> say up front that I am unqualified to discuss details given my
> complete ignorance of coding but I think my high-level points are
> still cogent.

Here's what we're doing, and why.  We're starting to implement TLS, but
we haven't gone threaded yet.  I don't think we should go threaded
until/unless these issues are settled.

The reasons in favor of TLS are pretty clear: it's a well analyzed link
protocol that gives us integrity, forward security, and mutual
authentication, all for free.

The reason that TLS links might argue for a multithreaded design is
that, with all the TLS implementations I know of, I don't know any good
way to separate the SSL handshake crypto operations from the SSL
handshake IO operations.  Thus, if we want to put the RSA operations in
a separate execution context (i.e., process or thread) in order to avoid
high-latency operations, we'll need to put the socket operations there
too.  But you can't pass sockets back and forth across processes; hence
threads seem smart.

Another option is to go heavily multiprocess: stick _all_ link
encryption operations in separate processes from the main onion routing
process.  The link encryption processes could send unencrypted cells
back and forth from the main OR process.  This could scale poorly, by
requiring many processes.

Another option is to accept hiccups while handshakes are going on.  If
we later impose some kind of proof of work requirement before beginning
the handshake, this might not be so bad.  On the box that the ORs are
currently running on, 1024 bit RSA decrypt takes 6 ms, 1536 bit RSA
decrypt takes 19 ms, and 2048 bit RSA decrypt takes 36 ms.

Another option is to try to hack up OpenSSL to separate the crypto parts
from the IO parts.  I don't have a lot of confidence here--OpenSSL
internals can get complicated, and I don't see any hooks that do what we
want.

Is there a better solution to this problem?  I would also like to avoid
going multithreaded if possible, but I'm not sure it's possible.

> I am very hesitant to move to a threading model without a clear idea
> that relevant problems can be handled. I was especially concerned
> about portability as soon as it was brought up and was not heartened
> to see Ben's comment that FreeBSD handles threads poorly.

While I don't question Ben's experience, I must add another data point
--- Mixminion uses a lightly multithreaded design, and seems to run fine
on FreeBSD.  ("I've never had any complaints." :P )  Ben: can you point
me towards the discussion where apache2 decided that FreeBSD threads
were inadequate to their needs?  Perhaps they discuss specifically what
doesn't work right.

Another data point: stunnel seems to use threads when it can.

>                                                             I'm also
> skittish because we started using threads in a previous incarnation of
> onion routing because it seemed to make alot of sense but had to
> switch back because of assorted problems, portability, crypto
> libraries we wanted to allow people to choose from weren't thread
> safe, and so on.  I would like to hear a more about why it is thought
> that these issues will not be big problems this time.

"assorted problems" -- Can't comment.
"portability" -- If we got pthreads and windows threads, we'd have
almost the whole market.  Of course, we'd have to stick to the common
subset of all such implementations, which could indeed be tricky. 
Nonetheless, there are enough examples of portable thread wrappers
(Apache,Python,Perl,glib,stunnel,etc.) that we one of us can probably
reverse engineer a threading discipline 
"Non-threadsafe libraries" -- OpenSSL (which we're using now) can
supposedly be compiled in a thread-safe mode.  GnuTLS is supposedly
threadsafe.  NSS is supposed to be threadsafe, if I read their
documentation correctly.

So as you can probably tell, I currently regard threading as the least
unworkable solution.  But if there's an even less unworkable solution,
I'd rather do that instead.

-- 
Nick