[minion-cvs] HALFTESTED] Make the server slightly multithreaded to k...

[nickm@seul.org (Nick Mathewson)]

Update of /home/minion/cvsroot/src/minion
In directory moria.mit.edu:/tmp/cvs-serv10668

Modified Files:
	TODO 
Log Message:
[HALFTESTED] Make the server slightly multithreaded to keep it from stalling.

Previously, we'd become unresponsive to the network while processing
packets, overwriting files, or sending messages via SMTP or Mixmaster.
Now we have a single separate thread to handle each of those, so we
will no longer see stalls of 30ms up to 5 minutes (!!).

The ideal is still for networking code to use the async server in
the main thread, whenever convenient and possible.

(NOTE: THIS CODE HAS WORKED FOR ME ON MY LAPTOP.  It is not tested or
documented enough to convince me that it is correct.  It has not been
tested on Linux. There are subtle issues about synchronizing access to
server queues that I need to hash out with myself before I trust this
code.  By all means feel free to play with it.... but if it breaks [as
Linus likes to say], you get to keep both pieces.)

BuildMessage:
- Make use of new Crypto.getTrueRNG.

Common:
- Import 'Queue' from the python distribution here.
- Make blocking securedelete work well with sigchild handler.
- Add 'blocking' option to waitForChildren.
- Remove unused, unusable signal stuff.

Crypto:
- Optimize getCommonPRNG for the common case
- Add a lock to TRNG
- Add an accessor function: getTrueRNG.

benchmark:
- Add note about limitations of hashlog benchmarks.

test, server.Queue<deleted>, server.ServerQueue<new>:
- Rename Queue.py to ServerQueue.py to avoid conflicting with Queue from
  the standard python distribution.
- Add locking -- possibly not completely right, and certainly not well
  documented, to StandardQueue.

HashLog:
- Add locks to hashlog.

ServerMain, Modules:
- The big threading patch, part 1.  Go from a single-thread model to one
  with 4 threads:
        * Main thread that runs the async server
	* Processing thread that handles all the crypto and packet magic
	* Cleaning thread that overwrites dead files
	* Delivery thread that handles all outgoing messages not bound
	  for the main server.  (Right now, that's everything but MMTP,
	  including SMTP and SMTP-via-Mixmaster).
  The code works for me, but needs some rethinking and cleanup in places.
- Catch HUP and TERM; take appropriate action on TERM.



Index: TODO
===================================================================
RCS file: /home/minion/cvsroot/src/minion/TODO,v
retrieving revision 1.60
retrieving revision 1.61
diff -u -d -r1.60 -r1.61
--- TODO	9 Jan 2003 05:51:19 -0000	1.60
+++ TODO	9 Jan 2003 06:28:57 -0000	1.61
@@ -53,9 +53,11 @@
                   down explicitly so they can't die halfway.
                 - Test like crazy
         - Signals
-                - Cleaner shutdown on TERM.
-                - Good reset handling for HUP.
-			- Don't die
+                . Cleaner shutdown on TERM.
+                        o Implement
+                        - Test
+                . Good reset handling for HUP.
+			o Don't die
                         - Restart logs
                         - Check configuration file for changeable
                           things.  (First, decide what's changeable on