[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[minion-cvs] Resolve numerous TODO items, including path selection, ...



Update of /home/minion/cvsroot/src/minion/lib/mixminion
In directory moria.mit.edu:/tmp/cvs-serv25930/lib/mixminion

Modified Files:
	ClientDirectory.py ClientMain.py ClientUtils.py Config.py 
	MMTPClient.py Main.py NetUtils.py Packet.py ServerInfo.py 
	test.py 
Log Message:
Resolve numerous TODO items, including path selection, unit tests,
bugfixes, and UI issues.

ClientDirectory:
  - Add configurable timeouts for directory retrieval
  - Make ClientDirectories, when available, help map keyids to nicer,
    more readily displayable strings.
  - Debug getNicknameByKeyID
  - Debug compressFormatMap
  - Make syntax for exit addresses more flexible
  - Only warn once for each unrecognized exit type.
  - Add 'getAvgLength' to PathElement
  - Minimize ~ elements to 1 hop
  - Add a kludge to allow paths of the form ~N to work as forward
    paths.

ClientMain:
  - Deal with function renaming
  - Add (partial) support for sending messages for user-side
    fragmentation
  - ***ALWAYS*** scramble packet order before sending them.
  - Rename Messages to Packets throughout as appropriate.
  - Bugfix: only de-queue  messages when delivery is successful.
  - Refactor flushQueue to use existing support functions.
  - Rename list-servers options again: --cascade and
    --cascade-features are probably more readable than --cascade=1
    and --cascade=2.
  
ClientUtils:
  - Rename LazyEncryptedPickle to LazyEncryptedStore; begin refactoring
  - Use displayServer() as appropriate

Config:
  - Make _formatEntry use new unparsing functionality.
  - Add options for user-configurable timeout.

MMTPClient:
  - Rename 'message' to 'packet' as appropriate.
  - Use new displayServer() function to pretty-print server names
    instead of IPs whenever possible.
  - Change error-printing options to handle timeouts and ssl version 
    errors better.

Main:
  - Add stubbed code to handle invocation as 'mixminiond': defer for
    now.

NetUtils: 
  - Add strings to TimeoutErrors

Packet:
  - Rename 'message' to 'packet' as appropriate.
  - Improve display of reply blocks.

ServerInfo:
  - Add new displayServer() function to pretty-print servers, looking
    up their actual nicknames whereever possible.  Servers can't take
    full advantage of this yet, since they don't download directories,
    but they can still call displayServer without harm.


test:
  - Rename 'message' to 'packet' as appropriate
  - Adjust ServerInfo tests to not try to make server with 'frag' but
    no actual delivery module, since this isn't allowed any more.
  - Add tests for getFeature()
  - Add tests for featureMap-related functions
  - Make tests for DNS lookup less time-dependent
  - Tests for new literal exit type behavior.

MMTPServer:
  - Rename 'message' to 'packet' as appropriate
  - Don't alias the LOG.* functions any more.  Apparently, this was
    allowing a long-standing bug where all DEBUG messages in
    MMTPServer were getting tagged as INFO messages.
  - (Partial) Use displayServer() as appropriate.

Modules:
  - Don't allow Delivery/Fragmented unless some other kind of delivery
    method is supported.

PacketHandler, ServerMain:
  - s/message/packet/ as appropriate

  
  

  



Index: ClientDirectory.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/ClientDirectory.py,v
retrieving revision 1.14
retrieving revision 1.15
diff -u -d -r1.14 -r1.15
--- ClientDirectory.py	10 Nov 2003 04:12:20 -0000	1.14
+++ ClientDirectory.py	19 Nov 2003 09:48:09 -0000	1.15
@@ -92,18 +92,19 @@
         finally:
             mixminion.ClientMain.clientUnlock() # XXXX
 
-    def updateDirectory(self, forceDownload=0, now=None):
+    def updateDirectory(self, forceDownload=0, timeout=None, now=None):
         """Download a directory from the network as needed."""
         if now is None:
             now = time.time()
 
         if forceDownload or self.lastDownload < previousMidnight(now):
-            self.downloadDirectory()
+            self.downloadDirectory(timeout=timeout)
         else:
             LOG.debug("Directory is up to date.")
-    def downloadDirectory(self, timeout=15):
+
+    def downloadDirectory(self, timeout=None):
         """Download a new directory from the network, validate it, and
-           rescan its servers."""
+           rescan its servers. DOCDOC timeout"""
         # Start downloading the directory.
         url = MIXMINION_DIRECTORY_URL
         LOG.info("Downloading directory from %s", url)
@@ -127,7 +128,7 @@
                 if mixminion.NetUtils.exceptionIsTimeout(e):
                     raise UIError("Connection to directory server timed out")
                 else:
-                    raise UIError("Error connecting: %s"%e)
+                    raise UIError("Error connecting to directory server: %s"%e)
         finally:
             if timeout:
                 mixminion.NetUtils.unsetGlobalTimeout()
@@ -277,6 +278,10 @@
                 self.digestMap)
         writePickled(os.path.join(self.dir, "cache"), data)
 
+    def _installAsKeyIDResolver(self):
+        """DOCDOC"""
+        mixminion.ServerInfo._keyIDToNicknameFn = self.getNicknameByKeyID
+
     def importFromFile(self, filename):
         """Import a new server descriptor stored in 'filename'"""
 
@@ -453,9 +458,9 @@
         if not s:
             return None
         r = []
-        for d in s:
-            if d.getNickname().lower() not in r:
-                r.append(d.getNickname())
+        for (desc,_) in s:
+            if desc.getNickname().lower() not in r:
+                r.append(desc.getNickname())
         return "/".join(r)
 
     def getNameByRelay(self, routingType, routingInfo):
@@ -858,6 +863,7 @@
         if not result[nickname]: continue
         
         ritems = result[nickname].items()
+        ritems.sort()
         minva = min([ va for (va,vu),features in ritems ])
         maxvu = max([ vu for (va,vu),features in ritems ])
         rfeatures = {}
@@ -910,7 +916,7 @@
                         sep.join([fmap[f] for f in features])))
             elif cascade==2:
                 if showTime:
-                    lines.append("  [%s]"%ftime)    
+                    lines.append("  [%s]"%ftime)
                 for f in features:
                     v = fmap[f]
                     lines.append("    %s:%s"%(f,v))
@@ -929,6 +935,9 @@
     "mbox", "smtp", "drop"
 ]
 
+# Map from (type, nickname) to 1 for servers we've already warned about.
+WARN_HISTORY = {}
+
 class ExitAddress:
     """An ExitAddress represents the target of a Mixminion message or SURB.
        It also encodes other properties off the message that must be known to
@@ -1060,8 +1069,11 @@
             pass
         else:
             if not verbose: return
+            if WARN_HISTORY.has_key((self.exitType, nickname)):
+                return
             LOG.warn("No way to tell if server %s supports exit type %s.",
                      nickname, self.getPrettyExitType())
+            WARN_HISTORY[(self.exitType, nickname)] = 1
 
     def getPrettyExitType(self):
         """Return a human-readable representation of the exit type."""
@@ -1114,11 +1126,23 @@
            OR <email address> (smtp is implicit)
            OR drop
            OR 0x<routing type>:<routing info>
+           OR 0x<routing type>
     """
     if s.lower() == 'drop':
         return ExitAddress('drop',"")
     elif s.lower() == 'test':
         return ExitAddress(0xFFFE, "")
+    elif s.startswith("0x") or s.startswith("0X"):
+        # Address of the form 0xABCD and 0xABCD:address
+        if len(s) < 6 or (len(s)>=7 and s[6] != ':'):
+            raise ParseError("Invalid address %r"%s)
+        try:
+            tp = int(s[2:6],16)
+        except ValueError:
+            raise ParseError("Invalid hexidecimal value %r"%s[2:6])
+        if not (0x0000 <= tp <= 0xFFFF):
+            raise ParseError("Invalid type: 0x%04x"%tp)
+        return ExitAddress(tp, s[7:])
     elif ':' not in s:
         if isSMTPMailbox(s):
             return ExitAddress('smtp', s)
@@ -1126,15 +1150,7 @@
             raise ParseError("Can't parse address %s"%s)
     tp,val = s.split(':', 1)
     tp = tp.lower()
-    if tp.startswith("0x"):
-        try:
-            tp = int(tp[2:], 16)
-        except ValueError:
-            raise ParseError("Invalid hexidecimal value %s"%tp)
-        if not (0x0000 <= tp <= 0xFFFF):
-            raise ParseError("Invalid type: 0x%04x"%tp)
-        return ExitAddress(tp, val)
-    elif tp == 'mbox':
+    if tp == 'mbox':
         if "@" in val:
             mbox, server = val.split("@",1)
             return ExitAddress('mbox', parseMBOXInfo(mbox).pack(), server)
@@ -1168,6 +1184,10 @@
         """Return the fewest number of servers that this element might
            contain."""
         raise NotImplemented()
+    def getAvgLength(self):
+        """Return the likeliest number of servers for this element to
+           contain."""
+        return self.getMinLength()
 
 class ServerPathElement(PathElement):
     """A path element for a single server specified by filename or nickname"""
@@ -1225,12 +1245,17 @@
         else:
             prng = mixminion.Crypto.getCommonPRNG()
             n = int(prng.getNormal(self.approx,1.5)+0.5)
+            if n < 1: n = 1
         return [ None ] * n
     def getMinLength(self):
-        #XXXX006 need getAvgLength too, probably.  Ugh.
         if self.n is not None: 
             return self.n
         else:
+            return 1
+    def getAvgLength(self):
+        if self.n is not None:
+            return self.n
+        else:
             return self.approx
     def __repr__(self):
         if self.n:
@@ -1239,13 +1264,13 @@
         else:
             return "RandomServersPathElement(approx=%r)"%self.approx
     def __str__(self):
-        if self.n == 1:
-            return "?"
-        elif self.n > 1:
-            return "*%d"%self.n
-        else:
+        if self.n is None:
             assert self.approx
             return "~%d"%self.approx
+        elif self.n == 1:
+            return "?"
+        else:
+            return "*%d"%self.n
 
 #----------------------------------------------------------------------
 class PathSpecifier:
@@ -1400,7 +1425,7 @@
         if "*" in pathEntries[starPos+1:]:
             raise UIError("Only one '*' is permitted in a single path")
         approxHops = reduce(operator.add,
-                            [ ent.getMinLength() for ent in pathEntries
+                            [ ent.getAvgLength() for ent in pathEntries
                               if ent not in ("*", "<swap>") ], 0)
         myNHops = nHops or defaultNHops or 6
         extraHops = max(myNHops-approxHops, 0)
@@ -1429,6 +1454,17 @@
         firstLegLen = 0
         lateSplit = 1
 
+    # This is a kludge to convert paths of the form ~N to ?,~(N-1) when we've
+    # got a full path.
+    if (len(pathEntries) == 1
+        and not halfPath
+        and isinstance(pathEntries[0], RandomServersPathElement)
+        and pathEntries[0].approx):
+        n_minus_1 = max(pathEntries[0].approx-1,0)
+        pathEntries = [ RandomServersPathElement(n=1),
+                        RandomServersPathElement(approx=n_minus_1) ]
+        lateSplit = 1 # XXXX Is this redundant?
+        
     # Split the path into 2 legs.
     path1, path2 = pathEntries[:firstLegLen], pathEntries[firstLegLen:]
 

Index: ClientMain.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/ClientMain.py,v
retrieving revision 1.129
retrieving revision 1.130
diff -u -d -r1.129 -r1.130
--- ClientMain.py	10 Nov 2003 04:12:20 -0000	1.129
+++ ClientMain.py	19 Nov 2003 09:48:09 -0000	1.130
@@ -30,10 +30,11 @@
      parseReplyBlocks, parseSMTPInfo, parseTextEncodedMessages, \
      parseTextReplyBlocks, ReplyBlock, MBOX_TYPE, SMTP_TYPE, DROP_TYPE, \
      parseMessageAndHeaders
+from mixminion.ServerInfo import displayServer
 
 #----------------------------------------------------------------------
 # Global variable; holds an instance of Common.Lockfile used to prevent
-# concurrent access to the directory cache, message queue, or SURB log.
+# concurrent access to the directory cache, packet queue, or SURB log.
 _CLIENT_LOCKFILE = None
 
 def clientLock():
@@ -68,7 +69,7 @@
             passwordManager = mixminion.ClientUtils.CLIPasswordManager()
         createPrivateDir(keyDir)
         fn = os.path.join(keyDir, "keyring")
-        self.keyring = mixminion.ClientUtils.LazyEncryptedPickled(
+        self.keyring = mixminion.ClientUtils.LazyEncryptedStore(
             fn, passwordManager, pwdName="ClientKeyring",
             queryPrompt="Enter password for keyring:",
             newPrompt="Entrer new keyring password:",
@@ -145,7 +146,8 @@
 #FileParanoia: yes
 
 [DirectoryServers]
-# Not yet implemented
+DirectoryTimeout: 1 minute
+# Other options not yet implemented
 
 [User]
 ## By default, mixminion puts your files in ~/.mixminion.  You can override
@@ -167,8 +169,7 @@
 #SURBPath: ?,?,?,FavoriteExit
 
 [Network]
-ConnectionTimeout: 20 seconds
-
+ConnectionTimeout: 60 seconds
 """)
 
 class MixminionClient:
@@ -197,16 +198,28 @@
         self.prng = mixminion.Crypto.getCommonPRNG()
         self.queue = mixminion.ClientUtils.ClientQueue(os.path.join(userdir, "queue"))
 
-    def _sortPackets(self, packets):
-        """[(packet,firstHop),...] -> [ (routing, [packet,...]), ...]"""
-        r = {}
+    def _sortPackets(self, packets, shuffle=1):
+        """Helper function.  Takes a list of tuples of (packet, routingInfo),
+           groups packets with the same routingInfos, and returns a list of
+           tuples of (routingInfo, [packet list]).
+
+           If 'shuffle' is true, then the packets within each list, and the
+           tuples themselves, are returned in a scrambled order.
+        """
+        d = {}
         for packet, firstHop in packets:
             ri = firstHop.getRoutingInfo()
-            r.setdefault(ri,[]).append(packet)
-        return r.items()
+            d.setdefault(ri,[]).append(packet)
+        result = d.items()
+        if shuffle:
+            self.prng.shuffle(result)
+            for _, pktList in result:
+                self.prng.shuffle(pktList)
+        return result
 
     def sendForwardMessage(self, directory, address, pathSpec, message,
-                           startAt, endAt, forceQueue=0, forceNoQueue=0):
+                           startAt, endAt, forceQueue=0, forceNoQueue=0,
+                           forceNoServerSideFragments=0):
         """Generate and send a forward message.
             address -- the results of a parseAddress call
             payload -- the contents of the message to send
@@ -215,17 +228,21 @@
             forceQueue -- if true, do not try to send the message; simply
                queue it and exit.
             forceNoQueue -- if true, do not queue the message even if delivery
-               fails."""
+               fails.
+
+            DOCDOC forceNoServerSideFragments
+        """
         assert not (forceQueue and forceNoQueue)
 
         allPackets = self.generateForwardPackets(
-            directory, address, pathSpec, message, startAt, endAt)
+            directory, address, pathSpec, message, forceNoServerSideFragments,
+            startAt, endAt)
 
         for routing, packets in self._sortPackets(allPackets):
             if forceQueue:
-                self.queueMessages(packets, routing)
+                self.queuePackets(packets, routing)
             else:
-                self.sendMessages(packets, routing, noQueue=forceNoQueue)
+                self.sendPackets(packets, routing, noQueue=forceNoQueue)
 
     def sendReplyMessage(self, directory, address, pathSpec, surbList, message,
                          startAt, endAt, forceQueue=0,
@@ -248,9 +265,9 @@
 
         for routing, packets in self._sortPackets(allPackets):
             if forceQueue:
-                self.queueMessages(packets, routing)
+                self.queuePackets(packets, routing)
             else:
-                self.sendMessages(packets, routing, noQueue=forceNoQueue)
+                self.sendPackets(packets, routing, noQueue=forceNoQueue)
 
     def generateReplyBlock(self, address, servers, name="", expiryTime=0):
         """Generate an return a new ReplyBlock object.
@@ -269,26 +286,27 @@
         return block
 
     def generateForwardPackets(self, directory, address, pathSpec, message,
-                               startAt, endAt):
-        """Generate a forward message, but do not send it.  Returns a
-           list of tuples of (the packet body, a ServerInfo for the
-           first hop.)
-
+                               noSSFragments, startAt, endAt):
+        """Generate packets for a forward message, but do not send
+           them.  Return a list of tuples of (the packet body, a
+           ServerInfo for the first hop.)
+           
            DOCDOC
             """
-
-        #XXXX006 handle user-side fragmentation.
-
         #XXXX006 we need to factor this long-message logic out to the
         #XXXX006 common code.  For now, this is a temporary measure.
-        fragmentedMessagePrefix = address.getFragmentedMessagePrefix()
+
+        if noSSFragments:
+            fragmentedMessagePrefix = ""
+        else:
+            fragmentedMessagePrefix = address.getFragmentedMessagePrefix()
         LOG.info("Generating payload(s)...")
         r = []
         if address.hasPayload():
             payloads = mixminion.BuildMessage.encodeMessage(message, 0,
                                 fragmentedMessagePrefix)
             if len(payloads) > 1:
-                address.setFragmented(1,len(payloads))
+                address.setFragmented(not noSSFragmented, len(payloads))
             else:
                 address.setFragmented(0,1)
         else:
@@ -302,17 +320,17 @@
         for p, (path1,path2) in zip(payloads, directory.generatePaths(
             len(payloads), pathSpec, address, startAt, endAt)):
 
-            msg = mixminion.BuildMessage.buildForwardPacket(
+            pkt = mixminion.BuildMessage.buildForwardPacket(
                 p, routingType, routingInfo, path1, path2,
                 self.prng)
-            r.append( (msg, path1[0]) )
+            r.append( (pkt, path1[0]) )
 
         return r
 
     def generateReplyPackets(self, directory, address, pathSpec, message,
                              surbList, startAt, endAt):
         """Generate a reply message, but do not send it.  Returns
-           a tuple of (the message body, a ServerInfo for the first hop.)
+           a tuple of (packet body, ServerInfo for the first hop.)
 
             address -- the results of a parseAddress call
             payload -- the contents of the message to send  (None for DROP
@@ -321,7 +339,7 @@
             surbList -- a list of SURBs to consider for the second leg of
                the path.  We use the first one that is neither expired nor
                used, and mark it used.
-               DOCDOC
+               DOCDOC: generates multiple packets
             """
         #XXXX write unit tests
         assert address.isReply
@@ -342,11 +360,11 @@
                                           startAt,endAt)):
                 assert path1 and not path2
                 LOG.info("Generating packet...")
-                msg = mixminion.BuildMessage.buildReplyPacket(
+                pkt = mixminion.BuildMessage.buildReplyPacket(
                     payload, path1, surb, self.prng)
                 
                 surbLog.markSURBUsed(surb)
-                result.append( (msg, path1[0]) )
+                result.append( (pkt, path1[0]) )
             
         finally:
             surbLog.close() #implies unlock
@@ -376,18 +394,18 @@
         except MixProtocolError, e:
             return 0, "Couldn't connect to server: %s" % e
 
-    def sendMessages(self, msgList, routingInfo, noQueue=0, lazyQueue=0,
+    def sendPackets(self, pktList, routingInfo, noQueue=0, lazyQueue=0,
                      warnIfLost=1):
         """Given a list of packets and an IPV4Info object, sends the
            packets to the server via MMTP.
 
-           If noQueue is true, do not queue the message even on failure.
-           If lazyQueue is true, only queue the message on failure.
-           Otherwise, insert the message in the queue, and remove it on
+           If noQueue is true, do not queue the packets even on failure.
+           If lazyQueue is true, only queue the packets on failure.
+           Otherwise, insert the packets in the queue, and remove them on
            success.
 
            If warnIfLost is true, log a warning if we fail to deliver
-           the message, and we don't queue it.
+           the packets, and we don't queue them.
 
            DOCDOC never raises
            """
@@ -401,23 +419,21 @@
         if noQueue or lazyQueue:
             handles = []
         else:
-            handles = self.queueMessages(msgList, routingInfo)
+            handles = self.queuePackets(pktList, routingInfo)
 
-        if len(msgList) > 1:
+        if len(pktList) > 1:
             mword = "packets"
         else:
             mword = "packet"
 
         try:
-            success = 0
             try:
                 # May raise TimeoutError
                 LOG.info("Connecting...")
-                mixminion.MMTPClient.sendMessages(routingInfo,
-                                                  msgList,
-                                                  timeout)
+                mixminion.MMTPClient.sendPackets(routingInfo,
+                                                 pktList,
+                                                 timeout)
                 LOG.info("... %s sent", mword)
-                success = 1
             except:
                 e = sys.exc_info()
                 if noQueue and warnIfLost:
@@ -425,30 +441,31 @@
                 elif lazyQueue:
                     LOG.info("Error while delivering %s; %s queued",
                              mword,mword)
-                    self.queueMessages(msgList, routingInfo)
+                    self.queuePackets(pktList, routingInfo)
                 else:
                     LOG.info("Error while delivering %s; leaving in queue",
                              mword)
                 LOG.info("Error was: %s",e[1])
-                return
-            try:
-                clientLock()
-                for h in handles:
-                    if self.queue.packetExists(h):
-                        self.queue.removePacket(h)
-                if handles:
-                    self.queue.cleanQueue()
-            finally:
-                clientUnlock()
+            else:
+                try:
+                    clientLock()
+                    for h in handles:
+                        if self.queue.packetExists(h):
+                            self.queue.removePacket(h)
+                    if handles:
+                        self.queue.cleanQueue()
+                finally:
+                    clientUnlock()
         except MixProtocolError, e:
             raise UIError(str(e))
 
-    def flushQueue(self, maxMessages=None):
-        """Try to send end all messages in the queue to their destinations.
+    def flushQueue(self, maxPackets=None):
+        """Try to send all packets in the queue to their destinations.
+           DOCDOC maxPackets
         """
         #XXXX write unit tests
 
-        class MessageProxy:
+        class PacketProxy:
             def __init__(self,h,queue):
                 self.h = h
                 self.queue = queue
@@ -457,48 +474,36 @@
             def __cmp__(self,other):
                 return cmp(id(self),id(other))
 
-        LOG.info("Flushing message queue")
+        LOG.info("Flushing packet queue")
         clientLock()
         try:
             handles = self.queue.getHandles()
-            LOG.info("Found %s pending messages", len(handles))
-            if maxMessages is not None:
+            LOG.info("Found %s pending packets", len(handles))
+            if maxPackets is not None:
                 handles = mixminion.Crypto.getCommonPRNG().shuffle(handles,
-                                                               maxMessages)
+                                                               maxPackets)
             LOG.info("Flushing %s", len(handles))
-            messagesByServer = {}
+            packets = []
             for h in handles:
                 try:
                     routing = self.queue.getRouting(h)
                 except mixminion.Filestore.CorruptedFile: 
                     continue
-                message = MessageProxy(h,self.queue)
-                messagesByServer.setdefault(routing, []).append((message, h))
+                packet = PacketProxy(h,self.queue)
+                packets.append((packet,routing))
         finally:
             clientUnlock()
 
         sentSome = 0; sentAll = 1
-        for routing in messagesByServer.keys():
-            LOG.info("Sending %s messages to %s:%s...",
-                     len(messagesByServer[routing]), routing.ip, routing.port)
-            msgs = [ m for m, _ in messagesByServer[routing] ]
-            handles = [ h for _, h in messagesByServer[routing] ]
+        for routing, packets in self._sortPackets(packets):
+            LOG.info("Sending %s packets to %s...",
+                     len(packets), displayServer(routing))
             try:
-                self.sendMessages(msgs, routing, noQueue=1, warnIfLost=0)
-##                 #XXXX006 is this part needed?
-##                 try:
-##                     clientLock()
-##                     for h in handles:
-##                         if self.queue.packetExists(h):
-##                             self.queue.removePacket(h)
-##                     if handles:
-##                         self.queue.cleanQueue()
-##                 finally:
-##                     clientUnlock()
+                self.sendPackets(packets, routing, noQueue=1, warnIfLost=0)
                 sentSome = 1
             except MixError, e:
-                LOG.error("Can't deliver messages to %s:%s: %s; leaving messages in queue",
-                          routing.ip, routing.port, str(e))
+                LOG.error("Can't deliver packets to %s: %s; leaving in queue",
+                          displayServer(routing), str(e))
                 sentAll = 0
 
         if sentAll:
@@ -506,10 +511,10 @@
         elif sentSome:
             LOG.info("Queue partially flushed")
         else:
-            LOG.info("No messages delivered")
+            LOG.info("No packets delivered")
 
     def cleanQueue(self, maxAge, now=None):
-        """Remove all messages older than maxAge seconds from the
+        """Remove all packets older than maxAge seconds from the
            client queue."""
         try:
             clientLock()
@@ -517,28 +522,30 @@
         finally:
             clientUnlock()
 
-    def queueMessages(self, msgList, routing):
-        """Insert all the messages in msgList into the queue, to be sent
-           to the server identified by the IPV4Info object 'routing'.
+    def queuePackets(self, pktList, routing):
+        """Insert all the packets in pktList into the queue, to be sent
+           to the server identified by the IPV4Info or MMTPHostInfo object
+           'routing'.
         """
         #XXXX write unit tests
-        LOG.trace("Queueing messages")
+        LOG.trace("Queueing packets")
         handles = []
         try:
             clientLock()
-            for msg in msgList:
-                h = self.queue.queuePacket(str(msg), routing)
+            for pkt in pktList:
+                h = self.queue.queuePacket(str(pkt), routing)
                 handles.append(h)
         finally:
             clientUnlock()
-        if len(msgList) > 1:
-            LOG.info("Messages queued")
+        if len(pktList) > 1:
+            LOG.info("Pacekts queued")
         else:
-            LOG.info("Message queued")
+            LOG.info("Packet queued")
         return handles
 
+    #XXXX006 rename to decodePacket ?
     def decodeMessage(self, s, force=0, isatty=0):
-        """Given a string 's' containing one or more text-encoed messages,
+        """Given a string 's' containing one or more text-encoded messages,
            return a list containing the decoded messages.
 
            Raise ParseError on malformatted messages.  Unless 'force' is
@@ -811,13 +818,16 @@
             assert self.wantConfig
             LOG.debug("Configuring server list")
             self.directory = mixminion.ClientDirectory.ClientDirectory(userdir)
+            self.directory._installAsKeyIDResolver()
 
         if self.wantDownload:
             assert self.wantClientDirectory
+            timeout = int(self.config['DirectoryServers']['DirectoryTimeout'])
             if self.download != 0:
                 try:
                     clientLock()
-                    self.directory.updateDirectory(forceDownload=self.download)
+                    self.directory.updateDirectory(forceDownload=self.download,
+                                                   timeout=timeout)
                 finally:
                     clientUnlock()
 
@@ -991,6 +1001,7 @@
 
     inFile = None
     h_subject = h_from = h_irt = h_references = None
+    no_ss_fragment = 0
     for opt,val in options:
         if opt in ('-i', '--input'):
             inFile = val
@@ -1002,6 +1013,8 @@
             h_irt = val
         elif opt == '--references':
             h_references = val
+        elif opt == '?????????':
+            no_ss_fragment = 1
 
     if args:
         sendUsageAndExit(cmd,"Unexpected arguments")
@@ -1087,9 +1100,8 @@
     else:
         client.sendForwardMessage(
             parser.directory, parser.exitAddress, parser.pathSpec,
-            message, parser.startAt, parser.endAt, forceQueue, forceNoQueue)
-            
-            
+            message, parser.startAt, parser.endAt, forceQueue, forceNoQueue,
+            no_ss_fragment)
 
 _PING_USAGE = """\
 Usage: mixminion ping [options] serverName
@@ -1202,11 +1214,11 @@
 
 def listServers(cmd, args):
     """[Entry point] Print info about """
-    options, args = getopt.getopt(args, "hf:D:vF:c:TVs:",
+    options, args = getopt.getopt(args, "hf:D:vF:TVs:cC",
                                   ['help', 'config=', "download-directory=",
-                                   'verbose', 'feature=', 'cascade=',
+                                   'verbose', 'feature=', 
                                    'with-time', "no-collapse", "valid",
-                                   "separator="])
+                                   "separator=", "cascade","cascade-features"])
     try:
         parser = CLIArgumentParser(options, wantConfig=1,
                                    wantClientDirectory=1,
@@ -1223,13 +1235,6 @@
     for opt,val in options:
         if opt in ('-F', '--feature'):
             features.extend(val.split(","))
-        elif opt in ('-c', '--cascade'):
-            try:
-                cascade = int(val)
-            except ValueError:
-                raise UIError("%s requires an integer"%opt)
-            if not (0 <= cascade <= 2):
-                raise UIError("Cascade level must be between 0 and 2")
         elif opt == ('-T'):
             showTime += 1
         elif opt == ('--with-time'):
@@ -1240,6 +1245,10 @@
             validOnly = 1
         elif opt in ('-s', '--separator'):
             separator = val
+        elif opt in ('-c', '--cascade'):
+            cascade = 1
+        elif opt in ('-C', '--cascade-features'):
+            cascade = 2
 
     if not features:
         if validOnly:
@@ -1310,9 +1319,11 @@
 
     parser.init()
     directory = parser.directory
+    config = parser.config
+    timeout = int(config['DirectoryServers']['DirectoryTimeout'])
     try:
         clientLock()
-        directory.updateDirectory(forceDownload=1)
+        directory.updateDirectory(forceDownload=1, timeout=timeout)
     finally:
         clientUnlock()
     print "Directory updated"
@@ -1556,10 +1567,10 @@
   -v, --verbose              Display extra debugging messages.
   -f <file>, --config=<file> Use a configuration file other than ~.mixminionrc
                                (You can also use MIXMINIONRC=FILE)
-  -n <n>, --count=<n>        Send no more than <n> messages from the queue.
+  -n <n>, --count=<n>        Send no more than <n> packets from the queue.
 
 EXAMPLES:
-  Try to send all currently queued messages.
+  Try to send all currently queued packets.
       %(cmd)s
 """.strip()
 
@@ -1593,10 +1604,10 @@
   -v, --verbose              Display extra debugging messages.
   -f <file>, --config=<file> Use a configuration file other than ~.mixminionrc
                                (You can also use MIXMINIONRC=FILE)
-  -d <n>, --days=<n>         Remove all messages older than <n> days old.
+  -d <n>, --days=<n>         Remove all packets older than <n> days old.
 
 EXAMPLES:
-  Remove all pending messages older than one week.
+  Remove all pending packets older than one week.
       %(cmd)s -d 7
 """.strip()
 
@@ -1638,11 +1649,12 @@
 """.strip()
 
 def listQueue(cmd, args):
-    options, args = getopt.getopt(args, "hvf:",
-                                  ["help", "verbose", "config=", ])
+    options, args = getopt.getopt(args, "hvf:D:",
+                                  ["help", "verbose", "config=",
+                                   'download-directory=',])
     try:
         parser = CLIArgumentParser(options, wantConfig=1, wantLog=1,
-                                   wantClient=1)
+                                   wantClient=1, wantClientDirectory=1)
     except UsageError, e:
         e.dump()
         print _LIST_QUEUE_USAGE % { 'cmd' : cmd }

Index: ClientUtils.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/ClientUtils.py,v
retrieving revision 1.8
retrieving revision 1.9
diff -u -d -r1.8 -r1.9
--- ClientUtils.py	8 Nov 2003 05:35:57 -0000	1.8
+++ ClientUtils.py	19 Nov 2003 09:48:09 -0000	1.9
@@ -22,6 +22,7 @@
 
 from mixminion.Common import LOG, MixError, UIError, createPrivateDir, \
      floorDiv, previousMidnight, readFile, writeFile
+from mixminion.ServerInfo import displayServer
 
 #----------------------------------------------------------------------
 class BadPassword(MixError):
@@ -193,12 +194,12 @@
     data = cPickle.dumps(obj, 1)
     writeEncryptedFile(fname, password, magic, data)
 
-class LazyEncryptedPickled:
-    """Wrapper for a file containing an encrypted pickled object, to
+class LazyEncryptedStore:
+    """Wrapper for a file containing an encrypted object, to
        perform password querying and loading on demand."""
     def __init__(self, fname, pwdManager, pwdName, queryPrompt, newPrompt,
                  magic, initFn):
-        """Create a new LazyEncryptedPickled
+        """Create a new LazyEncryptedStore
               fname -- The name of the file to hold the encrypted object.
               pwdManager -- A PasswordManager instance.
               pwdName, queryPrompt, newPrompt -- Arguments used when getting
@@ -281,6 +282,10 @@
         assert self.loaded and self.password is not None
         writeEncryptedPickled(self.fname, self.password, self.magic,
                               self.object)
+    def _encode(self,obj):
+        return cPickle.dumps(obj,1)
+    def _decode(self,obj):
+        return cPickle.loads(obj,1)
         
 # ----------------------------------------------------------------------
 
@@ -501,8 +506,8 @@
             days = floorDiv(now - oldest, 24*60*60)
             if days < 1:
                 days = "<1"
-            print "%2d packets for server at %s:%s (oldest is %s days old)"%(
-                count, s.ip, s.port, days)
+            print "%2d packets for %s (oldest is %s days old)"%(
+                count, displayServer(s), days)
 
     def cleanQueue(self, maxAge=None, now=None):
         """Remove all packets older than maxAge seconds from this queue."""

Index: Config.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Config.py,v
retrieving revision 1.65
retrieving revision 1.66
diff -u -d -r1.65 -r1.66
--- Config.py	10 Nov 2003 04:12:20 -0000	1.65
+++ Config.py	19 Nov 2003 09:48:09 -0000	1.66
@@ -55,8 +55,6 @@
 import binascii
 import os
 import re
-import socket # for inet_aton and error
-import string # for atoi
 try:
     import pwd
 except ImportError:
@@ -547,16 +545,16 @@
 
     return sections
 
-def _formatEntry(key,val,w=79,ind=4):
+def _formatEntry(key,val,w=79,ind=4,strict=0):
     """Helper function.  Given a key/value pair, returns a NL-terminated
        entry for inclusion in a configuration file, such that no line is
        avoidably longer than 'w' characters, and with continuation lines
        indented by 'ind' spaces.
     """
-    ind_s = " "*(ind-1)
-    if len(str(val))+len(key)+2 <= 79:
+    if strict or len(str(val))+len(key)+2 <= 79:
         return "%s: %s\n" % (key,val)
 
+    ind_s = " "*(ind-1)
     lines = [  ]
     linecontents = [ "%s:" % key ]
     linelength = len(linecontents[0])
@@ -903,8 +901,12 @@
         for s in self._sectionNames:
             lines.append("[%s]\n"%s)
             for k,v in self._sectionEntries[s]:
-                lines.append(_formatEntry(k,v))
-            lines.append("\n")
+                tp = self._syntax[s][k][1]
+                if tp:
+                    v = self.CODING_FNS[tp][1](v)
+                lines.append(_formatEntry(k,v,strict=self._restrictFormat))
+            if not self._restrictFormat:
+                lines.append("\n")
 
         return "".join(lines)
 
@@ -922,7 +924,8 @@
         'DirectoryServers' :
                    { '__SECTION__' : ('REQUIRE', None, None),
                      'ServerURL' : ('ALLOW*', None, None),
-                     'MaxSkew' : ('ALLOW', "interval", "10 minutes") },
+                     'MaxSkew' : ('ALLOW', "interval", "10 minutes"),
+                     'DirectoryTimeout' : ('ALLOW', "interval", "1 minute") },
         'User' : { 'UserDir' : ('ALLOW', "filename", "~/.mixminion" ) },
         'Security' : { 'PathLength' : ('ALLOW', "int", "8"),
                        'SURBAddress' : ('ALLOW', None, None),
@@ -932,7 +935,7 @@
                        'ReplyPath' : ('ALLOW', None, "*4"),
                        'SURBPath' : ('ALLOW', None, "*4"),
                        },
-        'Network' : { 'ConnectionTimeout' : ('ALLOW', "interval", None) }
+        'Network' : { 'ConnectionTimeout' : ('ALLOW', "interval", "2 minutes")}
         }
     def __init__(self, fname=None, string=None):
         _ConfigFile.__init__(self, fname, string)

Index: MMTPClient.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/MMTPClient.py,v
retrieving revision 1.41
retrieving revision 1.42
diff -u -d -r1.41 -r1.42
--- MMTPClient.py	7 Nov 2003 07:03:28 -0000	1.41
+++ MMTPClient.py	19 Nov 2003 09:48:09 -0000	1.42
@@ -4,20 +4,22 @@
 
    This module contains a single, synchronous implementation of the client
    side of the Mixminion Transfer protocol.  You can use this client to
-   upload messages to any conforming Mixminion server.
+   upload packets to any conforming Mixminion server.
 
    (We don't use this module for transferring packets between servers;
    in fact, MMTPServer makes it redundant.  We only keep this module
    around [A] so that clients have an easy (blocking) interface to
-   introduce messages into the system, and [B] so that we've got an
+   introduce packets into the system, and [B] so that we've got an
    easy-to-verify reference implementation of the protocol.)
    """
 
-__all__ = [ "BlockingClientConnection", "sendMessages" ]
+__all__ = [ "BlockingClientConnection", "sendPackets" ]
 
 import socket
 import mixminion._minionlib as _ml
 import mixminion.NetUtils
+import mixminion.Packet
+import mixminion.ServerInfo
 from mixminion.Crypto import sha1, getCommonPRNG
 from mixminion.Common import MixProtocolError, MixProtocolReject, \
      MixProtocolBadAuth, LOG, MixError, formatBase64, TimeoutError
@@ -40,7 +42,8 @@
     # PROTOCOL_VERSIONS: (static) a list of protocol versions we allow,
     #     in decreasing order of preference.
     PROTOCOL_VERSIONS = ['0.3']
-    def __init__(self, targetFamily, targetAddr, targetPort, targetKeyID):
+    def __init__(self, targetFamily, targetAddr, targetPort, targetKeyID,
+                 serverName=None):
         """Open a new connection."""
         self.targetFamily = targetFamily
         self.targetAddr = targetAddr
@@ -53,6 +56,12 @@
         self.tls = None
         self.sock = None
         self.certCache = PeerCertificateCache()
+        if serverName:
+            self.serverName = serverName
+            #DOCDOC
+        else:
+            self.serverName = mixminion.ServerInfo.displayServer(
+                mixminion.Packet.IPV4Info(targetAddr, targetPort, targetKeyID))
 
     def connect(self, connectTimeout=None):
         """Connect to the server, perform the TLS handshake, check the server
@@ -72,10 +81,15 @@
         """Helper method: given an exception (err) and an action string (e.g.,
            'connecting'), raises an appropriate MixProtocolError.
         """
+        errstr = str(err)
         if isinstance(err, socket.error):
             tp = "Socket"
+            if mixminion.NetUtils.exceptionIsTimeout(err):
+                tp = "Timeout"
         elif isinstance(err, _ml.TLSError):
             tp = "TLS"
+            if errstr == 'wrong version number':
+                errstr = 'wrong version number (or failed handshake)'
         elif isinstance(err, _ml.TLSClosed):
             tp = "TLSClosed"
         elif isinstance(err, _ml.TLSWantRead):
@@ -84,8 +98,8 @@
             tp = "Unexpected TLSWantWrite"
         else:
             tp = str(type(err))
-        e = MixProtocolError("%s error while %s to %s:%s: %s" %(
-                             tp, action, self.targetAddr, self.targetPort, err))
+        e = MixProtocolError("%s error while %s to %s: %s" %(
+                             tp, action, self.serverName, errstr))
         e.base = err
         raise e
 
@@ -94,20 +108,19 @@
         # Connect to the server
         self.sock = socket.socket(self.targetFamily, socket.SOCK_STREAM)
         self.sock.setblocking(1)
-        LOG.debug("Connecting to %s:%s", self.targetAddr, self.targetPort)
+        LOG.debug("Connecting to %s", self.serverName)
 
         # Do the TLS handshaking
         mixminion.NetUtils.connectWithTimeout(
             self.sock, (self.targetAddr, self.targetPort), connectTimeout)
         
-        LOG.debug("Handshaking with %s:%s",self.targetAddr, self.targetPort)
+        LOG.debug("Handshaking with %s:", self.serverName)
         self.tls = self.context.sock(self.sock.fileno())
         self.tls.connect()
         LOG.debug("Connected.")
         # Check the public key of the server to prevent man-in-the-middle
         # attacks.
-        self.certCache.check(self.tls, self.targetKeyID,
-                             "%s:%s"%(self.targetAddr,self.targetPort))
+        self.certCache.check(self.tls, self.targetKeyID, self.serverName)
 
         ####
         # Protocol negotiation
@@ -131,7 +144,8 @@
                 break
         if not self.protocol:
             raise MixProtocolError("Protocol negotiation failed")
-        LOG.debug("MMTP protocol negotiated: version %s", self.protocol)
+        LOG.debug("MMTP protocol negotiated with %s: version %s",
+                  self.serverName, self.protocol)
 
     def renegotiate(self):
         """Re-do the TLS handshake to renegotiate a new connection key."""
@@ -157,7 +171,7 @@
         """Helper method: implements sendPacket and sendJunkPacket.
               packet -- a 32K string to send
               control -- a 6-character string ending with CRLF to
-                  indicate the type of message we're sending.
+                  indicate the type of packet we're sending.
               serverControl -- a 10-character string ending with CRLF that
                   we expect to receive if we've sent correctly.
               hashExtra -- a string to append to the packet when computing
@@ -188,8 +202,7 @@
 
     def shutdown(self):
         """Close this connection."""
-        LOG.debug("Shutting down connection to %s:%s",
-                  self.targetAddr, self.targetPort)
+        LOG.debug("Shutting down connection to %s", self.serverName)
         try:
             if self.tls is not None:
                 self.tls.shutdown()
@@ -200,8 +213,8 @@
             self._raise(e, "closing connection")
         LOG.debug("Connection closed")
 
-def sendMessages(routing, packetList, connectTimeout=None, callback=None):
-    """Sends a list of messages to a server.  Raise MixProtocolError on
+def sendPackets(routing, packetList, connectTimeout=None, callback=None):
+    """Sends a list of packets to a server.  Raise MixProtocolError on
        failure.
 
        routing -- an instance of mixminion.Packet.IPV4Info or
@@ -224,7 +237,9 @@
         elif p == 'RENEGOTIATE':
             packets.append(("RENEGOTIATE", None))
         else:
-            packets.append(("MSG", p))
+            packets.append(("PKT", p))
+
+    serverName = mixminion.ServerInfo.displayServer(routing)
 
     if isinstance(routing, IPV4Info):
         family, addr = socket.AF_INET, routing.ip
@@ -236,7 +251,8 @@
             raise MixProtocolError("Couldn't resolve hostname %s: %s",
                                    routing.hostname, addr)
 
-    con = BlockingClientConnection(family,addr,routing.port,routing.keyinfo)
+    con = BlockingClientConnection(family,addr,routing.port,routing.keyinfo,
+                                   serverName=serverName)
     try:
         con.connect(connectTimeout=connectTimeout)
         for idx in xrange(len(packets)):
@@ -257,7 +273,7 @@
 
        May raise MixProtocolBadAuth, or other MixProtocolError if server
        isn't up."""
-    sendMessages(routing, ["JUNK"], connectTimeout=connectTimeout)
+    sendPackets(routing, ["JUNK"], connectTimeout=connectTimeout)
 
 class PeerCertificateCache:
     """A PeerCertificateCache validates certificate chains from MMTP servers,
@@ -267,17 +283,19 @@
     def __init__(self):
         self.cache = {}
 
-
-    def check(self, tls, targetKeyID, address):
+    #XXXX006 use displayName to 
+    def check(self, tls, targetKeyID, serverName):
         """Check whether the certificate chain on the TLS connection 'tls'
            is valid, current, and matches the keyID 'targetKeyID'.  If so,
            return.  If not, raise MixProtocolBadAuth.
         """
+        
         # First, make sure the certificate is neither premature nor expired.
         try:
             tls.check_cert_alive()
         except _ml.TLSError, e:
-            raise MixProtocolBadAuth("Invalid certificate: %s" % str(e))
+            raise MixProtocolBadAuth("Invalid certificate from %s: %s" % (
+                serverName, str(e)))
 
         # If we don't care whom we're talking to, we don't need to check
         # them out.
@@ -293,15 +311,13 @@
         # compatibility as well.
         if targetKeyID == hashed_peer_pk:
             raise MixProtocolBadAuth(
-               "Pre-0.0.4 (non-rotatable) certificate from server at %s" %
-               address)
+               "Pre-0.0.4 (non-rotatable) certificate from %s" % serverName)
 
         try:
             if targetKeyID == self.cache[hashed_peer_pk]:
                 # We recognize the key, and have already seen it to be
                 # signed by the target identity.
-                LOG.trace("Got a cached certificate from server at %s",
-                          address)
+                LOG.trace("Got a cached certificate from %s", serverName)
                 return # All is well.
             else:
                 # We recognize the key, but some other identity signed it.
@@ -315,13 +331,12 @@
         try:
             identity = tls.verify_cert_and_get_identity_pk()
         except _ml.TLSError, e:
-            raise MixProtocolBadAuth("Invalid KeyID from server at %s: %s"
-                                   %(address, e))
+            raise MixProtocolBadAuth("Invalid KeyID (allegedly) from %s: %s"
+                                   %serverName)
 
         # Okay, remember who has signed this certificate.
         hashed_identity = sha1(identity.encode_key(public=1))
-        LOG.trace("Remembering valid certificate for server at %s",
-                  address)
+        LOG.trace("Remembering valid certificate for %s", serverName)
         self.cache[hashed_peer_pk] = hashed_identity
 
         # Note: we don't need to worry about two identities signing the
@@ -332,4 +347,4 @@
 
         # Was the signer the right person?
         if hashed_identity != targetKeyID:
-            raise MixProtocolBadAuth("Invalid KeyID for server at %s" %address)
+            raise MixProtocolBadAuth("Invalid KeyID for %s" % serverName)

Index: Main.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Main.py,v
retrieving revision 1.60
retrieving revision 1.61
diff -u -d -r1.60 -r1.61
--- Main.py	7 Nov 2003 10:43:18 -0000	1.60
+++ Main.py	19 Nov 2003 09:48:09 -0000	1.61
@@ -253,6 +253,13 @@
 
     correctPath(args[0])
 
+##     if len(args) > 2 and args[1] == 'mixminiond':
+##         if _COMMANDS.has_key("server-"+args[2]):
+##             args[1:3] = "server-"+args[2]
+##         else:
+##             printUsage()
+##             sys.exit(1)
+
     # Check whether we have a recognized command.
     if len(args) == 1  or not _COMMANDS.has_key(args[1]):
         printUsage()

Index: NetUtils.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/NetUtils.py,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -d -r1.3 -r1.4
--- NetUtils.py	10 Nov 2003 04:12:20 -0000	1.3
+++ NetUtils.py	19 Nov 2003 09:48:09 -0000	1.4
@@ -106,7 +106,7 @@
                 return sock.connect(dest)
             except socket.error, e:
                 if e[0] in IN_PROGRESS_ERRNOS:
-                    raise TimeoutError()
+                    raise TimeoutError("Connection timed out")
                 else:
                     raise
         finally:
@@ -125,7 +125,7 @@
             except select.error, e:
                 raise
             if not wfds:
-                raise TimeoutError()
+                raise TimeoutError("Connection timed out")
 ##             try:
 ##                 sock.connect(dest)
 ##             except select.error, e:
@@ -287,13 +287,13 @@
         try:
             val = normalizeIP6(name)
             return (AF_INET6, val, time.time())
-        except ValueError, e:
+        except ValueError:
             return None
     elif name and name[0].isdigit():
         try:
             val = normalizeIP4(name)
             return (AF_INET, val, time.time())
-        except ValueError, e:
+        except ValueError:
             return None
     else:
         return None

Index: Packet.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Packet.py,v
retrieving revision 1.64
retrieving revision 1.65
diff -u -d -r1.64 -r1.65
--- Packet.py	10 Nov 2003 04:12:20 -0000	1.64
+++ Packet.py	19 Nov 2003 09:48:09 -0000	1.65
@@ -3,7 +3,7 @@
 """mixminion.Packet
 
    Functions, classes, and constants to parse and unparse Mixminion
-   messages and related structures.
+   messages, packets, and related structures.
 
    For functions that handle client-side generation and decoding of
    packets, see BuildMessage.py.  For functions that handle
@@ -49,12 +49,12 @@
 # Major and minor number for the understood packet format.
 MAJOR_NO, MINOR_NO = 0,3
 
-# Length of a Mixminion message
-MESSAGE_LEN = 1 << 15
+# Length of a Mixminion packet
+PACKET_LEN = 1 << 15
 # Length of a header section
 HEADER_LEN  = 128 * 16
 # Length of a single payload
-PAYLOAD_LEN = MESSAGE_LEN - HEADER_LEN*2
+PAYLOAD_LEN = PACKET_LEN - HEADER_LEN*2
 
 # Bytes taken up by OAEP padding in RSA-encrypted data
 OAEP_OVERHEAD = 42
@@ -79,10 +79,10 @@
 #----------------------------------------------------------------------
 # Values for the 'Routing type' subheader field
 # Mixminion types
-DROP_TYPE          = 0x0000     # Drop the current message
-FWD_IPV4_TYPE      = 0x0001 # Forward the msg to an IPV4 addr via MMTP
+DROP_TYPE          = 0x0000 # Drop the packet
+FWD_IPV4_TYPE      = 0x0001 # Forward the packet to an IPV4 addr via MMTP
 SWAP_FWD_IPV4_TYPE = 0x0002 # SWAP, then FWD_IPV4
-FWD_HOST_TYPE      = 0x0003 # Forward the msg to a hostname, via MMTP.
+FWD_HOST_TYPE      = 0x0003 # Forward the pkt to a hostname, via MMTP.
 SWAP_FWD_HOST_TYPE = 0x0004 # SWAP, then FWD_HOST
 
 # Exit types
@@ -112,8 +112,8 @@
 def parsePacket(s):
     """Given a 32K string, returns a Packet object that breaks it into
        two headers and a payload."""
-    if len(s) != MESSAGE_LEN:
-        raise ParseError("Bad message length")
+    if len(s) != PACKET_LEN:
+        raise ParseError("Bad packet length")
 
     return Packet(s[:HEADER_LEN],
                    s[HEADER_LEN:HEADER_LEN*2],
@@ -124,13 +124,13 @@
 
        Fields: header1, header2, payload"""
     def __init__(self, header1, header2, payload):
-        """Create a new Message object from three strings."""
+        """Create a new Packet object from three strings."""
         self.header1 = header1
         self.header2 = header2
         self.payload = payload
 
     def pack(self):
-        """Return the 32K string value of this message."""
+        """Return the 32K string value of this packet."""
         return "".join([self.header1,self.header2,self.payload])
 
 def parseHeader(s):
@@ -532,15 +532,18 @@
         self.encryptionKey = key
 
     def format(self):
+        from mixminion.ServerInfo import displayServer
         hash = binascii.b2a_hex(sha1(self.pack()))
         expiry = formatTime(self.timestamp)
         if self.routingType == SWAP_FWD_IPV4_TYPE:
-            server = parseIPV4Info(self.routingInfo).format()
+            routing = parseIPV4Info(self.routingInfo)
+        elif self.routingType == SWAP_FWD_HOST_TYPE:
+            routing = parseMMTPHostInfo(self.routingInfo)
         else:
-            server = "????"
+            routing = None
         return """Reply block hash: %s
 Expires at: %s GMT
-First server is: %s""" % (hash, expiry, server)
+First server is: %s""" % (hash, expiry, displayServer(routing))
 
     def pack(self):
         """Returns the external representation of this reply block"""

Index: ServerInfo.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/ServerInfo.py,v
retrieving revision 1.61
retrieving revision 1.62
diff -u -d -r1.61 -r1.62
--- ServerInfo.py	10 Nov 2003 04:12:20 -0000	1.61
+++ ServerInfo.py	19 Nov 2003 09:48:09 -0000	1.62
@@ -8,10 +8,11 @@
    descriptors.
    """
 
-__all__ = [ 'ServerInfo', 'ServerDirectory' ]
+__all__ = [ 'ServerInfo', 'ServerDirectory', 'displayServer' ]
 
 import re
 import time
+import types
 
 import mixminion.Config
 import mixminion.Crypto
@@ -38,6 +39,48 @@
 PACKET_KEY_BYTES = 2048 >> 3
 # Length of MMTP key
 MMTP_KEY_BYTES = 1024 >> 3
+
+# ----------------------------------------------------------------------
+def displayServer(s):
+    """DOCDOC"""
+    #XXXX006 unit tests are needed
+    if isinstance(s, types.StringType):
+        return s
+    elif isinstance(s, ServerInfo):
+        if s.getHostname():
+            addr = "%s:%s" % (s.getHostname(), s.getPort())
+        else:
+            addr = "%s:%s" % (s.getIP(), s.getPort())
+        nickname = "'%s'" % s.getNickname()
+    elif isinstance(s, (mixminion.Packet.IPV4Info,
+                       mixminion.Packet.MMTPHostInfo)):
+        nickname = getNicknameByKeyID(s.keyinfo)
+        if nickname:
+            nickname = "'%s'" % nickname
+        else:
+            nickname = 'server'
+        if isinstance(s, mixminion.Packet.IPV4Info):
+            addr = "%s:%s" % (s.ip, s.port)
+        else:
+            addr = "%s:%s" % (s.hostname, s.port)
+    elif s is None:
+        return "unknown server"
+    else:
+        assert 0
+
+    return "%s at %s" % (nickname, addr)
+
+def getNicknameByKeyID(keyid):
+    """DOCDOC"""
+    if _keyIDToNicknameFn:
+        return _keyIDToNicknameFn(keyid)
+    else:
+        return None
+
+#DOCDOC
+_keyIDToNicknameFn = None
+
+# ----------------------------------------------------------------------
 
 # tmp alias to make this easier to spell.
 C = mixminion.Config

Index: test.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/test.py,v
retrieving revision 1.164
retrieving revision 1.165
diff -u -d -r1.164 -r1.165
--- test.py	10 Nov 2003 04:12:20 -0000	1.164
+++ test.py	19 Nov 2003 09:48:10 -0000	1.165
@@ -2030,7 +2030,7 @@
 
         message = consMsg(secrets1, secrets2, h1, h2, pld)
 
-        self.assertEquals(len(message), mixminion.Packet.MESSAGE_LEN)
+        self.assertEquals(len(message), mixminion.Packet.PACKET_LEN)
         msg = mixminion.Packet.parsePacket(message)
         head1, head2, payload = msg.header1, msg.header2, msg.payload
         self.assert_(h1 == head1)
@@ -2057,7 +2057,7 @@
         ######
         ### Reply case
         message = consMsg(secrets1, None, h1, h2, pld)
-        self.assertEquals(len(message), mixminion.Packet.MESSAGE_LEN)
+        self.assertEquals(len(message), mixminion.Packet.PACKET_LEN)
         msg = mixminion.Packet.parsePacket(message)
         head1, head2, payload = msg.header1, msg.header2, msg.payload
         self.assert_(h1 == head1)
@@ -2550,7 +2550,7 @@
                     routinginfo: sequence of expected routinginfo, excl tags
                     payload: beginning of expected final payload."""
         for sp, rt, ri in zip(sps,routingtypes,routinginfo):
-            res = sp.processMessage(m)
+            res = sp.processPacket(m)
             self.assert_(isinstance(res, DeliveryPacket) or
                          isinstance(res, RelayedPacket))
             if rt in (FWD_IPV4_TYPE, SWAP_FWD_IPV4_TYPE):
@@ -2753,23 +2753,23 @@
         m = bfm(zPayload, MBOX_TYPE, "hello\000bye",
                 [self.server2, server1X, self.server3],
                 [server1X, self.server2, self.server3])
-        self.failUnlessRaises(ParseError, self.sp2.processMessage, m)
+        self.failUnlessRaises(ParseError, self.sp2.processPacket, m)
 
         # Duplicate messages need to fail.
         m = bfm(zPayload, SMTP_TYPE, "nobody@invalid",
                 [self.server1, self.server2], [self.server3])
-        self.sp1.processMessage(m)
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m)
+        self.sp1.processPacket(m)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m)
 
         # Duplicate reply blocks need to fail
         reply,s,tag = brbi([self.server3], SMTP_TYPE, "fred@invalid")
         yPayload = BuildMessage.encodeMessage("Y",0)[0]
         m = brm(yPayload, [self.server2], reply)
         m2 = brm(yPayload, [self.server1], reply)
-        m = self.sp2.processMessage(m).getPacket()
-        self.sp3.processMessage(m)
-        m2 = self.sp1.processMessage(m2).getPacket()
-        self.failUnlessRaises(ContentError, self.sp3.processMessage, m2)
+        m = self.sp2.processPacket(m).getPacket()
+        self.sp3.processPacket(m)
+        m2 = self.sp1.processPacket(m2).getPacket()
+        self.failUnlessRaises(ContentError, self.sp3.processPacket, m2)
 
         # Even duplicate secrets need to go.
         prng = AESCounterPRNG(" "*16)
@@ -2778,29 +2778,29 @@
         reply2,s,t = brbi([self.server2], MBOX_TYPE, "foo",0,prng)
         m = brm(yPayload, [self.server3], reply1)
         m2 = brm(yPayload, [self.server3], reply2)
-        m = self.sp3.processMessage(m).getPacket()
-        self.sp1.processMessage(m)
-        m2 = self.sp3.processMessage(m2).getPacket()
-        self.failUnlessRaises(ContentError, self.sp2.processMessage, m2)
+        m = self.sp3.processPacket(m).getPacket()
+        self.sp1.processPacket(m)
+        m2 = self.sp3.processPacket(m2).getPacket()
+        self.failUnlessRaises(ContentError, self.sp2.processPacket, m2)
 
         # Drop gets dropped.
         m = bfm(zPayload, DROP_TYPE, "", [self.server2], [self.server2])
-        m = self.sp2.processMessage(m).getPacket()
-        res = self.sp2.processMessage(m)
+        m = self.sp2.processPacket(m).getPacket()
+        res = self.sp2.processPacket(m)
         self.assertEquals(res,None)
 
         # Wrong server.
         m = bfm(zPayload, DROP_TYPE, "", [self.server1], [self.server2])
-        self.failUnlessRaises(CryptoError, self.sp2.processMessage, m)
-        self.failUnlessRaises(CryptoError, self.sp2_3.processMessage, m)
+        self.failUnlessRaises(CryptoError, self.sp2.processPacket, m)
+        self.failUnlessRaises(CryptoError, self.sp2_3.processPacket, m)
 
         # Plain junk in header
         m_x = ("XY"*64)+m[128:]
-        self.failUnlessRaises(CryptoError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(CryptoError, self.sp1.processPacket, m_x)
 
         # Bad message length
         m_x = m+"Z"
-        self.failUnlessRaises(ParseError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ParseError, self.sp1.processPacket, m_x)
 
         # Bad internal type
         try:
@@ -2813,43 +2813,43 @@
             m_x = bfm(zPayload, 500, "", [self.server1], [self.server2])
         finally:
             SWAP_FWD_IPV4_TYPE = save
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m_x)
 
         # Subhead with bad length
         m_x = pk_encrypt("foo", self.pk1)+m[256:]
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m_x)
 
         # Subhead we can't parse.
         m_x = pk_encrypt("f"*(256-42), self.pk1)+m[256:]
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m_x)
 
         # Bad IPV4 info
         subh_real = pk_decrypt(m[:256], self.pk1)
         subh = parseSubheader(subh_real)
         subh.setRoutingInfo(subh.routinginfo + "X")
         m_x = pk_encrypt(subh.pack()+subh.underflow[:-1], self.pk1)+m[256:]
-        self.failUnlessRaises(ParseError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ParseError, self.sp1.processPacket, m_x)
 
         # Bad Major or Minor
         subh = parseSubheader(subh_real)
         subh.major = 255
         m_x = pk_encrypt(subh.pack()+subh.underflow, self.pk1)+m[256:]
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m_x)
 
         # Bad digest
         subh = parseSubheader(subh_real)
         subh.digest = " "*20
         m_x = pk_encrypt(subh.pack()+subh.underflow, self.pk1)+m[256:]
-        self.failUnlessRaises(ContentError, self.sp1.processMessage, m_x)
+        self.failUnlessRaises(ContentError, self.sp1.processPacket, m_x)
 
         # Corrupt payload
         m = bfm(zPayload, MBOX_TYPE, "Z", [self.server1, self.server2],
                 [self.server3])
         m_x = m[:-30] + " "*30
         assert len(m_x) == len(m)
-        m_x = self.sp1.processMessage(m_x).getPacket()
-        m_x = self.sp2.processMessage(m_x).getPacket()
-        self.failUnlessRaises(CryptoError, self.sp3.processMessage, m_x)
+        m_x = self.sp1.processPacket(m_x).getPacket()
+        m_x = self.sp2.processPacket(m_x).getPacket()
+        self.failUnlessRaises(CryptoError, self.sp3.processPacket, m_x)
 
 #----------------------------------------------------------------------
 # FILESTORE and QUEUE
@@ -3707,28 +3707,28 @@
         self.doTest(self._testRejected)
 
     def _testBlockingTransmission(self):
-        server, listener, messagesIn, keyid = _getMMTPServer()
+        server, listener, packetsIn, keyid = _getMMTPServer()
         self.listener = listener
         self.server = server
 
-        messages = ["helloxxx"*4096, "helloyyy"*4096]
+        packets = ["helloxxx"*4096, "helloyyy"*4096]
 
         # Send m1, then junk, then renegotiate, then m2.
         server.process(0.1)
         routing = IPV4Info("127.0.0.1", TEST_PORT, keyid)
         t = threading.Thread(None,
-                             mixminion.MMTPClient.sendMessages,
+                             mixminion.MMTPClient.sendPackets,
                              args=(routing,
-                             [messages[0],"JUNK","RENEGOTIATE",messages[1]]))
+                             [packets[0],"JUNK","RENEGOTIATE",packets[1]]))
         t.start()
-        while len(messagesIn) < 2:
+        while len(packetsIn) < 2:
             server.process(0.1)
         t.join()
 
         for _ in xrange(3):
             server.process(0.1)
 
-        self.failUnless(messagesIn == messages)
+        self.failUnless(packetsIn == packets)
         self.assertEquals(1, server.nJunkPackets)
 
         # Now, with bad keyid.
@@ -3736,8 +3736,8 @@
         t = threading.Thread(None,
                              self.failUnlessRaises,
                              args=(MixProtocolError,
-                                   mixminion.MMTPClient.sendMessages,
-                                   routing, messages))
+                                   mixminion.MMTPClient.sendPackets,
+                                   routing, packets))
         t.start()
         while t.isAlive():
             server.process(0.1)
@@ -3773,7 +3773,7 @@
         try:
             try:
                 routing = IPV4Info("127.0.0.1", TEST_PORT, "Z"*20)
-                mixminion.MMTPClient.sendMessages(routing, ["JUNK"],
+                mixminion.MMTPClient.sendPackets(routing, ["JUNK"],
                                                    connectTimeout=1)
                 timedout = 0
             except mixminion.MMTPClient.TimeoutError:
@@ -3787,14 +3787,14 @@
         self.assert_(timedout)
 
     def _testNonblockingTransmission(self):
-        server, listener, messagesIn, keyid = _getMMTPServer()
+        server, listener, packetsIn, keyid = _getMMTPServer()
         self.listener = listener
         self.server = server
 
         # Send m1, then junk, then renegotiate, then junk, then m2.
         tlscon = mixminion.server.MMTPServer.SimpleTLSConnection
-        messages = ["helloxxx"*4096, "helloyyy"*4096]
-        deliv = [FakeDeliverable(m) for m in messages]
+        packets = ["helloxxx"*4096, "helloyyy"*4096]
+        deliv = [FakeDeliverable(m) for m in packets]
         async = mixminion.server.MMTPServer.AsyncServer()
         clientcon = mixminion.server.MMTPServer.MMTPClientConnection(
            _getTLSContext(0), "127.0.0.1", TEST_PORT, keyid,
@@ -3811,7 +3811,7 @@
 
         c = None
         t.start()
-        while len(messagesIn) < 2:
+        while len(packetsIn) < 2:
             if c is None and len(server.readers) > 1:
                 c = [ c for c in server.readers.values() if
                       isinstance(c, tlscon) ]
@@ -3821,8 +3821,8 @@
         t.join()
         endTime = time.time()
 
-        self.assertEquals(len(messagesIn), len(messages))
-        self.failUnless(messagesIn == messages)
+        self.assertEquals(len(packetsIn), len(packets))
+        self.failUnless(packetsIn == packets)
         self.failUnless(c is not None)
         self.failUnless(len(c) == 1)
         self.failUnless(startTime <= c[0].lastActivity <= endTime)
@@ -3831,7 +3831,7 @@
         self.assert_(deliv[1]._succeeded)
 
         # Again, with bad keyid.
-        deliv = [FakeDeliverable(m) for m in messages]
+        deliv = [FakeDeliverable(p) for p in packets]
         clientcon = mixminion.server.MMTPServer.MMTPClientConnection(
             _getTLSContext(0), "127.0.0.1", TEST_PORT, "Z"*20,
             deliv[:], None)
@@ -3856,11 +3856,11 @@
         self.assert_(deliv[1]._failed)
 
     def _testTimeout(self):
-        server, listener, messagesIn, keyid = _getMMTPServer()
+        server, listener, packetsIn, keyid = _getMMTPServer()
         self.listener = listener
         self.server = server
 
-        # This function wraps MMTPClient.sendMessages, but catches exceptions.
+        # This function wraps MMTPClient.sendPackets but catches exceptions.
         # Since we're going to run this function in a thread, we pass the
         # exception back through a list argument.
         def sendSlowlyAndCaptureException(exlst, pausing, targetIP, targetPort,
@@ -3891,7 +3891,7 @@
         timedOut = 0 # flag: have we really timed out?
         try:
             suspendLog() # stop logging, but wait for the timeout message.
-            while len(messagesIn) < 2:
+            while len(packetsIn) < 2:
                 server.process(0.1)
                 # If the number of connections changes around the call
                 # to tryTimeout, the timeout has occurred.
@@ -3907,7 +3907,7 @@
         # Did we log the timeout?
         self.assert_(stringContains(logMessage, "timed out"))
         # Was the one message we expected in fact transmitted?
-        self.assertEquals([messagesIn[0]], ["helloxxx"*4096])
+        self.assertEquals([packetsIn[0]], ["helloxxx"*4096])
 
         # Now stop the transmitting thread.  It will notice that its
         # connection has been forcibly closed.
@@ -3924,18 +3924,18 @@
             server.process(0.1)
 
     def _testRejected(self):
-        server, listener, messagesIn, keyid = _getMMTPServer(reject=1)
+        server, listener, packetsIn, keyid = _getMMTPServer(reject=1)
         self.listener = listener
         self.server = server
 
-        messages = ["helloxxx"*4096, "helloyyy"*4096]
-        # Send 2 messages -- both should be rejected.
+        packets = ["helloxxx"*4096, "helloyyy"*4096]
+        # Send 2 packets -- both should be rejected.
         server.process(0.1)
         routing = IPV4Info("127.0.0.1", TEST_PORT, keyid)
         ok=[0];done=[0]
-        def _t(routing=routing, messages=messages,ok=ok,done=done):
+        def _t(routing=routing, packets=packets, ok=ok, done=done):
             try:
-                mixminion.MMTPClient.sendMessages(routing,messages)
+                mixminion.MMTPClient.sendPackets(routing,packets)
             except mixminion.Common.MixProtocolReject:
                 ok[0] = 1
             done[0] = 1
@@ -3949,11 +3949,11 @@
             server.process(0.1)
 
         self.failUnless(ok[0])
-        self.failUnless(len(messagesIn) == 0)
+        self.failUnless(len(packetsIn) == 0)
 
         # Send m1, then junk, then renegotiate, then junk, then m2.
-        messages = ["helloxxx"*4096, "helloyyy"*4096]
-        deliv = [FakeDeliverable(m) for m in messages]
+        packets = ["helloxxx"*4096, "helloyyy"*4096]
+        deliv = [FakeDeliverable(p) for p in packets]
         async = mixminion.server.MMTPServer.AsyncServer()
 
         clientcon = mixminion.server.MMTPServer.MMTPClientConnection(
@@ -3971,7 +3971,7 @@
         while t.isAlive():
             server.process(0.1)
         t.join()
-        self.assertEquals(len(messagesIn), 0)
+        self.assertEquals(len(packetsIn), 0)
         self.assertEquals(deliv[0]._retriable, 1)
         self.assertEquals(deliv[1]._retriable, 1)
 
@@ -4388,11 +4388,18 @@
 Enabled = yes
 Allow: *
 Retry: every 1 hour for 1 day, every 1 day for 1 week
+"""
 
+MBOX_SEC = """
 [Delivery/MBOX]
-Enabled: no
+Enabled: yes
+AddressFile: %s
+ReturnAddress: a@b.c
+RemoveContact: b@c.d
 Retry: every 1 hour for 1 day, every 1 day for 1 week
+"""
 
+FRAGMENTED_SEC = """
 [Delivery/Fragmented]
 Enabled yes
 MaximumSize: 100k
@@ -4459,9 +4466,45 @@
                                            ])
         eq(info['Delivery/MBOX'].get('Version'), None)
 
+        # Now add frag and mbox
+        af = mix_mktemp()
+        writeFile(af, "")
+        try:
+            suspendLog()
+            conf = mixminion.server.ServerConfig.ServerConfig(
+                string=(SERVER_CONFIG % mix_mktemp()+FRAGMENTED_SEC+
+                        (MBOX_SEC%af)))
+        finally:
+            resumeLog()
+        if not os.path.exists(d):
+            os.mkdir(d, 0700)
+
+        inf = generateServerDescriptorAndKeys(conf,
+                                              identity,
+                                              d,
+                                              "key2",
+                                              d)
+        info = mixminion.ServerInfo.ServerInfo(string=inf)
+        eq(info['Delivery/MBOX'].get('Version'), '0.1')
         eq(info['Delivery/Fragmented'].get('Version'), '0.1')
         eq(info['Delivery/Fragmented'].get('Maximum-Fragments'), 6)
-        
+
+        # Test features
+        rfn = mixminion.Config.resolveFeatureName
+        SI = mixminion.ServerInfo.ServerInfo
+        self.assertEquals(rfn("softwarE",SI), ("Server", "Software"))
+        self.assertEquals(rfn("delivery/mbox:version",SI),
+                          ("Delivery/MBOX", "Version"))
+        self.assertEquals(rfn("caps",SI), ('-','caps'))
+        self.assertRaises(UIError, rfn, "version", SI)
+        self.assertRaises(UIError, rfn, "versiojdkasldj", SI)
+        self.assertRaises(UIError, rfn, "Server:foob", SI)
+        self.assertRaises(UIError, rfn, "Beep:version", SI)
+        self.assertEquals(info.getFeature("Server", "Packet-Key"),
+                          "<public key>")
+        self.assertEquals(info.getFeature("Incoming/MMTP", "Port"), "48099")
+        self.assertEquals(info.getFeature("-", "caps"), "mbox relay frag")
+
         # Check the more complex helpers.
         self.assert_(info.isValidated())
         self.assertEquals(info.getIntervalSet(),
@@ -4503,7 +4546,7 @@
         self.assert_(info.isNewerThan(time.time()-60*60))
         self.assert_(not info.isNewerThan(time.time()+60))
 
-        self.assertUnorderedEq(info.getCaps(), ["relay", "frag"])
+        self.assertUnorderedEq(info.getCaps(), ["frag", "relay", "mbox"])
 
         self.assertEquals(info.getIncomingMMTPProtocols(), ["0.3"])
         self.assertEquals(info.getOutgoingMMTPProtocols(), ["0.3"])
@@ -4528,9 +4571,9 @@
         self.assert_(stringContains(s,"Unrecognized key Unexpected-Key on line"))
 
         # Now make sure everything was saved properly
-        keydir = os.path.join(d, "key_key1")
+        keydir = os.path.join(d, "key_key2")
         eq(inf, readFile(os.path.join(keydir, "ServerDesc")))
-        mixminion.server.ServerKeys.ServerKeyset(d, "key1", d) # Can we load?
+        mixminion.server.ServerKeys.ServerKeyset(d, "key2", d) # Can we load?
         packetKey = Crypto.pk_PEM_load(
             os.path.join(keydir, "mix.key"))
         eq(packetKey.get_public_key(),
@@ -4681,6 +4724,7 @@
         eq(info3['Incoming/MMTP']['IP'], "192.168.100.4")
 
 
+
     def test_directory(self):
         eq = self.assertEquals
         examples = getExampleServerDescriptors()
@@ -5891,6 +5935,7 @@
             lock.release()
         try:
             DELAY = 0.2
+            LATENCY = 1.0
             overrideDNS({'foo'    : '10.2.4.11',
                          'bar'    : '18:0FFF::4:1',
                          'baz.com': '10.99.22.8'},
@@ -5912,14 +5957,14 @@
                               (socket.AF_INET, '10.2.4.11'))
             self.assertEquals(receiveDict['bar'][:2],
                               (mixminion.NetUtils.AF_INET6, '18:0FFF::4:1'))
-            self.assertFloatEq(receiveDict['foo'][2]-start, DELAY, .3)
-            self.assertFloatEq(receiveDict['bar'][2]-start, DELAY, .3)
+            self.assert_(DELAY <= receiveDict['foo'][2]-start <= DELAY+LATENCY)
+            self.assert_(DELAY <= receiveDict['bar'][2]-start <= DELAY+LATENCY)
             self.assertEquals(receiveDict['nowhere.noplace'][0], "NOENT")
             self.assertEquals(cache.getNonblocking("foo"),
                               receiveDict['foo'])
             self.assertEquals(cache.getNonblocking("baz.com")[:2],
                               (socket.AF_INET, '10.99.22.8'))
-            self.assertFloatEq(receiveDict['baz.com'][2]-start, DELAY*1.25, .3)
+            self.assert_(DELAY*1.25 <= receiveDict['baz.com'][2]-start <= DELAY*1.24 + LATENCY)
             cache.cleanCache(receiveDict['foo'][2]+
                              mixminion.server.DNSFarm.MAX_ENTRY_TTL+.001)
             self.assertEquals(cache.getNonblocking('foo'), None)
@@ -6155,7 +6200,7 @@
         self.assertEquals({9:10,11:12},
                           CU.readEncryptedPickled(f2,"pswd","ZZZ"))
         
-        # Test LazyEncryptedPickle
+        # Test LazyEncryptedStore
         class DummyPasswordManager(CU.PasswordManager):
             def __init__(self,d):
                 mixminion.ClientUtils.PasswordManager.__init__(self)
@@ -6167,8 +6212,8 @@
 
         f3 = os.path.join(d, "Baz")
         dpm = DummyPasswordManager({"Password1" : "p1"})
-        lep = CU.LazyEncryptedPickled(f3, dpm, "Password1", "Q:", "N:",
-                                     "magic0", lambda: "x"*3)
+        lep = CU.LazyEncryptedStore(f3, dpm, "Password1", "Q:", "N:",
+                                    "magic0", lambda: "x"*3)
         # Don't create.
         self.assert_(not lep.isLoaded())
         lep.load(create=0)
@@ -6178,7 +6223,7 @@
         self.assertEquals("x"*3, lep.get())
         self.assertEquals("x"*3, CU.readEncryptedPickled(f3,"p1","magic0"))
 
-        lep = CU.LazyEncryptedPickled(f3, dpm, "Password1", "Q:", "N:",
+        lep = CU.LazyEncryptedStore(f3, dpm, "Password1", "Q:", "N:",
                                      "magic0", lambda: "x"*3)
         lep.load()
         self.assertEquals("x"*3, lep.get())
@@ -6269,16 +6314,8 @@
         ks = mixminion.ClientDirectory.ClientDirectory(dirname)
 
         ## Write the descriptors to disk.
+        impdirname = self.writeDescriptorsToDisk()
         edesc = getExampleServerDescriptors()
-        impdirname = mix_mktemp()
-        createPrivateDir(impdirname)
-        for server, descriptors in edesc.items():
-            for idx in xrange(len(descriptors)):
-                fname = os.path.join(impdirname, "%s%s" % (server,idx))
-                writeFile(fname, descriptors[idx])
-                f = gzip.GzipFile(fname+".gz", 'wb')
-                f.write(descriptors[idx])
-                f.close()
 
         ## Test empty keystore
         eq(None, ks.getServerInfo("Fred"))
@@ -6718,6 +6755,103 @@
         ks.clean(now=now+oneDay*500) # Should zap all of imported servers.
         raises(MixError, ks.getServerInfo, "Lola")
 
+    def testFeatureMaps(self):
+        from mixminion.ClientDirectory import compressFeatureMap
+        from mixminion.ClientDirectory import formatFeatureMap
+        d = self.writeDescriptorsToDisk()
+        direc = mixminion.ClientDirectory.ClientDirectory(mix_mktemp())
+        self.loadDirectory(direc, d)
+        edesc = getExampleServerDescriptors()
+
+        bob3 = mixminion.ServerInfo.ServerInfo(string=edesc['Bob'][3])
+        bob4 = mixminion.ServerInfo.ServerInfo(string=edesc['Bob'][4])
+        features1 = [ 'software' , 'caps', 'status' ]
+        # Simple tests for getFeatureMap
+        fm = direc.getFeatureMap(features1)
+        self.assertUnorderedEq(fm.keys(), ['Fred','Lola','Alice','Bob','Lisa'])
+        self.assertEquals(len(fm['Bob']), 2)
+        self.assertUnorderedEq(fm['Bob'].keys(),
+                               [(bob3['Server']['Valid-After'],
+                                 bob3['Server']['Valid-Until']),
+                                (bob4['Server']['Valid-After'],
+                                 bob4['Server']['Valid-Until'])])
+        self.assertEquals(fm['Bob'].values()[0],
+                          { 'software' : 'Mixminion %s' %mixminion.__version__,
+                            'caps' : 'relay',
+                            'status' : '(not recommended)' })
+
+        # Now let's try compressing.
+        fm = { 'Alice' : { (100,200) : { "A" : "xx", "B" : "yy" },
+                           (200,300) : { "A" : "xx", "B" : "yy" },
+                           (350,400) : { "A" : "xx", "B" : "yy" } },
+               'Bob'   : { (100,200) : { "A" : "zz", "B" : "ww" },
+                           (200,300) : { "A" : "zz", "B" : "kk" } } }
+        fm2 = compressFeatureMap(fm, ignoreGaps=0, terse=0)
+        self.assertEquals(fm2,
+            { 'Alice' : { (100,300) : { "A" : "xx", "B" : "yy" },
+                          (350,400) : { "A" : "xx", "B" : "yy" } },
+              'Bob'   : { (100,200) : { "A" : "zz", "B" : "ww" },
+                          (200,300) : { "A" : "zz", "B" : "kk" } } })
+        fm3 = compressFeatureMap(fm, ignoreGaps=1, terse=0)
+        self.assertEquals(fm3,
+            { 'Alice' : { (100,400) :  { "A" : "xx", "B" : "yy" } },
+              'Bob'   : { (100,200) : { "A" : "zz", "B" : "ww" },
+                          (200,300) : { "A" : "zz", "B" : "kk" } } })
+
+        fm4 = compressFeatureMap(fm, terse=1)
+        self.assertEquals(fm4,
+            { 'Alice' : { (100,400) : { "A" : "xx", "B" : "yy" } },
+              'Bob'   : { (100,300) : { "A" : "zz", "B" : "ww / kk" } } })
+
+        # Test formatFeatureMap.
+        self.assertEquals(formatFeatureMap(["A","B"],fm4,showTime=0,cascade=0),
+          [ "Alice:xx yy", "Bob:zz ww / kk" ])
+        self.assertEquals(formatFeatureMap(["A","B"],fm3,showTime=1,cascade=0),
+          [ "Alice:1970/01/01 to 1970/01/01:xx yy",
+            "Bob:1970/01/01 to 1970/01/01:zz ww",
+            "Bob:1970/01/01 to 1970/01/01:zz kk" ])
+
+        day1 = 24*60*60
+        day2 = 2*24*60*60
+        day3 = 3*24*60*60
+        fmx = { 'Alice' : { (day1,day3)  : { "A" : "aa", "B" : "bb" } },
+                'Bob' : { (day1,day2) : { "A" : "a1", "B" : "b1" },
+                          (day2,day3) : { "A" : "a2", "B" : "b2" } } }
+        self.assertEquals(formatFeatureMap(["A","B"],fmx,cascade=1,sep="##"),
+          [ "Alice:", "  [1970/01/02 to 1970/01/04] aa##bb",
+            "Bob:",
+            "  [1970/01/02 to 1970/01/03] a1##b1",
+            "  [1970/01/03 to 1970/01/04] a2##b2" ])
+        
+        self.assertEquals(formatFeatureMap(["A","B"],fmx,showTime=1,cascade=2),
+          [ "Alice:", "  [1970/01/02 to 1970/01/04]", "    A:aa", "    B:bb",
+            "Bob:",
+            "  [1970/01/02 to 1970/01/03]", "    A:a1", "    B:b1",
+            "  [1970/01/03 to 1970/01/04]", "    A:a2", "    B:b2" ])
+
+    def writeDescriptorsToDisk(self):
+        edesc = getExampleServerDescriptors()
+        d = mix_mktemp()
+        createPrivateDir(d)
+        for server, descs in edesc.items():
+            for idx in xrange(len(descs)):
+                fname = os.path.join(d, "%s%s"%(server,idx))
+                writeFile(fname, descs[idx])
+                f = gzip.GzipFile(fname+".gz", 'wb')
+                f.write(descs[idx])
+                f.close()
+        return d
+
+    def loadDirectory(self, direc, d, now=None):
+        identity = getRSAKey(0,2048)
+        fingerprint = Crypto.pk_fingerprint(identity)
+        fname = getDirectory(
+            [os.path.join(d, s) for s in
+             ("Fred1", "Fred2", "Lola2", "Alice0", "Alice1",
+              "Bob3", "Bob4", "Lisa1", "Lisa2") ], identity)
+        mixminion.ClientDirectory.MIXMINION_DIRECTORY_URL = fileURL(fname)
+        mixminion.ClientDirectory.MIXMINION_DIRECTORY_FINGERPRINT = fingerprint
+        direc.updateDirectory(now=now)
 
     def assertSameSD(self, s1, s2):
         self.assert_(self.isSameServerDesc(s1,s2))
@@ -6759,8 +6893,9 @@
         parseEq("drop", DROP_TYPE, "", None)
         parseEq("test:foobar", 0xFFFE, "foobar", None)
         parseEq("test", 0xFFFE, "", None)
-        parseEq("0x999:zymurgy", 0x999, "zymurgy", None)
-        parseEq("0x999:", 0x999, "", None)
+        parseEq("0x0999:zymurgy", 0x999, "zymurgy", None)
+        parseEq("0x0999", 0x999, "", None)
+        parseEq("0x0999:", 0x999, "", None)
 
         def parseFails(s, f=self.failUnlessRaises):
             f(ParseError, mixminion.ClientDirectory.parseAddress, s)
@@ -6778,7 +6913,9 @@
         parseFails(":oom") # Missing module
         parseFails("0xZZ:zymurgy") # Bad hex literal
         parseFails("0xZZ") # Bad hex literal, no data.
-        parseFails("0x9999") # No data
+        parseFails("0x999")
+        parseFails("0x99999")
+        parseFails("0x9999z")
         parseFails("0xFEEEF:zymurgy") # Hex literal out of range
 
 
@@ -6848,7 +6985,7 @@
 
         pathSpec1 = parsePath(usercfg, "lola,joe:alice,joe")
 
-        ##  Test generateForwardMessage.
+        ##  Test generateForwardPacket.
         # We replace 'buildForwardPacket' to make this easier to test.
         replaceFunction(mixminion.BuildMessage, "buildForwardPacket",
                         lambda *a, **k:"X")
@@ -6862,13 +6999,13 @@
                 directory,
                 parseAddress("joe@cledonism.net"),
                 pathSpec1,
-                payload, time.time(), time.time()+200)
+                payload, 0, time.time(), time.time()+200)
             client.generateForwardPackets(
                 directory,
                 parseAddress("smtp:joe@cledonism.net"),
                 pathSpec1,
                 "Hey Joe, where you goin' with that gun in your hand?",
-                time.time(), time.time()+200)
+                0, time.time(), time.time()+200)
 
             for fn, args, kwargs in getCalls():
                 self.assertEquals(fn, "buildForwardPacket")
@@ -6885,14 +7022,13 @@
                 directory,
                 parseAddress("mbox:granola"),
                 parsePath(usercfg, "lola,joe:alice,lola"),
-                payload, time.time(), time.time()+200)
+                payload, 0, time.time(), time.time()+200)
             # And an mbox message with a last hop implicit in the address
             client.generateForwardPackets(
                 directory,
                 parseAddress("mbox:granola@Lola"),
                 parsePath(usercfg, "Lola,Joe:Alice"),
-                payload, time.time(), time.time()+200)
-                
+                payload, 0, time.time(), time.time()+200)
 
             for fn, args, kwargs in getCalls():
                 self.assertEquals(fn, "buildForwardPacket")
@@ -6908,13 +7044,13 @@
             undoReplacedAttributes()
             clearCalls()
 
-        ### Now try some failing cases for generateForwardMessage:
+        ### Now try some failing cases for generateForwardPackets
 
         # Temporarily replace BlockingClientConnection so we can try the client
         # without hitting the network.
         class FakeBCC:
             PROTOCOL_VERSIONS=["0.3"]
-            def __init__(self, family, addr, port, keyid):
+            def __init__(self, family, addr, port, keyid, serverName=None):
                 global BCC_INSTANCE
                 BCC_INSTANCE = self
                 self.family = family
@@ -7171,7 +7307,7 @@
     tc = loader.loadTestsFromTestCase
 
     if 0:
-        suite.addTest(tc(PacketTests))
+        suite.addTest(tc(ServerInfoTests))
         return suite
     testClasses = [MiscTests,
                    MinionlibCryptoTests,
@@ -7214,7 +7350,7 @@
 
     #DOCDOC
     if os.environ.get("MM_COVERAGE"):
-        allmods = [ mod for name, mod in sys.modules.items() 
+        allmods = [ mod for name, mod in sys.modules.items()
                     if (mod is not None and 
                         name.startswith("mixminion") and
                         name != 'mixminion._minionlib') ]