[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: SURB integration patches
Nick Mathewson <nickm@freehaven.net> writes:
> Hi, Brian! Thanks for the patches! I like your approach, but would
> you be able to rewrite your first patch to use something more like
> GPG's --status-fd mechanism?
Here you go. I added --status-fd outputs for generate-surb, inspect-surb, and
count-surbs. This patch contains both the --status-fd and the count-surbs
changes.
FYI, I didn't export the new 'STATUS' object from mixminion/Common.py: I
thought it better not to pollute the namespace.
It would be nice if there was some kind of common option processing for
things that are used everywhere, like --verbose and --config . As it stands I
added --status-fd options to the commands that I thought most likely to use
them, but really it ought to be accepted regardless of what client command is
being run. (but not for server commands.. it makes less sense to specify
--status-fd when running mixminion as a server).
cheers,
-Brian
? doc/statusfd.txt
Index: lib/mixminion/BuildMessage.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/BuildMessage.py,v
retrieving revision 1.73
diff -u -r1.73 BuildMessage.py
--- lib/mixminion/BuildMessage.py 23 Mar 2004 00:07:02 -0000 1.73
+++ lib/mixminion/BuildMessage.py 12 May 2004 20:18:26 -0000
@@ -9,11 +9,12 @@
import operator
import sys
import types
+import binascii
import mixminion.Crypto as Crypto
import mixminion.Fragments
from mixminion.Packet import *
-from mixminion.Common import MixError, MixFatalError, LOG, UIError
+from mixminion.Common import MixError, MixFatalError, LOG, STATUS, UIError
import mixminion.Packet
import mixminion._minionlib
@@ -296,8 +297,10 @@
prng = Crypto.AESCounterPRNG(Crypto.sha1(seed+userKey+"Generate")[:16])
- return _buildReplyBlockImpl(path, exitType, exitInfo, expiryTime, prng,
- seed)[0]
+ replyblock, secrets, tag = _buildReplyBlockImpl(path, exitType, exitInfo,
+ expiryTime, prng, seed)
+ STATUS.log("GENERATED_SURB", binascii.b2a_base64(tag))
+ return replyblock
def checkPathLength(path1, path2, exitType, exitInfo, explicitSwap=0,
suppressTag=0):
Index: lib/mixminion/ClientMain.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/ClientMain.py,v
retrieving revision 1.178
diff -u -r1.178 ClientMain.py
--- lib/mixminion/ClientMain.py 2 May 2004 18:45:15 -0000 1.178
+++ lib/mixminion/ClientMain.py 12 May 2004 20:18:26 -0000
@@ -26,7 +26,7 @@
MixFatalError, MixProtocolBadAuth, MixProtocolError, UIError, \
UsageError, createPrivateDir, englishSequence, floorDiv, isPrintingAscii,\
isSMTPMailbox, readFile, stringContains, succeedingMidnight, writeFile, \
- previousMidnight
+ previousMidnight, STATUS
from mixminion.Packet import encodeMailHeaders, ParseError, parseMBOXInfo, \
parseReplyBlocks, parseSMTPInfo, parseTextEncodedMessages, \
parseTextReplyBlocks, ReplyBlock, parseMessageAndHeaders, \
@@ -1072,6 +1072,7 @@
to the recipient instead of having the server
reassemble the message.
--reply-block-fd=<N> Read reply blcoks from file descriptor <N>.
+ --status-fd=<N> Send parseable status to file descriptor <N>.
%(extra)s
EXAMPLES:
@@ -1143,7 +1144,7 @@
"to=", "hops=", "path=", "reply-block=", "reply-block-fd=",
"input=", "queue", "no-queue",
"subject=", "from=", "in-reply-to=", "references=",
- "deliver-fragments", ])
+ "deliver-fragments", "status-fd=", ])
if not options:
sendUsageAndExit(cmd)
@@ -1164,6 +1165,8 @@
h_references = val
elif opt == '--deliver-fragments':
no_ss_fragment = 1
+ elif opt == '--status-fd':
+ STATUS.setFD(int(val))
if args:
sendUsageAndExit(cmd,"Unexpected arguments")
@@ -1554,6 +1557,7 @@
-i <file>, --input=<file> Read the results from <file>.
--passphrase-fd=<N> Read passphrase from file descriptor N instead
of asking on the console.
+ --status-fd=<N> Send parseable status to file descriptor <N>.
EXAMPLES:
Decode message(s) stored in 'NewMail', writing the result to stdout.
@@ -1566,7 +1570,7 @@
"""[Entry point] Decode a message."""
options, args = getopt.getopt(args, "hvQf:o:Fi:",
['help', 'verbose', 'quiet', 'config=',
- 'output=', 'force', 'input=', 'passphrase-fd=',])
+ 'output=', 'force', 'input=', 'passphrase-fd=', 'status-fd=',])
outputFile = '-'
inputFile = '-'
@@ -1578,6 +1582,8 @@
force = 1
elif o in ('-i', '--input'):
inputFile = v
+ elif o in ('--status-fd',):
+ STATUS.setFD(int(v))
try:
parser = CLIArgumentParser(options, wantConfig=1, wantClient=1,
@@ -1641,6 +1647,7 @@
--identity=<name> Specify a pseudonymous identity.
--passphrase-fd=<N> Read passphrase from file descriptor N instead
of asking on the console.
+ --status-fd=<N> Send parseable status to file descriptor <N>.
EXAMPLES:
Generate a reply block to deliver messages to the address given in
@@ -1669,7 +1676,7 @@
options, args = getopt.getopt(args, "hvQf:D:t:H:P:o:bn:",
['help', 'verbose', 'quiet', 'config=', 'download-directory=',
'to=', 'hops=', 'path=', 'lifetime=', 'passphrase-fd=',
- 'output=', 'binary', 'count=', 'identity='])
+ 'output=', 'binary', 'count=', 'identity=', 'status-fd=', ])
outputFile = '-'
binary = 0
@@ -1688,6 +1695,8 @@
sys.exit(1)
elif o in ('--identity',):
identity = v
+ elif o in ('--status-fd',):
+ STATUS.setFD(int(v))
try:
parser = CLIArgumentParser(options, wantConfig=1, wantClient=1,
wantLog=1, wantClientDirectory=1,
@@ -1734,6 +1743,7 @@
-v, --verbose Display extra debugging messages.
-f <file>, --config=<file> Use a configuration file other than ~/.mixminionrc
(You can also use MIXMINIONRC=FILE)
+ --status-fd=<N> Send parseable status to file descriptor <N>.
EXAMPLES:
Examine properties of reply blocks stored in 'FredsBlocks'.
@@ -1742,7 +1752,11 @@
def inspectSURBs(cmd, args):
options, args = getopt.getopt(args, "hvQf:",
- ["help", "verbose", "quiet", "config=", ])
+ ["help", "verbose", "quiet", "config=", "status-fd=", ])
+
+ for opt,val in options:
+ if opt == "--status-fd":
+ STATUS.setFD(int(val))
try:
parser = CLIArgumentParser(options, wantConfig=1, wantLog=1,
@@ -1770,11 +1784,52 @@
used = surblog.isSURBUsed(surb) and "yes" or "no"
print surb.format()
print "Used:", used
+ STATUS.log("INSPECT_SURB", "%s %d %s",
+ surb.getDigest(),
+ surb.timestamp,
+ surblog.isSURBUsed(surb))
except ParseError, e:
print "Error while parsing: %s"%e
finally:
surblog.close()
+_COUNT_SURBS_USAGE = """\
+Usage: %(cmd)s [options]
+
+Options:
+ -h, --help Print this usage message and exit.
+ -i <file>, --input=<file> Read the message from <file>. (Defaults to stdin.)
+ --status-fd=<N> Send parseable status to file descriptor <N>.
+
+EXAMPLES:
+ Report how many SURBs are necessary to send the message in file <data>
+ %(cmd)s -i data
+""".strip()
+
+def countSURBs(cmd, args):
+ options, args = getopt.getopt(args, "hi:",
+ ["help", "input=", "status-fd=", ])
+ inFile = '-'
+
+ for opt,val in options:
+ if opt in ('-i', '--input'):
+ inFile = val
+ elif opt == "--status-fd":
+ STATUS.setFD(int(val))
+
+ if inFile == '-':
+ if os.isatty(sys.stdin.fileno()):
+ print "Enter your message. Type %s when you are done."%(
+ EOF_STR)
+ message = sys.stdin.read()
+ else:
+ message = readFile(inFile)
+ payloads = mixminion.BuildMessage.encodeMessage(message, 0, "")
+ STATUS.log("COUNT_SURBS", "%d %d", len(message), len(payloads))
+ print "Message of length %d will require %d SURBs" % (len(message),
+ len(payloads))
+
+
_FLUSH_QUEUE_USAGE = """\
Usage: %(cmd)s [options] [servername] ...
-h, --help Print this usage message and exit.
@@ -1925,6 +1980,7 @@
-v, --verbose Display extra debugging messages.
-f <file>, --config=<file> Use a configuration file other than ~/.mixminionrc
(You can also use MIXMINIONRC=FILE)
+ --status-fd=<N> Send parseable status to file descriptor <N>.
EXAMPLES:
Describe the state of fragmented messages currently being reassembled.
@@ -1934,7 +1990,12 @@
def listFragments(cmd, args):
options, args = getopt.getopt(args, "hvQf:D:",
["help", "verbose", "quiet", "config=",
- 'download-directory=',])
+ 'download-directory=', "status-fd=",])
+
+ for opt,val in options:
+ if opt == "--status-fd":
+ STATUS.setFD(int(val))
+
try:
parser = CLIArgumentParser(options, wantConfig=1, wantLog=1,
wantClient=1, wantClientDirectory=1)
@@ -1966,13 +2027,14 @@
-v, --verbose Display extra debugging messages.
-f <file>, --config=<file> Use a configuration file other than ~/.mixminionrc
(You can also use MIXMINIONRC=FILE)
+ --status-fd=<N> Send parseable status to file descriptor <N>.
""".strip()
def reassemble(cmd, args):
options, args = getopt.getopt(args, "hvQf:D:Po:F",
["help", "verbose", "quiet", "config=",
'download-directory=','--purge',
- '--output', '--force'])
+ '--output', '--force', "status-fd="])
reassemble = 1
if cmd.endswith("purge-fragments") or cmd.endswith("purge-fragment"):
reassemble = 0
@@ -1998,8 +2060,10 @@
outfilename = v
elif o in ("-P", "--purge"):
purge = 1
- elif o ("-F", "--force"):
+ elif o in ("-F", "--force"):
force = 1
+ elif o in ("--status-fd",):
+ STATUS.setFD(int(v))
if not args:
print "No message IDs provided."
Index: lib/mixminion/Common.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Common.py,v
retrieving revision 1.138
diff -u -r1.138 Common.py
--- lib/mixminion/Common.py 2 May 2004 18:45:15 -0000 1.138
+++ lib/mixminion/Common.py 12 May 2004 20:18:26 -0000
@@ -1087,6 +1087,68 @@
def flush(self): pass
def close(self): pass
+class Status:
+ """I am used to emit machine-parseable status messages to the file
+ descriptor specifed by the --status-fd argument. The set of valid
+ messages and their format is described in doc/statusfd.txt .
+
+ Each message consists of a 'name', in all caps, and (possibly) a set of
+ arguments that depend upon the name. To maintain compatibility,
+ arguments must only be added, never removed or changed.
+
+ Messages should be emitted by importing the global STATUS instance from
+ this module and doing something like:
+
+ STATUS.log('SURB_GENERATED', '%s' % surbid)
+
+ Each message should be independent (it seems safer to not assume any
+ particular ordering between messages).
+
+ Each message is a single line: the '[MIXMINION]:' header and the newline
+ are provided by the log() function. Keep machine-parseability in mind
+ when adding these messages: make sure the name is unique, separate
+ arguments with spaces (assuming the arguments are guaranteed to not
+ contain spaces), and consider providing a regular expression in
+ doc/statusfd.txt for integrators to use in their status parsers.
+ """
+
+ def __init__(self):
+ self.fd = None
+ self.__lock = threading.Lock()
+
+ def setFD(self, fdnum):
+ # this will fail if the invoking user did not actually open the file
+ # descriptor they ask us to use, for example if they did:
+ # mixminion send --status-fd=3
+ # instead of:
+ # mixminion send --status-fd=3 3>somewhere.txt
+ #
+ # remember, this is not meant for use by a shell, it is for
+ # front-end programs that are running mixminion in a child process.
+ # There will be a pipe connected to this fd which the parent process
+ # will be reading from.
+
+ self.fd = os.fdopen(fdnum, "w", 0)
+
+ def log(self, name, message, *args):
+ if self.fd is None:
+ return
+
+ if args is None:
+ m = message
+ else:
+ m = message % args
+
+ self.__lock.acquire()
+ try:
+ self.fd.write("[MIXMINION:] %s %s\n" % (name, m))
+ finally:
+ self.__lock.release()
+
+# The global Status instance for the mixminion client. Status messages
+# should be emitted with STATUS.log()
+STATUS = Status()
+
#----------------------------------------------------------------------
# Time processing
Index: lib/mixminion/Main.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Main.py,v
retrieving revision 1.72
diff -u -r1.72 Main.py
--- lib/mixminion/Main.py 6 Mar 2004 00:04:38 -0000 1.72
+++ lib/mixminion/Main.py 12 May 2004 20:18:27 -0000
@@ -127,6 +127,7 @@
"generate-surbs" : ( 'mixminion.ClientMain', 'generateSURB' ),
"inspect-surb" : ( 'mixminion.ClientMain', 'inspectSURBs' ),
"inspect-surbs" : ( 'mixminion.ClientMain', 'inspectSURBs' ),
+ "count-surbs" : ( 'mixminion.ClientMain', 'countSURBs' ),
"flush" : ( 'mixminion.ClientMain', 'flushQueue' ),
"inspect-queue" : ( 'mixminion.ClientMain', 'listQueue' ),
"clean-queue" : ( 'mixminion.ClientMain', 'cleanQueue' ),
Index: lib/mixminion/Packet.py
===================================================================
RCS file: /home/minion/cvsroot/src/minion/lib/mixminion/Packet.py,v
retrieving revision 1.75
diff -u -r1.75 Packet.py
--- lib/mixminion/Packet.py 6 Mar 2004 00:04:38 -0000 1.75
+++ lib/mixminion/Packet.py 12 May 2004 20:18:27 -0000
@@ -525,7 +525,7 @@
def format(self):
import mixminion.ServerInfo
- digest = binascii.b2a_hex(sha1(self.pack()))
+ digest = self.getDigest()
expiry = formatTime(self.timestamp)
if self.routingType == SWAP_FWD_IPV4_TYPE:
routing = parseIPV4Info(self.routingInfo)
@@ -538,6 +538,9 @@
Expires at: %s GMT
First server is: %s""" % (digest, expiry, server)
+ def getDigest(self):
+ return binascii.b2a_hex(sha1(self.pack()))
+
def pack(self):
"""Returns the external representation of this reply block"""
return struct.pack(RB_UNPACK_PATTERN,
--- /dev/null 1969-12-31 16:00:00.000000000 -0800
+++ doc/statusfd.txt 2004-05-12 12:59:52.000000000 -0700
@@ -0,0 +1,54 @@
+
+The following messages are emitted on whatever file descriptor 'N' you pass
+(by number) into the --status-fd=N argument. They are designed to be machine
+parseable. Each message, once made available in an official mixminion
+release, will not be removed or shortened. New arguments may be added, so
+parse these with regular expressions which match from the left. All
+status-fd messages are prefixed with the string '[MIXMINION:] ', which may
+help distinguish them if for some reason you are forced to intermingle them
+with stderr.
+
+These are intended for use by front-end programs. Each message is listed
+with a python-style regular expression that can be used to parse the line.
+
+
+
+COUNT_SURBS <msglen> <surbcount>
+
+ r'COUNT_SURBS (\d+) (\d+)'
+
+ Emitted in the 'count-surbs' command. Determines the number of SURBs which
+ will be required to send the given message. <msglen> is the message length
+ in bytes, <surbcount> is the number of SURBs that will be required.
+
+
+GENERATED_SURB <surbid>
+
+ r'GENERATED_SURB (\S+)'
+
+ Emitted in the 'generate-surb' and 'generate-surbs' commands, one per SURB
+ generated. <surbid> is the SURB's base64-encoded "decoding handle", which
+ is only available to the creator of the SURB (at generation time) and the
+ final recipient of the reply message, both of which are usually the same
+ person. It is not visible in the finished SURB, and the user who sends a
+ message through this SURB will not be able to see it.
+
+ The decoding handle will eventually be visible in the final delivered
+ message. When that last message is delivered in ascii-armored form, it will
+ be the value of the "Decoding-Handle:" header.
+
+
+INSPECT_SURB <surbdigest> <expiretime> <used>
+
+ r'INSPECT_SURB (\w+) (\d+) (\d+)'
+
+ Emitted in the 'inspect-surb' command, one per SURB inspected. <surbdigest>
+ is the (hex) hash of the SURB itself, useable as an identifying handle.
+ Unlike the decoding handle, this digest is for use by the sender of the
+ reply message, and is visible by anyone who holds the SURB. <expiretime> is
+ the timestamp (seconds since the epoch) of the last time at which the SURB
+ is guaranteed to be useable. After this time, one or more of the component
+ remailer keys may have expired, rendering the SURB incapable of reaching
+ its final destination. <used> is 1 if this instance of mixminion remembers
+ sending a message with this SURB (thus rendering it unusable, as SURBs are
+ Single-Use), otherwise it is 0.