[cc'ing to the list, because many people seem to have these questions.]
On Thu, 2003-05-08 at 08:32, Colin Tuckley wrote:
> At 08:14 08/05/03 -0400, Nick Mathewson wrote:
>
> >Two guesses:
> > 1) The messages are destined for nodes that are down,
> > and your server is happily queueing them. This might
> > just be okay...
> > 2) ...but maybe there's a bug in the retry logic, or
> > something. I'll check it out.
>
> But wouldn't messages for nodes that are down be in the pool rather than
> the output Q? Or is the Q for nodes that were in the pool and are ready for
> onward transmission?
Hm. Perhaps I can explain.
First off, the client and the server keep a different set of
directories. The client queue (the one you can flush manually) belongs
to a ***completely different program*** from the server's pool and
queues.
The client queue exists for only two reasons: to allow users to batch
messages for bulk delivery, and to hang on to messages when servers are
down. Flushing is manual. If you aren't running a server, this is the
only pile-of-messages you have.
The server, on the other hand, maintains several buckets of messages,
***completely independently*** from the client. Data flows like this:
1) As the server receives messages over the network, it dumps them
into an "incoming" queue.
2) In the background, it decrypts and processes messages from the
"incoming" queue and inserts them into the "mix" pool, where they
wait until it's time to send them. (If you've enabled Cottrell
[Mixmaster-style] batching, they may wait for some while.)
3) Every time the mix interval elapses, some (or all) of the messages
from the "mix" pool get moved into the "outgoing" queue or one of
the "delivery" queues. Immediately, the server attempts to
deliver these messages to the appropriate nodes/MTAs/etc. If
delivery fails, the server retries failed messages at
subsequent mix intervals, according to the configured retry
rules.
After we've successfully delivered a message, or given up retrying, we
delete it from the outgoing or delivery queue.
> I've noticed that I've got messages in the pool which can't be sent because
> the target node is down. These seem to sit there for ever unless you run a
> flush, so the retry logic there seems broken.
The client queue and the server pool are *different*. There is no
automatic flushing from the client code. There is not retry logic in
the client code. (See comments at end.)
> But I've got a :lot: more messages in the output Q (over 70 compared with 5
> in the pool). There doesn't seem to be a way to force a retry for them in
> the way that "flush" does for pool messages.
They *are* getting retried periodically, and not getting delivered.
> I think I'm confused about why we have both a pool and an output Q.
Does this message explain it? Several people have had this confusion,
and I want to understand how to keep it from reappearing.
I suspect that the problem may be that people have gained some
expectations with Mixmaster that aren't met with Mixminion.
[I'm pulling this all out of thin air and guesswork. Mixmaster people:
correct me if I'm wrong?]
In Mixmaster, there's only one kind of pool. This is largely because
Mixmaster uses your MTA as its delivery engine, and so it hides some of
the complexity in the MTA's delivery logic. Our "incoming" queue
corresponds to Mixmaster's "sitting in our mbox, not yet processed."
Our "mix" pool corresponds (roughly) to Mixmaster's pool. Our
"outgoing" queue corresponds to the messages that Mixmaster has already
passed to sendmail, and doesn't need to think about anymore.
The other distinction that we make and Mixmaster doesn't is that we have
a separate client queue that receives the messages you put there, and
the messages that the client can't deliver, until you tell the client to
flush all those messages. It doesn't have automatic scheduled
redelivery. It doesn't have spiffy configurable batching policies. It
doesn't work by inserting messages into the pool of a local server.
The rationale is like this:
* The client program needs its own queue because I don't want the
client to know whether you've got a server running on the same
machine, and I sure as heck don't want the client to need to run
as a longstanding daemon in the way the server does.
* The client queue doesn't support automatic retrying because you
can't get it to reattempt, say, on the hour, without (A) a
daemon, or (B) a cron job, and the cron job is probably a better
way to go.
* The client's queue doesn't have mix-style batching because it
doesn't make any sense for anonymity. All the messages in the
client's queue come from the same origin, so strategies aimed at
decorrelating origin from destination are kind of silly.
(I _am_ sympathetic to attempts to conceal message timing, but
you can do that with the client code as it stands: queue all
your messages, and set a cron job to flush the pool every hour.)
* The client queue doesn't reattempt delivery on all previously
unsuccessful messages every time you say 'send' because...
because...
Actually, this one is probably the most debatable part. My gut
tells me that "send" means "send this message", not "send this
message and oh by the way reattempt delivery on all the
undeliverable messages you have lying around." If I have
messages for 5 defunct servers in the client queue, I surely
don't want to retry delivery on all of them every time I send a
new message!
I think. :)
Comments would be greatly appreciated,
--
Nick "0.0.4b1-any-day-now" Mathewson
Attachment:
signature.asc
Description: This is a digitally signed message part