[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: Is tor being slowed up to ~20ms per hop by TCP delayed ACK?

This pretty much mirrors my experiences. I have a tor node running on a dedi used for other purposes as well. It's capped to 300 KB/s currently, and marked as fast, guard, and stable by all directories, so it handles that much traffic 24/7. Tor generally has pretty high memory usage (over 200mb before on 0.1.x, about 100mb on 0.2.1.x), but very low CPU usage (<3% on average). This is not on a powerful CPU by any means.

It seems pretty obvious to me that the bottleneck in tor lies mostly with insufficient bandwidth, not overhead from encryption (minimal) or tcp (seriously?). Improvements in those two areas could help, but probably only in regards to reducing the load of bandwidth on nodes. What the network really seems to need is more high-speed nodes, so more connections can be handled without bottlenecking at some slower or too busy node. There is probably room for improvement in the node selection algorithm, too.

I think energy would be better spent trying to recruit more node providers, finding a way to give back to node providers, and ensuring their legal status. Technical improvements only go so far when, by design,  tor will add 3 very busy bottleneck points to the route.

As of writing, 260 KB/s steadily coming from tor.

- John Brooks

On Thu, Jul 24, 2008 at 1:01 AM, Scott Bennett <bennett@xxxxxxxxxx> wrote:
    On Wed, 23 Jul 2008 21:24:43 -0400 phobos@xxxxxxxxxx wrote:
>On Sun, Jul 20, 2008 at 10:52:44AM -0700, adam_richter2004@xxxxxxxxx wrote 2.5K bytes in 53 lines about:
>: tor- or libevent-1.3e.  Have I missed some other library
>: that tor is using to avoid unnecessary Nagle delays?
>:      I am thinking about modifying tor so that, _when it has
>: nothing else to do_, it will set and clear TCP_NODELAY on any sockets
>: that the outgoing data is flushed.  Outgoing TCP data would continue
>: to be coalesced when tor has more input pending from any connection.
>: The pushes would only be done if tor is about to go to sleep in
>: select() and has transmitted unpushed data pending.
>:      Am I overlooking something?  Has anyone else looked into this?
>Others have raised the possibility that tcp overhead and delays are the
>leading cause of tor slowness, not encryption overhead.  I don't know
>that anyone has done a rigorous analysis of tcp congestion/nagle/delays
>over Tor.
    My server is running under FreeBSD 6.3-STABLE on a Dell Inspiron XPS,
which has a 3.4 GHz P4 with HTT enabled and 2GB of memory.  My ISP
connection is an ADSL currently limited to ~4.56 Mb/s reception and ~698
Kb/s transmission.  tor input traffic varies from a few KB/s to, say, 180
KB/s, but peak average over 10 - 15 seconds is probably ~110 KB/s.  The
busier times usually have sustained rates running from 45 - 80 KB/s.  It
rarely gets up to such levels for more than a few seconds until after it
has been running long enough to get the Guard flag in consensus from the
directory authorities, but then may spend a lot of the time in that
activity range.  The server offers exit service for many ports, but not
port 80.  It also offers directory mirror service, so some of the limited
transmit bandwidth can be tied up by directory replies.  tor is configured
with NumCPUs 2 and has 6 threads most of the time.  When my system is
essentially inactive except for tor with its client side idle, top(1)
usually displays 99% - 100% idle time.  It occasionally drops lower by 1%
to 6% for as long as a few seconds, but no longer.  I don't know what it
tor is doing at those times that is different from the rest of the time.
Once in a great while, a couple of tor threads will chew up considerably
more, usually while incorporating cached-descriptors.new into
cached-descriptors.  Note that by using the system idle time as an
indicator, all overhead required by networking activities is already
included with the tor process's CPU time.  There is undoubtedly a certain
amount of time consumed by other, mostly idle processes in the system,
too, but that is so trivial that it can be safely ignored for the
conditions described above.
    So my impression of tor is that it is basically a highly network-bound
application, contrary to many other reports of high CPU consumption by tor
that I've read over time on this list.  It looks to me as though my system
would easily handle ten times my current available bandwidth while still
leaving 90% or more CPU time available for other activities.  Of course,
it would be fun to try it out on a connection with huge bandwidth available
for a few weeks to see whether those performance characteristics hold up,
but until fiber optics installed to residences is available to me and at a
rate affordable by me, which is not currently detectable on the horizon:-(,
I have no way of finding out.  But it does make me curious to know whether
my little machine could handle a load like that of blutmagie, for example,
if only it had a 100 Mb/s or faster connection. :-)

                                 Scott Bennett, Comm. ASMELG, CFIAG
* Internet:       bennett at cs.niu.edu                              *
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *