On 2014-12-02 21:45, l.m wrote:
Perhaps the new implementation of the hidden services will be better
?
How is it going ?
I don't see anything in the improvements suggested for hidden
services
that would help this situation. Though I would be grateful for being
corrected.
First, I just want to say I only meant sheep(s) to emphasize that you
don't know how many black sheep you have participating. I mentioned
the part about this potentially being an attack external to Tor out
of
concern for your participation in a de-anonymizing attack on your
hosted HS. I see your HS's are offline while you troubleshoot this so
that's good. Next, I'm confused by what you describe. Sorry I deleted
your previous email so I may repeat some things you already said.
- no evidence of any HS being flooded from logs. No evidence of a
load
on any particular HS.
and
- next to no bandwidth consumption. Does this include no processor
use? I don't recall if that was mentioned before.
- no evidence is apparent from checking REND_QUERY=HS. So no request
to rendezvous. Might make sense given little/no traffic.
- your guards go offline. This is contradictory. If the attack is
within Tor via a HS it implies the HS traffic *reliably* makes it to
at least your guard before you experience the symptomatic overload
and
timeout. Meaning there must be traffic you can detect. Otherwise the
attacker would likely lose their connection to the rendezvous point
(at least sometimes) by committing to the attack. What I mean is in
order for this to be an attack via malicious HS it would need to
succeed in not timing out until the traffic reaches your guard and
server. That's two circuits that must work before failing at your
guard. Not to mention you already tried changing the guards. It just
seems implausible to occur reliably enough to take your server down.
This assumes little/no traffic and no heavy cpu usage.
-- Now I don't know how you setup your logging but I assume you would
know if there was a load on any particular site or flood. I can
suggest beefing up this part of the auditing trail. You could use
proxy (on the same server) in your server blocks (Nginx?) for each HS
(or in batches). Then you can use SPI to analyses the traffic of each
proxy for a build up of use that might be causing your timeouts.
Though I don't see the use if your logging is as good as you
mentioned.
If there's no traffic, no cpu usage, no evidence of HS load except
your guards are timing out--I'm back to the implausible. Two circuits
that reliably take down your guards. There *has* to be traffic on
your
side you can measure or some load indicator. Either that or the
attack
is external to Tor. On the other hand you could reply and say 'yeh
lots of cpu use'. In which case sorry for wasting your time. If there
is alot of cpu use the VM-partitioning solution is the best solution
as it would guarantee at least some guards available to your other
HS's. It also provides you more granular control over hardware
allocation. Either way you have to assume at some point you will be
targeted externally (from Tor) to de-anonymize your HS's. Shared
hosting.. many HS's.. you're an eavesdropping goldmine.
-- leeroy bearr
Hi, thanks for supporting.
As a reminder, here is the archive for this thread:
https://lists.torproject.org/pipermail/tor-talk/2014-December/035807.html
And that's December:
https://lists.torproject.org/pipermail/tor-talk/2014-December/thread.html
Problem is still going on, let's recap:
Tor goes 100% CPU only when (re)starting and publishing the HS, this
takes around 5 minutes,
after that, it uses about 1% - 5% of 1 core.
We _are_ able to see the access.log and error.log of each virtual
host,
since they are simple nginx vhosts.
We also have a script that records the access.log size on mysql, we
use
that script to delete unused accounts.
We confirm that both access.log and error.log doesn't move of an inch,
basically there is no http traffic.
Bandwidth usage is consistently around 5KB - 15KB / second.
Currently we disabled from torrc the most visited vhosts, have a look
at
this:
http://fuckyouhotwkd3xh.onion/ (working)
http://ijaw6sx25rzose3c.onion/ (working)
http://3xsrosyv52vefh2l.onion/ (working)
http://v5uhidj456rn4cra.onion/ (working)
http://za3uovobxijq4grb.onion/ (not working)
http://4vxnpigkblvmud4i.onion/ (not working)
http://3jfbvyg5pggg7mfg.onion/ (not working)
this is a list of the HS we are currently hosting. We tried more
addresses but in short, it looks like only 10% of the HS are working.
Those are the Tor logs of the current session:
Dec 03 XXX [notice] Bootstrapped 90%: Establishing a Tor circuit.
Dec 03 XXX [notice] Tor has successfully opened a circuit. Looks like
client functionality is working.
Dec 03 XXX [notice] Bootstrapped 100%: Done.
Dec 03 XXX [notice] Your Guard regar42 (XXX) is failing more circuits
than usual. Most likely this means the Tor network is overloaded.
Success counts are 120/181. Use counts are 453/453. 120 circuits
completed, 0 were unusable, 0 collapsed, and 0 timed out. For
reference,
your timeout cutoff is 91 seconds.
Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing a very large
amount of circuits. Most likely this means the Tor network is
overloaded, but it could also mean an attack against you or
potentially
the guard itself. Success counts are 120/241. Use counts are 453/453.
120 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out.
For reference, your timeout cutoff is 91 seconds.
Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing an extremely
large
amount of circuits. This could indicate a route manipulation attack,
extreme network overload, or a bug. Success counts are 60/241. Use
counts are 453/453. 60 circuits completed, 0 were unusable, 0
collapsed,
and 0 timed out. For reference, your timeout cutoff is 91 seconds.
Dec 03 XXX [notice] We'd like to launch a circuit to handle a
connection, but we already have 32 general-purpose client circuits
pending. Waiting until some finish.
Dec 03 XXX [notice] Your Guard TorKuato (XXX) is failing more circuits
than usual. Most likely this means the Tor network is overloaded.
Success counts are 105/151. Use counts are 81/81. 105 circuits
completed, 0 were unusable, 0 collapsed, and 2063 timed out. For
reference, your timeout cutoff is 60 seconds.
(after 5 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a
connection, but we already have 32 general-purpose client circuits
pending. Waiting until some finish. [749254 similar message(s)
suppressed in last 600 seconds]
(after another 5 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a
connection, but we already have 32 general-purpose client circuits
pending. Waiting until some finish. [398 similar message(s) suppressed
in last 600 seconds]
(after 20 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a
connection, but we already have 32 general-purpose client circuits
pending. Waiting until some finish. [6 similar message(s) suppressed
in
last 600 seconds]
We are running all this on a small VPS with 1GB of RAM.
The RAM finishes quickly but there is plenty of free swap.
The service was running fine with much more hidden services, on the
same
hardware.
We could increase the RAM but doubt that's the problem.
Again .. any Tor developer willing to dig into this ?
Ours is an experimental project, there is nothing business-oriented
here.
If we are pushing the limits of hidden services capabilities, or
really
are under a low-level Tor based attack, isn't this a good chance to
improve Tor's hidden service code
or just improve Tor's hidden service _logging_ code ?
Perhaps we should post this on tor-dev ?