-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jobiwan Kenobi:
> I've been running a relay for about months now. It runs on an 1.6
> Ghz single core Atom with hyperthreading, 1GB of RAM. It's on my
> home connection; I advertise only 176 KB/sec.
>
> Under normal conditions, it pumps around 100KB/sec, it has around
> 600 connections, it uses around 20% CPU. Everything looks very
> healthy.
>
> Lately I have seen a couple of incidents where the number of
> connections suddenly goes up to over 3000, traffic increases
> heavily, CPU usage goes well over 150% (out of possible 200).
> Traffic can go up to between 500 and 1000 KB/sec for long periods
> of time. Sometimes it seems that my relay just can't take it
> anymore. In the log, the ratio of TAP handshakes goes wild, and I
> get clock jump warnings. My clock does not jump. This is Tor
> hanging while allocating memory.
Welcome to the world of the Raspberry Pi / BeagleBone / CubieBoard
operator, except normally we'd have crashed (without some defenseive
measures) before the clock jump thing - the Pi in particular has a
known dodgy "clock."
> Today I had a really bad episode, where the box started thrashing.
> When it became responsive again, Tor was left in a state where it
> was constantly downloading about 400KB/sec more than it was
> uploading. Normally I have a little bit more up than down, because
> I'm a directory server as well. I can not explain having a lot
> more down than up. I can fantasize that those hangs/clock jumps
> ("assuming established circuits no longer work") could leave
> 'half' circuits.
As best I can tell, probably that's a flood of incoming TAP requests
or TLS handshakes.
> Finally I restarted my relay, (which I really don't like to do)
> and after a while it stabilized. At this point, my router shows a
> peak of almost 8000 NAT sessions.
>
> Is this normal behavior of the network (esp. the sudden increase
> in connections) or is this another kind of attack/probe like what
> we've seen in early September? Is this because this machine is
> just too underpowered? Should I collect/provide any diagnostics?
> Have others seen similar events?
Have a look at the Raspberry Pi threads and search for "circuit
creation storms." I'm slowly developing a set of defensive iptables
rules for low-power relays which you might want to have a look at, but
as your machine is far more capable than a Pi, you'll need to adjust
accordingly (and then, I hope, contribute back!)
https://github.com/gordon-morehouse/cipollini/tree/master/contrib/90_slowboards/tcp_syn_limit
(Ignore the fail2ban stuff for now, I found a more efficient way to
handle the problem with the help of a list reader.)
> Nov 16 22:39:52.000 [notice] Heartbeat: Tor's uptime is 36 days
> 6:00 hours, with 615 circuits open. I've sent 192.16 GB and
> received 171.74 GB. Nov 16 22:39:52.000 [notice] Average packaged
> cell fullness: 87.305% Nov 16 22:39:52.000 [notice] TLS write
> overhead: 9% Nov 16 23:39:28.000 [notice] Circuit handshake stats
> since last time: 6011/6013 TAP, 35/35 NTor. Nov 17 00:39:28.000
> [notice] Circuit handshake stats since last time: 12521/12521 TAP,
> 47/47 NTor. Nov 17 01:24:45.000 [warn] Your system clock just
> jumped 138 seconds forward; assuming established circuits no
> longer work. Nov 17 01:39:28.000 [notice] Circuit handshake stats
> since last time: 49689/1454481 TAP, 51/51 NTor. Nov 17
> 02:28:44.000 [warn] Your system clock just jumped 174 seconds
> forward; assuming established circuits no longer work. Nov 17
> 02:33:12.000 [warn] Your system clock just jumped 268 seconds
> forward; assuming established circuits no longer work. Nov 17
> 02:35:17.000 [warn] Your system clock just jumped 125 seconds
> forward; assuming established circuits no longer work. Nov 17
> 02:39:28.000 [notice] Circuit handshake stats since last time:
> 55822/3477278 TAP, 37/37 NTor.
>
>
> Nov 25 06:53:31.000 [notice] Circuit handshake stats since last
> time: 28194/28200 TAP, 32/32 NTor. Nov 25 07:29:57.000 [warn] Your
> system clock just jumped 114 seconds forward; assuming established
> circuits no longer work. Nov 25 07:31:58.000 [warn] Your system
> clock just jumped 121 seconds forward; assuming established
> circuits no longer work. Nov 25 07:53:31.000 [notice] Circuit
> handshake stats since last time: 24207/2692265 TAP, 42/42 NTor.
> Nov 25 08:53:31.000 [notice] Circuit handshake stats since last
> time: 76451/4271579 TAP, 49/49 NTor. Nov 25 09:53:31.000 [notice]
> Circuit handshake stats since last time: 84956/3666170 TAP, 42/42
> NTor. Nov 25 10:45:15.000 [warn] Your system clock just jumped 302
> seconds forward; assuming established circuits no longer work. Nov
> 25 11:01:59.000 [warn] Your system clock just jumped 1004 seconds
> forward; assuming established circuits no longer work. Nov 25
> 11:02:04.000 [notice] Circuit handshake stats since last time:
> 64816/3151447 TAP, 38/38 NTor. Nov 25 11:02:18.000 [notice]
> Heartbeat: Tor's uptime is 5 days 11:42 hours, with 2450 circuits
> open. I've sent 46.92 GB and received 46.11 GB. Nov 25
> 11:02:18.000 [notice] Average packaged cell fullness: 82.623% Nov
> 25 11:02:18.000 [notice] TLS write overhead: 8% Nov 25
> 11:53:31.000 [notice] Circuit handshake stats since last time:
> 18710/1099205 TAP, 193/202 NTor. Nov 25 13:22:03.000 [warn] Your
> system clock just jumped 1923 seconds forward; assuming established
> circuits no longer work. Nov 25 13:27:23.000 [notice] Circuit
> handshake stats since last time: 70795/3452514 TAP, 138/140 NTor.
> Nov 25 13:35:51.000 [warn] Your system clock just jumped 828
> seconds forward; assuming established circuits no longer work. Nov
> 25 13:42:15.000 [warn] Your system clock just jumped 384 seconds
> forward; assuming established circuits no longer work. Nov 25
> 13:43:15.000 [notice] Received reload signal (hup). Reloading
> config and resetting internal state. Nov 25 13:43:16.000 [notice]
> Read configuration file "/opt/etc/tor/torrc". Nov 25 13:46:09.000
> [notice] Interrupt: we have stopped accepting new connections, and
> will shut down in 30 seconds. Interrupt again to exit now. Nov 25
> 13:46:39.000 [notice] Clean shutdown finished. Exiting.
>
> _______________________________________________ tor-relays mailing
> list tor-relays@xxxxxxxxxxxxxxxxxxxx
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
- --
Sent from my thing that sends email.
-----BEGIN PGP SIGNATURE-----
iQEcBAEBCgAGBQJSk53yAAoJED/jpRoe7/ujaqsH/1AO35w15Gp9hvvUab9BgHnc
ckr7XvpOhm56FJ4ThOUuULAsc0XE/zHywb8MRY7CWPqjPirLC0auV9op2VWq3dUJ
NoVy9AG0WzKNIwk+3xDcebUrPLOgWdGJPPQ9jHaqD56zSSSGKcKmq9k2VnCM/Ypc
4NMTgIAWeA7PyzFJxGhelDI+6TVPBFXgrmUeAlqX8MtrSEgLjv2F3qeAcwSkU/oa
bL6Rtj0FB8Pqlk72mjjbCP2Cp49Xn1fMNWccrfW5agvyFhsNLXUgInM+onHbAqYS
x71Nkys9weiEeUO5JZdD+EcHzvV5dCWFys1p1B9WaMp+rK/RvMdhPRsOfYdtZ8A=
=c3H3
-----END PGP SIGNATURE-----
Attachment:
0x1EEFFBA3.asc
Description: application/pgp-keys
_______________________________________________ tor-relays mailing list tor-relays@xxxxxxxxxxxxxxxxxxxx https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays