[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-relays] max TCP interruption before Tor circuit teardown?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Tor CPU usage has dropped way back to 15-40% in the last few minutes,
even as I was increasing my total inbound connection limit to 150
connections.
This is a hell of a log line on a Raspberry Pi, which also just popped
out:
Oct 31 10:13:49.000 [notice] Circuit handshake stats since last time:
61533/218956 TAP, 30/30 NTor.
Wow.
Best,
- -Gordon M.
Gordon Morehouse:
> I've just seen the most amazing headshot of my Tor relay by a
> sudden massive SYN flood yet. I was online and started noticing
> problems with DNS on my local router. I checked my so-called
> monitoring setup, a window with a permanent ping to my router, and
> noticed a lot of timeouts. Obviously, that means trouble.
>
> Checked my Raspberry Pi tor relay setup, and there was an
> incredible SYN flood just starting. I have attached an image where
> the vertical scale reaches up to 5 megabits per second and where is
> column is two seconds. This is absolutely not established tor
> connection behavior. I don't know what *all* of it is, since once
> the Tor daemon dies, the SYN traffic seems to be steady at about
> 50KB/sec (of *just* SYNs inbound, and 100+KB/sec of outbound ICMP
> port unreachable packets). But that huge tsunami marks when the
> flood / circuit creation storm really got going.
>
> My relay crashed faster than I've ever seen it crash before, even
> with my newer protections in place. In under 5 minutes the out of
> memory killer reaped Tor. In previous situations, I've observed
> during floods that Tor's share of physical memory doesn't seem to
> increase. I could be wrong about that, but I think the thing eating
> all the RAM is TCP open/half-open sockets and/or associated tables
> in the Linux kernel - once RAM pressure becomes too intense, Tor is
> just the biggest thing around, so the oom-killer picks it and bam.
>
> The truly amazing and disturbing thing is that it's an hour and a
> half later now, and my router is still under extreme load from the
> incoming SYN packets. It hasn't yet crashed.
>
> In the meantime I added an iptables rule right under the
> "ESTABLISHED" rule suggested by David Serrano:
>
>
> Chain INPUT (policy ACCEPT) target prot opt source
> destination
>
> ACCEPT all -- anywhere anywhere
> state ESTABLISHED
>
> DROP tcp -- anywhere anywhere tcpflags:
> FIN,SYN,RST,ACK/SYN #conn src/0 > 75
>
> SYN_THROTTLE tcp -- anywhere anywhere multiport
> dports 31923,31924 state NEW
>
>
> (those weird params are from a connlimit suggestion I found for
> limiting the total number of TCP connections which may be handled
> over a chain.) I started off at 50, and am now up to 100. This
> is obviously a stopgap solution for an ongoing event, but it
> suggests some further ways that slower single-board computers can
> be made to weather such storms, possibly without (see earlier
> discussion on this thread) using fail2ban at all, which is very
> inefficient.
>
> What's quite alarming is that when I raise the limit a bit, to get
> the restarted Tor relay better connected, the SYN flood logs go
> crazy for a minute or two before instantaneously stopping when, I
> presume, the connection limit has been reached. Since the dropped
> packets above the global inbound connection limit are not logged,
> the sudden start/stop of the SYN flood logging (in the SYN_THROTTLE
> chain, they're logged) tells me I am still under intense SYN
> flood.
>
> After adding connection limits on the Tor box, my router recovered
> and is seeing ping times, ballpark 2x normal (0.8-1.2ms is normal,
> now it's more like 1.0-2.0,s), but things are working handily. I
> have also been able to connect to other services through the Tor
> relay again, with considerable difficulty.
>
> I notice that Tor is consuming all available CPU, even though it
> is, for the moment, relaying a pretty consistent 50-80KB/sec. I
> suspect that this is mostly circuit creation requests coming in
> over established Tor connections, which Tor is ... processing, I
> don't know how, but unless there's been some turnover and they get
> lucky, once another peer attempts a TCP/TLS handshake, its packets
> are likely to be dropped. This is probably not ideal.
>
> As long as the Raspberry Pi manages to stay up and keep logging
> I'll have all the data to go through later. (I already captured a
> lot.) I also have the logs from the other incident that I caught
> and watched in real time. I'm planning to do an analysis on the
> number of IPs involved, whether they are Tor relays are not, and
> other interesting things that can be gleaned from the logs. I
> promise some graphs and charts, punch and pie not so much.
> Unfortunately my life is quite busy right now so the report may
> take a while although it's kind of a high priority thing for me at
> the moment. This is pretty crazy. I can't verify it, but my
> suspicion is this is happening when I get my Stable flag (I have no
> idea if I'd gotten it back this morning or not) or shortly
> thereafter.
>
> Node is VastCatbox, flood started around 8am Pacific.
>
> Best, -Gordon M.
>
>
>
>
> _______________________________________________ tor-relays mailing
> list tor-relays@xxxxxxxxxxxxxxxxxxxx
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
- --
Sent from my thing that sends email.
-----BEGIN PGP SIGNATURE-----
iQEcBAEBCgAGBQJScpDnAAoJED/jpRoe7/ujHW0H/0ylbsd2e1r5FighEgfzCxMd
aCoSsJkv6GH1H+P/TWgGHDS7H2f6DxuKgoWehTp/S9hO5IhYJrxQ2+451tEsVKH+
fCM3GvJZgZHCtjtBpOZA64Avna6KW4AC7siN/LWiYYsf8tSfrFvmMbV0j7zUNJak
y2Vrz3+GcxauZ2o7lWkRAih4dQQQjtuFAyzv66hwtA7jsfx9T2hko1D4LIQfCQWE
gDPogVjxTvJCyG731VrY2ev4ileOwPFbIzWlcTqCBg69gDwBxwMlaUgsrgNcShjl
LLolqsGFRn0NIk36LBy8X4kmPpVbkZpIKWZitkUaPKASsgAnbomb3ztEENQSGB8=
=d6rN
-----END PGP SIGNATURE-----
_______________________________________________
tor-relays mailing list
tor-relays@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays