On 22 Oct (09:00:27), TinCan via tor-relays wrote: > Hi everyone, Hello TinCan! dgoulet here, from the Tor network team. First, thanks for this report, very useful! We are currently working on several fixes for this ongoing BEGIN_DIR DDoS badly affecting the network. Next release (soon!) will most likely have them in. It will be a fun one because to be effective, we will need most of the relays to upgrade so to avoid degradation. I have couple questions for you. > > The attack started targeting my relay yesterday at 12:00 UTC, 15 hours after > the relay achieved the HSDir flag. This time I had implemented a rate cap (on > version 0.4.9.14) for BEGIN_DIR which allowed me to try out a few different > things. The rate limiter has a configurable rate/burst limit and two > configurable defense types: > Either > - Refuse the stream by sending END_STREAM_REASON_NOTDIRECTORY back to the offender > Or > - Close the offending circuit by sending -END_CIRC_REASON_RESOURCELIMIT back to the offender > > At the beginning of the attack I was limiting BEGIN_DIR requests to 200/s and > refusing anything beyond that. The attacker was spawning 300 circuits per > second on average and hitting me with ~55,000 BEGIN_DIR requests per second > on average. After 8 hours of being attacked the relay logged "1647224804 > BEGIN_DIR rejected". During these first 8 hours, the relay remained > responsive, with memory usage staying flat at or under 1000MB, main thread > CPU usage 20-80% and traffic 200-850 Mbps with a bandwidth ratio of around > 2:1 down:up. CPU usage was highly correlated with the amount of traffic being > moved. There were also occasional latency spikes while the bandwidth was > close to the Gbit line rate. Are you able to tell me if these BEGIN_DIR are for onion service descriptors as in your relay being an HSDir? Or they are normal regular BEGIN_DIR requests from a client asking for other directory information (ex: consensus)? Also, that limit you put in is global to all circuits or per circuit? > > At 8 hours into the attack I reloaded Tor with the other defense type (close > BEGIN_DIR circuits) and lowering the limit to 100 requests per second. > Bandwidth usage immediately dropped to 80-100 Mbps in both directions and CPU > usage plummeted to <10%. The attack now changed its strategy and started > creating circuits at an increasing rate, topping out at 1500 circuits per > second, still no match for 12 Ryzen cores though. At the time of writing the > attacker is still spawning around 1000 circuits per second and the main > thread is at 10% of one core with the circuit work being spread out over the > rest of the cores. The circuit rate here is wild! We have DoS defenses in place for rate limiting circuit creation per IP address so are these 1000/sec circuits coming from 1000 different addresses? Note here that if they come from tor clients, you will have a single **unauthenticated channel** (tor channel) per client IP address. This would imply they are a single hop request to your relay so client -> your relay. And thus generic directory requests such as consensus fetch for instance. If they come from a relay, it would be on an **authenticated** channel and this means these BEGIN_DIR request come from a multi-hop circuit implying onion service descriptor get/post. I'm giving you these details because if you point a LLM at your 500MB debug logs ;), it is possible to tell if the channel those circuits are on is authenticated vs not. The difference is very important here because in both cases, we will apply different fixes. And it would be very useful to learn what you are seeing. Thanks a lot for your help! If you want to chat, I'm "dgoulet" on our #tor-dev IRC channel which you can access via Matrix as well. Cheers! David -- Fwj9yfk1MF4rtih0k2x/SemjPPhGxtNXDVaAx9n7TgQ=
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ tor-relays mailing list -- tor-relays@xxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to tor-relays-leave@xxxxxxxxxxxxxxxxxxxx