On Tuesday, 18 March 2025 18:31 Tor at 1AEO via tor-relays wrote: > Listed general information below. What other information is helpful? > > Didn't want to log but seems will need something to troubleshoot issues. > Will work on metricsport, prometheus and grafana. From a quick glance > through htop sorting by memory and restarting the 7 relays at the top, ~4GB > of RAM frees up (3 in RAM and 1 in Swap) per relay. > > 66 Tor relays (all middle/guard). All less than 40 days old. I have 2x10G: 80 instances (40 guards/40 bridges) using 130G RAM Welcome to the Internet, I suspect DDoS. Do you have ip/nftables? With routed IPs I have no conntrack / no table filter https://paste.systemli.org/?075c3c469bfefa4b#AKMZk5mDeWPj6f7S3JEDz5cLuBYhMeDjJWsMJ4vHKGP9 On other 1G servers I have dynamic NFT rules. Search for information about YOUR 10G network card driver. e.g: pre-up /sbin/ethtool commands may be required. > General Setup: > EPYC 7702P. > > 256GB RAM > > Software Versions: > Tor version 0.4.8.14 > Ubuntu OS 24.04.2 > > Tor configuration: > SOCKSPort 0 SocksPolicy reject * ^ # I'm paranoid ;-) and I don't need ControlPort ControlPort 0 > ControlPort xxx.xxx.xxx.xxx:xxx > HashedControlPassword xxx > > ORPort xxx.xxx.xxx.xxx:xxxxx > Address xxx.xxx.xxx.xxx > OutboundBindAddress xxx.xxx.xxx.xxx RelayBandwidthRate 100 MBytes RelayBandwidthBurst 200 MBytes > Nickname ... > > ContactInfo ... > > MyFamily... > ExitPolicy reject *:* Only a hint if you have several dozen relays, you can use one file. eg: ## Include MyFamily & ContactInfo %include /etc/tor/torrc.all ## Include Exit Policy %include /etc/tor/torrc.exit > On Tuesday, March 18th, 2025 at 3:44 AM, mail--- via tor-relays <tor-relays@xxxxxxxxxxxxxxxxxxxx> wrote: > > Hi, > > > > To be honest I think something might be wrong. Maybe some memory leak or > > another issue because 66 relays with such low bandwidth shouldn't even > > come close to a memory footprint of 256 GB. We use different operating > > systems and most likely also slightly different relay configurations, but > > our relays use *significantly* less memory. You shouldn't need more than > > 128 GB of memory for ~10 Gb/s of Tor traffic, although 256 GB is > > recommended for some headroom for attacks and spikes and such. > > > > Could you share your general setup, software versions and Tor > > configuration? Perhaps someone on this mailinglist will be able to help > > you. Also using node_exporter and MetricsPort (for example with a few > > Grafana dashboards) would probably yield valuable information about this > > excessive memory footprint. For example: if the memory footprint > > increases linearly over time, it might be some software memory leak that > > requires a fix. > > > > Cheers, > > > > tornth > > > > Mar 18, 2025, 10:21 by tor-relays@xxxxxxxxxxxxxxxxxxxx: > >> Somewhat of a surprise based on the 2-4x RAM to core/threads/relay ratio > >> in this email thread, ran out of 256GB RAM with 66 Tor relays (roughly > >> ~4x ratio). > >> > >> Something misconfigured or this expected as part of the relay ramping up > >> behavior or just regular relay behavior? > >> > >> Summary: 64 cores / 128 threads (EPYC 7702P) running 66 Tor relays (all > >> middle/guard), Tor version 0.4.8.14, used 256GB RAM and went to swap, on > >> a 10 Gbps unmetered connection. > >> > >> Details: > >> > >> - ~30 relays are 30 days old and within 24 hours of adding ~30 new > >> relays, used up 256GB RAM. > >> > >> - Average Advertised Bandwidth is ~3 MiB/s per 33 relays and the other 33 > >> are unmeasured / not listed advertised bandwidth yet. > >> > >> - Swap was set at default 8GB and maxed out. Changed to 256G temporarily. > >> Swap usage is slowly climbing and reached 20G within first hour of > >> increasing size. > >> > >> - Ubuntu 24.04.2 LTS default server install. > >> > >> - Nothing else running. > >> > >> - Will upgrade RAM this server to 768GB within next few days > >> > >> Have a different server, with 88 cores/threads and ~88 relays, all less > >> than 30 days old, hovering around 240GB RAM, same average advertised > >> bandwidth of ~3 MiB/s per half the relays, but have already upgraded RAM > >> to 384GB and plan to take it to 512GB within the next week or two. Same > >> Ubuntu and Tor versions and software configuration. > >> > >> Will keep sharing back as more relays and servers ramp up traffic. > >> > >>> On Tuesday, February 18th, 2025 at 11:23 PM, mail--- via tor-relays <tor-relays@xxxxxxxxxxxxxxxxxxxx> wrote: > >>>> Hi, > >>>> > >>>> Many people already replied, but here are my (late) two cents. > >>>> > >>>>> 1) If a full IPv4 /24 Class C was available to host Tor relays, what > >>>>> are some optimal ways to allocate bandwidth, CPU cores and RAM to > >>>>> maximize utilization of the IPv4 /24 for Tor?>>>> > >>>> "Optimal" depends on your preferences and goals. Some examples: > >>>> > >>>> - IP address efficiency: run 8 relays per IPv4 address. > >>>> - Use the best ports: 256 relays (443) or 512 relays (443+80). > >>>> - Lowest kernel/system congestion: 1 locked relay per core/SMT thread > >>>> combination, ideally on high clocked CPUs. - Easiest to manage: as few > >>>> relays as possible. > >>>> - Memory efficiency: only run middle relays on very high clocked CPUs > >>>> (4-5 Ghz). - Cost efficiency: run many relays on 1-2 generations old > >>>> Epyc CPUs with a high core count (64 or more). > >>>> > >>>> There are always constraints. The hardware/CPU/memory and > >>>> bandwidth/routing capability available to you are probably not > >>>> infinite. Also the Tor Project maximizes bandwidth contributions to > >>>> 20% and 10% for exit relay and overall consensus weight respectively. > >>>> > >>>> With 256 IP addresses on modern hardware, it will be very hard to not > >>>> run in to one of these limitations long before you can make it > >>>> 'optimal'. Hardware wise, one modern/current gen high performance > >>>> server only running exit relays will easily push enough Tor traffic to > >>>> do more than half of the total exit bandwidth of the Tor network. > >>>> > >>>> My advice would be: > >>>> 1) Get the fastest/best hardware with current-ish generation CPU IPC > >>>> capabilities you can get within your budget. To lower complexity with > >>>> keeping congestion in control, one socket is easier to deal with than > >>>> a dual socket system. > >>>> > >>>> (tip for NIC: if your switch/router has 10 Gb/s or 25 Gb/s ports, get > >>>> some of the older Mellanox cards. They are very stable (more so than > >>>> their Intel counterparts in my experience) and extremely affordable > >>>> nowadays because of all the organizations that throw away their > >>>> digital sovereignty and privacy of their employees/users to move to > >>>> the cloud). > >>>> > >>>> 3) Start with 1 Tor relay per physical core (ignoring SMT). When the > >>>> Tor relays have ramped up (this takes 2-3 months for guard relays) and > >>>> there still is considerable headroom on the CPU (Tor runs extremely > >>>> poorly at scale sadly, so this would be my expectation) then move to 1 > >>>> Tor relay per thread (SMT included). > >>>> > >>>> (tip: already run/'train' some Tor relays with a very limited bandwidth > >>>> (2 MB/s or something) parallel to your primary ones and pin them all > >>>> to 1-2 cores to let them ramp up in parallel to your primary ones. > >>>> This makes it *much* less cumbersome to scale up your Tor contribution > >>>> when you need/want/can do that in the future). > >>>> > >>>> 4) Assume at least 1 GB of RAM per relay on modern CPUs + 32 GB > >>>> additionally for OS, DNS, networking and to have some headroom for DoS > >>>> attacks. This may sound high, especially considering the advice in the > >>>> Tor documentation. But on modern CPUs (especially with a high > >>>> clockspeed) guard relays can use a lot more than 512 MB of RAM, > >>>> especially when they are getting attacked. Middle and exit relays > >>>> require less RAM. > >>>> > >>>> Don't skimp out on system memory capacity. DDR4 RDIMMs with decent > >>>> clockspeeds are so cheap nowadays. For reference: we ran our smaller > >>>> Tor servers (16C@3.4Ghz) with 64 GB of RAM and had to upgrade it to > >>>> 128 GB because during attacks RAM usage exceeded the amount available > >>>> and killed processes. > >>>> > >>>> 5) If you have the IP space available, use one IPv4 address per relay > >>>> and use all the good ports such as 443. If IP addresses are more > >>>> scarce, it's also not bad to run 4 or 8 relays per IP address. > >>>> Especially for middle and exit relays the port doesn't matter (much). > >>>> Guard relays should ideally always run on a generally used (and > >>>> generally unblocked) port.>>>> > >>>>> 2) If a full 10 Gbps connection was available for Tor relays, how many > >>>>> CPU cores, RAM and IPv4 addresses would be required to saturate the > >>>>> 10 Gbps connection?>>>> > >>>> That greatly depends on the CPU and your configuration. I can offer 3 > >>>> references based on real world examples. They all run a mix of > >>>> guard/middle/exit relays. > >>>> > >>>> 1) Typical low core count (16+SMT) with higher clockspeed (3.4 Ghz) > >>>> saturates a 10 Gb/s connection with ~18.5 physical cores + SMT. 2) > >>>> Typical higher core count (64+SMT) with lower clockspeed (2.25 Ghz) > >>>> saturates a 10 Gb/s connection with ~31.5 physical cores + SMT. 3) > >>>> Typical energy efficient/low performance CPU with low core count (16) > >>>> with very low clockspeed (2.0 Ghz) used often in networking appliances > >>>> saturates a 10 Gb/s connection with ~75 physical cores (note: no SMT). > >>>> > >>>> The amount of IP addresses required also depends on multiple factors. > >>>> But I'd say that you would need between the amount and double the > >>>> amount of relays of the mentioned core+SMT count in order to saturate > >>>> 10 Gb/s. This would be 37-74, 63-126 and 75-150 relays respectively. > >>>> So between 5 and 19 IPv4 addresses would be required at minimum, > >>>> depending on CPU performance level. > >>>> > >>>> RAM wise the more relays you run, the more RAM overhead you will have. > >>>> So in general it's better to run less relays at a higher speed each > >>>> than run many at a low clock speed. But since Tor scales so badly you > >>>> need more Relays anyway so optimizing this isn't easy in practice.>>>> > >>>>> 3) Same for a 20 Gbps connection, how many CPU cores, RAM and IPv4 > >>>>> addresses are required to saturate?>>>> > >>>> Double the amount compared to 10 Gb/s. > >>>> > >>>> Good luck with your Tor adventure. And let us know your findings with > >>>> achieving 10 Gb/s when you get there :-). > >>>> > >>>> Cheers, > >>>> > >>>> tornth > >>>> > >>>> Feb 3, 2025, 18:14 by tor-relays@xxxxxxxxxxxxxxxxxxxx: > >>>>> Hi All, > >>>>> > >>>>> Looking for guidance around running high performance Tor relays on > >>>>> Ubuntu. > >>>>> > >>>>> Few questions: > >>>>> 1) If a full IPv4 /24 Class C was available to host Tor relays, what > >>>>> are some optimal ways to allocate bandwidth, CPU cores and RAM to > >>>>> maximize utilization of the IPv4 /24 for Tor? > >>>>> > >>>>> 2) If a full 10 Gbps connection was available for Tor relays, how many > >>>>> CPU cores, RAM and IPv4 addresses would be required to saturate the > >>>>> 10 Gbps connection? > >>>>> > >>>>> 3) Same for a 20 Gbps connection, how many CPU cores, RAM and IPv4 > >>>>> addresses are required to saturate? > >>>>> > >>>>> Thanks! > >>>>> > >>>>> Sent with [Proton Mail](https://proton.me/mail/home) secure email. -- ╰_╯ Ciao Marco! Debian GNU/Linux It's free software and it gives you freedom!
Attachment:
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ tor-relays mailing list -- tor-relays@xxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to tor-relays-leave@xxxxxxxxxxxxxxxxxxxx