Hi forest, and everyone,
I would like to follow up on my earlier post about the Tor BEGIN_DIR Storm Sentinel with observations from another severe BEGIN_DIR storm affecting two of my non-exit ARM relays on October 12–13.
First, forest, you were right to raise the issue of MetricsPort responsiveness.
Our Sentinel polls Tor’s MetricsPort with a four-second HTTP timeout. During both storms, many requests timed out, leaving substantial gaps in our BEGIN_DIR measurements. In other words, our monitoring was partially blinded precisely when we needed it most.
In the eight-hour observation windows examined shortly after the storms:
These percentages refer to collected samples over the entire eight-hour windows, not just the active storms. Many failures were caused by MetricsPort timeouts. Consequently, our recorded maximum BEGIN_DIR rates should be regarded as the highest successfully observed values, not necessarily the actual peaks.
Incident timeline (all times UTC):
| Event | Cloud2 / Hetzner | Cloud3 / Oracle |
|---|---|---|
| HSDir first observed (0→1) | Oct 12, 21:23:20 | Oct 12, 23:12:45 |
| First WATCH event | Oct 12, 23:09:20 | Oct 12, 23:20:45 |
| STORM_START | Oct 13, 00:01:24 | Oct 12, 23:20:55 |
| HSDir loss observed (1→0) | Oct 13, 01:22:30 | Oct 13, 01:22:25 |
| STORM_END | Oct 13, 02:40:50 | Oct 13, 02:43:15 |
| Guard recovery observed | Oct 13, 03:05:20 | Oct 13, 03:13:15 |
These timestamps are based on Sentinel observations. Flag transitions indicate the first successful sample showing the new value, rather than the exact moment of a directory consensus change.
Cloud3 entered STORM just 8 minutes and 10 seconds after HSDir was first observed. Cloud2 entered STORM approximately 2 hours and 38 minutes after acquiring HSDir, with several smaller WATCH episodes beforehand.
Particularly striking is the near-simultaneous observed loss of HSDir on both relays, only five seconds apart, despite their different HSDir acquisition and storm onset times.
Measurements during the storms
Cloud2 — Hetzner ARM:
Cloud3 — Oracle Ampere ARM:
These dropped-handshake figures come from Tor’s own overload notices, rather than from Sentinel. They should not be interpreted as exact totals for the entire storm.
The substantial outbound traffic asymmetry, together with Tor’s circuit-creation overload warnings and dropped handshake requests, indicates severe resource pressure on both relays.
Relay status and recovery
Both affected relays were running Tor 0.4.9.14. Neither Tor process restarted during the incident, as confirmed by unchanged PIDs in our Sentinel records and the uptime reported in Tor’s notice logs.
Cloud2 received a manually initiated configuration reload at 01:22:13 UTC, following a reduction of its bandwidth limits. Cloud3 had no corresponding reload at that time in its notice logs. Interestingly, both relays were first observed losing their HSDir and Guard flags within seconds of each other, despite their Tor processes remaining alive.
Both relays subsequently recovered their Guard flags — Cloud2 at 03:05:20 UTC and Cloud3 at 03:13:15 UTC — while HSDir remained absent. BEGIN_DIR activity eventually returned to normal on both machines.
Our third relay, a smaller Oracle AMD instance without the HSDir flag, remained in NORMAL state throughout the same observation period, with no comparable BEGIN_DIR storm.
An additional observation: StaleDesc
Later on October 13, I noticed that both affected relays had acquired the StaleDesc flag while still retaining Fast, Guard, Running, Stable, V2Dir, and Valid.
Interestingly, the archived 13:00 UTC network consensus listed server descriptors with publication timestamps of 01:22:19 UTC for Cloud2 and 01:22:27 UTC for Cloud3 — only eight seconds apart, and very close to the observed loss of HSDir on both relays.
As I understand it, StaleDesc indicates that the directory authorities consider a relay’s published server descriptor too old (normally more than 18 hours). If no newer descriptors were accepted, those timestamps would explain the appearance of StaleDesc later that evening.
We have not yet established why newer descriptors were not reflected in the consensus, whether the same thing happened during previous storms, or whether this is related to the overload.
Has anyone else observed StaleDesc appearing after a severe BEGIN_DIR storm, particularly without a Tor process restart?
Monitoring limitations and possible improvements
Our current Sentinel relies on successful Tor MetricsPort responses to measure BEGIN_DIR activity and detect storms. Although it also collects Tor process information directly from Linux /proc, these measurements are tied to the same sampling loop.
During prolonged MetricsPort timeouts, we lose the ability to observe the actual BEGIN_DIR rate. When MetricsPort eventually responds again, the calculated rate represents an average over the elapsed interval, potentially hiding much higher short-lived peaks.
We have also identified a CPU calculation issue following failed MetricsPort scrapes, which may underestimate CPU utilization during recovery.
We are therefore considering adding an independent, lightweight Linux monitoring component that would continue collecting:
The goal would be to preserve useful host-level measurements even when Tor’s MetricsPort becomes temporarily unresponsive, without interfering with Tor or collecting user-identifying traffic information.
This is currently only a design proposal; no changes have been deployed.
A few questions for forest and the Tor developers
MetricsPort responsiveness: Have other relay operators experienced similar prolonged MetricsPort timeouts during BEGIN_DIR storms? Is this a known consequence of Tor’s main event loop becoming overloaded?
HSDir and consensus behavior: Could the near-simultaneous loss of HSDir and Guard on two separately hosted relays be explained by the way directory authorities evaluate overloaded relays? Have others observed similar timing?
StaleDesc: Has anyone observed the StaleDesc flag appearing after severe BEGIN_DIR storms, particularly when the Tor process itself has not restarted?
forest’s experimental patch: I noticed that both of my non-exit relays report [DoSStreamCreationEnabled disabled] in their heartbeat logs. Forest, when you tested your patch moving BEGIN_DIR handling behind the existing stream-creation rate limiter, was DoSStreamCreationEnabled actually active on the test relay? Could this explain why the patch appeared to have no effect, or is there another issue with the per-circuit limiting mechanism?
I would appreciate any technical insights, particularly regarding ways to observe these events more reliably without interfering with Tor’s normal operation.
I can share additional Sentinel measurements and relevant Tor notice-log excerpts if they would be useful.
Regards,
Khan Krum
Tor BEGIN_DIR Storm Sentinel:
https://github.com/YankoNach/tor-begindir-sentinel
On Sun, 4 Oct 2026 23:57:11 +0300, Khan Krum khankrum359@xxxxxxxxx wrote:
Hi,
I run three hobby relays and I seem to be seeing the same behavior on
one of them. I compared two similar ARM, 2-core, non-exit Guard+HSDir
relays, both running Tor 0.4.9.13.On the affected relay I measured 41,240 BEGIN_DIR streams in 61 seconds
(~676/s). The Tor main thread reaches about 100% CPU, ntor onionskins
are being dropped, and traffic is strongly asymmetric — roughly 2.2 MB/s
read vs 7.1 MB/s written.On my other Guard+HSDir relay, measured at roughly the same time, I saw
only 148 BEGIN_DIR streams in 60 seconds (~2.5/s). That relay carries
much more traffic (~15 MB/s in both directions) and appears healthy.So in my case, simply having the HSDir flag does not seem sufficient to
trigger the issue. One relay is affected, while another Guard+HSDir
relay is not.I’m only a hobby relay operator, so I’m posting these numbers mainly in
case they are useful for comparison.Regards,
KhanKrum
_______________________________________________ tor-relays mailing list -- tor-relays@xxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to tor-relays-leave@xxxxxxxxxxxxxxxxxxxx