[tor-relays] Re: Ebury 1.8.3 on two Quetzalcoatl relay servers: timeline, lessons, open questions

["Jonathan D. Proulx via tor-relays" <tor-relays@xxxxxxxxxxxxxxxxxxxx>]

Hi Zimmer

Thanks for the excellent write up.

We got hit by Ebury more than a decade ago now (on our research
infrastructure not tor relays) and it still stings, good luck with
your clean up. We'll definitely review our detection rules based on
your updated details.

One of our key remediations which you didn't mention was adding
multifactor authenitcation (MFA) to our ssh, so even if a password or
key leaks that's no longer sufficient.

Nothing is perfect but MFA seems like a substantial mitigation to the
typical Ebury vector.

-Jon

-- 
Jonathan Proulx (he/him)
Sr. Technical Architect
The Infrastructure Group
MIT CSAIL

On Wed, Oct 07, 2026 at 01:29:42PM +0100, Dr Remmiz via tor-relays wrote:
:Hi all,
:
:Following hiro's "Security incident affecting some relay operators" post on
:Oct 5, here is what we found on our own infrastructure, in case it helps
:others check theirs. Two of our servers were rooted with the Ebury OpenSSH
:backdoor (self-reported version 1.8.3), and we use these two names for them
:throughout:
:
:- the management server: our Contabo VPS, which was our WireGuard VPN
:gateway and SSH jump host and also ran an obfs4 bridge;
:- Switzerland2: a FranTech VPS running two of our Quetzalcoatl exit relays
:(29FEFE36A5F66A6C93D24775B9D2364A3831B597 and
:8427937D5A39E15699C850F26FED3CD59C379C48).
:
:When we examined them, neither server showed the NAT redirect, new users or
:SSH keys, cron jobs, systemd units or modified tor binary from the
:advisory's checklist. Both were examined read-only through several
:independent AI-assisted review passes, cross-checked against logs, process
:memory and public Tor Metrics data. All times are UTC.
:
:One caveat up front: the attacker truncated some logs on the management
:server, timestomped the files they planted, and turned off sshd's login
:logging on Switzerland2, and Ebury's backdoor logins leave no login records
:or shell history at all. Some details are therefore missing, and parts of
:this timeline are inferred from what was left; we say so where it matters.
:
:
:1. TIMELINE
:
:- Background (management server): SSH password authentication was enabled,
:and an encrypted backup of our admin workstation was stored on the
:management server. That backup held our operator SSH private key, our
:ansible inventory and copies of our relays' offline Ed25519 master keys.
:
:- Background (both servers): both trusted the same operator SSH key, and
:the operator account's password was also its sudo password, sat in
:cleartext in a helper script on both servers, and was the backup's
:passphrase. We don't know which of our other relays had that script.
:
:- Aug 28, 03:19: the attacker logged in to the management server from
:185.225.226.62 (VikHost, AS207560) with the operator account's password on
:the first try, with no failed attempt from that IP ever recorded, so they
:almost certainly knew the password in advance. By 03:24 they had root via
:sudo su.
:
:- Aug 28, 03:25-03:43: the attacker installed Ebury on the management
:server, truncated some logs and installed gcc. They then tried apt inside
:our WireGuard dashboard (WGDashboard) container, which runs on Alpine Linux
:and only has Alpine's apk package manager; when apt failed, an SSH client
:and nmap appeared in the container within minutes (dated by file change
:times).
:
:- Aug 28, 04:33-04:37: the same IP opened two SSH connections to
:Switzerland2 as the operator account and disconnected before
:authenticating; Switzerland2 only accepted SSH keys.
:
:- Aug 28, 04:38: Switzerland2 accepted a login for the operator account
:from the management server's IP address (also our VPN exit), using an RSA
:key that was in no authorized_keys file we found on Switzerland2 (how it
:was accepted is still open; see section 3). One second later, sudo bash -c
:'echo <base64> | base64 -d | perl' installed the same Ebury build on
:Switzerland2.
:
:- The management server's logs show no SSH login or sudo at 04:38, and the
:connection's source port fits the management server's own port range rather
:than our VPN clients', so the attacker most likely made this hop through
:the management server's backdoor.
:
:- Aug 31 - Sep 6 (management server): root actions with no sudo record
:behind them installed nmap on the management server itself on Aug 31. On
:Sep 6, starting while no one was logged in, more such root actions pulled
:an alpine image, loaded NFS client kernel modules, installed duplicity and
:decrypted the workstation backup's metadata (whether the backup itself was
:copied out is open).
:
:- Sep 30 (Switzerland2): the attacker restarted sshd from inside the
:backdoor, after which sshd stopped writing its "Accepted ..." lines, the
:ones that record the user, key and source address of each login, because
:its LogLevel was ERROR. The PAM "session opened" lines kept appearing, but
:they record neither key nor source address.
:
:- Sep 30 - Oct 1 (Switzerland2): the attacker installed curl and
:wireguard-tools outside sudo, loaded the WireGuard kernel module, and
:started hidden processes named guard, then failover and snf, none of which
:left binaries on disk (what they did is open).
:
:- Oct 1, 15:35-15:43 (Switzerland2): with no SSH login, sudo command or
:cron job in the logs, the attacker prepended these lines to /etc/tor/torrc,
:sent Tor SIGHUPs and then restarted it:
:    # ==== OUR_POLICY_BEGIN 2026-10-01 ====
:    ExitPolicy reject *:25
:    ExitPolicy reject *:465
:    ExitPolicy reject *:587
:    ExitPolicy reject *:110
:    ExitPolicy reject *:143
:    ExitPolicy reject *:993
:    ExitPolicy reject *:995
:    ExitPolicy reject *:3389
:    ExitPolicy reject *:135
:    ExitPolicy reject *:137-139
:    ExitPolicy reject *:445
:    ExitPolicy accept *:*
:    # ==== OUR_POLICY_END ====
:  That file is unused on Switzerland2: the two relays run as Debian tor@
:instances that read /etc/tor/instances/<instance>/torrc, and the only unit
:that reads /etc/tor/torrc (tor@default) was masked. The published policy
:did not change.
:
:- Oct 2, 09:34 (Switzerland2): again with no SSH login, sudo command or
:cron job in the logs, both instance torrcs were rewritten. Our
:ansible-generated accept-list (60 accepted port ranges and a final reject
:*:*) was removed and five lines were appended after "# end of torrc":
:ExitPolicy reject *:25, *:465, *:587, *:993 and *:995.
:
:- Oct 2, 09:34:04: both Switzerland2 relays published the result, Tor's
:default rejects plus those five ports and then accept *:* (confirmed in the
:public CollecTor archive). At 09:42 the attacker tidied up: they emptied
:the OUR_POLICY block in /etc/tor/torrc, appended the same five reject lines
:to that file, and saved copies of all three files as *.bak20261002.
:
:- Our reading of Oct 1-2, a hypothesis the logs can't prove: the attacker
:assumed Tor reads the default /etc/tor/torrc, saw that reloads and a full
:restart didn't change the published policy, and came back the next day to
:edit the instance files. The policy they tried first also differed: it
:blocked plaintext POP3/IMAP and RDP, while the one that went live opened
:those and blocked the TLS mail ports instead.
:
:- Oct 4, 05:46 (Switzerland2): a routine kernel upgrade and reboot killed
:the hidden tools, but Ebury came straight back.
:
:
: Oct 4, 13:56 (Switzerland2): after the Tor Project's network health team
:spotted the exit-policy change and contacted us, we confirmed we hadn't
:made it, asked for the relays to be marked bad and stopped Tor on
:Switzerland2. The open policy had been live for about 52 hours on two
:Guard+Exit relays moving roughly 3.4 TiB a day.
:
:- Oct 5, 08:56-10:26: relays on 55 of our family's hosts were restarted in
:one scripted sweep, in text-sorted IP order (who ran it is open). On
:Switzerland2 that restart came from inside the backdoor, the directory
:authorities rejected both relays' descriptor uploads (HTTP 400), and we
:stopped Tor again.
:
:- Oct 6: Onionoo's family query listed 168 of our relays on 57 IPs, all
:down and all advertising that same near-open policy. Both servers are now
:offline and deleted, and all of our relays and our bridge are retired and
:blocklisted.
:
:
:2. WHAT WE LEARNED
:
:- The root cause was ours: one password reused for SSH login, sudo and the
:backup passphrase, stored in plaintext on both servers, plus SSH password
:authentication left enabled on the internet-facing management server. The
:attacker already had the password when they first connected.
:
:- The management server was a single point of failure. From it the attacker
:reached Switzerland2 from an IP address that blended in with our
:automation, every device on our WireGuard VPN was in reach, and Ebury also
:hooks the SSH client on the management server.
:
:- This Ebury is one unowned, setuid file,
:/usr/lib/x86_64-linux-gnu/libkeyutils.so.1.10.2 (63,368 bytes,
:timestomped), with the libkeyutils.so.1 symlink re-pointed at it. The
:packaged library is untouched, so package verification (dpkg -V, debsums)
:won't catch it, and sshd, sudo and PAM matched their package checksums on
:both servers.
:
:- On Debian 13 (Switzerland2) it is loaded by sshd-session through libkrb5,
:so ldd /usr/sbin/sshd and the sshd listener process look clean; from there
:it spreads into every process in an SSH session, sudo included.
:
:- Infected processes are lied to (a shell can't see the library in its own
:/proc/self/maps). The credential collector runs as a fake /sbin/rsyslogd
:(earlier as systemd-udevd or arpd) on abstract socket @event-0YsOG9WSOE,
:inside ssh.service's cgroup on Debian (Switzerland2) and, after a reboot,
:inside init.scope on Ubuntu 24.04 (the management server), where
:sshd-socket-generator starts it at boot.
:
:- Backdoor logins bypass PAM, logging and shell history; the implant's
:config includes UsePAM no, LogLevel QUIET, HISTFILE=/dev/null,
:AuthorizedKeysFile /proc/self/environ and PermitTunnel yes. The main traces
:were root actions no logged-in user could have made: apt history entries
:without "Requested-By:", su sessions opened by uid 0, systemctl calls from
:the ssh.service cgroup, Tor reloads nobody had asked for, and systemd's
:"remains running after unit stopped" when sshd restarted.
:
:- SSH encrypts passwords in transit, but sshd has to decrypt them to check
:them, and Ebury runs inside sshd. So it captures every login password
:exactly as typed, failed attempts included, notes which key logged in from
:where for key logins, and it captured the password our investigators used
:from every address they logged in from.
:
:
:- Version 1.8.3 is newer than ESET's public IOCs (last updated May 2024, up
:to 1.8.2), its socket name isn't on their list, and ESET's detect_ebury.sh
:would not flag it, although their YARA rule libkeyutils_with_ctor still
:matches. Each install stamps an 8-byte victim ID into the file, so the
:implant's hash differs on every server.
:
:- The policy flip opened plaintext POP3/IMAP, telnet, IRC, SSH and RDP
:while closing IMAPS/POP3S, so mail retrieval through the Switzerland2
:relays could only happen in cleartext. A hidden process named snf was
:running on Switzerland2 for most of that window.
:
:- It looks like the attacker went well beyond these two servers: every
:relay Onionoo listed in our family on Oct 6 advertised the same near-open
:policy, and the Oct 5 restart sweep covered 55 of our hosts. The management
:server and Switzerland2 shared an operator key and password; we didn't
:examine our other hosts, so we can't say whether they did too.
:
:- Some of our review passes got it wrong. One concluded there was no
:intrusion (it dismissed the setuid library as normal, trusted a clean dpkg
:-V and grouped logins by IP instead of key), and others concluded the
:policy change never reached the published descriptors; the public CollecTor
:descriptor archive and reads of process memory settled it.
:
:- Our "offline" master keys were not offline: copies sat in the workstation
:backup on the management server, encrypted with the password the attacker
:already had.
:
:- systemctl disable tor is not enough on Debian multi-instance relays,
:because instance units can be enabled on their own (on Switzerland2 they
:were, and the attacker added another boot link); mask the tor@ units, and
:since a live backdoor can undo that, take the affected server offline too.
:
:- On Switzerland2, sudo logged the entire base64 installer across 67
:"(command continued)" journal records, so we could decode it. On the
:management server the attacker ran sudo su first, and nothing of the
:install command was logged.
:
:
:3. OPEN QUESTIONS
:
:- Where did the attacker get the operator password before Aug 28? Not by
:brute force, and not through the WGDashboard on the management server (its
:web UI had been failing to start since April); the candidates are another
:of our relays, the admin workstation, or reuse somewhere we haven't found.
:
:- How did Switzerland2's sshd accept a key that was in no authorized_keys
:file, one second before Ebury was installed on Switzerland2? ~/.ssh and
:sshd_config.d on Switzerland2 never changed, which leaves a line in the
:main sshd_config (replaced on Sep 30, so we can't check it) or an earlier
:backdoor.
:
:- What did guard, failover and snf do on Switzerland2? Their binaries died
:with the Oct 4 reboot, so we don't know whether the newly opened port 22
:was used to intercept SSH or whether a WireGuard tunnel was built (udev
:logged two network-interface events on Sep 30, but no interface or config
:survived).
:
:
:- Where do the stolen credentials go? ESET's research on earlier versions
:describes operators pulling them through the backdoor, with an automatic
:encrypted send over UDP port 53 if nobody collects them for two weeks. On
:our servers, memory holds an https://%s/xpost.php template but no host, and
:a sandboxed run made no outbound connection, but the implant checks for
:packet sniffers and strace, so that test doesn't settle it.
:
:- Which of our other relays had their exit policy changed, when, and were
:any non-exits turned into exits? We are pulling the public CollecTor
:consensus and descriptor history for the whole family, back to May, and
:will follow up.
:
:- How many of the other 56 hosts in our family carry the implant, and was
:any infected before Aug 28 03:25 (an earlier implant birth time would mean
:an earlier victim)?
:
:- Who ran the Oct 5 restart sweep, and from where: the management server's
:backdoor, the admin workstation, or relay to relay? The logs on the
:management server and Switzerland2 can't tell these apart, and we haven't
:examined the workstation yet.
:
:- Did the 5.3 GB workstation backup leave the management server? Its
:metadata was decrypted on the management server on Sep 6 and NFS client
:modules were loaded that night, but nothing in the logs shows a transfer.
:
:- Was there a NAT redirect on Switzerland2 before the Oct 4 reboot?
:iptables was installed outside sudo on Sep 2 (we don't know by whom), and
:unsaved rules would not survive a reboot.
:
:- Is this the same actor as the other cases behind hiro's notice (NAT
:redirects, a fake uppercase CRON)? If you have seen libkeyutils.so.1.10.2,
:@event-0YsOG9WSOE, 185.225.226.62 or the SSH keys listed below, please tell
:us or bad-relays@.
:
:
:4. BEST PRACTICES
:
:- Never put passwords in scripts, and never reuse one across SSH login,
:sudo and backups. Use key-only automation with narrowly scoped NOPASSWD
:sudo rules.
:
:- Use key-only SSH everywhere, especially on jump and management servers,
:and reach them only through a VPN or an IP allow-list. Check the effective
:config with sshd -T, because sshd takes the first value it finds (on
:Switzerland2, a PermitRootLogin yes left near the top by the VPS image
:silently beat the PermitRootLogin no our setup script appended).
:
:- Treat your management/VPN server as the crown jewels: nothing web-facing
:on it, no key backups on it, and no admin tooling in its containers. The
:WGDashboard 4.3.0.1 on our management server was exposed to the internet
:(its web UI happened to be broken since April), has an unauthenticated
:file-read bug (CVE-2026-44343, fixed in 4.3.2) and logs every sign-in
:password in cleartext.
:
:- Keep OfflineMasterKey material genuinely offline. If a machine that ever
:held it is compromised, retire those identities and roll a new family key
:and FamilyId.
:
:
:- Check for this Ebury from a rescue system or an offline disk image if you
:can. On a live server, log in with a key (never a password) and use ESET's
:clean-shell trick, H=1 LD_PRELOAD=":" LD_DEBUG=":" bash; a setuid
:libkeyutils file, a libkeyutils.so.1 symlink to a file no package owns, an
:@event- abstract socket, or a non-sshd process lingering in the ssh.service
:cgroup is a strong sign of infection:
:    find / -xdev -name 'libkeyutils*' -perm -4000 2>/dev/null
:    dpkg -S "$(readlink -f /usr/lib/x86_64-linux-gnu/libkeyutils.so.1)"
:(rpm -qf on RPM systems)
:    grep -E '@event-|@/dev/(event|stats)-' /proc/net/unix
:    systemctl status ssh | sed -n '/CGroup/,$p'   (on Ubuntu 24.04, check
:init.scope too)
:
:- Search your journal for apt transactions with no "Requested-By:"
:(unattended-upgrades also lacks it), su sessions opened by uid 0,
:ssh.service "remains running after unit stopped" or unusually large
:"Consumed ... CPU time" lines (Switzerland2's showed 41 minutes and 281
:MB), "Reload requested from client ... (unit ssh.service)", and Tor reloads
:nobody on your team caused. On Ubuntu, the install also leaves
:apparmor="DENIED" operation="bind" profile="rsyslogd" addr="@event-..." in
:the kernel log.
:
:- Look for sudo running "base64 -d | perl" (long commands are split across
:"(command continued)" records) or "echo 3 > /proc/sys/vm/drop_caches", a
:missing /etc/falco/rules.d/ebury.yaml if you run Falco, and logins from
:185.225.226.62, and review accepted logins by key fingerprint, not just
:source IP. Keys: RSA SHA256:dvACokMYcXJ2om4wuS+TvdtjrAgPK7rC7ZQHDgvBHzk
:(used to log in to Switzerland2) and the two embedded in the implant, RSA
:SHA256:KlICz/B59619CpuQdGzgmmvzWBfTMUJ77EwihTYobuo and ED25519
:SHA256:OfLY0uF9SwOW9xVjiMXZGDfk6Atmz6CbHy4epknLm8Y.
:
:- Monitor your relays from outside, as the Tor Project's network health
:team did for us. Alert when the exit policy, family or contact info in your
:published descriptors (Onionoo or CollecTor) differs from what you
:deployed, since files on a compromised server can be rewritten.
:
:- Ship logs off your servers, keep NTP on, alert if sshd's LogLevel drops
:below INFO, and watch for unowned or setuid files in library directories
:(AIDE or similar).
:
:- Never type a password into a server you suspect, because Ebury will
:capture it. Investigate with keys from a clean machine or from an offline
:image, and capture memory before rebooting or wiping.
:
:- If you find it, take the relays down and mask the tor units, snapshot the
:disk, and rebuild from scratch rather than cleaning in place. Then rotate
:every SSH key, password and WireGuard key, generate new relay identities,
:and report to bad-relays@xxxxxxxxxxxxxxxxxxxx.
:
:Thanks to the Tor Project's network health team for catching the change and
:acting quickly. We're happy to share more detail, including the de-fanged
:installer, with anyone investigating a similar case.
:
:Zimmer
:Quetzalcoatl relays

:_______________________________________________
:tor-relays mailing list -- tor-relays@xxxxxxxxxxxxxxxxxxxx
:To unsubscribe send an email to tor-relays-leave@xxxxxxxxxxxxxxxxxxxx

_______________________________________________
tor-relays mailing list -- tor-relays@xxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to tor-relays-leave@xxxxxxxxxxxxxxxxxxxx