Hi,
Marc-Etienne here, I've been looking at Ebury for quite some time (13 years now, yikes!). I'm the author of the latest (2024) report on Ebury and co-author of the original paper from 2014.
First, I'd like to thank you for coming forward about the compromise and detailing what you found. **A lot** of servers have been compromised by Ebury in the past but most victims, for good or bad reasons, decided _not_ to disclose they got comprised. I think if we actually work together, we can better remediate threats like Ebury.
I'd be happy to assist in any way. Here are some partial answers to your questions:
> Where did the attacker get the operator password before Aug 28?
That I couldn't say for sure. My best guess is a login from a host already compromised by Ebury.
Another possibility is a compromised hosting provider. We identified a couple of them in the past where their management servers were compromised by Ebury and their customers' VPSes, containers, etc., ended up running Ebury as well.
> How did Switzerland2's sshd accept a key that was in no authorized_keys
I'm really not sure, but maybe Ebury was already present before and was updated on Sept 30? But then if it was Ebury letting them in, that log line wouldn't even be there :-/
> What did guard, failover and snf do on Switzerland2?
I couldn't say. snf may stand for "sniff", so maybe some sort of traffic sniffing on the egress of the exit node?
> Where do the stolen credentials go?
They are kept in memory in a process spawn by Ebury until they are fetched using Ebury's "Xcat" command. There are daily SSH connections from the attacker infrastructure pulling them. After 14 days without Xcat, it will send them via UDP/53 to some exfil server. The "xpost.php" is exclusively for stuff captured from HTTP POST requests made by libcurl (Apache, PHP, etc). This additional module is not activated on non-HTTP servers.
> On Switzerland2, sudo logged the entire base64 installer across 64 "(command continued)"
Any chance you could share this publicly or privately?
> If you find it, take the relays down and mask the tor units, snapshot the disk, and rebuild from scratch rather than cleaning in place.
+1 on this. I will also add: Ebury will spider everywhere. Look at your .ssh/known_hosts file to see where this system connected to before. They certainly did, using every username password and key combination possible.
Also please try to capture anything that could help us better understand this group's impact and intent.
A few notes and advice for any other relay operators in the same situation:
- Defusing the userland rootkit currently works with "I=1 bash" too. They replaced H with I after our 2024 publication. Now that I say this, next Ebury version will likely use J :P
- From your description it seems it is magling with network settings. The Ebury operators are known to play with them for capturing and redirecting traffic. Consider dumping netfilter tables (iptables-save, iptables-nft-save, nft list ruleset) as well routing tables and rules (ip link show, ip addr show, ip rule show, ip route show table all, ip tunnel show).
- If a process is running but the executable file is missing from disk, procfs will happily extract it from memory for you. Find the PID, and use `cp /proc/$PID/exe suspicious.elf` to dump the original file into suspicious.elf. gcore (just `gcore $PID`), part of gdb, can also be used to dump non-filebacked memory (heap, stack, rw mem). This could be helpful to extract configuration from running processes as well as reveil some details about the context of the execution (example: the SSH_CONNECTION envvar).
That's what I can think of off the top of my head for now. Given how many servers had Ebury between 2009 and 2023 (400 000!) there are most likely other compromised Tor relays out there. I wouldn't be surprised Ebury operators automated the process of installing "special tools" on Tor relay specifically, maybe to capture more SSH credentials from users connecting to SSH servers via Tor.
Feel free to reach out to me for any questions or if you find something interesting that may help better understand what Ebury is up to now :)
Cheers!
M-E