[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-relays] Relay not connecting

Hi,


I didn't see your original email and I don't currently have it in my
inbox to reply to it directly but I can tell you by looking at your
metrics history for "Alberta", indeed you had the same problem on Dec.18
that happened to my relays.


The attackers seem to be picking a group of relays at a time and move on
to others and it may be a while before they come back again. However the
problem gets amplified if you're running an exit relay without having
large enough resources to deal with all this traffic.


I'm not running an exit relay, so I have to deal with the problem for a
few hours, but all this outgoing traffic from my relay and other relays
that are under attack will inevitably end up at exit relays. This means
that even if they move on from my relay to another one, the traffic will
still continue to poor into exit relays, so the duration of the attack
for an exit relay may be much longer.


This shouldn't stop you from bringing your rely back up. You're not
doing anything wrong and there's nothing wrong with your set up. If
you're running other services on the same server of course the heavy
load might disrupt those services as well, which is why it's always a
good idea to only run Tor on the server and not to mix it with other
services.


My suggestion for the time being, keep running my firewall scripts as
they do help making the duration of the attack shorter by putting some
of those IP addresses in the block list and help making it less painful
and see how it goes. And in the long run, if you find that you don't
have the resources or patience to deal with such heavy load, consider
running a middle/Guard relay instead of an Exit. They're much easier to
deal with during an attack. But don't quit. It'll pass.


Cheers.


On 1/15/2024 8:14 PM, denny.obreham@xxxxxxxxxxxxxxxxx wrote:
>
> Sorry, but I'm going to vent a little bit. I'm not a network
> specialist, so 11 days ago I sent the following email to this mailing
> list asking for help because two of my Tor exit relays were completely
> frozen and I was unable to put them online again.
>
>
> Nobody answered, not even a comment. No wait, there was one person -
> very active on this mailing list recently - who sent me an email
> personally to tell me that I was an idiot (referring to what, I don't
> know) who should kill himself. Adding furthermore that if I didn't
> commit suicide within 72 hours, he would personally come to my house
> and kill me with a Glock 9 mm. Fun stuff, very disturbing.
>
>
> Anyway, no serious answers, someone calling m e an idiot: I tried to
> look as best as I could at what I did wrong. Couldn't find anything.
> My only available solution was to rebuild completely my server,
> something I wanted to do for a while because of other undesired quirks
> that were bothering me with my setup. I knew it would take a long time
> - which I didn't really have - but I finally finished my new setup
> yesterday. (Don't look for 25FC41154DCB2CAE3ABD74A8DFCD5B90D2CFFD57 or
> the bridge, they have been shut down for the moment.)
>
>
> Today, I read a line from Chris Endiku-6 saying: "There’s something
> going on for a while and I haven’t seen any mentions of it." The exact
> problem I mentioned! He says it goes "as early as Dec.23"; my problem
> goes to Dec 18 as shown in my previous email. Also, not mentioned in
> my previous email, before I renewed my setup, my tor-ddos firewall
> rules (I use the ones f rom Endiku-6) had blocked about 5 times more
> IP than usual - if that can be useful information to anyone.
>
>
> Sorry for the rent, I just needed to vent a little: There was
> something wrong, and I wasn't crazy or incompetent!
>
>
> I still would like to know how to restart such a relay, if this
> happens again in the future - other than reinstalling the entire
> server, that is.
>
>
> Thanks in advance.
>
>
> On 01/06/2024 01:52 PM denny.obreham@xxxxxxxxxxxxxxxxx wrote ..
>
>     I manage the two following exit relays: *
>     https://metrics.torproject.org/rs.html#details/25FC41154DCB2CAE3ABD74A8DFCD5B90D2CFFD57*
>     https://metrics.torproject.org/rs.html#details/3B85067588C3F017D5CCF7D8F65B5881B7D4C97C
>     They are both on the same IP and servers. A few hours ago they
>     lost contact even though I have both Apache and I2P servers on the
>     same machine that are reachable. The weird log messages seem to go
>     back to Dec 18, after a restart: ``` Dec 18 07:49:47 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bootstrapped 100% (done): Done Dec 18
>     07:54:18 a-n-o-n-y-m-e Tor-Alberta[904715]: Your computer is too
>     slow to handle this many circuit creation requests! Please
>     consider using the MaxAdvertisedBandwidth config option or
>     choosing a more restricted exit policy. Dec 18 07:57:11
>     a-n-o-n-y-m-e Tor-Alberta[904715]: Your computer is too slow to
>     handle this many circuit creation requests! Please consider using
>     the MaxAdvertisedBandwidth config option or choosing a more
>     restricted exit policy. [14254 similar message(s) suppressed in
>     last 180 seconds] ``` There are a few more of these messages
>     appearing afterward (My bandwidth is unlimited). This is the
>     typical Heartbeat that comes afterward: ``` Dec 18 13:49:42
>     a-n-o-n-y-m-e Tor-Alberta[904715]: Heartbeat: Tor's uptime is 6:00
>     hours, with 476 circuits open. I've sent 30.87 GB and received
>     9.04 GB. I've received 13401 connections on IPv4 and 0 on IPv6.
>     I've made 46754 connections with IPv4 and 0 with IPv6. Dec 18
>     13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: While not
>     bootstrapping, fetched this many bytes: 6412201 (server descriptor
>     fetch); 1424 (server descriptor upload); 376002 (consensus
>     network-status fetch); 57564 (microdescriptor fetch) Dec 18
>     13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Circuit handshake
>     stats since last time: 8/8 TAP, 1356688/6457307 NTor. Dec 18
>     13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Since startup we
>     initiated 0 and received 0 v1 connections; initiated 0 and
>     received 0 v2 connections; initiated 0 and received 0 v3
>     connections; initiated 0 and received 429 v4 connections;
>     initiated 285 and received 11137 v5 connections. Dec 18 13:49:42
>     a-n-o-n-y-m-e Tor-Alberta[904715]: Heartbeat: DoS mitigation since
>     startup: 0 circuits killed with too many cells, 0 circuits
>     rejected, 0 marked addresses, 0 marked addresses for max queue, 0
>     same address concurrent connections rejected, 0 connections
>     rejected, 0 single hop clients refused, 0 INTRODUCE2 rejected. ```
>     Then I have this kind of message appearing: ``` Dec 18 14:13:23
>     a-n-o-n-y-m-e Tor-Alberta[904715]: No circuits are opened. Relaxed
>     timeout for circuit 4487 (a Measuring circuit timeout 3-hop
>     circuit in state doing handshakes with channel state open) to
>     60000ms. However, it appears the circuit has timed out anyway. [4
>     similar message(s) suppressed in last 3300 seconds] ``` Then a few
>     days later, this bug report: ``` Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: tor_bug_occurred_(): Bug:
>     ../src/core/or/conflux.c:567: conflux_pick_first_leg: Non-fatal
>     assertion !(smartlist_len(cfx->legs) <= 0) failed. (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     Tor 0.4.8.10: Non-fatal assertion !(smartlist_len(cfx->legs) <= 0)
>     failed in conflux_pick_first_leg at ../src/core/or/conflux.c:567.
>     Stack trace: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(log_backtrace_impl+0x5b)
>     [0x55651f95b37b] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_bug_occurred_+0x18a)
>     [0x55651f97294a] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(conflux_decide_next_circ+0x40e) [0x55651fa12afe] (on
>     Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     Bug: /usr/bin/tor(circuit_get_package_window+0x75)
>     [0x55651fa12ec5] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x9ed63) [0x55651f908d63]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(connection_edge_package_raw_inbuf+0xae)
>     [0x55651f90b80e] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(connection_edge_process_inbuf+0x6f) [0x55651fa2b9df]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x1c2fb4) [0x55651fa2cfb4]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x73ffc) [0x55651f8ddffc]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x1ff58) [0x7fcee899bf58]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x577)
>     [0x7fcee899d8a7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(do_main_loop+0x127)
>     [0x55651f8de7c7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_run_main+0x215)
>     [0x55651f8e2805] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_main+0x4d)
>     [0x55651f8e2c6d] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(main+0x1d) [0x55651f8d4dcd]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcee80a8d90] (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)
>     [0x7fcee80a8e40] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(_start+0x25)
>     [0x55651f8d4e25] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching
>     client sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux
>     566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown:
>     0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21
>     15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     conflux_pick_first_leg(): Bug: Matching server sets: (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0
>     launched. Delivered: 75; teardown: 0; Current: (nil), Previous:
>     (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: End conflux
>     set dump (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: circuit_get_package_window(): Bug: Conflux
>     has no circuit to send on. Circuit 0x556536e76c20 idx 138 marked
>     at line ../src/core/or/command.c:663 (on Tor 0.4.8.10 ) Dec 21
>     15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: tor_bug_occurred_():
>     Bug: ../src/core/or/conflux.c:567: conflux_pick_first_leg:
>     Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed. (on
>     Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     Bug: Tor 0.4.8.10: Non-fatal assertion !(smartlist_len(cfx->legs)
>     <= 0) failed in conflux_pick_first_leg at
>     ../src/core/or/conflux.c:567. Stack trace: (on Tor 0.4.8.10 ) Dec
>     21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(log_backtrace_impl+0x5b) [0x55651f95b37b] (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(tor_bug_occurred_+0x18a) [0x55651f97294a] (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(conflux_decide_next_circ+0x40e) [0x55651fa12afe] (on
>     Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     Bug: /usr/bin/tor(circuit_get_package_window+0x75)
>     [0x55651fa12ec5] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x9ed63) [0x55651f908d63]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(connection_edge_package_raw_inbuf+0xae)
>     [0x55651f90b80e] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /usr/bin/tor(connection_edge_process_inbuf+0x6f) [0x55651fa2b9df]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x1c34dd) [0x55651fa2d4dd]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x73ffc) [0x55651f8ddffc]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x1ff58) [0x7fcee899bf58]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x577)
>     [0x7fcee899d8a7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(do_main_loop+0x127)
>     [0x55651f8de7c7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_run_main+0x215)
>     [0x55651f8e2805] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_main+0x4d)
>     [0x55651f8e2c6d] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(main+0x1d) [0x55651f8d4dcd]
>     (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcee80a8d90] (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug:
>     /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)
>     [0x7fcee80a8e40] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Bug: /usr/bin/tor(_start+0x25)
>     [0x55651f8d4e25] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching
>     client sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux
>     566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown:
>     0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21
>     15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     conflux_pick_first_leg(): Bug: Matching server sets: (on Tor
>     0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]:
>     conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0
>     launched. Delivered: 75; teardown: 0; Current: (nil), Previous:
>     (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: End conflux
>     set dump (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: circuit_get_package_window(): Bug: Conflux
>     has no circuit to send on. Circuit 0x556536e76c20 idx 138 marked
>     at line ../src/core/or/command.c:663 (on Tor 0.4.8.10 ) ``` It is
>     then an alternation of these three last messages (Heartbeat + "No
>     circuits are opened" + bug) until yesterday where I get a series
>     of these messages: ``` Jan 5 21:58:59 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Failed to find node for hop #1 of our path.
>     Discarding this circuit. Jan 5 21:58:59 a-n-o-n-y-m-e
>     Tor-Alberta[904715]: Our circuit 0 (id: 355722) died due to an
>     invalid selected path, purpose Unlinked conflux circuit. This may
>     be a torrc configuration issue, or a bug. ``` At this point, I
>     learned that my relays were down via Tor Weather. I restarted the
>     relays - even rebooted the whole server - and both relays seem to
>     be unable to connect to the network. Here's a sample: ``` Jan 6
>     09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: 254 connections have
>     failed: Jan 6 09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: 254
>     connections died in state connect()ing with SSL state (No SSL
>     object) Jan 6 09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: Problem
>     bootstrapping. Stuck at 5% (conn): Connecting to a relay.
>     (Connection timed out; TIMEOUT; count 256; recommendation warn;
>     host 65369D044C659CD299E35763914FFD0FC9AD4509 at
>     92.205.161.164:80) ``` I also have a bridge on this server
>     (different IP) and it seems OK (
>     https://metrics.torproject.org/rs.html#details/7CEB9D16C5218FE1A9BAB8E8A6EA9471D2E1F9B8
>     ). Last messages after the server reboot are: ``` Jan 6 08:34:17
>     a-n-o-n-y-m-e Tor-Nestor[998]: Bootstrapped 100% (done): Done Jan
>     6 08:35:21 a-n-o-n-y-m-e Tor-Nestor[998]: Performing bandwidth
>     self-test...done. Jan 6 09:54:22 a-n-o-n-y-m-e Tor-Nestor[998]: No
>     circuits are opened. Relaxed timeout for circuit 527 (a Measuring
>     circuit timeout 3-hop circuit in state doing handshakes with
>     channel state open) to 60000ms. However, it appears the circuit
>     has timed out anyway. ``` I have stopped both relays until I
>     figure out what is going on. Help would be appreciated.
>
>
_______________________________________________
tor-relays mailing list
tor-relays@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays