[tor-relays] unbound bogs down strangely, degrading exit relay

[Dhalgren Tor <dhalgren.tor@xxxxxxxxx>]

Hit a repeat of an earlier incident:

https://lists.torproject.org/pipermail/tor-relays/2016-January/008621.html

message from tor daemon is

   Resolved [scrubbed] which was already resolved; ignoring

About 5400 of these messages over 37 hours, during which the relay
dropped down to 30% of usual load and did not work for Tor Browser
when the exit was specifically set.  Unlike the earlier incident,
DNSSEC was disabled.  Probably the IP address for root-server H was
incorrect because it changed in December and 'unbound' was built in
September.  Just installed and configured the current 'named.root'
hints file.  Both "iterator" and "validator" modules were loaded,
though validation was disabled.  Changed the config to load only
"iterator".

Unbound 1.5.4, tor 0.2.6.10.  Running 164 days.

If I see this again I'm planning to attempt purging the 'unbound'
request queue with

   unbound-control dump_requestlist >reqlist.snap

   unbound-control flush_requestlist

and if that doesn't work, bouncing 'unbound' without stopping the 'tor' daemon.

Interesting stats were recorded, starting with the good pre-incident entries:

   Mar 11 05:53
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 15 17:45
   server stats for thread: 1029423 queries, 415824 answers from
cache, 613599 recursions, 0 prefetch
   server stats for thread: requestlist max 132 avg 22.5692 exceeded 0 jostled 0
   average recursion processing time 4.070100 sec

   Mar 16 05:46
   Resolved [scrubbed] which was already resolved; ignoring

   Mar 16 17:45
   server stats for thread: 1162172 queries, 287662 answers from
cache, 874510 recursions, 0 prefetch
   server stats for thread: requestlist max 421 avg 198.684 exceeded 0 jostled 0
   average recursion processing time 25.833646 sec

   Mar 16 21:08
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 17 16:52
   notice: sendto failed: Invalid argument
   notice: remote address is 0.0.0.1 port 53
   ...repeated 4 times

   Mar 17 17:45
   server stats for thread: 1078496 queries, 144557 answers from
cache, 933939 recursions, 0 prefetch
   server stats for thread: requestlist max 459 avg 296.668 exceeded 0 jostled 0
   average recursion processing time 40.621863 sec

   service stopped (unbound 1.5.4).

   Mar 17 19:28
   server stats for thread: 53991 queries, 3455 answers from cache,
50536 recursions, 0 prefetch
   server stats for thread: requestlist max 342 avg 290.977 exceeded 0 jostled 0
   average recursion processing time 35.947563 sec

If anyone has seen this or knows anything please comment.  Tried
searching but came up with nothing but thread referenced above.
_______________________________________________
tor-relays mailing list
tor-relays@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays