[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories

Subject: Re: [tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories
From: "Tor Bug Tracker & Wiki" <blackhole@xxxxxxxxxxxxxx>
Date: Fri, 21 Oct 2016 09:26:44 -0000
Auto-submitted: auto-generated
Delivered-to: archiver@xxxxxxxx
Delivery-date: Fri, 21 Oct 2016 05:26:55 -0400
In-reply-to: <047.e5f16da8813d3f253adc866a3fa899ff@torproject.org>
List-archive: <http://lists.torproject.org/pipermail/tor-bugs/>
List-help: <mailto:tor-bugs-request@lists.torproject.org?subject=help>
List-id: "auto: Tor bug tracker status mails" <tor-bugs.lists.torproject.org>
List-post: <mailto:tor-bugs@lists.torproject.org>
List-subscribe: <https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs>, <mailto:tor-bugs-request@lists.torproject.org?subject=subscribe>
List-unsubscribe: <https://lists.torproject.org/cgi-bin/mailman/options/tor-bugs>, <mailto:tor-bugs-request@lists.torproject.org?subject=unsubscribe>
References: <047.e5f16da8813d3f253adc866a3fa899ff@torproject.org>
Reply-to: no-reply@xxxxxxxxxxxxxx, tor-assistants@xxxxxxxxxxxxxx
Sender: "tor-bugs" <tor-bugs-bounces@xxxxxxxxxxxxxxxxxxxx>

#20421: Investigate invalid descriptors in out/ and recent/ subdirectories
-------------------------------+---------------------
 Reporter:  karsten            |          Owner:
     Type:  defect             |         Status:  new
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------

Comment (by karsten):

 Ah, some more information from the CollecTor logs:

 {{{
 2016-10-19 04:05:00,001 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs
 module of CollecTor.
 2016-10-19 04:05:10,870 INFO o.t.c.r.ArchiveWriter:842 Could not parse
 relay descriptor out/relay-descriptors/server-
 descriptor/2016/10/5/7/57077357a8bc1d431e89e29e3cec7d9a31d59332 before
 storing it to disk.  Storing anyway.
 org.torproject.descriptor.DescriptorParseException: Either keyword
 'accept' or 'reject' must be contained at least once.
         at
 org.torproject.descriptor.impl.ServerDescriptorImpl.<init>(ServerDescriptorImpl.java:49)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayServerDescriptorImpl.<init>(RelayServerDescriptorImpl.java:35)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayServerDescriptorImpl.parseDescriptors(RelayServerDescriptorImpl.java:24)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:61)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.storeServerDescriptor(ArchiveWriter.java:752)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:266)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 [na:1.7.0_111]
         at
 java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_111]
         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
 2016-10-19 04:05:18,322 INFO o.t.c.r.ArchiveWriter:842 Could not parse
 relay descriptor out/relay-descriptors/extra-
 info/2016/10/4/8/481a2c9b4db3a29e499b9ce0b8bf26ca0c344e0f before storing
 it to disk.  Storing anyway.
 org.torproject.descriptor.DescriptorParseException: Mismatch between
 identity-ed25519 and master-key-ed25519.
         at
 org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.parseIdentityEd25519CryptoBlock(ExtraInfoDescriptorImpl.java:789)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.parseDescriptorBytes(ExtraInfoDescriptorImpl.java:260)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.<init>(ExtraInfoDescriptorImpl.java:35)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.<init>(RelayExtraInfoDescriptorImpl.java:35)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.parseDescriptors(RelayExtraInfoDescriptorImpl.java:24)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:71)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.storeExtraInfoDescriptor(ArchiveWriter.java:782)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:305)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 [na:1.7.0_111]
         at
 java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_111]
         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
 2016-10-19 04:05:18,590 INFO o.t.c.r.CachedRelayDescriptorReader:255
 Finished importing relay descriptors from local Tor data directories:
 }}}

 {{{
 2016-10-19 07:05:00,001 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs
 module of CollecTor.
 2016-10-19 07:05:10,500 INFO o.t.c.r.ArchiveWriter:842 Could not parse
 relay descriptor out/relay-descriptors/server-
 descriptor/2016/10/4/6/4630ff545faaf16f8102311c4a1506a7497a70f2 before
 storing it to disk.  Storing anyway.
 org.torproject.descriptor.DescriptorParseException: Unrecognized character
 in keyword 'hidden-@uploaded-at' in line 'hidden-@uploaded-at 2016-10-19
 06:53:16'.
         at
 org.torproject.descriptor.impl.ParseHelper.parseKeyword(ParseHelper.java:30)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.ServerDescriptorImpl.parseDescriptorBytes(ServerDescriptorImpl.java:228)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.ServerDescriptorImpl.<init>(ServerDescriptorImpl.java:30)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayServerDescriptorImpl.<init>(RelayServerDescriptorImpl.java:35)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayServerDescriptorImpl.parseDescriptors(RelayServerDescriptorImpl.java:24)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:61)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.storeServerDescriptor(ArchiveWriter.java:752)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:266)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 [na:1.7.0_111]
         at
 java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_111]
         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
 2016-10-19 07:05:10,978 INFO o.t.c.r.ArchiveWriter:842 Could not parse
 relay descriptor out/relay-descriptors/extra-
 info/2016/10/3/6/36d5e078af899d327f286e5cd0bdc48189ea2a1d before storing
 it to disk.  Storing anyway.
 org.torproject.descriptor.DescriptorParseException: Keyword 'published' is
 contained 2 times, but must be contained exactly once.
         at
 org.torproject.descriptor.impl.DescriptorImpl.checkExactlyOnceKeywords(DescriptorImpl.java:293)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.ExtraInfoDescriptorImpl.<init>(ExtraInfoDescriptorImpl.java:40)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.<init>(RelayExtraInfoDescriptorImpl.java:35)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.RelayExtraInfoDescriptorImpl.parseDescriptors(RelayExtraInfoDescriptorImpl.java:24)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorImpl.parseDescriptors(DescriptorImpl.java:71)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.descriptor.impl.DescriptorParserImpl.parseDescriptors(DescriptorParserImpl.java:26)
 ~[collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.store(ArchiveWriter.java:836)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.storeExtraInfoDescriptor(ArchiveWriter.java:782)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.RelayDescriptorParser.parse(RelayDescriptorParser.java:305)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.CachedRelayDescriptorReader.<init>(CachedRelayDescriptorReader.java:216)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.relaydescs.ArchiveWriter.startProcessing(ArchiveWriter.java:166)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 org.torproject.collector.cron.CollecTorMain.run(CollecTorMain.java:56)
 [collector-1.0.2.jar:1.0.2-c43d0ad]
         at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 [na:1.7.0_111]
         at
 java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 [na:1.7.0_111]
         at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_111]
         at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_111]
         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
 2016-10-19 07:05:14,710 INFO o.t.c.r.CachedRelayDescriptorReader:255
 Finished importing relay descriptors from local Tor data directories:
 }}}

 So, we can probably expect invalid descriptors to be contained in the
 tarballs since we allowed descriptors that we cannot parse.  Heh, makes
 more sense now, though we probably cannot do much about this.  Still, ugh!

 These descriptors confuse CollecTor's reference checker which relies on
 metrics-lib, making it think it's missing a whole lot of descriptors.  I
 wonder if we need to make it more robust and continue reading until it
 finds the next valid descriptor.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20421#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

References:
- [tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories
  - From: Tor Bug Tracker & Wiki

Prev by Author: Re: [tor-bugs] #18504 [Internal Services/Service - git]: Download snapshots from gitweb
Next by Author: Re: [tor-bugs] #20404 [Metrics/metrics-lib]: DescriptorIndexCollector should be default implementation
Previous by thread: [tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories
Next by thread: Re: [tor-bugs] #20421 [Metrics/CollecTor]: Investigate invalid descriptors in out/ and recent/ subdirectories
Index(es):
- Author
- Thread