[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances

Subject: Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
From: "Tor Bug Tracker & Wiki" <blackhole@xxxxxxxxxxxxxx>
Date: Mon, 17 Oct 2016 18:04:32 -0000
Auto-submitted: auto-generated
Delivered-to: archiver@xxxxxxxx
Delivery-date: Mon, 17 Oct 2016 14:04:42 -0400
In-reply-to: <046.efa4d874c0d1b588ace4f821ae2d5546@torproject.org>
List-archive: <http://lists.torproject.org/pipermail/tor-bugs/>
List-help: <mailto:tor-bugs-request@lists.torproject.org?subject=help>
List-id: "auto: Tor bug tracker status mails" <tor-bugs.lists.torproject.org>
List-post: <mailto:tor-bugs@lists.torproject.org>
List-subscribe: <https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs>, <mailto:tor-bugs-request@lists.torproject.org?subject=subscribe>
List-unsubscribe: <https://lists.torproject.org/cgi-bin/mailman/options/tor-bugs>, <mailto:tor-bugs-request@lists.torproject.org?subject=unsubscribe>
References: <046.efa4d874c0d1b588ace4f821ae2d5546@torproject.org>
Reply-to: no-reply@xxxxxxxxxxxxxx, tor-assistants@xxxxxxxxxxxxxx
Sender: "tor-bugs" <tor-bugs-bounces@xxxxxxxxxxxxxxxxxxxx>

#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:  CollecTor 1.1.0
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------------------

Comment (by iwakeh):

 Replying to [comment:46 karsten]:

 I found the a good reference for the paths topic: it's in the
 [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh#n68 create tars script]:
 {{{
   $OUTDIR/exit-lists/$YEARONE/$MONTHONE/
   $OUTDIR/exit-lists/$YEARTWO/$MONTHTWO/
   $OUTDIR/torperf/$YEARONE/$MONTHONE/
   $OUTDIR/torperf/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/certs/
   $OUTDIR/relay-descriptors/microdesc/$YEARONE/$MONTHONE
   $OUTDIR/relay-descriptors/microdesc/$YEARTWO/$MONTHTWO
   $OUTDIR/relay-descriptors/consensus/$YEARONE/$MONTHONE
   $OUTDIR/relay-descriptors/consensus/$YEARTWO/$MONTHTWO
   $OUTDIR/relay-descriptors/vote/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/vote/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/server-descriptor/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/server-descriptor/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/extra-info/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/extra-info/$YEARTWO/$MONTHTWO/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/statuses/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/statuses/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/server-descriptors/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/server-descriptors/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/extra-infos/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/extra-infos/
 }}}

 The above makes an inconsistency between relay descriptor and bridge
 descriptor paths visible.

 As the new structure from #20228, i.e. votes also grouped and all dates in
 recent paths are derived from the acquisition time, will be in place soon,
 I'd like to use the 'new way' already for the merge.

 With these as background the test diff has some soon to be outdated
 corrections (comments added after '//'):
 >
 > {{{
 > diff --git
 a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > index ad5ee6a..cdb90b8 100644
 > ---
 a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > +++
 b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > @@ -58,15 +58,15 @@ public class SyncPersistenceTest {
 >    @Parameters
 >    public static Collection<Object[]> pathFilename() {
 >      return Arrays.asList(new Object[][] {
 > -        {"exit-lists/2016-10-05-19-06-17", // expected recent path  //
 acqu-date, which will be ok
 > +        {"exit-lists/2016-09-20-13-02-00", // expected recent path
 >           "exit-lists/2016/09/20/2016-09-20-13-02-00", // expected out
 path
 >           "2016-09-20-13-02-00", // test-filename
 >           Integer.valueOf(1), // expected recent count of descs files
 >           Integer.valueOf(1)}, // expected output count of descs files
 >
 >          {"relay-descriptors/microdescs/consensus-microdesc/"
 > -             + "2016-10-05-19-06-17-consensus-microdesc", // acqu-date,
 which will be ok
 > -         "relay-descriptors/microdescs/2016/10/consensus-microdesc/"
 > +             + "2016-10-02-17-00-00-consensus-microdesc",
 > +         "relay-descriptors/microdesc/2016/10/consensus-microdesc/"
 >               + "02/2016-10-02-17-00-00-consensus-microdesc",
 >           "2016-10-02-17-00-00-consensus-microdesc",
 >           Integer.valueOf(1),
 > @@ -74,7 +74,7 @@ public class SyncPersistenceTest {
 >
 >          {"bridge-descriptors/server-descriptors/"
 >               + "2016-10-05-19-06-17-server-descriptors",
 > -         "bridge-descriptors/server-descriptor/2016/10/"
 > +         "bridge-descriptors/2016/10/server-descriptor/"
 >               + "A/8/A8A5509AD1393C8F36ABD2D8F0DE1BB751926872",
 >           "bridge-2016-10-02-16-09-00-server-descriptors",
 >           Integer.valueOf(1),
 > @@ -88,28 +88,30 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1),
 >           Integer.valueOf(10)},
 >
 > -        {"relay-descriptors/consensuses/2016-10-05-19-06-17-consensus",
 // acqu-date, which will be ok
 > +        {"relay-descriptors/consensuses/2016-09-20-13-00-00-consensus",
 >           "relay-
 descriptors/consensus/2016/09/20/2016-09-20-13-00-00-consensus",
 >           "2016-09-20-13-00-00-consensus",
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 >          {"bridge-descriptors/statuses/"
 > -             +
 "20161005-190617-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1", // acqu-date,
 which will be ok
 > -         "bridge-descriptors/statuses/2016/09/20/"
 > +             +
 "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 > +         "bridge-descriptors/2016/09/statuses/20/"
 >               +
 "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 >           "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 >          {"relay-
 descriptors/microdescs/micro/2016-10-05-19-06-17-micro",
 > -         "relay-descriptors/microdescs/2016/10/micro/f/b/"
 > +         "relay-descriptors/microdesc/2016/10/micro/f/b/"
 >               +
 "fbf0c4cb8216d950d267866ea879880e69216bd0b46878ff0f51941ee4eca707",
 >           "2016-09-20-15-31-19-micro",
 >           Integer.valueOf(1),
 >           Integer.valueOf(5)},
 >
 > -        {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
 date, which will be ok
 > +        {"relay-descriptors/votes/2016/10/01/2016-10-01-16-00-00-vote"
 > +             + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
 > +             + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
 >           "relay-descriptors/vote/2016/10/01/2016-10-01-16-00-00-vote"
 >               + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
 >               + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
 > @@ -118,7 +120,9 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 > -        {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
 date, which will be ok
 > +        {"relay-descriptors/votes/2016/09/20/2016-09-20-13-00-00-vote-"
 > +             + "49015F787433103580E3B66A1707A00E60F2D15B"
 > +             + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
 >           "relay-descriptors/vote/2016/09/20/2016-09-20-13-00-00-vote-"
 >               + "49015F787433103580E3B66A1707A00E60F2D15B"
 >               + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
 > @@ -128,7 +132,7 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1)},
 >
 >          {"bridge-descriptors/extra-infos/2016-10-05-19-06-17-extra-
 infos",
 > -         "bridge-descriptors/extra-info/2016/10/9/F/"
 > +         "bridge-descriptors/2016/10/extra-info/9/F/"
 >               + "9F88A7C2ABE6665D204137BA8C2661D42E7C2829",
 >           "bridge-2016-10-02-08-09-00-extra-infos",
 >           Integer.valueOf(1),
 > }}}

 When changing the structure now shouldn't relays and bridges be stored in
 similar ways?  I think the current difference (in 'out') caused some of
 the confusion here.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:49>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Prev by Author: Re: [tor-bugs] #18222 [Applications/Tor Browser]: Tor Browser sets different Guard for one of the tabs when goes crazy (was: Circuit visualizer shows different Guard for one of the tabs!)
Next by Author: Re: [tor-bugs] #14828 [Core Tor/Tor]: Multiple hidden services can share a pk_digest/service_id.
Previous by thread: Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
Next by thread: Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
Index(es):
- Author
- Thread