[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances



#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
 Reporter:  iwakeh             |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:  CollecTor 1.1.0
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:  ctip               |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+---------------------------------

Comment (by iwakeh):

 Replying to [comment:46 karsten]:

 I found the a good reference for the paths topic: it's in the
 [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh#n68 create tars script]:
 {{{
   $OUTDIR/exit-lists/$YEARONE/$MONTHONE/
   $OUTDIR/exit-lists/$YEARTWO/$MONTHTWO/
   $OUTDIR/torperf/$YEARONE/$MONTHONE/
   $OUTDIR/torperf/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/certs/
   $OUTDIR/relay-descriptors/microdesc/$YEARONE/$MONTHONE
   $OUTDIR/relay-descriptors/microdesc/$YEARTWO/$MONTHTWO
   $OUTDIR/relay-descriptors/consensus/$YEARONE/$MONTHONE
   $OUTDIR/relay-descriptors/consensus/$YEARTWO/$MONTHTWO
   $OUTDIR/relay-descriptors/vote/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/vote/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/server-descriptor/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/server-descriptor/$YEARTWO/$MONTHTWO/
   $OUTDIR/relay-descriptors/extra-info/$YEARONE/$MONTHONE/
   $OUTDIR/relay-descriptors/extra-info/$YEARTWO/$MONTHTWO/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/statuses/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/statuses/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/server-descriptors/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/server-descriptors/
   $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/extra-infos/
   $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/extra-infos/
 }}}

 The above makes an inconsistency between relay descriptor and bridge
 descriptor paths visible.

 As the new structure from #20228, i.e. votes also grouped and all dates in
 recent paths are derived from the acquisition time, will be in place soon,
 I'd like to use the 'new way' already for the merge.

 With these as background the test diff has some soon to be outdated
 corrections (comments added after '//'):
 >
 > {{{
 > diff --git
 a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > index ad5ee6a..cdb90b8 100644
 > ---
 a/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > +++
 b/src/test/java/org/torproject/collector/sync/SyncPersistenceTest.java
 > @@ -58,15 +58,15 @@ public class SyncPersistenceTest {
 >    @Parameters
 >    public static Collection<Object[]> pathFilename() {
 >      return Arrays.asList(new Object[][] {
 > -        {"exit-lists/2016-10-05-19-06-17", // expected recent path  //
 acqu-date, which will be ok
 > +        {"exit-lists/2016-09-20-13-02-00", // expected recent path
 >           "exit-lists/2016/09/20/2016-09-20-13-02-00", // expected out
 path
 >           "2016-09-20-13-02-00", // test-filename
 >           Integer.valueOf(1), // expected recent count of descs files
 >           Integer.valueOf(1)}, // expected output count of descs files
 >
 >          {"relay-descriptors/microdescs/consensus-microdesc/"
 > -             + "2016-10-05-19-06-17-consensus-microdesc", // acqu-date,
 which will be ok
 > -         "relay-descriptors/microdescs/2016/10/consensus-microdesc/"
 > +             + "2016-10-02-17-00-00-consensus-microdesc",
 > +         "relay-descriptors/microdesc/2016/10/consensus-microdesc/"
 >               + "02/2016-10-02-17-00-00-consensus-microdesc",
 >           "2016-10-02-17-00-00-consensus-microdesc",
 >           Integer.valueOf(1),
 > @@ -74,7 +74,7 @@ public class SyncPersistenceTest {
 >
 >          {"bridge-descriptors/server-descriptors/"
 >               + "2016-10-05-19-06-17-server-descriptors",
 > -         "bridge-descriptors/server-descriptor/2016/10/"
 > +         "bridge-descriptors/2016/10/server-descriptor/"
 >               + "A/8/A8A5509AD1393C8F36ABD2D8F0DE1BB751926872",
 >           "bridge-2016-10-02-16-09-00-server-descriptors",
 >           Integer.valueOf(1),
 > @@ -88,28 +88,30 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1),
 >           Integer.valueOf(10)},
 >
 > -        {"relay-descriptors/consensuses/2016-10-05-19-06-17-consensus",
 // acqu-date, which will be ok
 > +        {"relay-descriptors/consensuses/2016-09-20-13-00-00-consensus",
 >           "relay-
 descriptors/consensus/2016/09/20/2016-09-20-13-00-00-consensus",
 >           "2016-09-20-13-00-00-consensus",
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 >          {"bridge-descriptors/statuses/"
 > -             +
 "20161005-190617-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1", // acqu-date,
 which will be ok
 > -         "bridge-descriptors/statuses/2016/09/20/"
 > +             +
 "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 > +         "bridge-descriptors/2016/09/statuses/20/"
 >               +
 "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 >           "20160920-063816-1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1",
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 >          {"relay-
 descriptors/microdescs/micro/2016-10-05-19-06-17-micro",
 > -         "relay-descriptors/microdescs/2016/10/micro/f/b/"
 > +         "relay-descriptors/microdesc/2016/10/micro/f/b/"
 >               +
 "fbf0c4cb8216d950d267866ea879880e69216bd0b46878ff0f51941ee4eca707",
 >           "2016-09-20-15-31-19-micro",
 >           Integer.valueOf(1),
 >           Integer.valueOf(5)},
 >
 > -        {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
 date, which will be ok
 > +        {"relay-descriptors/votes/2016/10/01/2016-10-01-16-00-00-vote"
 > +             + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
 > +             + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
 >           "relay-descriptors/vote/2016/10/01/2016-10-01-16-00-00-vote"
 >               + "-0232AF901C31A04EE9848595AF9BB7620D4C5B2E"
 >               + "-FEE63B4AB7CE5A6BDD09E9A5C4F01BD61EB7E4F1",
 > @@ -118,7 +120,9 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1),
 >           Integer.valueOf(1)},
 >
 > -        {"relay-descriptors/votes/2016-10-05-19-06-17-votes", // acqu-
 date, which will be ok
 > +        {"relay-descriptors/votes/2016/09/20/2016-09-20-13-00-00-vote-"
 > +             + "49015F787433103580E3B66A1707A00E60F2D15B"
 > +             + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
 >           "relay-descriptors/vote/2016/09/20/2016-09-20-13-00-00-vote-"
 >               + "49015F787433103580E3B66A1707A00E60F2D15B"
 >               + "-60ADC6BEC262AE921A1037D54C8A3976367DBE87",
 > @@ -128,7 +132,7 @@ public class SyncPersistenceTest {
 >           Integer.valueOf(1)},
 >
 >          {"bridge-descriptors/extra-infos/2016-10-05-19-06-17-extra-
 infos",
 > -         "bridge-descriptors/extra-info/2016/10/9/F/"
 > +         "bridge-descriptors/2016/10/extra-info/9/F/"
 >               + "9F88A7C2ABE6665D204137BA8C2661D42E7C2829",
 >           "bridge-2016-10-02-08-09-00-extra-infos",
 >           Integer.valueOf(1),
 > }}}

 When changing the structure now shouldn't relays and bridges be stored in
 similar ways?  I think the current difference (in 'out') caused some of
 the confusion here.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:49>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs