Re: [tor-relays] Metrics for assessing EFF's Tor relay challenge?

On Tue, Apr 8, 2014 at 12:59 PM, Karsten Loesing <karsten@xxxxxxxxxxxxxx> wrote:

On 05/04/14 17:46, Lukas Erlacher wrote:
> Hello Nikita, Karsten,
>
> On 04/05/2014 05:03 PM, Nikita Borisov wrote:
>> On Sat, Apr 5, 2014 at 3:58 PM, Karsten Loesing
>> <karsten@xxxxxxxxxxxxxx> wrote:
>>> Installing packages using Python-specific package managers is
>>> going to make our sysadmins sad, so we should have a very good
>>> reason for wanting such a package. ÂIn general, we don't need
>>> the latest and greatest package. ÂUnless we do.
>> What about virtualenv? Part of the premise behind it is that you
>> can configure appropriate packages as a developer / operator
>> without having to bother sysadmins and making them worried about
>> system-wide effects.
>>
>> - Nikita
>
> I was going to mention virtualenv as well, but I have to admit that
> I find it weird and scary, especially since I haven't found good
> documentation for it. If there is somebody who is familiar with
> virtualenv that would probably be the best solution.

I'm afraid I don't know enough about Python or virtualenv. ÂSo far, it
was almost zero effort for our sysadmins to install a package from the
repositories and keep that up-to-date. ÂI'd like to stick with the
apt-get approach and save the virtualenv approach for situations when
we really need a package that is not contained in the repositories.

Thanks for the suggestion, though!

> On 04/05/2014 04:58 PM, Karsten Loesing wrote:
>> My hope with challenger is that it's written quickly, working
>> quietly for a year, and then disappearing without anybody
>> noticing. ÂI'd rather not want to maintain yet another thing.
>> So, maybe Weather is a better candidate for using onion-py than
>> challenger.
>
> Yes, I understand.
>
>> Yeah, I think we'll want to define a maximum lifetime of cache
>> entries, or the poor cache will explode pretty soon.
>
> What usage patterns do we have to expect? Do we want to hit onionoo
> to check if the cache is still valid for every request, or should
> we do "hard caching" for several minutes? The best UX solution
> would be to have a background task that keeps the cache current so
> user requests can be delivered without hitting onionoo at all.

That's a fine question. ÂI can see various caching approaches here.
But I just realize that this is premature optimization. ÂLet's first
build the thing and download whatever we need and whenever we need it.
ÂAnd once we know what caching needs we have, let's build the cache.

> In other words, unless we do something intelligent with the cache,
> the cache is not actually going to be very useful.

Valid point. :)

>> Great, your help would be much appreciated! ÂWant to send me a
>> pull request whenever you have something to merge?
>
> Will do.

Great. ÂThanks!

Hi Karsten and others,

I got to run the challenger script by chance[1], and spotted a small mistake that was preventing Lukas' onion.py downloader code from working. Ended up forking and creating a separate branch:

https://github.com/wfn/challenger/commits/wfn_fix_luk3s_download

Relevant commits:

Â -Â38d88bcb1136f97881f81152d3d883c4e9480188[2] (enables downloader)

Â -Â39c800643c040474402fc62d2a2db75c25889dfc[3] (this is the one with the small thingie-fix)

(It was a very small thing with the way the 'requests' module handles/provides json documents.)

I was doing this to be able to give Roger the 'combined-*.json' files for currently vulnerable (re: openssl) relays (he wanted to see which part of the combined weight fraction they comprise, etc.)

Fingerprints for those relays are here, fwiw:Âhttp://ravinesmp.com/volatile/challenger-stuff/vuln_fingerprints.txt (the original link that Roger gave me was http://fpaste.org/92688/ )

(count: 1024.)

If you download these fingerprints, you can just run `python challenge.py -f vuln_fingerprints.txt`

(for anyone using virtualenv, you might need to `pip install requests`, and then things should work. For anyone who's just cloned the thing, everything should probably work after simply installing the 'requests' python module, if it's not there. I see that 'python-requests' is available in the repos.)

I guess the code hasn't been tested for those amounts of fingerprints before. Good news: it works (where 'works' means 'i opened the resulting files and they contained all those fingerprints, and/or they contained lots of numbers.') Kinda-bad news: Onionoo doesn't seem to share the enthusiasm, and hiccups, and spitsÂ502 Proxy Error some time after the lookups for the first document (combined bandwidth) are made.

My cheap quick hack was to insert time.sleep() here and there:

Â - 7425ef6fc00dedf3b2b7f2649e832fb4c93909ae[4]

(cheap hack is cheap, but it worked. Note: takes time to download everything. Didn't time it yet - sorry.)

For anyone interested, these are the resulting 'combined-*.json' files from all those fingerprints:

Â -Âhttp://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-bandwidth.json

Â -Âhttp://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-weights.json

Â -Âhttp://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-clients.json [uh oh, this one's empty. Why is it empty? Didn't look into it.]

Â -Âhttp://ravinesmp.com/volatile/challenger-stuff/vuln1024-combined-uptime.json

I haven't much looked into them, at least not yet.

Roger wants to get some information about those vulnerable relays, and he thinks this challenger stuff can help with that. Those combined-* documents seem useful. I made a separate ML thread for this:

https://lists.torproject.org/pipermail/tor-relays/2014-April/004262.html [wow, i should switch to plaintext email probably..]

[1]: where 'by chance' means 'fell under arma's irc-Jedi spells somehow' / didn't plan to / i might be wrong about things or the things i did to the script, so beware

[2]:Âhttps://github.com/wfn/challenger/commit/38d88bcb1136f97881f81152d3d883c4e9480188

[3]:Âhttps://github.com/wfn/challenger/commit/39c800643c040474402fc62d2a2db75c25889dfc

[4]:Âhttps://github.com/wfn/challenger/commit/7425ef6fc00dedf3b2b7f2649e832fb4c93909ae

take it easy

kostas / wfn

0x0e5dce45Â@Âpgp.mit.edu