Re: [tor-dev] Crux: Privacy-preserving statistics for Tor

Hi,

so you were right the databases were corrupt, but they shouldn't have been there in the first place. :-)

I didn't want to include large files in the git repo (~120mb in total), so there is a generation script in the tools directory (now added).

I added some instructions on the readme file too. Of course, this procedure is done only once.

I tried it and it should work now, but let me know if there is still a problem.

There is a plan to optimize the way the pre-computed values are used, and this could potentially shrink the size of the databases by a lot.

Vasilis

On 17 October 2015 at 15:56, str4d <str4d@xxxxxxxxxxx> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

str4d wrote:
> Vasilios Mavroudis wrote:
>> Hello,
>
>> I would like to introduce our project "Crux", which enables the
>> computation of privacy preserving statistics on sensitive data.
>> The project was developed at University College London (UCL) by
>> me, in close collaboration with George Danezis.
>
> This is fantastic work. Haven't read all of your thesis yet, but
> it looks similar to the stats collection system that we ran for a
> while inside I2P, but with the privacy protections that I wanted it
> to have. Well done!
>
>
>> "Crux" was designed with Tor hidden services in mind (in
>> accordance with this
>> <https://research.torproject.org/techreports/hidden-service-stats-201
5
>
>>
- -04-28.pdf>
>
>
> Tor
>> Tech Report), but it can be applied on various kinds of data
>> (given an appropriate parser).
>
> Excellent. I want to use this in I2P :)
>
>
>> It supports three statistics (i.e. mean, median, variance) and
>> its threat model is compatible Tor's (see technical document
>> below).
>
> I will look into this, but I doubt there will be any
> insurmountable incompatibilities with I2P's threat model.
>
>> Currently, we don't have a parser specific for the data
>> collected by the Tor relays, but we plan to keep adding
>> functionality.
>
>> The source code can be found here:
>> https://github.com/mavroudisv/Crux The theoretical background,
>> threat model, security argument and technical details can be
>> found here: link <http://mavroudisv.eu/files/thesis.pdf>
>
>
>> Ideas, comments, feedback much appreciated!
>
> Firstly: needs more docs. I shouldn't have to read your entire
> thesis in order to figure out how to run it :P The README is a good
> start, but the command parameters need clarification/definitions,
> and it is not at all obvious how the relays work. An XLS file is
> also mentioned without explanation.

I have now tried running the code, but the "unit test" run by
authority.py fails because bsbdb.btopen() returns an empty dict. I
don't know if this is because the table files are broken or because of
a local issue with my setup (I am using a virtualenv, but I get the
same empty dict when I try to open the tables manually outside the
virtualenv). I am not going to try and fix the latter on my own,
because bsddb is deprecated since Python 2.6, so you shouldn't really
be using it :) (But it must be something else, because I tried using
bsddb3 instead and that also returns an empty dict).

str4d

>
> Secondly, you could probably simplify configuration by managing
> hidden service connections yourself. I believe the Stem library can
> do this? I2P has a python library that can be used to listen on and
> connect to I2P Destinations, with a socket-like interface.
>
> It's not clear to me from your thesis whether you plan to merge
> the relay component into the Tor relay code (and have the Tor
> relays talk directly to the query processors), or have your relays
> running separately and interfacing with the Tor relays to fetch
> stats. This will potentially alter how it would be used with other
> networks. In I2P's case I'd imagine it would be easiest (in terms
> of usability by those setting up relays) to write an I2P plugin in
> Java that performs the relay role, but I wouldn't want to be
> duplicating a lot of logic that subsequently needs to be kept
> in-sync with upstream :)
>
> str4d
>
>
>> Vasilis
>
>
>
>> _______________________________________________ tor-dev mailing
>> list tor-dev@xxxxxxxxxxxxxxxxxxxx
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
> _______________________________________________ tor-dev mailing
> list tor-dev@xxxxxxxxxxxxxxxxxxxx
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
>
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCgAGBQJWItIgAAoJEBO17ljAn7PglsQP/3tOrL9EZfYMb5fxxVI6/wJm
qDvA8RpbEJDhoMjOc9sLNyMVgvGGPxEdma8I/R8H37yIPOp2GJg82MoXfUk+AGFe
5ZIIXwf2EzHH0RFLRI5xaQ4DW3Ru1yIzH+E3vhuGclL87YRKnAM0xkMQcueR5xo5
/ETe9TI7oPCdETI4J6iFoM+Ob9qsRiEVeuT5HEFFKwQhBvl0NSpfMHV9ftuxQHuu
I2B9jomnBdzF9OXnSQrYXg3ho2rxX4hO1TZfQHg5DgHvkt8TQYQM3cwdGE7/rsqC
d4+kBW2RuAz68wq+fLPeeZ2U2ff2QqmDLqVyUJvpP5RqeUcv66J/iYm7md886tTc
palwVGodKiTJ6DBM0z1K97x6Sl1jJXEAkZQAr2yQ8jrC/BJqQRPu2vHeZILqbMEQ
UU3JLjIk/Jc2RGK6zitvj4kTrLdfACEQMYy3brQUawZhaM42QNQgb3LQq1bJXzUo
vISZckX/CVfF+R63oO085LYOO7oTLyY3EhA1PCXiV09Usl/To66Oh4AVYtz/lPLH
6uoWR/ic5X9gFIXt1t+617+i/L/ILMfqkqN7GzoSbQ8QfyBJrjIikuj575M7AH3o
RuhRYHUTmaCRL+9niDFG/mSVGBxBqiSfW1eS5pGE3hIAcLQd0eFT/38HRdonkfLP
Ug5ihMFSKfKmCIljQ6mm
=jDmS
-----END PGP SIGNATURE-----