Re: Writing geoip stats to disk on directories

[Karsten Loesing <karsten.loesing@xxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/26/2009 06:55 AM, Roger Dingledine wrote:
> [...] this approach sounds like what we should be doing both for the
> geoip aggregate stats you write to the file and analyze, and also for
> the aggregate stats that bridges collect.

The changes for directories writing stats about their usage to disk are
now in master (commit dfebc88). Right now we have 11 nodes measuring
these stats which should give us some good first insights. I'm going to
analyze these data in about 2 weeks from now.

If people here want to help out with getting some stats on their nodes
(or just want to know how stats are measured), please do the following:

- - git clone git://git.torproject.org/git/tor
- - ./autogen.sh && ./configure --enable-geoip-stats && make
- - possibly make install, or use the executable in src/or/tor
- - configure the node as directory mirror
- - (there are no further configurations necessary to measure geoip stats)
- - wait for 24 hours for the directory to write a file called geoip-stats
to its data directory; this file is extended every 24 hours
- - make the content of the geoip-stats file available to me/this list by
no later than June 13 (but not much earlier to include at least 1 week
of requests, better 10 days)
- - let me/this list know about some basic bandwidth information of your
node: BandwidthRate, BandwidthBurst, and MaxAdvertisedBandwidth if used

The geoip-stats file will contain entries like this:

written 2009-06-02 18:53:02
started-at 2009-06-01 18:53:00
ns-ips us=1088,de=728,fr=392,cn=248,kr=240,it=208,gb=192,se=176,...
ns-v2-ips us=352,de=208,cn=192,kr=104,gb=80,ca=56,fr=56,ru=56,au=40,...
requests-start 2009-06-01 18:53:00
n-ns-reqs us=1184,de=744,fr=408,kr=336,cn=280,it=216,gb=192,se=176,...
n-v2-ns-reqs us=376,de=208,cn=192,kr=144,gb=88,ru=64,ca=56,fr=56,...
v2-ns-share 0.27%
v3-ns-share 0.27%

This node has a configured bandwidth rate of 200 KB/s and burst of 500 KB/s.

Very brief explanation: The ns-[v2-]ips lines show the number of IP
addresses per country seen in the past 24 hours requesting either a v2
or v3 directory. The n-[v2-]ns-reqs show numbers of v2 or v3 directory
requests per country in the past 24 hours which are slightly higher as
the same IP address can request the directory more than once a day. The
v[23]-ns-share lines show the fraction of such requests by already
bootstrapped clients that we should see. written, started-at, and
requests-start should be self-explanatory.

>> The next step would be to add the geoip-stats lines (or a subset of
>> them) to extra-info documents. I think a proposal for that would be in
>> place, but first I think it's fine to start with measuring on a few
>> nodes and working with files.
> 
> You mean for bridges, or for normal relays? I don't think we're at the
> point yet where we want normal relays to turn on the ENABLE_GEOIP_STATS
> compile-time flag. I want to be much more sure that we've got all the
> details right first. (Like, have run it for several months at a test
> relay and have a good handle on exactly what it's getting.)

Yup. The plan is to analyze the geoip-stats files first and see if we
are happy with the kind of results. The next step would be to write a
proposal for having normal relays report aggregate stats, too.

Speaking of, I didn't change the behavior of bridges writing stats to
their extra-info documents. I don't want to mess with that without
knowing for sure what else will be changed in extra-info documents. All
this will go into a proposal to be discussed here.

Best,
- --Karsten

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkomfDgACgkQ0M+WPffBEmXTrACfa3QGe+enSGPjXkk0YNVj8uEW
RR0An1O9dk7UQ49Iru/K+v5CYK6ozXz0
=2JV9
-----END PGP SIGNATURE-----