Consensus Reports List

[Damian Johnson <atagar1@xxxxxxxxx>]

Hi, recently I wrote a script for daily consensus summary emails to help avoid issues like the Planet Labs incident a while back. See the following for an example (if you're viewing it as a plain text email you're gonna be missing most of the content). It's currently available via the list at:
http://groups.google.com/group/tor-consensus-tracker

If you'd like to join but don't have a google account then let me know (it's only necessary for joining via the web interface).

The script that generates it is available at "https://svn.torproject.org/svn/arm/trunk/init/consensusTracker.py". Most of the information is pretty self explanatory, but here's a description:
- The counts are the total number of relays in the category, new are relays with a fingerprint we've never seen before, and bandwidth is the sum of the new relay's observed bandwidths.
- Total bandwidth (the blue column) is the total network capacity (sum of all relay's observed bandwidth) and the deltas throughout the day.
- An exit is defined as any relay with an accept policy before the reject-all.
- I only wanted relays put into a single bucket (no double counting) so exits with the guard flag are counted as exits (not guard) since that tends to me the more important metric for nefarious activity.
- The script sends an alert if 20 new exits or 50 Mbit/s of new exit bandwidth appear between consensuses, otherwise it's just a daily summary. It starts cropping result at a week of data.
- The attached archive has details on the relays that appear in each time period (fingerprints, ip, or port, bandwidth, flags, and the exit polcy) for investigations in case anything looks fishy.

If you have suggestions for improving it then please let me know. Cheers! -Damian

---------- Forwarded message ----------
From: <atagar1@xxxxxxxxx>
Date: Sun, Aug 8, 2010 at 4:21 PM
Subject: Daily Consensus Report for 08/08/2010
To: atagar1@xxxxxxxxx

Greetings from your friendly consensus monitoring daemon. At present there's no breaches to report. See below for a summary of consensus additions.

---

	Guards			Middle			Exits			Total
Date:	Count:	New:	Bandwidth:	Count:	New:	Bandwidth:	Count:	New:	Bandwidth:	Bandwidth:
2010-08-08	315	0	0 bits/s	624	27	8 MBit/s	921	115	27 MBit/s	4.82 GBit/s
23:00:00	305	0	0 bits/s	628	0	0 bits/s	911	3	739 KBit/s	-21 MBit/s
22:00:00	319	0	0 bits/s	626	2	475 KBit/s	956	11	3 MBit/s	-24 MBit/s
21:00:00	334	0	0 bits/s	625	0	0 bits/s	1014	6	826 KBit/s	+27 MBit/s
20:00:00	333	0	0 bits/s	629	2	424 KBit/s	1025	6	2 MBit/s	+11 MBit/s
19:00:00	337	0	0 bits/s	627	1	51 KBit/s	1007	7	1 MBit/s	-36 MBit/s
18:00:00	338	0	0 bits/s	630	1	685 KBit/s	1001	4	1 MBit/s	+7 MBit/s
17:00:00	343	0	0 bits/s	623	5	2 MBit/s	1017	2	0 bits/s	+3 MBit/s
16:00:00	336	0	0 bits/s	627	3	591 KBit/s	994	5	1 MBit/s	+16 MBit/s
15:00:00	332	0	0 bits/s	622	1	416 KBit/s	966	6	1 MBit/s	-28 MBit/s
14:00:00	334	0	0 bits/s	622	2	731 KBit/s	986	0	0 bits/s	+23 MBit/s
13:00:00	335	0	0 bits/s	619	0	0 bits/s	974	7	2 MBit/s	-6 MBit/s
12:00:00	334	0	0 bits/s	624	0	0 bits/s	963	4	1 MBit/s	+6 MBit/s
11:00:00	329	0	0 bits/s	619	2	609 KBit/s	944	8	2 MBit/s	+14 MBit/s
10:00:00	315	0	0 bits/s	623	1	0 bits/s	926	3	512 KBit/s	-7 MBit/s
09:00:00	305	0	0 bits/s	618	1	275 KBit/s	898	3	279 KBit/s	+10 MBit/s
08:00:00	305	0	0 bits/s	623	0	0 bits/s	857	5	1 MBit/s	+4 MBit/s
07:00:00	292	0	0 bits/s	624	0	0 bits/s	853	5	1 MBit/s	+10 MBit/s
06:00:00	287	0	0 bits/s	625	1	90 KBit/s	825	10	3 MBit/s	-5 MBit/s
05:00:00	286	0	0 bits/s	629	0	0 bits/s	808	4	750 KBit/s	-4 MBit/s
04:00:00	286	0	0 bits/s	625	0	0 bits/s	814	3	813 KBit/s	+751 KBit/s
03:00:00	286	0	0 bits/s	631	2	417 KBit/s	813	4	1 MBit/s	-34 MBit/s
02:00:00	288	0	0 bits/s	625	1	576 KBit/s	827	1	420 KBit/s	-10 MBit/s
01:00:00	302	0	0 bits/s	617	1	165 KBit/s	854	3	576 KBit/s	+3 MBit/s
00:00:00	303	0	0 bits/s	620	1	447 KBit/s	873	5	1 MBit/s
2010-08-07	313	0	0 bits/s	617	24	6 MBit/s	914	102	23 MBit/s	4.85 GBit/s
23:00:00	304	0	0 bits/s	633	1	419 KBit/s	888	6	999 KBit/s	-13 MBit/s
22:00:00	309	0	0 bits/s	635	1	441 KBit/s	922	4	160 KBit/s	-22 MBit/s
21:00:00	316	0	0 bits/s	625	2	268 KBit/s	955	10	2 MBit/s	-199 KBit/s
20:00:00	320	0	0 bits/s	617	2	664 KBit/s	955	4	1 MBit/s	-2 MBit/s
19:00:00	332	0	0 bits/s	614	0	0 bits/s	969	7	1 MBit/s	+10 MBit/s
18:00:00	329	0	0 bits/s	612	0	0 bits/s	969	4	875 KBit/s	-4 MBit/s
17:00:00	331	0	0 bits/s	608	2	341 KBit/s	993	6	2 MBit/s	+8 MBit/s
16:00:00	329	0	0 bits/s	607	0	0 bits/s	981	7	2 MBit/s	-18 MBit/s
15:00:00	332	0	0 bits/s	616	1	493 KBit/s	984	8	952 KBit/s	-2 MBit/s
14:00:00	329	0	0 bits/s	620	2	432 KBit/s	973	4	1 MBit/s	-688 KBit/s
13:00:00	328	0	0 bits/s	614	1	0 bits/s	961	1	410 KBit/s	+4 MBit/s
12:00:00	321	0	0 bits/s	617	1	209 KBit/s	964	4	916 KBit/s	-14 MBit/s
11:00:00	322	0	0 bits/s	609	0	0 bits/s	956	3	714 KBit/s	+9 MBit/s
10:00:00	326	0	0 bits/s	613	1	289 KBit/s	938	9	3 MBit/s	+14 MBit/s
09:00:00	315	0	0 bits/s	623	2	397 KBit/s	910	2	385 KBit/s	+16 MBit/s
08:00:00	307	0	0 bits/s	613	0	0 bits/s	880	4	1 MBit/s	+5 MBit/s
07:00:00	300	0	0 bits/s	619	1	381 KBit/s	856	2	454 KBit/s	+337 KBit/s
06:00:00	299	0	0 bits/s	617	2	713 KBit/s	844	3	1 MBit/s	+979 KBit/s
05:00:00	286	0	0 bits/s	621	2	885 KBit/s	832	5	764 KBit/s	+9 MBit/s
04:00:00	297	0	0 bits/s	606	2	160 KBit/s	820	1	578 KBit/s	-7 MBit/s
03:00:00	287	0	0 bits/s	619	0	0 bits/s	826	4	1 MBit/s	-45 MBit/s
02:00:00	299	0	0 bits/s	615	1	0 bits/s	837	1	440 KBit/s	-4 MBit/s
01:00:00	301	0	0 bits/s	614	0	0 bits/s	868	2	423 KBit/s	+1 MBit/s
00:00:00	301	0	0 bits/s	626	0	0 bits/s	874	1	0 bits/s
2010-08-06	315	0	0 bits/s	610	21	4 MBit/s	913	88	19 MBit/s	4.90 GBit/s
23:00:00	310	0	0 bits/s	629	0	0 bits/s	909	4	387 KBit/s	-8 MBit/s
22:00:00	319	0	0 bits/s	628	1	0 bits/s	920	6	2 MBit/s	-10 MBit/s
21:00:00	326	0	0 bits/s	614	1	272 KBit/s	951	9	2 MBit/s	-28 MBit/s
20:00:00	335	0	0 bits/s	609	1	61 KBit/s	985	7	1 MBit/s	+2 MBit/s
19:00:00	336	0	0 bits/s	608	3	733 KBit/s	984	1	236 KBit/s	+13 MBit/s
18:00:00	331	0	0 bits/s	606	1	160 KBit/s	982	4	1 MBit/s	-8 MBit/s
17:00:00	332	0	0 bits/s	611	5	600 KBit/s	975	6	904 KBit/s	-21 MBit/s
16:00:00	327	0	0 bits/s	603	1	160 KBit/s	978	7	631 KBit/s	-17 MBit/s
15:00:00	329	0	0 bits/s	607	0	0 bits/s	966	0	0 bits/s	+1 MBit/s
14:00:00	326	0	0 bits/s	615	1	182 KBit/s	958	3	0 bits/s	-275 KBit/s
13:00:00	323	0	0 bits/s	611	1	0 bits/s	936	3	432 KBit/s	+3 MBit/s
12:00:00	322	0	0 bits/s	610	1	408 KBit/s	935	6	1 MBit/s	+5 MBit/s
11:00:00	313	0	0 bits/s	612	2	446 KBit/s	921	7	2 MBit/s	+16 MBit/s
10:00:00	314	0	0 bits/s	601	0	0 bits/s	909	4	861 KBit/s	-5 MBit/s
09:00:00	309	0	0 bits/s	603	1	418 KBit/s	894	9	2 MBit/s	+6 MBit/s
08:00:00	299	0	0 bits/s	610	0	0 bits/s	844	2	579 KBit/s	+8 MBit/s
07:00:00	297	0	0 bits/s	610	1	529 KBit/s	831	2	423 KBit/s	-517 KBit/s
06:00:00	298	0	0 bits/s	605	1	160 KBit/s	819	2	637 KBit/s	-13 MBit/s
05:00:00	286	0	0 bits/s	611	0	0 bits/s	815	3	203 KBit/s	+2 MBit/s
04:00:00	296	0	0 bits/s	602	0	0 bits/s	821	3	870 KBit/s	-32 MBit/s
03:00:00	297	0	0 bits/s	606	0	0 bits/s	844	0	0 bits/s

Attachment: newRelays.tar.gz
Description: Binary data