[or-cvs] r9885: Reorganize scanned data storage. (in torflow/trunk: . data data/soat)

[mikeperry@xxxxxxxx]

Author: mikeperry
Date: 2007-03-20 01:04:45 -0400 (Tue, 20 Mar 2007)
New Revision: 9885

Added:
   torflow/trunk/data/soat/
   torflow/trunk/data/soat/certs/
   torflow/trunk/data/soat/docs/
   torflow/trunk/data/speedraces/
   torflow/trunk/data/stats/
Removed:
   torflow/trunk/certs/
   torflow/trunk/docs/
Modified:
   torflow/trunk/TODO
   torflow/trunk/metatroller.py
   torflow/trunk/soat.pl
   torflow/trunk/speedracer.pl
Log:
Reorganize scanned data storage.



Modified: torflow/trunk/TODO
===================================================================
--- torflow/trunk/TODO	2007-03-20 04:48:10 UTC (rev 9884)
+++ torflow/trunk/TODO	2007-03-20 05:04:45 UTC (rev 9885)
@@ -56,6 +56,8 @@
      using this scanner
    - Improve checking of changes to documents outside of Tor
    - Improve SSL handling/verification. openssl client is broken.
+     - Also the way we store certs is lame. No need to store so many copies
+       for diff IPs if they are all the same.
    - Parallelize scanning
      - Improve interaction between soat+metatroller so soat knows
        which exit was responsible for a given ip/url

Copied: torflow/trunk/data/soat/certs (from rev 9602, torflow/trunk/certs)

Modified: torflow/trunk/metatroller.py
===================================================================
--- torflow/trunk/metatroller.py	2007-03-20 04:48:10 UTC (rev 9884)
+++ torflow/trunk/metatroller.py	2007-03-20 05:04:45 UTC (rev 9885)
@@ -719,7 +719,7 @@
       s.write("250 OK\r\n")
     elif command == "SAVESTATS":
       if arg: filename = arg
-      else: filename = "./data/stats-"+time.strftime("20%y-%m-%d-%H:%M:%S")
+      else: filename="./data/stats/stats-"+time.strftime("20%y-%m-%d-%H:%M:%S")
       def notlambda(this): this.write_stats(filename)
       h.schedule_low_prio(notlambda)
       s.write("250 OK\r\n")

Modified: torflow/trunk/soat.pl
===================================================================
--- torflow/trunk/soat.pl	2007-03-20 04:48:10 UTC (rev 9884)
+++ torflow/trunk/soat.pl	2007-03-20 05:04:45 UTC (rev 9885)
@@ -10,8 +10,8 @@
 my $META_HOST = "127.0.0.1";
 
 my $WORD_LIST = "./wordlist.txt";
-my $CERT_DIR = "./certs";
-my $DOC_DIR = "./docs";
+my $CERT_DIR = "./data/soat/certs";
+my $DOC_DIR = "./data/soat/docs";
 my $USER_AGENT = "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1) Gecko/20061010 Firefox/2.0";
 #my $USER_AGENT = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)";
 

Modified: torflow/trunk/speedracer.pl
===================================================================
--- torflow/trunk/speedracer.pl	2007-03-20 04:48:10 UTC (rev 9884)
+++ torflow/trunk/speedracer.pl	2007-03-20 05:04:45 UTC (rev 9885)
@@ -246,7 +246,7 @@
         plog "DEBUG", "Reset stats\n";
         speedrace($mcp, $pct, $pct+$PCT_STEP); 
         plog "DEBUG", "speedroced\n";
-        print $mcp "SAVESTATS ./speed/stats-$pct:".($pct+$PCT_STEP)."\r\n";
+        print $mcp "SAVESTATS ./data/speedraces/stats-$pct:".($pct+$PCT_STEP)."\r\n";
         $line = <$mcp>;
         die "Error on SAVESTATS: $line" if (not $line =~ /^250/);
         plog "DEBUG", "Wrote stats\n";