[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[or-cvs] r15462: Added an interface for monitoring soat test results. Minor a (in torflow/branches/gsoc2008: . data/soat data/soat/http data/soat/ssh data/soat/ssl data/soat/ssl/certs)



Author: aleksei
Date: 2008-06-25 14:30:58 -0400 (Wed, 25 Jun 2008)
New Revision: 15462

Added:
   torflow/branches/gsoc2008/data/soat/http/
   torflow/branches/gsoc2008/data/soat/http/failed/
   torflow/branches/gsoc2008/data/soat/http/inconclusive/
   torflow/branches/gsoc2008/data/soat/http/successful/
   torflow/branches/gsoc2008/data/soat/http/tags/
   torflow/branches/gsoc2008/data/soat/ssh/
   torflow/branches/gsoc2008/data/soat/ssh/failed/
   torflow/branches/gsoc2008/data/soat/ssh/inconclusive/
   torflow/branches/gsoc2008/data/soat/ssh/successful/
   torflow/branches/gsoc2008/data/soat/ssl/failed/
   torflow/branches/gsoc2008/data/soat/ssl/inconclusive/
   torflow/branches/gsoc2008/data/soat/ssl/successful/
   torflow/branches/gsoc2008/soatstats.py
Removed:
   torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem
   torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem
   torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem
   torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem
   torflow/branches/gsoc2008/data/soat/ssl/nodesPositive/
   torflow/branches/gsoc2008/data/soat/ssl/nodesResults/
Modified:
   torflow/branches/gsoc2008/soat.py
Log:
Added an interface for monitoring soat test results. Minor adjustments in the scanner.

Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem	2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/addons.mozilla.org.pem	2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDAjCCAmugAwIBAgIDCG47MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
-MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
-aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDcxMjEwMTgwMjMzWhcNMDkxMjEwMTgwMjMz
-WjCBjDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcT
-DU1vdW50YWluIFZpZXcxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xGjAY
-BgNVBAsTEVNlY3VyZSBXZWIgU2VydmVyMRYwFAYDVQQDFA0qLm1vemlsbGEub3Jn
-MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQChC0WH3YhyVVYKWEjAfYT0q19W
-c89J2McBEX9UvBUX5vOujghbDdyDTahVSf5sgkX54mVDoLq/rv28fd8Op8Wf54ZU
-oN/n+05M/F3+j52hja3UysHgtEsCCiTUd73oWTh0GiwNhbchqGWo52rs5xKGxaEQ
-HhW+gJFC2jrEYsRzbQIDAQABo4GuMIGrMA4GA1UdDwEB/wQEAwIE8DAdBgNVHQ4E
-FgQU441V52B+eQnnlTrlE0IcnvaS77QwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDov
-L2NybC5nZW90cnVzdC5jb20vY3Jscy9zZWN1cmVjYS5jcmwwHwYDVR0jBBgwFoAU
-SOZo+SvSspXXR9gjIBBPM5iQn9QwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
-BwMCMA0GCSqGSIb3DQEBBQUAA4GBAD7DR9FVBTK8jxLKebbAIetW95SIc9r1Nl5/
-PecjkP6RfKHbnCQ32j213HC9nZZR8nwCGgS1ryTKp/l0HFVWuF1L/y2fLEU7mzVP
-3cwNb/QYgOvNRh2+VDTP+rXGnLmm+MNvQoqzQNJrUlta8qxitaUFk/AL/anne8dp
-f+07Xq2p
------END CERTIFICATE-----

Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem	2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/mail.google.com.pem	2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDIjCCAougAwIBAgIQbldpChBPqv+BdPg4iwgN8TANBgkqhkiG9w0BAQUFADBM
-MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
-THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wODA1MDIxNjMyNTRaFw0w
-OTA1MDIxNjMyNTRaMGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
-MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRgw
-FgYDVQQDEw9tYWlsLmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
-AoGBALlkxdh2QXegdElukCSOV2+8PKiONIS+8Tu9K7MQsYpqtLNC860zwOPQ2NLI
-3Zp4jwuXVTrtzGuiqf5Jioh35Ig3CqDXtLyZoypjZUQcq4mlLzHlhIQ4EhSjDmA7
-Ffw9y3ckSOQgdBQWNLbquHh9AbEUjmhkrYxIqKXeCnRKhv6nAgMBAAGjgecwgeQw
-KAYDVR0lBCEwHwYIKwYBBQUHAwEGCCsGAQUFBwMCBglghkgBhvhCBAEwNgYDVR0f
-BC8wLTAroCmgJ4YlaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVNHQ0NBLmNy
-bDByBggrBgEFBQcBAQRmMGQwIgYIKwYBBQUHMAGGFmh0dHA6Ly9vY3NwLnRoYXd0
-ZS5jb20wPgYIKwYBBQUHMAKGMmh0dHA6Ly93d3cudGhhd3RlLmNvbS9yZXBvc2l0
-b3J5L1RoYXd0ZV9TR0NfQ0EuY3J0MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEF
-BQADgYEAsRwpLg1dgCR1gYDK185MFGukXMeQFUvhGqF8eT/CjpdvezyKVuz84gSu
-6ccMXgcPQZGQN/F4Xug+Q01eccJjRSVfdvR5qwpqCj+6BFl5oiKDBsveSkrmL5dz
-s2bn7TdTSYKcLeBkjXxDLHGBqLJ6TNCJ3c4/cbbG5JhGvoema94=
------END CERTIFICATE-----

Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem	2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/www.fastmail.fm.pem	2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,21 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDfDCCAuWgAwIBAgIQbfxRUFCOzMrPD9Y7lr8MaTANBgkqhkiG9w0BAQUFADCB
-zjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ
-Q2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UE
-CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhh
-d3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNl
-cnZlckB0aGF3dGUuY29tMB4XDTA4MDUxNDIyMjkyMloXDTA5MDUyMDA5MjE0Nlow
-gYAxCzAJBgNVBAYTAkFVMRgwFgYDVQQIEw9OZXcgU291dGggV2FsZXMxEzARBgNV
-BAcTCkNyb3dzIE5lc3QxKDAmBgNVBAoTH09wdGltYWwgRGVjaXNpb25zIEdyb3Vw
-IFB0eSBMdGQxGDAWBgNVBAMTD3d3dy5mYXN0bWFpbC5mbTCBnzANBgkqhkiG9w0B
-AQEFAAOBjQAwgYkCgYEAyJMTIn/14CHIrAYGi3913dMk1khF1C1M4f6/JlmIv7Xl
-HlOT+GJNcal38wsS/VxXTSrKgPcXDDK0kYGg/1WU6wF8HJ64LEXHou558PDBDnnp
-4pw2ayB5mVy9E3YFez0a/NHEBDKrkngmQmWQxmcVzBLNDYhoR/NQPQ6JZjtR1kMC
-AwEAAaOBpjCBozAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwQAYDVR0f
-BDkwNzA1oDOgMYYvaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVByZW1pdW1T
-ZXJ2ZXJDQS5jcmwwMgYIKwYBBQUHAQEEJjAkMCIGCCsGAQUFBzABhhZodHRwOi8v
-b2NzcC50aGF3dGUuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEA
-IKus8RSbHoHfE36J44uEyGPMxFnfaEZLoZs+k6aZPnfVyvhDwNKPmfoEkaFWtqQ1
-Q+5IzVWGXCQOd5R4bWiiJRhG33KCDmRfqnVrCX0X+V6N72e9Zw51ac1Tv+Noipan
-s/yFWHIOod8fw6ELGcXzOd5mEt5XlCFWm/2IKtRCG9E=
------END CERTIFICATE-----

Deleted: torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem
===================================================================
--- torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem	2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/data/soat/ssl/certs/www.paypal.com.pem	2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,34 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIF5jCCBM6gAwIBAgIQbmuco/dSNbSVN4bU5RNUqTANBgkqhkiG9w0BAQUFADCB
-vjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
-ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug
-YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykwNjE4MDYGA1UEAxMv
-VmVyaVNpZ24gQ2xhc3MgMyBFeHRlbmRlZCBWYWxpZGF0aW9uIFNTTCBTR0MgQ0Ew
-HhcNMDcwMTMwMDAwMDAwWhcNMDkwMTI5MjM1OTU5WjCCAR4xEDAOBgNVBAUTBzMw
-MTQyNjcxEzARBgsrBgEEAYI3PAIBAxMCVVMxGTAXBgsrBgEEAYI3PAIBAhMIRGVs
-YXdhcmUxCzAJBgNVBAYTAlVTMRMwEQYDVQQRFAo5NTEzMS0yMDIxMQswCQYDVQQI
-EwJDQTERMA8GA1UEBxQIU2FuIEpvc2UxFjAUBgNVBAkUDTIyMTEgTiAxc3QgU3Qx
-FDASBgNVBAoUC1BheXBhbCBJbmMuMRwwGgYDVQQLFBNJbmZvcm1hdGlvbiBTeXN0
-ZW1zMTMwMQYDVQQLFCpUZXJtcyBvZiB1c2UgYXQgd3d3LnZlcmlzaWduLmNvbS9y
-cGEgKGMpMDYxFzAVBgNVBAMUDnd3dy5wYXlwYWwuY29tMIGfMA0GCSqGSIb3DQEB
-AQUAA4GNADCBiQKBgQC0ldZuxq9UVX26JfEnMM+U7pM+husarblH+rS18AyT8Aqf
-0oOZJ+439TESAQ521/9zDqXYpm5++VoNN5/M3HbQ63ksuqDILhA+G1sU0jA9RD38
-z992oazPJQDTfmeK+amv50+4okUT8QTDkb9WHjUI/gpf14AYQ628UFQQGO6WlQID
-AQABo4IB/zCCAfswCQYDVR0TBAIwADAdBgNVHQ4EFgQU7VBMXqBNGpJwm/AUUdLW
-nWziydAwCwYDVR0PBAQDAgWgMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9FVklu
-dGwtY3JsLnZlcmlzaWduLmNvbS9FVkludGwyMDA2LmNybDBEBgNVHSAEPTA7MDkG
-C2CGSAGG+EUBBxcGMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWdu
-LmNvbS9ycGEwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMCBglghkgBhvhC
-BAEGCisGAQQBgjcKAwMwHwYDVR0jBBgwFoAUTkPIHXbvN1N6T/JYb5TzOOLVvd8w
-dgYIKwYBBQUHAQEEajBoMCsGCCsGAQUFBzABhh9odHRwOi8vRVZJbnRsLW9jc3Au
-dmVyaXNpZ24uY29tMDkGCCsGAQUFBzAChi1odHRwOi8vRVZJbnRsLWFpYS52ZXJp
-c2lnbi5jb20vRVZJbnRsMjAwNi5jZXIwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcw
-VRYJaW1hZ2UvZ2lmMCEwHzAHBgUrDgMCGgQUj+XTGoasjY5rw8+AatRIGCx7GS4w
-JRYjaHR0cDovL2xvZ28udmVyaXNpZ24uY29tL3ZzbG9nby5naWYwDQYJKoZIhvcN
-AQEFBQADggEBADGj5/xJrl7mzkgcE6zf44jzT0Iw+/BLAzcerNixC3xZ57rPPhxu
-WRMKVzgIDYQzu/ed2oLRz6eefjF2Xs9XqLK1RDYRyDldksxP/AxB52AEGoBbufQ1
-xKWziRJXaUmqA27FBbx89lYrj3m6VQkkJ7bK+IpP9vQzpg86ZBQs0UWtxsQbO9zp
-D8IaeI+K97d5CgRfXixgfUR3OttmhofzmR9Zfawcui+MoEu0lrjeZNgAkEaCVlcH
-m0iL1rOpmhX/7b5DUpTWN3jopw9/VQTCqxcu+0zoEcNU02yadEVCNumWAEllTJ5D
-ePLh8FkejTsIkNexlknGZX4eQJYBo90wUok=
------END CERTIFICATE-----

Modified: torflow/branches/gsoc2008/soat.py
===================================================================
--- torflow/branches/gsoc2008/soat.py	2008-06-25 17:14:32 UTC (rev 15461)
+++ torflow/branches/gsoc2008/soat.py	2008-06-25 18:30:58 UTC (rev 15462)
@@ -1,5 +1,6 @@
 #!/usr/bin/python
 import httplib
+import md5
 import os
 import random
 import re
@@ -12,8 +13,10 @@
 import urllib
 import urllib2
 
+import soatstats
+from soatstats import *
+
 from TorCtl import TorUtil, TorCtl, PathSupport
-
 from TorCtl.TorUtil import meta_port, meta_host, control_port, control_host, tor_port, tor_host
 from TorCtl.TorUtil import *
 from TorCtl.PathSupport import *
@@ -28,7 +31,7 @@
 # Try to use system openssl first
 try:
   from OpenSSL import *
-except e:
+except:
   sys.path.append("./tools/")
   from OpenSSL import *
 
@@ -47,15 +50,16 @@
 
 same_origin_policy = True
 
-ssl_certs_directory = './data/soat/ssl/certs/';
-ssl_nodes_results_directory = './data/soat/ssl/nodesResults/';
-ssl_nodes_positive_directory = './data/soat/ssl/nodesPositive/';
+#
+# data storage
+#
 
+
 #
 # links of interest
 #
 
-docs_http = ['http://www.torproject.org']
+docs_http = ['http://www.torproject.org','http://www.math.ut.ee','http://www.mozilla.com']
 docs_https = ['mail.google.com','addons.mozilla.org','www.paypal.com','www.fastmail.fm']
 
 #
@@ -98,21 +102,6 @@
             s = s[:-1]
         return s
 
-# a class for saving ssl test results
-class OpenSSLTestResult:
-    def __init__(self, exit_node, ssl_site, cert_file):
-        self.exit_node = exit_node
-        self.ssl_site = ssl_site
-        self.timestamp = time.time()
-        self.cert = cert_file
-
-# a class for saving http test results
-class HttpTestResult:
-    def __init__(self, exit_node, website):
-        self.exit_node = exit_node
-        self.website = website
-        self.timestamp = time.time()
-
 # The scanner class
 class ExitNodeScanner:
 
@@ -222,7 +211,8 @@
         socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, tor_host, tor_port)
         socket.socket = socks.socksocket
 
-        plog('INFO', 'Opening ' + address + ' using exit node ' + self.get_exit_node())
+        exit_node = self.get_exit_node()
+        plog('INFO', 'Opening ' + address + ' using exit node ' + exit_node)
         try:
             g = urllib2.urlopen(request)
         except Exception, e:
@@ -232,7 +222,7 @@
             return 0
 
         pcontent = g.read()
-        pcontent.decode('ascii', 'ignore')
+        pcontent = pcontent.decode('ascii', 'ignore')
 
         node_page = BeautifulSoup(pcontent)
 
@@ -241,7 +231,27 @@
 
         # nasty tags: a, applet, div, embed, form, frame, iframe, img, link, script
         # also check DOM event stuff
+        
+        # address[7:] means we're leaving out the 'http://' from the address part
+        #
+        # just some temporary stuff (compare the whole content) that will be replaced soon
+        #
+        if md5.new(pcontent).digest() != md5.new(content).digest():
+            plog('ERROR', 'The retrieved pages dont match')
+            
+            content_file_handle = open(http_tags_dir + address[7:] + '.result','w')
+            content_file_handle.write(content)
+            content_file_handle.close()
 
+            pcontent_file_handle = open(http_tags_dir + `exit_node` + '_' + address[7:] + '.result','w')
+            pcontent_file_handle.write(pcontent)
+            pcontent_file_handle.close()
+
+        else:
+            result = HttpTestResult(exit_node, address, 0, TEST_SUCCESS)
+            result_file = open(http_s_dir + `exit_node` + '_' + address[7:] + '.result','w')
+            pickle.dump(result, result_file)
+
         return 0
 
     def check_openssh(self, address):
@@ -268,16 +278,21 @@
         c.set_connect_state()
 
         plog('INFO', 'Opening a direct ssl connection to ' + address)
+        
+        try:
+            c.connect((address, 443))
+            c.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+        except socket.error, e:
+            plog('ERROR','An error occured while opening a direct ssl connection to ' + address)
+            plog('ERROR', e)
+            return 0
 
-        c.connect((address, 443))
-        c.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
-
         cert = c.get_peer_certificate()
         cert_pem = crypto.dump_certificate(crypto.FILETYPE_PEM, cert)
 
         # save the cert 
 
-        cert_file_handle = open(ssl_certs_directory + address + '.pem', 'w')
+        cert_file_handle = open(ssl_certs_dir + address + '.pem', 'w')
         cert_file_handle.write(cert_pem)
         cert_file_handle.close()
 
@@ -289,7 +304,7 @@
         # check whether we already have a circuit.
         # if yes, open a connection via tor, otherwise skip to the next test
         exit_node = self.get_exit_node()
-        if exit_node == 0:
+        if exit_node == 0 or exit_node == '0':
             plog('INFO', 'We have no exit node to test, skipping to the next test.')
             return 0
 
@@ -303,8 +318,23 @@
  
         plog('INFO', 'Opening an ssl connection to ' + address + ' using exit node ' + `exit_node`)
 
-        c2.connect((address, 443))
-        c2.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+        try:
+            c2.connect((address, 443))
+        except Exception, e:
+            plog('ERROR','An error occured while opening a direct ssl connection to ' + address)
+            plog('ERROR', e)
+            return 0
+        
+        try:
+            c2.send(crypto.dump_certificate_request(crypto.FILETYPE_ASN1,request))
+        except Exception, e:
+            plog('ERROR', 'Got the following ssl error while connecting via tor:')
+            plog('ERROR', e)
+            cert_file = ssl_certs_dir + address + '.pem'
+            result = OpenSSLTestResult(exit_node, address, cert_file, TEST_INCONCLUSIVE)
+            result_file = open(ssl_i_dir + `exit_node` + '_' + address + '.result','w')
+            pickle.dump(result, result_file)
+            return 0
 
         cert2 = c2.get_peer_certificate()
         cert2_pem = crypto.dump_certificate(crypto.FILETYPE_PEM, cert2)
@@ -317,20 +347,20 @@
         # save the received cert for inspection
 
         if cert_pem == cert2_pem:
-            cert_file = ssl_certs_directory + address + '.pem'
-            result = OpenSSLTestResult(exit_node, address, cert_file)
-            result_file = open(ssl_nodes_results_directory + `exit_node` + '_' + address + '.result','w')
+            cert_file = ssl_certs_dir + address + '.pem'
+            result = OpenSSLTestResult(exit_node, address, cert_file, TEST_SUCCESS)
+            result_file = open(ssl_s_dir + `exit_node` + '_' + address + '.result','w')
             pickle.dump(result, result_file)
         else:
             plog('ERROR', 'Exit node ' + `exit_node` + ' seems to be meddling with certificates. (' + address + ')')
 
-            cert_file = ssl_certs_directory + address + '_' + `exit_node` + '.pem'
+            cert_file = ssl_certs_dir + address + '_' + `exit_node` + '.pem'
             cert_file_handle = open(cert_file, 'w')
             cert_file_handle.write(cert2_pem)
             cert_file_handle.close()
 
-            result = OpenSSLTestResult(exit_node, address, cert_file)
-            result_file = open(ssl_nodes_positive_directory + `exit_node` + '_' + address + '.result','w')
+            result = OpenSSLTestResult(exit_node, address, cert_file, TEST_FAILURE)
+            result_file = open(ssl_f_dir + `exit_node` + '_' + address + '.result','w')
             pickle.dump(result, result_file)
 
         plog('INFO', 'Test complete. Moving on...')
@@ -370,18 +400,6 @@
          
     return urllist
 
-def load_cert():
-    filehandler = open('./data/soat/ssl/certs/addons.mozilla.org.pem','r')
-    string = filehandler.read()
-
-    ctx = SSL.Context(SSL.SSLv23_METHOD)
-    ctx.use_certificate_file('./data/soat/ssl/certs/addons.mozilla.org.pem')
-
-    cert = crypto.load_certificate(crypto.FILETYPE_PEM, string)
-    print cert.get_subject()
-
-    return 0
-
 '''
 Find links to files related to a query
 '''
@@ -433,6 +451,8 @@
     # consistency test
     # scanner.check_all_exits_port_consistency()
     
+    while 1:
+
     # find sites for http testing if necessary
     #
     # global doc_urls
@@ -441,12 +461,12 @@
     # plog('NOTICE', 'Final URL list: ' + '\n'.join(doc_urls) + '\n')
     
     # https test
-    for ssl_site in docs_https:
-        scanner.check_openssl(ssl_site)
-
+        for ssl_site in docs_https:
+            scanner.check_openssl(ssl_site)
+    
     # http test
-    # for http_site in docs_http:
-    #   scanner.check_http(http_site)
+        for http_site in docs_http:
+            scanner.check_http(http_site)
 
 #
 # initiate the program

Added: torflow/branches/gsoc2008/soatstats.py
===================================================================
--- torflow/branches/gsoc2008/soatstats.py	                        (rev 0)
+++ torflow/branches/gsoc2008/soatstats.py	2008-06-25 18:30:58 UTC (rev 15462)
@@ -0,0 +1,440 @@
+import operator
+import os
+import pickle
+import sys
+import time
+
+import sets
+from sets import Set
+
+#
+# Data storage
+#
+
+# data locations
+
+data_dir = './data/soat/'
+
+ssl_certs_dir = data_dir + 'ssl/certs/'
+ssl_s_dir = data_dir + 'ssl/successful/'
+ssl_f_dir = data_dir + 'ssl/failed/'
+ssl_i_dir = data_dir + 'ssl/inconclusive/'
+
+http_tags_dir = data_dir + 'http/tags/'
+http_s_dir = data_dir + 'http/successful/'
+http_f_dir = data_dir + 'http/failed/'
+http_i_dir = data_dir + '/http/inconclusive/'
+
+ssh_s_dir = data_dir + 'ssh/successful/'
+ssh_f_dir = data_dir + 'ssh/failed/'
+ssh_i_dir = data_dir + 'ssh/inconclusive/'
+
+result_dirs = [ssl_s_dir, ssl_f_dir, ssl_i_dir,
+               http_s_dir, http_f_dir, http_i_dir]
+
+# classes to use with pickle to dump test results into files
+
+TEST_SUCCESS = 0
+TEST_INCONCLUSIVE = 1
+TEST_FAILURE = 2
+
+class TestResult(object):
+    def __init__(self, exit_node, site, status):
+        self.exit_node = exit_node
+        self.site = site
+        self.timestamp = time.time()
+        self.status = status
+
+class OpenSSLTestResult(TestResult):
+    def __init__(self, exit_node, ssl_site, cert_file, status):
+        super(OpenSSLTestResult, self).__init__(exit_node, ssl_site, status)
+        self.cert = cert_file
+
+class HttpTestResult(TestResult):
+    def __init__(self, exit_node, website, tag_prints, status):
+        super(HttpTestResult, self).__init__(exit_node, website, status)
+        self.tag_prints = tag_prints
+
+class OpenSSHTestResult(TestResult):
+    def __init__(self, exit_node, ssh_site, status):
+        super(OpenSSHTestResult, self).__init__(exit_node, ssh_site, status)
+
+#
+# Class for saving and managing data
+#
+class DataHandler():
+
+    # function for filtering results based on protocol and success level
+    def filterResults(self, results, 
+            show_ssh, show_http, show_ssl, 
+            show_good, show_bad, show_unsure):
+
+        filters = []
+        if not show_ssh:
+            filters.append(lambda x: not x.__class__.__name__ == 'OpenSSHTestResult')
+        if not show_ssl:
+            filters.append(lambda x: not x.__class__.__name__ == 'OpenSSLTestResult')
+        if not show_http:
+            filters.append(lambda x: not x.__class__.__name__ == 'HttpTestResult')
+        if not show_good:
+            filters.append(lambda x: not x.status == TEST_SUCCESS)
+        if not show_bad:
+            filters.append(lambda x: not x.status == TEST_FAILURE)
+        if not show_unsure:
+            filters.append(lambda x: not x.status == TEST_INCONCLUSIVE) 
+
+        filtered = []
+        if len(filters) > 0:
+            filter = lambda x: reduce(operator.__and__, [f(x) for f in filters]) 
+            filtered = [x for x in results if filter(x)]
+        else:
+            filtered = list(results)
+
+        return filtered
+
+    # get all available results
+    def getAll(self):
+        results = []
+
+        for dir in result_dirs:
+            files = os.listdir(dir)
+            for file in files:
+                fh = open(dir + file,'r')
+                result = pickle.load(fh)
+                results.append(result)
+
+        return results
+    
+    # filter by node
+    def filterByNode(self, results, id):
+        return filter(lambda x: x.exit_node == id, results)
+
+#
+# Displaying stats on the console
+#
+
+#
+# Class to display statistics on the console
+#
+# It can show the summary 
+#
+class StatsConsole():
+    def Listen(self):
+        while 1:
+            input = raw_input(">>>")
+            if input == 'e' or input == 'exit':
+                exit()
+            elif input == 's' or input == 'summary':
+                self.Summary()
+            elif input == 'h' or input == 'help' or len(input) > 6:
+                self.Help() 
+            else:
+                self.Reply(input)
+    
+    def Summary(self):
+        dh = DataHandler()
+        data = dh.getAll()
+        nodeSet = Set([])
+
+        total = len(data)
+        good = 0
+        bad = 0
+        inconclusive = 0
+        ssh = 0
+        http = 0
+        ssl = 0
+
+        for result in data:
+            nodeSet.add(result.exit_node)
+            
+            if result.status == 0:
+                good += 1
+            elif result.status == 1:
+                inconclusive += 1
+            elif result.status == 2:
+                bad += 1
+            
+            if result.__class__.__name__ == 'OpenSSHTestResult':
+                ssh += 1
+            elif result.__class__.__name__ == 'HttpTestResult':
+                http += 1
+            elif result.__class__.__name__ == 'OpenSSLTestResult':
+                ssl += 1
+
+        swidth = 25
+        nwidth = 10
+        width = swidth + nwidth
+
+        header_format = '%-*s%*s'
+        format = '%-*s%*i'
+
+        print '=' * width
+        print header_format % (swidth, 'Parameter', nwidth, 'Count')
+        print '-' * width
+
+        print format % (swidth, 'Tests completed', nwidth, total)
+        print format % (swidth, 'Nodes tested', nwidth, len(nodeSet))
+        print format % (swidth, 'Failed tests', nwidth, bad)
+        print format % (swidth, 'Succeeded tests', nwidth, good)
+        print format % (swidth, 'Inconclusive tests', nwidth, inconclusive)
+        print format % (swidth, 'SSH tests', nwidth, ssh)
+        print format % (swidth, 'HTTP tests', nwidth, http)
+        print format % (swidth, 'SSL tests', nwidth, ssl)
+
+        print '=' * width
+
+    def Reply(self, input):
+
+        ssh = False 
+        http = False 
+        ssl = False 
+        good = False 
+        bad = False
+        inconclusive = False
+
+        if 'a' in input:
+            ssh = http = ssl = good = bad = inconclusive = True
+        else:
+            if 's' in input:
+                ssh = True
+            if 'h' in input:
+                http = True
+            if 'l' in input:
+                ssl = True
+            if 'g' in input:
+                good = True
+            if 'b' in input:
+                bad = True
+            if 'i' in input:
+                inconclusive = True
+
+        dh = DataHandler()
+        data = dh.getAll()
+        filtered = dh.filterResults(data, ssh, http, ssl, good, bad, inconclusive)
+
+        nodewidth = 45
+        typewidth = 10
+        sitewidth = 30
+        timewidth = 30
+        statuswidth = 6
+        width = nodewidth + typewidth + sitewidth + timewidth + statuswidth
+
+        format = '%-*s%-*s%-*s%-*s%-*s'
+
+        print '=' * width 
+        print format % (nodewidth, 'Exit node', typewidth, 'Test type', sitewidth, 'Remote site', 
+                timewidth, 'Time', statuswidth, 'Status')
+        print '-' * width
+        for result in filtered:
+            print format % (nodewidth, `result.exit_node`, 
+                    typewidth, result.__class__.__name__[:-10],
+                    sitewidth, result.site, 
+                    timewidth, time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime(result.timestamp)), 
+                    statuswidth, `result.status`)
+        print '=' * width
+
+    def Help(self):
+        print ''
+        print 'Options:'
+        print '* summmary (s) - display a short summary about all tests done so far'
+        print '* exit (e) - terminate the program'
+        print '* help (h) - display this help text'
+        print '* all (a) - list all the results'
+        print '* (shlgbi) - display a filtered list of test results. Letters are optional and mean the following:'
+        print '  s - show ssh results'
+        print '  h - show http results'
+        print '  l - show ssl results'
+        print '  g - show good results'
+        print '  b - show bad results'
+        print '  i - show inconclusive results'
+        print ''
+
+#
+# Displaying stats in a graphical setting (first check if we have wx)
+#
+
+nowx = False
+try:
+    import wx
+    from wx.lib.mixins.listctrl import ListCtrlAutoWidthMixin, ColumnSorterMixin
+except:
+    nowx = True
+
+if not nowx:
+
+    # The list model to be used for the result listing
+    class ListMixin(wx.ListCtrl, ListCtrlAutoWidthMixin, ColumnSorterMixin):
+        def __init__(self, parent, map):
+            wx.ListCtrl.__init__(self, parent, -1, style=wx.LC_REPORT)
+            ListCtrlAutoWidthMixin.__init__(self)
+            ColumnSorterMixin.__init__(self, len(map))
+            self.itemDataMap = map
+
+        def GetListCtrl(self):
+            return self
+
+    # menu item ids
+    ID_EXIT = 1
+
+    ID_SHOW_GOOD = 11
+    ID_SHOW_BAD = 12
+    ID_SHOW_UNSURE = 13
+
+    ID_SHOW_SSL = 21
+    ID_SHOW_HTTP = 22
+    ID_SHOW_SSH = 23
+
+    ID_NODE = 31
+
+    # the main application window
+    class MainFrame(wx.Frame):
+        def __init__(self):
+            wx.Frame.__init__(self, None, title="Soat test results", size=(900,500))
+         
+            # get the data
+
+            self.dataHandler = DataHandler()
+            self.dataList = self.dataHandler.getAll()
+            self.filteredList = self.dataList
+
+            # display it
+        
+            self.CreateStatusBar()
+            self.initMenuBar()
+            self.initContent()
+
+            self.Center()
+            self.Show()
+    
+        def initMenuBar(self):
+            fileMenu = wx.Menu()
+            fileMenu.Append(ID_EXIT, "E&xit", "Exit the program")
+        
+            viewMenu = wx.Menu()
+            self.showGood = viewMenu.Append(ID_SHOW_GOOD, 'Show &Good', 'Show sucessful test results', kind=wx.ITEM_CHECK)
+            self.showBad = viewMenu.Append(ID_SHOW_BAD, 'Show &Bad', 'Show unsucessful test results', kind=wx.ITEM_CHECK)
+            self.showUnsure = viewMenu.Append(ID_SHOW_UNSURE, 'Show &Inconclusive', 'Show inconclusive test results', kind=wx.ITEM_CHECK)
+            viewMenu.AppendSeparator()
+            self.showSSL = viewMenu.Append(ID_SHOW_SSL, 'Show SS&L', 'Show SSL test results', kind=wx.ITEM_CHECK)
+            self.showHTTP = viewMenu.Append(ID_SHOW_HTTP, 'Show &HTTP', 'Show HTTP test results', kind=wx.ITEM_CHECK)
+            self.showSSH = viewMenu.Append(ID_SHOW_SSH, 'Show &SSH', 'Show SSH test results', kind=wx.ITEM_CHECK)
+            viewMenu.AppendSeparator()
+            viewMenu.Append(ID_NODE, '&Find node...', 'View test results for a given node [NOT IMPLEMENTED]')
+    
+            viewMenu.Check(ID_SHOW_GOOD, True)
+            viewMenu.Check(ID_SHOW_BAD, True)
+            viewMenu.Check(ID_SHOW_UNSURE, True)
+            viewMenu.Check(ID_SHOW_SSL, True)
+            viewMenu.Check(ID_SHOW_HTTP, True)
+            viewMenu.Check(ID_SHOW_SSH, True)
+
+            menuBar = wx.MenuBar()
+            menuBar.Append(fileMenu,"&File")
+            menuBar.Append(viewMenu,"&View")
+
+            self.SetMenuBar(menuBar)
+
+            wx.EVT_MENU(self, ID_EXIT, self.OnExit)
+
+            wx.EVT_MENU(self, ID_SHOW_GOOD, self.GenerateFilteredList)
+            wx.EVT_MENU(self, ID_SHOW_BAD, self.GenerateFilteredList)
+            wx.EVT_MENU(self, ID_SHOW_UNSURE, self.GenerateFilteredList)
+            wx.EVT_MENU(self, ID_SHOW_SSL, self.GenerateFilteredList)
+            wx.EVT_MENU(self, ID_SHOW_HTTP, self.GenerateFilteredList)
+            wx.EVT_MENU(self, ID_SHOW_SSH, self.GenerateFilteredList)
+
+        def initContent(self):
+        
+            base = wx.Panel(self, -1)
+            sizer = wx.GridBagSizer(0,0)
+
+            box = wx.StaticBox(base, -1, 'Summary')
+            boxSizer = wx.StaticBoxSizer(box, wx.HORIZONTAL)
+
+            total = wx.StaticText(base, -1, 'Total tests: ' + `len(self.filteredList)`)
+            boxSizer.Add(total, 0, wx.LEFT | wx.TOP | wx.BOTTOM, 10)
+
+            nodes = wx.StaticText(base, -1, 'Nodes scanned: ' + `len(Set([x.exit_node for x in self.filteredList]))`)
+            boxSizer.Add(nodes, 0, wx.LEFT | wx.TOP | wx.BOTTOM , 10)
+
+            bad = wx.StaticText(base, -1, 'Failed tests: ' + `len([x for x in self.filteredList if x.status == 2])`)
+            boxSizer.Add(bad, 0, wx.LEFT | wx.TOP | wx.BOTTOM, 10)
+
+            suspicious = wx.StaticText(base, -1, 'Inconclusive tests: ' + `len([x for x in self.filteredList if x.status == 1])`)
+            boxSizer.Add(suspicious, 0, wx.ALL, 10)
+
+            sizer.Add(boxSizer, (0,0), (1, 5), wx.EXPAND | wx.ALL, 15)
+
+            dataMap = {}
+            self.fillDataMap(dataMap)
+        
+            self.listCtrl = ListMixin(base, dataMap)
+            self.listCtrl.InsertColumn(0, 'exit node', width=380)
+            self.listCtrl.InsertColumn(1, 'type', width=70)
+            self.listCtrl.InsertColumn(2, 'site', width=180)
+            self.listCtrl.InsertColumn(3, 'time', width=180)
+            self.listCtrl.InsertColumn(4, 'status', wx.LIST_FORMAT_CENTER, width=50)
+
+            self.fillListCtrl(dataMap)
+        
+            sizer.Add(self.listCtrl, (1,0), (1,5), wx.EXPAND | wx.LEFT | wx.BOTTOM | wx.RIGHT, border=15)
+
+            sizer.AddGrowableCol(3)
+            sizer.AddGrowableRow(1)
+
+            base.SetSizerAndFit(sizer)
+
+        # make a nasty dictionary from the current self.filteredList object so columns would be sortable
+        def fillDataMap(self, dataMap):
+            for i in range(len(self.filteredList)):
+                dataMap.update([(i,(self.filteredList[i].exit_node, 
+                                self.filteredList[i].__class__.__name__[:-10],
+                                self.filteredList[i].site, 
+                                time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime(self.filteredList[i].timestamp)), 
+                                self.filteredList[i].status))])
+
+        # fill the result listing with data
+        def fillListCtrl(self, dataMap):
+            if self.listCtrl.GetItemCount() > 0:
+                self.listCtrl.DeleteAllItems()
+
+            for k, i in dataMap.items():
+                index = self.listCtrl.InsertStringItem(sys.maxint, `i[0]`)
+                self.listCtrl.SetStringItem(index, 1, i[1])
+                self.listCtrl.SetStringItem(index, 2, `i[2]`) 
+                self.listCtrl.SetStringItem(index, 3, i[3])
+                self.listCtrl.SetStringItem(index, 4, `i[4]`)
+                self.listCtrl.SetItemData(index,k)
+
+        def OnExit(self,e):
+            self.Close(True)
+
+        def GenerateFilteredList(self, e):
+            self.filteredList = list(self.dataHandler.filterResults(self.dataList, 
+                self.showSSH.IsChecked(), self.showHTTP.IsChecked(), self.showSSL.IsChecked(), 
+                self.showGood.IsChecked(), self.showBad.IsChecked(), self.showUnsure.IsChecked()))
+            dataMap = {}
+            self.fillDataMap(dataMap)
+            self.fillListCtrl(dataMap)
+            self.listCtrl.RefreshItems(0, len(dataMap)) 
+
+if __name__ == "__main__":
+    if len(sys.argv) == 1:
+        console = StatsConsole()
+        console.Listen()
+    elif len(sys.argv) == 2 and sys.argv[1] == 'wx':
+        if nowx:
+            print 'wxpython doesn\'t seem to be installed on your system'
+            print 'you can use the console interface instead (see help)'
+        else:
+            app = wx.App(0)
+            MainFrame()
+            app.MainLoop()
+    else:
+        print ''
+        print 'This app displays results of tests carried out by soat.py (in a user-friendly way).'
+        print ''
+        print 'Usage:'
+        print 'python soatstats.py - app starts console-only'
+        print 'python soatstats.py wx - app starts with a wxpython gui'
+        print ''


Property changes on: torflow/branches/gsoc2008/soatstats.py
___________________________________________________________________
Name: svn:executable
   + *