[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[or-cvs] r18508: {torflow} Implement and document --rescan and --restart, for scanning (torflow/trunk/NetworkScanners)
Author: mikeperry
Date: 2009-02-12 11:21:20 -0500 (Thu, 12 Feb 2009)
New Revision: 18508
Modified:
torflow/trunk/NetworkScanners/README.ExitScanning
torflow/trunk/NetworkScanners/libsoat.py
torflow/trunk/NetworkScanners/snakeinspector.py
torflow/trunk/NetworkScanners/soat.py
Log:
Implement and document --rescan and --restart, for scanning
only failed nodes and for restarting a crashed soat run,
respectively.
Modified: torflow/trunk/NetworkScanners/README.ExitScanning
===================================================================
--- torflow/trunk/NetworkScanners/README.ExitScanning 2009-02-12 16:07:43 UTC (rev 18507)
+++ torflow/trunk/NetworkScanners/README.ExitScanning 2009-02-12 16:21:20 UTC (rev 18508)
@@ -146,14 +146,31 @@
Yahoo and Google begin to hate you, you can't scan at all.
-B. Handling Results
+B. Handling Crashes
At this stage in the game, your primary task will be to periodically
check the scanner for exceptions and hangs. For that you'll just want
to tail the soat.log file to make sure it is putting out recent loglines
and is continuing to run. If there are any issues, please mail me your
-soat.log.
+soat.log.
+If/When SoaT crashes, you should be able to restart it exactly where it
+left off with:
+
+# ./soat.py --restart --ssl --html --http --dnsrebind >& soat.log &
+
+Keeping the same options during a --restart is a Really Good Idea.
+
+Soat actually saves a snapshot to a unique name each time you run it, so
+you can go back in time and restart arbitrary runs by specifying their
+number:
+
+# ls ./data/soat/
+# ./soat.py --restart 2 --ssl --html --http --dnsrebind >& soat.log &
+
+
+C. Handling Results
+
As things stabilize, you'll want to begin grepping your soat.log for
ERROR lines. These indicate serious scanning errors and content
modifications. There will likely be false positives at first, and these
@@ -162,10 +179,41 @@
# tar -jcf soat-data.tbz2 ./data/soat ./soat.log
+If you're feeling adventurous, you can inspect the results yourself by
+running snakeinspector.py. Running it with no arguments will dump all
+failures to your screen in a semi-human readable format. You can add a
+--verbose to get unified diffs of content modifications, and you can
+filter on specific Test Result types with --resultfilter, and on
+specific exit idhexes with --exit. Ex:
+
+# ./snakeinspector.py --verbose --exit 80972D30FE33CB8AD60726C5272AFCEBB05CD6F7
+ --resultfilter SSLTestResult
+
+or just:
+
+# ./snakeinspector.py | less
+
At some point in the future, I hope to have a script prepared that will
mail false positives and actual results to me when you run it. Later
still, soat will automatically mail these results to an email list we
are all subscribed to as they happen.
-Alright, let's get those motherfuckin snakes off this motherfuckin Tor!
+D. Verifying Results
+
+If you would like to verify a set of results, you can use the --rescan
+option of soat, which crawls your data directory and creates a list of
+nodes to scan that consist only of failures, and then scans those with
+fresh URLs:
+
+# ./soat.py --rescan --ssl --html --http --dnsrebind >& soat.log &
+
+Rescans can also be restarted with --restart should they fail.
+
+By default, soat also does a rescan at the end of every loop through the
+node list, though this may change, because we can't apply the network-wide
+false positive filter to URLs during rescan mode.
+
+
+Alright that covers the basics. Let's get those motherfuckin snakes off
+this motherfuckin Tor!
Modified: torflow/trunk/NetworkScanners/libsoat.py
===================================================================
--- torflow/trunk/NetworkScanners/libsoat.py 2009-02-12 16:07:43 UTC (rev 18507)
+++ torflow/trunk/NetworkScanners/libsoat.py 2009-02-12 16:21:20 UTC (rev 18508)
@@ -95,9 +95,11 @@
try:
basename = os.path.basename(file)
new_file = to_dir+basename
+ if not os.path.exists(file) and os.path.exists(new_file):
+ return new_file # Already moved by another test (ex: content file)
os.rename(file, new_file)
return new_file
- except:
+ except Exception, e:
traceback.print_exc()
plog("WARN", "Error moving "+file+" to "+to_dir)
return file
@@ -532,10 +534,8 @@
return pickle.load(fh)
def uniqueFilename(afile):
- if not os.path.exists(afile):
- return afile
(prefix,suffix)=os.path.splitext(afile)
- i=1
+ i=0
while os.path.exists(prefix+"."+str(i)+suffix):
i+=1
return prefix+"."+str(i)+suffix
@@ -581,7 +581,32 @@
pickle.dump(result, result_file)
result_file.close()
+ def __testFilename(self, test, position=-1):
+ if position == -1:
+ return DataHandler.uniqueFilename(self.data_dir+test.__class__.__name__+".test")
+ else:
+ return self.data_dir+test.__class__.__name__+"."+str(position)+".test"
+ def loadTest(self, testname, position=-1):
+ filename = self.data_dir+testname
+ if position == -1:
+ i=0
+ while os.path.exists(filename+"."+str(i)+".test"):
+ i+=1
+ position = i-1
+
+ test_file = open(filename+"."+str(position)+".test", 'r')
+ test = pickle.load(test_file)
+ test_file.close()
+ return test
+
+ def saveTest(self, test):
+ if not test.filename:
+ test.filename = self.__testFilename(test)
+ test_file = open(test.filename, 'w')
+ pickle.dump(test, test_file)
+ test_file.close()
+
# These three bits are needed to fully recursively strain the parsed soup.
# For some reason, the SoupStrainer does not get applied recursively..
__first_strainer = SoupStrainer(lambda name, attrs: name in tags_to_check or
Modified: torflow/trunk/NetworkScanners/snakeinspector.py
===================================================================
--- torflow/trunk/NetworkScanners/snakeinspector.py 2009-02-12 16:07:43 UTC (rev 18507)
+++ torflow/trunk/NetworkScanners/snakeinspector.py 2009-02-12 16:21:20 UTC (rev 18508)
@@ -16,8 +16,12 @@
from libsoat import *
sys.path.append("../")
+
+import TorCtl.TorUtil
from TorCtl.TorUtil import *
+TorCtl.TorUtil.loglevel="NOTICE"
+
def usage():
# TODO: Don't be a jerk.
print "Use teh src, luke."
@@ -35,8 +39,8 @@
use_file=None
node=None
reason=None
- result=None
- verbose=0
+ result=2
+ verbose=1
proto=None
resultfilter=None
for o,a in opts:
Modified: torflow/trunk/NetworkScanners/soat.py
===================================================================
--- torflow/trunk/NetworkScanners/soat.py 2009-02-12 16:07:43 UTC (rev 18507)
+++ torflow/trunk/NetworkScanners/soat.py 2009-02-12 16:21:20 UTC (rev 18508)
@@ -69,6 +69,8 @@
from soat_config import *
search_cookies=None
+metacon=None
+datahandler=None
linebreak = '\r\n'
@@ -91,6 +93,7 @@
def connect(self):
try:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
+ self.sock.settimeout(read_timeout) # Mnemotronic tonic
if self.debuglevel > 0:
print "connect: (%s, %s)" % (self.host, self.port)
self.sock.connect((str(self.host), self.port))
@@ -118,6 +121,7 @@
new_cookies = []
mime_type = ""
try:
+ plog("DEBUG", "Starting request for: "+address)
if cookie_jar != None:
opener = urllib2.build_opener(NoDNSHTTPHandler, urllib2.HTTPCookieProcessor(cookie_jar))
reply = opener.open(request)
@@ -132,7 +136,12 @@
plog("WARN", "Max content size exceeded for "+address+": "+length)
return (reply.code, [], "", "")
mime_type = reply.info().type
+ plog("DEBUG", "Mime type is "+mime_type+", length "+str(length))
content = decompress_response_data(reply)
+ except socket.timeout, e:
+ plog("WARN", "Socket timeout for "+address+": "+str(e))
+ traceback.print_exc()
+ return (666, [], "", e.__class__.__name__+str(e))
except urllib2.HTTPError, e:
plog('NOTICE', "HTTP Error during request of "+address+": "+str(e))
traceback.print_exc()
@@ -161,24 +170,23 @@
class Test:
""" Base class for our tests """
- def __init__(self, mt, proto, port):
+ def __init__(self, proto, port):
self.proto = proto
self.port = port
- self.mt = mt
- self.datahandler = DataHandler()
self.min_targets = min_targets
self.exit_limit_pct = max_exit_fail_pct
self.dynamic_limit = max_dynamic_failure
- self.marked_nodes = sets.Set([])
- self.exit_fails = {}
- self.successes = {}
- self.results = []
- self.dynamic_fails = {}
+ self.filename = None
+ self.rescan_nodes = sets.Set([])
+ self.nodes = []
+ self.node_map = {}
self.banned_targets = sets.Set([])
self.total_nodes = 0
- self.nodes = []
- self.node_map = {}
- self.all_nodes = sets.Set([])
+ self.scan_nodes = 0
+ self.nodes_to_mark = 0
+ self.tests_per_node = num_tests_per_node
+ self._reset()
+ self._pickle_revision = 0 # Will increment as fields are added
def run_test(self):
raise NotImplemented()
@@ -186,9 +194,6 @@
def get_targets(self):
raise NotImplemented()
- def get_node(self):
- return random.choice(self.nodes)
-
def remove_target(self, target, reason="None"):
self.banned_targets.add(target)
if target in self.targets: self.targets.remove(target)
@@ -211,28 +216,79 @@
except:
pass
r.mark_false_positive(reason)
- self.datahandler.saveResult(r)
+ datahandler.saveResult(r)
self.results.remove(r)
+ def load_rescan(self, type, since=None):
+ self.rescan_nodes = sets.Set([])
+ results = datahandler.getAll()
+ for r in results:
+ if r.status == type:
+ if not since or r.timestamp >= since:
+ self.rescan_nodes.add(r.exit_node[1:])
+ plog("INFO", "Loaded "+str(len(self.rescan_nodes))+" nodes to rescan")
+ if self.nodes and self.rescan_nodes:
+ self.nodes &= self.rescan_nodes
+ self.scan_nodes = len(self.nodes)
+ self.tests_per_node = num_rescan_tests_per_node
+ self.nodes_to_mark = self.scan_nodes*self.tests_per_node
+
+ def toggle_rescan(self):
+ if self.rescan_nodes:
+ plog("NOTICE", self.proto+" rescan complete. Switching back to normal scan")
+ self.rescan_nodes = sets.Set([])
+ self.tests_per_node = num_tests_per_node
+ self.update_nodes()
+ else:
+ plog("NOTICE", self.proto+" switching to recan mode.")
+ self.load_rescan(TEST_FAILURE, self.run_start)
+
+ def get_node(self):
+ return random.choice(list(self.nodes))
+
def update_nodes(self):
- self.nodes = self.mt.node_manager.get_nodes_for_port(self.port)
+ nodes = metacon.node_manager.get_nodes_for_port(self.port)
self.node_map = {}
- for n in self.nodes:
+ for n in nodes:
self.node_map[n.idhex] = n
- self.total_nodes = len(self.nodes)
- self.all_nodes = sets.Set(self.nodes)
+ self.total_nodes = len(nodes)
+ self.nodes = sets.Set(map(lambda n: n.idhex, nodes))
+ # Only scan the stuff loaded from the rescan
+ if self.rescan_nodes: self.nodes &= self.rescan_nodes
+ if not self.nodes:
+ plog("ERROR", "No nodes remain after rescan load!")
+ self.scan_nodes = len(self.nodes)
+ self.nodes_to_mark = self.scan_nodes*self.tests_per_node
- def mark_chosen(self, node):
+ def mark_chosen(self, node, result):
+ exit_node = metacon.get_exit_node()[1:]
+ if exit_node != node:
+ plog("ERROR", "Asked to mark a node that is not current: "+node+" vs "+exit_node)
self.nodes_marked += 1
- self.marked_nodes.add(node)
+ if not node in self.node_results: self.node_results[node] = []
+ self.node_results[node].append(result)
+ if len(self.node_results[node]) >= self.tests_per_node:
+ self.nodes.remove(node)
def finished(self):
- return not self.marked_nodes ^ self.all_nodes
+ return not self.nodes
+
+ def percent_complete(self):
+ return round(100.0*self.nodes_marked/(self.nodes_to_mark), 1)
- def percent_complete(self):
- return round(100.0*self.nodes_marked/self.total_nodes, 1)
+ def _reset(self):
+ self.exit_fails = {}
+ self.successes = {}
+ self.dynamic_fails = {}
+ self.results = []
+ self.targets = []
+ self.tests_run = 0
+ self.nodes_marked = 0
+ self.node_results = {}
+ self.run_start = time.time()
def rewind(self):
+ self._reset()
self.targets = self.get_targets()
if not self.targets:
raise NoURLsFound("No URLS found for protocol "+self.proto)
@@ -244,16 +300,7 @@
else:
targets = "\n\t".join(self.targets)
plog("INFO", "Using the following urls for "+self.proto+" scan:\n\t"+targets)
- self.tests_run = 0
- self.nodes_marked = 0
- self.marked_nodes = sets.Set([])
- self.exit_fails = {}
- self.successes = {}
- self.dynamic_fails = {}
- # TODO: report these results as BadExit before clearing
- self.results = []
-
def register_exit_failure(self, address, exit_node):
if address in self.exit_fails:
self.exit_fails[address].add(exit_node)
@@ -291,9 +338,9 @@
class SearchBasedTest(Test):
- def __init__(self, mt, proto, port, wordlist_file):
+ def __init__(self, proto, port, wordlist_file):
self.wordlist_file = wordlist_file
- Test.__init__(self, mt, proto, port)
+ Test.__init__(self, proto, port)
def rewind(self):
self.wordlist = load_wordlist(self.wordlist_file)
@@ -409,9 +456,9 @@
return list(urllist)
class HTTPTest(SearchBasedTest):
- def __init__(self, mt, wordlist, filetypes=scan_filetypes):
+ def __init__(self, wordlist, filetypes=scan_filetypes):
# FIXME: Handle http urls w/ non-80 ports..
- SearchBasedTest.__init__(self, mt, "HTTP", 80, wordlist)
+ SearchBasedTest.__init__(self, "HTTP", 80, wordlist)
self.fetch_targets = urls_per_filetype
self.httpcode_fails = {}
self.httpcode_limit_pct = max_exit_httpcode_pct
@@ -434,13 +481,13 @@
for cookie in self.cookie_jar:
plain_cookies += "\t"+cookie.name+":"+cookie.domain+cookie.path+" discard="+str(cookie.discard)+"\n"
if tor_cookies != plain_cookies:
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
plog("ERROR", "Cookie mismatch at "+exit_node+":\nTor Cookies:"+tor_cookies+"\nPlain Cookies:\n"+plain_cookies)
result = CookieTestResult(exit_node, TEST_FAILURE,
FAILURE_COOKIEMISMATCH, plain_cookies,
tor_cookies)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_FAILURE
return TEST_SUCCESS
@@ -553,6 +600,8 @@
if not content:
plog("WARN", "Failed to direct load "+address)
+ # Just remove it
+ self.remove_target(address, INCONCLUSIVE_NOLOCALCONTENT)
# Restore cookie jar
self.cookie_jar = orig_cookie_jar
self.tor_cookie_jar = orig_tor_cookie_jar
@@ -591,7 +640,7 @@
# reset the connection to direct
socket.socket = defaultsocket
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if exit_node == 0 or exit_node == '0' or not exit_node:
plog('WARN', 'We had no exit node to test, skipping to the next test.')
# Restore cookie jars
@@ -608,7 +657,7 @@
result = HttpTestResult(exit_node, address, TEST_INCONCLUSIVE,
INCONCLUSIVE_TORBREAKAGE+str(pcontent))
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_INCONCLUSIVE
else:
BindingSocket.bind_to = refetch_ip
@@ -624,7 +673,7 @@
result = HttpTestResult(exit_node, address, TEST_FAILURE,
FAILURE_BADHTTPCODE+str(pcode)+":"+str(pcontent))
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_httpcode_failure(address, exit_node)
return TEST_FAILURE
@@ -634,7 +683,7 @@
result = HttpTestResult(exit_node, address, TEST_FAILURE,
FAILURE_NOEXITCONTENT)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
# Restore cookie jars
self.cookie_jar = orig_cookie_jar
self.tor_cookie_jar = orig_tor_cookie_jar
@@ -645,7 +694,7 @@
if psha1sum.hexdigest() == sha1sum.hexdigest():
result = HttpTestResult(exit_node, address, TEST_SUCCESS)
self.results.append(result)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -664,7 +713,7 @@
result = HttpTestResult(exit_node, address, TEST_INCONCLUSIVE,
INCONCLUSIVE_NOLOCALCONTENT)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_INCONCLUSIVE
sha1sum_new = sha.sha(content_new)
@@ -696,7 +745,7 @@
if psha1sum.hexdigest() == sha1sum_new.hexdigest():
result = HttpTestResult(exit_node, address, TEST_SUCCESS)
self.results.append(result)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -746,7 +795,7 @@
psha1sum.hexdigest(), content_prefix+".content",
exit_content_file.name)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
@@ -762,7 +811,7 @@
content_prefix+'.content-old',
sha1sum.hexdigest())
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
# The HTTP Test should remove address immediately.
plog("WARN", "HTTP Test is removing dynamic URL "+address)
@@ -770,12 +819,12 @@
return TEST_FAILURE
class HTMLTest(HTTPTest):
- def __init__(self, mt, wordlist, recurse_filetypes=scan_filetypes):
- HTTPTest.__init__(self, mt, wordlist, recurse_filetypes)
+ def __init__(self, wordlist, recurse_filetypes=scan_filetypes):
+ HTTPTest.__init__(self, wordlist, recurse_filetypes)
self.fetch_targets = num_html_urls
self.proto = "HTML"
self.recurse_filetypes = recurse_filetypes
- self.fetch_queue = Queue.Queue()
+ self.fetch_queue = []
def run_test(self):
# A single test should have a single cookie jar
@@ -795,9 +844,9 @@
# Keep a trail log for this test and check for loops
address = random.choice(self.targets)
- self.fetch_queue.put_nowait(("html", address, first_referer))
- while not self.fetch_queue.empty():
- (test, url, referer) = self.fetch_queue.get_nowait()
+ self.fetch_queue.append(("html", address, first_referer))
+ while self.fetch_queue:
+ (test, url, referer) = self.fetch_queue.pop(0)
if referer: self.headers['Referer'] = referer
# Technically both html and js tests check and dispatch via mime types
# but I want to know when link tags lie
@@ -811,6 +860,10 @@
result = self.check_cookies()
if result > ret_result:
ret_result = result
+
+ # Need to clear because the cookiejars use locks...
+ self.tor_cookie_jar = None
+ self.cookie_jar = None
return ret_result
def get_targets(self):
@@ -851,7 +904,7 @@
for i in sets.Set(targets):
if self._is_useable_url(i[1], html_schemes):
plog("NOTICE", "Adding "+i[0]+" target: "+i[1])
- self.fetch_queue.put_nowait((i[0], i[1], orig_addr))
+ self.fetch_queue.append((i[0], i[1], orig_addr))
else:
plog("NOTICE", "Skipping "+i[0]+" target: "+i[1])
@@ -884,7 +937,7 @@
if not has_js_changes:
result = JsTestResult(exit_node, address, TEST_SUCCESS)
self.results.append(result)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -903,7 +956,7 @@
exit_content_file.name,
content_prefix+'.content-old')
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
plog("ERROR", "Javascript 3-way failure at "+exit_node+" for "+address)
return TEST_FAILURE
@@ -949,7 +1002,7 @@
plog("INFO", "Successful soup comparison after SHA1 fail for "+address+" via "+exit_node)
result = HtmlTestResult(exit_node, address, TEST_SUCCESS)
self.results.append(result)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -960,7 +1013,7 @@
result = HtmlTestResult(exit_node, address, TEST_INCONCLUSIVE,
INCONCLUSIVE_NOLOCALCONTENT)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_INCONCLUSIVE
new_soup = FullyStrainedSoup(content_new)
@@ -976,7 +1029,7 @@
FAILURE_EXITONLY, content_prefix+".content",
exit_content_file.name)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
@@ -1022,7 +1075,7 @@
plog("NOTICE", "False positive detected for dynamic change at "+address+" via "+exit_node)
result = HtmlTestResult(exit_node, address, TEST_SUCCESS)
self.results.append(result)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -1036,16 +1089,16 @@
exit_content_file.name,
content_prefix+'.content-old')
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_dynamic_failure(address, exit_node)
return TEST_FAILURE
class SSLTest(SearchBasedTest):
- def __init__(self, mt, wordlist):
+ def __init__(self, wordlist):
self.test_hosts = num_ssl_hosts
- SearchBasedTest.__init__(self, mt, "SSL", 443, wordlist)
+ SearchBasedTest.__init__(self, "SSL", 443, wordlist)
def run_test(self):
self.tests_run += 1
@@ -1060,6 +1113,7 @@
# specify the context
ctx = SSL.Context(SSL.TLSv1_METHOD)
+ ctx.set_timeout(int(read_timeout))
ctx.set_verify_depth(1)
# ready the certificate request
@@ -1069,10 +1123,15 @@
# FIXME: Hrmmm. handshake considerations
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ # SSL has its own timeouts handled above. Undo ours from BindingSocket
+ s.settimeout(None)
c = SSL.Connection(ctx, s)
c.set_connect_state()
c.connect((address, 443)) # DNS OK.
c.send(crypto.dump_certificate_request(crypto.FILETYPE_PEM,request))
+ except socket.timeout, e:
+ plog('WARN','Socket timeout for '+address+": "+str(e))
+ return e
except socket.error, e:
plog('WARN','An error occured while opening an ssl connection to '+address+": "+str(e))
return e
@@ -1094,7 +1153,7 @@
return c.get_peer_certificate()
def get_resolved_ip(self, hostname):
- mappings = self.mt.control.get_address_mappings("cache")
+ mappings = metacon.control.get_address_mappings("cache")
ret = None
for m in mappings:
if m.from_addr == hostname:
@@ -1135,23 +1194,38 @@
ssl_domain = SSLDomain(address)
check_ips = []
- resolved = socket.getaddrinfo(address, 443)
+ # Make 3 resolution attempts
+ for attempt in xrange(1,4):
+ try:
+ resolved = []
+ resolved = socket.getaddrinfo(address, 443)
+ break
+ except socket.gaierror:
+ plog("NOTICE", "Local resolution failure #"+str(attempt)+" for "+address)
+
for res in resolved:
if res[0] == socket.AF_INET and res[2] == socket.IPPROTO_TCP:
check_ips.append(res[4][0])
+ if not check_ips:
+ plog("WARN", "Local resolution failure for "+address)
+ self.remove_target(address, INCONCLUSIVE_NOLOCALCONTENT)
+ return TEST_INCONCLUSIVE
+
try:
if self._update_cert_list(ssl_domain, check_ips):
ssl_file = open(ssl_file_name, 'w')
pickle.dump(ssl_domain, ssl_file)
ssl_file.close()
except OpenSSL.crypto.Error:
- plog('WARN', 'Crypto error.')
+ plog('WARN', 'Local crypto error for '+address)
traceback.print_exc()
+ self.remove_target(address, INCONCLUSIVE_NOLOCALCONTENT)
return TEST_INCONCLUSIVE
if not ssl_domain.cert_map:
plog('WARN', 'Error getting the correct cert for ' + address)
+ self.remove_target(address, INCONCLUSIVE_NOLOCALCONTENT)
return TEST_INCONCLUSIVE
if ssl_domain.cert_changed:
@@ -1163,8 +1237,9 @@
pickle.dump(ssl_domain, ssl_file)
ssl_file.close()
except OpenSSL.crypto.Error:
- plog('WARN', 'Crypto error.')
+ plog('WARN', 'Local crypto error for '+address)
traceback.print_exc()
+ self.remove_target(address, INCONCLUSIVE_NOLOCALCONTENT)
return TEST_INCONCLUSIVE
if ssl_domain.cert_changed:
plog("NOTICE", "Fully dynamic certificate host "+address)
@@ -1172,7 +1247,7 @@
result = SSLTestResult("NoExit", address, ssl_file_name,
TEST_INCONCLUSIVE,
INCONCLUSIVE_DYNAMICSSL)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.results.append(result)
self.remove_target(address, FALSEPOSITIVE_DYNAMIC)
return TEST_INCONCLUSIVE
@@ -1182,7 +1257,7 @@
result = SSLTestResult("NoExit", address, ssl_file_name,
TEST_INCONCLUSIVE,
INCONCLUSIVE_NOLOCALCONTENT)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.results.append(result)
self.remove_target(address, FALSEPOSITIVE_DEADSITE)
return TEST_INCONCLUSIVE
@@ -1197,7 +1272,7 @@
# reset the connection method back to direct
socket.socket = defaultsocket
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if not exit_node or exit_node == '0':
plog('WARN', 'We had no exit node to test, skipping to the next test.')
return TEST_INCONCLUSIVE
@@ -1207,7 +1282,7 @@
result = SSLTestResult(exit_node, address, ssl_file_name,
TEST_INCONCLUSIVE,
INCONCLUSIVE_TORBREAKAGE)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.results.append(result)
return TEST_INCONCLUSIVE
@@ -1217,7 +1292,7 @@
result = SSLTestResult(exit_node, address, ssl_file_name,
TEST_FAILURE,
FAILURE_NOEXITCONTENT)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.results.append(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
@@ -1227,7 +1302,7 @@
result = SSLTestResult(exit_node, address, ssl_file_name, TEST_FAILURE,
FAILURE_MISCEXCEPTION+":"+cert.__class__.__name__+str(cert))
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
@@ -1239,14 +1314,14 @@
result = SSLTestResult(exit_node, address, ssl_file_name, TEST_FAILURE,
FAILURE_MISCEXCEPTION+":"+e.__class__.__name__+str(e))
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
# if certs match, everything is ok
if ssl_domain.seen_cert(cert_pem):
result = SSLTestResult(exit_node, address, ssl_file_name, TEST_SUCCESS)
- #self.datahandler.saveResult(result)
+ #datahandler.saveResult(result)
if address in self.successes: self.successes[address]+=1
else: self.successes[address]=1
return TEST_SUCCESS
@@ -1258,7 +1333,7 @@
FAILURE_DYNAMICCERTS,
self.get_resolved_ip(address), cert_pem)
self.results.append(result)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.register_dynamic_failure(address, exit_node)
return TEST_FAILURE
@@ -1266,14 +1341,14 @@
result = SSLTestResult(exit_node, address, ssl_file_name, TEST_FAILURE,
FAILURE_EXITONLY, self.get_resolved_ip(address),
cert_pem)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
self.results.append(result)
self.register_exit_failure(address, exit_node)
return TEST_FAILURE
class POP3STest(Test):
- def __init__(self, mt):
- Test.__init__(self, mt, "POP3S", 110)
+ def __init__(self):
+ Test.__init__(self, "POP3S", 110)
def run_test(self):
self.tests_run += 1
@@ -1371,7 +1446,7 @@
socket.socket = defaultsocket
# check whether the test was valid at all
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if exit_node == 0 or exit_node == '0':
plog('INFO', 'We had no exit node to test, skipping to the next test.')
return TEST_SUCCESS
@@ -1450,16 +1525,16 @@
if (capabilities_ok != capabilities_ok_d or starttls_present != starttls_present_d or
tls_started != tls_started_d or tls_succeeded != tls_succeeded_d):
result = POPTestResult(exit_node, address, TEST_FAILURE)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_FAILURE
result = POPTestResult(exit_node, address, TEST_SUCCESS)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_SUCCESS
class SMTPSTest(Test):
- def __init__(self, mt):
- Test.__init__(self, mt, "SMTPS", 587)
+ def __init__(self):
+ Test.__init__(self, "SMTPS", 587)
def run_test(self):
self.tests_run += 1
@@ -1511,7 +1586,7 @@
socket.socket = defaultsocket
# check whether the test was valid at all
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if exit_node == 0 or exit_node == '0':
plog('INFO', 'We had no exit node to test, skipping to the next test.')
return TEST_SUCCESS
@@ -1549,17 +1624,17 @@
# compare
if ehlo1_reply != ehlo1_reply_d or has_starttls != has_starttls_d or ehlo2_reply != ehlo2_reply_d:
result = SMTPTestResult(exit_node, address, TEST_FAILURE)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_FAILURE
result = SMTPTestResult(exit_node, address, TEST_SUCCESS)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_SUCCESS
class IMAPSTest(Test):
- def __init__(self, mt):
- Test.__init__(self, mt, "IMAPS", 143)
+ def __init__(self):
+ Test.__init__(self, "IMAPS", 143)
def run_test(self):
self.tests_run += 1
@@ -1646,7 +1721,7 @@
socket.socket = defaultsocket
# check whether the test was valid at all
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if exit_node == 0 or exit_node == '0':
plog('INFO', 'We had no exit node to test, skipping to the next test.')
return TEST_SUCCESS
@@ -1716,11 +1791,11 @@
if (capabilities_ok != capabilities_ok_d or starttls_present != starttls_present_d or
tls_started != tls_started_d or tls_succeeded != tls_succeeded_d):
result = IMAPTestResult(exit_node, address, TEST_FAILURE)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_FAILURE
result = IMAPTestResult(exit_node, address, TEST_SUCCESS)
- self.datahandler.saveResult(result)
+ datahandler.saveResult(result)
return TEST_SUCCESS
class DNSTest(Test):
@@ -1734,7 +1809,7 @@
ip = tor_resolve(address)
# check whether the test was valid at all
- exit_node = self.mt.get_exit_node()
+ exit_node = metacon.get_exit_node()
if exit_node == 0 or exit_node == '0':
plog('INFO', 'We had no exit node to test, skipping to the next test.')
return TEST_SUCCESS
@@ -2159,6 +2234,9 @@
if len(argv) < 2:
print ''
print 'Please provide at least one test option:'
+ print '--pernode <n>'
+ print '--restart [<n>]'
+ print '--rescan [<n>]'
print '--ssl'
print '--http'
print '--html'
@@ -2172,17 +2250,19 @@
print ''
return
- opts = ['ssl','html','http','ssh','smtp','pop','imap','dns','dnsrebind','policies','exit=']
+ opts = ['ssl','rescan', 'pernode=', 'restart', 'html','http','ssh','smtp','pop','imap','dns','dnsrebind','policies','exit=']
flags, trailer = getopt.getopt(argv[1:], [], opts)
# get specific test types
+ do_restart = False
+ do_rescan = ('--rescan','') in flags
do_ssl = ('--ssl','') in flags
do_http = ('--http','') in flags
do_html = ('--html','') in flags
- do_ssh = ('--ssh','') in flags
- do_smtp = ('--smtp','') in flags
- do_pop = ('--pop','') in flags
- do_imap = ('--imap','') in flags
+ #do_ssh = ('--ssh','') in flags
+ #do_smtp = ('--smtp','') in flags
+ #do_pop = ('--pop','') in flags
+ #do_imap = ('--imap','') in flags
do_dns_rebind = ('--dnsrebind','') in flags
do_consistency = ('--policies','') in flags
@@ -2190,20 +2270,38 @@
for flag in flags:
if flag[0] == "--exit":
scan_exit = flag[1]
+ if flag[0] == "--pernode":
+ global num_tests_per_node
+ num_tests_per_node = int(flag[1])
+ if flag[0] == "--rescan" and flag[1]:
+ global num_rescan_tests_per_node
+ num_rescan_tests_per_node = int(flag[1])
+ if flag[0] == "--restart":
+ do_restart = True
+ if flag[1]:
+ restart_run=int(flag[1])
+ else:
+ restart_run=-1
+ # Make logs go to disk so restarts are less painful
+ #TorUtil.logfile = open(log_file_name, "a")
+
# initiate the connection to the metatroller
- mt = Metaconnection()
+ global metacon
+ metacon = Metaconnection()
+ global datahandler
+ datahandler = DataHandler()
# initiate the passive dns rebind attack monitor
if do_dns_rebind:
- mt.check_dns_rebind()
+ metacon.check_dns_rebind()
# check for sketchy exit policies
if do_consistency:
- mt.check_all_exits_port_consistency()
+ metacon.check_all_exits_port_consistency()
# maybe only the consistency test was required
- if not (do_ssl or do_html or do_http or do_ssh or do_smtp or do_pop or do_imap):
+ if not (do_ssl or do_html or do_http):
plog('INFO', 'Done.')
return
@@ -2216,46 +2314,53 @@
tests = {}
- if do_ssl:
- tests["SSL"] = SSLTest(mt, ssl_wordlist_file)
+ if do_restart:
+ if do_ssl:
+ tests["SSL"] = datahandler.loadTest("SSLTest", restart_run)
- if do_http:
- tests["HTTP"] = HTTPTest(mt, filetype_wordlist_file)
+ if do_http:
+ tests["HTTP"] = datahandler.loadTest("HTTPTest", restart_run)
- if do_html:
- tests["HTML"] = HTMLTest(mt, html_wordlist_file)
+ if do_html:
+ tests["HTML"] = datahandler.loadTest("HTMLTest", restart_run)
+ else:
+ if do_ssl:
+ tests["SSL"] = SSLTest(ssl_wordlist_file)
- if do_smtp:
- tests["SMTPS"] = SMTPSTest(mt)
-
- if do_pop:
- tests["POPS"] = POP3STest(mt)
+ if do_http:
+ tests["HTTP"] = HTTPTest(filetype_wordlist_file)
- if do_imap:
- tests["IMAPS"] = IMAPSTest(mt)
+ if do_html:
+ tests["HTML"] = HTMLTest(html_wordlist_file)
+
# maybe no tests could be initialized
- if not (do_ssl or do_html or do_http or do_ssh or do_smtp or do_pop or do_imap):
+ if not tests:
plog('INFO', 'Done.')
sys.exit(0)
- for test in tests.itervalues():
- test.rewind()
+ if do_rescan:
+ for test in tests.itervalues():
+ test.load_rescan(TEST_FAILURE)
+
+ if not do_restart:
+ for test in tests.itervalues():
+ test.rewind()
if scan_exit:
plog("NOTICE", "Scanning only "+scan_exit)
- mt.set_new_exit(scan_exit)
- mt.get_new_circuit()
+ metacon.set_new_exit(scan_exit)
+ metacon.get_new_circuit()
while 1:
for test in tests.values():
result = test.run_test()
plog("INFO", test.proto+" test via "+scan_exit+" has result "+str(result))
-
+
# start testing
while 1:
avail_tests = tests.values()
- if mt.node_manager.has_new_nodes():
+ if metacon.node_manager.has_new_nodes():
plog("INFO", "Got signal for node update.")
for test in avail_tests:
test.update_nodes()
@@ -2270,39 +2375,44 @@
common_nodes = None
# Do set intersection and reuse nodes for shared tests
for test in to_run:
- if not common_nodes: common_nodes = Set(map(lambda n: n.idhex, test.nodes))
- else: common_nodes &= Set(map(lambda n: n.idhex, test.nodes))
+ if not common_nodes: common_nodes = copy.copy(test.nodes)
+ else: common_nodes &= test.nodes
if common_nodes:
current_exit_idhex = random.choice(list(common_nodes))
plog("DEBUG", "Chose to run "+str(n_tests)+" tests via "+current_exit_idhex+" (tests share "+str(len(common_nodes))+" exit nodes)")
- mt.set_new_exit(current_exit_idhex)
- mt.get_new_circuit()
+ metacon.set_new_exit(current_exit_idhex)
+ metacon.get_new_circuit()
for test in to_run:
# Keep testing failures and inconclusives
result = test.run_test()
- if result == TEST_SUCCESS:
- test.mark_chosen(test.node_map[current_exit_idhex])
+ if result != TEST_INCONCLUSIVE:
+ test.mark_chosen(current_exit_idhex, result)
+ datahandler.saveTest(test)
plog("INFO", test.proto+" test via "+current_exit_idhex+" has result "+str(result))
- plog("INFO", test.proto+" attempts: "+str(test.tests_run)+". Completed: "+str(test.nodes_marked)+"/"+str(test.total_nodes)+" ("+str(test.percent_complete())+"%)")
+ plog("INFO", test.proto+" attempts: "+str(test.tests_run)+". Completed: "+str(test.nodes_marked)+"/"+str(test.nodes_to_mark)+" ("+str(test.percent_complete())+"%)")
else:
plog("NOTICE", "No nodes in common between "+", ".join(map(lambda t: t.proto, to_run)))
for test in to_run:
current_exit = test.get_node()
- mt.set_new_exit(current_exit.idhex)
- mt.get_new_circuit()
+ metacon.set_new_exit(current_exit.idhex)
+ metacon.get_new_circuit()
# Keep testing failures and inconclusives
result = test.run_test()
+ if result != TEST_INCONCLUSIVE:
+ test.mark_chosen(current_exit_idhex, result)
+ datahandler.saveTest(test)
plog("INFO", test.proto+" test via "+current_exit.idhex+" has result "+str(result))
- plog("INFO", test.proto+" attempts: "+str(test.tests_run)+". Completed: "+str(test.nodes_marked)+"/"+str(test.total_nodes)+" ("+str(test.percent_complete())+"%)")
- if result == TEST_SUCCESS:
- test.mark_chosen(current_exit)
+ plog("INFO", test.proto+" attempts: "+str(test.tests_run)+". Completed: "+str(test.nodes_marked)+"/"+str(test.nodes_to_mark)+" ("+str(test.percent_complete())+"%)")
# Check each test for rewind
for test in tests.itervalues():
if test.finished():
- plog("NOTICE", test.proto+" test has finished all nodes. Rewinding")
+ plog("NOTICE", test.proto+" test has finished all nodes.")
+ datahandler.saveTest(test)
+ if not do_rescan and rescan_at_finish:
+ test.toggle_rescan()
test.rewind()