> Scanning methodology:
>
> The first scans to be implemented are HTTP, HTML, _javascript_, and
> SSL scans.
>
> The HTTP scan scrapes Google for common filetype urls such as exe, msi,
> doc, dmg, etc. It then fetches these urls through Non-Tor and Tor, and
> compares the SHA1 hases of the resulting content.
^^hashes
> The SSL scan downloads certificates for all IPs a domain will locally
> resolve to and compares these certificates to those seen over Tor. The
> scanner notes if a domain had rotated certificates locally in the
> results for each scan.
>
> The HTML scan checks HTML, _javascript_, and plugin content for
> modifications. Because of the dynamic nature of most of the web, the
> scanner has a number of mechanisms built in to filter out false
> positives that are used when a change is noticed between Tor and
> Non-Tor.
As an eventual feature, for the above tests, it probably makes sense
to be able to imitate a few different popular browsers as the scanner
does its checks. If an adversary can recognize the scanner, it can
MITM everything _but_ the scanner.
You're right. It would be best to imitate different browsers.
I got one thing to add to this. Simply using a "User-Agent:" header isn't good enough. You will need to look at the order in which the headers are sent. As an example (which may or may not be accurate) FF may send "User-Agent:" before a "Accept-Encoding:" where IE may do the opposite. It would be feasible for an attacker to analyze the order of the HTTP headers to determine which browser is being used, or in this case, that a scanner is being used.
I looked at this some time back and was able to guess, very accurately at that, which browser was being used. Specifically, I was looking for FF agents pretending to be a different browser.
Just my $0.02...
- Kyle