[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-bugs] #29565 [Obfuscation/Snowflake]: Fix broker robots.txt to disallow crawling
#29565: Fix broker robots.txt to disallow crawling
---------------------------------------+--------------------
Reporter: dcf | Owner: (none)
Type: defect | Status: new
Priority: Medium | Milestone:
Component: Obfuscation/Snowflake | Version:
Severity: Normal | Keywords: easy
Actual Points: | Parent ID:
Points: | Reviewer:
Sponsor: |
---------------------------------------+--------------------
From comment:11:ticket:28848 and https://github.com/ahf/snowflake-
notes/blob/fb4304a7df08c6ddeeb103f38fc9103721a20cd9/Broker.markdown#the-
robotstxt-handler:
> - Was the question about crawling ever answered? I can't think of a very
good reason not to allow it. Even if censors were crawling the web for
Snowflake brokers, they could get this information much more easily just
from the source code.
I believe the intention behind the robots.txt handler is to prevent search
engines from indexing any pages on the site, because there's no permanent
information there, not for any security or anti-enumeration reason.
ahf points out that the current robots.txt achieves the opposite: it
allows crawling of all pages by anyone. Instead of
{{{
User-agent: *
Disallow:
}}}
it should be
{{{
User-agent: *
Disallow: /
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29565>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs