[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: IP address blocked on certain site

On 2/4/2011 2:42 PM, cmeclax-sazri wrote:
robots.txt is a file put on some websites as a directive to robots. If you run
a wiki, and you want only current versions, not the hundreds of previous
versions of every page, indexed, you could put a directive in robots.txt, or
label the pages themselves as "noindex nofollow". Automated software that
ignores such directives is likely to eat up huge amounts of bandwidth and
create copies that are many times bigger than the original.

So how does that relate to Tor? Do sites using this see Tor as a robot? I didn't even get on the site. Plus, there're no files to d/l from it - at least not from an avg users capability.
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxxx with
unsubscribe or-talk    in the body. http://archives.seul.org/or/talk/