On 2/4/2011 2:42 PM, cmeclax-sazri wrote:
robots.txt is a file put on some websites as a directive to robots. If you runSo how does that relate to Tor? Do sites using this see Tor as a robot? I didn't even get on the site. Plus, there're no files to d/l from it - at least not from an avg users capability.a wiki, and you want only current versions, not the hundreds of previous versions of every page, indexed, you could put a directive in robots.txt, or label the pages themselves as "noindex nofollow". Automated software that ignores such directives is likely to eat up huge amounts of bandwidth and create copies that are many times bigger than the original. cmeclax
*********************************************************************** To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxxx with unsubscribe or-talk in the body. http://archives.seul.org/or/talk/