On 09/19/2012 10:13 AM, tor@xxxxxxxxxxxxxxxxxx wrote:
> On 19/09/12 06:36, grarpamp wrote:
>
>>> People use robots.txt to indicate that they don't want their site
>>> to be added to indexes.
>
>> They use it to indicate that they don't want their site to be
>> crawled.
>
> In almost all cases (99% or higher), robots.txt is used to indicate
> that a site shouldn't be crawled, *because* they don't want it to be
> indexed. The intention is painfully clear...
If website owners don't want a page to be indexed, they should use the
noindex meta tag: http://en.wikipedia.org/wiki/Noindex . robots.txt is only
for crawlers that automatically follow links from one page to others.
Neither standard prevents or discourages manually setting a link to the page.
Best regards
Christian
--
|------- Dr. Christian Siefkes ------- christian@xxxxxxxxxxx -------
| Homepage: http://www.siefkes.net/ | Blog: http://www.keimform.de/
| Peer Production Everywhere: http://peerconomy.org/wiki/
|---------------------------------- OpenPGP Key ID: 0x346452D8 --
Politics is for people who have a passion for changing life but lack a
passion for living it.
-- Tom Robbins, Even Cowgirls Get the Blues
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ tor-talk mailing list tor-talk@xxxxxxxxxxxxxxxxxxxx https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk