[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-talk] Hidden Services

On 09/19/2012 10:13 AM, tor@xxxxxxxxxxxxxxxxxx wrote:
> On 19/09/12 06:36, grarpamp wrote:
>>> People use robots.txt to indicate that they don't want their site
>>> to be added to indexes.
>> They use it to indicate that they don't want their site to be
>> crawled.
> In almost all cases (99% or higher), robots.txt is used to indicate
> that a site shouldn't be crawled, *because* they don't want it to be
> indexed. The intention is painfully clear...

If website owners don't want a page to be indexed, they should use the
noindex meta tag: http://en.wikipedia.org/wiki/Noindex . robots.txt is only
for crawlers that automatically follow links from one page to others.

Neither standard prevents or discourages manually setting a link to the page.

Best regards

|------- Dr. Christian Siefkes ------- christian@xxxxxxxxxxx -------
| Homepage: http://www.siefkes.net/ | Blog: http://www.keimform.de/
|    Peer Production Everywhere:       http://peerconomy.org/wiki/
|---------------------------------- OpenPGP Key ID: 0x346452D8 --
Politics is for people who have a passion for changing life but lack a
passion for living it.
        -- Tom Robbins, Even Cowgirls Get the Blues

Attachment: signature.asc
Description: OpenPGP digital signature

tor-talk mailing list