Re: [tor-talk] Hidden Services

[tor@xxxxxxxxxxxxxxxxxx]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 19/09/12 06:36, grarpamp wrote:

>> People use robots.txt to indicate that they don't want their site
>> to be added to indexes.
> 
> They use it to indicate that they don't want their site to be
> crawled.

In almost all cases (99% or higher), robots.txt is used to indicate
that a site shouldn't be crawled, *because* they don't want it to be
indexed. The intention is painfully clear...

- -- 
Mike Cardwell  https://grepular.com/     http://cardwellit.com/
OpenPGP Key    35BC AF1D 3AA2 1F84 3DC3  B0CF 70A5 F512 0018 461F
XMPP OTR Key   8924 B06A 7917 AAF3 DBB1  BF1B 295C 3C78 3EF1 46B4
-----BEGIN PGP SIGNATURE-----

iQGGBAEBCgBwBQJQWX7AMBSAAAAAACAAB3ByZWZlcnJlZC1lbWFpbC1lbmNvZGlu
Z0BwZ3AuY29tcGdwbWltZTgUgAAAAAAVABpwa2EtYWRkcmVzc0BnbnVwZy5vcmdt
aWtlLmNhcmR3ZWxsQGdyZXB1bGFyLmNvbQAKCRCdJiMBwdHnBOh6B/0bF8fGP3xt
UmflWZ3wCK0dBj0AQyQpdvrbqI6yHEVlzvs1M3qKtAaSInuz/Bb/tFOsd8JVhoEG
/bu+YsnOwk5qEifywe34EFjFcUIeLu+badF+O57aCJoFdu4YcK4k7/7ZdmxAvpj8
9U73BywJO2HM3fe2pdMeEf0eNZUkOkjFJ2DvpaiRU4T/rv5x/zP1HrsEyRaHsLVp
pbZTSb8QEgNJDUJ+r5+1BQVgKX1lw84Qp+MOD1VfyzqZImdu4ah4y3zrEhSweyLa
qlNDqs1BZot7rIJhJKrJIuGSWnyWj1DWSkv9cyXmhwYEgDcK8K1+sDrrVGoQdSBq
7ms4gtZhioaH
=btT8
-----END PGP SIGNATURE-----
_______________________________________________
tor-talk mailing list
tor-talk@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk