[tor-commits] [metrics-web/master] Add a robots.txt to discourage indexing database lookups.

[karsten@xxxxxxxxxxxxxx]

commit 46264713a550421c358e471a1dd1c50f20a46e5f
Author: Karsten Loesing <karsten.loesing@xxxxxxx>
Date:   Mon Sep 12 08:21:42 2011 +0200

    Add a robots.txt to discourage indexing database lookups.
---
 web/robots.txt |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/web/robots.txt b/web/robots.txt
new file mode 100644
index 0000000..3b8e76e
--- /dev/null
+++ b/web/robots.txt
@@ -0,0 +1,12 @@
+User-agent: *
+Disallow: /networkstatus.html
+Disallow: /routerdetail.html
+Disallow: /relay.html
+Disallow: /csv/
+Disallow: /serverdesc
+Disallow: /extrainfodesc
+Disallow: /descriptor.html
+Disallow: /consensus
+Disallow: /votes
+Disallow: /consensus-health.html
+

_______________________________________________
tor-commits mailing list
tor-commits@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits