[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Ht://Dig for go live
1) Fixed problem with htdig not indexing the site... turnedout to be
password related.
2) Ht://Dig currently places 114 documents in it's database which
exlcudess /mirror/. I hope people realize that this is completely screwy.
It turns out the reason for this is that we have a lot of documents
linking to other documents in a circular fasion but the URL changes!
I've actually known about this for sometime, but figured that it wasn't
important for us to worry about, because there is a work around.
The workaround is that for *EVERY* page that is NOT an actual
article/external link we need the following in the <HEAD></HEAD> block:
<meta name="robots" content="noindex">
This is very important from a search results listing and performance
standpoint. I would also suggest that article/ext links pages include the:
<meta name="robobts" content="nofollow">
tag so htdig won't try to (re)index links off of these pages.
--
Aaron Turner, Core Developer http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization http://www.linuxkb.org/
Because world domination requires quality open documentation.
aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
The difference between `Unstable' and `Usable' is only two characters: NT