[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Ht://Dig for go live
At 05:55 PM 01/23/2000 -0800, you wrote:
>
>1) Fixed problem with htdig not indexing the site... turnedout to be
>password related.
>
>2) Ht://Dig currently places 114 documents in it's database which
>exlcudess /mirror/. I hope people realize that this is completely screwy.
>It turns out the reason for this is that we have a lot of documents
>linking to other documents in a circular fasion but the URL changes!
>
>I've actually known about this for sometime, but figured that it wasn't
>important for us to worry about, because there is a work around.
>
>The workaround is that for *EVERY* page that is NOT an actual
>article/external link we need the following in the <HEAD></HEAD> block:
>
><meta name="robots" content="noindex">
>
>This is very important from a search results listing and performance
>standpoint. I would also suggest that article/ext links pages include the:
>
><meta name="robobts" content="nofollow">
That should actually be:
<meta name="robots" content="nofollow">
>
>tag so htdig won't try to (re)index links off of these pages.
>
>
>--
>Aaron Turner, Core Developer http://vodka.linuxkb.org/~aturner/
>Linux Knowledge Base Organization http://www.linuxkb.org/
>Because world domination requires quality open documentation.
>aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
>The difference between `Unstable' and `Usable' is only two characters: NT
>