Re: Analog report available

[Aaron Turner <aturner@linuxkb.org>]

On Thu, 1 Jul 1999, Micah Yoder wrote:

> Aaron Turner wrote:
> > 
> > http://admin.linuxkb.org/analog.html
> 
> Doesn't exist

Sorry, https://....
 
> > 1)  Lots of spiders are hitting us already
> 
> Do we have a /robots.txt?  Does anyone know how to make them?

No we don't.  Yes I do.

Problem is that robots.txt isn't a solution.  Basically there's a lot of
web spiders out there that ignore the file.  There's even spiders that
read the file just to findout what to index!

A true solution is to use a mod_perl module in Apache to do some kind of
rate-limiting.  If someone does 50 page requests in 10 seconds it's not
human and we should do something (what exactly I'm not sure).

> > 3)  People came to us from virtually every point on the globe- from
> > Argentina to Quatar.
> 
> Sweet beans.