Apache robots.txt

[Karl Latiss]

On Wed, 2007-07-11 at 02:52 +0000, debu wrote:
>   Hi Group,
> 
> I was just trying to find all the tags that can be used inside apache robots.txt, any link, or any addition to the following?
> 
> i have these,
> 
> User-agent: 
> Disallow:
> Crawl-delay:
> Request-rate:
> Visit-time: 
> 
> Thanks
> Debajit kataki

According to http://www.robotstxt.org/wc/norobots.html only User-agent
and Disallow are valid. All other entries are user-agent specific (see
for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/)

-- 
Karl Latiss <karl.latiss at atvert.com.au>
Atvert Systems