[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Apache robots.txt



On Wed, 2007-07-11 at 02:52 +0000, debu wrote:
>   Hi Group,
> 
> I was just trying to find all the tags that can be used inside apache robots.txt, any link, or any addition to the following?
> 
> i have these,
> 
> User-agent: 
> Disallow:
> Crawl-delay:
> Request-rate:
> Visit-time: 
> 
> Thanks
> Debajit kataki

According to http://www.robotstxt.org/wc/norobots.html only User-agent
and Disallow are valid. All other entries are user-agent specific (see
for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/)

-- 
Karl Latiss <karl latiss atvert com au>
Atvert Systems



[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]