Apache robots.txt

[Waleed Harbi]

http://www.redhat.com/robots.txt

On 7/11/07, Karl Latiss <karl.latiss at atvert.com.au> wrote:
>
> On Wed, 2007-07-11 at 02:52 +0000, debu wrote:
> >   Hi Group,
> >
> > I was just trying to find all the tags that can be used inside apache
> robots.txt, any link, or any addition to the following?
> >
> > i have these,
> >
> > User-agent:
> > Disallow:
> > Crawl-delay:
> > Request-rate:
> > Visit-time:
> >
> > Thanks
> > Debajit kataki
>
> According to http://www.robotstxt.org/wc/norobots.html only User-agent
> and Disallow are valid. All other entries are user-agent specific (see
> for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/)
>
> --
> Karl Latiss <karl.latiss at atvert.com.au>
> Atvert Systems
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>



-- 
:. Best Wishes
:. Waleed Harbi
---------------------------------------------
:. Never too old to learn
:. Every why has a wherefore
:. Grasp all, lose all