stop web spiders
Ed Wilts
ewilts at ewilts.org
Wed Jan 25 18:36:57 UTC 2006
On Wed, Jan 25, 2006 at 07:19:18PM +0100, Anze Vidmar wrote:
> Is it possible to stop spiders to access specific folder on my web
> server? e.g. http://myweb.com is public, but
> http://myweb.com/confidential has some confidential files in it and I
> don't want them to show while searching google for example. The password
> protection (.htaccess) is *not* an option. I have "-Indexes" option set
> on this folder, but still not sure if spiders can read them or not?
http://www.google.com/webmasters/remove.html
Please note that although the robots.txt file is a standard, there is no
guarantee that all spiders will honor it. Google does though.
.../Ed
--
Ed Wilts, RHCE
Mounds View, MN, USA
mailto:ewilts at ewilts.org
Member #1, Red Hat Community Ambassador Program
More information about the redhat-list
mailing list