stop web spiders

[Ryan Golhar]

Look up robots.exe on google.  You can use this file to indicate what
should be indexed and what shouldn't be.


-----Original Message-----
From: redhat-list-bounces at redhat.com
[mailto:redhat-list-bounces at redhat.com] On Behalf Of Anze Vidmar
Sent: Wednesday, January 25, 2006 1:19 PM
To: Redhat List
Subject: stop web spiders


hi!

I know this topic sounds like a spam, but it's not ;-)

I'm talking about web spiders that are collecting information from web
sites.

Is it possible to stop spiders to access specific folder on my web
server? e.g. http://myweb.com is public, but
http://myweb.com/confidential has some confidential files in it and I
don't want them to show while searching google for example. The password
protection (.htaccess) is *not* an option. I have "-Indexes" option set
on this folder, but still not sure if spiders can read them or not?

Thanks.
-- 
Anze Vidmar <anzevi at gmail.com>

-- 
redhat-list mailing list
unsubscribe mailto:redhat-list-request at redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list