stop web spiders

[Sean O Sullivan]

Anze Vidmar wrote:
> Is it possible to stop spiders to access specific folder on my web
> server? e.g. http://myweb.com is public, but
> http://myweb.com/confidential has some confidential files in it and I
> don't want them to show while searching google for example. The password
> protection (.htaccess) is *not* an option. I have "-Indexes" option set
> on this folder, but still not sure if spiders can read them or not?
> 

Look into robots.txt

Regards,

Sean