Why doesn't "wget -r" work for /scratch/ directory of koji.fedoraproject.org?

[Michael Schwendt]

On Wed, 12 Mar 2008 03:06:59 +0900, Mamoru Tasaka wrote:

> Hello.
> 
> I remember that recursive and pattern matching download by "wget"
> against /scratch/ directory of koji.fedoraproject.org like:
> 
> $ wget -N -r -A.i386.rpm http://koji.fedoraproject.org/scratch/mtasaka/task_511288/
> 
> was working before. However it seems now this does not work, only returning
> like:
> -------------------------------------------------------------------------------------
> [tasaka1 at localhost gambas2]$ LANG=C wget -N -r -A.i386.rpm http://koji.fedoraproject.org/scratch/mtasaka/task_511288/
> --03:00:54--  http://koji.fedoraproject.org/scratch/mtasaka/task_511288/
>            => `koji.fedoraproject.org/scratch/mtasaka/task_511288/index.html'
> Resolving koji.fedoraproject.org... 209.132.176.98
> Connecting to koji.fedoraproject.org|209.132.176.98|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: unspecified [text/html]
> 
>     [  <=>                                                               ] 12,205        37.25K/s             
> 
> Last-modified header missing -- time-stamps turned off.
> 03:00:56 (37.14 KB/s) - `koji.fedoraproject.org/scratch/mtasaka/task_511288/index.html' saved [12205]
> 
> Loading robots.txt; please ignore errors.
> --03:00:56--  http://koji.fedoraproject.org/robots.txt
>            => `koji.fedoraproject.org/robots.txt'
> Connecting to koji.fedoraproject.org|209.132.176.98|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 26 [text/plain]
> Server file no newer than local file `koji.fedoraproject.org/robots.txt' -- not retrieving.
> 
> Removing koji.fedoraproject.org/scratch/mtasaka/task_511288/index.html since it should be rejected.
> 
> FINISHED --03:00:56--
> Downloaded: 12,205 bytes in 1 files
> -------------------------------------------------------------------------------------
> 
> Does someone know why? I really want to use this wget usage (perhaps
> browsing the above URL will show the reason :) )

Examine the contents of the robots.txt file.