Wget question...

bruce bedouglas at earthlink.net
Thu Jun 22 16:32:25 UTC 2006


hi...

this question is for the wget guru!

i'm testing wget on a test site.. i'm using the recursive function of wget
to crawl through a portion of the site...

it appears that wget is hitting a link within the crawl that's causing it to
begin to crawl through the section of the site again...

i know wget isn't as robust as nutch, but can someone tell me if wget keeps
a track of the URLs that it's bben through so it doesn't repeat/get stuck in
a never ending processs...

i haven't run across anything in the docs that seems to speak to this
point..

thanks

-bruce




More information about the fedora-list mailing list