http download automation script?

[Tobias Speckbacher]

> -----Original Message-----
> From: redhat-list-bounces at redhat.com [mailto:redhat-list-
> bounces at redhat.com] On Behalf Of Magnus Andersen
> Sent: Tuesday, December 06, 2005 11:38 AM
> To: General Red Hat Linux discussion list
> Subject: http download automation script?
> 
> I need to download files from a website and I'd like to automate it.
I
> have
> to login to the website, navigate to the download section and download
the
> files.  They do not have an ftp site and I have to do this over http.
The
> system I'll be doing this from is a RHEL 3 As system.

Ages ago I wrote this to automate the download of a text file.
Obviously I did not use it exactly as outlined below, it should however
give you the basic idea of how it can work.

You may or may not need to work with cookies, depending on the site you
are working with.

Hope this helps.

use LWP::UserAgent;
use HTTP::Request::Common qw(POST);
use HTTP::Cookies;

my $url1 ='http://www.example.com/login';
my $url2 ='http://www.example.com/download/big/file/here';

my $ua = new LWP::UserAgent(keep_alive => 1);
$ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
                                   autosave => 1,
                                   ignore_discard => 1));
$ua->agent("Quova Data Retrieval/0.1");
my $req = POST $url1,
                    [
                        passwd  =>      'password',
                        user    =>      'username',
                        ];
my $response = $ua->request($req);
if (!$response->is_success){
	die $response->error_as_HTML;
            
}
$response = $ua->get($url2);
if (!$response->is_success){
        die $response->error_as_HTML;
}
open (FILE,">$http_fn");
print FILE $response->content;
close FILE;



> 
> Any ideas on how this can be accomplished.
> 
> --
> Magnus Andersen
> Systems Administrator / Oracle DBA
> Walker & Associates, Inc.
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request at redhat.com?subject=subscribe
> https://www.redhat.com/mailman/listinfo/redhat-list




======================
Tobias Speckbacher
Manager, IT Operations
Quova, Inc.
p: 650-528-3704
e: tobias at quova.com
f: 650-625-9801 
======================