app to look/view/inspect http traffic...
bruce
bedouglas at earthlink.net
Wed Jun 9 23:41:48 UTC 2004
mark...
after a thorough inspection of the headers... (finally!!).. the server
returns a 302 (object moved) along with the new location of the page...
so my question is, how can libwww be used to detect/process this
information. i assumed that the information should somehow be in the
request/response method of the useragent.. but i couldn't seem to get my
hands around how it should perform...
the headers are as follows:
========================================================
http://soc.our.psu.edu/soc/
GET /soc/ HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Cache-Control: max-age=0
HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:22:18 GMT
X-Powered-By: ASP.NET
Connection: close
Content-Type: text/html
Page-Completion-Status: Normal, Normal
----------------------------------------------------------
http://soc.our.psu.edu/soc/socheader.gif
GET /soc/socheader.gif HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept: image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
If-Modified-Since: Tue, 02 Oct 2001 20:42:12 GMT
If-None-Match: "70228fb6824bc11:13e4"
Cache-Control: max-age=0
HTTP/1.x 304 Not Modified
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:24:07 GMT
X-Powered-By: ASP.NET
Etag: "70228fb6824bc11:bd9"
Content-Length: 0
----------------------------------------------------------
http://soc.our.psu.edu/soc/main_search.cfm
POST /soc/main_search.cfm HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Content-Type: application/x-www-form-urlencoded
Content-Length: 91
Semester=FALL+2004&CrseLoc=AA%3A%3AAltoona+Campus&CECrseLoc=&CourseAbbrevAnd
Num=&CrseAlpha=
HTTP/1.x 200 OK
Server: Microsoft-IIS/5.0
Date: Wed, 09 Jun 2004 23:24:13 GMT
X-Powered-By: ASP.NET
Connection: close
Content-Type: text/html
Page-Completion-Status: Normal, Normal
----------------------------------------------------------
http://soc.our.psu.edu/soc/main_search.cfm
POST /soc/main_search.cfm HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/main_search.cfm
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
Content-Type: application/x-www-form-urlencoded
Content-Length: 115
Semester=FALL+2004&CrseLoc=AA%3A%3AAltoona+Campus&CECrseLoc=&CourseAbbrevAnd
Num=&CrseAlpha=a-c&Search=View+schedule
HTTP/1.x 302 Object Moved
Location:
act_main_search.cfm?Semester=FALL%202004&CrseLoc=AA%3A%3AAltoona%20Campus&CE
CrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum=0
Server: Microsoft-IIS/5.0
Content-Type: text/html
Content-Length: 270
----------------------------------------------------------
http://soc.our.psu.edu/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=
AA%3A%3AAltoona%20Campus&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum
=0
GET
/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=AA%3A%3AAltoona%20Camp
us&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum=0 HTTP/1.1
Host: soc.our.psu.edu
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040608
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=
0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://soc.our.psu.edu/soc/main_search.cfm
Cookie: SOCCECRSELOCCOOKIE=AA%3A%3AAltoona+Campus;
SOCCRSELOCCOOKIE=AA%3A%3AAltoona+Campus; SOCSEMCOOKIE=FALL+2004; CFID=9420;
CFTOKEN=86341880
========================================================
++++>>>>>>>>HTTP/1.x 302 Object Moved <<<<<<+++++++
thanks...
-bruce
tom...
the following url works in a browser window. you can select it, and the
appropriate page appears...
http://soc.our.psu.edu/soc/act_main_search.cfm?Semester=FALL%202004&CrseLoc=
AA%3A%3AAltoona%20Campus&CECrseLoc=&CourseAbbrev=0&CrseAlpha=a%2Dc&CourseNum
=0
what i'm trying to get a better understanding of, is why when this is used
with libwww/wget/etc.. the correct page is not returned.. i've disabled
cookies on my test IE browser so it shouldn't be a cookie issue..
(and yeah.. you were correct when you pointed out that all this really does
is redirect to a basic tree structure!!! which kind of makes this effort
pointless.. but i need to get a better understanding as to what the
problem/issue is that i'm missing...)
if anybody has any insight, i'd appreciate it!
i took a quick look at the headers using windump... (i've cut out alot of
the garbage to give a feel as to what appears to be happening...)
flags [DF], length: 691, bad cksum 0 (->a17c)!) sys2.mesa.com.2327 >
soc.our.psu.edu.80: P 366:1017(651) ack 165 win 17356
0x0000: 4500 02b3 6ad1 4000 8006 0000 c0a8 0103 E...j. at .........
0x0010: 92ba 9791 0917 0050 4c75 f500 da9b f421 .......PLu.....!
0x0020: 5018 43cc ee9c 0000 504f 5354 202f 736f P.C.....POST./so
0x0030: 632f 6d61 696e 5f73 6561 7263 682e 6366 c/main_search.cf
0x0040: 6d20 4854 5450 2f31 2e31 0d0a 4163 6365 m.HTTP/1.1..Acce
0x0050: 7074 pt
.
.
.
0x0000: 4500 0028 6ad5 4000 8006 0000 c0a8 0103 E..(j. at .........
0x0010: 92ba 9791 0917 0050 4c75 f78b da9b f491 .......PLu......
0x0020: 5010 435c ec11 0000 P.C\....
12:21:17.747572 IP (tos 0x0, ttl 128, id 27351, offset 0, flags [DF],
length: 617, bad cksum 0 (->a1c0)!) sys2.mesa.com.2327 > soc.our.psu.edu.80:
P 1017:1594(577) ack 788 win 16733
0x0000: 4500 0269 6ad7 4000 8006 0000 c0a8 0103 E..ij. at .........
0x0010: 92ba 9791 0917 0050 4c75 f78b da9b f690 .......PLu......
0x0020: 5018 415d ee52 0000 4745 5420 2f73 6f63 P.A].R..GET./soc
0x0030: 2f61 6374 5f6d 6169 6e5f 7365 6172 6368 /act_main_search
0x0040: 2e63 666d 3f53 656d 6573 7465 723d 4641 .cfm?Semester=FA
0x0050: 4c4c LL
12:21:17.747619 IP (tos 0x0, ttl 128, id 27351, offset 0, flags [DF],
length: 617, bad cksum 0 (->a1c0)!) sys2.mesa.com.2327 > soc.our.psu.edu.80:
P 1017:1594(577) ack 788 win 16733
0x0000: 4500 0269 6ad7 4000 8006 0000 c0a8 0103 E..ij. at .........
0x0010: 92ba 9791 0917 0050 4c75 f78b da9b f690 .......PLu......
0x0020: 5018 415d ee52 0000 4745 5420 2f73 6f63 P.A].R..GET./soc
0x0030: 2f61 6374 5f6d 6169 6e5f 7365 6172 6368 /act_main_search
0x0040: 2e63 666d 3f53 656d 6573 7465 723d 4641 .cfm?Semester=FA
0x0050: 4c4c LL
.
.
.
12:21:18.270385 IP (tos 0x0, ttl 128, id 27360, offset 0, flags [DF],
length: 40, bad cksum 0 (->a3f8)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
. [bad tcp cksum ec11 (->e378)!] 0:0(0) ack 1 win 17520
0x0000: 4500 0028 6ae0 4000 8006 0000 c0a8 0103 E..(j. at .........
0x0010: 92ba 9791 091a 0050 4c95 8117 dab9 ea23 .......PL......#
0x0020: 5010 4470 ec11 0000 P.Dp....
12:21:18.272296 IP (tos 0x0, ttl 128, id 27361, offset 0, flags [DF],
length: 522, bad cksum 0 (->a215)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
P 0:482(482) ack 1 win 17520
0x0000: 4500 020a 6ae1 4000 8006 0000 c0a8 0103 E...j. at .........
0x0010: 92ba 9791 091a 0050 4c95 8117 dab9 ea23 .......PL......#
0x0020: 5018 4470 edf3 0000 4745 5420 2f73 6f63 P.Dp....GET./soc
0x0030: 2f66 616c 6c2f 6161 2f61 2d63 2f69 6e64 /fall/aa/a-c/ind
0x0040: 6578 2e68 746d 6c20 4854 5450 2f31 2e31 ex.html.HTTP/1.1
0x0050: 0d0a ..
12:21:18.272342 IP (tos 0x0, ttl 128, id 27361, offset 0, flags [DF],
length: 522, bad cksum 0 (->a215)!) sys2.mesa.com.2330 > soc.our.psu.edu.80:
P 0:482(482) ack 1 win 17520
0x0000: 4500 020a 6ae1 4000 8006 0000 c0a8 0103 E...j. at .........
0x0010: 92ba 9791 091a 0050 4c95 8117 dab9 ea23 .......PL......#
0x0020: 5018 4470 edf3 0000 4745 5420 2f73 6f63 P.Dp....GET./soc
0x0030: 2f66 616c 6c2f 6161 2f61 2d63 2f69 6e64 /fall/aa/a-c/ind
0x0040: 6578 2e68 746d 6c20 4854 5450 2f31 2e31 ex.html.HTTP/1.1
0x0050: 0d0a ..
as you can see... the client initially sends a "..main_search.cfm.. which is
redirected towards ..act_main_search.cfm...to finally
.../fall/aa/a-c/index.html.."
the overall issue i have is how to set this up using libwww/mechanize/etc...
thanks...
-bruce
-----Original Message-----
From: perl-win32-users-bounces at listserv.ActiveState.com
[mailto:perl-win32-users-bounces at listserv.ActiveState.com]On Behalf Of
Thomas, Mark - BLS CTR
Sent: Wednesday, June 09, 2004 11:25 AM
To: 'bedouglas at earthlink.net'; 'General Red Hat Linux discussion list';
perl-win32-users at listserv.ActiveState.com
Subject: RE: app to look/view/inspect http traffic...
Bruce,
I suggest looking at the 'Live HTTP headers' plugin for Mozilla/Firefox. It
may be just what you're looking for. It allows you to see the HTTP headers
in real time, and lets you edit request headers.
The extension is one of many available here:
http://extensionroom.mozdev.org/
--
Mark Thomas Thomas.Mark at bls.gov
Internet Systems Architect DigitalNet, Inc.
$_=q;KvtuyboopuifeyQQfeemyibdlfee;; y.e.s. ;y+B-x+A-w+s; ;y;y; ;;print;;
_______________________________________________
Perl-Win32-Users mailing list
Perl-Win32-Users at listserv.ActiveState.com
To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
More information about the redhat-list
mailing list