Forum: Ruby How do I debug WWW::Mechanize problems?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
5730f209b34b8474639e0c2020f54513?d=identicon&s=25 dan (Guest)
on 2005-12-02 19:55
(Received via mailing list)
Hi, I'm trying to scrape my mileage summary from United.  Mechanize
starts out correctly, but then gets redirected back to the United
homepage instead of the Mileage Summary page.

Is there an rdoc for mechanize, as gems doesn't install one?  The only
examples I could find (besides the one it ships with) were
<http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc>  and
<http://www.zenspider.com/pipermail/ruby/2005-July/....  So,
I'm having trouble just figuring out the process to debug these errors.
 Is there anything equivalent to the  mech-dump that Perl's Mechanize
ships with
<http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/...?

Anyway, here's the code and the debugging output (run with my username
and password).  (If you have a United frequent flyer account, you can
replace username and password with your own to recreate.)



require 'mechanize'
agent = WWW::Mechanize.new{|a| a.log = Logger.new(STDERR) }

page = agent.get('http://www.united.com')
agent.user_agent_alias = 'Windows IE 6'

form = page.forms.find {|f| f.name == 'MileagePlus'}
form.fields.find {|f| f.name == 'userId'}.value = username
form.fields.find {|f| f.name == 'password'}.value = password
puts page.forms
page = agent.submit(form, form.buttons.first)

output = File.open("output.html", "w")
output.puts page.body
puts page.body



>ruby united.rb
I, [2005-12-02T10:02:10.791000 #4816]  INFO -- : GET:
http://www.united.com
D, [2005-12-02T10:02:10.961000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:12 GMT
D, [2005-12-02T10:02:11.021000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
I, [2005-12-02T10:02:11.512000 #4816]  INFO -- : status: 200
I, [2005-12-02T10:02:18.091000 #4816]  INFO -- : POST:
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOK...
warning: peer certificate won't be verified in this SSL session
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : query:
{"submit2"=>"Login", "sel_return_to"=>"&return_to=ff_acct_hist",
"userId"=>"username", "password"=>"password"}
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
referer => http://www.united.com
D, [2005-12-02T10:02:19.313000 #4816] DEBUG -- : request-header:
content-length => 92
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : cookie received:
testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: set-cookie :
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/, testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG -- : header: location :
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&s...
I, [2005-12-02T10:02:19.553000 #4816]  INFO -- : status: 302
I, [2005-12-02T10:02:19.553000 #4816]  INFO -- : follow redirect to:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&s...
I, [2005-12-02T10:02:19.553000 #4816]  INFO -- : GET:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&s...
warning: peer certificate won't be verified in this SSL session
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG -- : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOK...
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:20.174000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: set-cookie :
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.224000 #4816] DEBUG -- : header: location :
http://www.united.com/index/Home?stamp=NEWCOOKY*it...
I, [2005-12-02T10:02:20.234000 #4816]  INFO -- : status: 302
I, [2005-12-02T10:02:20.234000 #4816]  INFO -- : follow redirect to:
http://www.united.com/index/Home?stamp=NEWCOOKY*it...
I, [2005-12-02T10:02:20.234000 #4816]  INFO -- : GET:
http://www.united.com/index/Home?stamp=NEWCOOKY*it...
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG -- : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&s...
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received:
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : cookie received:
sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: connection :
close
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-type :
text/html
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: set-cookie :
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/, sessionid=NEWCOOKY*itn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: content-length
: 0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG -- : header: location :
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816]  INFO -- : status: 302
I, [2005-12-02T10:02:20.435000 #4816]  INFO -- : follow redirect to:
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816]  INFO -- : GET:
http://www.united.com/
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
sessionid=NEWCOOKY*itn/ord=NEWREC
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: accept
=> */*
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.665000 #4816] DEBUG -- : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e#<WWW::Form:0x34644c0>
#<WWW::Form:0x35fcd78>
#<WWW::Form:0x35d1530>
#<WWW::Form:0x35a71e8>
#<WWW::Form:0x35a3c78>
#<WWW::Form:0x35e8300>

          - dan
This topic is locked and can not be replied to.