How do I debug WWW::Mechanize problems?


#1

Hi, I’m trying to scrape my mileage summary from United. Mechanize
starts out correctly, but then gets redirected back to the United
homepage instead of the Mileage Summary page.

Is there an rdoc for mechanize, as gems doesn’t install one? The only
examples I could find (besides the one it ships with) were
http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc and
http://www.zenspider.com/pipermail/ruby/2005-July/002068.html. So,
I’m having trouble just figuring out the process to debug these errors.
Is there anything equivalent to the mech-dump that Perl’s Mechanize
ships with
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod?

Anyway, here’s the code and the debugging output (run with my username
and password). (If you have a United frequent flyer account, you can
replace username and password with your own to recreate.)

require ‘mechanize’
agent = WWW::Mechanize.new{|a| a.log = Logger.new(STDERR) }

page = agent.get(‘http://www.united.com’)
agent.user_agent_alias = ‘Windows IE 6’

form = page.forms.find {|f| f.name == ‘MileagePlus’}
form.fields.find {|f| f.name == ‘userId’}.value = username
form.fields.find {|f| f.name == ‘password’}.value = password
puts page.forms
page = agent.submit(form, form.buttons.first)

output = File.open(“output.html”, “w”)
output.puts page.body
puts page.body

ruby united.rb
I, [2005-12-02T10:02:10.791000 #4816] INFO – : GET:
http://www.united.com
D, [2005-12-02T10:02:10.961000 #4816] DEBUG – : request-header: accept
=> /
D, [2005-12-02T10:02:11.021000 #4816] DEBUG – : header: connection :
close
D, [2005-12-02T10:02:11.021000 #4816] DEBUG – : header: content-type :
text/html
D, [2005-12-02T10:02:11.021000 #4816] DEBUG – : header: date : Fri, 02
Dec 2005 18:02:12 GMT
D, [2005-12-02T10:02:11.021000 #4816] DEBUG – : header: server :
Netscape-Enterprise/6.0
I, [2005-12-02T10:02:11.512000 #4816] INFO – : status: 200
I, [2005-12-02T10:02:18.091000 #4816] INFO – : POST:
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKYitn/ord=NEWREC,itn/air/united
warning: peer certificate won’t be verified in this SSL session
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : query:
{“submit2”=>“Login”, “sel_return_to”=>"&return_to=ff_acct_hist",
“userId”=>“username”, “password”=>“password”}
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : request-header: accept
=> /
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : request-header:
content-type => application/x-www-form-urlencoded
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : request-header:
referer => http://www.united.com
D, [2005-12-02T10:02:19.313000 #4816] DEBUG – : request-header:
content-length => 92
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : cookie received:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : cookie received:
testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: connection :
close
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: content-type :
text/html
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: set-cookie :
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
path=/, testcookie=1
D, [2005-12-02T10:02:19.553000 #4816] DEBUG – : header: location :
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY
itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:19.553000 #4816] INFO – : status: 302
I, [2005-12-02T10:02:19.553000 #4816] INFO – : follow redirect to:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKYitn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:19.553000 #4816] INFO – : GET:
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKY
itn/ord=NEWREC,itn/air/united
warning: peer certificate won’t be verified in this SSL session
D, [2005-12-02T10:02:19.824000 #4816] DEBUG – : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG – : request-header: accept
=> /
D, [2005-12-02T10:02:19.824000 #4816] DEBUG – : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:19.824000 #4816] DEBUG – : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1
D, [2005-12-02T10:02:19.824000 #4816] DEBUG – : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?stamp=NEWCOOKYitn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : cookie received:
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : cookie received:
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : header: connection :
close
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : header: content-type :
text/html
D, [2005-12-02T10:02:20.174000 #4816] DEBUG – : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.224000 #4816] DEBUG – : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.224000 #4816] DEBUG – : header: set-cookie :
rememberMP=-1; domain=.ua2go.com; expires=Friday, 02-Dec-2005 18:02:23
GMT; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
domain=.ua2go.com; path=/,
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
domain=.ua2go.com; path=/
D, [2005-12-02T10:02:20.224000 #4816] DEBUG – : header: location :
http://www.united.com/index/Home?stamp=NEWCOOKY
itn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:20.234000 #4816] INFO – : status: 302
I, [2005-12-02T10:02:20.234000 #4816] INFO – : follow redirect to:
http://www.united.com/index/Home?stamp=NEWCOOKYitn/ord=NEWREC,itn/air/united
I, [2005-12-02T10:02:20.234000 #4816] INFO – : GET:
http://www.united.com/index/Home?stamp=NEWCOOKY
itn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.325000 #4816] DEBUG – : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG – : request-header: accept
=> /
D, [2005-12-02T10:02:20.325000 #4816] DEBUG – : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.325000 #4816] DEBUG – : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1
D, [2005-12-02T10:02:20.325000 #4816] DEBUG – : request-header:
referer =>
https://www.ua2go.com/ci/DoLogin.jsp?setCookie=1&stamp=NEWCOOKYitn/ord=NEWREC,itn/air/united
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : cookie received:
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : cookie received:
sessionid=NEWCOOKY
itn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: connection :
close
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: content-type :
text/html
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: date : Fri, 02
Dec 2005 18:02:21 GMT
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: server :
Netscape-Enterprise/6.0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: set-cookie :
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
path=/, sessionid=NEWCOOKYitn/ord=NEWREC; domain=.united.com; path=/
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: content-length
: 0
D, [2005-12-02T10:02:20.435000 #4816] DEBUG – : header: location :
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816] INFO – : status: 302
I, [2005-12-02T10:02:20.435000 #4816] INFO – : follow redirect to:
http://www.united.com/
I, [2005-12-02T10:02:20.435000 #4816] INFO – : GET:
http://www.united.com/
D, [2005-12-02T10:02:20.665000 #4816] DEBUG – : use cookie:
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5c6931842a1f415c7d123f896c82db46a0;
ciToken=792ad6bc2fb42b7d178c34e2f98fab5cf8cf3f5e1ce627d58e2ed9576b828ef1;
united=DQMujVlTtYtjKEjdBoIwk2nLn1vdm82kMn1pSndGtFNa7nIBOEFy!-232476364;
sessionid=NEWCOOKY
itn/ord=NEWREC
D, [2005-12-02T10:02:20.665000 #4816] DEBUG – : request-header: accept
=> /
D, [2005-12-02T10:02:20.665000 #4816] DEBUG – : request-header:
user-agent => Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
D, [2005-12-02T10:02:20.665000 #4816] DEBUG – : request-header: cookie
=>
JSESSIONID=DQMtsIaIkMsQbuTwzMK9Nciy064tQh1KBNY1EB1Q3M3ibLa5g7J9!151650267;
testcookie=1; rememberMP=-1;
ciToken=792ad6bc2fb42b7d178c34e#WWW::Form:0x34644c0
#WWW::Form:0x35fcd78
#WWW::Form:0x35d1530
#WWW::Form:0x35a71e8
#WWW::Form:0x35a3c78
#WWW::Form:0x35e8300

      - dan