We’re about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we’re finally
putting a bounty on AOL’s head.
Comparing the Firefox session with Mechanize’s shows they’re nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.
We’re offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.
If you’d like a starting point (which might not help much since it
ends up at an error page), here’s our code:
=============
require ‘rubygems’
require ‘mechanize’
agent = WWW::Mechanize.new
agent.user_agent_alias = ‘Windows IE 6’
page = agent.get( ‘http://webmail.aol.com’ )
page = agent.get( page.body.scan(/snsRedir(“(https:[^”]
+)/).first.first )
form = page.forms.name(‘AOLLoginForm’).first
form.loginId = ‘’
form.password = ‘’
page = agent.submit(form, form.buttons.first)
page = agent.get( page.body.scan(/onLoad="checkError[^)]
+/).first.scan(/‘([^’]+)'/).last.first )
=============
That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.
Good luck. In addition to your bounty, your work will also net the
community another gem.
Dave
Hi Dave,
I don’t think I can help you re: Mechanize and AOL Webmail (at the
moment at least).
Just wanted to make sure you’ve seen this:
http://rubyforge.org/projects/contacts/
… in case any pooling of efforts would help. Looking forward to
seeing your results!
Cheers,
[email protected] wrote:
We’re about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
Do you absolutely need mechanize?
Cheers,
Peter
http://www.rubyrailways.com
http://scrubyt.org
Just wanted to make sure you’ve seen this:http://rubyforge.org/projects/contacts/
We hadn’t - and looking now I’m not sure how we missed it. We’re doing
things a little different (we plan on having different exporters too
like csv, xml, etc) so I’m not sure how much merging could happen (but
I’ll check in with them). Unfortunately, they don’t have AOL figured
out either.
It’s weird that these PHP scripts (using curl) seem to be able to log
in fine but we can’t (and it seems MOG couldn’t either).
Quoth [email protected]:
Just wanted to make sure you’ve seen
this:http://rubyforge.org/projects/contacts/
We hadn’t - and looking now I’m not sure how we missed it. We’re doing
things a little different (we plan on having different exporters too
like csv, xml, etc) so I’m not sure how much merging could happen (but
I’ll check in with them). Unfortunately, they don’t have AOL figured
out either.
It’s weird that these PHP scripts (using curl) seem to be able to log
in fine but we can’t (and it seems MOG couldn’t either).
Could AOL be rejecting you based on UserAgent?
On Oct 31, 6:39 pm, Konrad M. [email protected] wrote:
out either.
1KDownload
This line
agent.user_agent_alias = ‘Windows IE 6’
in the sample code results in a UserAgent that matches IE 6.
Do you absolutely need mechanize?
It’s helpful and the other completed importers (Gmail, Yahoo, Hotmail)
use it already. To keep dependencies down I’d like it to use
Mechanize, but if you’ve got something using scRUBYt then I’m sure it
could be translated (or at least help highlight where Mechanize is
falling down).
Dave
AOL webmail using Mechanize, to the point where it greets you with "Hi,
". If that’s what you needed, then I have the solution for you.
mortee
If you want to send it directly to me, that’s fine. Also feel free to
paste it here.
Dave
I think I have it figured it out. I have a script that can log you into
AOL webmail using Mechanize, to the point where it greets you with "Hi,
". If that’s what you needed, then I have the solution for
you.
mortee
[email protected] wrote:
AOL webmail using Mechanize, to the point where it greets you with "Hi,
". If that’s what you needed, then I have the solution for you.
mortee
If you want to send it directly to me, that’s fine. Also feel free to
paste it here.
I’ve sent a private mail to you; have you received it?..
Sorry for the noise.
mortee
You’re the man, Marton! Consider the bounty won!
Marton found a bug in Mechanize’s #to_absolute_uri method caused by
the use of #zip which caused already-escaped characters beyond the
first to be dropped. He replaced it with some SyncEnumerator hackery.
I’ll let the Aaron P. know about the fix to Mechanize.
Good job Marton. We (and Blackbook and probably Mechanize, too)
appreciate your genius (and tolerance level for reading through that
nasty bit of Javascript to highlight the problem).
Dave
[email protected] wrote:
You’re the man, Marton! Consider the bounty won!
My pleasure (:
Marton found a bug in Mechanize’s #to_absolute_uri method caused by
the use of #zip which caused already-escaped characters beyond the
first to be dropped. He replaced it with some SyncEnumerator hackery.
I’ll let the Aaron P. know about the fix to Mechanize.
Something like that. Thanks if you let him know, then I won’t have to.
Good job Marton. We (and Blackbook and probably Mechanize, too)
appreciate your genius (and tolerance level for reading through that
nasty bit of Javascript to highlight the problem).
It was a nice task. Let me point out here that Dave and his company are
really generous, we got along pretty nicely.
regards
mortee