Need advice for a script to scrap my Verizon account

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have
found one Perl and one Java/Firefox solution that does the same, but I
thought it would be a good project to learn more about Ruby. I would
rather not use external shells (to wget, for example) and rely on the
internal Ruby libraries instead. I’m looking for advice on what
libraries to use to get me started down the right path. Thank you.

On 5/4/07, Steve [email protected] wrote:

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have

I haven’t used it myself, but if you might want to look at
WWW::Mechanize, which handles things like cookies for you. I found a
good looking blog post about it here:

http://neurogami.com/cafe-fetcher/

Another excellent library is open-uri, which comes standard with Ruby.
If Verizon does not have complex cookie requirements, you might be
able to handle the details yourself and use open-uri.

Good luck!

Justin

I would suggest Net::HTTP

Although if you really want to scrap your account you could just call
them
:wink:

On 5/4/07, Steve [email protected] wrote:


“Hey brother christian with your high and mighty errand, Your actions
speak
so loud, I can’t hear a word you’re saying.”

-Greg Graffin (Bad Religion)

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have
found one Perl and one Java/Firefox solution that does the same, but I
thought it would be a good project to learn more about Ruby. I would
rather not use external shells (to wget, for example) and rely on the
internal Ruby libraries instead. I’m looking for advice on what
libraries to use to get me started down the right path. Thank you.

These will probably help…

for fetching the page…

http://www.ruby-doc.org/core/classes/OpenURI.html
http://curb.rubyforge.org/

for parsing it:

On May 4, 5:49 pm, Steve [email protected] wrote:

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have
found one Perl and one Java/Firefox solution that does the same, but I
thought it would be a good project to learn more about Ruby. I would
rather not use external shells (to wget, for example) and rely on the
internal Ruby libraries instead. I’m looking for advice on what
libraries to use to get me started down the right path. Thank you.

It’s not fancy, but I did pretty much the same thing you are talking
about with curb (a wrapper for lib curl) and hpricot. You can see a
quick solution I can up with here:
http://exawkuser.blogspot.com/2007/02/stickittodamanitis.html.
Verizon has probably changed their site in a way that now breaks my
script but you should be able to see the basics of web scraping in
ruby.

-Brenton

Check out Watir also.

On Friday 04 May 2007 21:49, Steve wrote:

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month. This would involve opening an
HTTPS connection, then scraping and calculating the results. I have
found one Perl and one Java/Firefox solution that does the same, but I
thought it would be a good project to learn more about Ruby. I would
rather not use external shells (to wget, for example) and rely on the
internal Ruby libraries instead. I’m looking for advice on what
libraries to use to get me started down the right path. Thank you.

try http://www.scrubyt.org - great ruby toolkit for webscraping
and Net:Mechanize

On Sunday 06 May 2007 17:47, Peter S. wrote:

Cheers,
Peter
__
http://www.rubyrailways.com :: Ruby and Web2.0 blog
http://scrubyt.org :: Ruby web scraping framework
http://rubykitchensink.ca/ :: The indexed archive of all things Ruby

i had problem with site with javascripts and i must say most of it is
easly
fixable - using firewatir is overkill in my opinion

try http://www.scrubyt.org - great ruby toolkit for webscraping

Yeah, scRUBYt! is really great and all, but currently it has one major
shortcoming from the viewpoint of verizon scraping: it is built on
WWW::Mechanize, which can not handle javascript/AJAX and it seems the
verizon page has plenty of those.

FireWatir is just being integrated into scRUBYt! and scraping such a
site won’t be a problem once it will be finished - however, until then I
would suggest to use either Watir or FireWatir to handle the AJAXy
stuff.

Cheers,
Peter
__
http://www.rubyrailways.com :: Ruby and Web2.0 blog
http://scrubyt.org :: Ruby web scraping framework
http://rubykitchensink.ca/ :: The indexed archive of all things Ruby

Steve wrote:

I am new to Ruby. I would like to write a script that will login to
my Verizon account and grab my minute details and calculate my
remaining unused minutes for the month.

I just did something similar for our electronic
time sheet – I had to negotiate a frameset, viz,

URL = ‘https://site.somewhere.gov/
require ‘rubygems’; require ‘mechanize’
agent = WWW::Mechanize.new
form = agent.get("#{URL}").forms.name(‘loginForm’).first
form.userid = ARGV[0] # 1st commandline argument
form.password = ARGV[1] # 2nd " "
frame = agent.submit(form).frames.name(‘Main’).href
form = agent.get("#{URL}/#{frame}").forms.name(‘timeCard’).first
form.ENTRY_24378_1_3 = ‘2.25’ # set hours for 1st week, 3rd day
agent.submit(form)

The next step was to use Hpricot to scrape the relations
of table inputs to time codes, but I haven’t gotten there
yet…

Regards,

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs