Scrubyt scraper help

Hello all. I’m trying to build a simple web scraper to mine some data
off the yellow pages. Specifically, this link:

I’m scraping all the information that I need correctly. I’m very
pleased about that! However, I’m only able to scrape the first page. I
want my script to automatically go to the next page after the first one
has been scraped, and the next after that. Scrubyt’s “next_page”
function can do this, but it can only use a full URL. On this website,
however, the “Next” link at the bottom is a relative link. Is there any
way I might be able to grab the URL of the website and add the relative
link onto it, and then go to the next page? Or is there another way of
doing it? I really appreciate the help! Thanks so much.

My code is as follows:

require ‘rubygems’
require ‘scrubyt’

yellowpages_data = Scrubyt::Extractor.define do

#Perform the action(s)
fetch ‘

This part does the scraping

listing “//div[@class=‘listing_content’]” do
name “Pascucci”
#street “792 State St,”
street “//span[@class=‘street-address’]”
city “//span[@class=‘locality’]”
state “//span[@class=‘region’]”
zip_code “//span[@class=‘postal-code’]”
phone “//span[@class=‘business-phone phone’]”

       # This is the function I was talking about.  It needs a full

link to work, but I only have a relative one!
next_page “Next”, :limit => 2

puts yellowpages_data.to_xml.write($stdout, 1)

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs