Alexander Y. wrote:
services APIs that eBay makes available? There may be much easier ways
to
get at data than screen scraping. Look at the “Developers” link at the
bottom of their homepage.
This is unquestionably true: no web scraper will ever (?) give you 100%
reliability and robustness and … that a WS API can offer. If your
thing
is mission critical, and the API offers everything you need, no one is
questioning that it is the way to go - even if the other alternative is
scRUBYt! 
Hmm looks like good advice, could you perhaps give me an example of how
I could apply this with ruby? (I’m still new)
Do you have the Ruby Cookbook? There is one recipe for using the ebay
API…
The scraping i will need
to do is going to be pretty advanced I think
The present version of scRUBYt! (about 0.0.8) works like this:
ebay_data = Scrubyt::Extractor define do
#Navigate to our page
fetch ‘ebay.com’
fill_textfield ‘dell laptop’
submit
click_link ‘Laptops/Notebooks’
#Construct the scraper
record do
name “DELL LATITUDE C600 P3 1.0 LAPTOP NOTEBOOK 256MB 20GB HD”
price “$192.50”
shipping “$37.00”
end.ensure_presence_of_pattern(:price)
next_page “8 9 Next”, :limit => 5
end
So, this extractor does the following: first navigates to your page of
interest automatically as you would do it (powered by Mechanize - from
0.2.0 I would like to add Watir as well so it can handle JavaScript,
then later on Selenium), Then the scraper is constructed.
As you can see, there is no need for any technical knowledge (powered by
Hpricot); You just show an example record and all the others are
extracted automatically. Moreover, only those records are extracted
which have a price (i.e. not the ‘buy now’ ones). Another goodie for
free is the navigation to the next pages - in this case the first 5
pages (you can do also unlimited as well).
Now, this is just a part of the functionality even of the version which
is on my HDD - I am planning to add more stuff into 0.1.0 - and even
that will be just the top of the iceberg of the features I am planning
to add long term.
And of course the best thing is that it works - it’s not a toy (at
least not on the pages I have tried it so far) - and though I am sure
there will be problems with the earlier versions, I count on the
community to help to catch these and propose features important for them
so I guess the future might be bright for scRUBYt!
Cheers,
Peter
__
http://www.rubyrailways.com
- basically I am