# Page crawling and URL grabbing

**URL:** <https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704>\
**Category:** Ruby\
**Created:** [January 27, 2009, 1:57am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704 "2009-01-27T01:57:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lynchnco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/c77e96/32.png) [@lynchnco](https://www.ruby-forum.com/u/lynchnco)\
**Post date:** [January 27, 2009, 1:57am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704/1 "2009-01-27T01:57:21Z")

</div>

Hey guys,  
I’m trying to write an application that goes onto a website (istockphoto  
specifically), opens up [istockphoto.com/file\_browse.php](http://istockphoto.com/file_browse.php) and grabs the  
URLs of the photos that appear there.

It’s my first time doing something like this. I’m reading some  
documentation right now…but a hand would be greatly appreciated. I’m  
not really sure how to do regex on an html file…or even find the right  
stuff within that file. I’m guessing its…

open(‘[http://www.istockphoto.com/file\_browse.php/](http://www.istockphoto.com/file_browse.php/)’) do |f|  
f.find # dot something something  
end

but I really have no idea. Any help would be great - thanks in advance!

---

<div class="post-metadata">

**Author:** ![lynchnco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/c77e96/32.png) [@lynchnco](https://www.ruby-forum.com/u/lynchnco)\
**Post date:** [January 27, 2009, 9:39am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704/2 "2009-01-27T09:39:07Z")

</div>

On Tue, Jan 27, 2009 at 1:55 AM, Patrick L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Hey guys,  
> I’m trying to write an application that goes onto a website (istockphoto  
> specifically), opens up [istockphoto.com/file\_browse.php](http://istockphoto.com/file_browse.php) and grabs the  
> URLs of the photos that appear there.
> 
> It’s my first time doing something like this. I’m reading some  
> documentation right now…but a hand would be greatly appreciated. I’m  
> not really sure how to do regex on an html file…or even find the right  
> stuff within that file. I’m guessing its…

Generally speaking, regular expressions are not the best tool to extract  
information from HTML. Take a look at these other tools:

Mechanize  
Hpricot  
Scrubyt  
Nokogiri

This is an example that might get you started, although I recommend  
taking  
a look at the above tools:

require ‘open-uri’  
require ‘hpricot’

h = Hpricot(open(“[http://www.istockphoto.com/file\_browse.php](http://www.istockphoto.com/file_browse.php)”))  
imgs = h.search(“//[@class = searchImg]”)  
imgs.map {|img| img[“src”]}

# =\>

[“[http://www2.istockphoto.com/file\_thumbview\_approve/8137463/1/istockphoto\_8137463-budapest-by-night.jpg](http://www2.istockphoto.com/file_thumbview_approve/8137463/1/istockphoto_8137463-budapest-by-night.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139472/1/istockphoto\_8139472-four-antique-wood-tennis-racquets.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139472/1/istockphoto_8139472-four-antique-wood-tennis-racquets.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/6731990/1/istockphoto\_6731990-two-female-lovers.jpg](http://www2.istockphoto.com/file_thumbview_approve/6731990/1/istockphoto_6731990-two-female-lovers.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8308377/1/istockphoto\_8308377-beauty.jpg](http://www2.istockphoto.com/file_thumbview_approve/8308377/1/istockphoto_8308377-beauty.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/6349299/1/istockphoto\_6349299-lovers-interested-in-smth.jpg](http://www2.istockphoto.com/file_thumbview_approve/6349299/1/istockphoto_6349299-lovers-interested-in-smth.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322403/1/istockphoto\_8322403-happy-piggy-bank.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322403/1/istockphoto_8322403-happy-piggy-bank.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8138976/1/istockphoto\_8138976-tower-guard-of-cetara-little-town-in-amalfi-coast-italy.jpg](http://www2.istockphoto.com/file_thumbview_approve/8138976/1/istockphoto_8138976-tower-guard-of-cetara-little-town-in-amalfi-coast-italy.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322394/1/istockphoto\_8322394-yellow-red-paper.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322394/1/istockphoto_8322394-yellow-red-paper.jpg)”,  
“[http://www1.istockphoto.com/file\_thumbview\_approve/4660654/1/istockphoto\_4660654-the-art-of-eye-shadows.jpg](http://www1.istockphoto.com/file_thumbview_approve/4660654/1/istockphoto_4660654-the-art-of-eye-shadows.jpg)”,  
“[http://www1.istockphoto.com/file\_thumbview\_approve/8301075/1/istockphoto\_8301075-3d-render-of-the-olive-tree.jpg](http://www1.istockphoto.com/file_thumbview_approve/8301075/1/istockphoto_8301075-3d-render-of-the-olive-tree.jpg)”,  
“[http://www1.istockphoto.com/file\_thumbview\_approve/6921717/1/istockphoto\_6921717-manicure.jpg](http://www1.istockphoto.com/file_thumbview_approve/6921717/1/istockphoto_6921717-manicure.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322391/1/istockphoto\_8322391-pomegranate.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322391/1/istockphoto_8322391-pomegranate.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8138975/1/istockphoto\_8138975-junger-mann-seitlich.jpg](http://www2.istockphoto.com/file_thumbview_approve/8138975/1/istockphoto_8138975-junger-mann-seitlich.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139815/1/istockphoto\_8139815-winter.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139815/1/istockphoto_8139815-winter.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8137153/1/istockphoto\_8137153-beadworkafrican\_pictureframe\_p3406-jpg.jpg](http://www2.istockphoto.com/file_thumbview_approve/8137153/1/istockphoto_8137153-beadworkafrican_pictureframe_p3406-jpg.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139787/1/istockphoto\_8139787-statue-of-liberty.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139787/1/istockphoto_8139787-statue-of-liberty.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322388/1/istockphoto\_8322388-cold-winter-day.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322388/1/istockphoto_8322388-cold-winter-day.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139602/1/istockphoto\_8139602-statue-of-liberty.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139602/1/istockphoto_8139602-statue-of-liberty.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8137801/1/istockphoto\_8137801-litchi.jpg](http://www2.istockphoto.com/file_thumbview_approve/8137801/1/istockphoto_8137801-litchi.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139406/1/istockphoto\_8139406-statue-of-liberty.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139406/1/istockphoto_8139406-statue-of-liberty.jpg)”,  
“[http://www1.istockphoto.com/file\_thumbview\_approve/6850893/1/istockphoto\_6850893-polka-dot-wedding-cake.jpg](http://www1.istockphoto.com/file_thumbview_approve/6850893/1/istockphoto_6850893-polka-dot-wedding-cake.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139802/1/istockphoto\_8139802-snow-woman.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139802/1/istockphoto_8139802-snow-woman.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322364/1/istockphoto\_8322364-white-cherry-blossom.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322364/1/istockphoto_8322364-white-cherry-blossom.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139808/1/istockphoto\_8139808-airport.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139808/1/istockphoto_8139808-airport.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8322357/1/istockphoto\_8322357-ciruit.jpg](http://www2.istockphoto.com/file_thumbview_approve/8322357/1/istockphoto_8322357-ciruit.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8139597/1/istockphoto\_8139597-cheese-and-wine.jpg](http://www2.istockphoto.com/file_thumbview_approve/8139597/1/istockphoto_8139597-cheese-and-wine.jpg)”,  
“[http://www2.istockphoto.com/file\_thumbview\_approve/8138075/1/istockphoto\_8138075-employee-of-office.jpg](http://www2.istockphoto.com/file_thumbview_approve/8138075/1/istockphoto_8138075-employee-of-office.jpg)”]

You should customize the criteria to choose the images (in my little  
example I selected all tags which had a class searchImg, which at a  
quick glance seemed what you wanted, but double check).

I recall reading somewhere that nokogiri has better XPath support than  
Hpricot, so check it out.

Jesus.

---

<div class="post-metadata">

**Author:** ![lynchnco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/c77e96/32.png) [@lynchnco](https://www.ruby-forum.com/u/lynchnco)\
**Post date:** [January 28, 2009, 12:43am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704/3 "2009-01-28T00:43:22Z")

</div>

Miroslaw N. wrote:

> 2009/1/27 Patrick L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):
> 
> > open(‘[http://www.istockphoto.com/file\_browse.php/](http://www.istockphoto.com/file_browse.php/)’) do |f|  
> > f.find # dot something something  
> > end
> 
> Try Mechanize.  
> It’s easy :
> 
> agent = WWW::Mechanize.new  
> agent.user\_agent\_alias=‘Mac Safari’  
> page = agent.get(‘[http://www.istockphoto.com/file\_browse.php](http://www.istockphoto.com/file_browse.php)’);  
> page.links.text(/jpg/)  
> …

That’s great, or it sounds great. Is there any documentation aside from  
blog posts and this: [http://mechanize.rubyforge.org/mechanize/](http://mechanize.rubyforge.org/mechanize/) ? What  
did you use to learn it?

---

<div class="post-metadata">

**Author:** ![lynchnco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/c77e96/32.png) [@lynchnco](https://www.ruby-forum.com/u/lynchnco)\
**Post date:** [January 28, 2009, 12:47am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704/4 "2009-01-28T00:47:26Z")

</div>

mechanize is very easy and intuitive … you could basically learn to  
use mechanize just by playing with it in irb . Combine that with reading  
some/the docs , and you’re good to go .

---

<div class="post-metadata">

**Author:** ![lynchnco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/c77e96/32.png) [@lynchnco](https://www.ruby-forum.com/u/lynchnco)\
**Post date:** [January 27, 2009, 9:42am UTC](https://www.ruby-forum.com/t/page-crawling-and-url-grabbing/158704/5 "2009-01-27T09:42:06Z")

</div>

2009/1/27 Patrick L. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> open(‘[http://www.istockphoto.com/file\_browse.php/](http://www.istockphoto.com/file_browse.php/)’) do |f|  
> f.find # dot something something  
> end

Try Mechanize.  
It’s easy :

agent = WWW::Mechanize.new  
agent.user\_agent\_alias=‘Mac Safari’  
page = agent.get(‘[http://www.istockphoto.com/file\_browse.php](http://www.istockphoto.com/file_browse.php)’);  
page.links.text(/jpg/)  
…