# Newbie to Ruby

**URL:** <https://www.ruby-forum.com/t/newbie-to-ruby/113065>\
**Category:** Ruby\
**Created:** [August 27, 2007, 5:29am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065 "2007-08-27T05:29:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 27, 2007, 5:29am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/1 "2007-08-27T05:29:03Z")

</div>

Hi,  
I was reading through Learning R. and was trying to get the example on  
page 119 to work to scrape Google. But when I run it nothing happens.  
Any help would be appreciated. Thanks.

require ‘open-uri’

url = “[ruby - Google Search](http://www.google.com/search?q=ruby)”

open(url) { |page| page\_content = page.read()

links = page\_content.scan(/\<a class=1._?href="(._?)"/).flatten

links.each {|link| puts link}

}

---

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 27, 2007, 6:01am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/2 "2007-08-27T06:01:54Z")

</div>

Hi Charles,

Charles P. wrote:

> links = page\_content.scan(/\<a class=1._?href="(._?)"/).flatten

I don’t think that regular expression (regexp) works. Maybe google has  
changed their code since the book was written. I think it goes “href”  
then “class”.

If you work out the correct regexp, let us know.

Cheers,  
Mark

---

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 27, 2007, 6:27am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/3 "2007-08-27T06:27:04Z")

</div>

Mark G. wrote:

> Cheers,  
> Mark

As Mark said, google changed their code somewhat. If you work out the  
correct regular expression and it still seems to give erratic results,  
here is a hint: the naive solution uses "._?" in a certain place, but  
that will still match too many results. Try [^"]_? instead, because you  
probably don’t want to match quotes. (I just tried this, and that was  
the problem I encountered.)

By the way, a robust regex to match all HTML links looks kind of nasty,  
but perhaps you should try writing one–it’s a good exercise. (Of  
course, that’s not what you want for this–you want to match all links  
of class=l.)

Regards,  
Dan

---

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 27, 2007, 7:14am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/4 "2007-08-27T07:14:22Z")

</div>

As those guys said, Google probably changed their code since the book  
was written.  
That’s not to prevent web scraping, it’s just that web sites are  
pretty transitory. They change all the time and very easily. This  
makes web sophisticated web scraping a moving target.

---

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 27, 2007, 8:24pm UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/5 "2007-08-27T20:24:08Z")

</div>

2007/8/27, John J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> That’s not to prevent web scraping, it’s just that web sites are  
> pretty transitory. They change all the time and very easily. This  
> makes web sophisticated web scraping a moving target.

Yes, web scraping using just open-uri and regular expressions is  
pretty low-level.

Try Hpricot or scRUBYt for a higher level, more flexible, scraping.¡

---

<div class="post-metadata">

**Author:** ![animate123](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/e19b73/32.png) [@animate123](https://www.ruby-forum.com/u/animate123)\
**Post date:** [August 30, 2007, 12:15am UTC](https://www.ruby-forum.com/t/newbie-to-ruby/113065/6 "2007-08-30T00:15:06Z")

</div>

Jaime I. wrote:

> 2007/8/27, John J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):
> 
> > That’s not to prevent web scraping, it’s just that web sites are  
> > pretty transitory. They change all the time and very easily. This  
> > makes web sophisticated web scraping a moving target.
> 
> Yes, web scraping using just open-uri and regular expressions is  
> pretty low-level.
> 
> Try Hpricot or scRUBYt for a higher level, more flexible, scraping.�

So I was trying out what everyone said and I got it to work. Here is  
what I did.

page\_content.scan(/\<a href="([^"]_?)" class=l[^"]_?/).flatten