# Getting all google results with hpricot and connecting two g

**URL:** <https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280>\
**Category:** Ruby\
**Created:** [August 29, 2007, 2:46pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280 "2007-08-29T14:46:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 29, 2007, 2:46pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/1 "2007-08-29T14:46:32Z")

</div>

Hi,  
I’m trying to fetch all google results with hpricot. For the first page  
of results I wrote this here:

#!/usr/bin/env ruby  
$Verbose=true

require ‘hpricot’  
require ‘open-uri’

google =  
Hpricot(open(“[Google](http://www.google.com/search?name=f&hl=en&q=#%7B$*%7D)”))  
(google/“h2.r/a”).each {|line| puts  
line.to\_s.gsub(/^.+href=“/,‘’).gsub(/” .+$/,‘’)}

So my first question is can I connect the both gsub statments above in  
just one gsub which should increase the speed? Or is there even a better  
way than using gsub for cleaning the results?

And the next question is: how can I get all results not just from the  
first page?

greets

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 29, 2007, 4:12pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/2 "2007-08-29T16:12:50Z")

</div>

On Wed, Aug 29, 2007 at 09:45:04PM +0900, kazaam wrote:

> (google/“h2.r/a”).each {|line| puts line.to\_s.gsub(/^.+href=“/,‘’).gsub(/” .+$/,‘’)}
> 
> So my first question is can I connect the both gsub statments above in  
> just one gsub which should increase the speed? Or is there even a better  
> way than using gsub for cleaning the results?
> 
> And the next question is: how can I get all results not just from the  
> first page?

Look into mechanize or scrubyt for this. They sit on top of hpricot, but  
are much better suited to screen scraping applications than hpricot  
alone.

> greets  
> kazaam [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> –Greg

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 29, 2007, 9:34pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/3 "2007-08-29T21:34:50Z")

</div>

Gregory S. wrote:

> > google = Hpricot(open(“[Google](http://www.google.com/search?name=f&hl=en&q=#%7B$*%7D)”))  
> > Look into mechanize or scrubyt for this. They sit on top of hpricot, but  
> > are much better suited to screen scraping applications than hpricot alone.
> 
> > greets  
> > kazaam [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> > –Greg

I always thought the Google Terms of Service (around 5.3) suggested that  
you  
shouldn’t be running google searches in scripts?

TerryP.

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 29, 2007, 11:44pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/4 "2007-08-29T23:44:36Z")

</div>

Hi Kazaam,

On 29 Aug 2007, at 13:45, kazaam wrote:

> So my first question is can I connect the both gsub statments above  
> in just one gsub which should increase the speed? Or is there even  
> a better way than using gsub for cleaning the results?

why covered this a while ago in one of his blog posts: http://  
[redhanded.hobix.com/inspect/nostrils.html](http://redhanded.hobix.com/inspect/nostrils.html)

The non-graphic pastie version can be seen here: http://  
pastie.caboo.se/54741

And just for completeness, I wrapped it up in a rails  
controller:[http://douglasfshearer.com/blog/site-search-using-google-](http://douglasfshearer.com/blog/site-search-using-google-)  
in-ruby-on-rails

Cheers.

Douglas F Shearer  
removed\_email\_address@domain.invalid

> **[Douglas F Shearer](https://douglasfshearer.com/)**
>
> Software developer, mountainbiker.

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 30, 2007, 4:41pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/5 "2007-08-30T16:41:40Z")

</div>

On Wed, 29 Aug 2007 16:10:06 +0200, Gregory S.  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > google =  
> > Hpricot(open(“[Google](http://www.google.com/search?name=f&hl=en&q=#%7B$*%7D)”))  
> > (google/“h2.r/a”).each {|line| puts  
> > line.to\_s.gsub(/^.+href=“/,‘’).gsub(/” .+$/,‘’)}

This doesn’t work?

(google/“h2.r/a”).each {|line| puts line[‘href’]}

> > So my first question is can I connect the both gsub statments above in  
> > just one gsub which should increase the speed? Or is there even a better  
> > way than using gsub for cleaning the results?
> > 
> > And the next question is: how can I get all results not just from the  
> > first page?

Can’t you teach Google to show more results on a page?  
If not, extract the “next page” link and fetch that page. Proceed as  
long  
as long want 🙂

> Look into mechanize or scrubyt for this. They sit on top of hpricot, but  
> are much better suited to screen scraping applications than hpricot  
> alone.

Nope. For this purpose, Hpricot is the best you can use!!! And it is  
really great!  
Mechanize doesn’t give you much advantage if you just want to parse  
pages  
(and I say that as the author who wrong the initial version of Mechanize  
:).

scrubyt never worked on my machine, but I also find it too complicated  
or  
maybe I am just too stupid :).

Regards,

Michael

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 30, 2007, 10:24pm UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/6 "2007-08-30T22:24:50Z")

</div>

From the Google Terms of Service which as far as I know cover all Google  
Products inloo of a seperate service agreement.

quote:

1. Use of the Services by you

:un quote

Doesn’t that apply to this?? 😕

Cheers.

---

<div class="post-metadata">

**Author:** ![kazaam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/b4bc9f/32.png) [@kazaam](https://www.ruby-forum.com/u/kazaam)\
**Post date:** [August 30, 2007, 12:27am UTC](https://www.ruby-forum.com/t/getting-all-google-results-with-hpricot-and-connecting-two-g/113280/7 "2007-08-30T00:27:47Z")

</div>

Douglas F Shearer wrote:

> why covered this a while ago in one of his blog posts:  
> [http://redhanded.hobix.com/inspect/nostrils.html](http://redhanded.hobix.com/inspect/nostrils.html)
> 
> The non-graphic pastie version can be seen here:  
> [Parked at Loopia](http://pastie.caboo.se/54741)
> 
> And just for completeness, I wrapped it up in a rails  
> controller:[Site Search Using Google In Ruby On Rails](http://douglasfshearer.com/blog/site-search-using-google-in-ruby-on-rails)

I would not suggest doing that as google doesn’t allow bot requests and  
your IP will simply get blacklisted. For site search I recommend google  
coop: [Programmable Search Engine by Google](http://www.google.com/coop/)

Daniel