# Hpricot won't scrape! (newb question)

**URL:** <https://www.ruby-forum.com/t/hpricot-wont-scrape-newb-question/164429>\
**Category:** Rails\
**Created:** [April 1, 2009, 10:29am UTC](https://www.ruby-forum.com/t/hpricot-wont-scrape-newb-question/164429 "2009-04-01T10:29:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jrgoodner](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/53a042/32.png) [@jrgoodner](https://www.ruby-forum.com/u/jrgoodner)\
**Post date:** [April 1, 2009, 10:29am UTC](https://www.ruby-forum.com/t/hpricot-wont-scrape-newb-question/164429/1 "2009-04-01T10:29:25Z")

</div>

Hey all! Just to preface, I am fairly new to RoR, and brand new to  
using hpricot.

I am using the following code to scrape this xpath:  
“/html/body/div/div[5]/div/div[2]/div[2]/div[2]”

from this url:  
“[http://www.greatnonprofits.org/](http://www.greatnonprofits.org/)”

Here is my code to do so (taken from [igvita.com](http://igvita.com)’s related blogpost):

* * *

require ‘rubygems’

require ‘open-uri’

require ‘hpricot’

@url = “[http://www.greatnonprofits.org/](http://www.greatnonprofits.org/)”

@response = ‘’

begin

# open-uri RDoc:

[http://stdlib.rubyonrails.org/libdoc/open-uri/rdoc/index.html](http://stdlib.rubyonrails.org/libdoc/open-uri/rdoc/index.html)

open(@url, “User-Agent” =\> “Ruby/#{RUBY\_VERSION}”,  
“From” =\> “removed\_email\_address@domain.invalid”,  
“Referer” =\> “[Ilya Grigorik](http://www.igvita.com/blog/)”) { |f|

```
puts "Fetched document: #{f.base_uri}"

puts "\t Content Type: #{f.content_type}\n"

puts "\t Charset: #{f.charset}\n"

puts "\t Content-Encoding: #{f.content_encoding}\n"

puts "\t Last Modified: #{f.last_modified}\n\n"

# Save the response body

@response = f.read

```

}

# HPricot RDoc: [http://code.whytheluckystiff.net/hpricot/](http://code.whytheluckystiff.net/hpricot/)

doc = Hpricot(@response)

# Retrieve content

puts (doc/“/html/body/div/div[5]/div/div[2]/div[2]/div[2]”).to\_html  
()

rescue Exception =\> e

print e, “\n”

end

* * *

In my irb terminal, I get this:

* * *

irb(main):031:0\> load ‘greatnonprofitsscraper.rb’  
Fetched document: [http://www.greatnonprofits.org/](http://www.greatnonprofits.org/)  
Content Type: text/html  
Charset: utf-8  
Content-Encoding:  
Last Modified: Tue Mar 31 23:43:52 -0700 2009

=\> true

* * *

Anyone know why this is happening? The code works with other urls/  
xpaths. Can anyone specify for me why [www.greatnonprofits.com](http://www.greatnonprofits.com) is  
different?

Thanks a million! I am quite frustrated, and I appreciate any help!!!

---

<div class="post-metadata">

**Author:** ![jrgoodner](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/53a042/32.png) [@jrgoodner](https://www.ruby-forum.com/u/jrgoodner)\
**Post date:** [April 1, 2009, 10:51am UTC](https://www.ruby-forum.com/t/hpricot-wont-scrape-newb-question/164429/2 "2009-04-01T10:51:44Z")

</div>

On Apr 1, 7:50 am, jrgoodner [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Anyone know why this is happening? The code works with other urls/  
> xpaths. Can anyone specify for me whywww.greatnonprofits.comis  
> different?

Well, just asking the stupid question: are you sure the html on that  
page matches the structure in that xpath ?

Fred

---

<div class="post-metadata">

**Author:** ![jrgoodner](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/53a042/32.png) [@jrgoodner](https://www.ruby-forum.com/u/jrgoodner)\
**Post date:** [April 1, 2009, 2:50pm UTC](https://www.ruby-forum.com/t/hpricot-wont-scrape-newb-question/164429/3 "2009-04-01T14:50:50Z")

</div>

jrgoodner wrote:

> Hey all! Just to preface, I am fairly new to RoR, and brand new to  
> using hpricot.
> 
> I am using the following code to scrape this xpath:  
> “/html/body/div/div[5]/div/div[2]/div[2]/div[2]”
> 
> from this url:  
> “[http://www.greatnonprofits.org/](http://www.greatnonprofits.org/)”

Switch to Nokogiri.