# Hrpicot - cant extract what i want from page

**URL:** <https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142>\
**Category:** Ruby\
**Created:** [March 28, 2008, 8:11am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142 "2008-03-28T08:11:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 8:11am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/1 "2008-03-28T08:11:16Z")

</div>

Hi im starting to use hrpicot and im having problems extracting  
descriptions of various concert events from a page. Here is a sample of  
the html

Concerts   
 POPULAR   
   
 Middle Field! Vol.4   
Featuring electric-pop band The Stealth, Mac and Masaru, and others. Mar 28, 7pm, ¥2,500 (adv)/ ¥3,000 (door). Shibuya O-Nest. Tel: 03-3498-9999.  
   
 Philip Woo featuring Brenda Vaughn   
Japanese pianist and soul singer performing with Andy Wulf and Kaori Kobayashi. Mar 28 & 29, 7 & 9:30pm, ¥3,150. Cotton Club, Marunouchi. Tel: 03-3215-1555.  
... ... ... etc

I can get the artist band names fine using  
names = doc.search("//span[@class=‘textbold’]")

 

but i cant get teh descriptions. In fact the descriptions aren’t  
indvidually wrapped up in any tags but rather just clumped together  
under the paragraph tab with line breaks

 

So I thought id just try  
descriptions =  
doc.search("/html/body/div/table/tbody/tr[4]/td/table/tbody/tr/td[2]/table/tbody/tr/td/span/p")  
but when i try to puts descriptions nothing is printed to the screen.

 

How would i go about getting this info??? any tips or ideas?

 

Thanks

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 10:10am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/2 "2008-03-28T10:10:41Z")

</div>

more info…

the original website can be found at  
[http://metropolis.co.jp/tokyo/recent/listings.asp](http://metropolis.co.jp/tokyo/recent/listings.asp)

i used firebug to retrieve the xpath address of the desired paragraph  
(excerpted above). When I put it in doc.search it doesnt retrieve  
anything, nothing at all???

Does anyone know why i cant??? Im banging my head against the wall

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 10:35am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/3 "2008-03-28T10:35:32Z")

</div>

On Fri, Mar 28, 2008 at 2:11 AM, Adam A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Kobayashi. Mar 28 & 29, 7 & 9:30pm, ¥3,150. Cotton Club, Marunouchi. but i cant get teh descriptions. In fact the descriptions aren't Thanks

Wow! It looks nice, but the html is really ugly. This would be  
pretty hard to scrape on a regular basis. For artists, there are a  
mix of tags, tags,  
and I noticed one artist with no surrounding tags at all (Ex-press  
Ver.2).

It can be really hard to work with inconsistent html, but I suppose it  
could be done to some degree of accuracy. Any hpricot masters out  
there? I’m sure you’d have to attack with regexps as well. Maybe  
turning into text and then parsing is a better idea after all.

Todd

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 11:05am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/4 "2008-03-28T11:05:33Z")

</div>

thanks tod for the reply. Yes even I thought that it was badly designed  
and I dont have any web desing experience at all. In fact i learn the  
basics of html, xml and xpath just for this.

Although those inconsitencies will prove to be a problem in the future  
the one im having right now is getting any information at all. Surely  
when i pass the xpath address for the paragraph element which contains  
all the artists names and event descriptinos it should return something  
rather than nothing. Is that right??? Every time a try to print to  
screen the result of the search it just comes blank. Does anyone know  
why???

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 11:42am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/5 "2008-03-28T11:42:43Z")

</div>

Firebug puts in tbody’s into xpath’s that reach into tables even if the

 tag is not in the html source. Try removing the tbody path and debug using shorter xpaths to initially address content further up in the hierarchy.

You might have some success addressing text nodes combined with some  
subsequent regexp processing:

b = doc.search("//text()")

I think you might be more successful using a css selector instead of an  
xpath selector. To overcome hpricot not supporting all xpath axes you  
can sometimes find a way to address the elements with a clever css  
selector.

It can be a challenge to use hpricot with malformed html or if there are  
no containers wrapping items that otherwise appear visually as a list or  
table. I haven’t tried it yet, but running the html through something  
like tidy before parsing so might create some of the missing structure.

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 5:46pm UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/6 "2008-03-28T17:46:34Z")

</div>

ok i have tried taking out the tbody tags completely and got some of the  
text back. Ill experiment to see if i can get all of it.

Re: Tidy

I installed the gem and i got the example code

require ‘tidy’  
Tidy.path = ‘/usr/lib/libtidy.so’  
html = ‘titleBody’  
xml = Tidy.open(:show\_warnings=\>true) do |tidy|  
tidy.options.output\_xml = true  
puts tidy.options.show\_warnings  
xml = tidy.clean(html)  
puts tidy.errors  
puts tidy.diagnostics  
xml  
end  
puts xml

now i have to change the path to whereever the lib is…well i foudn  
tidys folder in my lib directory and changed the above to this

Tidy.path = ‘C:\ruby\lib\ruby\gems\1.8\gems\tidy-1.1.2\lib\tidy\tidylib’

and its complaining saying no such file… i tried

Tidy.path =  
‘C:\ruby\lib\ruby\gems\1.8\gems\tidy-1.1.2\lib\tidy\tidylib.rb’

as thats the proper extension of the tidylib file but again it wont  
work.

I cant find any tidylib file with an extenision .so

banging my head even more now 😉

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 5:54pm UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/7 "2008-03-28T17:54:46Z")

</div>

just downloaded a dll which i needed. Why doesnt that come with the  
\*\*\*\*\*\*\* gem.

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 28, 2008, 12:10pm UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/8 "2008-03-28T12:10:47Z")

</div>

On Fri, Mar 28, 2008 at 11:42 AM, Dan D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Firebug puts in tbody’s into xpath’s that reach into tables even if the tag is not in the html source. Try removing the tbody path and debug using shorter xpaths to initially address content further up in the hierarchy.

Yes, Firefox does it to make it more (X)HTML-conform. It took me a  
while to get the hang of it. You might download the page using  
open-uri and open it with your favourite editor, search the text and  
work your way up through the tags.  
Most sites don’t use , so just try it without it.

---

<div class="post-metadata">

**Author:** ![scudco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [March 31, 2008, 2:27am UTC](https://www.ruby-forum.com/t/hrpicot-cant-extract-what-i-want-from-page/133142/9 "2008-03-31T02:27:08Z")

</div>

On Mar 28, 6:11 pm, Adam A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Tel: 03-3215-1555. indvidually wrapped up in any tags but rather just clumped together -- Posted viahttp://www.ruby-forum.com/.

Once you have the ‘name’ node you can use next\_node to get the next  
elements in the document  
This method should work for your example:

def print\_names\_and\_descriptions(hpricot\_doc)  
names = hpricot\_doc.search(“//span[@class=‘textbold’]”)

names.each do |name|  
node = name.next\_node  
node = node.next\_node until node.text? and node.inner\_text =~ /\w  
+/

```
puts name.inner_text
puts node.to_s.strip
puts

```

end  
end