# Hpricot help - parsing malformed HTML **URL:** \ **Category:** Rails\ **Created:** [November 17, 2006, 1:06am UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258 "2006-11-17T01:06:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 1:06am UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/1 "2006-11-17T01:06:46Z")
I am having a complicated issue here. I am trying to fetch a page from Froogle and parse it via Hpricot to collect data from the products in the search results. sample page: [http://froogle.google.com/froogle?q=magnets&btnG=Search](http://froogle.google.com/froogle?q=magnets&btnG=Search) The problem is that the HTML on Froogle is seriously broken. I need to get the table row (tr) for each product, and then look in each of that rows td’s for data. But google’s html is full of unclosed tags for their tables that makes Hpricot freak out. Hpricot thinks the tr’s are empty: “\\n” So I guess the question is how do I make Hpricot cope with this markup? It obviously works great in the browser. Are there any tools that will convert a string of html to a valid XML or DOM equivalent? It must be possible because web browsers handle it all the time. What I need to be able to do: html = open(‘[http://foo.com/](http://foo.com/)’).read html = html.clean\_markup html = Hpricot(html) * * * Here is an oversimplified example of froogle’s of malformed markup: | foo | bar | | baz | boo | ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 1:12am UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/2 "2006-11-17T01:12:06Z")
Alex W. wrote: > I am having a complicated issue here. I am trying to fetch a page from > Froogle and parse it via Hpricot to collect data from the products in > the search results. > > sample page: [http://froogle.google.com/froogle?q=magnets&btnG=Search](http://froogle.google.com/froogle?q=magnets&btnG=Search) > > The problem is that the HTML on Froogle is seriously broken. I need to > get the table row (tr) for each product, and then look in each of that > rows td’s for data. But google’s html is full of unclosed tags for > their tables that makes Hpricot freak out. Hpricot thinks the tr’s are > empty: > > “\\n” Heres a better illustration of the problem, from irb: pp Hpricot(‘ | foo | bar | ’) # =\> #\ the is empty, and the 's are considered direct children of . So the selector "table tr td" wont work. There is no way to groud td's by row in this case. ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 9:39am UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/3 "2006-11-17T09:39:26Z")
On 17 Nov 2006, at 00:06, Alex W. wrote: > The problem is that the HTML on Froogle is seriously broken. Agreed! > html = html.clean\_markup > html = Hpricot(html) I had a similar problem last week and ended up doing exactly what you are proposing, i.e. a pre-processing step to clean up the HTML before feeding it to Hpricot. > Here is an oversimplified example of froogle’s of malformed markup: > > | foo | bar | > | baz | boo | I believe there are Ruby libraries for cleaning up HTML though I’m not familiar with them. Perhaps you could just treat it as a long string and walk over it doing the following: 1. Scan forward until you find a tag (either opening or closing). 2. If the tag is a known potentially-broken one (’’, ‘’, ‘ ’, etc) set a flag for that tag to indicate it is open (or push it onto a per-tag stack somewhere). Clear the flag (or pop the stack) if/when you see the matching closing tag. 3. When you see that tag again, if it hasn’t been closed in the meantime, insert the closing tag yourself and clear your flag (pop your stack). I think it will be easier to do than it sounds 😉 Hope that helps, Andy ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 4:10pm UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/4 "2006-11-17T16:10:21Z")
Take a look at scrapi - if not to actually use then to steal Assaf’s ideas. =) I THINK he has some sort of way to pre-process HTML with Tidy in there; might want to crib those ideas. On 11/16/06, Alex W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > their tables that makes Hpricot freak out. Hpricot thinks the tr’s are > > foo – I think it is inevitable that people program poorly. Training will not substantially help matters. We have to learn to live with it. – Alan Perlis ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 3:56pm UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/5 "2006-11-17T15:56:19Z")
Andrew S. wrote: > On 17 Nov 2006, at 00:06, Alex W. wrote: > > > The problem is that the HTML on Froogle is seriously broken. > > Agreed! Disagree! The example given is not malformed. It’s perfectly acceptable HTML 4.01. The end tags for and can be omitted. Unless the DTD declaration claims it to be something newer than HTML 4.01, it is fine. I would say this is a bug in Hpricot. - Mark. ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 4:10pm UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/6 "2006-11-17T16:10:22Z")
On Nov 17, 2006, at 9:51 AM, Thomas, Mark - BLS CTR wrote: > The end tags for and can be omitted. > > Unless the DTD declaration claims it to be something newer than HTML > 4.01, it is fine. > > I would say this is a bug in Hpricot. > > - Mark. You can use RubyfulSoup to deal with HTML even when it isn’t completely correct. It is packaged as a gem, but I unpacked it into the plugin directory and it’s working for me. (Hpricot didn’t exist at the time or I might have tried it.) #Rubyful Soup #Elixir and Tonic #“The Screen-Scraper’s Friend” #v1.0.4 #[http://www.crummy.com/software/RubyfulSoup/](http://www.crummy.com/software/RubyfulSoup/) # #Rubyful Soup is a port to the Ruby language and idiom of the Python #library Beautiful Soup. #See [Beautiful Soup: We called him Tortoise because he taught us.](http://www.crummy.com/software/BeautifulSoup/) for details on the original. -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 4:42pm UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/7 "2006-11-17T16:42:32Z")
On 11/17/06, Michael C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Take a look at scrapi - if not to actually use then to steal Assaf’s > ideas. =) I THINK he has some sort of way to pre-process HTML with > Tidy in there; might want to crib those ideas. We also use tidy for cleaning up invalid xhtml with MasterView project. You can get the ruby tidy wrapper here [http://rubyforge.org/projects/tidy](http://rubyforge.org/projects/tidy) [http://tidy.rubyforge.org/](http://tidy.rubyforge.org/) (for usage info) Note that it also requires that the tidy library available on the server as well. It is available for both windows and \*nix. It works well at cleaning up invalid xhtml and the ruby tidy wrapper is simple to use. The only disadvantage is that you need to have the lib available and you need to set the path to the lib so that it can load it. I wish that could be automated some how, because it is a manual setup step. Jeff ---
**Author:** ![squeegy](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/dec6dc/32.png) [@squeegy](https://www.ruby-forum.com/u/squeegy)\ **Post date:** [November 17, 2006, 6:21pm UTC](https://www.ruby-forum.com/t/hpricot-help-parsing-malformed-html/76258/8 "2006-11-17T18:21:49Z")
Jeff B. wrote: > On 11/17/06, Michael C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > Take a look at scrapi - if not to actually use then to steal Assaf’s > > ideas. =) I THINK he has some sort of way to pre-process HTML with > > Tidy in there; might want to crib those ideas. > > We also use tidy for cleaning up invalid xhtml with MasterView project. > > You can get the ruby tidy wrapper here > [http://rubyforge.org/projects/tidy](http://rubyforge.org/projects/tidy) > [http://tidy.rubyforge.org/](http://tidy.rubyforge.org/) (for usage info) > > Jeff I seem to be having some luck with tidy and cleaning it before I send it to Hpricot. This little code snippet seems to handle keeping the Tidy.path assigned. I just have to include the linux and windows tody libs in my /lib directory. require ‘tidy’ if RUBY\_PLATFORM =~ /mswin/ Tidy.path = “#{RAILS\_ROOT}/lib/tidy.dll” else Tidy.path = “#{RAILS\_ROOT}/lib/tidy” end Thanks for the tip!