Forum: Ruby Help with HTML parsing

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Vivek N. (Guest)
on 2009-01-01 23:43
Hello,

I'm new to Watir\Ruby and need to resolve something that involves HTML
parsing - you could also call it screen scraping. I haven't used either
library before, but I wanted to know if it is better to use Hpricot or
open_uri. The problem is similar to below:

let's say I'm searching Google for some string, "Dungeons & Dragons" for
instance. I want to parse through the first results page and get the
title text and url for the top 5 results. How would I do this using
Hpricot or open_uri or both?

Please help!


Viv.
Dan D. (Guest)
on 2009-01-02 00:38
(Received via mailing list)
open-uri gives you access to http/https/ftp while Hpricot is a html
parser. So for your intended application you would potentially use both
- open-uri would fetch the html document and whereas Hpricot would do
the parsing using xpath/css selectors etc. open-uri is used when you are
loading your html from the web url. The open-uri part happens in the
blink of an eye and then you are off manipulating your hpricot doc:
require 'open-uri'
doc = open("http://qwantz.com/") { |f| Hpricot(f) }
Vivek N. (Guest)
on 2009-01-03 00:27
ok, i've started to work with hpricot. i'm debating if i should use
open-uri or watir. but thats another discussion. my current issue is,
though, with xpath. this is what i have so far...

#!ruby
require 'watir'
require 'open-uri'
require 'rubygems'
require 'hpricot'


Watir.options_file = 'c:/ruby/options.yml'
Watir::Browser.default = 'ie'

test_site = "http://www.google.com"

br = Watir::Browser.new

br.goto test_site
br.text_field(:name, 'q').set("Dungeons & Dragons")
br.button(:name, 'btnG').click

doc = Hpricot(br.html)

#after the above, i'm trying to store all result elements in an array

x = Array.new
x << doc.search("//li[@class='g w0']/h3/a")

the XPath checker in firefox shows that I have the right path. but the
command doesn't work in irb. how do i drill down to specific elements so
that i can extract the title text and url? and wats wrong with my
xpath??

plz help!


Viv.
Phlip (Guest)
on 2009-01-03 01:21
(Received via mailing list)
Vivek Netha wrote:

> x = Array.new
> x << doc.search("//li[@class='g w0']/h3/a")
>
> the XPath checker in firefox shows that I have the right path. but the
> command doesn't work in irb. how do i drill down to specific elements so
> that i can extract the title text and url? and wats wrong with my
> xpath??

Don't use Hpricot - its XPath support only covers a very few predicate
and path
types. Use libxml-ruby, if you can install it, or REXML, if you don't
mind the
slow speed (the RE stands for Regular Expressions!), or nokogiri, which
I don't
know how to recommend yet, but I will soon.

BTW Google for my street-name and XPath for all kinds of fun in Ruby
with them.
Vivek N. (Guest)
on 2009-01-08 00:29
Hi Philip,

could you give me more pointers on how exactly you would do it using
REXML. with actual code, if possible.

thx.

Phlip wrote:
> Vivek Netha wrote:
>
>> x = Array.new
>> x << doc.search("//li[@class='g w0']/h3/a")
>>
>> the XPath checker in firefox shows that I have the right path. but the
>> command doesn't work in irb. how do i drill down to specific elements so
>> that i can extract the title text and url? and wats wrong with my
>> xpath??
>
> Don't use Hpricot - its XPath support only covers a very few predicate
> and path
> types. Use libxml-ruby, if you can install it, or REXML, if you don't
> mind the
> slow speed (the RE stands for Regular Expressions!), or nokogiri, which
> I don't
> know how to recommend yet, but I will soon.
>
> BTW Google for my street-name and XPath for all kinds of fun in Ruby
> with them.
Phlip (Guest)
on 2009-01-08 06:50
(Received via mailing list)
Vivek Netha wrote:

> could you give me more pointers on how exactly you would do it using
> REXML. with actual code, if possible.

>>> x = Array.new
>>> x << doc.search("//li[@class='g w0']/h3/a")

Sure, but this is just tutorial-level REXML (hint hint), and uncompiled:

   require 'rexml/document'
   doc = REXML::Document.new(my_xml_string)
   x << REXML::XPath.first(doc, '//li[ @class = "g w0" ]/h3/a').text

Now we come to the very nub of the gist. A @class of "g w0" is the same
as "w0
g", or any other permutation, yet XPath is literal-minded, and unaware
of CSS,
so it will only match one of those permutations, when any other could
have been
just as significant.

You could try contains(@class, "w0"), but that would match "w0p0p",
which is a
different class.

This is why nokogiri is interesting - it might do CSS Selector notation,
which
is less exact than XPath, and more aware of CSS rules. (Look up
assert_select()
to see what I mean.)

For further REXML abuse, try this...

   http://www.google.com/codesearch?q=REXML%3A%3AXPath.first

...but be warned (again, I think), it's _almost_ as slow as Windows
Vista with
more than one program running, so /caveat emptor/.
Tom M. (Guest)
on 2009-01-10 07:00
(Received via mailing list)
On 2009-01-08, Phlip <removed_email_address@domain.invalid> wrote:
> Vivek Netha wrote:
>>>> x << doc.search("//li[@class='g w0']/h3/a")
>
> You could try contains(@class, "w0"), but that would match "w0p0p", which is a
> different class.
>

XPath that solves this:
contains(concat(' ', @class, ' '), ' w0 ')

You can join multiple classes together like this:
contains(concat(' ', @class, ' '), ' w0 ') && contains(concat(' ',
@class, ' '), ' g ')

It's not pretty. I've written it too much in my life.
I do wish XPath had a class selector.
Phlip (Guest)
on 2009-01-10 11:50
(Received via mailing list)
Tom M. wrote:

> XPath that solves this:
> contains(concat(' ', @class, ' '), ' w0 ')
>
> You can join multiple classes together like this:
> contains(concat(' ', @class, ' '), ' w0 ') && contains(concat(' ',
> @class, ' '), ' g ')
>
> It's not pretty. I've written it too much in my life.
> I do wish XPath had a class selector.

Noted. I'm writing an XPath DSL, above the level of a raw XML library,
and I
just added to its pending feature list these line-items:

#  TODO  :class => :symbol should do the trick contains(concat(' ',
@class, '
'), ' w0 ')
#  TODO  :class => [] should do the trick contains(concat(' ', @class, '
'), '
w0 ') && contains(concat(' ',@class, ' '), ' g ')
#  TODO  :class => a string should be raw.

Those won't fix low-level XPath, but they will be useful when my DSL
targets
XHTML...
Aaron P. (Guest)
on 2009-01-10 20:23
(Received via mailing list)
On Sat, Jan 10, 2009 at 06:49:32PM +0900, Phlip wrote:
>> I do wish XPath had a class selector.
> Those won't fix low-level XPath, but they will be useful when my DSL
> targets XHTML...

Couldn't you use CSS selectors when targeting XHTML (or XML even)?  In
Nokogiri, I have a CSS to XPath converter and just push the XPath down
to libxml.  For example:

  require 'nokogiri'

  doc = Nokogiri::HTML(<<-eoxml)
  <html>
    <body>
      <div class="one two">
        Hello
      </div>
    </body>
  </html>
  eoxml

  puts doc.at('.one.two').content

It also doesn't care if you're using XML or HTML.  The CSS selectors
will get
converted to XPath either way.
Phlip (Guest)
on 2009-01-10 21:11
(Received via mailing list)
> Couldn't you use CSS selectors when targeting XHTML (or XML even)?

>   puts doc.at('.one.two').content

I'm going to. As soon as I get off my lazy butt, I will make nokogiri
into a
pluggable back-end to the DSL. Then the CSS selection notation is free.

But I can't make the DSL itself into a CSS hybrid because that would
disadvantage the other pluggable XML parsers. So if you want pure XML
compliance, use REXML, if you want some XHTML awareness use LibXML, and
if you
want CSS notation use Nokogiri. (After I'm done!)
Aaron P. (Guest)
on 2009-01-10 22:49
(Received via mailing list)
On Sun, Jan 11, 2009 at 04:09:54AM +0900, Phlip wrote:
> compliance, use REXML, if you want some XHTML awareness use LibXML, and
> if you want CSS notation use Nokogiri. (After I'm done!)

The CSS to XPath conversion can stand alone as a pure ruby library.
Feel free to take it from my source if you like.  Then you can give CSS
notation to anything that supports XPath. :-)
Jun Y. Kim (Guest)
on 2009-01-12 02:18
(Received via mailing list)
you can also use ruby library Sanitize (http://wonko.com/post/sanitize)

This library can make you parse html template very easily.

let's see the following examples.

Using Sanitize is easy. First, install it:
  sudo gem install sanitize

Then call it like so:

  require 'rubygems'
  require 'sanitize'

  html = '<b><a href="http://foo.com/">foo</a></b><img
src="http://foo.com/bar.jpg
" />'

  Sanitize.clean(html) # => 'foo'

By default, Sanitize removes all HTML. You can use one of the built-in
configs to tell Sanitize to allow certain attributes and elements:

  Sanitize.clean(html, Sanitize::Config::RESTRICTED)
  # => '<b>foo</b>'

  Sanitize.clean(html, Sanitize::Config::BASIC)
  # => '<b><a href="http://foo.com/" rel="nofollow">foo</a></b>'

  Sanitize.clean(html, Sanitize::Config::RELAXED)
  # => '<b><a href="http://foo.com/">foo</a></b><img
src="http://foo.com/bar.jpg
" />'

Or, if you¡¯d like more control over what¡¯s allowed, you can provide
your own custom configuration:

  Sanitize.clean(html, :elements => ['a', 'span'],
      :attributes => {'a' => ['href', 'title'], 'span' => ['class']},
      :protocols => {'a' => {'href' => ['http', 'https', 'mailto']}})

good one :)

2009. 01. 02, ¿ÀÀü 6:42, Vivek Netha ÀÛ¼º:
Santosh T. (Guest)
on 2009-11-05 06:29
Hi,

  I am using Sanitize.clean(), for freeing contents from html tags, but
the difficulty is I want to preserve some of the tags from removing.. I
have given like this.

html = File.new(file).read
      soup = BeautifulSoup.new(html)
      soup.title.contents=['']
      soup.find_all.each do |tag|
        if tag.string!= nil
          tag.contents  = ['<strong>'+tag.contents.to_s+'</strong>'] if
(tag['style'] =~ /bold/)
          tag.contents  = ['<em>'+tag.contents.to_s+'</em>'] if
(tag['style'] =~ /italic/)
          tag.contents  = ['<u>'+tag.contents.to_s+'</u>'] if
(tag['style'] =~ /underline/)
     end
    end
soup_string = str_replace(soup.html.to_s)

return Sanitize.clean(soup_string.to_s, :elements =>
['div','p','span','center','table','tr','th','td','blockquote', 'br',
'cite', 'code', 'dd', 'dl', 'dt','em','i', 'li', 'ol','pre', 'q',
'small', 'strike','strong', 'sub','sup', 'u', 'ul','tbody']),
  but the problem is that I want to preserver the center and right
justifications also, which is not happening if I give 'center' here. If
any body know how to preserve justifications pls help me.

Thanks In Advance,
Santosh




Jun Y. Kim wrote:
> you can also use ruby library Sanitize (http://wonko.com/post/sanitize)
>
> This library can make you parse html template very easily.
>
> let's see the following examples.
>
> Using Sanitize is easy. First, install it:
>   sudo gem install sanitize
>
> Then call it like so:
>
>   require 'rubygems'
>   require 'sanitize'
>
>   html = '<b><a href="http://foo.com/">foo</a></b><img
> src="http://foo.com/bar.jpg
> " />'
>
>   Sanitize.clean(html) # => 'foo'
>
> By default, Sanitize removes all HTML. You can use one of the built-in
> configs to tell Sanitize to allow certain attributes and elements:
>
>   Sanitize.clean(html, Sanitize::Config::RESTRICTED)
>   # => '<b>foo</b>'
>
>   Sanitize.clean(html, Sanitize::Config::BASIC)
>   # => '<b><a href="http://foo.com/" rel="nofollow">foo</a></b>'
>
>   Sanitize.clean(html, Sanitize::Config::RELAXED)
>   # => '<b><a href="http://foo.com/">foo</a></b><img
> src="http://foo.com/bar.jpg
> " />'
>
> Or, if you��d like more control over what��s allowed, you can provide
> your own custom configuration:
>
>   Sanitize.clean(html, :elements => ['a', 'span'],
>       :attributes => {'a' => ['href', 'title'], 'span' => ['class']},
>       :protocols => {'a' => {'href' => ['http', 'https', 'mailto']}})
>
> good one :)
>
> 2009. 01. 02, ���� 6:42, Vivek Netha �ۼ�:
This topic is locked and can not be replied to.