Mechanize - extract href

Ruby
1

Hey there everyone. I’m having a slight problem using Mechanize. I’m
trying to scrape the yellowpages.com, and extract information about each
business listing. I’m extracting all the information I want, except for
one small portion: the business’s website. It is the href inside of a
link that I am trying to scrape. As far as I know, I’m following the
correct xpath rules, but I can’t seem to get the part I want. One
tricky thing that I’ve had to deal with is that not every listing has a
website. The website link and the “learn more” link are very similar,
xpath-wise, so I have to use an if statement to check the inner text of
both of them to make sure that I’m extracting the xpath one.

I’m scraping from
Best 30 Restaurants in Santa Barbara, CA with Reviews and my code
is attached.

Thanks so much for your help!

2

The example file might not help if your not using windows, but it might
give you some ideas. Sorry if this isn’t what you’re looking for at
all…

  • jethrow
3

Jethrow, thanks but that’s not quite what I need. I need to extract
this link’s href attribute, which is the website of the buisness. I’m
using xpath, using the “…/a/@href” method which I believe is the
correct one. But it just doesn’t extract anything! Any other ideas?

4

On Sat, Oct 16, 2010 at 3:50 PM, Corey W. [email protected]
wrote:

both of them to make sure that I’m extracting the xpath one.

I’m scraping from
Best 30 Restaurants in Santa Barbara, CA with Reviews and my code
is attached.

Your xpath:

website = website.search(“/a/@href”)

should be:

website = website.search(“./a/@href”)

a leading “/” means that you want the xpath search to begin from the
root of the document. “./” means to start from the context node, in
this case website.

5

Thanks Mike. I’ve updated the code. Putting “./” in my code messed up
what the code grabbed, so I eliminated all leading “/” and it worked.
However, the website part still doesn’t work. It looks like it is
grabbing what is inside of the a tags, instead of grabbing the href
address. A typical listing output looks like:

McDonald’s
1213 State St # B,
Santa Barbara
CA
93101
(805) 962-6976
»
?
Website
[“Restaurants”, “Fast Food Restaurants”, “American Restaurants”]

The three lines: , ?, and Website are all grabbed from the website
= website.search(…) line, but it’s grabbing the wrong thing! Do you
have any more suggestions?

6

I still haven’t figured this out. Perhaps I should phrase the question
a different way…

What is the preferred method of extracting the href attribute from a
link? I’ve tried doing it using .search() and searching for the xml
@href attribute. For some reason that’s not working for me.

Is there a different way of extracting this attribute, without using
.search and an xml path? I’m sure mechanize has some other method
too…

7

On Wed, Oct 27, 2010 at 2:57 AM, Corey W. [email protected]
wrote:

I still haven’t figured this out. Perhaps I should phrase the question
a different way…

What is the preferred method of extracting the href attribute from a
link? I’ve tried doing it using .search() and searching for the xml
@href attribute. For some reason that’s not working for me.

Is there a different way of extracting this attribute, without using
.search and an xml path? I’m sure mechanize has some other method
too…

With this and a local version of the page I was able to get the info you
want:

#!/bin/env ruby19

require ‘nokogiri’

raw = File.read(“restaurants.html”, mode: “r:UTF-8”)
puts raw.encoding

raw.force_encoding ‘UTF-8’

doc = Nokogiri.parse raw

doc.xpath(‘//div[@class=“listing_content”]’).each do |listing|
puts ‘----------------------------------------’

p listing.to_s[0…10]+“…”

puts listing
puts ‘----------------------------------------’

p listing.xpath(‘.//a//text()’).map(&:to_s)

listing.xpath(‘.//a[@href and contains(text(),“Website”)]/@href’).each
do |a|
p a.value
end
puts
end

Cheers

robert

8

On Thu, Oct 28, 2010 at 8:54 PM, Corey W. [email protected]
wrote:

Thanks! I eventually came up with something like this:

website =
page.search(“//div[@class=‘listing_content’]/ul[@class=‘features’]/li[1]”)

Wouldn’t it be better to first search for all div with
class=listing_content and then extract data for that single listing
from there?

initializes the link variable with something

link = “super”

“link = nil” is sufficient.

The segment below grabs the URL of the listing’s website, if it has

one. Otherwise it

puts ‘nil’ into the link variable

if website.inner_text() == " Website"

You can use my XPath to find those links.

website.search(“a”).map do |l|

#map is nonsense here - you only need #each.

link = l[‘href’]
end
else
link = “nil”

Didn’t you mean “link = nil”?

Kind regards

robert

9

Thanks! I eventually came up with something like this:

website =
page.search("//div[@class=‘listing_content’]/ul[@class=‘features’]/li[1]")

initializes the link variable with something

link = “super”

The segment below grabs the URL of the listing’s website, if it has

one. Otherwise it

puts ‘nil’ into the link variable

if website.inner_text() == “» Website”
website.search(“a”).map do |l|
link = l[‘href’]
end
else
link = “nil”
end

10

Robert, thank you so much for your help. I’m just starting out in ruby,
so I have a lot to learn! I’ve attached my code with the changes you
recommended. It works wonderfully! Are there any other changes or
optimizations I could make?

11

Robert,

Thanks again for your help. This script above is actually part of a
larger script, but I was just showing you the smaller portion for
clarity. I’ve attached the whole script, in order to put the for loop
in context. I’m using the for loop to grab multiple pages in
succession. Hopefully this makes it more clear.

I’ve made all the changes you recommended, except consolidating the puts
statements. Could you explain how I would do that? Thank you!

12

On Fri, Oct 29, 2010 at 9:15 AM, Corey W. [email protected]
wrote:

Robert, thank you so much for your help. I’m just starting out in ruby,
so I have a lot to learn! I’ve attached my code with the changes you
recommended.

Attachments:
http://www.ruby-forum.com/attachment/5263/mech.rb

Few remarks:

for x in 1…2 will execute the loop body exactly once. Are you sure
this is what you want?

You can combine all the puts at the end in a single statement.

You can replace all the “.gsub(/^\s+/, “”).gsub(/\s+$/, $/)” by
".strip. You do not need to append a newline because puts does that
already.

Why do you use ‘entry.search(“a”)’ and do not include the “a” in the
path expression?

Cheers

robert