Yahoo API and Ruby


#1

I’m working on a couple of large sites that aren’t sending the correct
response codes for missing pages. I want to use the Yahoo API key to
search Yahoo’s cache to see if it has any clues about what pages are
sending bad responses. So I need to get all 1000 results from the
Yahoo cache and write it to a spreadsheet. Then I can sort the URL
data and response codes in the spreadsheet.

I can only request 100 results at a time and can set a different
“start” number for each request. The first request would be start=1,
the second, start=101 and so on.

The other problem is that it won’t get the response codes. I just get
this unhelpful error message:
c:/ruby/lib/ruby/1.8/net/http.rb:1467:in initialize': HTTP request path y (ArgumentError) from c:/ruby/lib/ruby/1.8/net/http.rb:1585:ininitialize’
from hpricot_test.rb:32:in new' from hpricot_test.rb:32:inget_headers’
from hpricot_test.rb:80:in generate_workbook from hpricot_test.rb:70:ineach’
from hpricot_test.rb:70:in `generate_workbook
from hpricot_test.rb:94

Here is the code:

#!/usr/bin/ruby -w

require ‘net/http’
require ‘uri’
require ‘hpricot’
require ‘spreadsheet/excel’
include Spreadsheet

def get_cache
# set variables for POST request
appid = ‘yahooAPI-key’ # a Yahoo API key goes here
query = ‘http://www.example.com’ # a Web site to check goes here

# this gets the first 100 results, but I want to loop through
#  it 10 times with a different "start" number to get all 1000
#  available results
results = 100
start = 1

post_args = {
  'appid' => appid,
  'query' => query,
  'results' => results,
  'start' => start
}
url =

URI.parse(‘http://search.yahooapis.com/SiteExplorerService/V1/pageData’)

# send post request
@resp, @data = Net::HTTP.post_form(url, post_args)

# read XML
@doc = Hpricot(@data)

end

def get_headers(url)
# This gets the response code for the page to see if it exists
(200, 301, 404, etc.)
page = URI.parse(url)
req = Net::HTTP::Get.new(page.path)
res = Net::HTTP.start(page.host, page.port) { |http|
http.request(req)
}
return res.code
end

def generate_workbook
# create new workbook and worksheet
workbook = Spreadsheet::Excel.new(“yahoo_cache.xls”)
worksheet = workbook.add_worksheet(‘Yahoo Cache’)

# set variables
current_row = 2
format_nil = nil
format_header = Format.new(
  :color => 'white',
  :bg_color => 'gray',
  :bold  => true
)
workbook.add_format(format_header)
workbook.add_format(format_nil)

# Add header row
worksheet.write(0,0,"Yahoo's Cache for Site", format_nil)
worksheet.write(1,0,"TITLE",format_header)
worksheet.write(1,1,"URL", format_header)
# worksheet.write(1,2,"CODE", format_header)
# worksheet.write(1,3,"LOCATION", format_header) # coming soon

# Add xml_data to worksheet
(@doc/"result").each do |el|
  result_title = (el/"title").text
  result_url   = (el/"url").text
  worksheet.write(current_row, 0, result_title, format_nil)
  worksheet.write(current_row, 1, result_url, format_nil)

  # get response codes -- this is causing an error with

“result_url” – maybe it isn’t a URL in a string?
# see error message at top of this post
# response_code ||= 0
# response_code = get_headers(result_url) # this works if I put
a URL here, but not with the result_url variable
# worksheet.write(current_row, 2, response_code, format_nil)

  # move to the next row in the spreadsheet before going to the

next XML item
current_row += 1
end

# finished, close the workbook
workbook.close

end

====
The above code works (except the part that gets response codes). The
following code is a previous version where I tried to loop through all
1000 results. (It was using xmlsimple.) I couldn’t figure out how to
store each set of XML – each request is an entire XML file. I tried
@pass[count], but it wasn’t working. Any ideas about a good way to
store each request?

# prepare to loop through 100 results
count = 1
start = 1

# pass[] = each of the 10 requests to Yahoo
@pass = []

# perform the loop
while count < 11 do
  post_args = {
    'appid' => appid,
    'query' => query,
    'results' => results,
    'start' => start
  }

  # send post request
  @resp, @data = Net::HTTP.post_form(url, post_args)

  # read XML
  xml_data = XmlSimple.xml_in(@data)
  @pass[count] = xml_data
  # puts "Count: #{count}"
  # print @pass[count]

  # puts "Start: #{start}"
  # puts
  count += 1
  start += 100
end