# Yahoo API and Ruby

**URL:** <https://www.ruby-forum.com/t/yahoo-api-and-ruby/80915>\
**Category:** Ruby\
**Created:** [January 1, 2007, 10:52pm UTC](https://www.ruby-forum.com/t/yahoo-api-and-ruby/80915 "2007-01-01T22:52:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [January 1, 2007, 10:52pm UTC](https://www.ruby-forum.com/t/yahoo-api-and-ruby/80915/1 "2007-01-01T22:52:12Z")

</div>

I’m working on a couple of large sites that aren’t sending the correct  
response codes for missing pages. I want to use the Yahoo API key to  
search Yahoo’s cache to see if it has any clues about what pages are  
sending bad responses. So I need to get all 1000 results from the  
Yahoo cache and write it to a spreadsheet. Then I can sort the URL  
data and response codes in the spreadsheet.

I can only request 100 results at a time and can set a different  
“start” number for each request. The first request would be start=1,  
the second, start=101 and so on.

The other problem is that it won’t get the response codes. I just get  
this unhelpful error message:  
c:/ruby/lib/ruby/1.8/net/http.rb:1467:in `initialize': HTTP request path y (ArgumentError) from c:/ruby/lib/ruby/1.8/net/http.rb:1585:in `initialize’  
from hpricot\_test.rb:32:in `new' from hpricot_test.rb:32:in `get\_headers’  
from hpricot\_test.rb:80:in `generate_workbook from hpricot_test.rb:70:in `each’  
from hpricot\_test.rb:70:in `generate\_workbook  
from hpricot\_test.rb:94

Here is the code:

#!/usr/bin/ruby -w

require ‘net/http’  
require ‘uri’  
require ‘hpricot’  
require ‘spreadsheet/excel’  
include Spreadsheet

def get\_cache  
# set variables for POST request  
appid = ‘yahooAPI-key’ # a Yahoo API key goes here  
query = ‘[http://www.example.com](http://www.example.com)’ # a Web site to check goes here

```
# this gets the first 100 results, but I want to loop through
# it 10 times with a different "start" number to get all 1000
# available results
results = 100
start = 1

post_args = {
  'appid' => appid,
  'query' => query,
  'results' => results,
  'start' => start
}
url =

```

URI.parse(‘[http://search.yahooapis.com/SiteExplorerService/V1/pageData](http://search.yahooapis.com/SiteExplorerService/V1/pageData)’)

```
# send post request
@resp, @data = Net::HTTP.post_form(url, post_args)

# read XML
@doc = Hpricot(@data)

```

end

def get\_headers(url)  
# This gets the response code for the page to see if it exists  
(200, 301, 404, etc.)  
page = URI.parse(url)  
req = Net::HTTP::Get.new(page.path)  
res = Net::HTTP.start(page.host, page.port) { |http|  
http.request(req)  
}  
return res.code  
end

def generate\_workbook  
# create new workbook and worksheet  
workbook = Spreadsheet::Excel.new(“yahoo\_cache.xls”)  
worksheet = workbook.add\_worksheet(‘Yahoo Cache’)

```
# set variables
current_row = 2
format_nil = nil
format_header = Format.new(
  :color => 'white',
  :bg_color => 'gray',
  :bold => true
)
workbook.add_format(format_header)
workbook.add_format(format_nil)

# Add header row
worksheet.write(0,0,"Yahoo's Cache for Site", format_nil)
worksheet.write(1,0,"TITLE",format_header)
worksheet.write(1,1,"URL", format_header)
# worksheet.write(1,2,"CODE", format_header)
# worksheet.write(1,3,"LOCATION", format_header) # coming soon

# Add xml_data to worksheet
(@doc/"result").each do |el|
  result_title = (el/"title").text
  result_url = (el/"url").text
  worksheet.write(current_row, 0, result_title, format_nil)
  worksheet.write(current_row, 1, result_url, format_nil)

  # get response codes -- this is causing an error with

```

“result\_url” – maybe it isn’t a URL in a string?  
# see error message at top of this post  
# response\_code ||= 0  
# response\_code = get\_headers(result\_url) # this works if I put  
a URL here, but not with the result\_url variable  
# worksheet.write(current\_row, 2, response\_code, format\_nil)

```
  # move to the next row in the spreadsheet before going to the

```

next XML item  
current\_row += 1  
end

```
# finished, close the workbook
workbook.close

```

end

====  
The above code works (except the part that gets response codes). The  
following code is a previous version where I tried to loop through all  
1000 results. (It was using xmlsimple.) I couldn’t figure out how to  
store each set of XML – each request is an entire XML file. I tried  
@pass[count], but it wasn’t working. Any ideas about a good way to  
store each request?

```
# prepare to loop through 100 results
count = 1
start = 1

# pass[] = each of the 10 requests to Yahoo
@pass = []

# perform the loop
while count < 11 do
  post_args = {
    'appid' => appid,
    'query' => query,
    'results' => results,
    'start' => start
  }

  # send post request
  @resp, @data = Net::HTTP.post_form(url, post_args)

  # read XML
  xml_data = XmlSimple.xml_in(@data)
  @pass[count] = xml_data
  # puts "Count: #{count}"
  # print @pass[count]

  # puts "Start: #{start}"
  # puts
  count += 1
  start += 100
end

```