Question about Mechanize and Google API

Ruby
1

Hi,

I’m trying to use Mechanize to retrieve spreadsheets from Google Docs
using the Google API. So far I got the ClientLogin to work, but I
can’t seem to be able to use the Auth token they give me back. This is
what I have:

require ‘logger’
require ‘mechanize’

agent = WWW::Mechanize.new {|a| a.log = Logger.new(“test.log”)}
response = agent.post(“https://www.google.com/accounts/ClientLogin”,
“Email” => “[email protected]”, “Passwd” => “xxxxx”, “service” => “wise”,
“source” => “personal-test-1.0”, “accountType” => “GOOGLE”)
h = {}
response.body.each {|line| a = line.split(“=”); h[a[0]] = a[1]}
auth = h[“Auth”]
agent.basic_auth(“GoogleLogin”, auth)

this didn’t work either

#agent.auth(“GoogleLogin”, auth)
page =
agent.get(“http://spreadsheets.google.com/feeds/spreadsheets/private/full”)

The problem is, according to google I have to:
"After a successful authentication request, use the Auth value to
create an Authorization header for each request:

Authorization: GoogleLogin auth=yourAuthValue"

I’ve checked and the ClientLogin is working and returning me an Auth
token, but I can’t get Mechanize to use it. In the Mechanize logs I
see this:
I, [2007-10-08T17:01:52.211778 #24929] INFO – : Net::HTTP::Post:
/accounts/ClientLogin
D, [2007-10-08T17:01:52.394047 #24929] DEBUG – : request-header:
accept-language => en-us,en;q0.5
D, [2007-10-08T17:01:52.394214 #24929] DEBUG – : request-header:
connection => keep-alive
D, [2007-10-08T17:01:52.394305 #24929] DEBUG – : request-header: accept
=> /
D, [2007-10-08T17:01:52.394394 #24929] DEBUG – : request-header:
accept-encoding => gzip,identity
D, [2007-10-08T17:01:52.394482 #24929] DEBUG – : request-header:
user-agent => WWW-Mechanize/0.6.10
(http://rubyforge.org/projects/mechanize/)
D, [2007-10-08T17:01:52.394570 #24929] DEBUG – : request-header:
content-type => application/x-www-form-urlencoded
D, [2007-10-08T17:01:52.394656 #24929] DEBUG – : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,;q=0.7
D, [2007-10-08T17:01:52.394742 #24929] DEBUG – : request-header:
content-length => 104
D, [2007-10-08T17:01:52.394828 #24929] DEBUG – : request-header:
keep-alive => 300
D, [2007-10-08T17:01:52.613138 #24929] DEBUG – : Read 563 bytes
D, [2007-10-08T17:01:52.613358 #24929] DEBUG – : response-header:
cache-control => no-cache
D, [2007-10-08T17:01:52.613483 #24929] DEBUG – : response-header:
date => Mon, 08 Oct 2007 15:00:17 GMT
D, [2007-10-08T17:01:52.613603 #24929] DEBUG – : response-header:
content-type => text/plain
D, [2007-10-08T17:01:52.613721 #24929] DEBUG – : response-header:
server => GFE/1.3
D, [2007-10-08T17:01:52.613837 #24929] DEBUG – : response-header:
content-length => 563
D, [2007-10-08T17:01:52.613969 #24929] DEBUG – : response-header:
pragma => no-cache
I, [2007-10-08T17:01:52.614363 #24929] INFO – : status: 200
I, [2007-10-08T17:13:15.370472 #24929] INFO – : Net::HTTP::Get:
/feeds/spreadsheets/private/full
D, [2007-10-08T17:13:15.482357 #24929] DEBUG – : request-header:
accept-language => en-us,en;q0.5
D, [2007-10-08T17:13:15.482481 #24929] DEBUG – : request-header:
connection => keep-alive
D, [2007-10-08T17:13:15.482534 #24929] DEBUG – : request-header: accept
=> /
D, [2007-10-08T17:13:15.482587 #24929] DEBUG – : request-header:
accept-encoding => gzip,identity
D, [2007-10-08T17:13:15.482639 #24929] DEBUG – : request-header:
user-agent => WWW-Mechanize/0.6.10
(http://rubyforge.org/projects/mechanize/)
D, [2007-10-08T17:13:15.482692 #24929] DEBUG – : request-header:
referer => https://www.google.com/accounts/ClientLogin
D, [2007-10-08T17:13:15.482745 #24929] DEBUG – : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,;q=0.7
D, [2007-10-08T17:13:15.482796 #24929] DEBUG – : request-header:
keep-alive => 300
D, [2007-10-08T17:13:15.652098 #24929] DEBUG – : Read 141 bytes
D, [2007-10-08T17:13:15.652313 #24929] DEBUG – : response-header:
cache-control => no-cache, no-store, max-age=0, must-revalidate
D, [2007-10-08T17:13:15.652438 #24929] DEBUG – : response-header:
expires => Fri, 01 Jan 1990 00:00:00 GMT
D, [2007-10-08T17:13:15.652604 #24929] DEBUG – : response-header:
date => Mon, 08 Oct 2007 15:11:40 GMT
D, [2007-10-08T17:13:15.652725 #24929] DEBUG – : response-header:
content-type => text/html; charset=UTF-8
D, [2007-10-08T17:13:15.652859 #24929] DEBUG – : response-header:
www-authenticate => No credentials were included in your request.
D, [2007-10-08T17:13:15.652977 #24929] DEBUG – : response-header:
server => GFE/1.3
D, [2007-10-08T17:13:15.653094 #24929] DEBUG – : response-header:
set-cookie => S=trix=Xc7oom0LVJ8; Path=/
D, [2007-10-08T17:13:15.653211 #24929] DEBUG – : response-header:
content-length => 141
D, [2007-10-08T17:13:15.653326 #24929] DEBUG – : response-header:
pragma => no-cache
D, [2007-10-08T17:13:15.659559 #24929] DEBUG – : saved cookie:
S=trix=Xc7oom0LVJ8
I, [2007-10-08T17:13:15.659765 #24929] INFO – : status: 404

So, it seems that the credentials are not correctly set. Anybody knows
how to do this?

Thanks,

Jesus.

2

On 10/8/07, Jesús Gabriel y Galán [email protected] wrote:

Hi,

I’m trying to use Mechanize to retrieve spreadsheets from Google Docs
using the Google API. So far I got the ClientLogin to work, but I
can’t seem to be able to use the Auth token they give me back. This is
what I have:

From my experience Mechanize is a bit low level, especially in this
area.
You are expected to write the code yourself to handle, redirects,
authentication,
SSL etc.

You might want to try open-uri or Hpricot as well. But I think there is
no
right-sized solution - you will still need to do a bit of code writing
yourself.

BTW - you should check out the new RESTful Web Services book (O’Reilly).
In addition to just being an excellent book, it has a lot of practical
advice
on HTTP clients and directly manipulating HTTP headers, which is I
suspect what you will ultimately need to do.

3

Jesus,

the Gdata gem works with the google Gdata API , I’m currently
implementing
the gcalendar API for that gem , but luckely for you, I think the
0.4version has some spreadsheet implementation.

4

On 10/8/07, Matias Pablo B. [email protected] wrote:

the Gdata gem works with the google Gdata API , I’m currently implementing
the gcalendar API for that gem , but luckely for you, I think the
0.4version has some spreadsheet implementation.

Hi, yes, I had checked the Gdata gem, but saw it was pre alpha. In any
case I installed it and checked the spreadsheet class but I just saw a
couple of methods to demo how to evaluate a cell, and not the complete
functionality of the Google API. Am I right?

I’m thinking of stop using Mechanize for this and use just net/http
for the simple post and get I need to do, so I can customize the
headers myself, as I’ve seen you do in the gdata classes.

Thanks,

Jesus.

5

Richard C. wrote:

From my experience Mechanize is a bit low level, especially in this area.
You are expected to write the code yourself to handle, redirects,
authentication,
SSL etc.

You might want to try open-uri or Hpricot as well. But I think there is no
right-sized solution - you will still need to do a bit of code writing yourself.

Just for the record, last time I checked, Mechanize did use Hpricot to
parse pages. AFAIK, you can even provide Mechainze with your own page
parser class before fetching the stuff.

mortee

6

On 10/9/07, mortee [email protected] wrote:

parse pages. AFAIK, you can even provide Mechainze with your own page
parser class before fetching the stuff.

Yes, Mechanize parses pages with Hpricot, and also allows you to use
an Hpricot-like API on them for search:

agent.get(‘http://rubyforge.org’).search(“//strong”)

The problem I was having with it is the fact that I didn’t know how to
specify arbitrary headers for my request, or make the Google
authentication stuff to work.

Working directly with net/https worked for me (code “inspired” by the
Gdata gem :-):

require ‘net/http’
require ‘net/https’
require ‘uri’
require ‘hpricot’

module Net
class HTTPS < HTTP
def initialize(address, port = nil)
super(address, port)
self.use_ssl = true
end
end
end

h = {“Email” => “[email protected]”,
“Passwd” => “xxxxx”,
“service” => “wise”,
“source” => “personal-test-1.0”,
“accountType” => “GOOGLE”
}

url = URI.parse(‘https://www.google.com/accounts/ClientLogin’)
response = Net::HTTPS.post_form(url, h)
auth = “”
response.body.each do |line|
if line =~ /Auth=/
auth = line.split(“=”)[1].chomp
end
end

headers = {“Authorization” => “GoogleLogin auth=#{auth}”}
http = Net::HTTP.new(“spreadsheets.google.com”, 80)
response, data = http.get(“/feeds/spreadsheets/private/full”, headers)
document = Hpricot(data)
document.search(“//entry”).each do |entry|
p entry
end

This gives you a document with the list of private spreadsheets of the
user. Then you can use Hpricot or whatever you want to parse the atom
feed.

Jesus.