Net/HTTP is flaky?

maz · July 19, 2006, 1:51pm

Inexplicably, the following code fails:

http = Net::HTTP.new(“somesite.com”)
puts http.get(“/”)[1]

...with this error:

ruby/1.8/net/http.rb:925:in ‘[]’: undefined method ‘downcase’ for
1:Fixnum
(NoMethodError)
from web.rb:5

Now, I do have access to a Linux system with Ruby 1.6 and it

sometimes
succeeds but often fails with this error:

/usr/lib/ruby/1.6/net/protocol.rb:221:in error!': 403 "Forbidden" (Net::ProtoFatalError) from /usr/lib/ruby/1.6/net/http.rb:1217:in value’
from /usr/lib/ruby/1.6/net/http.rb:605:in `get’
from ./web.rb:16

How is this possible?  How can it sometimes succeed and sometimes

fail?
Why can’t this reliably work?
Thank you…

maz · July 19, 2006, 2:50pm

On Jul 19, 2006, at 7:50 AM, Just Another Victim of the Ambient
Morality wrote:

ruby/1.8/net/http.rb:925:in ‘[]’: undefined method ‘downcase’ for
1:Fixnum
(NoMethodError)
from web.rb:5

The interface to Net::HTTP changed between 1.6 and 1.8
You want
http.get(’/’).body

    from ./web.rb:16


How is this possible?  How can it sometimes succeed and  
sometimes fail?
Why can’t this reliably work?
Thank you…

It doesn’t reliably work because this is the internet. Also the
particular error you are seeing means that the webserver has
forbidden access to the url you are trying to, um, access.

maz · July 19, 2006, 11:26pm

On Jul 19, 2006, at 1:20 PM, Just Another Victim of the Ambient
Morality wrote:

    from /usr/lib/ruby/1.6/net/http.rb:605:in `get'
particular error you are seeing means that the webserver has
forbidden
access to the url you are trying to, um, access.
But I can access the page just fine with a web browser. For instance,
go to “/” at “en.wikipedia.org” and it will reliably work with a
browser.

No, it redirects to /wiki/Main_Page

Try it with Ruby and it will usually fail, although I did get it to
work
occasionally, last night, which I don’t get.

$ ruby -rnet/http
Net::HTTP.start ‘en.wikipedia.org’ do |http| res = http.get ‘/’; p
res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Matches browser behavior.

Hell, “/” at “music.com” has never worked and there doesn’t
appear to
be any kind of redirect or anything.

$ ruby -rnet/http
Net::HTTP.start ‘music.com’ do |http| res = http.get ‘/’; p res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Redirects to www.music.com in the browser, Net::HTTP matches browser
behavior.

–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

maz · July 19, 2006, 10:20pm

“Logan C.” [email protected] wrote in message
news:[email protected]…

The interface to Net::HTTP changed between 1.6 and 1.8
You want
http.get(‘/’).body

Ah, thanks...

How is this possible?  How can it sometimes succeed and  sometimes 
fail?
Why can’t this reliably work?
Thank you…
It doesn’t reliably work because this is the internet. Also the
particular error you are seeing means that the webserver has forbidden
access to the url you are trying to, um, access.

But I can access the page just fine with a web browser.  For

instance,
go to “/” at “en.wikipedia.org” and it will reliably work with a
browser.
Try it with Ruby and it will usually fail, although I did get it to work
occasionally, last night, which I don’t get.
Hell, “/” at “music.com” has never worked and there doesn’t appear
to
be any kind of redirect or anything.
Does any of this make sense?
Thank you…

maz · July 20, 2006, 12:46am

On 7/19/06, Just Another Victim of the Ambient M.
[email protected] wrote:

Okay, so the site doesn't exist where I think it does and I get
redirected… in my browser. Net::HTTP doesn’t get redirected, it simply
fails. This behaviour doesn’t match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?

You might want to check out the API docs for Net::HTTP.

Copy/pasted from
http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html:

require 'net/http'
require 'uri'

def fetch(uri_str, limit = 10)
  # You should choose better exception.
  raise ArgumentError, 'HTTP redirect too deep' if limit == 0

  response = Net::HTTP.get_response(URI.parse(uri_str))
  case response
  when Net::HTTPSuccess     then response
  when Net::HTTPRedirection then fetch(response['location'], limit -

else
response.error!
end
end

print fetch(‘http://www.ruby-lang.org’)

maz · July 20, 2006, 12:49am

On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient
Morality wrote:

Okay, so the site doesn’t exist where I think it does and I get
redirected… in my browser. Net::HTTP doesn’t get redirected, it
simply
fails.

No, it doesn’t fail, it gives you a redirect. Follow it.

This behaviour doesn’t match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?

Follow the redirect. Look at open-uri for example code.

–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

maz · July 19, 2006, 11:58pm

“Eric H.” [email protected] wrote in message
news:[email protected]…

$ ruby -rnet/http
Net::HTTP.start ‘music.com’ do |http| res = http.get ‘/’; p res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

Redirects to www.music.com in the browser, Net::HTTP matches browser
behavior.

Okay, so the site doesn't exist where I think it does and I get

redirected… in my browser. Net::HTTP doesn’t get redirected, it
simply
fails. This behaviour doesn’t match my browser. Is there any way I
can
get redirected or find out where it wants to redirect me and go there,
myself?
Thanks…

maz · July 20, 2006, 1:05am

On 7/19/06, Joe Van D. [email protected] wrote:

Ok, so Net:HTTP doesn’t fail, it just returns pretty low-level
information. If you want browser behavior for redirects, you have to
roll it yourself. I ran into something similar 2 days ago when I was
trying to fetch pages which required basic authentication (Which, I
just learned, is the mechanism behind those urername & password login
popups that the browser generates). I did manage to roll my own,
based on some examples from this mailing list.
Then I had to write a wrapper around that because sometimes my URI’s
pointed to file:// instead of http:// - another thing the browser
handles invisibly.
Then there are those pages that use cookies to track logins…

So my question is, is there an existing library out there that
implements all these things a browser does? Where get(uri).page
returns the source of the same page I would see if I put the URI in
firefox?

-Adam

maz · July 20, 2006, 1:55am

“Eric H.” [email protected] wrote in message
news:[email protected]…

On Jul 19, 2006, at 2:55 PM, Just Another Victim of the Ambient Morality
wrote:

No, it doesn’t fail, it gives you a redirect. Follow it.

You know, it would be easier to follow the redirect if I knew that

it
was giving me a redirect and if I knew how to follow one were I to be
given
it…

This behaviour doesn’t match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go there,
myself?

Follow the redirect. Look at open-uri for example code.

You know, this is where Ruby's lack of documentation is really

biting me
in the ass. I found some rudimentary docs on how to use it but, you
know
what? For the life of me, no amount of googling can reveal exactly
where I
can get the open-uri module… Seriously, where do I download this
thing
from and how could I have known that?
Thank you, from a very frustrated would-be Ruby programmer…

maz · July 20, 2006, 6:40am

On Jul 19, 2006, at 7:03 PM, Adam S. wrote:

there,
popups that the browser generates). I did manage to roll my own,
based on some examples from this mailing list.
Then I had to write a wrapper around that because sometimes my URI’s
pointed to file:// instead of http:// - another thing the browser
handles invisibly.
Then there are those pages that use cookies to track logins…

So my question is, is there an existing library out there that
implements all these things a browser does? Where get(uri).page
returns the source of the same page I would see if I put the URI in
firefox?

WWW::Mechanize is a programmatic web browser for ruby:
http://mechanize.rubyforge.org/
http://rubyforge.org/projects/mechanize

maz · July 20, 2006, 2:56am

On Jul 19, 2006, at 4:55 PM, Just Another Victim of the Ambient
Morality wrote:

that it
was giving me a redirect and if I knew how to follow one were I to
be given
it…

Well, if you expect to use Net::HTTP you need to know both HTTP and
the library. You might prefer WWW::Mechanize instead.

You’ll get Top Quality Answers if you tell us what you really want to
do. Help with the intricacies of a library may not let us guide you
down the right path

Quoting myself:

$ ruby -rnet/http
Net::HTTP.start ‘en.wikipedia.org’ do |http| res = http.get ‘/’; p
res; end
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true>

That shows Net::HTTP returning a redirect when you perform a get.

This behaviour doesn’t match my browser. Is there any way I can
get redirected or find out where it wants to redirect me and go
there,
myself?

Follow the redirect. Look at open-uri for example code.

You know, this is where Ruby’s lack of documentation is really
biting me in the ass. I found some rudimentary docs on how to use
it but, you know what?

Unfortunately the Net::HTTP documentation isn’t enabled in 1.8. You
can use the 1.9 documentation however:

http://ruby-doc.org/core-1.9/classes/Net/HTTP.html

As luck would have it, it has a “Following Redirection” section a few
pages down.

For the life of me, no amount of googling can reveal exactly where
I can get the open-uri module… Seriously, where do I download
this thing from and how could I have known that?

It ships with ruby, require ‘open-uri’.

Thank you, from a very frustrated would-be Ruby programmer…

I’ve been working on getting lots more documentation enabled in 1.8.
I hope to get the net/ stuff turned on, but I don’t know if I’ll have
enough time to review/merge documentation from HEAD. (I’d really,
really like help with this, and Hugh S. has brought in a ton of
new documentation.)

–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

maz · July 20, 2006, 10:17pm

On 7/19/06, Just Another Victim of the Ambient M.
[email protected] wrote:

it…
what? For the life of me, no amount of googling can reveal exactly where I
can get the open-uri module… Seriously, where do I download this thing
from and how could I have known that?
Thank you, from a very frustrated would-be Ruby programmer…

open-uri is included in the Ruby standard library.

The process for finding its documentation isn’t that difficult.

Go to http://www.ruby-doc.org. Click on the 1.8.4 standard library
link. Click on the open-uri library link that’s in the left frame.
Tada!

maz · July 20, 2006, 9:35pm

> pages down. >

It’s in the regular standard library documentation as well:

http://ruby-doc.org/stdlib/
http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html

-Justin