Forum: Ruby 400 "Bad Request"

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Arun K. (Guest)
on 2009-03-26 17:25
Hi,
I'm developing a program to fetch the html contents of a site using
'net/http'. Everyting works fine except for http://www.youtube.com. When
i pass that url an error like this is found

/usr/lib/ruby/1.8/net/http.rb:2097:in `error!': 400 "Bad Request"
(Net::HTTPServerException)

I think it is because i'm not using any user agent. Can any body please
tell me any auggestion. I'll be really greatful. This is my code
snippet.

response = Net::HTTP.get_response(URI.parse("http://www.youtube.com"))
                    case response
                        when Net::HTTPSuccess     then response
                        when Net::HTTPRedirection then response =
Net::HTTP.get(URI.parse(response['location']))
                    else
                        response.error!
                    end

Thanks

regards
Arun K.
Luis P. (Guest)
on 2009-03-26 18:09
(Received via mailing list)
Hi

On Thu, Mar 26, 2009 at 12:21 PM, Arun K.
<removed_email_address@domain.invalid> wrote:
> snippet.
You can try getting the page with Mechanize [1].

irb -rubygems -rmechanize
irb(main):001:0> agent=WWW::Mechanize.new
irb(main):002:0> agent.get('http://www.youtube.com').code
=> "200"

Anyway, it seems it's as you said, without a User-Agent, youtube returns
a 400:
irb(main):003:0> agent.user_agent=nil
=> nil
irb(main):004:0> agent.get('http://www.youtube.com').code
WWW::Mechanize::ResponseCodeError: 400 => Net::HTTPBadRequest
  from
/usr/lib/ruby/gems/1.8/gems/mechanize-0.9.2/lib/www/mechanize.rb:229:in
`get'
  from (irb):4
irb(main):005:0>


[1] http://mechanize.rubyforge.org/
Coey M. (Guest)
on 2009-03-26 18:32
(Received via mailing list)
Arun K. writes:
 > Hi,
 > I'm developing a program to fetch the html contents of a site using
 > 'net/http'. Everyting works fine except for http://www.youtube.com.
When
 > i pass that url an error like this is found
 >
 > /usr/lib/ruby/1.8/net/http.rb:2097:in `error!': 400 "Bad Request"
 > (Net::HTTPServerException)
 >
 > I think it is because i'm not using any user agent. Can any body
please
 > tell me any auggestion. I'll be really greatful. This is my code
 > snippet.
 >
 > response =
Net::HTTP.get_response(URI.parse("http://www.youtube.com"))
 >                     case response
 >                         when Net::HTTPSuccess     then response
 >                         when Net::HTTPRedirection then response =
 > Net::HTTP.get(URI.parse(response['location']))
 >                     else
 >                         response.error!
 >                     end
 >
 > Thanks
 >
 > regards
 > Arun K.
 > --
 > Posted via http://www.ruby-forum.com/.
 >

From a quick look at the Net::HTTP RDoc, it would look like your best
option would be to use the public instance 'get' method.

Here is a suggested rework of your code snippet:

Net::HTTP.start('www.youtube.com', 80) {|http|
  response = http.get('/', {'User-Agent'=>'ruby/net::http'})
  case response
    when Net::HTTPSuccess     then response
    when Net::HTTPRedirection then response =
Net::HTTP.get(URI.parse(response['location']))
  else
      response.error!
  end
}

Of course, this quick snippet does not account for when you get a
Net::HTTPRedirection and the redirected host is 'www.youtube.com',
again.  (Then again, your original code didn't error check the
redirection response either.)  So this isn't a complete solution, but
it will at least show you how to send the "User-Agent" header; I used
a variation of this code and was able to get content from
www.youtube.com.

Coey
This topic is locked and can not be replied to.