Using net/HTTP - How to login and then retrieve data?

dubstep · January 21, 2011, 8:20pm

I’m new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I’m working in a Windows environment with Ruby 1.8 and RubyMine.

I’ve been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

Please review my code and let me know what I need to do for the POST to
work.

Thanks!

–Bob

require ‘net/http’

#LOGIN
url1 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login’)
req = Net::HTTP::Get.new(url1.path)
req.basic_auth ‘myuser’, ‘mypass’
req.set_form_data({“RETS-Version” => “RETS/1.7”, ‘Accept’ => ‘/’,
‘User-Agent’ => ‘Mozilla/4.0’})
res1 = Net::HTTP.start(url1.host, url1.port) {|http| http.request(req)}
puts res1.body # This returns a 200 response and a listing of methods
that can be called, therefore this Login snippet works

#POST
url2Str = ‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search’
url2 = URI.parse(url2Str)
path = ‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search’
http = Net::HTTP.new(url2.host, 80)
data =
‘SearchType=Property&Limit=15000&Format=COMPACT&Query=%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29&Class=BC_4’
headers = {
‘Accept’ => ‘/’,
‘User-Agent’ => ‘Mozilla/4.0’,
‘RETS-Version’ => ‘RETS/1.7’
}
res2 = http.post(url2Str, data, headers)
puts res2.body # This returns error <Net::HTTPUnauthorized 401 Invalid
Username/Password combination readbody=true>

#LOGOUT
url3 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout’)
req = Net::HTTP::Get.new(url3.path)
req.basic_auth ‘myuser’, ‘mypass’
req.set_form_data({“RETS-Version” => “RETS/1.7”, ‘Accept’ => ‘/’,
‘User-Agent’ => ‘Mozilla/4.0’})
res3 = Net::HTTP.start(url3.host, url3.port) {|http| http.request(req)}
puts res3.body # This returns a 200 response

mrrobert · January 21, 2011, 8:59pm

On Jan 21, 2011, at 2:20 PM, Bob C. wrote:

to take place for a successful Login-POST-Logout.

#LOGIN
url1 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/
login’)
req = Net::HTTP::Get.new(url1.path)
req.basic_auth ‘myuser’, ‘mypass’

Do you need to also set .basic_auth on the POST request?
Do you need to store a cookie? (In which case, you might prefer using
httpclient gem rather than net/http directly.)

You can also do:

 http = Net::HTTP.new(url.host, url.port)
 http.read_timeout = 120
 if url.scheme == 'https' || url.port == 443
   http.use_ssl = true
   http.verify_mode = OpenSSL::SSL::VERIFY_NONE
   http.timeout = 120
 end
 http.start do |connection|
   request = Net::HTTP::Post.new(url.request_uri)
   request.basic_auth 'user', 'password'

   request.body = 'something if needed'

   response = connection.request(request)

   # do other stuff with the same connection...
 end

url2 = URI.parse(url2Str)
}
req.set_form_data({“RETS-Version” => “RETS/1.7”, ‘Accept’ => ‘/’,
‘User-Agent’ => ‘Mozilla/4.0’})
res3 = Net::HTTP.start(url3.host, url3.port) {|http|
http.request(req)}
puts res3.body # This returns a 200 response

–
Posted via http://www.ruby-forum.com/.

Rob B.
[email protected] http://AgileConsultingLLC.com/
[email protected] http://GaslightSoftware.com/

mrrobert · January 22, 2011, 1:01am

Yes, POST also requires basic_auth.

No, a cookie is not required.

For now I’d like to stick with using net/http, and possibly later look
at things like httpclient.

When I code according to your example (see below) I am still failing on
the SEARCH operation due to error <Net::HTTPPreconditionFailed 412
Precondition Failed-User not logged in. readbody=true>. This doesn’t
make sense because I have successfully logged in, and I am re-using the
connection.

I appreciate your help Rob, because I didn’t know how to re-use the
connection - Thanks!

–Bob

Here’s my current code:

require ‘net/http’

url1 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login’)
http = Net::HTTP.new(url1.host, url1.port)
http.read_timeout = 120
http.start do |connection|

#LOGIN
request1 = Net::HTTP::Post.new(url1.request_uri)
request1.basic_auth ‘myuser’, ‘mypass’
request1.set_form_data({‘RETS-Version’ => ‘RETS/1.7’, ‘Accept’ =>
‘/’, ‘User-Agent’ => ‘Mozilla/4.0’})
#request1.body = ‘something if needed’
response1 = connection.request(request1)
puts response1.body

#SEARCH
url2 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search’)
request2 = Net::HTTP::Post.new(url2.request_uri)
request2.basic_auth ‘myuser’, ‘mypass’
request2.set_form_data({‘RETS-Version’ => ‘RETS/1.7’, ‘Accept’ =>
‘/’, ‘User-Agent’ => ‘Mozilla/4.0’,
‘SearchType’ => ‘Property’,
‘Limit’ => ‘15000’,
‘Format’ => ‘COMPACT’,
‘Query’ =>
‘%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29’,
‘Class’ => ‘BC_4’
})
response2 = connection.request(request2)
puts response2.inspect

#LOGOUT
url3 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout’)
request3 = Net::HTTP::Post.new(url3.request_uri)
request3.set_form_data({‘RETS-Version’ => ‘RETS/1.7’, ‘Accept’ =>
‘/’, ‘User-Agent’ => ‘Mozilla/4.0’})
request3.basic_auth ‘myuser’, ‘mypass’
response3 = connection.request(request3)
puts response3.body
end
#<Net::HTTPOK 200 OK readbody=true>

#<Net::HTTPPreconditionFailed 412 Precondition Failed-User not logged
in. readbody=true>

#<Net::HTTPOK 200 OK readbody=true>

Rob B. wrote in post #976647:

On Jan 21, 2011, at 2:20 PM, Bob C. wrote:

to take place for a successful Login-POST-Logout.

#LOGIN
url1 =
URI.parse(‘http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/
login’)
req = Net::HTTP::Get.new(url1.path)
req.basic_auth ‘myuser’, ‘mypass’

Do you need to also set .basic_auth on the POST request?
Do you need to store a cookie? (In which case, you might prefer using
httpclient gem rather than net/http directly.)

You can also do:
 http = Net::HTTP.new(url.host, url.port)
 http.read_timeout = 120
 if url.scheme == 'https' || url.port == 443
   http.use_ssl = true
   http.verify_mode = OpenSSL::SSL::VERIFY_NONE
   http.timeout = 120
 end
 http.start do |connection|
   request = Net::HTTP::Post.new(url.request_uri)
   request.basic_auth 'user', 'password'

   request.body = 'something if needed'

   response = connection.request(request)

   # do other stuff with the same connection...
 end
url2 = URI.parse(url2Str)
}
req.set_form_data({“RETS-Version” => “RETS/1.7”, ‘Accept’ => ‘/’,
‘User-Agent’ => ‘Mozilla/4.0’})
res3 = Net::HTTP.start(url3.host, url3.port) {|http|
http.request(req)}
puts res3.body # This returns a 200 response

–
Posted via http://www.ruby-forum.com/.

Rob B.
[email protected] http://AgileConsultingLLC.com/
[email protected] http://GaslightSoftware.com/

mrrobert · January 21, 2011, 9:16pm

On Fri, Jan 21, 2011 at 7:20 PM, Bob C. [email protected] wrote:

Well Net::HTTP is a bit on the low level side. You really need to know
the
details of HTTP in order to be productive.

There are a lot of alternative HTTP client libraries out there that may
give
you what you want a lot easier (abstracting the HTTP details away);
OTOH:
RestClient
HttpClient
Patron
Httparty
Resourceful
All of those are pure Ruby, and should work fine on windows.
For pure web scraping/form interacting clients you can take it up a
level.
WATIR is well worth a look if you are on windows (your QA guys will want
a
demo, as it has a visual mode)
There is also Webrat and Capybara which are outstanding, normally used
for
assertion level testing.

If Java is your thing, you might want to consider JRuby specific options
as
well, like Celerity (which uses native Java HttpClient IIRC)

Net::HTTP forces you down into a very low level of HTTP mechanics. For
most
use cases this is not necessary, and its a tricky library to work with
if
your needs are simpler.
Many of those other options are easier to learn and may be more directly
applicable to your problem (logging in, interacting & leaving)

mrrobert · January 23, 2011, 4:06am

http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

mrrobert · January 24, 2011, 10:43pm

soldier.coder wrote in post #976863:

http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

Thanks for that - looks like an interesting gem. I will try it out.

BTW: I have solved my problem - was using an encoded “Query” param that
was screwing up the server processing.

–Bob