Posting over https

Ruby
1

Hello Rubyists,

I am struggling to post data over https and would be grateful for any
help.

I know the server works because I can hit it with curl and get the
response I expect:

curl -d url

When I try to do the same from my Ruby code the server doesn’t like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require ‘net/https’
require ‘uri’

url =
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { ‘key1’ => ‘value1’,
‘key2’ => ‘value2’,
‘keyn’ => ‘valuen’ }.map {|k,v| “#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}”}.join(‘&’)
response = request.post(“#{uri.path}?#{uri.query}”, data)

When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

Thanks in advance,
Andy S.

2

On Tue, Oct 10, 2006 at 12:04:26AM +0900, Andrew S. wrote:

When I try to do the same from my Ruby code the server doesn’t like it.
url =
When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

Try using Mechanize. You can post forms over https with it, and you
won’t have to deal with encoding query parameters or setting up
Net::HTTP.

http://mechanize.rubyforge.org/

–Aaron

3

Hi,

I am struggling to post data over https and would be grateful for any
help.

Try using Mechanize. You can post forms over https with it, and you
won’t have to deal with encoding query parameters or setting up
Net::HTTP.

OK, but I would still like to understand what’s wrong with my code so
I can learn and improve.

Thanks anyway for the pointer,

Andy S.

4

On 10/9/06, Andrew S. [email protected] wrote:

OK, but I would still like to understand what’s wrong with my code so
I can learn and improve.

Its admirable, but I think Net::HTTP is a bit too low level, and
requires a
bit too much HTTP knowledge for many applications. Mechanize seems
to be the business, and possibly Hpricot might also have similar
functionality.

I found Net::HTTP a bit too frustrating to use and the lack of good
examples
bothered me a lot. It seems that the real work along these lines
(programmatic
HTTP/HTML processing) is done with Mechanize & Hpricot.

5

On Tue, 10 Oct 2006, Andrew S. wrote:

uri = URI.parse(url)
data and everything was fine.

Any ideas?

Thanks in advance,
Andy S.

fortytwo :/var/www/html > cat form.cgi
#! /usr/local/bin/ruby
require ‘cgi’
require ‘yaml’
(c=CGI.new).out(“type” => “text/plain”){ c.params.to_yaml }

fortytwo :/var/www/html > cat post.rb
require ‘cgi’
require ‘net/https’
require ‘uri’

url = ARGV.shift || “https://fortytwo.merseine.nu/form.cgi”
uri = URI.parse url

data = {
‘key1’ => ‘value1’,
‘key2’ => ‘value2’,
‘keyn’ => ‘valuen’,
}

e = lambda{|x| CGI.escape x}
q = lambda{|h| h.map{|k,v| [e[k], e[v]].join ‘=’}.join(‘&’)}
form = q[data]

class HTTPS < Net::HTTP
def initialize *a, &b
super
self.use_ssl = true
self.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
end

HTTPS.start(uri.host, uri.port) do |https|
res = https.post uri.path, form
puts res.body
end

fortytwo :/var/www/html > ruby post.rb
https://fortytwo.merseine.nu/form.cgi

key1:

  • value1
    key2:
  • value2
    keyn:
  • valuen

-a

6

Hello again,

fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/
form.cgi

key1:

  • value1
    key2:
  • value2
    keyn:
  • valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-
fledged code sample. I do appreciate your taking the time to
respond, and so quickly.

Adapting my code in the light of Ara’s code above, with
‘self.set_debug_output $stderr’ added to the HTTPS initialize method,
I compared the client-server conversation with curl’s, which I knew
worked. This led me to discover that the server expected the content-
type header to be set to ‘application/x-www-form-urlencoded’.

For the record, here’s the code which works with the server I am
talking to (Protx’s payment gateway). Now that I have it working,
I’ll take Aaron and Richard’s advice and look into replacing it with
higher-level code using Mechanize.

require ‘net/https’
require ‘uri’

class HTTPS < Net::HTTP
def initialize *a, &b
super
self.use_ssl = true
self.verify_mode = OpenSSL::SSL::VERIFY_NONE
#self.set_debug_output $stderr
end
end

url = ‘https://some.host.com/foo/bar.asp?service=someservice’
uri = URI.parse url
data = {
‘key1’ => ‘value1’,
‘key2’ => ‘value2’,
‘keyn’ => ‘valuen’
}
e = lambda {|x| URI.escape x}
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join ‘=’}.join(‘&’)}
form = q[data]
HTTPS.start(uri.host, uri.port) do |https|
response = https.post(“#{uri.path}?#{uri.query}”, form, {‘content-
type’ => ‘application/x-www-form-urlencoded’})
puts response.body
end

Regards,
Andy S.

7

On 10/10/06, Andrew S. [email protected] wrote:

keyn:
type header to be set to ‘application/x-www-form-urlencoded’.
def initialize *a, &b
‘key1’ => ‘value1’,
end

Regards,
Andy S.

Thanks for sharing this, I could not help you out as I lost my old POST
code, now I’ll just adapt your’s (thx Ara) :slight_smile:
Robert

–
Deux choses sont infinies : l’univers et la bêtise humaine ; en ce qui
concerne l’univers, je n’en ai pas acquis la certitude absolue.

  • Albert Einstein