Rsa sign and verify with openssl on windows

juuser · September 1, 2010, 12:38pm

Hello!

I’ve spent quite amount of time now to find out why can’t i verify at
server side (written in Java) a signature done in Ruby.

Here’s some of the code from Ruby side:
…
require ‘openssl’
require ‘base64’

private_key = OpenSSL::PKey::RSA.new(File.read(‘priv.pem’))
digest = Digest::SHA1.digest(data)
signed = Base64.encode64(private_key.private_encrypt(digest))
parameters.merge! ‘my_data’ => signed.gsub(/[\r\n]/, ‘’)
Net::HTTP.post_form(URI.parse(return_url), parameters).body
…

And at the Java side verification fails. It works with another real
services so the problem cannot be at Java side.

After some inspection, i’ve extracted public key from priv.pem with
openssl in PEM format and tried private_key.public_key() which also
returns a public key in PEM format, but it is different!

How is it possible that using Ruby’s OpenSSL::PKey::RSA#public_key
returns different public key than using openssl on command line?

I suspect that to be a culprit of the problem, but i’m not sure. I don’t
even understand how can that happen…

I also tried to sign the same data with the same private key at Java
side and got a different base64 output than Ruby…

I also tried to sign and verify with the same keys using openssl command
line tool and were successful.

Data has only ASCII characters although let’s not forget that in Java
UTF-8 is default. Could there be any encoding issues?

Anyway, any help or suggestion is welcome.

Best Regards,
Jarmo

juuser · September 1, 2010, 1:09pm

I suggest you generate a new throw-away private key, then you can post
the private key, complete programs in Ruby and Java and the openssl
command line version to replicate the problem, and the exact outputs you
see.

Also break it down bit by bit.

p data.bytesize # same as in Java?
p digest # should be 20 bytes, is it same as you get in Java?
p private_key.private_encrypt(digest)

After some inspection, i’ve extracted public key from priv.pem with
openssl in PEM format and tried private_key.public_key() which also
returns a public key in PEM format, but it is different!

Again, would like to see this replicated.

On those two PEM files, try somethine like

openssl rsa -in foo.pem -pubin -noout -text

to see how/if the contents are different.

I also tried to sign and verify with the same keys using openssl command
line tool and were successful.

So show the openssl command lines you used, then we can see if what
you’re doing in Ruby is different.

Data has only ASCII characters although let’s not forget that in Java
UTF-8 is default. Could there be any encoding issues?

Seems unlikely. Data is data, and Ruby’s openssl wrapper isn’t going to
be transcoding stuff before giving it to openssl (I’d certainly hope not
anyway)

juuser · September 1, 2010, 1:42pm

Here are some tests on ubuntu Lucid with ruby 1.8.7

$ openssl genrsa -out priv.pem 1024
Generating RSA private key, 1024 bit long modulus
…++++++
…++++++
e is 65537 (0x10001)

$ openssl rsa -in priv.pem -out pub.pem -pubout
writing RSA key

$ irb --simple-prompt

require ‘openssl’
=> true

private_key = OpenSSL::PKey::RSA.new(File.read(‘priv.pem’)); nil
=> nil

File.open(“pub2.pem”,“w”) { |f| f.write private_key.public_key }
=> 251

$ cat pub.pem
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCx5oz9tweJf4OZaM3y/0JRUeS3
Ctyy7zuPmrf2D/EK6GO4a+8OIG/Q++XskSIrqFUXAMjVDfM4zrGAwdzVRaKBo5an
2YskZFfT5YizOxyzyxjQ7+Z7kgNH7O7KEXCXOdSa8Bg3qQp40ChVI4kpdWTYlmS5
YY7lWqRLTAYYdVkVAQIDAQAB
-----END PUBLIC KEY-----

brian@zino:~$ cat pub2.pem
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

Now, I see that openssl is happy with pub.pem:

$ openssl rsa -in pub.pem -pubin -noout -text

but fails to read pub2.pem (even if you change “RSA PUBLIC KEY” to
“PUBLIC KEY”)
$ openssl rsa -in pub2.pem -pubin -noout -text

Ruby can read it back though (although interestingly, ruby 1.8.6 under
Ubuntu Hardy cannot)

public_key = OpenSSL::PKey::RSA.new(File.read(“pub2.pem”))
=> -----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

But I think this is probably a side-issue of outputting public keys.

The real question is, what algorithm are you trying to implement in
ruby? Are you trying to do something along the lines of “openssl rsautl
-sign” ? In that case, you should use the OpenSSL API for that, as it
deals with all the padding correctly.

pk = OpenSSL::PKey::RSA.new(File.read(“priv.pem”))
…

File.open(“enc1.dat”,“wb”) { |f| f.write pk.sign(“sha1”, “hello world”) }
=> 128

This file is quite happily read by openssl command line:

$ openssl rsautl -in enc1.dat -verify -inkey priv.pem -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 21 30
…0!0
0060 - 09 06 05 2b 0e 03 02 1a-05 00 04 14 2a ae 6c 35
…+…*.l5
0070 - c9 4f cf b4 15 db e9 5f-40 8b 9c e9 1e e8 46 ed
.O…_@…F.

And you can see that it includes the expected digest in the last 20
bytes:

$ echo -n “hello world” | openssl dgst -sha1 -hex
2aae6c35c94fcfb415dbe95f408b9ce91ee846ed

juuser · September 1, 2010, 3:36pm

As soon as i started using the #sign method with OpenSSL::Digest::SHA1
everything started to work

Excellent news

To be honest, I had to dig into the C source code to find out what the
arguments were to RSA#sign (Ruby told me it needed 2 arguments, but not
what they were)

But in general, the rule for getting Ruby and OpenSSL to work is:

Make it work using the OpenSSL command line tools
Translate those commands into the equivalent Ruby OpenSSL API calls

Regards,

Brian.

juuser · September 1, 2010, 3:13pm

Thank you for these great answers, Brian!

In Java the name of the algorithm was “SHA1withRSA” and i tried to do it

RSA(SHA1). Unfortunately i totally forgout about the padding issue and
Ruby OpenSSL documentation is just not there. I still couldn’t find a
documentation for this #sign method and that’s why i didn’t use it.

As soon as i started using the #sign method with OpenSSL::Digest::SHA1
everything started to work So, if someone is struggling with the same
problem then don’t try to use Digest::SHA1 and #private_encrypt
directly, but use #sign instead like this:

private_key.sign(OpenSSL::Digest::SHA1.new, “data”)

I wish that the documentation for some stdlibs would be better.

Thank You again, Brian!

Jarmo

Brian C. wrote:

Here are some tests on ubuntu Lucid with ruby 1.8.7

$ openssl genrsa -out priv.pem 1024
Generating RSA private key, 1024 bit long modulus
…++++++
…++++++
e is 65537 (0x10001)

$ openssl rsa -in priv.pem -out pub.pem -pubout
writing RSA key

$ irb --simple-prompt

require ‘openssl’
=> true

private_key = OpenSSL::PKey::RSA.new(File.read(‘priv.pem’)); nil
=> nil

File.open(“pub2.pem”,“w”) { |f| f.write private_key.public_key }
=> 251

$ cat pub.pem
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCx5oz9tweJf4OZaM3y/0JRUeS3
Ctyy7zuPmrf2D/EK6GO4a+8OIG/Q++XskSIrqFUXAMjVDfM4zrGAwdzVRaKBo5an
2YskZFfT5YizOxyzyxjQ7+Z7kgNH7O7KEXCXOdSa8Bg3qQp40ChVI4kpdWTYlmS5
YY7lWqRLTAYYdVkVAQIDAQAB
-----END PUBLIC KEY-----

brian@zino:~$ cat pub2.pem
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

Now, I see that openssl is happy with pub.pem:

$ openssl rsa -in pub.pem -pubin -noout -text

but fails to read pub2.pem (even if you change “RSA PUBLIC KEY” to
“PUBLIC KEY”)
$ openssl rsa -in pub2.pem -pubin -noout -text

Ruby can read it back though (although interestingly, ruby 1.8.6 under
Ubuntu Hardy cannot)

public_key = OpenSSL::PKey::RSA.new(File.read(“pub2.pem”))
=> -----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

But I think this is probably a side-issue of outputting public keys.

The real question is, what algorithm are you trying to implement in
ruby? Are you trying to do something along the lines of “openssl rsautl
-sign” ? In that case, you should use the OpenSSL API for that, as it
deals with all the padding correctly.

pk = OpenSSL::PKey::RSA.new(File.read(“priv.pem”))
…

File.open(“enc1.dat”,“wb”) { |f| f.write pk.sign(“sha1”, “hello world”) }
=> 128

This file is quite happily read by openssl command line:

$ openssl rsautl -in enc1.dat -verify -inkey priv.pem -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff
…
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 21 30
…0!0
0060 - 09 06 05 2b 0e 03 02 1a-05 00 04 14 2a ae 6c 35
…+…*.l5
0070 - c9 4f cf b4 15 db e9 5f-40 8b 9c e9 1e e8 46 ed
.O…_@…F.

And you can see that it includes the expected digest in the last 20
bytes:

$ echo -n “hello world” | openssl dgst -sha1 -hex
2aae6c35c94fcfb415dbe95f408b9ce91ee846ed

juuser · September 1, 2010, 4:03pm

Jarmo P. wrote:

private_key.sign(OpenSSL::Digest::SHA1.new, “data”)

You can also pass a string for the digest type (at least in 1.8.7p249)

private_key.sign(“sha1”, “data”)

juuser · September 1, 2010, 4:14pm

Not in 1.8.6p398.

Jarmo

Brian C. wrote:

Jarmo P. wrote:

private_key.sign(OpenSSL::Digest::SHA1.new, “data”)

You can also pass a string for the digest type (at least in 1.8.7p249)

private_key.sign(“sha1”, “data”)

juuser · September 1, 2010, 3:56pm

I agree that Ruby openssl’s documentation is abysmal.
If I find the time I’ll try to do something about that, I’ve had to
piece it together reading code for an earlier project.