Forum: Ruby rsa sign and verify with openssl on windows

Hello!

I've spent quite amount of time now to find out why can't i verify at
server side (written in Java) a signature done in Ruby.

Here's some of the code from Ruby side:
...
require 'openssl'
require 'base64'

private_key = OpenSSL::PKey::RSA.new(File.read('priv.pem'))
digest = Digest::SHA1.digest(data)
signed = Base64.encode64(private_key.private_encrypt(digest))
parameters.merge! 'my_data' => signed.gsub(/[\r\n]/, '')
Net::HTTP.post_form(URI.parse(return_url), parameters).body
...

And at the Java side verification fails. It works with another real
services so the problem cannot be at Java side.

After some inspection, i've extracted public key from priv.pem with
openssl in PEM format and tried private_key.public_key() which also
returns a public key in PEM format, but it is different!

How is it possible that using Ruby's OpenSSL::PKey::RSA#public_key
returns different public key than using openssl on command line?

I suspect that to be a culprit of the problem, but i'm not sure. I don't
even understand how can that happen...

I also tried to sign the same data with the same private key at Java
side and got a different base64 output than Ruby...

I also tried to sign and verify with the same keys using openssl command
line tool and were successful.

Data has only ASCII characters although let's not forget that in Java
UTF-8 is default. Could there be any encoding issues?

Anyway, any help or suggestion is welcome.

Best Regards,
Jarmo

I suggest you generate a new throw-away private key, then you can post 
the private key, complete programs in Ruby and Java and the openssl 
command line version to replicate the problem, and the exact outputs you 
see.

Also break it down bit by bit.

p data.bytesize  # same as in Java?
p digest         # should be 20 bytes, is it same as you get in Java?
p private_key.private_encrypt(digest)

> After some inspection, i've extracted public key from priv.pem with
> openssl in PEM format and tried private_key.public_key() which also
> returns a public key in PEM format, but it is different!

Again, would like to see this replicated.

On those two PEM files, try somethine like

openssl rsa -in foo.pem -pubin -noout -text

to see how/if the contents are different.

> I also tried to sign and verify with the same keys using openssl command
> line tool and were successful.

So show the openssl command lines you used, then we can see if what 
you're doing in Ruby is different.

> Data has only ASCII characters although let's not forget that in Java
> UTF-8 is default. Could there be any encoding issues?

Seems unlikely. Data is data, and Ruby's openssl wrapper isn't going to 
be transcoding stuff before giving it to openssl (I'd certainly hope not 
anyway)

Here are some tests on ubuntu Lucid with ruby 1.8.7

$ openssl genrsa -out priv.pem 1024
Generating RSA private key, 1024 bit long modulus
....++++++
...................++++++
e is 65537 (0x10001)

$ openssl rsa -in priv.pem -out pub.pem -pubout
writing RSA key

$ irb --simple-prompt
>> require 'openssl'
=> true
>> private_key = OpenSSL::PKey::RSA.new(File.read('priv.pem')); nil
=> nil
>> File.open("pub2.pem","w") { |f| f.write private_key.public_key }
=> 251

$ cat pub.pem
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCx5oz9tweJf4OZaM3y/0JRUeS3
Ctyy7zuPmrf2D/EK6GO4a+8OIG/Q++XskSIrqFUXAMjVDfM4zrGAwdzVRaKBo5an
2YskZFfT5YizOxyzyxjQ7+Z7kgNH7O7KEXCXOdSa8Bg3qQp40ChVI4kpdWTYlmS5
YY7lWqRLTAYYdVkVAQIDAQAB
-----END PUBLIC KEY-----

brian@zino:~$ cat pub2.pem
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

Now, I see that openssl is happy with pub.pem:

$ openssl rsa -in pub.pem -pubin -noout -text

but fails to read pub2.pem (even if you change "RSA PUBLIC KEY" to 
"PUBLIC KEY")
$ openssl rsa -in pub2.pem -pubin -noout -text

Ruby can read it back though (although interestingly, ruby 1.8.6 under 
Ubuntu Hardy cannot)

>> public_key = OpenSSL::PKey::RSA.new(File.read("pub2.pem"))
=> -----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
-----END RSA PUBLIC KEY-----

But I think this is probably a side-issue of outputting public keys.

The real question is, what algorithm are you trying to implement in 
ruby? Are you trying to do something along the lines of "openssl rsautl 
-sign" ? In that case, you should use the OpenSSL API for that, as it 
deals with all the padding correctly.

>> pk = OpenSSL::PKey::RSA.new(File.read("priv.pem"))
...
>> File.open("enc1.dat","wb") { |f| f.write pk.sign("sha1", "hello world") }
=> 128

This file is quite happily read by openssl command line:

$ openssl rsautl -in enc1.dat -verify -inkey priv.pem -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 21 30 
.............0!0
0060 - 09 06 05 2b 0e 03 02 1a-05 00 04 14 2a ae 6c 35 
...+........*.l5
0070 - c9 4f cf b4 15 db e9 5f-40 8b 9c e9 1e e8 46 ed 
.O....._@.....F.

And you can see that it includes the expected digest in the last 20 
bytes:

$ echo -n "hello world" | openssl dgst -sha1 -hex
2aae6c35c94fcfb415dbe95f408b9ce91ee846ed

Thank you for these great answers, Brian!

In Java the name of the algorithm was "SHA1withRSA" and i tried to do it 
- RSA(SHA1). Unfortunately i totally forgout about the padding issue and 
Ruby OpenSSL documentation is just not there. I still couldn't find a 
documentation for this #sign method and that's why i didn't use it.

As soon as i started using the #sign method with OpenSSL::Digest::SHA1 
everything started to work :) So, if someone is struggling with the same 
problem then don't try to use Digest::SHA1 and #private_encrypt 
directly, but use #sign instead like this:

private_key.sign(OpenSSL::Digest::SHA1.new, "data")

I wish that the documentation for some stdlibs would be better.

Thank You again, Brian!

Jarmo

Brian Candler wrote:
> Here are some tests on ubuntu Lucid with ruby 1.8.7
> 
> $ openssl genrsa -out priv.pem 1024
> Generating RSA private key, 1024 bit long modulus
> ....++++++
> ...................++++++
> e is 65537 (0x10001)
> 
> $ openssl rsa -in priv.pem -out pub.pem -pubout
> writing RSA key
> 
> $ irb --simple-prompt
>>> require 'openssl'
> => true
>>> private_key = OpenSSL::PKey::RSA.new(File.read('priv.pem')); nil
> => nil
>>> File.open("pub2.pem","w") { |f| f.write private_key.public_key }
> => 251
> 
> $ cat pub.pem
> -----BEGIN PUBLIC KEY-----
> MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCx5oz9tweJf4OZaM3y/0JRUeS3
> Ctyy7zuPmrf2D/EK6GO4a+8OIG/Q++XskSIrqFUXAMjVDfM4zrGAwdzVRaKBo5an
> 2YskZFfT5YizOxyzyxjQ7+Z7kgNH7O7KEXCXOdSa8Bg3qQp40ChVI4kpdWTYlmS5
> YY7lWqRLTAYYdVkVAQIDAQAB
> -----END PUBLIC KEY-----
> 
> brian@zino:~$ cat pub2.pem
> -----BEGIN RSA PUBLIC KEY-----
> MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
> 5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
> 7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
> -----END RSA PUBLIC KEY-----
> 
> Now, I see that openssl is happy with pub.pem:
> 
> $ openssl rsa -in pub.pem -pubin -noout -text
> 
> but fails to read pub2.pem (even if you change "RSA PUBLIC KEY" to 
> "PUBLIC KEY")
> $ openssl rsa -in pub2.pem -pubin -noout -text
> 
> Ruby can read it back though (although interestingly, ruby 1.8.6 under 
> Ubuntu Hardy cannot)
> 
>>> public_key = OpenSSL::PKey::RSA.new(File.read("pub2.pem"))
> => -----BEGIN RSA PUBLIC KEY-----
> MIGJAoGBALHmjP23B4l/g5lozfL/QlFR5LcK3LLvO4+at/YP8QroY7hr7w4gb9D7
> 5eyRIiuoVRcAyNUN8zjOsYDB3NVFooGjlqfZiyRkV9PliLM7HLPLGNDv5nuSA0fs
> 7soRcJc51JrwGDepCnjQKFUjiSl1ZNiWZLlhjuVapEtMBhh1WRUBAgMBAAE=
> -----END RSA PUBLIC KEY-----
> 
> But I think this is probably a side-issue of outputting public keys.
> 
> The real question is, what algorithm are you trying to implement in 
> ruby? Are you trying to do something along the lines of "openssl rsautl 
> -sign" ? In that case, you should use the OpenSSL API for that, as it 
> deals with all the padding correctly.
> 
>>> pk = OpenSSL::PKey::RSA.new(File.read("priv.pem"))
> ...
>>> File.open("enc1.dat","wb") { |f| f.write pk.sign("sha1", "hello world") }
> => 128
> 
> This file is quite happily read by openssl command line:
> 
> $ openssl rsautl -in enc1.dat -verify -inkey priv.pem -raw -hexdump
> 0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
> ................
> 0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
> ................
> 0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
> ................
> 0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
> ................
> 0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff 
> ................
> 0050 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 21 30 
> .............0!0
> 0060 - 09 06 05 2b 0e 03 02 1a-05 00 04 14 2a ae 6c 35 
> ...+........*.l5
> 0070 - c9 4f cf b4 15 db e9 5f-40 8b 9c e9 1e e8 46 ed 
> .O....._@.....F.
> 
> And you can see that it includes the expected digest in the last 20 
> bytes:
> 
> $ echo -n "hello world" | openssl dgst -sha1 -hex
> 2aae6c35c94fcfb415dbe95f408b9ce91ee846ed

> As soon as i started using the #sign method with OpenSSL::Digest::SHA1 
> everything started to work :)

Excellent news :-)

To be honest, I had to dig into the C source code to find out what the 
arguments were to RSA#sign (Ruby told me it needed 2 arguments, but not 
what they were)

But in general, the rule for getting Ruby and OpenSSL to work is:

1. Make it work using the OpenSSL command line tools
2. Translate those commands into the equivalent Ruby OpenSSL API calls

Regards,

Brian.

I agree that Ruby openssl's documentation is abysmal.
If I find the time I'll try to do something about that, I've had to
piece it together reading code for an earlier project.

Jarmo Pertman wrote:
> private_key.sign(OpenSSL::Digest::SHA1.new, "data")

You can also pass a string for the digest type (at least in 1.8.7p249)

private_key.sign("sha1", "data")

Not in 1.8.6p398.

Jarmo

Brian Candler wrote:
> Jarmo Pertman wrote:
>> private_key.sign(OpenSSL::Digest::SHA1.new, "data")
> 
> You can also pass a string for the digest type (at least in 1.8.7p249)
> 
> private_key.sign("sha1", "data")