Ruby vs JRuby Performance

Ruby
1

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve “hard”
problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

BTW, there is no GUI in my program. I hope to learn the GUI part with
JRuby.
That’s the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE,
Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

2

Victor R. schrieb:
[…]

jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does
not mean its overall performance is bad. I suggest you modify the
program so it runs for example 10 times and then run again. I am sure
the JRuby program will be very much below 10s

bye Jochen

3

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

Try to run it with jruby -J-server …

4

Victor R. wrote:

ruby -v
Elapsed Time: 1.058 Secs*

Send the source to Charlie Nutter … he is always looking for jRuby
performance test cases.

5

Victor R. wrote:

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve “hard”
problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:
Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

Am I in the ball-park?

According to the metrics posted here, JRuby runs about twice as fast,
takes longer to load and uses a lot more memory. Here is a little
something on the metrics:

http://www.ruby-forum.com/topic/129990#new
http://www.ruby-forum.com/topic/129996#new

6

I executed the program using jruby -J-server as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.

I didn’t realize how small the numbers already are. Have you checked
the result of

time ruby -v
time jruby -v

just to get a feeling for how much time is spent in startup alone?

7

Ruby vs JRuby Performance
http://www.ruby-forum.com/topic/155783#686221
Posted by Victor R. (Guest)
on 09.06.2008 15:41
[image: (Received via mailing list)]

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve “hard”

problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

BTW, there is no GUI in my program. I hope to learn the GUI part with

JRuby.
That’s the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE,
Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

Reply with
quotehttp://www.ruby-forum.com/topic/155783?reply_to=686221#postform
*Re: Ruby vs JRuby
Performancehttp://www.ruby-forum.com/topic/155783#686235
*
Posted by Jochen T. (Guest)
on 09.06.2008 16:15
[image: (Received via mailing list)]

Victor R. schrieb:
[…]

jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does

not mean its overall performance is bad. I suggest you modify the
program so it runs for example 10 times and then run again. I am sure
the JRuby program will be very much below 10s

bye Jochen

Reply with
quotehttp://www.ruby-forum.com/topic/155783?reply_to=686235#postform
*Re: Ruby vs JRuby
Performancehttp://www.ruby-forum.com/topic/155783#686241
*
Posted by M. Edward (Ed) Borasky (Guest)
on 09.06.2008 16:28
[image: (Received via mailing list)]

Victor R. wrote:

ruby -v
Elapsed Time: 1.058 Secs*

Send the source to Charlie Nutter … he is always looking for jRuby
performance test cases.

Reply with
quotehttp://www.ruby-forum.com/topic/155783?reply_to=686241#postform
*Re: Ruby vs JRuby
Performancehttp://www.ruby-forum.com/topic/155783#686245
*
Posted by ThoML (Guest)
on 09.06.2008 16:31

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

Try to run it with jruby -J-server …

Reply with
quotehttp://www.ruby-forum.com/topic/155783?reply_to=686245#postform
*Re: Ruby vs JRuby
Performancehttp://www.ruby-forum.com/topic/155783#686249
*
Posted by Lloyd L. (lloyd
http://www.ruby-forum.com/user/show/6864)

on 09.06.2008 16:44

Victor R. wrote:

The results are in:
Am I in the ball-park?
According to the metrics posted here, JRuby runs about twice as fast,
takes longer to load and uses a lot more memory. Here is a little
something on the metrics:

http://www.ruby-forum.com/topic/129990#new
http://www.ruby-forum.com/topic/129996#new

======================================================================================

Team,

First, thank you for all your comments.

As suggested by JoChen, I executed the program under JRuby 14 times.

To be fair to Ruby, I also executed it 14 times under Ruby.
You can see the results below.

As suggested by Ed Borasky, I will send the code to Charlie Nutter, if
I can get his email. Otherwise I can post it here. It is not great

code at all, but it is doing the work for now.

I executed the program using jruby -J-server as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.
I also can’t believe that jruby has been clocked at twice the speed of
ruby since after all, jruby has more overhead than ruby. Then again,

I am not expert on the subject (or any subject).

*JRuby executions:
*Elapsed Time: 0.888
Elapsed Time: 0.881
Elapsed Time: 0.977
Elapsed Time: 0.888
Elapsed Time: 0.872

Elapsed Time: 0.884
Elapsed Time: 0.869
Elapsed Time: 0.892
Elapsed Time: 0.876
Elapsed Time: 0.872
Elapsed Time: 0.882
Elapsed Time: 0.882
Elapsed Time: 0.867
Elapsed Time: 0.881
Elapsed Time: 0.906

Elapsed Time: 0.881

Ruby executions:
Elapsed Time: 0.422
Elapsed Time: 0.391
Elapsed Time: 0.407
Elapsed Time: 0.406
Elapsed Time: 0.407

Elapsed Time: 0.422
Elapsed Time: 0.406
Elapsed Time: 0.438
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.39
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.422

Elapsed Time: 0.39

jruby -J-server sudoku01final.rb
Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

Thank you

Victor

8

Jochen T. wrote:

well, looking at these numbers I would suggest you subscribe to the
jruby mailing list and ask why jruby is 100% slower than ruby :wink: The
server mode really looks awful… They need to know about this!

You are here trolling on the Ruby lists Jochen? Don’t you have some
Groovy bugs to be fixing? :slight_smile:

The server VM starts up a lot slower on most systems, which is the
reason for this test to be invalid. And in general, any benchmark under
5-10s is going to be a pretty poor test of JRuby performance, since the
code might not even get compiled or optimized on such a short run.

  • Charlie
9

Victor R. wrote:

jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

Far too short.

~/amsterdam âž” time ruby -e “puts 1”
1

real 0m0.006s
user 0m0.003s
sys 0m0.004s

~/amsterdam âž” time jruby -e “puts 1”
1

real 0m0.594s
user 0m0.460s
sys 0m0.101s

Is JRuby 100 times slower than Ruby? No. It is too short a test. Run
something longer and you’ll get more realistic results.

  • Charlie
10

Victor R. schrieb:
[…]

I executed the program using jruby -J-server as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.
I also can’t believe that jruby has been clocked at twice the speed of
ruby since after all, jruby has more overhead than ruby. Then again,

I am not expert on the subject (or any subject).

*JRuby executions:
*Elapsed Time: 0.888
Elapsed Time: 0.881
[…]
Elapsed Time: 0.881

Ruby executions:
Elapsed Time: 0.422
Elapsed Time: 0.391
[…]
Elapsed Time: 0.39

jruby -J-server sudoku01final.rb
Elapsed Time: 2.474
[…]
Elapsed Time: 2.414

well, looking at these numbers I would suggest you subscribe to the
jruby mailing list and ask why jruby is 100% slower than ruby :wink: The
server mode really looks awful… They need to know about this!

bye Jochen

11

Charles Oliver N. schrieb:

Jochen T. wrote:

well, looking at these numbers I would suggest you subscribe to the
jruby mailing list and ask why jruby is 100% slower than ruby :wink: The
server mode really looks awful… They need to know about this!

You are here trolling on the Ruby lists Jochen? Don’t you have some
Groovy bugs to be fixing? :slight_smile:

ah, I didn’t try to troll, I put so many :wink: in.

The server VM starts up a lot slower on most systems, which is the
reason for this test to be invalid. And in general, any benchmark under
5-10s is going to be a pretty poor test of JRuby performance, since the
code might not even get compiled or optimized on such a short run.

well, ok, but it was:

jruby -J-server sudoku01final.rb
Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

that’s >12s… unless he did start over again and again, but I assumed
that is not the case because of:

JRuby executions: Elapsed Time: 0.888 […]

vs.

JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs

but maybe I was wrong.

bye Jochen

12

On Tue, Jun 10, 2008 at 3:15 PM, Victor R. [email protected]
wrote:

Elapsed Time: 1.058 Secs*
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: jruby sudoku01final.rb multiple
times. Then I did the same for ruby: ruby sudoku01final.rb.
Although the code is not great code, that does not really matter since the
same “bad” or “good” code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

Just put a loop in your Ruby program:

12.times do

…main code

end

by restarting the JVM each time, you are not seeing any of JRuby’s
benefits.

Les

13

On Tue, Jun 10, 2008 at 8:19 AM, Jochen T. [email protected]
wrote:

bugs to be fixing? :slight_smile:
well, ok, but it was:
that’s >12s… unless he did start over again and again, but I assumed
but maybe I was wrong.

bye Jochen

Team,

I’ll be more than happy to send the code to someone or just put it here
as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: jruby sudoku01final.rb
multiple
times. Then I did the same for ruby: ruby sudoku01final.rb.
Although the code is not great code, that does not really matter since
the
same “bad” or “good” code was executed unmodified under both
environments.
Let me know if anyone wants to see and run the code for testing!

Thank you

Victor

14

The JVM is built for long running processes, optimizing things over time
(jruby takes advantage of this), MRI is going to remain the same speed
over
time. (for the most part, see below)

here is a quick example… first the somewhat silly example code:
#######################################
require ‘benchmark’
num = ARGV[0] || 10

puts “Running #{num} times”

puts Benchmark.measure {
num.to_i.times do
234 + 89239423 + 23482903 + 34892389423 + 234823
end
}.total
#######################################
now run it 10 times each: ( ruby/jruby speed.rb 10)
ruby: 0.0
jruby: 0.0519

10,000 times:
ruby:0.03
jruby:0.105

100,000 times:
ruby: 0.26
jruby: 0.217

1,000,000 times:
ruby: 2.61
jruby: 1.303

now with some additional optimization for the jvm: jruby -J-server
speed.rb
1000000
jruby: 0.845

This is obviously a silly little test, but can possibly shed light on
the
start-up cost diffs between JRuby and MRI… and where the benefit lies.
So
who is faster? MRI? (yes) JRuby? (yes) just depends on what problem
you
are trying to solve.

On Tue, Jun 10, 2008 at 9:24 AM, Leslie V.
[email protected]
wrote:

Les

–
Aloha!

Mike McKinney
(http://blog.huikau.com)

15

Victor R. wrote:

BTW, there is no GUI in my program. I hope to learn the GUI part with JRuby.
That’s the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, Etc.
I guess there is trade-off which one must be willing to accept.

For what it’s worth, 1.1.3 should include an update to the interpreter
that will help short-running apps a bit. But even then runs shorter than
5-10s aren’t going to really hit their stride as far as JRuby goes.

  • Charlie
16

Victor R. wrote:

I knew that there was a penalty to be paid when running JRuby, but I did not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve “hard” problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

Ugh, and I keep forgetting…yes, we’d love to see the code.

  • Charlie
17

Victor R. schrieb:
[…]

I’ll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: jruby sudoku01final.rb multiple
times.

ok, that was not what I intended. I did mean, what others also said, a
loop inside the program.

bye Jochen

18

Victor R. wrote:

I’ll be more than happy to send the code to someone or just put it here as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: jruby sudoku01final.rb multiple
times. Then I did the same for ruby: ruby sudoku01final.rb.
Although the code is not great code, that does not really matter since the
same “bad” or “good” code was executed unmodified under both environments.
Let me know if anyone wants to see and run the code for testing!

I’d love to give it a try. If it does turn out to be slower, it will be
fun to investigate and figure out why.

  • Charlie
19

On Fri, Jun 20, 2008 at 8:52 AM, Charles Oliver N. <
[email protected]> wrote:

I’d love to give it a try. If it does turn out to be slower, it will be fun
to investigate and figure out why.

  • Charlie

Charlie,

The code is attached to this thread.
However, I will send it to your email.
Also, remember that this is code written by a Rubist wannabe and not
necessary by one who knows the ins and outs of the language.
I’ll be interested in knowing what you find!

Thank you

Victor

20

Victor R. wrote:

The code is attached to this thread.
However, I will send it to your email.
Also, remember that this is code written by a Rubist wannabe and not
necessary by one who knows the ins and outs of the language.
I’ll be interested in knowing what you find!

Ok, I managed to get it to run a few times in a cycle. First here’s
MRI’s time for me:

Elapsed time: around 0.191s

Now the time running “jruby sudoku01final.rb”

(all using JRuby trunk, “soylatte” Java 6 on OS X)

Elapsed time: anywhere from 0.491s to 0.525s

If I specify --server, it degrades on this short run:

Elapsed time: 1.6s to 1.65s

However, trying both runs ten times produces a very different result:

Ten times without --server:
Elapsed time (final result): 0.105s

Ten times with --server:
Elapsed time (final result): 0.212s

So there’s some interesting slowdown for this short benchmark running
under server. Let’s try 100 iterations:

100 times without --server:
Elapsed time (final result): 0.103

100 times with --server:
Elapsed time (final result): 0.065s

It seems like the run is just too short for the server VM to really do
much optimization, which is why we have JRuby run with the client VM by
default.

I’ve attached the modified source.

  • Charlie