Forum: Ruby Ruby vs JRuby Performance

I knew that there was a penalty to be paid when running JRuby, but I did 
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard" 
problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

BTW, there is no GUI in my program. I hope to learn the GUI part with 
JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, 
Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

Victor Reyes schrieb:
[...]
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does
not mean its overall performance is bad. I suggest you modify the
program so it runs for example 10 times and then run again. I am sure
the JRuby program will be very much below 10s

bye Jochen

Victor Reyes wrote:
> ruby -v
> Elapsed Time: 1.058 Secs*
> 
Send the source to Charlie Nutter ... he is always looking for jRuby
performance test cases.

> jruby -v
> ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
> *JRuby:
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*

Try to run it with jruby -J-server ...

Victor Reyes wrote:
> I knew that there was a penalty to be paid when running JRuby, but I did 
> not
> know how high.
> I developed a Sudoku solver, which actually solves simple to medium
> difficult puzzles. I am still struggling to make it solve "hard" 
> problems.
> I just installed JRuby yesterday and wanted to compare it against Ruby.
> 
> I executed the same program:
> 
> The results are in:
> *Ruby:
> ruby sudoku01final.rb
> Elapsed Time: 0.437 Secs*
> 
> jruby -v
> ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
> *JRuby:
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*
> 
> Am I in the ball-park?

According to the metrics posted here, JRuby runs about twice as fast, 
takes longer to load and uses a lot more memory.  Here is a little 
something on the metrics:

http://www.ruby-forum.com/topic/129990#new
http://www.ruby-forum.com/topic/129996#new

*Ruby vs JRuby Performance 
<http://www.ruby-forum.com/topic/155783#686221>*
 Posted by Victor Reyes (Guest)
 on 09.06.2008 15:41
[image: (Received via mailing list)]

I knew that there was a penalty to be paid when running JRuby, but I did
not
know how high.
I developed a Sudoku solver, which actually solves simple to medium
difficult puzzles. I am still struggling to make it solve "hard"

problems.
I just installed JRuby yesterday and wanted to compare it against Ruby.

I executed the same program:

The results are in:

ruby -v
ruby 1.8.6 (2007-09-24 patchlevel 111) [i386-mswin32]

*Ruby:
ruby sudoku01final.rb
Elapsed Time: 0.437 Secs*

jruby -v
ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

BTW, there is no GUI in my program. I hope to learn the GUI part with

JRuby.
That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE,
Etc.
I guess there is trade-off which one must be willing to accept.

Am I in the ball-park?

Regards,

Victor

 Reply with 
quote<http://www.ruby-forum.com/topic/155783?reply_to=68...
  *Re: Ruby vs JRuby 
Performance<http://www.ruby-forum.com/topic/155783#686235>
*
 Posted by Jochen Theodorou (Guest)
 on 09.06.2008 16:15
[image: (Received via mailing list)]

Victor Reyes schrieb:
[...]
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*

jruby has certainly higher startup costs than native Ruby, but that does

not mean its overall performance is bad. I suggest you modify the
program so it runs for example 10 times and then run again. I am sure
the JRuby program will be very much below 10s

bye Jochen

 Reply with 
quote<http://www.ruby-forum.com/topic/155783?reply_to=68...
  *Re: Ruby vs JRuby 
Performance<http://www.ruby-forum.com/topic/155783#686241>
*
 Posted by M. Edward (Ed) Borasky (Guest)
 on 09.06.2008 16:28
[image: (Received via mailing list)]

Victor Reyes wrote:
> ruby -v
> Elapsed Time: 1.058 Secs*
>

Send the source to Charlie Nutter ... he is always looking for jRuby
performance test cases.

 Reply with 
quote<http://www.ruby-forum.com/topic/155783?reply_to=68...
  *Re: Ruby vs JRuby 
Performance<http://www.ruby-forum.com/topic/155783#686245>
*
 Posted by ThoML (Guest)
 on 09.06.2008 16:31

> jruby -v
> ruby 1.8.6 (2008-05-28 rev 6586) [x86-jruby1.1.2]
> *JRuby:
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*

Try to run it with jruby -J-server ...

 Reply with 
quote<http://www.ruby-forum.com/topic/155783?reply_to=68...
 *Re: Ruby vs JRuby 
Performance<http://www.ruby-forum.com/topic/155783#686249>
*
 Posted by Lloyd Linklater (lloyd 
<http://www.ruby-forum.com/user/show/6864>)

 on 09.06.2008 16:44

Victor Reyes wrote:
> The results are in:
> Am I in the ball-park?
According to the metrics posted here, JRuby runs about twice as fast,
takes longer to load and uses a lot more memory.  Here is a little
something on the metrics:

http://www.ruby-forum.com/topic/129990#new
http://www.ruby-forum.com/topic/129996#new


======================================================================================

Team,

First, thank you for all your comments.

As suggested by JoChen, I executed the program under JRuby 14 times.

To be fair to Ruby, I also executed it 14 times under Ruby.
You can see the results below.

As suggested by Ed Borasky, I will send the code to Charlie Nutter, if
I can get his email. Otherwise I can post it here. It is not great

code at all, but it is doing the work for now.

I executed the program using *jruby -J-server* as suggested by ThoML
and results are catastrophic! See the number belows! I only ran it 6
times.
I also can't believe that jruby has been clocked at twice the speed of
ruby since after all, jruby has more overhead than ruby. Then again,

I am not expert on the subject (or any subject).

*JRuby executions:
*Elapsed Time: 0.888
Elapsed Time: 0.881
Elapsed Time: 0.977
Elapsed Time: 0.888
Elapsed Time: 0.872

Elapsed Time: 0.884
Elapsed Time: 0.869
Elapsed Time: 0.892
Elapsed Time: 0.876
Elapsed Time: 0.872
Elapsed Time: 0.882
Elapsed Time: 0.882
Elapsed Time: 0.867
Elapsed Time: 0.881
Elapsed Time: 0.906

Elapsed Time: 0.881

*Ruby executions:*
Elapsed Time: 0.422
Elapsed Time: 0.391
Elapsed Time: 0.407
Elapsed Time: 0.406
Elapsed Time: 0.407

Elapsed Time: 0.422
Elapsed Time: 0.406
Elapsed Time: 0.438
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.39
Elapsed Time: 0.406
Elapsed Time: 0.406
Elapsed Time: 0.422

Elapsed Time: 0.39

*jruby -J-server sudoku01final.rb*
Elapsed Time: 2.474
Elapsed Time: 2.467
Elapsed Time: 2.347
Elapsed Time: 2.404
Elapsed Time: 2.326
Elapsed Time: 2.414

Thank you

Victor

> I executed the program using *jruby -J-server* as suggested by ThoML
> and results are catastrophic! See the number belows! I only ran it 6
> times.

I didn't realize how small the numbers already are. Have you checked
the result of

time ruby -v
time jruby -v

just to get a feeling for how much time is spent in startup alone?

Victor Reyes schrieb:
[...]
> I executed the program using *jruby -J-server* as suggested by ThoML
> and results are catastrophic! See the number belows! I only ran it 6
> times.
> I also can't believe that jruby has been clocked at twice the speed of
> ruby since after all, jruby has more overhead than ruby. Then again,
> 
> I am not expert on the subject (or any subject).
> 
> *JRuby executions:
> *Elapsed Time: 0.888
> Elapsed Time: 0.881
[...]
> Elapsed Time: 0.881
> 
> *Ruby executions:*
> Elapsed Time: 0.422
> Elapsed Time: 0.391
[...]
> Elapsed Time: 0.39
> 
> *jruby -J-server sudoku01final.rb*
> Elapsed Time: 2.474
[...]
> Elapsed Time: 2.414

well, looking at these numbers I would suggest you subscribe to the
jruby mailing list and ask why jruby is 100% slower than ruby ;) The
server mode really looks awful... They need to know about this!

bye Jochen

Jochen Theodorou wrote:
> well, looking at these numbers I would suggest you subscribe to the 
> jruby mailing list and ask why jruby is 100% slower than ruby ;) The 
> server mode really looks awful... They need to know about this!

You are here trolling on the Ruby lists Jochen? Don't you have some
Groovy bugs to be fixing? :)

The server VM starts up a lot slower on most systems, which is the
reason for this test to be invalid. And in general, any benchmark under
5-10s is going to be a pretty poor test of JRuby performance, since the
code might not even get compiled or optimized on such a short run.

- Charlie

Victor Reyes wrote:
> jruby sudoku01final.rb
> Elapsed Time: 1.058 Secs*

Far too short.

~/amsterdam âž” time ruby -e "puts 1"
1

real  0m0.006s
user  0m0.003s
sys  0m0.004s

~/amsterdam âž” time jruby -e "puts 1"
1

real  0m0.594s
user  0m0.460s
sys  0m0.101s

Is JRuby 100 times slower than Ruby? No. It is too short a test. Run
something longer and you'll get more realistic results.

- Charlie

Charles Oliver Nutter schrieb:
> Jochen Theodorou wrote:
>> well, looking at these numbers I would suggest you subscribe to the 
>> jruby mailing list and ask why jruby is 100% slower than ruby ;) The 
>> server mode really looks awful... They need to know about this!
> 
> You are here trolling on the Ruby lists Jochen? Don't you have some 
> Groovy bugs to be fixing? :)

ah, I didn't try to troll, I put so many ;) in.

> The server VM starts up a lot slower on most systems, which is the 
> reason for this test to be invalid. And in general, any benchmark under 
> 5-10s is going to be a pretty poor test of JRuby performance, since the 
> code might not even get compiled or optimized on such a short run.

well, ok, but it was:

> *jruby -J-server sudoku01final.rb*
> Elapsed Time: 2.474
> Elapsed Time: 2.467
> Elapsed Time: 2.347
> Elapsed Time: 2.404
> Elapsed Time: 2.326
> Elapsed Time: 2.414

that's  >12s... unless he did start over again and again, but I assumed
that is not the case because of:

*JRuby executions:* Elapsed Time: 0.888 [...]

vs.

*JRuby:
jruby sudoku01final.rb
Elapsed Time: 1.058 Secs*

but maybe I was wrong.

bye Jochen

On Tue, Jun 10, 2008 at 8:19 AM, Jochen Theodorou <blackdrag@uni.de> 
wrote:

>> bugs to be fixing? :)
> well, ok, but it was:
> that's  >12s... unless he did start over again and again, but I assumed
> but maybe I was wrong.
>
> bye Jochen
>
>
Team,

I'll be more than happy to send the code to someone or just put it here 
as
an attachment.
The way I executed the code was by opening a DOS (cmd) session on XP.
I typed the command as described above: *jruby sudoku01final.rb* 
multiple
times. Then I did the same for ruby:  *ruby sudoku01final.rb.*
Although the code is not great code, that does not really matter since 
the
same "bad" or "good" code was executed unmodified under both 
environments.
Let me know if anyone wants to see and run the code for testing!

Thank you

Victor

On Tue, Jun 10, 2008 at 3:15 PM, Victor Reyes <victor.reyes@gmail.com> 
wrote:
>>>
>>>
>>>
>> Elapsed Time: 1.058 Secs*
> The way I executed the code was by opening a DOS (cmd) session on XP.
> I typed the command as described above: *jruby sudoku01final.rb* multiple
> times. Then I did the same for ruby:  *ruby sudoku01final.rb.*
> Although the code is not great code, that does not really matter since the
> same "bad" or "good" code was executed unmodified under both environments.
> Let me know if anyone wants to see and run the code for testing!

Just put a loop in your Ruby program:

12.times do

...main code

end



by restarting the JVM each time, you are not seeing any of JRuby's 
benefits.

Les

The JVM is built for long running processes, optimizing things over time
(jruby takes advantage of this), MRI is going to remain the same speed 
over
time. (for the most part, see below)

here is a quick example... first the somewhat silly example code:
#######################################
  require 'benchmark'
  num = ARGV[0] || 10

  puts "Running #{num} times"

  puts Benchmark.measure {
    num.to_i.times do
      234 + 89239423 + 23482903 + 34892389423 + 234823
    end
  }.total
#######################################
now run it 10 times each: ( ruby/jruby speed.rb 10)
  ruby: 0.0
  jruby: 0.0519

10,000 times:
  ruby:0.03
  jruby:0.105

100,000 times:
  ruby: 0.26
  jruby: 0.217

1,000,000 times:
  ruby: 2.61
  jruby: 1.303

now with some additional optimization for the jvm: jruby -J-server 
speed.rb
1000000
  jruby: 0.845

This is obviously a silly little test, but can possibly shed light on 
the
start-up cost diffs between JRuby and MRI... and where the benefit lies. 
So
who is faster?  MRI? (yes) JRuby? (yes)  just depends on what problem 
you
are trying to solve.


On Tue, Jun 10, 2008 at 9:24 AM, Leslie Viljoen 
<leslieviljoen@gmail.com>
wrote:

>
> Les
>
>
--
Aloha!

Mike McKinney
(http://blog.huikau.com)

Victor Reyes wrote:
> BTW, there is no GUI in my program. I hope to learn the GUI part with JRuby.
> That's the main reason why I downloaded JRuby, NetBeans, Java JDK/JRE, Etc.
> I guess there is trade-off which one must be willing to accept.

For what it's worth, 1.1.3 should include an update to the interpreter
that will help short-running apps a bit. But even then runs shorter than
5-10s aren't going to really hit their stride as far as JRuby goes.

- Charlie

Victor Reyes wrote:
> I knew that there was a penalty to be paid when running JRuby, but I did not
> know how high.
> I developed a Sudoku solver, which actually solves simple to medium
> difficult puzzles. I am still struggling to make it solve "hard" problems.
> I just installed JRuby yesterday and wanted to compare it against Ruby.

Ugh, and I keep forgetting...yes, we'd love to see the code.

- Charlie

Victor Reyes schrieb:
[...]
> I'll be more than happy to send the code to someone or just put it here as
> an attachment.
> The way I executed the code was by opening a DOS (cmd) session on XP.
> I typed the command as described above: *jruby sudoku01final.rb* multiple
> times. 

ok, that was not what I intended. I did mean, what others also said, a
loop inside the program.

bye Jochen

Victor Reyes wrote:
> I'll be more than happy to send the code to someone or just put it here as
> an attachment.
> The way I executed the code was by opening a DOS (cmd) session on XP.
> I typed the command as described above: *jruby sudoku01final.rb* multiple
> times. Then I did the same for ruby:  *ruby sudoku01final.rb.*
> Although the code is not great code, that does not really matter since the
> same "bad" or "good" code was executed unmodified under both environments.
> Let me know if anyone wants to see and run the code for testing!

I'd love to give it a try. If it does turn out to be slower, it will be
fun to investigate and figure out why.

- Charlie

On Fri, Jun 20, 2008 at 8:52 AM, Charles Oliver Nutter <
charles.nutter@sun.com> wrote:

>>
>
> I'd love to give it a try. If it does turn out to be slower, it will be fun
> to investigate and figure out why.
>
> - Charlie
>
>
Charlie,

The code is attached to this thread.
However, I will send it to your email.
Also, remember that this is code written by a Rubist wannabe and not
necessary by one who knows the ins and outs of the language.
I'll be interested in knowing what you find!

Thank you

Victor

Victor Reyes wrote:
> The code is attached to this thread.
> However, I will send it to your email.
> Also, remember that this is code written by a Rubist wannabe and not
> necessary by one who knows the ins and outs of the language.
> I'll be interested in knowing what you find!

Ok, I managed to get it to run a few times in a cycle. First here's
MRI's time for me:

   Elapsed time: around 0.191s

Now the time running "jruby sudoku01final.rb"

(all using JRuby trunk, "soylatte" Java 6 on OS X)

   Elapsed time: anywhere from 0.491s to 0.525s

If I specify --server, it degrades on this short run:

   Elapsed time: 1.6s to 1.65s

However, trying both runs ten times produces a very different result:

   Ten times without --server:
   Elapsed time (final result): 0.105s

   Ten times with --server:
   Elapsed time (final result): 0.212s

So there's some interesting slowdown for this short benchmark running
under server. Let's try 100 iterations:

   100 times without --server:
   Elapsed time (final result): 0.103

   100 times with --server:
   Elapsed time (final result): 0.065s

It seems like the run is just too short for the server VM to really do
much optimization, which is why we have JRuby run with the client VM by
default.

I've attached the modified source.

- Charlie