Forum: JRuby Follow-up JRuby 1.7.0 performance question

This was prompted by reading the Topic headed "jruby 1.7.0 with
invokedynamic.all=true on HotSpot not faster than 1.6.8?"

Am I correct in thinking that the "speedup" parts of the JVM (including
invokeDynamic) only makes a difference for a loop of code that is
repeated so many times that it takes 2 minutes to run?

There can't be too many cases like that about. I have one program that
simulates charging and discharging lead-acid batteries and the whole
thing only takes about a minute to iterate over 6 batteries at simulated
ten minute intervals for 3 years.

Most of my programs have methods that run once and present a new screen
to the user e.g. downloading and displaying weather data.

Does this mean that my programs would see no significant performance
improvement with 1.7.x compared with 1.6.8?

...R

In broad strokes, if your program has no 'hot' code - e.g. methods
that are called many, many times, then a JIT has no effect on your
code.

When benchmarking, we generally try to run loops for a long time (e.g.
100 million times or more), and repeat them, to ensure that the code
has been compiled, and to remove any transient system load effects.

If your code is running hard for 1 minute (and not sleeping or doing
I/O), then there is a good possibility it will benefit from some
speedup.  On my ancient macbook, I get in the order of 30 million
function invocations/second, code running for a minute could have 1.8
billion+ method calls - so there's a good chance a lot of those will
be optimizable.

Invoke dynamic is used for more than method calls, so even things like
constant and global variable reads will be improved with 1.7.x.

TL;DR - try it and see.  Theorising on performance never works.
Actual results are what you need.

Thanks Wayne,
I get the impression from your reply that I am unlikely to see much of 
an improvement. I'm reminded of the adage "if you have to measure it, 
you haven't made a difference".

I am not interested to test my existing programs with different versions 
of JRuby - they perform well enough.

I am hoping to find the value of X in the statement "1.7.x runs X% 
faster than 1.6.8" as one of the factors in a decision about which 
version to use.

Before this I had the impression (from various bits I had read) that X 
would be a substantial number - but it looks I misunderstood.

...R

>Wayne Meissner wrote in post #1093941:
> In broad strokes, if your program has no 'hot' code - e.g. methods
> that are called many, many times, then a JIT has no effect on your
> code.
>
> When benchmarking, we generally try to run loops for a long time (e.g.
> 100 million times or more), and repeat them, to ensure that the code
> has been compiled, and to remove any transient system load effects.
>
> If your code is running hard for 1 minute (and not sleeping or doing
> I/O), then there is a good possibility it will benefit from some
> speedup.  On my ancient macbook, I get in the order of 30 million
> function invocations/second, code running for a minute could have 1.8
> billion+ method calls - so there's a good chance a lot of those will
> be optimizable.
>
> Invoke dynamic is used for more than method calls, so even things like
> constant and global variable reads will be improved with 1.7.x.
>
> TL;DR - try it and see.  Theorising on performance never works.
> Actual results are what you need.

Robin -

JIT (aka "hot spot") compiling is just one of several things that can 
affect performance factors in JRuby applications.

On other message threads in this forum, we've seen how changing a single 
line of code can potentially have a tremendous effect on program 
throughput.  More below...


On Jan 27, 2013, at 6:52 AM, Robin McKay <lists@ruby-forum.com> wrote:

> Thanks Wayne,
> I get the impression from your reply that I am unlikely to see much of
> an improvement. I'm reminded of the adage "if you have to measure it,
> you haven't made a difference".
>

There are lots of variables in play, many known only by those who are 
expert in JRuby and JVM internals.  And those variables have different 
degrees of effect depending on the runtime behavior of a program. That's 
why real world testing is important.


> I am not interested to test my existing programs with different versions
> of JRuby - they perform well enough.
>

If you're happy with your current performance, then fine, but do realize 
that by turning away from the issue you may be missing opportunities for 
improvement, possibly dramatic improvements.


> I am hoping to find the value of X in the statement "1.7.x runs X%
> faster than 1.6.8" as one of the factors in a decision about which
> version to use.

While it's our job to analyze, detect patterns, and make informed 
judgments based on them, we need to be careful not to oversimplify. 
While there may be a single value for X as a very coarse indicator, the 
real value you encounter may vary wildly depending on the runtime 
behavior of your program.

It seems that you are looking for shortcuts and generalizations. 
There's nothing wrong with that, it's our job to accomplish as much as 
possible in as little time as possible.  However, when it comes to 
JRuby/Java performance, shortcuts and generalizations are often 
unreliable.

- Keith

---
Keith R. Bennett
http://about.me/keithrbennett

Thanks Keith,

I understand perfectly what you say about performance varying widely
between cases. Nevertheless a generalized indication (perhaps a range)
would be very useful when each new version comes out (especially if
there are "dramatic improvements" to be had). You might be able to say
something like this:
"this changes in this version don't effect performance"
or
"in most cases this version should give a 5% performance improvement but
special attention has been given to XX and if you use that feature you
should see a 50% speed improvement in that part of your code"

Obviously another way of looking at things is if your program is
sluggish try it on a newer version to see if that improves things.

Interestingly I don't think I have ever seen advice about what things to
do or not to do in my JRuby code to minimize runtime.

And, of course, there may be many non-speed reasons to justify an 
upgrade.
...R

>Keith B. wrote in post #1093973:
> Robin -
>
> If you're happy with your current performance, then fine, but do realize
> that by turning away from the issue you may be missing opportunities for
> improvement, possibly dramatic improvements.

> While it's our job to analyze, detect patterns, and make informed
> judgments based on them, we need to be careful not to oversimplify.
> While there may be a single value for X as a very coarse indicator, the
> real value you encounter may vary wildly depending on the runtime
> behavior of your program.
>
> It seems that you are looking for shortcuts and generalizations.
> There's nothing wrong with that, it's our job to accomplish as much as
> possible in as little time as possible.  However, when it comes to
> JRuby/Java performance, shortcuts and generalizations are often
> unreliable.
>
> - Keith
>
> ---
> Keith R. Bennett
> http://about.me/keithrbennett

It sounds like, for you, the main advice for upgrading will be that
there will never be a JRuby 1.6.9.  If you run into a problem with
1.6.8 you will either have to upgrade or patch it yourself.  If your
code never changes then perhaps even this is not a factor, but most
people do change their code and run into the unexpected.  I typically
try and keep up so I am upgrading on my schedule (and terms) instead
of immediate need (e.g. hit a bug).

An interesting realization is that language runtimes are less prone to
upgrade rot than other programs because the behavior is so well
defined.  If you wait two major versions, then it may be zero pain to
upgrade.  This is rarely true with most software...

-Tom

On Mon, Jan 28, 2013 at 2:30 AM, Robin McKay <lists@ruby-forum.com> 
wrote:
> special attention has been given to XX and if you use that feature you
> ...R
>> While there may be a single value for X as a very coarse indicator, the
>>
>     http://xircles.codehaus.org/manage_email
>
>



--
blog: http://blog.enebo.com       twitter: tom_enebo
mail: tom.enebo@gmail.com

Great discussion!



Im going to try to bring my contribution...



Im implementing a multi-dimensional array class (MDArray), which by the
way, Im thinking to give to the community if there is any interest (more
on another thread) in line with Numpy.  The solution uses 
multi-dimensional
java array implemented by unidata Java NetCDF library. This library is
freely available and the source code is released under the (MIT-style)
netCDF C library
license<http://www.unidata.ucar.edu/software/netcdf/copyri....
Ive being doing some performance testing and would like to share the
results.  My machine: Intel Core i5-2400 CPU @ 3.10GHz. 4,00 GB Windows 
7
64 on top of cygwin.



The code bellow creates a new multi-dimensional array of type double 
with
4 dimensions with the sizes of 7, 500, 20, 320.  So, the total number of
elements in the array is: 7 x 500 x 20 x 320 = 22.400.000.  The
fromfunction block receives the dimensions and fills the given element 
with
the resulting value.  So in the example bellow @a[5, 10, 15, 20] = 5 + 
10 +
15 + 20 = 50.



@a = MDArray.fromfunction("double", [7, 500, 20, 320]) do |x, y, z, k|

        x + y + z + k

 end


The relevant ruby code called to execute this method is:


  def set_block(*args)

    get_args(*args) do |op_iterator, shape, *other_args|

      block = other_args[0]

      while (op_iterator.has_next?)

        op_iterator.next

        op_iterator.set_current(block.call(op_iterator.get_current_counter))

      end if block

    end

  end

         Get_args just parses the arguments to fromfunction.  In this 
case
the argument is the multi-dimension array @a and it calls the internal
block giving an iterator over @a  op_iterator, the shape of the array 
and
the remaining args if there are any;

         We then iterate over all elements of @a and set the current 
value
by calling the block with the current_counter value, e.g., [0, 0, 0, 0],
[0, 0, 0, 1], etc.

The methods set_current and get_current_counter are calls to the NetCDF
java methods and should be fairly fast.


Running this code with Jruby 1.6.8 as: /jruby-1.6.8/bin/jruby --server
-J-Djruby.compile.frameless=true -J-Djruby.compile.fastops=true 
-J-Xmn512m
-J-Xms1024m -J-Xmx1024m $1 *takes between 36 and 38 seconds*.


Running the same code with Jruby 1.7.2 as: /jruby-1.7.2/bin/jruby 
--server
-Xinvokedynamic.constants=true -J-Xmn512m -J-Xms1024m -J-Xmx1024m $1 
*takes
between 33 and 35 seconds*.


So, there is an improvement, but I wouldnt say it is very large and I
dont know if we can make any generalization.


35 seconds is actually a lot of time and in order to improve on this I
created java methods to execute the loop.


This is the Java method that does the same loop as the ruby method 
above:


    public static void setAll4(ArrayDouble array, D4 func) {

                IndexIterator iterator = array.getIndexIterator();

                int[] counter;

                while (iterator.hasNext()) {

                    iterator.next();

                    counter = iterator.getCurrentCounter();

                    iterator.setDoubleCurrent(func.call(counter[0],
counter[1], counter[2],


 counter[3]));

                }

    }


Unfortunately I had to create methods called setAll1, setAll2, setAll3
setAll7 for efficiency reasons as I dont think there is a way for Jruby 
to
finding the proper method.  In the example above, setAll4 will be called 
as
my array is 4 dimensions.


Now running this code with Jruby 1.6.8 with the same flags as before
executes in 4.23 seconds with very minor differences between runs.  So,
bringing the code to Java does actually make a huge difference.


Now, with Jruby 1.7.2  with invokedynamics it executes in 4.8 to 5.1
seconds.  So, actually it performs worst than 1.6.8.  Even changing the
flags to the same flags as in 1.6.8 does not improve performance.


So any comments and ideas why 1.7.2 is worst than 1.6.8 when the loop is 
in
Java?



Are there other interesting flags that should be used in order to 
improve
performance?


Thanks for all the comments and ideas....



Rodrigo

Thanks Tom,

I think you are off-Topic as you have not commented on speed 
differences:) I am well aware there are other issues such as elimination 
of bugs and security issues that justify upgrades. But my question was 
about speed - and I guess you know more about that than anyone else.

I have been using 1.7.x since it appeared, but I am now wondering if I 
really needed to change. I have myself organized so each project has its 
own copy of JRuby so versions and gems are completely independent. There 
is no need to upgrade a working application. But I might take the 
trouble to do so if there was a 30% speed increase!

This has really been prompted because I can get 1.6.8 to work on an 
Android device, but not 1.7.x and, in any case, 1.6.8 is considerably 
smaller.

Generally when a new version comes out the list of changes is pretty 
useless for deciding if the upgrade is valuable unless you have 
experienced one of the problems that it addresses and, as far as I know, 
there is never any advice about the performance improvement that has 
been achieved.

...R


>Thomas E Enebo wrote in post #1094092:
> It sounds like, for you, the main advice for upgrading will be that
> there will never be a JRuby 1.6.9.  If you run into a problem with
> 1.6.8 you will either have to upgrade or patch it yourself.  If your
> code never changes then perhaps even this is not a factor, but most
> people do change their code and run into the unexpected.  I typically
> try and keep up so I am upgrading on my schedule (and terms) instead
> of immediate need (e.g. hit a bug).
>
> An interesting realization is that language runtimes are less prone to
> upgrade rot than other programs because the behavior is so well
> defined.  If you wait two major versions, then it may be zero pain to
> upgrade.  This is rarely true with most software...
>
> -Tom
>
> On Mon, Jan 28, 2013 at 2:30 AM, Robin McKay <lists@ruby-forum.com>
> wrote:
>> special attention has been given to XX and if you use that feature you
>> ...R
>>> While there may be a single value for X as a very coarse indicator, the
>>>
>>     http://xircles.codehaus.org/manage_email
>>
>>
>
>
>
> --
> blog: http://blog.enebo.com       twitter: tom_enebo
> mail: tom.enebo@gmail.com

To be a little more on topic, performance in 1.7.x is a moving target.
 If we could claim a percentage it would make things simpler, but
unfortunately it is not that simple.

There are new improvements per point release like Charlie making
constant access nearly free for 1.7.2.

There is also changing performance numbers per 'u' release of the JVM
since the JVM engineers are actively working on invokedynamic (+ other
areas).  Invokedynamic gets special mention since they changes have
been substantial and it is still considered a new feature of the JVM.
Indy keeps improving but it is definitely changing each VM update.

The general conclusion to all 'is it worth it' threads is: try it and
see.  I know it takes work to try, but it is impossible to distill
this.  I have heard people talk about 4-6x improvement to people
saying things have gotten 10-15% slower than 1.6.8.

-Tom

On Mon, Jan 28, 2013 at 11:42 AM, Robin McKay <lists@ruby-forum.com> 
wrote:
> is no need to upgrade a working application. But I might take the
> been achieved.
>> try and keep up so I am upgrading on my schedule (and terms) instead
>> wrote:
>> --
>
>



--
blog: http://blog.enebo.com       twitter: tom_enebo
mail: tom.enebo@gmail.com