I just ran the benchmark suite that comes with Ruby 1.9 on my 32-bit
machine (1.3 GHz Athlon Thunderbird). The two versions of Ruby are
1.8.6-p110 and Ruby 1.9.0 revision 13500 from the SVN repository. Both
were compiled “-march=athlon-tbird -O3” with gcc 4.2.0. Here are the
numbers – if you’ve got OpenOffice, I’ve attached the spreadsheet:
benchmark 1.8.6-p110 1.9.0-r13500 ratio log ratio
app_answer 2.66 0.2 13.49 2.6
app_factorial 1.8 1.37 1.32 0.28
app_fib 16.54 2.44 6.77 1.91
app_mandelbrot 5.55 2.36 2.35 0.86
app_pentomino 256.81 80.46 3.19 1.16
app_raise 3.03 4.11 0.74 -0.31
app_strconcat 2.82 1.92 1.47 0.38
app_tak 21.28 3.82 5.57 1.72
app_tarai 17.02 3.07 5.54 1.71
loop_generator 83.06 3.57 23.27 3.15
loop_times 15.55 4.93 3.16 1.15
loop_whileloop 26.56 2.73 9.74 2.28
loop_whileloop2 3.65 0.55 6.67 1.9
so_ackermann 48.52 3.19 15.21 2.72
so_array 15.95 5.51 2.89 1.06
so_concatenate 6.84 1.73 3.96 1.38
so_count_words 0.94 0.01 72.38 4.28
so_exception 6.47 5.77 1.12 0.11
so_lists 2.44 1.25 1.95 0.67
so_matrix 4.35 1.47 2.97 1.09
so_nested_loop 15.57 4.5 3.46 1.24
so_object 14.24 6.45 2.21 0.79
so_random 4.86 1.75 2.79 1.03
so_sieve 1.2 0.3 3.97 1.38
vm1_block 63.43 9.74 6.51 1.87
vm1_const 34.22 4.3 7.95 2.07
vm1_ensure 47.44 3.37 14.1 2.65
vm1_length 35.18 7.36 4.78 1.56
vm1_rescue 32.69 2.86 11.41 2.43
vm1_simplereturn 55.13 6.5 8.49 2.14
vm1_swap 42.12 5.03 8.38 2.13
vm2_array 10.96 5.05 2.17 0.77
vm2_case 8.36 1.15 7.28 1.98
vm2_method 37.23 7.79 4.78 1.56
vm2_mutex 14.45 5.39 2.68 0.99
vm2_poly_method 43.42 8.46 5.13 1.64
vm2_poly_method_ov 8.04 1.4 5.75 1.75
vm2_proc 14.49 2.84 5.11 1.63
vm2_regexp 7.87 5.6 1.4 0.34
vm2_send 9.55 1.56 6.11 1.81
vm2_super 12.08 2.41 5.01 1.61
vm2_unif1 9.06 1.54 5.87 1.77
vm2_zsuper 13.18 2.52 5.22 1.65
vm3_thread_create_join 0.03 0.26 0.1 -2.31
vm3_thread_mutex 281.97 1.91 147.86 5
Min 0.1 -2.31
P05 1.1 0.14
P10 1.4 0.35
P25 2.8 1.02
P50 5.1 1.62
P75 6.9 1.93
P90 12.8 2.55
P95 15.0 2.71
Max 72.4 4.28
Notes:
Column 1 is the benchmark name
Column 2 is the time for 1.8
Column 3 is the time for 1.9
Column 4 is the ratio of 1.8 time / 1.9 time
Column 5 is the log to the base e (natural log) of the ratio.
The statistics below are derived from these natural logs. Specifically,
Min is the worst case ratio from the test suite, Max is the best case
ratio from the test suite, and Pxx are the percentiles. What these mean
is that xx percent of the time, you can expect the ratio to be below
that value. So, since P50 is 5.1, half of the time you can expect the
ratio to be below 5.1 and half of the time you can expect it to be above
5.1. Only 25 percent of the time is the ratio below 2.8 percent, etc.
So … in a nutshell … 1.9.0 is five times as fast as 1.8.6!
I can do fancier stuff in R, but I probably won’t – 5x is easy to
remember.
M. Edward (Ed) Borasky wrote:
So … in a nutshell … 1.9.0 is five times as fast as 1.8.6!
I’d be careful about making that assertion. The benchmarks in Ruby 1.9
are specific areas that Koichi has optimized to make 1.9 run them
faster. There are other areas, especially as regards core classes and
evaluated code, where 1.9 is no faster or even slower.
Not trying to rain on the parade, but even Koichi has recommended not
taking the 1.9 benchmark numbers as indicative of overall performance.
Charles Oliver N. wrote:
Well … I’ll agree with that after looking at the codes. They’re
definitely micro-benchmarks and probably won’t stress the cache/RAM
interface on modern chips. I’m hoping to get a Rails benchmark running
in the next few days – it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don’t suppose you have a
Rails benchmark laying around in your repositories. 
Meanwhile, I ran them on the Athlon64 X2 and got slightly better results
– which scale by cycle time as well. Unfortunately CodeAnalyst won’t
tell me much on the T-Bird – the chip is too ancient to interact
correctly with the profiler. But I should be able to get some pretty
good profiles on the Athlon64 X2 with it.
Hi,
M. Edward (Ed) Borasky wrote:
I can do fancier stuff in R, but I probably won’t – 5x is easy to remember.
Some warning:
- Results of vm[12]_* benchmarks should be subtracted
loop_whileloop[12] results.
Does anyone hack benchmark program?
- These benchmarks are only “micro” benchmarks. The purpose of
these benchmarks are only to measure VM basic performance.
Especially benchmarks focuses features which YARV achives
performance improvement on.
So practical application (such as using String manipulation) will
not be fast.
M. Edward (Ed) Borasky wrote:
Well … I’ll agree with that after looking at the codes. They’re
definitely micro-benchmarks and probably won’t stress the cache/RAM
interface on modern chips. I’m hoping to get a Rails benchmark running
in the next few days – it looks like it will be either Substruct or
rTPlan, whichever one is easiest to install. I don’t suppose you have a
Rails benchmark laying around in your repositories.
Alexey Verhovsky from ThoughtWorks does have a “Pet Store” app in Rails
that he’s been using to benchmark. I don’t know how complete it is or
how similar to a “real world” application, but it might be a good place
to start (and I suspect he’d be interested in having others try it and
improve it).
Probably easiest to search JRuby mailing list archive for it…I don’t
have a link handy.
On Sep 24, 2007, at 01:00 , SASADA Koichi wrote:
not be fast.
For what its worth, the RubyGems test suite runs about 20-25% faster
on a recent 1.9.0, as reported by test/unit.
Charles Oliver N. wrote:
how similar to a “real world” application, but it might be a good place
to start (and I suspect he’d be interested in having others try it and
improve it).
Probably easiest to search JRuby mailing list archive for it…I don’t
have a link handy.
Yeah … he told me about that on the Rails list. I downloaded it but
there aren’t any installation instructions. It came up but none of the
links worked, presumably because I didn’t do some necessary step.
Charles Oliver N. wrote:
is or how similar to a “real world” application, but it might be a
good place to start (and I suspect he’d be interested in having others
try it and improve it).
Probably easiest to search JRuby mailing list archive for it…I don’t
have a link handy.
Take a look at the project tw-commons in RubyForge.
Cheers
–
Ola B. (http://ola-bini.blogspot.com)
JRuby Core Developer
Developer, ThoughtWorks Studios (http://studios.thoughtworks.com)
Practical JRuby on Rails (http://apress.com/book/view/9781590598818)
“Yields falsehood when quined” yields falsehood when quined.
On 9/24/07, M. Edward (Ed) Borasky [email protected] wrote:
Yeah … he told me about that on the Rails list. I downloaded it but
there aren’t any installation instructions. It came up but none of the
links worked, presumably because I didn’t do some necessary step.
- yum install mysql-server
- mysql -u root
create database petstore_production
exit
- rake RAILS_ENV=production db:reset
- script/server -e production
- ab -c 1 -n 1000
http://localhost:3000/shop/viewCategory.shtml?category=DOGS
That’s about it.
On 9/24/07, Alexey V. [email protected] wrote:
That’s about it.
Also:
To run on Mongrel-JRuby:
jruby -O -J-server script/server -e production
To run on Jetty:
jruby -S rake jrubyworks:war
this sucks, will fix soon
mkdir -p war
cp petstore.war war/
jruby -S rake jrubyworks:retty
To run on Ruby 1.9: a fairly big exercise (Rails is not Ruby
1.9-compatible)
SASADA Koichi wrote:
Does anyone hack benchmark program?
Well … I’m putting together a benchmark suite, including the ones in
Ruby 1.9.0 and my own MatrixBenchmark. And I’m planning to duplicate
(and enhance) the logic in the spreadsheet in Ruby.
- These benchmarks are only “micro” benchmarks. The purpose of
these benchmarks are only to measure VM basic performance.
Especially benchmarks focuses features which YARV achives
performance improvement on.
So practical application (such as using String manipulation) will
not be fast.
Isn’t the String manipulation written in C already? It won’t be slower,
and overall a program that does a lot of String manipulation will run
faster because its spending more of its time doing String manipulation
and less of its time interpreting the Ruby code around the String
manipulation.
In general, yes, micro-benchmarks will have a higher speedup than
real-world programs for a variety of reasons. At the same time, though,
a faster VM helps everybody, while faster String operations only help
people doing String processing, faster Bignum operations will help my
Matrix benchmark but not a script that searches for patterns in a huge
text file using Oniguruma, and neither one of them will help a
multimedia codec that’s doing a discrete cosine transform.
Alexey V. wrote:
- script/server -e production
- ab -c 1 -n 1000 http://localhost:3000/shop/viewCategory.shtml?category=DOGS
That’s about it.
Thanks!! It was the MySQL piece I was missing – it’s installed but I
didn’t know how to build the database.
SASADA Koichi wrote:
- Results of vm[12]_* benchmarks should be subtracted
loop_whileloop[12] results.
thank you for this hint! I updated my online benchmark report:
http://www.rubychan.de/share/yarv_speedups.html
file://localhost/Users/murphy/ruby/bench/yarv_bench.rb
and removed the misleading “Average speedup”.
M. Edward (Ed) Borasky wrote:
So … in a nutshell … 1.9.0 is five times as fast as 1.8.6!
I don’t think this is fair to say. Some benchmarks I made with pure Ruby
libraries and the ones that the Kansai Rails people did:
eigenclass.org
indicate speedups around 1.1x - 1.5x.
[murphy]
Brent Roman wrote:
Column 1 is the benchmark name
ratio to be below 5.1 and half of the time you can expect it to be above
5.1. Only 25 percent of the time is the ratio below 2.8 percent, etc.
So … in a nutshell … 1.9.0 is five times as fast as 1.8.6!
I can get it – haven’t done so yet. Once I get the last benchmark in
the suite up and running (Rails app) I’ll start looking at memory – a
little bit anyhow. Quite frankly, I’m interested in raw speed at the
possible expense of RAM space, since one of my boxes is a 4 GB 64-bit
machine and I can’t even fill up RAM by running 10 compiles
concurrently.
Ed,
Did you do any investigation of the relative vmSizes between
v1.8 and v1.9?
Do you have any rough feel for the memory use of v1.9 vs. v1.8
running these benchmarks?