Hi
We all know: Ruby is not the fastest interpreted language out there. I
think, this isn’t the goal, but I also think, Ruby is too slow.
I would suggest, that a time, say two monthes or three, only speed
patches and bugfixes are accepted. So the development could be
concentrated on speed.
I’d like to hear what over users out there and the developers say to
this idea.
On 02/22/2010 20:30, Benedikt Müller wrote:
Ruby is not the fastest interpreted language out there.
If you have not already done so, switch to Ruby 1.9.
2010/2/22 Alexander J. [email protected]:
On 02/22/2010 20:30, Benedikt Müller wrote:
Ruby is not the fastest interpreted language out there.
If you have not already done so, switch to Ruby 1.9.
Done, and I know that there are improvements. But it’s not enough
On Mon, Feb 22, 2010 at 2:47 PM, Benedikt Müller
[email protected]wrote:
2010/2/22 Alexander J. [email protected]:
On 02/22/2010 20:30, Benedikt Müller wrote:
Ruby is not the fastest interpreted language out there.
If you have not already done so, switch to Ruby 1.9.
Done, and I know that there are improvements. But it’s not enough–
Gruß, BenediktIf you really want speed check out JRuby or MacRuby. From what I
understand they are usually a version behind regular Ruby but you have
the
power of the JVM. I agree with speeding Ruby up to the credit of the
core
team they did make an effort on the last release. I do think there are
a
few other options. (Don’t get me wrong I am not opposed to the speed
sprints
I just wanted to present other options) As far as I know the Ruby
runtime
is missing lookahead ability. If that is correct adding lookahead could
give the runtime a huge performance boost. Another alternative to
asking
the Ruby core team to do these sprints we could profile, make changes,
and
submit the commits (and test cases) ourselves. I have never worked on
open
source project before, but I am speaking with someone about how to get
involved and be a productive active member. It takes more than I
originally
thought.
Hi,
In message “Re: Speed sprint”
on Tue, 23 Feb 2010 04:30:55 +0900, Benedikt
Müller [email protected] writes:
|I would suggest, that a time, say two monthes or three, only speed
|patches and bugfixes are accepted. So the development could be
|concentrated on speed.
|I’d like to hear what over users out there and the developers say to this idea.
We are very interested in performance improvement. Although I don’t
think we are going to restrict changes related to performance, we are
VERY welcome the proposals, suggestions, ideas, and patches, that
related to performance.
matz.
On Mon, Feb 22, 2010 at 12:47 PM, Benedikt Müller
[email protected]wrote:
Done, and I know that there are improvements. But it’s not enough
Have you taken a look at some of the other Ruby implementations
available,
such as JRuby, Rubinius, and MacRuby?
JRuby is able to leverage the performance features of the Java Virtual
Machine. Rubinius and MacRuby are both using LLVM to produce native
code
from Ruby sources. Ruby 1.9, on the other hand, uses a stack machine
interpreter and does not compile to native code.
Depending on the specific nature of your performance problems, one of
these
alternative implementations may be addressing your issues already.
On 22.02.2010 20:47, Benedikt Müller wrote:
2010/2/22 Alexander J. [email protected]:
On 02/22/2010 20:30, Benedikt Müller wrote:
Ruby is not the fastest interpreted language out there.
If you have not already done so, switch to Ruby 1.9.
Done, and I know that there are improvements. But it’s not enough
You know that greed is one of the seven deadly sins, do you? 
Kind regards
robert
Alexander J. wrote:
On 02/22/2010 20:30, Benedikt Müller wrote:
Ruby is not the fastest interpreted language out there.
If you have not already done so, switch to Ruby 1.9.
I have seen this claim that Ruby 1.9 is somehow faster than Ruby 1.8
repeated over and over again, but I have never seen any credible
evidence for that, neither in my own benchmarks nor in Antonio
Cangiano’s (or any other, for that matter). Does anyone have any
evidence that this is actually the case? I would be very interested in
that.
jwm
On Feb 23, 2010, at 7:50 AM, Jörg W Mittag wrote:
that.
Are you looking at the same benchmarks and charts that I am?
http://antoniocangiano.com/category/ruby-benchmark-suite/
This clearly shows that 1.9.1 is consistently and measurably faster than
1.8 in nearly every benchmark. My own experience proves this out too.
I am a recent convert to 1.9.1 after I discovered that it ran all of
my scripts at least 50% faster than 1.8.7. Between 1.9.1 and JRuby I am
loving my ruby performance lately. I am also tracking Rubinius closely;
it is very fast but its performance is less consistent than MRI and
JRuby so far.
cr
2010/2/23 Jörg W Mittag
<[email protected][email protected]
that.
jwm
Hi,
“I have never seen” Maybe you didn’t look enough. Anyway, I think
there
are plenty of benchmarks …
Test yourself and you’ll see.
I think it’s most of the time faster or equal in Ruby 1.9. There is a
reason
for this speed improvement, I let you search it.
You mention usual Antonio C. Benchmarks:
Well, there is a long time I keep traces of these benchmarks:
MBP => MacBookPro, 2x2.26GHz, 2Go
P => patchlevel
R => revision
T => trunk
Time Computer OS 32/64bit RubyVersion
0.684 MBP Mac 64 1.9.2 2010-01-14 T 26319
0.836 MBP Lin 32 1.9.2 2009-07-18 T 24186
0.899 MBP Mac 32 1.9.2 2009-11-04 T 25635
1.719 MBP Win 32 1.9.2 2009-07-18
1.850 MBP Mac 32 1.8.6 2008-08-11 P 287
2.000 MBP Mac 32 1.8.7 2009-12-24 P 248
2.406 MBP Win 32 1.9.1 2009-01-30 R 21907
2.937 MBP Win 32 1.8.6 2007-09-24 P 111
If that is not clear …
All 1.9.2 are faster(more than 2 times here). The only exception is
probably
due to better implementation of 1.8 on Mac than Windows (the 1.9.1 is
probably an early version too).
Regards,
B.D.
Chuck R. wrote:
that.
Are you looking at the same benchmarks and charts that I am?
Yes, I am.
This clearly shows that 1.9.1 is consistently and measurably faster
than 1.8 in nearly every benchmark.
No, it doesn’t. It shows that 1.9.1 running on one VM is faster than
1.8 on a totally different VM. That’s shoddy benchmarking. If you
want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not 1.8
and some other things vs. 1.9 and some totally unrelated, completely
different other things.
This is statistics 101, and is usually called “controlling your
variables”, although the great Zed S. in his inimitable style said
it much better: “if you want to measure shit, don’t measure other
shit.”
You claim that the performance increase is due to Ruby 1.9. But how do
you know that? How do you know it is not due to YARV? Or different
compiler options? Or different ./configure options? After all, it
wouldn’t be the first time that Antonio had to withdraw or correct his
benchmark results.
My own experience proves this out too.
Then show your results! As I wrote before: I would love to see them!
jwm
http://shootout.alioth.debian.org/
These benchmarks show that Ruby 1.9 (MRI) is much faster than 1.8. In
fact, it is even faster than Python (the closest competing* language).
But keep in mind that dynamically-typed, everything-is-an-object
languages will always come with performance penalties.
*Not that we’re actually “competing”–we’re all friends, helping
eachother make great very-high-level languages with our respective
preferred syntaxes and feature-sets. But Ruby is still way better.
Benoit D. wrote:
evidence that this is actually the case? I would be very interested in
that.
“I have never seen” Maybe you didn’t look enough. Anyway, I think there
are plenty of benchmarks …Test yourself and you’ll see.
Did you even read the article you replied to? I already wrote that I
did my own tests and they I did not see any statistically
significant difference in performance.
I think it’s most of the time faster or equal in Ruby 1.9.
It’s interesting that you use the word “think” here. So, apparently,
you have your (subconcious) doubts, too.
There is a reason for this speed improvement, I let you search it.
Again, I would like to know. The only source I could find so far, is
a video from a BoF session with Jon Lam from maybe two years ago,
which goes something like this:
| Audience: What about 1.9 support?
| JL: Microsoft has commited to making IronRuby a real Ruby
| Implementation that runs real Ruby code. Right now, real Ruby
| code is written in Ruby 1.8. We would love to start with Ruby
| 1.9, because it has some semantic changes that make it much easier
| to implement and much easier to implement fast. But we have to
| stay with 1.8 for now.
[Don’t hold me to the exact wording. This is all from memory, and it
was just a minor sidenote in a video I watched two years ago.]
This is pretty much the only source I could find which actually names
a reason for the claimed speed improvements. However, “some semantic
changes” isn’t actually incredibly precise, after all, in a
programming language, pretty much everything is a “semantic change”.
So, if you know the reason, please share it. Saying “I let you search
it” isn’t exactly helpful. After all, I wouldn’t have asked if I
hadn’t searched first, that’s Netiquette 101.
Time Computer OS 32/64bit RubyVersion
0.684 MBP Mac 64 1.9.2 2010-01-14 T 26319
0.836 MBP Lin 32 1.9.2 2009-07-18 T 24186
0.899 MBP Mac 32 1.9.2 2009-11-04 T 25635
1.719 MBP Win 32 1.9.2 2009-07-18
1.850 MBP Mac 32 1.8.6 2008-08-11 P 287
2.000 MBP Mac 32 1.8.7 2009-12-24 P 248
2.406 MBP Win 32 1.9.1 2009-01-30 R 21907
2.937 MBP Win 32 1.8.6 2007-09-24 P 111
Unfortunately, this table doesn’t show the execution environment of
the individual benchmark runs. The only way in which this table is
useful, is if the benchmark runs were tightly controlled. Otherwise
you get uncontrolled variables, and all your benchmark results mean
squat.
In statistics, this is called “controlling your variables”, but Zed
Shaw said it much better: “If you want to measure shit, don’t measure
other shit.”
In other words, if you want to measure Ruby 1.9 vs. Ruby 1.8, measure
Ruby 1.9 vs. Ruby 1.8 and not Ruby 1.9 and YARV vs. Ruby 1.8 and MRI,
because that way you will never know whether the performance
difference came from Ruby 1.9, YARV or a combination of the two.
If that is not clear …
All 1.9.2 are faster(more than 2 times here). The only exception is probably
due to better implementation of 1.8 on Mac than Windows (the 1.9.1 is
probably an early version too).
Like I wrote above: if you actually did properly control your
variables for these benchmarks, then, yes, they show indeed a
performance increase for Ruby 1.9. And in that case, you really should
publish these results, because, like I wrote earlier, there is nothing
like this out there, at the moment.
jwm
Ruby is not the fastest interpreted language out there.
Are you looking at the same benchmarks and charts that I am?
than
1.8 on a totally different VM. That’s shoddy benchmarking. If you
want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not
1.8
and some other things vs. 1.9 and some totally unrelated,
completely
different other things.
The claim is that 1.9.1, in its entirety, is generally faster than
1.8.6, in its entirety. The data backs that up. (I’m looking at the “A
faster Ruby on Windows is possible (benchmarks for 4 implementations
inside)” benchmark.) The fact that a new VM (YARV) is part of what
makes 1.9.1 different from 1.8.6 is irrelevant. If you trying to assert
that the different VM is the only reason for the performance, then
yes, that would be a bad claim, because there are too many other
variables and changes involved. But they didn’t; the comparison was
between 1.9.1 and 1.8.6.
Your statement is a little akin to saying a speed comparison between a
Nissan Cube and a Nissan GT-R is invalid because they have different
engines. Well, of course they have different engines; that’s part of
what makes them different vehicles.
(If you were referring to the benchmark including IronRuby, well that is
why it was also compared with 1.8.6 and 1.9.1; that is a comparison of
the IronRuby release, with its use of the .NET CLR, with the Ruby native
releases, with their respective included VMs. Again, it’s the entire
release that is trying to be compared.)
This is statistics 101, and is usually called “controlling your
variables”, although the great Zed S. in his inimitable style
said
it much better: “if you want to measure shit, don’t measure other
shit.”
And the variables were controlled, as best they could be. For the “A
faster Ruby on Windows is possible (benchmarks for 4 implementations
inside)” benchmark, they use the same system to run all tests. They
didn’t run 1.9 on a faster system than 1.8, or vice versa. They also
include 2 different installer methods as a second variable in the mix.
The comparison of Windows to Linux (above the “faster Ruby on
Windows…” benchmark) were on identical systems using identical Ruby
releases. They even used 32-bit versions of Windows and Linux to remove
that variable as a difference. Obviously different compilers are
involved, because they are different platforms, but that gets stated as
part of the setup description.
You claim that the performance increase is due to Ruby 1.9. But how
do
you know that?
Because that is what the numbers are telling us.
How do you know it is not due to YARV? Or different
compiler options? Or different ./configure options? After all, it
wouldn’t be the first time that Antonio had to withdraw or correct
his
benchmark results.
As far as the bulk numbers are concerned, who cares? The test
conditions were given, and even the code used to get those results are
available. So take advantage of another scientific principle, and
repeat_the_experiment_for_yourself. If you get widely different
results, share them with a full explanation of your test setup and
methodology.
If someone were interested in a detailed report about which differences
between 1.8.6 and 1.9.1 provide what portion of the performance
differences (or lack of differences, for some tests), that would be a
different type of testing. That would be profiling, with the need to
identify which sections are comparable between 1.8.6 and 1.9.1 and which
are different.
(As to your point about Antonio withdrawing or correcting benchmark
results, that’s just a sign of responding to peer review. If you can
point out valid reasons that the results are invalid as given, or
produce a large disparity from your own runs, share that so further
corrections can be made.)
My own experience proves this out too.
Then show your results! As I wrote before: I would love to see
them!jwm
The flip-side is also fair. You claim that your own benchmarks show no
difference between 1.8 and 1.9. Fine. But you haven’t gone into any
detail about how you tested them (that I’ve seen): what was your test
setup; how did you compile and install them; where is your code that you
used to benchmark the two; show the data for the different runs that
will provide the statistical rigor to justify a statement that “there’s
no significant performance difference.” Antonio’s benchmarks have 51
different tests, some with different implementations, and most show that
1.9.1 is faster than 1.8.6, but not all. Maybe your one test (or two?)
happens to be similar to one of the ones which has 1.8.6 faster than
1.9.1. Well, if that is indicative of the type of work that you always
do, then yes, 1.8.6 would not be any faster than 1.9.1.
It was mentioned by at least one other, but here is another benchmark:
http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=yarv&lang2=ruby
It’s data shows that for the majority of the benchmarks, Ruby 1.9 (yarv)
takes less than half the time as 1.8 (MRI). But out of the 11 tests,
there were 3 where they were closer, and 1 where 1.9 took longer than
1.8 (pidigits). But one test out of 11 does not, IMO, invalidate the
claim that Ruby 1.9 is generally faster than Ruby 1.8.
Coey
Then show your results! As I wrote before: I would love to see them!
http://blog.pluron.com/2009/05/ruby-19-performance.html
Their rails runs 2x faster with 1.9
-r
So what your saying is that the VM is part of Ruby? I think your analogy
would be better stated that a speed comparison between a Nissan Cube and
a Nissan GT-R is invalid because they were driven on different tracks
for the time trials.
In that case, IT would be valid.
The claim is that 1.9.1, in its entirety, is generally faster than
1.8.6, in its entirety. The data backs that up. (I’m looking at the “A
faster Ruby on Windows is possible (benchmarks for 4 implementations
inside)” benchmark.) The fact that a new VM (YARV) is part of what
makes 1.9.1 different from 1.8.6 is irrelevant. If you trying to assert
that the different VM is the only reason for the performance, then
yes, that would be a bad claim, because there are too many other
variables and changes involved. But they didn’t; the comparison was
between 1.9.1 and 1.8.6.
Your statement is a little akin to saying a speed comparison between a
Nissan Cube and a Nissan GT-R is invalid because they have different
engines. Well, of course they have different engines; that’s part of
what makes them different vehicles.
In that case, IT would be valid.
Just to be clear, when I talked about VMs, I was referring to the part
of Ruby’s architecture that is considered the virtual machine. I was
not talking about the virtual machine that would be running XP or Linux
within which a Ruby (or any other application) would be run.
In my analogy, Ruby 1.8 and 1.9 are 2 different CARs. The VM within
Ruby (whether the original 1.8 one or YARV in 1.9) is the engine. The
different benchmark scripts are the different tracks the cars run on.
And the virtual machine (with Windows or Linux) would be Earth in some
universe or alternate reality.
Yes, trying to compare Car 1 running on Earth 1 with Car 2 running on
Earth 2 would be a bit invalid, without also reporting Car 1 on Earth 2
and Car 2 on Earth 1.
Coey
On Feb 23, 2010, at 10:00 AM, Jörg W Mittag wrote:
evidence that this is actually the case? I would be very interested in
that.
Are you looking at the same benchmarks and charts that I am?Yes, I am.
Oh, boy, here we go. By the tone of your response, I’m not sure if
you’re a jerk or I’m in a bad mood. I’ll assume you are not and that I
am. But just in case you are this guy…
This clearly shows that 1.9.1 is consistently and measurably faster
than 1.8 in nearly every benchmark.No, it doesn’t. It shows that 1.9.1 running on one VM is faster than
1.8 on a totally different VM. That’s shoddy benchmarking. If you
want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not 1.8
and some other things vs. 1.9 and some totally unrelated, completely
different other things.
I don’t understand your point here. How can I benchmark 1.9.1 against
1.8.7 without the VMs being different? The whole point of the move to
1.9.1 was YARV and its new interpreter. The VMs are different and unless
someone undertakes the effort to move the 1.8 syntax on top of YARV,
that won’t change. You mentioned this in two separate replies to this
thread so I know it is an important point for you. You cannot bench
1.8 versus 1.9 without YARV entering the equation.
I suppose you are saying that there is no valid comparison between 1.8.x
and 1.9.x because of this difference? If not, explain yourself better.
If you do believe that there is no valid comparison because the VMs
are different, then please explain how 1.8.x and 1.9.x could be
rigorously compared.
I don’t know about you, but I run my ruby code on MRI, YARV, Java6 and
LLVM (Rubinius and MacRuby). The VM doesn’t matter to me as long as the
results are the same and each platform gives me the performance I
require.
I don’t know how you run your code without a VM; sounds neat.
This is statistics 101, and is usually called “controlling your
variables”, although the great Zed S. in his inimitable style said
it much better: “if you want to measure shit, don’t measure other
shit.”You claim that the performance increase is due to Ruby 1.9. But how do
you know that? How do you know it is not due to YARV? Or different
compiler options? Or different ./configure options? After all, it
wouldn’t be the first time that Antonio had to withdraw or correct his
benchmark results.
You’re looking for a level of rigor that just doesn’t exist in software
benchmarking and particularly for ruby open source. I ran these
benchmarks on my system. I can give you the basic details but my machine
likely performs differently from the exact same machine with the same
software load on it. There are dozens of Linux distros all with
differing patch levels; same goes for BSD, Windows and OSX. Maybe my
machine runs a little hotter (therefore slower) than an exact clone; I’m
not about to get a temperature controlled room to bench this language.
All of these things are likely to effect the bench results.
I say, “So what.” Unless there is a bug in a major system component
(e.g. libc, pthreads, etc), the compiler toolchain, or a hardware
defect, the results across comparable systems are pretty close. That is
good enough.
Results from lots of independent sources show that 1.9 is faster than
1.8. Why? It’s worthy of investigation by someone other than me. 
My own experience proves this out too.
Then show your results! As I wrote before: I would love to see them!
I emailed my results to you using Cogniano’s benchmark suite available
from github. I used the source available via the “rvm” utility and
benched both 1.8.7-p248 and 1.9.1-p376 built on my system (OSX Intel,
10.6.2, dual quad-core, blah blah blah).
Ruby 1.9.1 was faster in 60 (of 100+) tests, the same in 2 or 3, and
slower in about 20. These are against the “micro” benchmarks in the test
suite. In my world, the usual scripts I run are all noticeably faster
with 1.9.1 versus any 1.8.x MRI branch.
cr
Jörg W Mittag wrote:
In other words, if you want to measure Ruby 1.9 vs. Ruby 1.8, measure
Ruby 1.9 vs. Ruby 1.8 and not Ruby 1.9 and YARV vs. Ruby 1.8 and MRI,
because that way you will never know whether the performance
difference came from Ruby 1.9, YARV or a combination of the two.
The confusion is understandable.
Historically, on the Matz lineage of interpreters, ruby 1.9 (formerly
ruby 2.0) had long been associated with several key feature bullet
points, particularly:
- new VM for increased speed
- multinationalization support
I joined ruby-talk in 2001, and it was being discussed even then.
Originally the new VM was to be called Rite instead of YARV, and it
was expected to debut in ruby 2.0 instead of 1.9:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/23267
These days, when we look at other ruby interpreters, such as JRuby,
it’s easy to see that 1.8 vs. 1.9 language semantics are orthogonal
to the VM. So, indeed, on JRuby (for example), one can benchmark
1.8 semantics vs. 1.9 semantics without changing the VM.
But in the Matz lineage, 1.8 implies MRI, and 1.9 implies YARV.
Further, the new VM was always understood to be the primary reason
to expect speed improvements from 1.8 → 1.9 in the Matz lineage.
Regards,
Bill
On 2/23/2010 11:07 AM, Miles S. wrote:
So what your saying is that the VM is part of Ruby? I think your analogy would be better stated that a speed comparison between a Nissan Cube and a Nissan GT-R is invalid because they were driven on different tracks for the time trials.
In that case, IT would be valid.
Yes, the YARV VM is an intrinsic part of Ruby 1.9, just as the JVM
is an intrinsic part of Java.
Thus the original analogy stands, and it makes perfect sense to compare
MRI Ruby 1.8 results with
Ruby 1.9 results using YARV because one of the key improvements in 1.9
is the usage of YARV.