(Sorry if this is a duplicate post...the original seems to have disappeared) Many folks still seem to be in the dark about JRuby performance relative to Ruby 1.8.x. I figured a short post with some basic info could start a discussion for anyone wanting to know more. Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and for most execution benchmarks it is the fastest 1.8-compatible Ruby implementation available. This is based on the YARV benchmarks, the Alioth benchmarks, a number of benchmarks we created for JRuby, and a few other external benchmarks we run from time to time. Many (most?) of these are microbenchmarks, but a number are nontrivial code. For example, M. Ed. Borasky's MatrixBenchmark: JRuby: Hilbert matrix of dimension 64 times its inverse = identity? true 18.784000 0.000000 18.784000 ( 18.784000) Ruby 1.8.6: Hilbert matrix of dimension 64 times its inverse = identity? true 32.560000 0.110000 32.670000 ( 32.776915) Or the YARV "pentomino" benchmark: ~/NetBeansProjects/jruby $ time jruby -J-server test/bench/yarv/bm_app_pentomino.rb real 1m31.770s user 1m32.016s sys 0m1.278s ~/NetBeansProjects/jruby $ time ruby test/bench/yarv/bm_app_pentomino.rb real 1m48.100s user 1m47.489s sys 0m0.287s Ola Bini of the JRuby team recently ran all the YARV benchmarks and reported the results here: http://ola-bini.blogspot.com/2007/10/mystery-expos... And I periodically post about performance progress on my blog here: http://headius.blogspot.com I expect we'll see Alioth updated soon after the 1.1 or 1.1 beta 1 releases. What about Rails? One of the biggest uses for JRuby recently has been Rails. Rails performance is a more complicated thing to measure; there's a lot of non-execution bottlenecks that get in the way. But both Ola and Nick Sieger (also of JRuby team) have found that JRuby on Rails performance is either very near or exceeding Ruby 1.8.x on Rails: http://ola-bini.blogspot.com/2007/10/current-state... (toward the bottom of the post are updated Rails numbers) http://blog.nicksieger.com/articles/2007/10/25/jru... Rails is probably the ultimate real-world benchmark, and we're excited to see that JRuby's starting to pass Ruby 1.8.x performance here as well. I won't drag this on any longer, but I'd encourage you to try out your own Ruby code and let us know how it performs. We've spent a lot of time on performance issues for the 1.1 release, and have many future plans to continue increasing performance. We want your input. Many people believed we'd never be faster than the C implementation, and many still think we're slower. Now that I've set that record straight, any questions? - Charlie
on 2007-10-31 19:30
on 2007-10-31 21:51
> any questions? Startup time and comparison with 1.9? :)
on 2007-10-31 21:53
Roger Pack wrote: >> any questions? > > Startup time and comparison with 1.9? :) Also memory use comparison compared to normal C version. Does the GC throw as often, too?
on 2007-11-01 01:01
Roger Pack wrote: > Roger Pack wrote: >>> any questions? >> Startup time and comparison with 1.9? :) > > Also memory use comparison compared to normal C version. Does the GC > throw as often, too? In profiling most apps I've tried, even pretty brutal ones generating a lot of garbage, the GC cost is almost invisible. A tiny blip once in a while. I don't have any really good concrete numbers on overall memory usage. Tom's done a comparison of a minimal/bare Rails app here: http://www.bloglines.com/blog/ThomasEEnebo?id=38 The general story is that per JRuby on Rails instance (usually you have many in a single JVM), we use less memory than an equivalent Mongrel + Ruby + Rails process. A quick check of "jirb" compared to "irb" gives me: jirb: 27MB irb: 2.8MB This is on OS X, Java 6 preview version, so it's safe to say that any JRuby process will have an initial cost of about 25MB to start up the JVM. We've also done a lot in 1.1 to reduce memory consumption, and there's more work coming up. - Charlie
on 2007-11-01 01:59
There was a recent thread which asked how fast ruby ran for each user. http://www.ruby-forum.com/topic/125344#new Josip Gracin wrote: > Here's something interesting. The processor is Intel T7200 @ 2.00GHz, > Linux Ubuntu. > > 1. With ruby 1.8.5. which comes packaged with Ubuntu: > > Ruby 1.8.5 on i486-linux > It took 20.346013 seconds to run. 49149 iterations per > second. > > 2. With ruby 1.9.0 (2006-06-08) which comes packaged in Ubuntu: > > Ruby 1.9.0 on i486-linux > It took 23.875774 seconds to run. 41883 iterations per > second. > > > 3. With ruby 1.8.6 compiled from sources on the machine: > > Ruby 1.8.6 on i686-linux > It took 7.107875 seconds to run. 140689 iterations per > second. > > > Obviously, the target instruction set matters quite a lot. I think I'll > be compiling my own ruby from now on. :-) Greg Halsey wrote: > windows xp. intel core2 1.86GHz > > JRUBY: > > C:\jruby\jruby-trunk\bin>jruby calculate.rb > 55 > > It took 41.125 seconds to run. 24316 iterations per > second. > > Not so good. Let's disable ObjectSpace: > C:\jruby\jruby-trunk\bin>jruby -O calculate.rb > 55 > > It took 11.109 seconds to run. 90017 iterations per > second. > > That was good. Now let's give it some more juice: > C:\jruby\jruby-trunk\bin>jruby -J-server -O calculate.rb > 55 > > It took 5.75 seconds to run. 173913 iterations per > second. > -------------------------------------------- > MRI:One-Click > ruby calculate.rb > 55 > > Ruby 1.8.6 patch 0 on i386-mswin32 > It took 15.828 seconds to run. 63179 iterations per > second. Now, I am thinking that jruby might work well if we know how to find the sweet spot. Why the pre compiled version is so very much slower than a home compiled version makes me wonder why they do not compile it better. As things can be so radically different in jruby if only you know how to tweak it perfectly is another thing that makes me think. So, do you find your own benchmarks and work your compiler to make it run at its fastest or is performance going to be "roll your own" for those with insider knowledge?
on 2007-11-01 05:14
Roger Pack wrote: >> any questions? > > Startup time and comparison with 1.9? :) Startup time varies widely between platforms. OS X is probably one of the fastest: ~/NetBeansProjects/jruby $ time ruby -e "puts 'hello'" hello real 0m0.015s user 0m0.007s sys 0m0.008s ~/NetBeansProjects/jruby $ time jruby -e "puts 'hello'" hello real 0m1.815s user 0m1.709s sys 0m0.183s There's been reports that on server-class machines the startup time can be as high as 5-10 seconds. The slow startup time is partially due to the JVM and partially our fault (JRuby's complicated, yo). As for Ruby 1.9...it's hard to make a comparison right now. Most of the benchmarks we use to measure Ruby 1.9 (like those I described previously) are usually a bit faster in Ruby 1.9. And of course almost all of Ruby 1.9's benchmarks are fastest in Ruby 1.9. Part of the difficulty comparing is that Ruby 1.9 contains a bunch of changes to Ruby language semantics that we have not yet put in JRuby, since we've focused on 1.8 compatibility. Things like: - fast math operations, that speed up +, -, etc, but you can't override them - constant-time case/when when literal values are used, but you can't override === for those literal types Here's A few numbers, for your edification. Some things are faster in Ruby 1.9: JRuby: ~/NetBeansProjects/jruby $ jruby -J-server test/bench/bench_fib_recursive.rb 0.869000 0.000000 0.869000 ( 0.869000) 0.677000 0.000000 0.677000 ( 0.677000) 0.652000 0.000000 0.652000 ( 0.652000) 0.648000 0.000000 0.648000 ( 0.648000) Ruby 1.9: ~/NetBeansProjects/jruby $ ../ruby1.9/ruby -I ../ruby1.9/lib test/bench/bench_fib_recursive.rb 0.400000 0.000000 0.400000 ( 0.405639) 0.400000 0.000000 0.400000 ( 0.406707) 0.400000 0.000000 0.400000 ( 0.400150) 0.400000 0.000000 0.400000 ( 0.398065) And some things are slower: JRuby: 1m loops yielding three fixnums 10 times to block splatting and accessing them 3.387000 0.000000 3.387000 ( 3.387000) 3.314000 0.000000 3.314000 ( 3.315000) 3.235000 0.000000 3.235000 ( 3.235000) Ruby 1.9: 1m loops yielding three fixnums 10 times to block splatting and accessing them 4.980000 0.020000 5.000000 ( 5.055369) 4.980000 0.020000 5.000000 ( 5.073821) 5.010000 0.020000 5.030000 ( 5.088681) And so on. However, for Rails, I've only seen one comparison that showed Ruby 1.9 was either slightly faster or slower than Ruby 1.8.x: http://eigenclass.org/hiki/non-synthetic-benchmarks-for-yarv I'm not sure if there's been a more recent comparison. I believe we'll be able to match Ruby 1.9 performance once we start making the same optimizations. - Charlie
on 2007-11-01 05:25
Lloyd Linklater wrote: >> Ruby 1.8.5 on i486-linux >> 3. With ruby 1.8.6 compiled from sources on the machine: >> windows xp. intel core2 1.86GHz >> C:\jruby\jruby-trunk\bin>jruby -O calculate.rb >> second. > sweet spot. Why the pre compiled version is so very much slower than a > home compiled version makes me wonder why they do not compile it better. > As things can be so radically different in jruby if only you know how to > tweak it perfectly is another thing that makes me think. > > So, do you find your own benchmarks and work your compiler to make it > run at its fastest or is performance going to be "roll your own" for > those with insider knowledge? We have been running whatever benchmarks we can, and they're all pretty generic. For all the numbers I've reported (and I believe most of the numbers others have reported for JRuby), it's been JRuby trunk (soon to be 1.1) with the Java 6 "server" VM. There's really no more tweaking than that required to get the same performance. As for the compiler...there's no additional user tweaking necessary to make it run best; it should run its best all the time. The additional tweaks you may want to apply would be using a different Java version, tweaking the JVM or GC settings, and so on. Normal server tuning most people won't need. The reality with the JVM, and especially with JRuby, is that short tests aren't going to show the real long-term performance. For most execution, JRuby starts out interpreted, and the interpreter is slower than Ruby 1.8.x. But after running for a while, code will eventually compile to Java bytecode (or you can precompile to skip that time). Then there's an additional delay while the JVM itself eventually compiles that bytecode down to platform-native code. We expect that for JRuby 1.1, the main sweet spot will be on longer-running apps e.g. for server-side apps or for heavy data processing.
on 2007-11-01 06:11
Charles Oliver Nutter wrote: > JRuby 1.1 is now generally faster than Ruby 1.8.6, and > for most execution benchmarks it is the fastest 1.8-compatible Ruby > implementation available. I look forward to getting my hands on JRuby 1.1 and running some Rails apps. Keep up the good work!