I just found out about Web Framework Benchmark at http://www.techempower.com/benchmarks/ I found it interesting to find a lot of their JRuby benchmarks performed worse than MRI one. I read their test methodology, they do allow some warm up period to make sure JIT based platform has enough time to kick in. Any thought why JRuby performs worse, when supposedly better than MRI one?
on 2013-11-13 21:52
on 2013-11-13 22:25
Had a look at their github repo... Gemfile for jruby contains jruby-jars... So it might be jruby running within MRI, not jruby running in pure java. So far over more than 2 years moving from MRI to jruby, I never found a case where jruby would perform so badly, and I am on Windows... Using a fully expanded or installed jruby is faster than jruby-jars or jruby-complete, especially with lots of gems.
on 2013-11-13 23:01
We only perform worse on the rack benchmarks for all others we perform better. Perhaps there is something in jruby-rack which is not performing well? I would think if a full Rails app warms up enough to perform better than MRI a smaller app would warmup faster? -Tom
on 2013-11-13 23:13
I've actually been looking at these benchmarks the past week in prototyping the next generation of TorqueBox. So far I've been looking mainly at the JSON serialization test and the plaintext test to get a benchmark. All the JRuby tests are run on top of Resin. Why Resin was chosen instead of Puma, Trinidad, TorqueBox / torquebox-lite, or something else more sane I'm not sure. That's the biggest problem with at least the JSON and plaintext tests. Techempower gets 69k req/s on their i7 hardware for the 'rack-ruby' JSON test that uses Ruby 2.0, Unicorn, and Nginx. As a baseline, on my local laptop I get 32k req/s for that same rack-ruby test with the same versions of MRI Ruby, Unicorn, and Nginx. On that same local laptop with torquebox-lite, I get 46k req/s. A nice improvement for sure, but not anything amazing. Still on the same laptop, using some unpushed prototype code that will replace torquebox-lite, I get 71k req/s. This prototype is based on JBoss Undertow, which you'll notice is on top or near the top of most of these benchmarks. Perhaps it's time to put a bow on this server and get it out there to let people play with? So, we should definitely work up a pull request to use a better JRuby server option. And it doesn't have to be TorqueBox by any means - these are just the simple examples I had locally. One optimization in all my local JRuby tests is I set the JRuby option "-Xjruby.ji.objectProxyCache=false" which will be the default in JRuby 9k. At lower throughput levels it doesn't really make a difference but once you start getting over 50k req/s or so the overhead from using the object proxy cache really starts to show up. Ben
on 2013-11-13 23:21
It's likely that the tests aren't CPU-bound. These tiny Rack apps are basically a test of how much the web server, language, and framework can get out of the way. And my guess is there's either something in JRuby-Rack or Resin that is getting in the way - excessive object allocation, synchronization in the request path, etc. Ben
on 2013-11-14 01:36
I wonder also what templating system they were using for rendering responses. When I first started using JRuby, my existing Rails apps were 10 times slower using JRuby 6, and I couldn't understand why, but I finally narrowed it down to anything using haml being terrible (and I was using a lot of haml). Erb templates rendered much faster, but still way slower than MRI. JRuby 7 got haml down to only 2 or 3 times slower, and Erb was very close, but like these benchmarks, still a little slower. This makes decent sense. It's really hard to see how Java is going to be faster at something like String manipulation than C. But unless an application is really template/view heavy, I think you'll make it up in other ways. Sent from my mobile device. Apologies for brevity. On Nov 13, 2013, at 3:51 PM, DJ <firstname.lastname@example.org> wrote: I just found out about Web Framework Benchmark at http://www.techempower.com/benchmarks/ I found it interesting to find a lot of their JRuby benchmarks performed worse than MRI one. I read their test methodology, they do allow some warm up period to make sure JIT based platform has enough time to kick in. Any thought why JRuby performs worse, when supposedly better than MRI one?
on 2013-11-15 04:27
These particular tests weren't using any templating - just strings returned directly from Rack applications.
on 2013-11-15 04:35
If anyone else is interested, I've released version 0.1.0 of the 'torqbox' gem, which scores drastically higher on the TechEmpower JSON and plaintext benchmarks on my machine. I'd like to get a couple more reports of its performance before submitting a pull request to the TechEmpower repo to replace its usage of Resin with torqbox. This should outperform any other JRuby web server, and I'd like to hear reports from anyone that can find an example showing otherwise. If you're curious, 'torqbox' is the codename for a future TorqueBox version. The funky gem name is just so we can cut releases of both without confusing current 'torquebox' gem users. Read more at https://github.com/torquebox/torquebox/tree/5x-dev
on 2013-11-15 13:21
As for the JRuby-Rack part ... yes there might be some "redundant" (as Tom mentioned) work in JRuby-Rack's Request/Response lifecycle. It was not an issue (at least for now) - it simply outperformed MRI as much as I've seen with Trinidad and some of the standalone servers - Warble-d (including JBoss). I have started some work on this locally ... e.g. rewriting JRuby::Rack::Response to "plain" Java, but time is a precious commodity managing a few OSS projects :) So we'll see, hopefully some of it gets out this year ... at least in a 1.2.0 pre-release. The other thing is, I've heard quite a lot of "praise buzz" about Resin but it's somehow one of the few servers I never actually seen at work even through my Java days. K.