So, I have managed to build several different versions of Ruby to compare all in some benchmarks. I am going to let the numbers talk for themselves. It was all tested on an old desktop computer with Windows XP, so I am not testing anything too impressive. Of importance is the seemingly poor performance of Ruby 1.8.7 for whatever reason. It's as if Ruby 1.8.7 were like Windows ME, a version between major versions, as an analogy. :-) Also, I was not able to run the Mongrel gem on Ruby 1.9.1. I am going to order the data by fastest to make it easier to compare it on text. I have collected the data on a Google Spreadsheet: http://spreadsheets.google.com/ccc?key=pmtaDyNoBut... I am going to keep it short here and for more detail on versions and whatnot please refer to the spreadsheet. == WEBrick (req/s) 22.39 - jruby 16.95 - jruby --server 12.82 - ruby oneclick 1.8.6 12.49 - ruby msvc 1.8.6 12.45 - ruby rubyinstaller 1.8.6 12.43 - ruby mingw 1.8.6 11.94 - ruby msvc 1.9.1 11.48 - ruby mingw 1.9.1 3.42 - ruby msvc 1.8.7 3.40 - ruby mingw 1.8.7 == Mongrel (req/s) 357.14 - ruby mingw 1.8.6 347.45 - ruby rubyinstaller 1.8.6 326.70 - ruby msvc 1.8.6 313.57 - ruby oneclick 1.8.6 274.68 - ruby mingw 1.8.7 232.39 - ruby msvc 1.8.7 71.24 - jruby 62.22 - jruby --server (crash when starting up) - ruby mingw 1.9.1 (error when building gem) - ruby msvc 1.9.1 == bench_pythag.rb (s) 19.09 - jruby --server 24.35 - jruby 31.72 - ruby mingw 1.9.1 31.91 - ruby mingw 1.8.6 32.73 - ruby mingw 1.8.7 33.94 - ruby msvc 1.9.1 35.67 - ruby rubyinstaller 1.8.6 35.83 - ruby msvc 1.8.7 39.79 - ruby msvc 1.8.6 48.30 - ruby oneclick 1.8.6 == bench_fractal.rb (s) 3.22 - jruby --server 3.89 - jruby 5.67 - ruby msvc 1.9.1 5.75 - ruby mingw 1.9.1 9.80 - ruby mingw 1.8.6 9.90 - ruby mingw 1.8.7 10.11 - ruby rubyinstaller 1.8.6 10.30 - ruby msvc 1.8.6 10.64 - ruby msvc 1.8.7 15.17 - ruby oneclick 1.8.6 == bench_float_math.rb (s) 103.44 - jruby --server 125.89 - jruby 158.64 - ruby msvc 1.9.1 175.41 - ruby mingw 1.9.1 176.61 - ruby mingw 1.8.6 179.94 - ruby mingw 1.8.7 184.73 - ruby rubyinstaller 1.8.6 195.37 - ruby msvc 1.8.7 197.39 - ruby msvc 1.8.6 279.62 - ruby oneclick 1.8.6 Since I collected these data, the Ruby 1.9.1 branch has had a number of updates for one thing. :-) Benchmarks can be kind of tough to do right and there are all possibilities when it comes to doing them. If anyone has an algorithm that he would like to test on all those Ruby implementations I would be glad to test it. With the exception of testing Rails, which I have never used and don't know enough about it to willingly make myself available when testing it. :-) Cheers, Joao
on 2008-11-30 18:20
on 2008-12-01 00:02
According to my own tests, "Thin" should be better than your current best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby win over everyone. http://www.tudbc.org/tudbcjrubyservlet You should also give more details about how you test.
on 2008-12-01 01:19
Joao Pedrosa wrote: > (crash when starting up) - ruby mingw 1.9.1 > (error when building gem) - ruby msvc 1.9.1 I'd love to figure out what makes mongrel slow for us, when all other cases are faster ;) GlassFish gem would be a nice one to test, but of course there's no equivalent comparison for the others. - Charlie
on 2008-12-01 20:23
Benchmarking ruby on windows is fairly useless. MRI 1.8.* is known to be very slow on windows and harldy anyone deploys production apps to windows because of this. If you want your benches to be taken seriously you need to run them on linux, bsd or osx where ruby performs much better. Also make sure to compile ruby with --disable- pthreads for best perf. Cheers- Ezra Zygmuntowicz firstname.lastname@example.org
on 2008-12-01 20:49
On Mon, Dec 1, 2008 at 2:17 PM, Ezra Zygmuntowicz <email@example.com>wrote: > firstname.lastname@example.org Though it might be slow, I really don't think it's useless. Where I work, we are stuck in windows no matter the cost or benefit of another platform. Because of this these windows numbers actually matter to me.
on 2008-12-01 23:51
Post Tudbc wrote: > According to my own tests, "Thin" should be better than your current > best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby > win over everyone. > > http://www.tudbc.org/tudbcjrubyservlet > > You should also give more details about how you test. I notice that you use the best of 5 runs to report numbers, which I think it is going to vary a lot. You can use my free tool to do web benchmarking, which I think is much more accurate. http://www.tudbc.org/tudbcwebbenchmark I think your Machine is about same as my old test machine. Here are some numbers: Using "Echo" test: (copied from http://www.tudbc.org/jrubywebapp) 0.98ms TudbcJRubyServlet_RB 1.73ms Thin 4.04ms Mongrel (estimated, because Mongrel crashed in the middle) 8.51ms WEBrick Using your "time now+version" test: 1.11ms TudbcJRubyServlet_RB 1.56ms Thin 4.00ms Mongrel (estimated, because Mongrel crashed in the middle) 8.18ms WEBrick As you can see in your "time now" test on your machine, your Mongrel results were probably inaccurate, because you used the "best of 5" to measure. Both my "Echo" and your "Time now+version" tests seem to produce similar benchmarks. As you can see, TudbcJRubyServlet_RB still win easily, and Thin is faster than Mongrel by a wide margin. Note: The default measurement of TudbcWebBenchmark is average of last 10 of 20 runs (with each run for 10 seconds). Because Mongrel crashed, so I only estimate it by 3rd run of 5 seconds each. Note: Ruby version used: JRuby 1.1.5 and Ruby 1.8.7-p72 on Windows
on 2008-12-02 00:24
Oops! Forgot to paste the codes I used for TudbcJRubyServlet --------- timeversion.rb -------- a =  a << 'Hello Rack!' a << Time.new a << RUBY_PLATFORM a << RUBY_VERSION ctx.print(a.join('<br />')) --------------------------------- Your "time now+version" uses 5 array ops. My guess is that maybe JRuby array processing is relatively not as efficient, so that ratio between TudbcJRubyServlet and Thin for "time now+version" is not as great as "Echo" test. Note: My 4ms means your 1000/4 or 250 reqs/sec.