Some benchmarks with MinGW, JRuby, MSVC

Ruby
1

So, I have managed to build several different versions of Ruby to
compare all
in some benchmarks. I am going to let the numbers talk for themselves.
It was
all tested on an old desktop computer with Windows XP, so I am not
testing
anything too impressive.

Of importance is the seemingly poor performance of Ruby 1.8.7 for
whatever
reason. It’s as if Ruby 1.8.7 were like Windows ME, a version between
major
versions, as an analogy. :slight_smile:

Also, I was not able to run the Mongrel gem on Ruby 1.9.1.

I am going to order the data by fastest to make it easier to compare
it on text.
I have collected the data on a Google Spreadsheet:

I am going to keep it short here and for more detail on versions and
whatnot
please refer to the spreadsheet.

== WEBrick (req/s)

22.39 - jruby
16.95 - jruby --server
12.82 - ruby oneclick 1.8.6
12.49 - ruby msvc 1.8.6
12.45 - ruby rubyinstaller 1.8.6
12.43 - ruby mingw 1.8.6
11.94 - ruby msvc 1.9.1
11.48 - ruby mingw 1.9.1
3.42 - ruby msvc 1.8.7
3.40 - ruby mingw 1.8.7

== Mongrel (req/s)

357.14 - ruby mingw 1.8.6
347.45 - ruby rubyinstaller 1.8.6
326.70 - ruby msvc 1.8.6
313.57 - ruby oneclick 1.8.6
274.68 - ruby mingw 1.8.7
232.39 - ruby msvc 1.8.7
71.24 - jruby
62.22 - jruby --server
(crash when starting up) - ruby mingw 1.9.1
(error when building gem) - ruby msvc 1.9.1

== bench_pythag.rb (s)

19.09 - jruby --server
24.35 - jruby
31.72 - ruby mingw 1.9.1
31.91 - ruby mingw 1.8.6
32.73 - ruby mingw 1.8.7
33.94 - ruby msvc 1.9.1
35.67 - ruby rubyinstaller 1.8.6
35.83 - ruby msvc 1.8.7
39.79 - ruby msvc 1.8.6
48.30 - ruby oneclick 1.8.6

== bench_fractal.rb (s)

3.22 - jruby --server
3.89 - jruby
5.67 - ruby msvc 1.9.1
5.75 - ruby mingw 1.9.1
9.80 - ruby mingw 1.8.6
9.90 - ruby mingw 1.8.7
10.11 - ruby rubyinstaller 1.8.6
10.30 - ruby msvc 1.8.6
10.64 - ruby msvc 1.8.7
15.17 - ruby oneclick 1.8.6

== bench_float_math.rb (s)

103.44 - jruby --server
125.89 - jruby
158.64 - ruby msvc 1.9.1
175.41 - ruby mingw 1.9.1
176.61 - ruby mingw 1.8.6
179.94 - ruby mingw 1.8.7
184.73 - ruby rubyinstaller 1.8.6
195.37 - ruby msvc 1.8.7
197.39 - ruby msvc 1.8.6
279.62 - ruby oneclick 1.8.6

Since I collected these data, the Ruby 1.9.1 branch has had a number
of updates
for one thing. :slight_smile:

Benchmarks can be kind of tough to do right and there are all
possibilities when it
comes to doing them.

If anyone has an algorithm that he would like to test on all those
Ruby implementations
I would be glad to test it. With the exception of testing Rails, which
I have never used
and don’t know enough about it to willingly make myself available when
testing it. :slight_smile:

Cheers,
Joao

2

According to my own tests, “Thin” should be better than your current
best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby
win over everyone.

http://www.tudbc.org/tudbcjrubyservlet

You should also give more details about how you test.

3

Joao P. wrote:

(crash when starting up) - ruby mingw 1.9.1
(error when building gem) - ruby msvc 1.9.1

I’d love to figure out what makes mongrel slow for us, when all other
cases are faster :wink:

GlassFish gem would be a nice one to test, but of course there’s no
equivalent comparison for the others.

  • Charlie
4

Benchmarking ruby on windows is fairly useless. MRI 1.8.* is known to
be very slow on windows and harldy anyone deploys production apps to
windows because of this. If you want your benches to be taken
seriously you need to run them on linux, bsd or osx where ruby
performs much better. Also make sure to compile ruby with --disable-
pthreads for best perf.

Cheers-
Ezra Z.
[email protected]

5

On Mon, Dec 1, 2008 at 2:17 PM, Ezra Z.
[email protected]wrote:

[email protected]
Though it might be slow, I really don’t think it’s useless. Where I
work, we
are stuck in windows no matter the cost or benefit of another platform.
Because of this these windows numbers actually matter to me.

6

Post Tudbc wrote:

According to my own tests, “Thin” should be better than your current
best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby
win over everyone.

http://www.tudbc.org/tudbcjrubyservlet

You should also give more details about how you test.

I notice that you use the best of 5 runs to report numbers, which I
think it is going to vary a lot. You can use my free tool to do web
benchmarking, which I think is much more accurate.
http://www.tudbc.org/tudbcwebbenchmark

I think your Machine is about same as my old test machine. Here are some
numbers:

Using “Echo” test: (copied from http://www.tudbc.org/jrubywebapp)

0.98ms TudbcJRubyServlet_RB
1.73ms Thin
4.04ms Mongrel (estimated, because Mongrel crashed in the middle)
8.51ms WEBrick

Using your “time now+version” test:

1.11ms TudbcJRubyServlet_RB
1.56ms Thin
4.00ms Mongrel (estimated, because Mongrel crashed in the middle)
8.18ms WEBrick

As you can see in your “time now” test on your machine, your Mongrel
results were probably inaccurate, because you used the “best of 5” to
measure.

Both my “Echo” and your “Time now+version” tests seem to produce similar
benchmarks. As you can see, TudbcJRubyServlet_RB still win easily, and
Thin is faster than Mongrel by a wide margin.

Note: The default measurement of TudbcWebBenchmark is average of last 10
of 20 runs (with each run for 10 seconds). Because Mongrel crashed, so I
only estimate it by 3rd run of 5 seconds each.

Note: Ruby version used: JRuby 1.1.5 and Ruby 1.8.7-p72 on Windows

7

Oops! Forgot to paste the codes I used for TudbcJRubyServlet

--------- timeversion.rb --------
a = []
a << ‘Hello Rack!’
a << Time.new
a << RUBY_PLATFORM
a << RUBY_VERSION
ctx.print(a.join(’
’))

Your “time now+version” uses 5 array ops. My guess is that maybe JRuby
array processing is relatively not as efficient, so that ratio between
TudbcJRubyServlet and Thin for “time now+version” is not as great as
“Echo” test.

Note: My 4ms means your 1000/4 or 250 reqs/sec.