Forum: Ruby Some benchmarks with MinGW, JRuby, MSVC...

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
A9b059eb8692215d448423606757ab09?d=identicon&s=25 Joao Pedrosa (jpedrosa)
on 2008-11-30 18:20
(Received via mailing list)
So, I have managed to build several different versions of Ruby to
compare all
in some benchmarks. I am going to let the numbers talk for themselves.
It was
all tested on an old desktop computer with Windows XP, so I am not
testing
anything too impressive.

Of importance is the seemingly poor performance of Ruby 1.8.7 for
whatever
reason. It's as if Ruby 1.8.7 were like Windows ME, a version between
major
versions, as an analogy. :-)

Also, I was not able to run the Mongrel gem on Ruby 1.9.1.

I am going to order the data by fastest to make it easier to compare
it on text.
I have collected the data on a Google Spreadsheet:
http://spreadsheets.google.com/ccc?key=pmtaDyNoBut...

I am going to keep it short here and for more detail on versions and
whatnot
please refer to the spreadsheet.

== WEBrick (req/s)

22.39 - jruby
16.95 - jruby --server
12.82 - ruby oneclick 1.8.6
12.49 - ruby msvc 1.8.6
12.45 - ruby rubyinstaller 1.8.6
12.43 - ruby mingw 1.8.6
11.94 - ruby msvc 1.9.1
11.48 - ruby mingw 1.9.1
3.42 - ruby msvc 1.8.7
3.40 - ruby mingw 1.8.7


== Mongrel (req/s)

357.14 - ruby mingw 1.8.6
347.45 - ruby rubyinstaller 1.8.6
326.70 - ruby msvc 1.8.6
313.57 - ruby oneclick 1.8.6
274.68 - ruby mingw 1.8.7
232.39 - ruby msvc 1.8.7
71.24 - jruby
62.22 - jruby --server
(crash when starting up) - ruby mingw 1.9.1
(error when building gem) - ruby msvc 1.9.1


== bench_pythag.rb (s)

19.09 - jruby --server
24.35 - jruby
31.72 - ruby mingw 1.9.1
31.91 - ruby mingw 1.8.6
32.73 - ruby mingw 1.8.7
33.94 - ruby msvc 1.9.1
35.67 - ruby rubyinstaller 1.8.6
35.83 - ruby msvc 1.8.7
39.79 - ruby msvc 1.8.6
48.30 - ruby oneclick 1.8.6


== bench_fractal.rb (s)

3.22 - jruby --server
3.89 - jruby
5.67 - ruby msvc 1.9.1
5.75 - ruby mingw 1.9.1
9.80 - ruby mingw 1.8.6
9.90 - ruby mingw 1.8.7
10.11 - ruby rubyinstaller 1.8.6
10.30 - ruby msvc 1.8.6
10.64 - ruby msvc 1.8.7
15.17 - ruby oneclick 1.8.6


== bench_float_math.rb (s)

103.44 - jruby --server
125.89 - jruby
158.64 - ruby msvc 1.9.1
175.41 - ruby mingw 1.9.1
176.61 - ruby mingw 1.8.6
179.94 - ruby mingw 1.8.7
184.73 - ruby rubyinstaller 1.8.6
195.37 - ruby msvc 1.8.7
197.39 - ruby msvc 1.8.6
279.62 - ruby oneclick 1.8.6


Since I collected these data, the Ruby 1.9.1 branch has had a number
of updates
for one thing. :-)

Benchmarks can be kind of tough to do right and there are all
possibilities when it
comes to doing them.

If anyone has an algorithm that he would like to test on all those
Ruby implementations
I would be glad to test it. With the exception of testing Rails, which
I have never used
and don't know enough about it to willingly make myself available when
testing it. :-)

Cheers,
Joao
A57b61a5b35b432421fdd6d6514c9ad7?d=identicon&s=25 Post Tudbc (tudbc)
on 2008-12-01 00:02
According to my own tests, "Thin" should be better than your current
best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby
win over everyone.

http://www.tudbc.org/tudbcjrubyservlet

You should also give more details about how you test.
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-01 01:19
(Received via mailing list)
Joao Pedrosa wrote:
> (crash when starting up) - ruby mingw 1.9.1
> (error when building gem) - ruby msvc 1.9.1

I'd love to figure out what makes mongrel slow for us, when all other
cases are faster ;)

GlassFish gem would be a nice one to test, but of course there's no
equivalent comparison for the others.

- Charlie
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2008-12-01 20:23
(Received via mailing list)
Benchmarking ruby on windows is fairly useless. MRI 1.8.* is known to
be very slow on windows and harldy anyone deploys production apps to
windows because of this. If you want your benches to be taken
seriously you need to run them on linux, bsd or osx where ruby
performs much better. Also make sure to compile ruby with --disable-
pthreads for best perf.


Cheers-
Ezra Zygmuntowicz
ez@engineyard.com
12a71a456ac3d464914a8267f11d43b3?d=identicon&s=25 Shane Emmons (Guest)
on 2008-12-01 20:49
(Received via mailing list)
On Mon, Dec 1, 2008 at 2:17 PM, Ezra Zygmuntowicz
<ezmobius@gmail.com>wrote:

> ez@engineyard.com
Though it might be slow, I really don't think it's useless. Where I
work, we
are stuck in windows no matter the cost or benefit of another platform.
Because of this these windows numbers actually matter to me.
A57b61a5b35b432421fdd6d6514c9ad7?d=identicon&s=25 Post Tudbc (tudbc)
on 2008-12-01 23:51
Post Tudbc wrote:
> According to my own tests, "Thin" should be better than your current
> best, which is Mongrel. Furthermore, TudbcJRubyServlet should help JRuby
> win over everyone.
>
> http://www.tudbc.org/tudbcjrubyservlet
>
> You should also give more details about how you test.

I notice that you use the best of 5 runs to report numbers, which I
think it is going to vary a lot. You can use my free tool to do web
benchmarking, which I think is much more accurate.
http://www.tudbc.org/tudbcwebbenchmark

I think your Machine is about same as my old test machine. Here are some
numbers:

Using "Echo" test: (copied from http://www.tudbc.org/jrubywebapp)

0.98ms TudbcJRubyServlet_RB
1.73ms Thin
4.04ms Mongrel (estimated, because Mongrel crashed in the middle)
8.51ms WEBrick


Using your "time now+version" test:

1.11ms TudbcJRubyServlet_RB
1.56ms Thin
4.00ms Mongrel (estimated, because Mongrel crashed in the middle)
8.18ms WEBrick

As you can see in your "time now" test on your machine, your Mongrel
results were probably inaccurate, because you used the "best of 5" to
measure.

Both my "Echo" and your "Time now+version" tests seem to produce similar
benchmarks. As you can see, TudbcJRubyServlet_RB still win easily, and
Thin is faster than Mongrel by a wide margin.

Note: The default measurement of TudbcWebBenchmark is average of last 10
of 20 runs (with each run for 10 seconds). Because Mongrel crashed, so I
only estimate it by 3rd run of 5 seconds each.

Note: Ruby version used: JRuby 1.1.5 and Ruby 1.8.7-p72 on Windows
A57b61a5b35b432421fdd6d6514c9ad7?d=identicon&s=25 Post Tudbc (tudbc)
on 2008-12-02 00:24
Oops! Forgot to paste the codes I used for TudbcJRubyServlet

--------- timeversion.rb --------
a = []
a << 'Hello Rack!'
a << Time.new
a << RUBY_PLATFORM
a << RUBY_VERSION
ctx.print(a.join('<br />'))
---------------------------------

Your "time now+version" uses 5 array ops. My guess is that maybe JRuby
array processing is relatively not as efficient, so that ratio between
TudbcJRubyServlet and Thin for "time now+version" is not as great as
"Echo" test.

Note: My 4ms means your 1000/4 or 250 reqs/sec.
This topic is locked and can not be replied to.