Hi All,
I owe my rather large company a performance evaluation of JRuby on Rails
versus a straightforward JDBC approach that we used on another project.
Please comment on my redacted report which I attach here in media wiki
format with attachements. My apologies for the formatting.
Since Java is the primary server side language at x, [John W.| x]
from
the x tools team offers this performance comparison. Please dispute or
add
comments about the results and John’s interpretations and workarounds.
If
you want the source code, email John. h1. Overview Quick Overall
Results
-
Java is consistently 15x faster than most any dynamic language as
proved
by popular and respected micro benchmarks. Straight Java over JDBC code
seems consistently 6x faster than the current JRuby on Rails without any
optimization running on JBoss on Oracle on dev linux machines. Possible
solutions are serving most of the data from memcached and for calls that
are
critical and can’t be cached easily (a fetch call to data that is
constantly
changing e.g.), re-writing the service in Java and deploying them in the
same war along with the JRuby code. Also JRuby can call Java in some
cases
as well. By doing this the performance difference between the solutions
can
be minimized and the productivity benefits of JRuby on Rails can be
maintained. This page will attempt to give an honest and broad
(although
not comprehensive) comparison of JRuby and Java when it comes to
performance
and scaleability. We will examine 3 areas: * Core Language Speed -
using
well known micro benchmarks * Web Application performance - the
results
and discussion of a simple simple CRUD test I wrote comparing JRuby on
Rails
to raw Java JDBC on top of Oracle running in JBoss. * Scaling Java
versus
JRuby - a discussion of of which one scales better. h1. Core Language
Speed
Let’s cut to the chase. Several benchmarks have shown that Java is for
non-math oriented micro-benchmarks the fastest language on earth after
C.
None of the dynamic languages, Ruby, Python, PHP come close. I did not
do my
own micro-benchmarks because several are out there. One of most
important
and well respected micro-benchmark studies are the ever-running [Great
Language shootout wars|http://shootout.alioth.debian.org/]. Yes, people
there is war going on constantly out there in IT. From their website you
can
compare every language to almost any language on any platform. Let’s
take a
look at the results of JRuby versus Java from this [page|
http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=jruby&lang2=lua]:
White bar indicates which language is faster, black bar indicates which
takes less memory. Each bar represents a key algorithm used in the
shootout. !Picture 102.png|align=centre! So Java is consistently 15x
faster
than JRuby. Java is also faster than Python and every other dynamic
language
out there by similar amounts: !Picture 103.png|align=centre! Yes Python
does
a bit better than JRuby here but other recent [tests |
http://blog.dhananjaynene.com/2008/07/performance-comparison-c-java-python-ruby-jython-jruby-groovy/]
show JRuby closer. The point is that no dynamic language comes close to
Java
in performance. The only things that consistently beat it are GNU C and
carefully written C++ (the Java compiler beats out a lot of C++
programmers
in a lot of cases). So in light of this why not use Java for everything
at
x? Well first of all remember, Facebook is written largely in PHP,
Twitter
in Ruby, Google and You-Tube were written largely in Python…highly
performing things are written in dynamic languages. This brings up the
point
that there are other considerations when choosing a language. *
Productivity for the problem domain - Different domains have different
needs that make Java more or less productive. Java is a great general
purpose language. E.g. game development requires lots of math
calculation at
which Java does not excel so many calculation intensive games have
traditionally been written in C++ with the scripting done in Lua or
other
nicely embeddable scripting languages. Web development can benefit from
the
rich reg-expression support, dynamic objects and other features of
python,
php, and ruby. These languages although slower than Java provide more
specialized tools for these domains. However languages can imitate each
other. Java is becoming more dynamic with core scripting support and
programmer friendly language constructs. * Memory Usage - if memory
usage
is a huge concern very lightweight programming may be needed in C++ or C
where each object in memory can be controlled and removed when not
needed. *
Extreme Concurrency - Erlang and some of the other functional
languages
have been designed to help model real world concurrent problems. *
Frameworks and Libraries available - Different languages have strong
frameworks at different periods of time. At one time all the best web
development frameworks were in PHP and Python or Java. Now Ruby on Rails
is
a very popular web framework. On projects where raw language performance
is
not an issue and where a framework can aid in rapid development, this
factor
can be a tipping point. These are just some of the other things to
consider
when looking at languages. However, let’s give Java it’s credit, it’s
the
way faster language in raw language performance (which is why JRuby is
written on top of it). Now let’s zoom in on a specific comparison
between
JRuby and Java in a simple web oriented crud application. h1. Web
Application Performance. Description - This test compared JRuby on
Rails
versus straight Java with JDBC on top of JBoss and Oracle. In particular
a
simple http get was perfomed retrieving one simple object called a
workitem
from Oracle. The workitem table has 3 attributes, an id (int), a name
(string) and description (string). 1,000 requests were issued in
sequence
(not in parallel) to load the item and compare the difference in
roundtrip
time between JRuby and a straight Java solution. The test was run from a
separate box (fb-linux1) [htttperf |
http://www.hpl.hp.com/research/linux/httperf/]. The Java solution
exposed a
JSP which called one class that used straight JDBC to fetch the data.
The
JRuby solution used RoR and Active Record to load the data. In both
cases
the same JBoss datasource file was used to serve pooled Oracle
connections.
Overall Results - A straight Java JDBC solution is 6x faster than
JRuby on
Rails. * Java - 152.7 req/s finishing in 6.551 seconds. * JRuby -
24.7
req/s finishing in 40.501 seconds Conclusion - A straight Java
solution is
much faster. (This same perf difference of 6x bore itself out for
different
number of serial requests, 1, 5000, 10000 etc.) The RoR solution is
running
inside a framework with a lot more layers than the Java solution. There
might be ways to optimize the JRuby solution to get a better result. A
test
that would be a bit more fair would be to compare Java/Hibernate with
RoR
JRuby so that you are testing equivalent functionality. Even then Java
would
be slower but by how much? Also much of the optimization in the web
layer is
about scaling out with caching, using memcached and other techniques.
Also
critical paths of services originally developed in JRuby on Rails can be
rewritten in straight Java and deployed in the same war with little
impact
since the services are stateless and use the same JBoss JNDI datasource.
Nevertheless, this test seems to clearly shows the performance penalty
of
going with the current JRuby on Rails solution over hand-crafted JDBC
code.
h3. Test Details. Test Tool - [htttperf |
http://www.hpl.hp.com/research/linux/httperf/] JBoss Version - JBoss
[Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417) h5.
JRuby Details JRuby design - the Ruby on Rails app from Brainstorm.
Deployed as war file to jboss * Rails version - 2.1 * JRuby version -
jruby
1.1.4 (ruby 1.8.6 patchlevel 114) (2008-08-28 rev 7570) [i386-java] *
JRuby
plugins - ActiveRecord-jdbc-adapter * Other notes - packaged with
warbler
JRuby httperf command - httperf --server localhost --port 8080 --uri
/brainstorm/workitems/show/10031?format=xml --hog --num-conns 1000 *
JRuby
Raw Results !Picture 106.png|align=centre! h5. Straight Java on JDBC
Details
- Oracle Driver - ojdbc14_g.jar * Number of Java classes - 1 JSP and one
Java. * ORM used - none * Java httperf command - httperf --server
localhost
–port 8080 --uri /bs_benchmark/get_work_item.jsp?id=10031 --hog
–num-conns
1000 *Java raw results !Picture 108.png|align=centre! h5. fb-linux2
details
This test was run on fb-linux2, a Redhat linux box with the following
specs:
CPU: HP DL360, Intel P4/Xeon 3.0GHz RAM: 2gb(4slots/2full), SWAP:
8gb/sda
DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto S/W:
RedHat
4WS (Nahant), 32-bit, LOADED: 6/12/08 s/w-all, locale -a/560 LOC:
W07-509,
OWNER: prakumar SN#: usm51405ds 1000-… During the tests 99.9% of the
CPU
was taken by httperf. Neither Java nor JRuby took much cpu or memory as
no
load was put on them. h1. Scaling Java vs JRuby Description - On x we
typically run JBoss in a 10 node cluster behind a F5 load balancer. I
could
not replicate this test so I put 10 concurrent user load on both JRuby
and
JBoss using the --num-calls part of the httperf command issued from
another
box (fb-linux1) httperf --server 10.0.28.134 --port 8080 --uri
/bs_benchmark/get_work_item.jsp?id=10031 --hog --num-conns 1000
–num-calls
10 Results - I got the exact same numbers as I did in the non-load
tests,
where Java was approx. 6x faster. This is because neither jboss or jruby
really handle any load very well without a load balancer. Conclusion -
Both JRuby and straight Java solutions will continue to need a load
balancer
proxy and caching solutions like memcached in order to scale out. Also,
remember that JRuby is one of the fastest known Ruby implementations at
this
time and there are several sites on the web running Ruby (e.g. Twitter)
that
serve several times the enterprise population of x. There are ways to
scale
Ruby out and still serve a large load.
