Straight Java/JDBC versus JRuby on Rails on JBoss Perf. Evaluation


#1

Hi All,
I owe my rather large company a performance evaluation of JRuby on Rails
versus a straightforward JDBC approach that we used on another project.
Please comment on my redacted report which I attach here in media wiki
format with attachements. My apologies for the formatting.

Since Java is the primary server side language at x, [John W.| x]
from
the x tools team offers this performance comparison. Please dispute or
add
comments about the results and John’s interpretations and workarounds.
If
you want the source code, email John.
h1. Overview Quick Overall
Results

  • Java is consistently 15x faster than most any dynamic language as
    proved
    by popular and respected micro benchmarks. Straight Java over JDBC code
    seems consistently 6x faster than the current JRuby on Rails without any
    optimization running on JBoss on Oracle on dev linux machines. Possible
    solutions are serving most of the data from memcached and for calls that
    are
    critical and can’t be cached easily (a fetch call to data that is
    constantly
    changing e.g.), re-writing the service in Java and deploying them in the
    same war along with the JRuby code. Also JRuby can call Java in some
    cases
    as well. By doing this the performance difference between the solutions
    can
    be minimized and the productivity benefits of JRuby on Rails can be
    maintained
    . This page will attempt to give an honest and broad
    (although
    not comprehensive) comparison of JRuby and Java when it comes to
    performance
    and scaleability. We will examine 3 areas: * Core Language Speed -
    using
    well known micro benchmarks * Web Application performance - the
    results
    and discussion of a simple simple CRUD test I wrote comparing JRuby on
    Rails
    to raw Java JDBC on top of Oracle running in JBoss. * Scaling Java
    versus
    JRuby
    - a discussion of of which one scales better. h1. Core Language
    Speed
    Let’s cut to the chase. Several benchmarks have shown that Java is for
    non-math oriented micro-benchmarks the fastest language on earth after
    C.
    None of the dynamic languages, Ruby, Python, PHP come close. I did not
    do my
    own micro-benchmarks because several are out there. One of most
    important
    and well respected micro-benchmark studies are the ever-running [Great
    Language shootout wars|http://shootout.alioth.debian.org/]. Yes, people
    there is war going on constantly out there in IT. From their website you
    can
    compare every language to almost any language on any platform. Let’s
    take a
    look at the results of JRuby versus Java from this [page|
    http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=jruby&lang2=lua]:
    White bar indicates which language is faster, black bar indicates which
    takes less memory. Each bar represents a key algorithm used in the
    shootout.
    !Picture 102.png|align=centre! So Java is consistently 15x
    faster
    than JRuby. Java is also faster than Python and every other dynamic
    language
    out there by similar amounts: !Picture 103.png|align=centre! Yes Python
    does
    a bit better than JRuby here but other recent [tests |
    http://blog.dhananjaynene.com/2008/07/performance-comparison-c-java-python-ruby-jython-jruby-groovy/]
    show JRuby closer. The point is that no dynamic language comes close to
    Java
    in performance. The only things that consistently beat it are GNU C and
    carefully written C++ (the Java compiler beats out a lot of C++
    programmers
    in a lot of cases). So in light of this why not use Java for everything
    at
    x? Well first of all remember, Facebook is written largely in PHP,
    Twitter
    in Ruby, Google and You-Tube were written largely in Python…highly
    performing things are written in dynamic languages. This brings up the
    point
    that there are other considerations when choosing a language. *
    Productivity for the problem domain - Different domains have different
    needs that make Java more or less productive. Java is a great general
    purpose language. E.g. game development requires lots of math
    calculation at
    which Java does not excel so many calculation intensive games have
    traditionally been written in C++ with the scripting done in Lua or
    other
    nicely embeddable scripting languages. Web development can benefit from
    the
    rich reg-expression support, dynamic objects and other features of
    python,
    php, and ruby. These languages although slower than Java provide more
    specialized tools for these domains. However languages can imitate each
    other. Java is becoming more dynamic with core scripting support and
    programmer friendly language constructs. * Memory Usage - if memory
    usage
    is a huge concern very lightweight programming may be needed in C++ or C
    where each object in memory can be controlled and removed when not
    needed. *
    Extreme Concurrency - Erlang and some of the other functional
    languages
    have been designed to help model real world concurrent problems. *
    Frameworks and Libraries available - Different languages have strong
    frameworks at different periods of time. At one time all the best web
    development frameworks were in PHP and Python or Java. Now Ruby on Rails
    is
    a very popular web framework. On projects where raw language performance
    is
    not an issue and where a framework can aid in rapid development, this
    factor
    can be a tipping point. These are just some of the other things to
    consider
    when looking at languages. However, let’s give Java it’s credit, it’s
    the
    way faster language in raw language performance (which is why JRuby is
    written on top of it). Now let’s zoom in on a specific comparison
    between
    JRuby and Java in a simple web oriented crud application. h1. Web
    Application Performance. Description - This test compared JRuby on
    Rails
    versus straight Java with JDBC on top of JBoss and Oracle. In particular
    a
    simple http get was perfomed retrieving one simple object called a
    workitem
    from Oracle. The workitem table has 3 attributes, an id (int), a name
    (string) and description (string). 1,000 requests were issued in
    sequence
    (not in parallel) to load the item and compare the difference in
    roundtrip
    time between JRuby and a straight Java solution. The test was run from a
    separate box (fb-linux1) [htttperf |
    http://www.hpl.hp.com/research/linux/httperf/]. The Java solution
    exposed a
    JSP which called one class that used straight JDBC to fetch the data.
    The
    JRuby solution used RoR and Active Record to load the data. In both
    cases
    the same JBoss datasource file was used to serve pooled Oracle
    connections.
    Overall Results - A straight Java JDBC solution is 6x faster than
    JRuby on
    Rails. * Java - 152.7 req/s finishing in 6.551 seconds. * JRuby -
    24.7
    req/s finishing in 40.501 seconds Conclusion - A straight Java
    solution is
    much faster. (This same perf difference of 6x bore itself out for
    different
    number of serial requests, 1, 5000, 10000 etc.) The RoR solution is
    running
    inside a framework with a lot more layers than the Java solution. There
    might be ways to optimize the JRuby solution to get a better result. A
    test
    that would be a bit more fair would be to compare Java/Hibernate with
    RoR
    JRuby so that you are testing equivalent functionality. Even then Java
    would
    be slower but by how much? Also much of the optimization in the web
    layer is
    about scaling out with caching, using memcached and other techniques.
    Also
    critical paths of services originally developed in JRuby on Rails can be
    rewritten in straight Java and deployed in the same war with little
    impact
    since the services are stateless and use the same JBoss JNDI datasource.
    Nevertheless, this test seems to clearly shows the performance penalty
    of
    going with the current JRuby on Rails solution over hand-crafted JDBC
    code.
    h3. Test Details. Test Tool - [htttperf |
    http://www.hpl.hp.com/research/linux/httperf/] JBoss Version - JBoss
    [Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417) h5.
    JRuby Details JRuby design - the Ruby on Rails app from Brainstorm.
    Deployed as war file to jboss * Rails version - 2.1 * JRuby version -
    jruby
    1.1.4 (ruby 1.8.6 patchlevel 114) (2008-08-28 rev 7570) [i386-java] *
    JRuby
    plugins - ActiveRecord-jdbc-adapter * Other notes - packaged with
    warbler
    JRuby httperf command - httperf --server localhost --port 8080 --uri
    /brainstorm/workitems/show/10031?format=xml --hog --num-conns 1000 *
    JRuby
    Raw Results !Picture 106.png|align=centre! h5. Straight Java on JDBC
    Details
  • Oracle Driver - ojdbc14_g.jar * Number of Java classes - 1 JSP and one
    Java. * ORM used - none * Java httperf command - httperf --server
    localhost
    –port 8080 --uri /bs_benchmark/get_work_item.jsp?id=10031 --hog
    –num-conns
    1000 *Java raw results !Picture 108.png|align=centre! h5. fb-linux2
    details
    This test was run on fb-linux2, a Redhat linux box with the following
    specs:
    CPU: HP DL360, Intel P4/Xeon 3.0GHz RAM: 2gb(4slots/2full), SWAP:
    8gb/sda
    DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto S/W:
    RedHat
    4WS (Nahant), 32-bit, LOADED: 6/12/08 s/w-all, locale -a/560 LOC:
    W07-509,
    OWNER: prakumar SN#: usm51405ds 1000-… During the tests 99.9% of the
    CPU
    was taken by httperf. Neither Java nor JRuby took much cpu or memory as
    no
    load was put on them. h1. Scaling Java vs JRuby Description - On x we
    typically run JBoss in a 10 node cluster behind a F5 load balancer. I
    could
    not replicate this test so I put 10 concurrent user load on both JRuby
    and
    JBoss using the --num-calls part of the httperf command issued from
    another
    box (fb-linux1) httperf --server 10.0.28.134 --port 8080 --uri
    /bs_benchmark/get_work_item.jsp?id=10031 --hog --num-conns 1000
    –num-calls
    10 Results - I got the exact same numbers as I did in the non-load
    tests,
    where Java was approx. 6x faster. This is because neither jboss or jruby
    really handle any load very well without a load balancer. Conclusion -
    Both JRuby and straight Java solutions will continue to need a load
    balancer
    proxy and caching solutions like memcached in order to scale out. Also,
    remember that JRuby is one of the fastest known Ruby implementations at
    this
    time and there are several sites on the web running Ruby (e.g. Twitter)
    that
    serve several times the enterprise population of x. There are ways to
    scale
    Ruby out and still serve a large load.

#2

Lots of content, and it’s late, so here’s a quick comment:

It’s entirely unsurprising that Java/JDBC is faster than JRuby/Rails,
but Rails developers generally don’t care. Most of the dynamic languages
are interesting not for performance reasons, but instead for ease of
development.

  • Charlie

John W. wrote:

workarounds. If you want the source code, email John._ h1. Overview
minimized and the productivity benefits of JRuby on Rails can be
dynamic languages, Ruby, Python, PHP come close. I did not do my own
shootout._ !Picture 102.png|align=centre! So Java is consistently 15x
in PHP, Twitter in Ruby, Google and You-Tube were written largely in
languages although slower than Java provide more specialized tools for
Java. Now Ruby on Rails is a very popular web framework. On projects
The workitem table has 3 attributes, an id (int), a name (string) and
finishing in 6.551 seconds. * JRuby - 24.7 req/s finishing in 40.501
in JRuby on Rails can be rewritten in straight Java and deployed in the
ActiveRecord-jdbc-adapter * Other notes - packaged with warbler *JRuby
8gb/sda DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto
–num-calls 10 Results - I got the exact same numbers as I did in the



To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#3

Yes, generally I think people are right when they say Ruby is “fast
enough”
for executing the MVC logic. In those cases where it isn’t, you need C
or
Java extensions (preferring the latter :wink:

I’m more concerned about the extremely high memory churn caused by Ruby
as a
dynamic language and particularily Rails. For instance looking at what
view
helper methods do in Rails makes me feel dizzy.

In contrast to the rather slow execution speed this can lead to a more
unpredictable runtime behaviour, challenging GC implementations.

Charles Oliver N.-2 wrote:

John W. wrote:

workarounds. If you want the source code, email John._ h1. Overview
minimized and the productivity benefits of JRuby on Rails can be
dynamic languages, Ruby, Python, PHP come close. I did not do my own
shootout._ !Picture 102.png|align=centre! So Java is consistently 15x
in PHP, Twitter in Ruby, Google and You-Tube were written largely in
languages although slower than Java provide more specialized tools for
Java. Now Ruby on Rails is a very popular web framework. On projects
The workitem table has 3 attributes, an id (int), a name (string) and
finishing in 6.551 seconds. * JRuby - 24.7 req/s finishing in 40.501
in JRuby on Rails can be rewritten in straight Java and deployed in the
ActiveRecord-jdbc-adapter * Other notes - packaged with warbler *JRuby
8gb/sda DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto
–num-calls 10 Results - I got the exact same numbers as I did in the

http://xircles.codehaus.org/manage_email


View this message in context:
http://www.nabble.com/Straight-Java-JDBC-versus-JRuby-on-Rails-on-JBoss-Perf.-Evaluation-tp20261058p20262231.html
Sent from the JRuby - User mailing list archive at Nabble.com.


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#4

John W. wrote:
-snip-

http://shootout.alioth.debian.org/

So Java is consistently 15x faster than JRuby.

Didn’t it seem strange that so many of the white bars should be exactly
15x?

As you can see from the table:

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=jruby&lang2=java14#ratio

: those measurements are off the chart - 24x 49x 101x 194x 88x 113x 26x
150x 16x 167x

As the JRuby measurements were made with Java 6 -server the appropriate
comparison is not ye olde Java 1.4 but Java 6 -server, and 12
measurements are off the chart faster: 20x 46x 102x 196x 109x 120x 43x
106x 10x 170x 21x 28x

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=jruby&lang2=java


#5

Like Charlie, I need to get back to this with some full attention…
there
are some assumptions that may be incorrect in your analysis.

Another thing to note, you mention some big traffic number sites using
dynamic languages… America Online uses/used the AOLServer which is
written
atop the TCL language (in C). The page logic and most modules are
written
in straight TCL. They have been doing the “dynamic language” thing
since
back when we used to call them scripting languages and of course the
sites
scaled just fine. It comes down to good design and “dipping down” to a
base
language when extra performance is needed. (common in this area)

When people say “fast enough”, I think the main meaning is how this
translates to the biz side of the house… hardware is usually cheaper
than
wetware!

software engineering time == $$$$$$$$
hardware processing time == $$$$

When things like time to market, cheaper manpower costs, relatively easy
O&M, quick/nimble new feature or bug fix turnaround are important to the
folks cutting the checks… then dynamic languages become an easier pill
to
swallow.

M

On Fri, Oct 31, 2008 at 5:10 AM, Christian S. removed_email_address@domain.invalid
wrote:

In contrast to the rather slow execution speed this can lead to a more

development.

easily (a fetch call to data that is constantly changing e.g.),
in JBoss. * Scaling Java versus JRuby - a discussion of of which one
[page|
other recent [tests |
brings up the point that there are other considerations when choosing a
becoming more dynamic with core scripting support and programmer
aid in rapid development, this factor can be a tipping point. These are
parallel) to load the item and compare the difference in roundtrip time
same perf difference of 6x bore itself out for different number of
same JBoss JNDI datasource. Nevertheless, this test seems to clearly
/brainstorm/workitems/show/10031?format=xml --hog --num-conns 1000 *
LOC: W07-509, OWNER: prakumar SN#: usm51405ds 1000-… During the
neither jboss or jruby really handle any load very well without a load


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


Aloha!

Mike McKinney
(http://blog.huikau.com)