Forum: JRuby Straight Java/JDBC versus JRuby on Rails on JBoss Perf. Evaluation

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
C17855d3dc27b4bd2a4ed6920a48f629?d=identicon&s=25 John Wright (Guest)
on 2008-10-31 08:14
(Received via mailing list)
Attachment: Picture_102.png (20 KB)
Attachment: Picture_103.png (20 KB)
Attachment: Picture_106.png (100 KB)
Attachment: Picture_108.png (100 KB)
Hi All,
I owe my rather large company a performance evaluation of JRuby on Rails
versus a straightforward JDBC approach that we used on another project.
 Please comment on my redacted report which I attach here in media wiki
format with attachements.  My apologies for the formatting.

_Since Java is the primary server side language at x, [John Wright| x]
from
the x tools team offers this performance comparison. Please dispute or
add
comments about the results and John's interpretations and workarounds.
If
you want the source code, email John._ h1. Overview *Quick Overall
Results*
- _Java is consistently 15x faster than most any dynamic language as
proved
by popular and respected micro benchmarks. Straight Java over JDBC code
seems consistently 6x faster than the current JRuby on Rails without any
optimization running on JBoss on Oracle on dev linux machines. Possible
solutions are serving most of the data from memcached and for calls that
are
critical and can't be cached easily (a fetch call to data that is
constantly
changing e.g.), re-writing the service in Java and deploying them in the
same war along with the JRuby code. Also JRuby can call Java in some
cases
as well. By doing this the performance difference between the solutions
can
be minimized and the productivity benefits of JRuby on Rails can be
maintained_. This page will attempt to give an honest and broad
(although
not comprehensive) comparison of JRuby and Java when it comes to
performance
and scaleability. We will examine 3 areas: * *Core Language Speed* -
using
well known micro benchmarks * *Web Application performance* - the
results
and discussion of a simple simple CRUD test I wrote comparing JRuby on
Rails
to raw Java JDBC on top of Oracle running in JBoss. * *Scaling Java
versus
JRuby* - a discussion of of which one scales better. h1. Core Language
Speed
Let's cut to the chase. Several benchmarks have shown that Java is for
non-math oriented micro-benchmarks the fastest language on earth after
C.
None of the dynamic languages, Ruby, Python, PHP come close. I did not
do my
own micro-benchmarks because several are out there. One of most
important
and well respected micro-benchmark studies are the ever-running [Great
Language shootout wars|http://shootout.alioth.debian.org/]. Yes, people
there is war going on constantly out there in IT. From their website you
can
compare every language to almost any language on any platform. Let's
take a
look at the results of JRuby versus Java from this [page|
http://shootout.alioth.debian.org/gp4/benchmark.ph...
_White bar indicates which language is faster, black bar indicates which
takes less memory. Each bar represents a key algorithm used in the
shootout._ !Picture 102.png|align=centre! So Java is consistently 15x
faster
than JRuby. Java is also faster than Python and every other dynamic
language
out there by similar amounts: !Picture 103.png|align=centre! Yes Python
does
a bit better than JRuby here but other recent [tests |
http://blog.dhananjaynene.com/2008/07/performance-...]
show JRuby closer. The point is that no dynamic language comes close to
Java
in performance. The only things that consistently beat it are GNU C and
carefully written C++ (the Java compiler beats out a lot of C++
programmers
in a lot of cases). So in light of this why not use Java for everything
at
x? Well first of all remember, Facebook is written largely in PHP,
Twitter
in Ruby, Google and You-Tube were written largely in Python...highly
performing things are written in dynamic languages. This brings up the
point
that there are other considerations when choosing a language. *
*Productivity for the problem domain* - Different domains have different
needs that make Java more or less productive. Java is a great general
purpose language. E.g. game development requires lots of math
calculation at
which Java does not excel so many calculation intensive games have
traditionally been written in C++ with the scripting done in Lua or
other
nicely embeddable scripting languages. Web development can benefit from
the
rich reg-expression support, dynamic objects and other features of
python,
php, and ruby. These languages although slower than Java provide more
specialized tools for these domains. However languages can imitate each
other. Java is becoming more dynamic with core scripting support and
programmer friendly language constructs. * *Memory Usage* - if memory
usage
is a huge concern very lightweight programming may be needed in C++ or C
where each object in memory can be controlled and removed when not
needed. *
*Extreme Concurrency* - Erlang and some of the other functional
languages
have been designed to help model real world concurrent problems. *
*Frameworks and Libraries available* - Different languages have strong
frameworks at different periods of time. At one time all the best web
development frameworks were in PHP and Python or Java. Now Ruby on Rails
is
a very popular web framework. On projects where raw language performance
is
not an issue and where a framework can aid in rapid development, this
factor
can be a tipping point. These are just some of the other things to
consider
when looking at languages. However, let's give Java it's credit, it's
the
way faster language in raw language performance (which is why JRuby is
written on top of it). Now let's zoom in on a specific comparison
between
JRuby and Java in a simple web oriented crud application. h1. Web
Application Performance. *Description* - This test compared JRuby on
Rails
versus straight Java with JDBC on top of JBoss and Oracle. In particular
a
simple http get was perfomed retrieving one simple object called a
workitem
from Oracle. The workitem table has 3 attributes, an id (int), a name
(string) and description (string). 1,000 requests were issued in
sequence
(not in parallel) to load the item and compare the difference in
roundtrip
time between JRuby and a straight Java solution. The test was run from a
separate box (fb-linux1) [htttperf |
http://www.hpl.hp.com/research/linux/httperf/]. The Java solution
exposed a
JSP which called one class that used straight JDBC to fetch the data.
The
JRuby solution used RoR and Active Record to load the data. In both
cases
the same JBoss datasource file was used to serve pooled Oracle
connections.
*Overall Results* - A straight Java JDBC solution is 6x faster than
JRuby on
Rails. * *Java* - 152.7 req/s finishing in 6.551 seconds. * *JRuby* -
24.7
req/s finishing in 40.501 seconds *Conclusion* - A straight Java
solution is
much faster. (This same perf difference of 6x bore itself out for
different
number of serial requests, 1, 5000, 10000 etc.) The RoR solution is
running
inside a framework with a lot more layers than the Java solution. There
might be ways to optimize the JRuby solution to get a better result. A
test
that would be a bit more fair would be to compare Java/Hibernate with
RoR
JRuby so that you are testing equivalent functionality. Even then Java
would
be slower but by how much? Also much of the optimization in the web
layer is
about scaling out with caching, using memcached and other techniques.
Also
critical paths of services originally developed in JRuby on Rails can be
rewritten in straight Java and deployed in the same war with little
impact
since the services are stateless and use the same JBoss JNDI datasource.
Nevertheless, this test seems to clearly shows the performance penalty
of
going with the current JRuby on Rails solution over hand-crafted JDBC
code.
h3. Test Details. *Test Tool* - [htttperf |
http://www.hpl.hp.com/research/linux/httperf/] *JBoss Version* - JBoss
[Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181417) h5.
JRuby Details *JRuby design* - the Ruby on Rails app from Brainstorm.
Deployed as war file to jboss * Rails version - 2.1 * JRuby version -
jruby
1.1.4 (ruby 1.8.6 patchlevel 114) (2008-08-28 rev 7570) [i386-java] *
JRuby
plugins - ActiveRecord-jdbc-adapter * Other notes - packaged with
warbler
*JRuby httperf command* - httperf --server localhost --port 8080 --uri
/brainstorm/workitems/show/10031?format=xml --hog --num-conns 1000 *
JRuby
Raw Results !Picture 106.png|align=centre! h5. Straight Java on JDBC
Details
* Oracle Driver - ojdbc14_g.jar * Number of Java classes - 1 JSP and one
Java. * ORM used - none * Java httperf command - httperf --server
localhost
--port 8080 --uri /bs_benchmark/get_work_item.jsp?id=10031 --hog
--num-conns
1000 *Java raw results !Picture 108.png|align=centre! h5. fb-linux2
details
This test was run on fb-linux2, a Redhat linux box with the following
specs:
CPU: HP DL360, Intel P4/Xeon 3.0GHz RAM: 2gb(4slots/2full), SWAP:
8gb/sda
DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto S/W:
RedHat
4WS (Nahant), 32-bit, LOADED: 6/12/08 s/w-all, locale -a/560 LOC:
W07-509,
OWNER: prakumar SN#: usm51405ds 1000-..... During the tests 99.9% of the
CPU
was taken by httperf. Neither Java nor JRuby took much cpu or memory as
no
load was put on them. h1. Scaling Java vs JRuby *Description* - On x we
typically run JBoss in a 10 node cluster behind a F5 load balancer. I
could
not replicate this test so I put 10 concurrent user load on both JRuby
and
JBoss using the --num-calls part of the httperf command issued from
another
box (fb-linux1) httperf --server 10.0.28.134 --port 8080 --uri
/bs_benchmark/get_work_item.jsp?id=10031 --hog --num-conns 1000
--num-calls
10 *Results* - I got the exact same numbers as I did in the non-load
tests,
where Java was approx. 6x faster. This is because neither jboss or jruby
really handle any load very well without a load balancer. *Conclusion* -
Both JRuby and straight Java solutions will continue to need a load
balancer
proxy and caching solutions like memcached in order to scale out. Also,
remember that JRuby is one of the fastest known Ruby implementations at
this
time and there are several sites on the web running Ruby (e.g. Twitter)
that
serve several times the enterprise population of x. There are ways to
scale
Ruby out and still serve a large load.
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-10-31 09:35
(Received via mailing list)
Lots of content, and it's late, so here's a quick comment:

It's entirely unsurprising that Java/JDBC is faster than JRuby/Rails,
but Rails developers generally don't care. Most of the dynamic languages
are interesting not for performance reasons, but instead for ease of
development.

- Charlie

John Wright wrote:
> workarounds. If you want the source code, email John._ h1. Overview
> minimized and the productivity benefits of JRuby on Rails can be
> dynamic languages, Ruby, Python, PHP come close. I did not do my own
> shootout._ !Picture 102.png|align=centre! So Java is consistently 15x
> in PHP, Twitter in Ruby, Google and You-Tube were written largely in
> languages although slower than Java provide more specialized tools for
> Java. Now Ruby on Rails is a very popular web framework. On projects
> The workitem table has 3 attributes, an id (int), a name (string) and
> finishing in 6.551 seconds. * *JRuby* - 24.7 req/s finishing in 40.501
> in JRuby on Rails can be rewritten in straight Java and deployed in the
> ActiveRecord-jdbc-adapter * Other notes - packaged with warbler *JRuby
> 8gb/sda DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto
> --num-calls 10 *Results* - I got the exact same numbers as I did in the
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
3c3fe2b74a5a634c9b191f9f6ccbf000?d=identicon&s=25 Christian Seiler (Guest)
on 2008-10-31 10:16
(Received via mailing list)
Yes, generally I think people are right when they say Ruby is "fast
enough"
for executing the MVC logic. In those cases where it isn't, you need C
or
Java extensions (preferring the latter ;-)

I'm more concerned about the extremely high memory churn caused by Ruby
as a
dynamic language and particularily Rails. For instance looking at what
view
helper methods do in Rails makes me feel dizzy.

In contrast to the rather slow execution speed this can lead to a more
unpredictable runtime behaviour, challenging GC implementations.


Charles Oliver Nutter-2 wrote:
> John Wright wrote:
>> workarounds. If you want the source code, email John._ h1. Overview
>> minimized and the productivity benefits of JRuby on Rails can be
>> dynamic languages, Ruby, Python, PHP come close. I did not do my own
>> shootout._ !Picture 102.png|align=centre! So Java is consistently 15x
>> in PHP, Twitter in Ruby, Google and You-Tube were written largely in
>> languages although slower than Java provide more specialized tools for
>> Java. Now Ruby on Rails is a very popular web framework. On projects
>> The workitem table has 3 attributes, an id (int), a name (string) and
>> finishing in 6.551 seconds. * *JRuby* - 24.7 req/s finishing in 40.501
>> in JRuby on Rails can be rewritten in straight Java and deployed in the
>> ActiveRecord-jdbc-adapter * Other notes - packaged with warbler *JRuby
>> 8gb/sda DISK: 160gb/sda, 24X cdrom/hda NET: eth0: 1000 Mbps full/auto
>> --num-calls 10 *Results* - I got the exact same numbers as I did in the
>> ------------------------------------------------------------------------
>>
>
>     http://xircles.codehaus.org/manage_email
>
>
>
>

--
View this message in context:
http://www.nabble.com/Straight-Java-JDBC-versus-JR...
Sent from the JRuby - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
3e412ab42690791643b657e80e499ab0?d=identicon&s=25 Mike McKinney (Guest)
on 2008-10-31 14:51
(Received via mailing list)
Like Charlie, I need to get back to this with some full attention...
there
are some assumptions that may be incorrect in your analysis.

Another thing to note, you mention some big traffic number sites using
dynamic languages... America Online uses/used the AOLServer which is
written
atop the TCL language (in C).  The page logic and most modules are
written
in straight TCL.    They have been doing the "dynamic language" thing
since
back when we used to call them scripting languages and of course the
sites
scaled just fine.  It comes down to good design and "dipping down" to a
base
language when extra performance is needed. (common in this area)

When people say "fast enough", I think the main meaning is how this
translates to the biz side of the house...  hardware is usually cheaper
than
wetware!

software engineering time == $$$$$$$$
hardware processing time == $$$$

When things like time to market, cheaper manpower costs, relatively easy
O&M, quick/nimble new feature or bug fix turnaround are important to the
folks cutting the checks... then dynamic languages become an easier pill
to
swallow.

M

On Fri, Oct 31, 2008 at 5:10 AM, Christian Seiler <chr.seiler@web.de>
wrote:

> In contrast to the rather slow execution speed this can lead to a more
> > development.
> >>
> >> easily (a fetch call to data that is constantly changing e.g.),
> >> in JBoss. * *Scaling Java versus JRuby* - a discussion of of which one
> >> [page|
> >> other recent [tests |
> >> brings up the point that there are other considerations when choosing a
> >> becoming more dynamic with core scripting support and programmer
> >> aid in rapid development, this factor can be a tipping point. These are
> >> parallel) to load the item and compare the difference in roundtrip time
> >> same perf difference of 6x bore itself out for different number of
> >> same JBoss JNDI datasource. Nevertheless, this test seems to clearly
> >> /brainstorm/workitems/show/10031?format=xml --hog --num-conns 1000 *
> >> LOC: W07-509, OWNER: prakumar SN#: usm51405ds 1000-..... During the
> >> neither jboss or jruby really handle any load very well without a load
> >>
> >>
> >
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>


--
Aloha!

Mike McKinney
(http://blog.huikau.com)
F79a7909dfca0088f4fdc01f109f497e?d=identicon&s=25 Isaac Gouy (igouy)
on 2008-10-31 20:59
John Wright wrote:
-snip-

> http://shootout.alioth.debian.org/

> So Java is consistently 15x faster than JRuby.

Didn't it seem strange that so many of the white bars should be exactly
15x?

As you can see from the table:

http://shootout.alioth.debian.org/gp4/benchmark.ph...

: those measurements are off the chart - 24x 49x 101x 194x 88x 113x 26x
150x 16x 167x


As the JRuby measurements were made with Java 6 -server the appropriate
comparison is not ye olde Java 1.4 but Java 6 -server, and 12
measurements are off the chart faster: 20x 46x 102x 196x 109x 120x 43x
106x 10x 170x 21x 28x

http://shootout.alioth.debian.org/gp4/benchmark.ph...
This topic is locked and can not be replied to.