Rails war performance

matthew · May 6, 2008, 6:23pm

Hi all.

IÂ¹ve been running a bunch of performance tests comparing JRails deployed
in
JBoss to Spring (a Java MVC framework), and the results IÂ¹ve gotten
arenÂ¹t
good for JRuby. IÂ¹d really like to be able to start moving towards
Rails,
but I there are a few deal-breakers IÂ¹d like to figure out. First, let
me
describe my environment:

Dev:
Mac OS X Leopard
Java 1.5.0_13

Prod:
Centos 5
JBoss 3.2.7 / Jetty 4.2.24

3G RAM and 256M PermGen space
-server -Djruby.compilation.mode=FORCE
-Djruby.objectspace.enabled=false
-Djruby.thread.pooling=true
war configured to start 10 JRails runtimes
Java 1.5.0_14

JRuby/Rails:
JRuby 1.1.1
Rails 2.0.2
Warbler 0.9.5

My webapp is very simple. ItÂ¹s a three page drilldown that uses a
native
Java DB class (shared between the JRails and Spring implementations) to
call stored procs in an Oracle DB.

IÂ¹m using Grinder to make repeated requests with increasing numbers of
threads, from 1 to 50 in increments of 5. At each thread count, IÂ¹m
making
about 150 requests (50 to each type of page).

What I see from the client side is many responses returning in a
reasonable
amount of time (the average request time under heavy load is around
1-1.5
seconds, but many requests return in 100-200 milliseconds), but many
threads
seemingly blocking for a long time (the standard deviation on request
time
is as high as 30 seconds).

On the server side, Rails reports some requests being processed for as
long
as a few seconds (as many as 5 or 6), but never for as many as 30. The
JVM
is also garbage collecting like crazy from the moment JRuby starts. I
suspect that the long delays are a result of threads being blocked
waiting
for garbage collecting, but IÂ¹m not positive. I eliminated the DB code
as a
potential cause by running the same suite of tests on a simple Â³Hello
worldÂ²
app, and got approximately the same performance results (Spring shows
about
a 5-fold increase in throughput).

My assumption is that the constant overhead of garbage collection is the
bottleneck, but IÂ¹m by no means convinced. IÂ¹ve read that IBM and
ThoughtWorks have deployed JRails apps in similar environments, so my
results are confusing and frustrating.

Here are my questions:

Is there a way to tame the garbage collection? I suspect that
objects
are being created and deleted with each request rather than being
reused.
Can Rails be fully compiled to Java? Can a Rails app? It seems like
thatÂ¹s a goal of jrubyc, but is it possible with the current version?
Am I missing some bit of configuration in the war deployment that
will
tell JRuby to be more conservative with object creation?

Sorry for the long email. Thanks for taking the time to read!

-Matt

matthew · May 6, 2008, 7:39pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt S. wrote:

| 3. Am I missing some bit of configuration in the war deployment that will
| tell JRuby to be more conservative with object creation?

Is Rails configured for production or the development environment?

In development, nothing gets cached by Rails (like HTML), and gets
regenerated every time the view is called, which, obviously, kills
performance.

Of course, if Rails is in its production environment, disregard this.

Though in such a case it might be worthwhile to prod into JBoss’s
caching mechanisms.

Also, Rails thread’s are blocking, too, on every request, so it isn’t
necessarily the GC that throws you off, but memory constrains, or so.

Phillip G.
Twitter: twitter.com/cynicalryan
Blog: http://justarubyist.blogspot.com

~ You know what we need, Hobbes? We need an attitude. Yeah, you can’t
be cool if you don’t have an attitude. – Calvin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkggl4YACgkQbtAgaoJTgL/wpACgn8I1pyEB2eT2fbsMke7Cq1lw
PZwAoKW8rPHeJVWgUaPWzDRshE3lC8nq
=ESnA
-----END PGP SIGNATURE-----

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 6, 2008, 8:17pm

On 5/6/08 1:38 PM, “Phillip G.” [email protected]
wrote:

Is Rails configured for production or the development environment?

In development, nothing gets cached by Rails (like HTML), and gets
regenerated every time the view is called, which, obviously, kills
performance.

Of course, if Rails is in its production environment, disregard this.

Rails is configured for the production environment.

Also, Rails thread’s are blocking, too, on every request, so it isn’t
necessarily the GC that throws you off, but memory constrains, or so.

I have Rails configured for a pool of 10 runtimes, so I guess that means
it
can only handle as many as 10 concurrent threads and the rest are
blocking.
That would explain some of the long response times. Memory constraints
are
not a big issue (I have 4G available for Jboss), but even 10 runtimes
under
load have resident memory usage at about 1.2G, so I can get up to 20 or
25
runtimes, but that’s not even close to what a native Java app can handle
(our traditional Jboss apps are configured for a pool of 300 threads).
Is
there any other way to scale up other than just adding more app servers?
I’ve hear my share of “rails doesn’t scale” comments. Is this what
they’re
referring to?

I’m not convinced that the small number of threads is the full answer.
Even
at 10 threads, the Spring app handily outperformed the JRails app. I’m
willing to accept that JRuby isn’t quite up to the speed of native Java
yet,
but it’s the size of the gap that makes me think I might be missing
something. Has anyone else tried this kind of deployment and would like
to
compare notes?

-Matt

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 6, 2008, 8:52pm

I’m seeing about a 5x increase in throughput (requests/second) with
Spring,
which is more than I expected. I’m going to see what performance looks
like
in Tomcat. Jboss is a bit of a heavyweight. Thanks for all the help so
far.

Matt

On 5/6/08 2:38 PM, “Phillip G.” [email protected]
wrote:

exception.

like to
How big is the gap you are seeing?
-----BEGIN PGP SIGNATURE-----
http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 6, 2008, 8:38pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt S. wrote:

| I have Rails configured for a pool of 10 runtimes, so I guess that
means it
| can only handle as many as 10 concurrent threads and the rest are
blocking.

That is my understanding, too. I;m not a Rails wizard, though.

| That would explain some of the long response times. Memory
constraints are
| not a big issue (I have 4G available for Jboss), but even 10 runtimes
under
| load have resident memory usage at about 1.2G, so I can get up to 20 or 25
| runtimes, but that’s not even close to what a native Java app can handle
| (our traditional Jboss apps are configured for a pool of 300 threads). Is
| there any other way to scale up other than just adding more app servers?

Well, you could use JRuby’s JIT, I guess (I have no idea how to enable
this, but seems to be on already).

http://wiki.jruby.org/wiki/JRuby_Compiler

This should provide you with more information than I could at this
stage.

| I’ve hear my share of “rails doesn’t scale” comments. Is this what
they’re
| referring to?

Sort of, yes. However, adding more application servers is one option.
Another would be to farm out heavy lifting you do in controllers (or was
it models?) to Java (or C in case you’d be using the MRI).

Another approach would be using more threads (I don’t know how JRuby
uses memory), if you don’t do stuff requiring large amounts of memory.

I haven’t checked how much RAM my Rails development environment
swallows, but it can’t be no where near the number you report, as I
don’t have that amount of RAM, and the JRE would throw an OutOfMemory
exception.

Other possible issues:

- JBoss does some odd stuff, and introduces a lot of overhead itself
  (I’ve noticed that JBoss uses a lot of memory itself, but that was 3
  years ago, and a data mining specific application, so take that with a
  grain of salt!)
- The JRE you use doesn’t optimize as good as, say the Sun JRE, or the
  Sun JRE allocates more memory than needed (I have no experience with
  other JREs than Sun’s).
- Ruby isn’t all that good with memory consumption in general.

| I’m not convinced that the small number of threads is the full answer.
Even
| at 10 threads, the Spring app handily outperformed the JRails app. I’m
| willing to accept that JRuby isn’t quite up to the speed of native
Java yet,
| but it’s the size of the gap that makes me think I might be missing
| something. Has anyone else tried this kind of deployment and would
like to
| compare notes?

Well, Spring is a Java framework, where Rails is a Ruby framework, and
JRuby is implemented on top of Java. So you have the overhead of Java,
JRuby and Rails at the same time. That doesn’t, however, translate to,
say, 3 times the overhead.

But that a Java specific application outperforms a JRuby application by
a big margin isn’t all that surprising to me.

How big is the gap you are seeing?

Phillip G.
Twitter: twitter.com/cynicalryan
Blog: http://justarubyist.blogspot.com

:zorkmid: /zork’mid/ n. The canonical unit of currency in
~ hacker-written games. This originated in {zork} but has spread
~ to {nethack} and is referred to in several other games.
~ – The AI Hackers Dictionary
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkggpZkACgkQbtAgaoJTgL+rqQCbBQeNFWmPSaqX7Zc6JXZDiy77
f/MAn0Jjc5Htt+L0cMjKXldf+nX+jYe/
=igoS
-----END PGP SIGNATURE-----

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 9:04am

On Tue, May 6, 2008 at 9:22 AM, Matt S. [email protected]
wrote:

Mac OS X Leopard
Java 1.5.0_13

Prod:
Centos 5
JBoss 3.2.7 / Jetty 4.2.24

3G RAM and 256M PermGen space

-server -Djruby.compilation.mode=FORCE -Djruby.objectspace.enabled=false

Try with just “-server”. compile = FORCE will probably only slow
things down, as ahead-of-time compiler is not always generating
optimal code (yet). objectspace is off by default with 1.1.x.

-Djruby.thread.pooling=true

Shouldn’t need thread pooling either, to start.

war configured to start 10 JRails runtimes

You should have at least as many runtimes as you plan to bench
concurrent users/transactions, otherwise you’re not going to see very
good scaling curves.

What I see from the client side is many responses returning in a reasonable
amount of time (the average request time under heavy load is around 1-1.5
seconds, but many requests return in 100-200 milliseconds), but many threads
seemingly blocking for a long time (the standard deviation on request time
is as high as 30 seconds).

Contention for a runtime is one reason, but you might also be getting
bit by a regexp cache bug that we recently fixed in trunk. You might
want to give trunk a try if you can – 1.1.2 will be out in a couple
weeks.

On our project, we have not seen this behavior. Generally requests
scale well up to 100 concurrent users, and performance/latency is
generally on par with matzruby. Of course, performance is always going
to be application-dependent.

On the server side, Rails reports some requests being processed for as long
as a few seconds (as many as 5 or 6), but never for as many as 30. The JVM
is also garbage collecting like crazy from the moment JRuby starts. I
suspect that the long delays are a result of threads being blocked waiting
for garbage collecting, but I’m not positive. I eliminated the DB code as a
potential cause by running the same suite of tests on a simple “Hello world”
app, and got approximately the same performance results (Spring shows about
a 5-fold increase in throughput).

You might want to open JConsole and connect it to your app server’s VM
to confirm or deny your garbage collection suspicion.

Is there a way to tame the garbage collection? I suspect that objects are
being created and deleted with each request rather than being reused.
Can Rails be fully compiled to Java? Can a Rails app? It seems like that’s
a goal of jrubyc, but is it possible with the current version?
Am I missing some bit of configuration in the war deployment that will tell
JRuby to be more conservative with object creation?

You should confirm that it’s garbage collection. You might also want
to try a memory profiler if you have access to one, or maybe just run
with -verbose:gc.

/Nick

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 12:33am

Matt,

I have a JRuby app running under JBoss 4.2.

I don’t think your problem is with JBoss itself…V4 uses Tomcat anyway
internally and I don’t believe they’ve hacked it that much. My
installation has a lot of the other JBoss stuff stripped out of the
config anyway.

Likewise I don’t think your JVM or GC is the problem either.

I haven’t actually compared the ‘same app’ side between a JRuby version
and a Java version, but I also find JRuby seems to be a lot slower
overall. And yes, the memory consumption is pretty heavy too. Because
the rails model is to just duplicate the process cgi-style to get more
listeners, (since ruby is not multithreaded), maybe the translation of
this pattern into Jruby/goldspike isn’t very efficient. In fact I wonder
if each process ends up being loaded under its own classloader so that
all the code is duplicated in memory?

One thing: you said you were using a database connection for your
test…what is the form of your DB connectivity on the JRuby side?

Have you thought about just doing native Ruby with Apache? If you still
need access to Java code perhaps you could use some sort of remoting
technique or other form of layering.

…Milt

Matt S. wrote:

I’m seeing about a 5x increase in throughput (requests/second) with Spring,
which is more than I expected. I’m going to see what performance looks like
in Tomcat. Jboss is a bit of a heavyweight. Thanks for all the help so
far.

Matt

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 9:25am

http://jira.codehaus.org/browse/JRUBY-2483 is the issue I filed. If
you have threads that seem to be locked up, send a “kill -QUIT” to the
process to inspect the thread stack traces.

/Nick

On Wed, May 7, 2008 at 12:15 AM, [email protected] wrote:

On our project, we have not seen this behavior. Generally requests
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 9:22am

Nick, do you have a JIRA issue that is tagged to this problem? I would
like to understand more about how this impacts a typical war app under
jruby…

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 4:42pm

First, thanks all for helping me get to the bottom of this.

On 5/7/08 3:04 AM, “Nick S.” [email protected] wrote:

On Tue, May 6, 2008 at 9:22 AM, Matt S. [email protected] wrote:

war configured to start 10 JRails runtimes

You should have at least as many runtimes as you plan to bench
concurrent users/transactions, otherwise you’re not going to see very
good scaling curves.

I’d do this, except the memory costs is prohibitive. I ramped up to 25
runtimes, and under heavy load, memory usage was heading past 2G. By
contrast, the Spring equivalent is configured with a pool of 300
response
handlers, and it fits tidily within 512M. Are you not seeing the high
memory usage? What does your deployed environment look like?

to confirm or deny your garbage collection suspicion.
to try a memory profiler if you have access to one, or maybe just run
with -verbose:gc.

I was alerted to the GC issue when tailing the logs with -verbose:gc.
JRails was completing only around 5 requests between each GC. I took a
look
at a heap dump from a high load period, and I saw a surprising number of
orphaned objects.

On 5/6/08 6:32 PM, “[email protected]” [email protected] wrote:

One thing: you said you were using a database connection for your
test…what is the form of your DB connectivity on the JRuby side?

It’s a pure java class that uses the Oracle thin client and returns a
JSON
object. We encapsulated the DB stuff in Java so we could use it the
same
way in Rails and Spring.

Matt

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 5:28pm

Hi!

On Tue, May 06, 2008 at 12:22:50PM -0400, Matt S. wrote:
[…]

for garbage collecting, but I¹m not positive. I eliminated the DB code as a
potential cause by running the same suite of tests on a simple ³Hello world²
app, and got approximately the same performance results (Spring shows about
a 5-fold increase in throughput).

You didnt mention mod_jk being part of your setup, but we had exactly
the same behaviour in our production environment with JBoss, Apache +
mod_jk.

The fix was to set the mod_jk recycle_timeout parameter to a high value
(300 worked well for us, while 30 didn’t). Not setting it at all but
leaving it alone might work too, but I never got a chance to try this
out since I have no direct access to this production environment
unfortunately.

Besides that, comparing the performance of a plain spring webapp with a
Rails one feels a bit like comparing apples with oranges to me. In any
performance testing I did with database intensive jruby/rails apps, its
performance was on the very same level as MRI/Rails (that is, after an
initial warmup period of several 100 requests). Of course you can get
better performance with plain servlets or something that comes near to
plain servlets.

Before going online our jruby/rails application has been load tested by
some third party company with up to 60 concurrent virtual clients (which
not only issued plain get requests on pages but also loaded images and
stuff like a real browser would, which matters in this case since assets
are delivered from oracle blobs by this rails app, too) and never showed
response times longer than 3 seconds during several hours of testing.
Unfortunately I don’t have the exact numbers, 3 seconds was the upper
margin that must not be hit in order to pass this test…

The app (www.medienservice.sachsen.de in case youre interested what im
talking about here) runs on a cluster of 2 virtualized app servers with
6
Rails workers per server and an Xmx setting of 1024m.

Cheers,
Jens

–
Jens Krämer
Finkenlust 14, 06449 Aschersleben, Germany
VAT Id DE251962952
http://www.jkraemer.net/ - Blog
http://www.omdb.org/ - The new free film database

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 7, 2008, 8:05pm

On Wed, May 7, 2008 at 7:41 AM, Matt S. [email protected]
wrote:

concurrent users/transactions, otherwise you’re not going to see very
good scaling curves.

I’d do this, except the memory costs is prohibitive. I ramped up to 25
runtimes, and under heavy load, memory usage was heading past 2G. By
contrast, the Spring equivalent is configured with a pool of 300 response
handlers, and it fits tidily within 512M. Are you not seeing the high
memory usage? What does your deployed environment look like?

I agree, the memory situation is not the best, but compared to
matzruby, JRuby deployments are actually favorable. We’re seeing
around 20MB per runtime in our deployment, so something seems wrong if
you’re seeing ~80MB per.

My point with the scaling and runtime pools is that you just can’t
expect good scaling for concurrent transactions beyond the number of
runtimes you have available. It’s not ideal compared to a Java-based
framework where you don’t have the same memory costs, but the Rails
landscape is quickly shifting toward producing a multi-thread-capable
Rails, which will all but eliminate these problems. Expect to see
something in Rails (2.1).next to address threading.

You might also want to check out Merb or some of the other Ruby web
frameworks if you’re looking for a Rails-like alternative that doesn’t
have the need to do runtime-pooling.

I was alerted to the GC issue when tailing the logs with -verbose:gc.
JRails was completing only around 5 requests between each GC. I took a look
at a heap dump from a high load period, and I saw a surprising number of
orphaned objects.

There are probably some memory and GC-related JVM flags that can help
you here, but I’m not an expert in this area. Some other things to
look into:

Try running with Java 6 (if you have a 64-bit capable mac), or try
Soylatte
Again, try JRuby trunk
Experiment with some different GC settings, like -XX:NewRatio:2,
-XX:+UseParallelGC, -XX:+UseParallelOldGC. Introduce these one at a
time to isolate the effects (if any).

/Nick

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 8, 2008, 6:31am

For what it’s worth, we have seen similar GC issues under Glassfish.
Increasing the minimum and initial heap sizes in addition to maximum
heap size has made a big difference for us.

On Tue, May 6, 2008 at 12:22 PM, Matt S. [email protected]
wrote:

Mac OS X Leopard

threads, from 1 to 50 in increments of 5. At each thread count, I’m making
is also garbage collecting like crazy from the moment JRuby starts. I

Sorry for the long email. Thanks for taking the time to read!

-Matt

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 8, 2008, 11:06pm

What are you setting your minimum and maximum heap size to, how many
rails instances?

James N. wrote:

For what it’s worth, we have seen similar GC issues under Glassfish.
Increasing the minimum and initial heap sizes in addition to maximum
heap size has made a big difference for us.

On Tue, May 6, 2008 at 12:22 PM, Matt S. [email protected] wrote:

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

matthew · May 10, 2008, 3:45am

The relevant jvm settings from our domain config file are given below:

-XX:MaxPermSize=1024m
-server
-Xmx2560m
-Xms2560m
-Xmn1024m
-XX:LargePageSizeInBytes=256m

On Thu, May 8, 2008 at 5:06 PM, [email protected] wrote:

wrote:

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email