Forum: JRuby JRuby on Rails experience

Just wanted to share my first JRuby experience: A few months earlier,
after all the deliberation, we chose JRuby over vanilla Ruby for a heavy
traffic application...JRuby seemed very promising, and so far so good,
I am pleased to say that JRuby has kept all its promises!
Recently, we successfully deployed our first JRuby on Rails application
in production (JRuby 1.5.0, Rails 2.3.5, Apache, Tomcat 6, PostgreSQL
8.4). Initially faced a lot of challanges to scale it for 2000+
concurrent users, but with excellent help from this forum, the JRuby
cookbook and other JRuby sources on internet, we were able to tune the
application and make those 2000+ concurrent users happy.

The most important learning for us has been the jruby.runtimes tuning.
The
configuration that scaled the best was (enable config.threadsafe! in
Rails) + (jruby min and max runtimes = 1). We faced performance issues
with JRuby to Java calls at higher concurrency and had to switch to
ServletFilter for most of the Java calls.

Overall, JRuby is superb!

Our special thanks to entire JRuby development team (Nick, Charles et
al) for giving us this fantastic bridge between Java and Ruby! It just
works!

Looking forward to further improvements in JRuby libraries...

Keep going JRuby team...!

-Darshan.

Thanks for sharing your experience with us! It's always reassuring to 
hear a success story. Let us know if there's anything you would like to 
see improved.

/Nick

On Sat, Feb 19, 2011 at 1:55 AM, Darshan Karandikar
<lists@ruby-forum.com> wrote:
> The most important learning for us has been the jruby.runtimes tuning.
> The
> configuration that scaled the best was (enable config.threadsafe! in
> Rails) + (jruby min and max runtimes = 1). We faced performance issues
> with JRuby to Java calls at higher concurrency and had to switch to
> ServletFilter for most of the Java calls.

I'm very interested in hearing more specifics about these performance
issues.  At what levels of parallel execution did you notice
performance degradation, and what exactly did you observe?

Anthony Juckel wrote in post #982645:
> On Sat, Feb 19, 2011 at 1:55 AM, Darshan Karandikar
> <lists@ruby-forum.com> wrote:
>> The most important learning for us has been the jruby.runtimes tuning.
>> The
>> configuration that scaled the best was (enable config.threadsafe! in
>> Rails) + (jruby min and max runtimes = 1). We faced performance issues
>> with JRuby to Java calls at higher concurrency and had to switch to
>> ServletFilter for most of the Java calls.
>
> I'm very interested in hearing more specifics about these performance
> issues.  At what levels of parallel execution did you notice
> performance degradation, and what exactly did you observe?


Hi Nick, Anthony,

Summarizing our observations during performance testing with 1500
concurrent users into the system, along with :
- comments on areas we would like to see improved in JRuby
- details of "JRuby to Java call" performance issue

1) With more number of jruby runtimes, the number of classes loaded and
JVM heap memory utilization grows substantially (analyzed using
jvisualvm). This is expected JRuby 1.5.0 behavior I suppose.

-  We started with 5 jruby min runtimes and max 25. As more users
entered the system, we noticed considerable stress on CPU and JVM heap
memory whenever a new runtime had to be created to serve those users
(new runtime creation was indicated by increase in number of classes
loaded, noticed using jvisualvm). We concluded that the resources
required to create a jruby runtime was a bit high and slows down things
under load. So we configured jruby.runtime min=max=25, so that they get
created during application deployment itself. This improved response
time & resources utilization during user ramp up, but slowed down
application deployment (which was ok for us).
- With jruby min runtimes = 5 & max = 25, (with jvisualvm) we noticed
that the classes created for 25 runtimes at peak were not unloaded (&
the memory utilization didn't reduce) even after the application usage
dropped down to a few users. I suppose this is known issue, and higher
JRuby version might have fixed it already (if yes, would like to know
which JRuby version to upgrade to for the fix?).
- Around 25 jruby runtimes (required to keep response time within
acceptable limit) consumed approx 40-50% of JVM heap memory, and with
approx around 1000 concurrent users in, the free heap memory reduced to
a few MBs, thus triggering GC quite often. It is desirable if the heap
memory consumption by a jruby runtime can be reduced further so that we
can accomodate more runtimes in less heap space, leaving enough heap
space for users' session data.
- Looking at the foreseen user growth & constraints on h/w we have, we
decided to reduce resource consumption by jruby runtimes and give those
resources for application functionality (mainly memory for session
data). The best option for this was to ensure we have thread-safe code
in place (= efforts[code review]), then enable config.threadsafe! in
Rails and set jruby.runtime.min=max=1. We got rid of some Rails plugins
as we weren't sure if they are thread-safe or not. This configuration
worked well and spared much more memory for usage by application logic.
We would like this to be default JRuby+Rails behavior/configuration
(like default Servlet behavior in JEE), though we understand this might
also depend   on the way Ruby+Rails handles threads/requests by default.

2) JRuby to Java calls: The code had a few simple Java calls (create a
Java object and call a method with some input params) from JRuby in one
functionality. Everything worked ok till around 1100 concurrent users
used that functionality, but beyond that concurrency application server
CPU utilization started increasing non-linearly, and response time
increased beyond acceptable limit. We started RCA to get to root cause
by commenting code in that particular functionality one by one, and
noticed that "without" the JRuby to Java calls, everything worked fine
even with 1500 concurrent users. Hence, we removed these calls, and
placed them in Servlet Filter. This change fixed the CPU utilization
issue under high load. This behavior has led us to conclude that the
JRuby to Java calls was the root cause of slow response and high CPU
utilization at high load. Not sure if this observation has anything to
do with this discussion:
http://www.mail-archive.com/dev@jruby.codehaus.org...

3) After the test, when we stop the Tomcat server, we get following
errors in the Tomcat logs (not sure the impact of these during heavy
load, but still would like to see these gone):

SEVERE: A web application created a ThreadLocal with key of type [null]
(value [com.kenai.jaffl.provider.StringIO$1@1bb90f6]) and a value of
type [java.lang.ref.SoftReference] (value
[java.lang.ref.SoftReference@12a2937]) but failed to remove it when the
web application was stopped. To prevent a memory leak, the ThreadLocal
has been forcibly removed.
xxxx  yyyy PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [null]
(value [org.joni.StackMachine$1@f8027b]) and a value of type
[java.lang.ref.WeakReference] (value
[java.lang.ref.WeakReference@6b06df]) but failed to remove it when the
web application was stopped. To prevent a memory leak, the ThreadLocal
has been forcibly removed.
xxxx  yyyy PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap