Just wanted to share my first JRuby experience: A few months earlier, after all the deliberation, we chose JRuby over vanilla Ruby for a heavy traffic application...JRuby seemed very promising, and so far so good, I am pleased to say that JRuby has kept all its promises! Recently, we successfully deployed our first JRuby on Rails application in production (JRuby 1.5.0, Rails 2.3.5, Apache, Tomcat 6, PostgreSQL 8.4). Initially faced a lot of challanges to scale it for 2000+ concurrent users, but with excellent help from this forum, the JRuby cookbook and other JRuby sources on internet, we were able to tune the application and make those 2000+ concurrent users happy. The most important learning for us has been the jruby.runtimes tuning. The configuration that scaled the best was (enable config.threadsafe! in Rails) + (jruby min and max runtimes = 1). We faced performance issues with JRuby to Java calls at higher concurrency and had to switch to ServletFilter for most of the Java calls. Overall, JRuby is superb! Our special thanks to entire JRuby development team (Nick, Charles et al) for giving us this fantastic bridge between Java and Ruby! It just works! Looking forward to further improvements in JRuby libraries... Keep going JRuby team...! -Darshan.
on 2011-02-19 08:55
on 2011-02-19 15:31
Thanks for sharing your experience with us! It's always reassuring to hear a success story. Let us know if there's anything you would like to see improved. /Nick
on 2011-02-19 15:36
On Sat, Feb 19, 2011 at 1:55 AM, Darshan Karandikar <email@example.com> wrote: > The most important learning for us has been the jruby.runtimes tuning. > The > configuration that scaled the best was (enable config.threadsafe! in > Rails) + (jruby min and max runtimes = 1). We faced performance issues > with JRuby to Java calls at higher concurrency and had to switch to > ServletFilter for most of the Java calls. I'm very interested in hearing more specifics about these performance issues. At what levels of parallel execution did you notice performance degradation, and what exactly did you observe?
on 2011-02-21 09:17
Anthony Juckel wrote in post #982645: > On Sat, Feb 19, 2011 at 1:55 AM, Darshan Karandikar > <firstname.lastname@example.org> wrote: >> The most important learning for us has been the jruby.runtimes tuning. >> The >> configuration that scaled the best was (enable config.threadsafe! in >> Rails) + (jruby min and max runtimes = 1). We faced performance issues >> with JRuby to Java calls at higher concurrency and had to switch to >> ServletFilter for most of the Java calls. > > I'm very interested in hearing more specifics about these performance > issues. At what levels of parallel execution did you notice > performance degradation, and what exactly did you observe? Hi Nick, Anthony, Summarizing our observations during performance testing with 1500 concurrent users into the system, along with : - comments on areas we would like to see improved in JRuby - details of "JRuby to Java call" performance issue 1) With more number of jruby runtimes, the number of classes loaded and JVM heap memory utilization grows substantially (analyzed using jvisualvm). This is expected JRuby 1.5.0 behavior I suppose. - We started with 5 jruby min runtimes and max 25. As more users entered the system, we noticed considerable stress on CPU and JVM heap memory whenever a new runtime had to be created to serve those users (new runtime creation was indicated by increase in number of classes loaded, noticed using jvisualvm). We concluded that the resources required to create a jruby runtime was a bit high and slows down things under load. So we configured jruby.runtime min=max=25, so that they get created during application deployment itself. This improved response time & resources utilization during user ramp up, but slowed down application deployment (which was ok for us). - With jruby min runtimes = 5 & max = 25, (with jvisualvm) we noticed that the classes created for 25 runtimes at peak were not unloaded (& the memory utilization didn't reduce) even after the application usage dropped down to a few users. I suppose this is known issue, and higher JRuby version might have fixed it already (if yes, would like to know which JRuby version to upgrade to for the fix?). - Around 25 jruby runtimes (required to keep response time within acceptable limit) consumed approx 40-50% of JVM heap memory, and with approx around 1000 concurrent users in, the free heap memory reduced to a few MBs, thus triggering GC quite often. It is desirable if the heap memory consumption by a jruby runtime can be reduced further so that we can accomodate more runtimes in less heap space, leaving enough heap space for users' session data. - Looking at the foreseen user growth & constraints on h/w we have, we decided to reduce resource consumption by jruby runtimes and give those resources for application functionality (mainly memory for session data). The best option for this was to ensure we have thread-safe code in place (= efforts[code review]), then enable config.threadsafe! in Rails and set jruby.runtime.min=max=1. We got rid of some Rails plugins as we weren't sure if they are thread-safe or not. This configuration worked well and spared much more memory for usage by application logic. We would like this to be default JRuby+Rails behavior/configuration (like default Servlet behavior in JEE), though we understand this might also depend on the way Ruby+Rails handles threads/requests by default. 2) JRuby to Java calls: The code had a few simple Java calls (create a Java object and call a method with some input params) from JRuby in one functionality. Everything worked ok till around 1100 concurrent users used that functionality, but beyond that concurrency application server CPU utilization started increasing non-linearly, and response time increased beyond acceptable limit. We started RCA to get to root cause by commenting code in that particular functionality one by one, and noticed that "without" the JRuby to Java calls, everything worked fine even with 1500 concurrent users. Hence, we removed these calls, and placed them in Servlet Filter. This change fixed the CPU utilization issue under high load. This behavior has led us to conclude that the JRuby to Java calls was the root cause of slow response and high CPU utilization at high load. Not sure if this observation has anything to do with this discussion: http://email@example.com... 3) After the test, when we stop the Tomcat server, we get following errors in the Tomcat logs (not sure the impact of these during heavy load, but still would like to see these gone): SEVERE: A web application created a ThreadLocal with key of type [null] (value [com.kenai.jaffl.provider.StringIO$1@1bb90f6]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@12a2937]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. xxxx yyyy PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [null] (value [org.joni.StackMachine$1@f8027b]) and a value of type [java.lang.ref.WeakReference] (value [java.lang.ref.WeakReference@6b06df]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. xxxx yyyy PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap