Hi all. I¹ve been running a bunch of performance tests comparing JRails deployed in JBoss to Spring (a Java MVC framework), and the results I¹ve gotten aren¹t good for JRuby. I¹d really like to be able to start moving towards Rails, but I there are a few deal-breakers I¹d like to figure out. First, let me describe my environment: Dev: Mac OS X Leopard Java 1.5.0_13 Prod: Centos 5 JBoss 3.2.7 / Jetty 4.2.24 * 3G RAM and 256M PermGen space * -server -Djruby.compilation.mode=FORCE -Djruby.objectspace.enabled=false -Djruby.thread.pooling=true * war configured to start 10 JRails runtimes Java 1.5.0_14 JRuby/Rails: JRuby 1.1.1 Rails 2.0.2 Warbler 0.9.5 My webapp is very simple. It¹s a three page drilldown that uses a native Java DB class (shared between the JRails and Spring implementations) to call stored procs in an Oracle DB. I¹m using Grinder to make repeated requests with increasing numbers of threads, from 1 to 50 in increments of 5. At each thread count, I¹m making about 150 requests (50 to each type of page). What I see from the client side is many responses returning in a reasonable amount of time (the average request time under heavy load is around 1-1.5 seconds, but many requests return in 100-200 milliseconds), but many threads seemingly blocking for a long time (the standard deviation on request time is as high as 30 seconds). On the server side, Rails reports some requests being processed for as long as a few seconds (as many as 5 or 6), but never for as many as 30. The JVM is also garbage collecting like crazy from the moment JRuby starts. I suspect that the long delays are a result of threads being blocked waiting for garbage collecting, but I¹m not positive. I eliminated the DB code as a potential cause by running the same suite of tests on a simple ³Hello world² app, and got approximately the same performance results (Spring shows about a 5-fold increase in throughput). My assumption is that the constant overhead of garbage collection is the bottleneck, but I¹m by no means convinced. I¹ve read that IBM and ThoughtWorks have deployed JRails apps in similar environments, so my results are confusing and frustrating. Here are my questions: 1. Is there a way to tame the garbage collection? I suspect that objects are being created and deleted with each request rather than being reused. 2. Can Rails be fully compiled to Java? Can a Rails app? It seems like that¹s a goal of jrubyc, but is it possible with the current version? 3. Am I missing some bit of configuration in the war deployment that will tell JRuby to be more conservative with object creation? Sorry for the long email. Thanks for taking the time to read! -Matt
on 06.05.2008 18:23
on 06.05.2008 19:39
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matt Singleton wrote: | 3. Am I missing some bit of configuration in the war deployment that will | tell JRuby to be more conservative with object creation? Is Rails configured for production or the development environment? In development, nothing gets cached by Rails (like HTML), and gets regenerated every time the view is called, which, obviously, kills performance. ;) Of course, if Rails is in its production environment, disregard this. Though in such a case it might be worthwhile to prod into JBoss's caching mechanisms. Also, Rails thread's are blocking, too, on every request, so it isn't necessarily the GC that throws you off, but memory constrains, or so. - -- Phillip Gawlowski Twitter: twitter.com/cynicalryan Blog: http://justarubyist.blogspot.com ~ You know what we need, Hobbes? We need an attitude. Yeah, you can't be cool if you don't have an attitude. -- Calvin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkggl4YACgkQbtAgaoJTgL/wpACgn8I1pyEB2eT2fbsMke7Cq1lw PZwAoKW8rPHeJVWgUaPWzDRshE3lC8nq =ESnA -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 06.05.2008 20:17
On 5/6/08 1:38 PM, "Phillip Gawlowski" <cmdjackryan@googlemail.com> wrote: > Is Rails configured for production or the development environment? > > In development, nothing gets cached by Rails (like HTML), and gets > regenerated every time the view is called, which, obviously, kills > performance. ;) > > Of course, if Rails is in its production environment, disregard this. > Rails is configured for the production environment. > Also, Rails thread's are blocking, too, on every request, so it isn't > necessarily the GC that throws you off, but memory constrains, or so. I have Rails configured for a pool of 10 runtimes, so I guess that means it can only handle as many as 10 concurrent threads and the rest are blocking. That would explain some of the long response times. Memory constraints are not a big issue (I have 4G available for Jboss), but even 10 runtimes under load have resident memory usage at about 1.2G, so I can get up to 20 or 25 runtimes, but that's not even close to what a native Java app can handle (our traditional Jboss apps are configured for a pool of 300 threads). Is there any other way to scale up other than just adding more app servers? I've hear my share of "rails doesn't scale" comments. Is this what they're referring to? I'm not convinced that the small number of threads is the full answer. Even at 10 threads, the Spring app handily outperformed the JRails app. I'm willing to accept that JRuby isn't quite up to the speed of native Java yet, but it's the size of the gap that makes me think I might be missing something. Has anyone else tried this kind of deployment and would like to compare notes? -Matt --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 06.05.2008 20:38
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Matt Singleton wrote: | I have Rails configured for a pool of 10 runtimes, so I guess that means it | can only handle as many as 10 concurrent threads and the rest are blocking. That is my understanding, too. I;m not a Rails wizard, though. | That would explain some of the long response times. Memory constraints are | not a big issue (I have 4G available for Jboss), but even 10 runtimes under | load have resident memory usage at about 1.2G, so I can get up to 20 or 25 | runtimes, but that's not even close to what a native Java app can handle | (our traditional Jboss apps are configured for a pool of 300 threads). Is | there any other way to scale up other than just adding more app servers? Well, you could use JRuby's JIT, I guess (I have no idea how to enable this, but seems to be on already). http://wiki.jruby.org/wiki/JRuby_Compiler This should provide you with more information than I could at this stage. | I've hear my share of "rails doesn't scale" comments. Is this what they're | referring to? Sort of, yes. However, adding more application servers is one option. Another would be to farm out heavy lifting you do in controllers (or was it models?) to Java (or C in case you'd be using the MRI). Another approach would be using more threads (I don't know how JRuby uses memory), if you don't do stuff requiring large amounts of memory. I haven't checked how much RAM my Rails development environment swallows, but it can't be no where near the number you report, as I don't have that amount of RAM, and the JRE would throw an OutOfMemory exception. Other possible issues: - - JBoss does some odd stuff, and introduces a lot of overhead itself (I've noticed that JBoss uses a lot of memory itself, but that was 3 years ago, and a data mining specific application, so take that with a grain of salt!) - - The JRE you use doesn't optimize as good as, say the Sun JRE, or the Sun JRE allocates more memory than needed (I have no experience with other JREs than Sun's). - - Ruby isn't all that good with memory consumption in general. | I'm not convinced that the small number of threads is the full answer. Even | at 10 threads, the Spring app handily outperformed the JRails app. I'm | willing to accept that JRuby isn't quite up to the speed of native Java yet, | but it's the size of the gap that makes me think I might be missing | something. Has anyone else tried this kind of deployment and would like to | compare notes? Well, Spring is a Java framework, where Rails is a Ruby framework, and JRuby is implemented on top of Java. So you have the overhead of Java, JRuby and Rails at the same time. That doesn't, however, translate to, say, 3 times the overhead. But that a Java specific application outperforms a JRuby application by a big margin isn't all that surprising to me. How big is the gap you are seeing? - -- Phillip Gawlowski Twitter: twitter.com/cynicalryan Blog: http://justarubyist.blogspot.com :zorkmid: /zork'mid/ n. The canonical unit of currency in ~ hacker-written games. This originated in {zork} but has spread ~ to {nethack} and is referred to in several other games. ~ -- The AI Hackers Dictionary -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkggpZkACgkQbtAgaoJTgL+rqQCbBQeNFWmPSaqX7Zc6JXZDiy77 f/MAn0Jjc5Htt+L0cMjKXldf+nX+jYe/ =igoS -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 06.05.2008 20:52
I'm seeing about a 5x increase in throughput (requests/second) with Spring, which is more than I expected. I'm going to see what performance looks like in Tomcat. Jboss is a bit of a heavyweight. Thanks for all the help so far. Matt On 5/6/08 2:38 PM, "Phillip Gawlowski" <cmdjackryan@googlemail.com> wrote: > > > > exception. > > like to > How big is the gap you are seeing? > -----BEGIN PGP SIGNATURE----- > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 00:33
Matt, I have a JRuby app running under JBoss 4.2. I don't think your problem is with JBoss itself...V4 uses Tomcat anyway internally and I don't believe they've hacked it that much. My installation has a lot of the other JBoss stuff stripped out of the config anyway. Likewise I don't think your JVM or GC is the problem either. I haven't actually compared the 'same app' side between a JRuby version and a Java version, but I also find JRuby seems to be a lot slower overall. And yes, the memory consumption is pretty heavy too. Because the rails model is to just duplicate the process cgi-style to get more listeners, (since ruby is not multithreaded), maybe the translation of this pattern into Jruby/goldspike isn't very efficient. In fact I wonder if each process ends up being loaded under its own classloader so that all the code is duplicated in memory? One thing: you said you were using a database connection for your test...what is the form of your DB connectivity on the JRuby side? Have you thought about just doing native Ruby with Apache? If you still need access to Java code perhaps you could use some sort of remoting technique or other form of layering. ...Milt Matt Singleton wrote: > I'm seeing about a 5x increase in throughput (requests/second) with Spring, > which is more than I expected. I'm going to see what performance looks like > in Tomcat. Jboss is a bit of a heavyweight. Thanks for all the help so > far. > > Matt > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 09:04
On Tue, May 6, 2008 at 9:22 AM, Matt Singleton <msingleton@wgen.net> wrote: > Mac OS X Leopard > Java 1.5.0_13 > > Prod: > Centos 5 > JBoss 3.2.7 / Jetty 4.2.24 > * 3G RAM and 256M PermGen space > * -server -Djruby.compilation.mode=FORCE -Djruby.objectspace.enabled=false Try with just "-server". compile = FORCE will probably only slow things down, as ahead-of-time compiler is not always generating optimal code (yet). objectspace is off by default with 1.1.x. > -Djruby.thread.pooling=true Shouldn't need thread pooling either, to start. > * war configured to start 10 JRails runtimes You should have at least as many runtimes as you plan to bench concurrent users/transactions, otherwise you're not going to see very good scaling curves. > What I see from the client side is many responses returning in a reasonable > amount of time (the average request time under heavy load is around 1-1.5 > seconds, but many requests return in 100-200 milliseconds), but many threads > seemingly blocking for a long time (the standard deviation on request time > is as high as 30 seconds). Contention for a runtime is one reason, but you might also be getting bit by a regexp cache bug that we recently fixed in trunk. You might want to give trunk a try if you can -- 1.1.2 will be out in a couple weeks. On our project, we have not seen this behavior. Generally requests scale well up to 100 concurrent users, and performance/latency is generally on par with matzruby. Of course, performance is always going to be application-dependent. > On the server side, Rails reports some requests being processed for as long > as a few seconds (as many as 5 or 6), but never for as many as 30. The JVM > is also garbage collecting like crazy from the moment JRuby starts. I > suspect that the long delays are a result of threads being blocked waiting > for garbage collecting, but I'm not positive. I eliminated the DB code as a > potential cause by running the same suite of tests on a simple "Hello world" > app, and got approximately the same performance results (Spring shows about > a 5-fold increase in throughput). You might want to open JConsole and connect it to your app server's VM to confirm or deny your garbage collection suspicion. > > Is there a way to tame the garbage collection? I suspect that objects are > being created and deleted with each request rather than being reused. > Can Rails be fully compiled to Java? Can a Rails app? It seems like that's > a goal of jrubyc, but is it possible with the current version? > Am I missing some bit of configuration in the war deployment that will tell > JRuby to be more conservative with object creation? You should confirm that it's garbage collection. You might also want to try a memory profiler if you have access to one, or maybe just run with -verbose:gc. /Nick --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 09:22
Nick, do you have a JIRA issue that is tagged to this problem? I would like to understand more about how this impacts a typical war app under jruby... > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 09:25
http://jira.codehaus.org/browse/JRUBY-2483 is the issue I filed. If you have threads that seem to be locked up, send a "kill -QUIT" to the process to inspect the thread stack traces. /Nick On Wed, May 7, 2008 at 12:15 AM, <mctozzy@gmail.com> wrote: > > On our project, we have not seen this behavior. Generally requests > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 16:42
First, thanks all for helping me get to the bottom of this. On 5/7/08 3:04 AM, "Nick Sieger" <nicksieger@gmail.com> wrote: > On Tue, May 6, 2008 at 9:22 AM, Matt Singleton <msingleton@wgen.net> wrote: >> * war configured to start 10 JRails runtimes > > You should have at least as many runtimes as you plan to bench > concurrent users/transactions, otherwise you're not going to see very > good scaling curves. > I'd do this, except the memory costs is prohibitive. I ramped up to 25 runtimes, and under heavy load, memory usage was heading past 2G. By contrast, the Spring equivalent is configured with a pool of 300 response handlers, and it fits tidily within 512M. Are you not seeing the high memory usage? What does your deployed environment look like? > to confirm or deny your garbage collection suspicion. > to try a memory profiler if you have access to one, or maybe just run > with -verbose:gc. > I was alerted to the GC issue when tailing the logs with -verbose:gc. JRails was completing only around 5 requests between each GC. I took a look at a heap dump from a high load period, and I saw a surprising number of orphaned objects. On 5/6/08 6:32 PM, "mctozzy@gmail.com" <mctozzy@gmail.com> wrote: > One thing: you said you were using a database connection for your > test...what is the form of your DB connectivity on the JRuby side? It's a pure java class that uses the Oracle thin client and returns a JSON object. We encapsulated the DB stuff in Java so we could use it the same way in Rails and Spring. Matt --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 17:28
Hi! On Tue, May 06, 2008 at 12:22:50PM -0400, Matt Singleton wrote: [..] > for garbage collecting, but I¹m not positive. I eliminated the DB code as a > potential cause by running the same suite of tests on a simple ³Hello world² > app, and got approximately the same performance results (Spring shows about > a 5-fold increase in throughput). You didnt mention mod_jk being part of your setup, but we had exactly the same behaviour in our production environment with JBoss, Apache + mod_jk. The fix was to set the mod_jk recycle_timeout parameter to a high value (300 worked well for us, while 30 didn't). Not setting it at all but leaving it alone might work too, but I never got a chance to try this out since I have no direct access to this production environment unfortunately. Besides that, comparing the performance of a plain spring webapp with a Rails one feels a bit like comparing apples with oranges to me. In any performance testing I did with database intensive jruby/rails apps, its performance was on the very same level as MRI/Rails (that is, after an initial warmup period of several 100 requests). Of course you can get better performance with plain servlets or something that comes near to plain servlets. Before going online our jruby/rails application has been load tested by some third party company with up to 60 concurrent virtual clients (which not only issued plain get requests on pages but also loaded images and stuff like a real browser would, which matters in this case since assets are delivered from oracle blobs by this rails app, too) and never showed response times longer than 3 seconds during several hours of testing. Unfortunately I don't have the exact numbers, 3 seconds was the upper margin that must not be hit in order to pass this test... The app (www.medienservice.sachsen.de in case youre interested what im talking about here) runs on a cluster of 2 virtualized app servers with 6 Rails workers per server and an Xmx setting of 1024m. Cheers, Jens -- Jens Krämer Finkenlust 14, 06449 Aschersleben, Germany VAT Id DE251962952 http://www.jkraemer.net/ - Blog http://www.omdb.org/ - The new free film database --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 07.05.2008 20:05
On Wed, May 7, 2008 at 7:41 AM, Matt Singleton <msingleton@wgen.net> wrote: > > concurrent users/transactions, otherwise you're not going to see very > > good scaling curves. > > > > I'd do this, except the memory costs is prohibitive. I ramped up to 25 > runtimes, and under heavy load, memory usage was heading past 2G. By > contrast, the Spring equivalent is configured with a pool of 300 response > handlers, and it fits tidily within 512M. Are you not seeing the high > memory usage? What does your deployed environment look like? I agree, the memory situation is not the best, but compared to matzruby, JRuby deployments are actually favorable. We're seeing around 20MB per runtime in our deployment, so something seems wrong if you're seeing ~80MB per. My point with the scaling and runtime pools is that you just can't expect good scaling for concurrent transactions beyond the number of runtimes you have available. It's not ideal compared to a Java-based framework where you don't have the same memory costs, but the Rails landscape is quickly shifting toward producing a multi-thread-capable Rails, which will all but eliminate these problems. Expect to see something in Rails (2.1).next to address threading. You might also want to check out Merb or some of the other Ruby web frameworks if you're looking for a Rails-like alternative that doesn't have the need to do runtime-pooling. > I was alerted to the GC issue when tailing the logs with -verbose:gc. > JRails was completing only around 5 requests between each GC. I took a look > at a heap dump from a high load period, and I saw a surprising number of > orphaned objects. There are probably some memory and GC-related JVM flags that can help you here, but I'm not an expert in this area. Some other things to look into: - Try running with Java 6 (if you have a 64-bit capable mac), or try Soylatte - Again, try JRuby trunk - Experiment with some different GC settings, like -XX:NewRatio:2, -XX:+UseParallelGC, -XX:+UseParallelOldGC. Introduce these one at a time to isolate the effects (if any). /Nick --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 08.05.2008 06:31
For what it's worth, we have seen similar GC issues under Glassfish. Increasing the minimum and initial heap sizes in addition to maximum heap size has made a big difference for us. On Tue, May 6, 2008 at 12:22 PM, Matt Singleton <msingleton@wgen.net> wrote: > Mac OS X Leopard > > threads, from 1 to 50 in increments of 5. At each thread count, I'm making > is also garbage collecting like crazy from the moment JRuby starts. I > > Sorry for the long email. Thanks for taking the time to read! > > -Matt --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 08.05.2008 23:06
What are you setting your minimum and maximum heap size to, how many rails instances? James Norton wrote: > For what it's worth, we have seen similar GC issues under Glassfish. > Increasing the minimum and initial heap sizes in addition to maximum > heap size has made a big difference for us. > > On Tue, May 6, 2008 at 12:22 PM, Matt Singleton <msingleton@wgen.net> wrote: > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
on 10.05.2008 03:45
The relevant jvm settings from our domain config file are given below: <jvm-options>-XX:MaxPermSize=1024m</jvm-options> <jvm-options>-server</jvm-options> <jvm-options>-Xmx2560m</jvm-options> <jvm-options>-Xms2560m</jvm-options> <jvm-options>-Xmn1024m</jvm-options> <jvm-options>-XX:LargePageSizeInBytes=256m</jvm-options> On Thu, May 8, 2008 at 5:06 PM, <mctozzy@gmail.com> wrote: >> wrote: >> > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email