Hi All, We are using jruby-1.7.3 with rails 2.3.17. Running this on tomcat server in the production environment.We are running it on two tomcat nodes. When load goes high on production environment. We are seeing lot of timeout errors and server response time goes high. Also, CPU utilization spikes out. Just wanted to check if something seriously wrong with jruby-1.7.3 with respect to time out. Or is there any issue in running the parallel threads. Thanks in Advance for help !!
on 2014-01-14 12:25
on 2014-01-14 22:56
Hey, that's very likely not a JRuby issue (although there should be no blocker for you guys to simply update to the latest of 1.7.x and see if it's the same) but an I/O one. Something is waiting for IO (a Ruby library/gem such as memcached or it's alternatives or a low pool setting) - you need to review your gems or even better monitor what's going on. K.
on 2014-01-14 23:00
I would tend to agree with Karol. Is there a stack trace in the logs that would clue you into where the timeout is happening in the code? Thanks, Ariel --- Sent from my mobile device. Please excuse any errors.
on 2014-01-15 13:08
Thanks Karol and Ariel, These are random stack traces which we are getting upon hitting the application multiple times. The exact same code base and gems were working fine on our older stack which was JBOSS and Jruby-1.4.x and Rails - 2.3.17 Status: 500 Internal Server Error could not obtain a database connection within 5 seconds. The max pool size is currently 90; consider increasing it. Status: 500 Internal Server Error Detected invalid hash contents due to unsynchronized modifications with concurrent users RuntimeError (can't add a new key into hash during iteration): org/jruby/RubyHash.java:946:in `=' org/jruby/RubyProc.java:261:in `call' org/jruby/RubyProc.java:209:in `call' So am thinking that it might be issue with jruby thread-safety Any pointers will be really helpful.
on 2014-01-15 14:42
I do recall there being many issues with rails 2.x around thread safety. I assume that your application is running with "config.threadsafe!"? Does the concurrent modification error also happen due to connection pool errors? It looks like there are not enough connections in the pool to satisfy the number of concurrent requests tomcat is dispatching and/or some requests take too long, which causes other threads to timeout. Have you tried increasing the checkout_timeout or the number of threads? What version of JBoss are you using where everything seems to work fine? If I recall correctly it uses tomcat as the servlet container. Are the versions and configuration the same (eg maxSpareThreads) Thanks, Ariel --- Sent from my mobile device. Please excuse any errors.
on 2014-01-15 15:14
Adding upon Ariel's suggestions - the concurrency issue your pointing out with the Hash is something that should be the first thing to look at. Unfortunately we do not see the full trace ... I'm guessing your app/code is not really thread-safe ... working on an old version of JRuby + JBoss does not mean it was all fine. Maybe you've run with JRuby-Rack and it pools runtimes by default ? On the other hand Rails 2.3 is not 100% (depends on it's usage) thread-safe e.g. I've seen issues with the 2.3 connection pool not behaving consistently when a connection is closed and re-required on the same thread (might happen with the active_record session store). I might be wrong in the end but do not expect to get all of this worked out if you're just shooting (helped some and their issues were mostly non-trivial - required understanding of the app on the level of user code or at least knowing the configuration/gems used) - it's best to get someone to look at the app or simply start with the "easy" fixes one thing at a time. K.
on 2014-01-15 15:30
I suggest you take a thread dump and check the java heap use and GC also. Thanks Alex
on 2014-01-15 15:46
A little snooping around rails defects it appears to be fixed in 3.2 https://github.com/rails/rails/pull/6467 Thanks, Ariel --- Sent from my mobile device. Please excuse any errors.
on 2014-01-15 21:57
I'm not sure it's only that - have tried to track down comparing against 3.2, but it's not straight-forward since code got "re-invented" (refactored) ... I've ended up carefully back-porting AR's 3.2 pool with AR-JDBC 1.3.x for Rails 2.3 and it worked without any (production) issues. The other thing we should have noted as well and I certainly recommend ( although I'm biased :) is to upgrade AR-JDBC to 1.3.x since some parts previously not completely AR compatible might cause issues with some adapters as well (e.g. reconnect!). K.