Re: RE: Rack Error


#1
Going to Rails 2.2 may have it's own issues though.

Just today we had to back rev to 2.1 for our first "beta" sight as we are getting a pariticular multithreaded problem.  (we are getting the body of our request set to nil underneath us).

I'm trying to characterize it... But it really only seems to happen under heavy load.  If you have a slow running job.  2.2 works pretty well.  I'd recommend it.  The memory improvement is compelling.

as well as the performance improvements.


Dec 18, 2008 12:36:40 PM, removed_email_address@domain.invalid wrote:
How about:

jthread = java.lang.Thread.new do puts "Running in a Java thread" end
jthread.start

Come to think of it, why not just use the Ruby thread? Thread.new?

Also, it sounds like if your conjecture is right, then the problem is not with
JRuby, but rather the blocking architecture of your app; Your only solution
would be to start up at least as many runtime as your expected peak
concurrency. If so, you should really upgrade to Rails 2.2, since that gives
some big memory saving.

Of course, this is assuming that your problem is with request blocking, and
not some kind of JRuby leak. Are you able to tell what the heap size is, so as
to figure out whether it is runtime concurrency exhaustion or memory
exhaustion?

Peter

From: AD [mailto:removed_email_address@domain.invalid]
Sent: Thursday, December 18, 2008 11:13 AM
To: removed_email_address@domain.invalid
Subject: Re: [jruby-user] Rack Error

So I think what *could* be causing this is we need to make a remote soap calls
(using soap/wsdldriver and soap/header/SimpleHandler) and it can take up to
30seconds to a minute to process (they dont support Async and we are in the
process of moving this to a JMS queue)

Is there an easy way to wrap this in a Java thread (aside from a rails 2.2
upgrade) ?  I think what could be happening is 8 simultaneous calls are being
made tying up all Jruby runtimes.  What I dont get is why the whole app server
dies after this happens.

We have 8 runtimes available.

Adam
On Thu, Dec 18, 2008 at 11:24 AM, Charles Oliver N.

Ok, now we can start looking at your problem in earnest :)

Can you describe your configuration? How many runtimes? How heavily are they
hit? The error below I believe means that there's no runtimes available in the
pool, and so the request can't be handled. What do you have to do to cause
this error?

AD wrote:
Sorry guys, same error again on 1.1.6 with rack 0.9.3

SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
       at
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplica
tionFactory.java:87)
       at
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
       at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:215)
       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:188)
       at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
       at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:172)
       at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

Adam


On Thu, Dec 18, 2008 at 9:01 AM, AD


   Thanks Charlie, I couldnt agree more here.  A lot of conversation on
   this group about different GC algorithms, PermGen, Newsize, etc it
   would be good to get a dedicated section on the Wiki.  I also think
   this will soon change with Rails 2.2 and multi-threading since we
   wont need as many runtimes in the heap so maybe now is a good time
   as any to address it.

   On your first comment , are you saying an Xmn of 128M is way too small ?

   Thx
   Adam


   On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver N.
   

       Yes, that's another area we really need more tuning
       help...perhaps we should start or update a page on the wiki and
       make it boldface and flashing?

       -Xmn is an important one because Ruby is so hard on the heap.
       There's soooo many objects being created and collected, it needs
       a pretty good sized new generation for most nontrivial apps.

       You should also get comfortable using jconsole. If you run JRuby
       in an app server or using the --manage flag, you can connect to
       it with jconsole and monitor heap utilization (various
       generations) GC time, and a whole host of JRuby statistics.

       I think this may be one of the last frontiers for getting JRuby
       to really peak performance (other than additional hackery and
       optimization, which we continue to do), since it's obvious that
       the default settings for the JVM are tuned to *Java*
       applications. We need to start exploring the hundreds of options
       (and perhaps dozens that are directly relevant) to know the
       right settings for *Ruby* applications, and make sure others
       know about them.

       - Charlie

       AD wrote:

           Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
           will see how things look.  I can say this, from a memory
           standpoint simply adding Xmn=128M to our JAVA_OPTS has
           dropped our average heap size from 3gb to about 1.7Gb .

           Will post my results later this week.

           adam

           On Wed, Dec 17, 2008 at 9:35 PM, Charles Oliver N.
           
           
           

              AD wrote:

                  Is there a way to determine the size of a runtime ?
            We keep
                  hitting OutofMem errors and just trying to figure out
           to tune
                  this thing.  We are going to force set the New Size
           via -Xmn to
                  something like 128M to force more frequent garbage
           collection,
                  will see if that helps.

                  we shouldnt really be hitting OutOfMem errors so easily ,
                  especially with a 4Gb heap on an 8Gb mem box.  We
           also turned on
                  thread.pooling to see how that behaves.


              OutOfMemoryErrors come in many flavors; oddly enough, you
           can also
              get OutOfMemoryError when the JVM can't spin up any more
           threads,
              which was the issue Peter Chan discovered. So try out
           1.1.6 and then
              get back to us if you still see issues.


              - Charlie

                       
---------------------------------------------------------------------
              To unsubscribe from this list, please visit:

                http://xircles.codehaus.org/manage_email





       ---------------------------------------------------------------------
       To unsubscribe from this list, please visit:

         http://xircles.codehaus.org/manage_email





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email



To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#2

We need to explore your problem in more detail, and probably enlist the
Rails guys. They’ll want to fix whatever it is…

Jay McGaffigan wrote:

as well as the performance improvements.

blocking, and
Subject: Re: [jruby-user] Rack Error
rails 2.2

AD wrote:
       at
:172)

   Thx
       -Xmn is an important one because Ruby is so hard on the heap.
       optimization, which we continue to do), since it's obvious that
           Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
           
                  something like 128M to force more frequent garbage
              OutOfMemoryErrors come in many flavors; oddly enough, you
                       
 ---------------------------------------------------------------------

--------------------------------------------------------------------- To
unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#3

Yes I agree! :slight_smile:

I am working to try and narrow it down.

I’ve posted to the rails email list but not gotten any real response.
I’ve asked at the chat room to the same effect. So I figured I’d wait a
couple of days before reposting and hopefully I Can get a better set of
information to share.

Jay


#4

Ping nzkoz on IRC or Twitter once you have something narrowed down. I
know he’ll want to get it repaired.

Jay McGaffigan wrote:

Sent: Thursday, December 18, 2008 11:27 PM

are getting a pariticular multithreaded problem. (we are getting the

solution
exhaustion?
(using soap/wsdldriver and soap/header/SimpleHandler) and it can
app server
are they
org.jruby.rack.RackInitializationException: timeout: all listeners busy
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
   this will soon change with Rails 2.2 and multi-threading since we
   On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver N.
       You should also get comfortable using jconsole. If you run JRuby
       right settings for *Ruby* applications, and make sure others


                  we shouldnt really be hitting OutOfMem errors so
              which was the issue Peter Chan discovered. So try out
                http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email