Forum: JRuby Min/Max Runtime Configuration

Hello,

We currently have the following configuration in our warbler.rb:

  config.webxml.jruby.min.runtimes = 2  config.webxml.jruby.max.runtimes 
= 6

Upon starting Tomcat, we receive the following messages in our localhost 
log file:

INFO: Info: using runtime pool timeout of 30 seconds
INFO: Info: received min runtimes = 2
INFO: Info: received max runtimes = 6
INFO: Info: add application to the pool. size now = 1
INFO: Info: add application to the pool. size now = 2


What specifically is it that will cause the pool size to grow and will 
we receive any indication that it has grown? I've looked for messages 
indicating applications have been added to the pool and run a variety of 
tests trying to cause applications to get added to the pool and have 
never seen additional applications added or don't know how to check the 
current pool size if a message is not printed to the localhost log.

The reason we've been looking into this is we receive the following 
stack trace periodically which I am attributing to more traffic than we 
are able/configured to handle:

SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
        at 
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplicationFactory.java:87)
        at 
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
        at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
        at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
        at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
        at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.InterruptedException
        ... 15 more


We are using jruby 1.2.0 and jruby-rack 0.9.3.

Thanks
Ryan

Ryan,

 I went through a bunch of this pain prior to rails thread-safety
(which trust me comes with its own pain).  I recommend using JConsole
to monitor the active threads and see if there is blocking anywhere.
You can also get a thread dump by doing a kill -QUIT on the java pid.
Reading these is by no means easy but may help see if you are blocking
all the runtimes and not releasing them for some reason.

 Are you integrating with any other components, like JMS, Memcache,
long file uploads, external HTTP connections that could be blocking
the thread ?

 Is this for rails?  If so you can look at enabling config.threadsafe!
and setting min/max to 1/1 which auto turns on threading.

Adam

On Fri, Apr 17, 2009 at 12:59 PM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
> INFO: Info: received min runtimes = 2
> pool size if a message is not printed to the localhost log.
> org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>         at
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

Yes, this is for rails. We are currently using 2.1.2 and as part of this 
I've done some testing getting it running under 2.2.2 and in general 
tried some different min/max configurations and also turned on 
config.threadsafe! The config.threadsafe! seemed to alleviate this 
specific issue but I'm concerned about other possible race condition 
type issues as once I turned it on for my tests I had to research 
memcache related threadsafe configurations. We are stable under the 
current configuration but have been playing with different numbers of 
runtimes and I'm interested in knowing if we are indeed growing to 6 
runtimes with our current configuration or staying at the minimum with 
2.

For the other questions, we are definitely using memcache and making 
external HTTP connections in several places. These external connections 
are indeed probably the root cause of our problems but at the same time, 
the customer requires that we pull this data in to the pages. We use 
timeouts but that is another problematic area from the research we've 
done.


On 4/17/09 12:09 PM, "AD" <straightflush@gmail.com> wrote:

Ryan,

 I went through a bunch of this pain prior to rails thread-safety
(which trust me comes with its own pain).  I recommend using JConsole
to monitor the active threads and see if there is blocking anywhere.
You can also get a thread dump by doing a kill -QUIT on the java pid.
Reading these is by no means easy but may help see if you are blocking
all the runtimes and not releasing them for some reason.

 Are you integrating with any other components, like JMS, Memcache,
long file uploads, external HTTP connections that could be blocking
the thread ?

 Is this for rails?  If so you can look at enabling config.threadsafe!
and setting min/max to 1/1 which auto turns on threading.

Adam

On Fri, Apr 17, 2009 at 12:59 PM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
> INFO: Info: received min runtimes = 2
> pool size if a message is not printed to the localhost log.
> org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
>         at
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

yea your best bet right now is JConsole to see what the threads are
doing and if they are blocking on something.

I wish i was better at this and could give you better advice, but if
you do find a blocking thread, post it here someone will certainly be
able to help.

Adam

On Fri, Apr 17, 2009 at 1:23 PM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
>
>
>
>>
>> INFO: Info: add application to the pool. size now = 1
>> The reason we’ve been looking into this is we receive the following stack
>> org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
>>         at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>>         at java.lang.Thread.run(Thread.java:637)
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

On Fri, Apr 17, 2009 at 11:59 AM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
> INFO: Info: received min runtimes = 2
> pool size if a message is not printed to the localhost log.
>
> The reason we’ve been looking into this is we receive the following stack
> trace periodically which I am attributing to more traffic than we are
> able/configured to handle:

JRuby-Rack should initialize a new runtime on demand if there are none
available when a request comes in, up to the max runtimes. Do you see
messages in the log file like this, indicating you've reached the max?

add application to the pool. size now = 6

/Nick

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

No, that is just it. I have never seen any message (grepping the log for 
the word "pool") other than the entries at initialization adding the 
"min" number to the pool.

I've also tried in my development environment to hammer a tomcat 
instance using httperf and never successfully gotten a message that the 
pool grew.



On 4/17/09 3:01 PM, "Nick Sieger" <nicksieger@gmail.com> wrote:

On Fri, Apr 17, 2009 at 11:59 AM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
> INFO: Info: received min runtimes = 2
> pool size if a message is not printed to the localhost log.
>
> The reason we've been looking into this is we receive the following stack
> trace periodically which I am attributing to more traffic than we are
> able/configured to handle:

JRuby-Rack should initialize a new runtime on demand if there are none
available when a request comes in, up to the max runtimes. Do you see
messages in the log file like this, indicating you've reached the max?

add application to the pool. size now = 6

/Nick

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

For your production config, I'd suggest setting min = max. Maybe try
with both set to 6 and see how you handle the requests?

/Nick

On Fri, Apr 17, 2009 at 3:07 PM, Howe, Ryan <ryan@productiveedge.com> 
wrote:
>
>> log
>> receive any indication that it has grown? I’ve looked for messages
> available when a request comes in, up to the max runtimes. Do you see
>
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email

My configuration in warble.rb is as follows

  config.webxml.jruby.min.runtimes = 2
  config.webxml.jruby.max.runtimes = 4


and under production.rb

#config.threadsafe!


Error :
org.jruby.rack.RackInitializationException: timeout: all listeners busy

I am suddenly getting errors as above  and when I searched through the 
logs. This was what I found.

Mar 24, 2011 10:50:46 PM org.apache.catalina.core.ApplicationContext log
INFO: Info: add application to the pool. size now = 2


Any suggestions much appreciated.

Thanks for the help in advance,
Kesh.

Keshav P. wrote in post #989324:
> My configuration in warble.rb is as follows
>
>   config.webxml.jruby.min.runtimes = 2
>   config.webxml.jruby.max.runtimes = 4
>
>
> and under production.rb
>
> #config.threadsafe!
>
>
> Error :
> org.jruby.rack.RackInitializationException: timeout: all listeners busy
>
> I am suddenly getting errors as above  and when I searched through the
> logs. This was what I found.
>
> Mar 24, 2011 10:50:46 PM org.apache.catalina.core.ApplicationContext log
> INFO: Info: add application to the pool. size now = 2
>
>
> Any suggestions much appreciated.
>
> Thanks for the help in advance,
> Kesh.


Did you find an answer to this? I have the same problem.
many thanks,
Adam.

I also ran into this using Rails 3.2.3 on JRuby 1.6.7 (ruby 1.9) 
deployed to Tomcat 7.0.x.

I had min=2, max=4 and threadsafe was not enabled (it was actually 
commented out). I've changed min/max to equal 1 and I turned threadsafe 
on...so far no problems.

Although I'm still wondering what the "proper" number of runtimes is if 
there is one.

Nick Sieger (Warbler creator) suggests min=max for production.

Also, a while back, I remember having read (though I don't remember 
where)
a suggestion to set them to 6, which I did.

  config.webxml.jruby.min.runtimes = 6
  config.webxml.jruby.max.runtimes = 6

I could perform load testing or tuning to determine the perfect 
settings,
but I haven't bothered.

Thanks,
Jamie