Upon starting Tomcat, we receive the following messages in our localhost
log file:
INFO: Info: using runtime pool timeout of 30 seconds
INFO: Info: received min runtimes = 2
INFO: Info: received max runtimes = 6
INFO: Info: add application to the pool. size now = 1
INFO: Info: add application to the pool. size now = 2
What specifically is it that will cause the pool size to grow and will
we receive any indication that it has grown? I’ve looked for messages
indicating applications have been added to the pool and run a variety of
tests trying to cause applications to get added to the pool and have
never seen additional applications added or don’t know how to check the
current pool size if a message is not printed to the localhost log.
The reason we’ve been looking into this is we receive the following
stack trace periodically which I am attributing to more traffic than we
are able/configured to handle:
SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
at
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplicationFactory.java:87)
at
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.InterruptedException
… 15 more
I went through a bunch of this pain prior to rails thread-safety
(which trust me comes with its own pain). I recommend using JConsole
to monitor the active threads and see if there is blocking anywhere.
You can also get a thread dump by doing a kill -QUIT on the java pid.
Reading these is by no means easy but may help see if you are blocking
all the runtimes and not releasing them for some reason.
Are you integrating with any other components, like JMS, Memcache,
long file uploads, external HTTP connections that could be blocking
the thread ?
Is this for rails? If so you can look at enabling config.threadsafe!
and setting min/max to 1/1 which auto turns on threading.
Adam
On Fri, Apr 17, 2009 at 12:59 PM, Howe, Ryan [email protected]
wrote:
INFO: Info: received min runtimes = 2
pool size if a message is not printed to the localhost log.
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
Yes, this is for rails. We are currently using 2.1.2 and as part of this
I’ve done some testing getting it running under 2.2.2 and in general
tried some different min/max configurations and also turned on
config.threadsafe! The config.threadsafe! seemed to alleviate this
specific issue but I’m concerned about other possible race condition
type issues as once I turned it on for my tests I had to research
memcache related threadsafe configurations. We are stable under the
current configuration but have been playing with different numbers of
runtimes and I’m interested in knowing if we are indeed growing to 6
runtimes with our current configuration or staying at the minimum with
2.
For the other questions, we are definitely using memcache and making
external HTTP connections in several places. These external connections
are indeed probably the root cause of our problems but at the same time,
the customer requires that we pull this data in to the pages. We use
timeouts but that is another problematic area from the research we’ve
done.
I went through a bunch of this pain prior to rails thread-safety
(which trust me comes with its own pain). I recommend using JConsole
to monitor the active threads and see if there is blocking anywhere.
You can also get a thread dump by doing a kill -QUIT on the java pid.
Reading these is by no means easy but may help see if you are blocking
all the runtimes and not releasing them for some reason.
Are you integrating with any other components, like JMS, Memcache,
long file uploads, external HTTP connections that could be blocking
the thread ?
Is this for rails? If so you can look at enabling config.threadsafe!
and setting min/max to 1/1 which auto turns on threading.
Adam
On Fri, Apr 17, 2009 at 12:59 PM, Howe, Ryan [email protected]
wrote:
INFO: Info: received min runtimes = 2
pool size if a message is not printed to the localhost log.
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
yea your best bet right now is JConsole to see what the threads are
doing and if they are blocking on something.
I wish i was better at this and could give you better advice, but if
you do find a blocking thread, post it here someone will certainly be
able to help.
Adam
On Fri, Apr 17, 2009 at 1:23 PM, Howe, Ryan [email protected]
wrote:
INFO: Info: add application to the pool. size now = 1
The reason we’ve been looking into this is we receive the following stack
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at java.lang.Thread.run(Thread.java:637)
No, that is just it. I have never seen any message (grepping the log for
the word “pool”) other than the entries at initialization adding the
“min” number to the pool.
I’ve also tried in my development environment to hammer a tomcat
instance using httperf and never successfully gotten a message that the
pool grew.
On Fri, Apr 17, 2009 at 11:59 AM, Howe, Ryan [email protected]
wrote:
INFO: Info: received min runtimes = 2
pool size if a message is not printed to the localhost log.
The reason we’ve been looking into this is we receive the following stack
trace periodically which I am attributing to more traffic than we are
able/configured to handle:
JRuby-Rack should initialize a new runtime on demand if there are none
available when a request comes in, up to the max runtimes. Do you see
messages in the log file like this, indicating you’ve reached the max?
On Fri, Apr 17, 2009 at 11:59 AM, Howe, Ryan [email protected]
wrote:
INFO: Info: received min runtimes = 2
pool size if a message is not printed to the localhost log.
The reason we’ve been looking into this is we receive the following stack
trace periodically which I am attributing to more traffic than we are
able/configured to handle:
JRuby-Rack should initialize a new runtime on demand if there are none
available when a request comes in, up to the max runtimes. Do you see
messages in the log file like this, indicating you’ve reached the max?
I also ran into this using Rails 3.2.3 on JRuby 1.6.7 (ruby 1.9)
deployed to Tomcat 7.0.x.
I had min=2, max=4 and threadsafe was not enabled (it was actually
commented out). I’ve changed min/max to equal 1 and I turned threadsafe
on…so far no problems.
Although I’m still wondering what the “proper” number of runtimes is if
there is one.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.