Forum: JRuby Rack Error

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-17 19:03
(Received via mailing list)
has anyone seen this error on rack 0.9.3 ? Using jruby 1.1.4 and rack
0.9.3

RackInitializationException: timeout: all listeners busy

Thx
Adam
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-17 19:23
(Received via mailing list)
AD wrote:
> has anyone seen this error on rack 0.9.3 ? Using jruby 1.1.4 and rack 0.9.3
>
> RackInitializationException: timeout: all listeners busy

Could be the same leak again :)

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-17 19:59
(Received via mailing list)
bah, OK.  My prod app servers keep running out of memory, hoping 1.1.6
and
Rails 2.2 fixes this.  Right now we have to restart our app servers
every
few days (max 10 days so far)  and clients are proving to lose patience
with
the instability at this point.

We have 8Gb RAM, 4GB Heap, 512Permsize, 8 max runtimes in 1 app and 4
max in
another, but still seem to keep hitting this.  JConsole showed threads
just
randomly jump through the roof and then the machine went kaput.

server -Xms4096m -Xmx4096m -XX ermSize=512m -XX:MaxPermSize=512m
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -XX:+HeapDumpOnOutOfMemoryError

We have a couple of heap dumps I will see what comes out of that..

Adam

On Wed, Dec 17, 2008 at 1:22 PM, Charles Oliver Nutter <
9ff4ad7c2fd1fc0fce18cb59f5649658?d=identicon&s=25 Peter K Chan (Guest)
on 2008-12-17 20:09
(Received via mailing list)
Did you try the 1.1.6 RC release? What do you mean by threads jumping
through
the roof?

If you have heap dump, I would suggest either yourkit or the Eclipse
Memory
Analyzer. Look at the largest size by class and see if you can find a
class of
object that is dominating memory usage.

Peter

From: AD [mailto:straightflush@gmail.com]
Sent: Wednesday, December 17, 2008 12:59 PM
To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error

bah, OK.  My prod app servers keep running out of memory, hoping 1.1.6
and
Rails 2.2 fixes this.  Right now we have to restart our app servers
every few
days (max 10 days so far)  and clients are proving to lose patience with
the
instability at this point.

We have 8Gb RAM, 4GB Heap, 512Permsize, 8 max runtimes in 1 app and 4
max in
another, but still seem to keep hitting this.  JConsole showed threads
just
randomly jump through the roof and then the machine went kaput.

server -Xms4096m -Xmx4096m -XX ermSize=512m -XX:MaxPermSize=512m
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -XX:+HeapDumpOnOutOfMemoryError

We have a couple of heap dumps I will see what comes out of that..

Adam
On Wed, Dec 17, 2008 at 1:22 PM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
AD wrote:
has anyone seen this error on rack 0.9.3 ? Using jruby 1.1.4 and rack
0.9.3

RackInitializationException: timeout: all listeners busy

Could be the same leak again :)

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-17 20:50
(Received via mailing list)
i didnt try 1.1.6RC yet, although based on Charles comments seems that I
should.  Essentially all of a sudden live threads went from about 150 to
400

Downloading heap dump now for use in YourKit.

Adam
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-17 22:13
(Received via mailing list)
Is there a way to determine the size of a runtime ?  We keep hitting
OutofMem errors and just trying to figure out to tune this thing.  We
are
going to force set the New Size via -Xmn to something like 128M to force
more frequent garbage collection, will see if that helps.

we shouldnt really be hitting OutOfMem errors so easily , especially
with a
4Gb heap on an 8Gb mem box.  We also turned on thread.pooling to see how
that behaves.

Adam
9ff4ad7c2fd1fc0fce18cb59f5649658?d=identicon&s=25 Peter K Chan (Guest)
on 2008-12-18 03:29
(Received via mailing list)
Not sure, but your best bet is to try out 1.1.6. Given some leaks fixed,
you
may find the problem solved without needing to dig into the heap dump.

Peter

From: AD [mailto:straightflush@gmail.com]
Sent: Wednesday, December 17, 2008 3:12 PM
To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error

Is there a way to determine the size of a runtime ?  We keep hitting
OutofMem
errors and just trying to figure out to tune this thing.  We are going
to
force set the New Size via -Xmn to something like 128M to force more
frequent
garbage collection, will see if that helps.

we shouldnt really be hitting OutOfMem errors so easily , especially
with a
4Gb heap on an 8Gb mem box.  We also turned on thread.pooling to see how
that
behaves.

Adam
On Wed, Dec 17, 2008 at 2:50 PM, AD <straightflush@gmail.com> wrote:
i didnt try 1.1.6RC yet, although based on Charles comments seems that I
should.  Essentially all of a sudden live threads went from about 150 to
400

Downloading heap dump now for use in YourKit.

Adam

On Wed, Dec 17, 2008 at 2:08 PM, Peter K Chan <peter@oaktop.com> wrote:
Did you try the 1.1.6 RC release? What do you mean by threads jumping
through
the roof?

If you have heap dump, I would suggest either yourkit or the Eclipse
Memory
Analyzer. Look at the largest size by class and see if you can find a
class of
object that is dominating memory usage.

Peter

From: AD [mailto:straightflush@gmail.com]
Sent: Wednesday, December 17, 2008 12:59 PM
To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error

bah, OK.  My prod app servers keep running out of memory, hoping 1.1.6
and
Rails 2.2 fixes this.  Right now we have to restart our app servers
every few
days (max 10 days so far)  and clients are proving to lose patience with
the
instability at this point.

We have 8Gb RAM, 4GB Heap, 512Permsize, 8 max runtimes in 1 app and 4
max in
another, but still seem to keep hitting this.  JConsole showed threads
just
randomly jump through the roof and then the machine went kaput.

server -Xms4096m -Xmx4096m -XX ermSize=512m -XX:MaxPermSize=512m
-Dcom.sun.management.jmxremote.port=8484
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -XX:+HeapDumpOnOutOfMemoryError

We have a couple of heap dumps I will see what comes out of that..

Adam
On Wed, Dec 17, 2008 at 1:22 PM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
AD wrote:
has anyone seen this error on rack 0.9.3 ? Using jruby 1.1.4 and rack
0.9.3

RackInitializationException: timeout: all listeners busy

Could be the same leak again :)

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-18 03:35
(Received via mailing list)
AD wrote:
> Is there a way to determine the size of a runtime ?  We keep hitting
> OutofMem errors and just trying to figure out to tune this thing.  We
> are going to force set the New Size via -Xmn to something like 128M to
> force more frequent garbage collection, will see if that helps.
>
> we shouldnt really be hitting OutOfMem errors so easily , especially
> with a 4Gb heap on an 8Gb mem box.  We also turned on thread.pooling to
> see how that behaves.

OutOfMemoryErrors come in many flavors; oddly enough, you can also get
OutOfMemoryError when the JVM can't spin up any more threads, which was
the issue Peter Chan discovered. So try out 1.1.6 and then get back to
us if you still see issues.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-18 05:09
(Received via mailing list)
Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow will see
how
things look.  I can say this, from a memory standpoint simply adding
Xmn=128M to our JAVA_OPTS has dropped our average heap size from 3gb to
about 1.7Gb .
Will post my results later this week.

adam

On Wed, Dec 17, 2008 at 9:35 PM, Charles Oliver Nutter <
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-18 07:10
(Received via mailing list)
Yes, that's another area we really need more tuning help...perhaps we
should start or update a page on the wiki and make it boldface and
flashing?

-Xmn is an important one because Ruby is so hard on the heap. There's
soooo many objects being created and collected, it needs a pretty good
sized new generation for most nontrivial apps.

You should also get comfortable using jconsole. If you run JRuby in an
app server or using the --manage flag, you can connect to it with
jconsole and monitor heap utilization (various generations) GC time, and
a whole host of JRuby statistics.

I think this may be one of the last frontiers for getting JRuby to
really peak performance (other than additional hackery and optimization,
which we continue to do), since it's obvious that the default settings
for the JVM are tuned to *Java* applications. We need to start exploring
the hundreds of options (and perhaps dozens that are directly relevant)
to know the right settings for *Ruby* applications, and make sure others
know about them.

- Charlie

AD wrote:
> <charles.nutter@sun.com <mailto:charles.nutter@sun.com>> wrote:
>         especially with a 4Gb heap on an 8Gb mem box.  We also turned on
>
>     ---------------------------------------------------------------------
>     To unsubscribe from this list, please visit:
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-18 15:02
(Received via mailing list)
Thanks Charlie, I couldnt agree more here.  A lot of conversation on
this
group about different GC algorithms, PermGen, Newsize, etc it would be
good
to get a dedicated section on the Wiki.  I also think this will soon
change
with Rails 2.2 and multi-threading since we wont need as many runtimes
in
the heap so maybe now is a good time as any to address it.

On your first comment , are you saying an Xmn of 128M is way too small ?

Thx
Adam

On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver Nutter <
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-18 16:28
(Received via mailing list)
Sorry guys, same error again on 1.1.6 with rack 0.9.3
SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
        at
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplicationFactory.java:87)
        at
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
        at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

Adam
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-18 17:23
(Received via mailing list)
AD wrote:
> Thanks Charlie, I couldnt agree more here.  A lot of conversation on
> this group about different GC algorithms, PermGen, Newsize, etc it would
> be good to get a dedicated section on the Wiki.  I also think this will
> soon change with Rails 2.2 and multi-threading since we wont need as
> many runtimes in the heap so maybe now is a good time as any to address it.
>
> On your first comment , are you saying an Xmn of 128M is way too small ?

No, in fact it's a pretty substantial size. And if it helps your app, it
sounds like a great size :)

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-18 17:24
(Received via mailing list)
Ok, now we can start looking at your problem in earnest :)

Can you describe your configuration? How many runtimes? How heavily are
they hit? The error below I believe means that there's no runtimes
available in the pool, and so the request can't be handled. What do you
have to do to cause this error?

AD wrote:
> 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>
>     this will soon change with Rails 2.2 and multi-threading since we
>     <charles.nutter@sun.com <mailto:charles.nutter@sun.com>> wrote:
>         in an app server or using the --manage flag, you can connect to
>         know about them.
>             Will post my results later this week.
>                    Is there a way to determine the size of a runtime ?
>                    especially with a 4Gb heap on an 8Gb mem box.  We
>                get back to us if you still see issues.
>
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-18 18:22
(Received via mailing list)
So I think what *could* be causing this is we need to make a remote soap
calls (using soap/wsdldriver and soap/header/SimpleHandler) and it can
take
up to 30seconds to a minute to process (they dont support Async and we
are
in the process of moving this to a JMS queue)
Is there an easy way to wrap this in a Java thread (aside from a rails
2.2
upgrade) ?  I think what could be happening is 8 simultaneous calls are
being made tying up all Jruby runtimes.  What I dont get is why the
whole
app server dies after this happens.

We have 8 runtimes available.

Adam

On Thu, Dec 18, 2008 at 11:24 AM, Charles Oliver Nutter <
9ff4ad7c2fd1fc0fce18cb59f5649658?d=identicon&s=25 Peter K Chan (Guest)
on 2008-12-18 18:36
(Received via mailing list)
How about:

jthread = java.lang.Thread.new do puts "Running in a Java thread" end
jthread.start

Come to think of it, why not just use the Ruby thread? Thread.new?

Also, it sounds like if your conjecture is right, then the problem is
not with
JRuby, but rather the blocking architecture of your app; Your only
solution
would be to start up at least as many runtime as your expected peak
concurrency. If so, you should really upgrade to Rails 2.2, since that
gives
some big memory saving.

Of course, this is assuming that your problem is with request blocking,
and
not some kind of JRuby leak. Are you able to tell what the heap size is,
so as
to figure out whether it is runtime concurrency exhaustion or memory
exhaustion?

Peter

From: AD [mailto:straightflush@gmail.com]
Sent: Thursday, December 18, 2008 11:13 AM
To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error

So I think what *could* be causing this is we need to make a remote soap
calls
(using soap/wsdldriver and soap/header/SimpleHandler) and it can take up
to
30seconds to a minute to process (they dont support Async and we are in
the
process of moving this to a JMS queue)

Is there an easy way to wrap this in a Java thread (aside from a rails
2.2
upgrade) ?  I think what could be happening is 8 simultaneous calls are
being
made tying up all Jruby runtimes.  What I dont get is why the whole app
server
dies after this happens.

We have 8 runtimes available.

Adam
On Thu, Dec 18, 2008 at 11:24 AM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
Ok, now we can start looking at your problem in earnest :)

Can you describe your configuration? How many runtimes? How heavily are
they
hit? The error below I believe means that there's no runtimes available
in the
pool, and so the request can't be handled. What do you have to do to
cause
this error?

AD wrote:
Sorry guys, same error again on 1.1.6 with rack 0.9.3

SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
       at
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplica
tionFactory.java:87)
       at
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
       at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:215)
       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:188)
       at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
       at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:172)
       at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

Adam


On Thu, Dec 18, 2008 at 9:01 AM, AD <straightflush@gmail.com
<mailto:straightflush@gmail.com>> wrote:

   Thanks Charlie, I couldnt agree more here.  A lot of conversation on
   this group about different GC algorithms, PermGen, Newsize, etc it
   would be good to get a dedicated section on the Wiki.  I also think
   this will soon change with Rails 2.2 and multi-threading since we
   wont need as many runtimes in the heap so maybe now is a good time
   as any to address it.

   On your first comment , are you saying an Xmn of 128M is way too
small ?

   Thx
   Adam


   On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver Nutter
   <charles.nutter@sun.com <mailto:charles.nutter@sun.com>> wrote:

       Yes, that's another area we really need more tuning
       help...perhaps we should start or update a page on the wiki and
       make it boldface and flashing?

       -Xmn is an important one because Ruby is so hard on the heap.
       There's soooo many objects being created and collected, it needs
       a pretty good sized new generation for most nontrivial apps.

       You should also get comfortable using jconsole. If you run JRuby
       in an app server or using the --manage flag, you can connect to
       it with jconsole and monitor heap utilization (various
       generations) GC time, and a whole host of JRuby statistics.

       I think this may be one of the last frontiers for getting JRuby
       to really peak performance (other than additional hackery and
       optimization, which we continue to do), since it's obvious that
       the default settings for the JVM are tuned to *Java*
       applications. We need to start exploring the hundreds of options
       (and perhaps dozens that are directly relevant) to know the
       right settings for *Ruby* applications, and make sure others
       know about them.

       - Charlie

       AD wrote:

           Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
           will see how things look.  I can say this, from a memory
           standpoint simply adding Xmn=128M to our JAVA_OPTS has
           dropped our average heap size from 3gb to about 1.7Gb .

           Will post my results later this week.

           adam

           On Wed, Dec 17, 2008 at 9:35 PM, Charles Oliver Nutter
           <charles.nutter@sun.com <mailto:charles.nutter@sun.com>
           <mailto:charles.nutter@sun.com
           <mailto:charles.nutter@sun.com>>> wrote:

              AD wrote:

                  Is there a way to determine the size of a runtime ?
            We keep
                  hitting OutofMem errors and just trying to figure out
           to tune
                  this thing.  We are going to force set the New Size
           via -Xmn to
                  something like 128M to force more frequent garbage
           collection,
                  will see if that helps.

                  we shouldnt really be hitting OutOfMem errors so
easily ,
                  especially with a 4Gb heap on an 8Gb mem box.  We
           also turned on
                  thread.pooling to see how that behaves.


              OutOfMemoryErrors come in many flavors; oddly enough, you
           can also
              get OutOfMemoryError when the JVM can't spin up any more
           threads,
              which was the issue Peter Chan discovered. So try out
           1.1.6 and then
              get back to us if you still see issues.


              - Charlie

                       
---------------------------------------------------------------------
              To unsubscribe from this list, please visit:

                http://xircles.codehaus.org/manage_email





     
 ---------------------------------------------------------------------
       To unsubscribe from this list, please visit:

         http://xircles.codehaus.org/manage_email





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2008-12-18 20:09
(Received via mailing list)
will wrapping this in a Thread.new remove the blocking on the Jruby
runtime
?
9ff4ad7c2fd1fc0fce18cb59f5649658?d=identicon&s=25 Peter K Chan (Guest)
on 2008-12-18 20:25
(Received via mailing list)
Yes, if you don't wait for the new Java thread in your code.



Note that once the SOAP call comes back asynchronously, the closure
context
may no longer be valid (since the request has already been processed),
so be
sure to test out your code for any dependency on accessing variables
from the
original request.



Peter



From: AD [mailto:straightflush@gmail.com]
Sent: Thursday, December 18, 2008 1:09 PM
To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error



will wrapping this in a Thread.new remove the blocking on the Jruby
runtime ?



On Thu, Dec 18, 2008 at 12:36 PM, Peter K Chan <peter@oaktop.com> wrote:

How about:

jthread = java.lang.Thread.new do puts "Running in a Java thread" end
jthread.start

Come to think of it, why not just use the Ruby thread? Thread.new?

Also, it sounds like if your conjecture is right, then the problem is
not with
JRuby, but rather the blocking architecture of your app; Your only
solution
would be to start up at least as many runtime as your expected peak
concurrency. If so, you should really upgrade to Rails 2.2, since that
gives
some big memory saving.

Of course, this is assuming that your problem is with request blocking,
and
not some kind of JRuby leak. Are you able to tell what the heap size is,
so as
to figure out whether it is runtime concurrency exhaustion or memory
exhaustion?


Peter

From: AD [mailto:straightflush@gmail.com]

Sent: Thursday, December 18, 2008 11:13 AM

To: user@jruby.codehaus.org
Subject: Re: [jruby-user] Rack Error

So I think what *could* be causing this is we need to make a remote soap
calls
(using soap/wsdldriver and soap/header/SimpleHandler) and it can take up
to
30seconds to a minute to process (they dont support Async and we are in
the
process of moving this to a JMS queue)

Is there an easy way to wrap this in a Java thread (aside from a rails
2.2
upgrade) ?  I think what could be happening is 8 simultaneous calls are
being
made tying up all Jruby runtimes.  What I dont get is why the whole app
server
dies after this happens.

We have 8 runtimes available.

Adam
On Thu, Dec 18, 2008 at 11:24 AM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
Ok, now we can start looking at your problem in earnest :)

Can you describe your configuration? How many runtimes? How heavily are
they
hit? The error below I believe means that there's no runtimes available
in the
pool, and so the request can't be handled. What do you have to do to
cause
this error?

AD wrote:
Sorry guys, same error again on 1.1.6 with rack 0.9.3

SEVERE: Exception caught
org.jruby.rack.RackInitializationException: timeout: all listeners busy
       at
org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplica
tionFactory.java:87)
       at
org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)
       at org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)
       at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
lterChain.java:215)
       at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
n.java:188)
       at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
:213)
       at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
:172)
       at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)

Adam


On Thu, Dec 18, 2008 at 9:01 AM, AD <straightflush@gmail.com
<mailto:straightflush@gmail.com>> wrote:

   Thanks Charlie, I couldnt agree more here.  A lot of conversation on
   this group about different GC algorithms, PermGen, Newsize, etc it
   would be good to get a dedicated section on the Wiki.  I also think
   this will soon change with Rails 2.2 and multi-threading since we
   wont need as many runtimes in the heap so maybe now is a good time
   as any to address it.

   On your first comment , are you saying an Xmn of 128M is way too
small ?

   Thx
   Adam


   On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver Nutter
   <charles.nutter@sun.com <mailto:charles.nutter@sun.com>> wrote:

       Yes, that's another area we really need more tuning
       help...perhaps we should start or update a page on the wiki and
       make it boldface and flashing?

       -Xmn is an important one because Ruby is so hard on the heap.
       There's soooo many objects being created and collected, it needs
       a pretty good sized new generation for most nontrivial apps.

       You should also get comfortable using jconsole. If you run JRuby
       in an app server or using the --manage flag, you can connect to
       it with jconsole and monitor heap utilization (various
       generations) GC time, and a whole host of JRuby statistics.

       I think this may be one of the last frontiers for getting JRuby
       to really peak performance (other than additional hackery and
       optimization, which we continue to do), since it's obvious that
       the default settings for the JVM are tuned to *Java*
       applications. We need to start exploring the hundreds of options
       (and perhaps dozens that are directly relevant) to know the
       right settings for *Ruby* applications, and make sure others
       know about them.

       - Charlie

       AD wrote:

           Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
           will see how things look.  I can say this, from a memory
           standpoint simply adding Xmn=128M to our JAVA_OPTS has
           dropped our average heap size from 3gb to about 1.7Gb .

           Will post my results later this week.

           adam

           On Wed, Dec 17, 2008 at 9:35 PM, Charles Oliver Nutter
           <charles.nutter@sun.com <mailto:charles.nutter@sun.com>
           <mailto:charles.nutter@sun.com
           <mailto:charles.nutter@sun.com>>> wrote:

              AD wrote:

                  Is there a way to determine the size of a runtime ?
            We keep
                  hitting OutofMem errors and just trying to figure out
           to tune
                  this thing.  We are going to force set the New Size
           via -Xmn to
                  something like 128M to force more frequent garbage
           collection,
                  will see if that helps.

                  we shouldnt really be hitting OutOfMem errors so
easily ,
                  especially with a 4Gb heap on an 8Gb mem box.  We
           also turned on
                  thread.pooling to see how that behaves.


              OutOfMemoryErrors come in many flavors; oddly enough, you
           can also
              get OutOfMemoryError when the JVM can't spin up any more
           threads,
              which was the issue Peter Chan discovered. So try out
           1.1.6 and then
              get back to us if you still see issues.


              - Charlie


---------------------------------------------------------------------
              To unsubscribe from this list, please visit:

                http://xircles.codehaus.org/manage_email





       ---------------------------------------------------------------------
       To unsubscribe from this list, please visit:

         http://xircles.codehaus.org/manage_email





---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email




---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-12-19 05:09
(Received via mailing list)
Yeah, I concur with Peter's assessment...tossing your long-running call
into a thread ought to work reasonably well. You're going to need to
make sure resources that thread might access later won't be concurrency
problems (like if it attempts to use AR on a pooled connection some new
request has already grabbed?) but if it's just to toss off a one-way
SOAP payload it ought to work ok.

Of course, if you need to get the responses back, you may want to look
at an offline queue you can toss jobs into. I think there's been some
discussion here about background processing in JRuby.

And yes, by all means, if you need this kind of concurrency you may very
well decrease memory and increase throughput subtantially by moving to
2.2 in threadsafe mode.

- Charlie

Peter K Chan wrote:
>
>
> jthread = java.lang.Thread.new do puts "Running in a Java thread" end
>
>
>
> On Thu, Dec 18, 2008 at 11:24 AM, Charles Oliver Nutter
> Sorry guys, same error again on 1.1.6 with rack 0.9.3
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
>        at
>
>    Adam
>        -Xmn is an important one because Ruby is so hard on the heap.
>        optimization, which we continue to do), since it's obvious that
>            Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
> <mailto:charles.nutter@sun.com <mailto:charles.nutter@sun.com>>
>                   this thing.  We are going to force set the New Size
>
>
>        ---------------------------------------------------------------------
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.