Wierd error on Tomcat


#1

Has anyone seen an error like this before ?

org.jruby.rack.RackInitializationException:
/usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb (Too
many
open files)


#2

AD wrote:

Has anyone seen an error like this before ?

org.jruby.rack.RackInitializationException:
/usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb (Too
many open files)

Yes! :slight_smile:

What setup are you running on? Does it have an unreasonably low ulimit?

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#3

redhat enterprise linux. How do I confirm ulimit ?
ulimit -n
1024

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 71679
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 71679
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

On Sun, Dec 7, 2008 at 5:22 AM, Charles Oliver N. <


#4

Can you elaborate on when you’re seeing that error? A server that’s been
up for a while and had multiple redeploys perhaps?

AD wrote:

pending signals (-i) 71679
file locks (-x) unlimited
(Too many open files)

  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#5

yea server was up for a week or two, in production and then just started
throwing these errors. Not sure if there were any redeploys, we usually
stop/start tomcat for new war deployments so probably not.

Adam

On Tue, Dec 9, 2008 at 5:31 AM, Charles Oliver N. <


#6

On Tue, Dec 9, 2008 at 10:43 AM, AD removed_email_address@domain.invalid wrote:

yea server was up for a week or two, in production and then just started
throwing these errors. Not sure if there were any redeploys, we usually
stop/start tomcat for new war deployments so probably not.

Adam

You should consider raising the limit for max number of files open.
You can probably edit /etc/security/limits.conf and insert something
for the user you’re running the server as with the nofile setting to
something like 32768 and adjust your start scripts to set the limit
accordingly. I’m not sure if redhat enterprise uses /etc/security or
not, but that’s the path I would investigate.

open files (-n) 1024
<removed_email_address@domain.invalid mailto:removed_email_address@domain.invalid> wrote:
Yes! :slight_smile:

What setup are you running on? Does it have an unreasonably low
ulimit?

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#7

Ok, there is a bug where we were leaking threads, which could cause this
error. It will be fixed in 1.1.6 final, and you can try it yourself by
building trunk or 1.1.6RC2 tag or downloading one of the nightly builds
from http://jruby.headius.com:8080/hudson in the “dist” project.

Ideally, you should test this ASAP, so if there’s a real problem it
might be fixable in 1.1.6.

  • Charlie

AD wrote:

been up for a while and had multiple redeploys perhaps?
    max nice                        (-e) 0
    max user processes              (-u) 71679
           Has anyone seen an error like this before ?
    low ulimit?



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#8

Very likely.

AD wrote:

"dist" project.
    redeploys, we usually stop/start tomcat for new war deployments
    that's
           data seg size           (kbytes, -d) unlimited
           cpu time               (seconds, -t) unlimited


    ---------------------------------------------------------------------
       To unsubscribe from this list, please visit:
  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#9

Interesting, I did notice that in Jconsole the number of threads were
never
decreasing, is this related? We are on 1.1.4

On Wed, Dec 10, 2008 at 5:10 AM, Charles Oliver N. <


#10

Saw this again with 1.1.4 is this the same bug ?

Happened after running smooth for about a week again. Didnt get a
chance to
change ulimit yet, what is a safe number to increase this to? Is 4096
enough ?

java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:312)
at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)
at
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)

Thx
Adam

On Thu, Dec 11, 2008 at 10:19 AM, Charles Oliver N. <


#11

is there anything we can do to clean up the thread usage aside from
restarting the app server once per week? We are looking to roll out a
new
release probably prior to 1.1.6 being GA (currently in 1.1.4). Will
certainly migrate to 1.1.6 when its out of RC state

Thx
Adam

On Mon, Dec 15, 2008 at 12:45 PM, Charles Oliver N. <


#12

Probably still the same bug :slight_smile: It wasn’t fixed until 1.1.6ish, and then
I think possibly only in RC2. Try 1.1.6RC1, and if that doesn’t resolve
it see if you can pull a nightly build of 1.1.7 from this URL:

http://jruby.headius.com:8080/hudson/job/jruby-dist/

AD wrote:

    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)                                                                                                                                    


    wrote:
    problem it
           redeploys, we usually stop/start tomcat for new war
    <mailto:removed_email_address@domain.invalid>>>>

                  open files                      (-n) 1024
                  <removed_email_address@domain.invalid
           <mailto:removed_email_address@domain.invalid
     /usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb

                   
     ---------------------------------------------------------------------

  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#13

If there’s any setting to get Tomcat to reuse threads more, that might
help. Otherwise I’m afraid there’s not really an answer.

Please, please try to confirm you don’t see the problem under a 1.1.7
nightly, so we can be assured it will fix your problem :slight_smile:

AD wrote:

    Happened after running smooth for about a week again.  Didnt get
         
                                                                   
    org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:876)

    threads
              Ok, there is a bug where we were leaking threads,


                  On Tue, Dec 9, 2008 at 5:31 AM, Charles Oliver N.
    <mailto:removed_email_address@domain.invalid>

                         max nice                        (-e) 0
                         max user processes              (-u) 71679
           <mailto:removed_email_address@domain.invalid
           <mailto:removed_email_address@domain.invalid
           <mailto:removed_email_address@domain.invalid
    org.jruby.rack.RackInitializationException:
                         low ulimit?



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#14

AD wrote:

thanks charles, are there jruby-compete’s that are built as well for this ?

Also confirmed via lsof that open files for tomcat was 1104, blowing the
default of 1024 out. Raised it to 4096, hopefully prevents the issue
until we can update production.

NOw just gotta fix this IOError :slight_smile:

With the 1.1.6 release, we’ve started to push jruby-complete jars as
well. Happy hunting!

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

#15

thanks charles, are there jruby-compete’s that are built as well for
this ?
Also confirmed via lsof that open files for tomcat was 1104, blowing the
default of 1024 out. Raised it to 4096, hopefully prevents the issue
until
we can update production.

NOw just gotta fix this IOError :slight_smile:

Adam

On Mon, Dec 15, 2008 at 2:00 PM, Charles Oliver N. <