Forum: JRuby wierd error on Tomcat

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
AD (Guest)
on 2008-12-05 22:16
(Received via mailing list)
Has anyone seen an error like this before ?

org.jruby.rack.RackInitializationException:
/usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb (Too
many
open files)
Charles Oliver N. (Guest)
on 2008-12-07 12:22
(Received via mailing list)
AD wrote:
> Has anyone seen an error like this before ?
>
> org.jruby.rack.RackInitializationException:
> /usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb (Too
> many open files)

Yes! :)

What setup are you running on? Does it have an unreasonably low ulimit?

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-09 05:59
(Received via mailing list)
redhat enterprise linux.  How do I confirm ulimit ?
ulimit -n
1024

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71679
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 71679
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

On Sun, Dec 7, 2008 at 5:22 AM, Charles Oliver N. <
Charles Oliver N. (Guest)
on 2008-12-09 12:32
(Received via mailing list)
Can you elaborate on when you're seeing that error? A server that's been
up for a while and had multiple redeploys perhaps?

AD wrote:
> pending signals                 (-i) 71679
> file locks                      (-x) unlimited
>         (Too many open files)
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-09 20:44
(Received via mailing list)
yea server was up for a week or two, in production and then just started
throwing these errors.  Not sure if there were any redeploys, we usually
stop/start tomcat for new war deployments so probably not.

Adam

On Tue, Dec 9, 2008 at 5:31 AM, Charles Oliver N. <
Phil R. (Guest)
on 2008-12-09 22:32
(Received via mailing list)
On Tue, Dec 9, 2008 at 10:43 AM, AD <removed_email_address@domain.invalid> 
wrote:
> yea server was up for a week or two, in production and then just started
> throwing these errors.  Not sure if there were any redeploys, we usually
> stop/start tomcat for new war deployments so probably not.
>
> Adam


  You should consider raising the limit for max number of files open.
You can probably edit /etc/security/limits.conf and insert something
for the user you're running the server as with the nofile setting to
something like 32768 and adjust your start scripts to set the limit
accordingly.  I'm not sure if redhat enterprise uses /etc/security or
not, but that's the path I would investigate.


>>>
>>> open files                      (-n) 1024
>>> <removed_email_address@domain.invalid <mailto:removed_email_address@domain.invalid>> 
wrote:
>>>    Yes! :)
>>>
>>>    What setup are you running on? Does it have an unreasonably low
>>> ulimit?
>>>
>>>    - Charlie
>>>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2008-12-10 12:10
(Received via mailing list)
Ok, there is a bug where we were leaking threads, which could cause this
error. It will be fixed in 1.1.6 final, and you can try it yourself by
building trunk or 1.1.6RC2 tag or downloading one of the nightly builds
from http://jruby.headius.com:8080/hudson in the "dist" project.

Ideally, you should test this ASAP, so if there's a real problem it
might be fixable in 1.1.6.

- Charlie

AD wrote:
>     been up for a while and had multiple redeploys perhaps?
>         max nice                        (-e) 0
>         max user processes              (-u) 71679
>                Has anyone seen an error like this before ?
>         low ulimit?
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe from this list, please visit:
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-11 05:34
(Received via mailing list)
Interesting, I did notice that in Jconsole the number of threads were
never
decreasing, is this related?  We are on 1.1.4

On Wed, Dec 10, 2008 at 5:10 AM, Charles Oliver N. <
Charles Oliver N. (Guest)
on 2008-12-11 17:19
(Received via mailing list)
Very likely.

AD wrote:
>     "dist" project.
>         redeploys, we usually stop/start tomcat for new war deployments
>         that's
>                data seg size           (kbytes, -d) unlimited
>                cpu time               (seconds, -t) unlimited
>
>
>         ---------------------------------------------------------------------
>            To unsubscribe from this list, please visit:
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-15 16:07
(Received via mailing list)
Saw this again with 1.1.4 is this the same bug ?

Happened after running smooth for about a week again.  Didnt get a
chance to
change ulimit yet, what is a safe number to increase this to?  Is 4096
enough ?

java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:312)
        at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:666)
        at
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:876)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
        at java.lang.Thread.run(Thread.java:619)


Thx
Adam

On Thu, Dec 11, 2008 at 10:19 AM, Charles Oliver N. <
Charles Oliver N. (Guest)
on 2008-12-15 19:46
(Received via mailing list)
Probably still the same bug :) It wasn't fixed until 1.1.6ish, and then
I think possibly only in RC2. Try 1.1.6RC1, and if that doesn't resolve
it see if you can pull a nightly build of 1.1.7 from this URL:

http://jruby.headius.com:8080/hudson/job/jruby-dist/

AD wrote:
>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>
>
>         wrote:
>         problem it
>                redeploys, we usually stop/start tomcat for new war
>         <mailto:removed_email_address@domain.invalid>>>>
>
>                       open files                      (-n) 1024
>                       <removed_email_address@domain.invalid
>                <mailto:removed_email_address@domain.invalid
>          /usr/local/tomcat/webapps/testapp/WEB-INF/config/environment.rb
>
>
>          ---------------------------------------------------------------------
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-15 20:56
(Received via mailing list)
is there anything we can do to clean up the thread usage aside from
restarting the app server once per week?  We are looking to roll out a
new
release probably prior to 1.1.6 being GA (currently in 1.1.4).  Will
certainly migrate to 1.1.6 when its out of RC state

Thx
Adam

On Mon, Dec 15, 2008 at 12:45 PM, Charles Oliver N. <
Charles Oliver N. (Guest)
on 2008-12-15 21:01
(Received via mailing list)
If there's any setting to get Tomcat to reuse threads more, that might
help. Otherwise I'm afraid there's not really an answer.

Please, please try to confirm you don't see the problem under a 1.1.7
nightly, so we can be assured it will fix your problem :)

AD wrote:
>
>         Happened after running smooth for about a week again.  Didnt get
>
>
>         org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:876)
>
>         threads
>                   Ok, there is a bug where we were leaking threads,
>
>
>                       On Tue, Dec 9, 2008 at 5:31 AM, Charles Oliver N.
>         <mailto:removed_email_address@domain.invalid>
>
>                              max nice                        (-e) 0
>                              max user processes              (-u) 71679
>                <mailto:removed_email_address@domain.invalid
>                <mailto:removed_email_address@domain.invalid
>                <mailto:removed_email_address@domain.invalid
>         org.jruby.rack.RackInitializationException:
>                              low ulimit?
>
>
>
>     ---------------------------------------------------------------------
>     To unsubscribe from this list, please visit:
>
>       http://xircles.codehaus.org/manage_email
>
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-16 06:43
(Received via mailing list)
thanks charles, are there jruby-compete's that are built as well for
this ?
Also confirmed via lsof that open files for tomcat was 1104, blowing the
default of 1024 out.  Raised it to 4096, hopefully prevents the issue
until
we can update production.

NOw just gotta fix this IOError :-)

Adam

On Mon, Dec 15, 2008 at 2:00 PM, Charles Oliver N. <
Charles Oliver N. (Guest)
on 2008-12-17 23:50
(Received via mailing list)
AD wrote:
> thanks charles, are there jruby-compete's that are built as well for this ?
>
> Also confirmed via lsof that open files for tomcat was 1104, blowing the
> default of 1024 out.  Raised it to 4096, hopefully prevents the issue
> until we can update production.
>
> NOw just gotta fix this IOError :-)

With the 1.1.6 release, we've started to push jruby-complete jars as
well. Happy hunting!

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.