Forum: JRuby Re: RE: Rack Error

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Jay McGaffigan (Guest)
on 2008-12-19 05:55
(Received via mailing list)
<html><head>

<link media="all" type="text/css"
href="/webmail/static/deg/css/wysiwyg-4230203003.css" rel="stylesheet">
</head><body>
Going to Rails 2.2 may have it's own issues though.<br><br>Just today we
had to back rev to 2.1 for our first "beta" sight as we are getting a
pariticular multithreaded problem.&nbsp; (we are getting the body of our
request set to nil underneath us).<br><br>I'm trying to characterize
it... But it really only seems to happen under heavy load.&nbsp; If you
have a slow running job.&nbsp; 2.2 works pretty well.&nbsp; I'd
recommend it.&nbsp; The memory improvement is compelling.<br><br>as well
as the performance improvements.<br><br><br>Dec 18, 2008 12:36:40 PM,
removed_email_address@domain.invalid wrote:<br><blockquote style="border-left: 
3px
solid rgb(102, 153, 204);">How about:<br><br>jthread =
java.lang.Thread.new do puts "Running in a Java thread"
end<br>jthread.start<br><br>Come to think of it, why not just use the
Ruby thread? Thread.new?<br><br>Also, it sounds like if your conjecture
is right, then the problem is not with<br>JRuby, but rather the blocking
architecture of your app; Your only solution<br>would be to start up at
least as many runtime as your expected peak<br>concurrency. If so, you
should really upgrade to Rails 2.2, since that gives<br>some big memory
saving.<br><br>Of course, this is assuming that your problem is with
request blocking, and<br>not some kind of JRuby leak. Are you able to
tell what the heap size is, so as<br>to figure out whether it is runtime
concurrency exhaustion or memory<br>exhaustion?
<br><br>Peter<br><br>From: AD [mailto:removed_email_address@domain.invalid] 
<br>Sent:
Thursday, December 18, 2008 11:13 AM<br>To:
removed_email_address@domain.invalid<br>Subject: Re: [jruby-user] Rack
Error<br><br>So I think what *could* be causing this is we need to make
a remote soap calls<br>(using soap/wsdldriver and
soap/header/SimpleHandler) and it can take up to<br>30seconds to a
minute to process (they dont support Async and we are in the<br>process
of moving this to a JMS queue)<br><br>Is there an easy way to wrap this
in a Java thread (aside from a rails 2.2<br>upgrade) ? &nbsp;I think
what could be happening is 8 simultaneous calls are being<br>made tying
up all Jruby runtimes. &nbsp;What I dont get is why the whole app
server<br>dies after this happens.<br><br>We have 8 runtimes
available.<br><br>Adam<br>On Thu, Dec 18, 2008 at 11:24 AM, Charles
Oliver N.<br><br>Ok, now we can start looking at your problem in
earnest :)<br><br>Can you describe your configuration? How many
runtimes? How heavily are they<br>hit? The error below I believe means
that there's no runtimes available in the<br>pool, and so the request
can't be handled. What do you have to do to cause<br>this
error?<br><br>AD wrote:<br>Sorry guys, same error again on 1.1.6 with
rack 0.9.3<br><br>SEVERE: Exception
caught<br>org.jruby.rack.RackInitializationException: timeout: all
listeners busy<br>&nbsp; &nbsp; &nbsp;
&nbsp;at<br>org.jruby.rack.PoolingRackApplicationFactory.getApplication(PoolingRackApplica<br>tionFactory.java:87)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.jruby.rack.DefaultRackDispatcher.process(DefaultRackDispatcher.java:31)<br>&nbsp;
&nbsp; &nbsp; &nbsp;at
org.jruby.rack.RackFilter.doFilter(RackFilter.java:51)<br>&nbsp; &nbsp;
&nbsp;
&nbsp;at<br>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi<br>lterChain.java:215)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai<br>n.java:188)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java<br>:213)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java<br>:172)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)<br>&nbsp;
&nbsp; &nbsp;
&nbsp;at<br>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)<br><br>Adam<br><br><br>On
Thu, Dec 18, 2008 at 9:01 AM, AD <br><br><br>&nbsp; &nbsp;Thanks
Charlie, I couldnt agree more here. &nbsp;A lot of conversation
on<br>&nbsp; &nbsp;this group about different GC algorithms, PermGen,
Newsize, etc it<br>&nbsp; &nbsp;would be good to get a dedicated section
on the Wiki. &nbsp;I also think<br>&nbsp; &nbsp;this will soon change
with Rails 2.2 and multi-threading since we<br>&nbsp; &nbsp;wont need as
many runtimes in the heap so maybe now is a good time<br>&nbsp; &nbsp;as
any to address it.<br><br>&nbsp; &nbsp;On your first comment , are you
saying an Xmn of 128M is way too small ?<br><br>&nbsp;
&nbsp;Thx<br>&nbsp; &nbsp;Adam<br><br><br>&nbsp; &nbsp;On Thu, Dec 18,
2008 at 1:10 AM, Charles Oliver N.<br>&nbsp; &nbsp;<br><br>&nbsp;
&nbsp; &nbsp; &nbsp;Yes, that's another area we really need more
tuning<br>&nbsp; &nbsp; &nbsp; &nbsp;help...perhaps we should start or
update a page on the wiki and<br>&nbsp; &nbsp; &nbsp; &nbsp;make it
boldface and flashing?<br><br>&nbsp; &nbsp; &nbsp; &nbsp;-Xmn is an
important one because Ruby is so hard on the heap.<br>&nbsp; &nbsp;
&nbsp; &nbsp;There's soooo many objects being created and collected, it
needs<br>&nbsp; &nbsp; &nbsp; &nbsp;a pretty good sized new generation
for most nontrivial apps.<br><br>&nbsp; &nbsp; &nbsp; &nbsp;You should
also get comfortable using jconsole. If you run JRuby<br>&nbsp; &nbsp;
&nbsp; &nbsp;in an app server or using the --manage flag, you can
connect to<br>&nbsp; &nbsp; &nbsp; &nbsp;it with jconsole and monitor
heap utilization (various<br>&nbsp; &nbsp; &nbsp; &nbsp;generations) GC
time, and a whole host of JRuby statistics.<br><br>&nbsp; &nbsp; &nbsp;
&nbsp;I think this may be one of the last frontiers for getting
JRuby<br>&nbsp; &nbsp; &nbsp; &nbsp;to really peak performance (other
than additional hackery and<br>&nbsp; &nbsp; &nbsp; &nbsp;optimization,
which we continue to do), since it's obvious that<br>&nbsp; &nbsp;
&nbsp; &nbsp;the default settings for the JVM are tuned to
*Java*<br>&nbsp; &nbsp; &nbsp; &nbsp;applications. We need to start
exploring the hundreds of options<br>&nbsp; &nbsp; &nbsp; &nbsp;(and
perhaps dozens that are directly relevant) to know the<br>&nbsp; &nbsp;
&nbsp; &nbsp;right settings for *Ruby* applications, and make sure
others<br>&nbsp; &nbsp; &nbsp; &nbsp;know about them.<br><br>&nbsp;
&nbsp; &nbsp; &nbsp;- Charlie<br><br>&nbsp; &nbsp; &nbsp; &nbsp;AD
wrote:<br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Fair assesment.
&nbsp;We are rolling out 1.1.6 to dev/qa tomorrow<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;will see how things look. &nbsp;I can say
this, from a memory<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;standpoint simply adding Xmn=128M to our JAVA_OPTS has<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;dropped our average heap size from 3gb
to about 1.7Gb .<br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Will
post my results later this week.<br><br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp;adam<br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;On
Wed, Dec 17, 2008 at 9:35 PM, Charles Oliver N.<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br><br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; AD wrote:<br><br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Is there a way to determine
the size of a runtime ?<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; We
keep<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
hitting OutofMem errors and just trying to figure out<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;to tune<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; this thing. &nbsp;We are going to force set
the New Size<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;via -Xmn
to<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
something like 128M to force more frequent garbage<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;collection,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; will see if that helps.<br><br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; we shouldnt
really be hitting OutOfMem errors so easily ,<br>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; especially with a 4Gb heap on
an 8Gb mem box. &nbsp;We<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;also turned on<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; thread.pooling to see how that behaves.<br><br><br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; OutOfMemoryErrors come in many
flavors; oddly enough, you<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;can also<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; get
OutOfMemoryError when the JVM can't spin up any more<br>&nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp;threads,<br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; which was the issue Peter Chan discovered. So try
out<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1.1.6 and then<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; get back to us if you still
see issues.<br><br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
- Charlie<br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp;
&nbsp;<br>---------------------------------------------------------------------<br>&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; To unsubscribe from this list,
please visit:<br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;
http://xircles.codehaus.org/manage_email<br><br><b...
&nbsp; &nbsp;
&nbsp;---------------------------------------------------------------------<br>&nbsp;
&nbsp; &nbsp; &nbsp;To unsubscribe from this list, please
visit:<br><br>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp;http://xircles.codehaus.org/manage_email<br><br><b...
unsubscribe from this list, please visit:<br><br>&nbsp;
http://xircles.codehaus.org/manage_email<br><br><b...
unsubscribe from this list, please visit:<br><br>
http://xircles.codehaus.org/manage_email<br><br><b...

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Charles Oliver N. (Guest)
on 2008-12-19 06:27
(Received via mailing list)
We need to explore your problem in more detail, and probably enlist the
Rails guys. They'll want to fix whatever it is...

Jay McGaffigan wrote:
> as well as the performance improvements.
>
>     blocking, and
>     Subject: Re: [jruby-user] Rack Error
>     rails 2.2
>
>     AD wrote:
>            at
>     :172)
>
>        Thx
>            -Xmn is an important one because Ruby is so hard on the heap.
>            optimization, which we continue to do), since it's obvious that
>                Fair assesment.  We are rolling out 1.1.6 to dev/qa tomorrow
>
>                       something like 128M to force more frequent garbage
>                   OutOfMemoryErrors come in many flavors; oddly enough, you
>
>      ---------------------------------------------------------------------
>
>
> --------------------------------------------------------------------- To
> unsubscribe from this list, please visit:
> http://xircles.codehaus.org/manage_email


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Jay McGaffigan (Guest)
on 2008-12-19 15:17
(Received via mailing list)
Yes I agree! :)

I am working to try and narrow it down.

I've posted to the rails email list but not gotten any real response.
I've asked at the chat room to the same effect.  So I figured I'd wait a
couple of days before reposting and hopefully I Can get a better set of
information to share.

Jay
Charles Oliver N. (Guest)
on 2008-12-19 17:46
(Received via mailing list)
Ping nzkoz on IRC or Twitter once you have something narrowed down. I
know he'll want to get it repaired.

Jay McGaffigan wrote:
> Sent: Thursday, December 18, 2008 11:27 PM
>> are getting a pariticular multithreaded problem.  (we are getting the
>>
>>     solution
>>     exhaustion?
>>     (using soap/wsdldriver and soap/header/SimpleHandler) and it can
>>     app server
>>     are they
>>     org.jruby.rack.RackInitializationException: timeout: all listeners busy
>>     org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
>>     org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>        this will soon change with Rails 2.2 and multi-threading since we
>>        On Thu, Dec 18, 2008 at 1:10 AM, Charles Oliver N.
>>            You should also get comfortable using jconsole. If you run JRuby
>>            right settings for *Ruby* applications, and make sure others
>>
>>
>>                       we shouldnt really be hitting OutOfMem errors so
>>                   which was the issue Peter Chan discovered. So try out
>>                     http://xircles.codehaus.org/manage_email
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.