Starting to see a ton of these in my tomcat logs. Not sure I
understand the origin of this error, but here is the backtrace.
Charles, is this related to the other propogated caching issue ?
Adam
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
On Tue, Feb 17, 2009 at 10:36 AM, AD [email protected] wrote:
Starting to see a ton of these in my tomcat logs. Not sure I
understand the origin of this error, but here is the backtrace.Charles, is this related to the other propogated caching issue ?
Not sure about the origin of the error. That message comes from code
in jruby-rack that catches an exception, and tries to rollback the
response so it can print an error message:
private void handleException(Exception re, RackApplicationFactory
rackFactory,
HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
if (response.isCommitted()) {
servletContext.log(“Error: Couldn’t handle error: response
committed”, re);
return;
}
response.reset();
… // send an error response here
}
Looks like something IO-related happened that prevented Tomcat from
allowing the response to be reset.
What version of JRuby are you using?
/Nick
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
jruby 1.1.6 and jruby-rack 0.9.3.
Could this be related to memcache issues ?
On Tue, Feb 17, 2009 at 1:56 PM, Nick S. [email protected]
wrote:
response so it can print an error message:
response.reset();To unsubscribe from this list, please visit:
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
AD wrote:
Starting to see a ton of these in my tomcat logs. Not sure I
understand the origin of this error, but here is the backtrace.Charles, is this related to the other propogated caching issue ?
java.net.SocketException: Broken pipe looks like what happens when the
client shuts down the connection before the server is done sending. At
any rate it doesn’t seem like anything we’ve done would make that any
more likely than before…
Perhaps jruby-rack isn’t handling the broken pipe gracefully? Or we’re
not making it easy to handle gracefully?
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
Possible. I am also seeing these with a similar error
ClientAbortException: java.net.SocketException: Connection reset
So i guess it could be an issue of Tomcat closing the connection too
early
On Tue, Feb 17, 2009 at 10:51 PM, Charles Oliver N.
[email protected] wrote:
client shuts down the connection before the server is done sending. At any
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
The bigger question here for me is, should we be disabling
“config.threadsafe!” until this issue is resolved? We seem to get
more issues under high load in this scenario but i just want to be
sure.
On Tue, Feb 17, 2009 at 10:55 PM, AD [email protected] wrote:
Starting to see a ton of these in my tomcat logs. Not sure I
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
The error seems to cause application issues under high load (we were
running a load test and things went haywire). Not so concerned about
logfiles filling up if the errors are legit. I am more concerned that
we moved backwards in performance by using Rails 2.2 and threadsafety
in the latest release, depending on how serious this error is.
We front Tomcat with apache using plain old ProxyPass to port 8080.
Keepalive is set to 1 (disabled) and maxthreads is set to around 600.
Machine is a quad core 8Gb RAM, Heap is set to 4Gb. When loading 1000
or so connections things just went horribly wrong and saw all these
errors.
Adam
On Wed, Feb 18, 2009 at 3:58 PM, Charles Oliver N.
[email protected] wrote:
a better catch-all exception handler that knows to ignore “broken pipe” if
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
AD wrote:
errors.
1000 concurrent connections? Were you able to handle that load under
pre-Rails 2.2? At that level it’s possible Tomcat itself is culling
sockets to try to keep up. It really doesn’t look like something we’re
doing, other than making too much noise when the client cuts off.
Can you try gradually ramping up to see if there’s a point where it
starts to fail?
To be honest it still seems like clients are dropping off early; maybe
the load is too high and the load tool is cutting them off?
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_emailwell its 1000 connections across 4 servers, each server with a
maxthread of about 500, and some sleep time put in (with JMeter).
I can try to ramp it up, but i also see a lot of those
“CacheDependant” Errors (we discussed on a previous thread fixed in
1.2) so just trying to sift through all of the noise and see if we
have a real problem.
We were able to handle that load to some degree, not really
concurrent b/c pre-2.2 was limited to max 8 runtimes per box * 4
boxes so requests would queue up a bit but still finish. This test
just went ugly and south fast which is why I got concerned. Let me
see what other data I can provide.
Adam
On Wed, Feb 18, 2009 at 10:03 PM, Charles Oliver N.
[email protected] wrote:
Machine is a quad core 8Gb RAM, Heap is set to 4Gb. When loading 1000
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
AD wrote:
The bigger question here for me is, should we be disabling
“config.threadsafe!” until this issue is resolved? We seem to get
more issues under high load in this scenario but i just want to be
sure.
Well it’s possible, but it seems unlikely. The individual requests
should be pretty well isolated. Is this error causing actual application
issues or just filling up logs? In either case it looks like perhaps
jruby-rack needs a better catch-all exception handler that knows to
ignore “broken pipe” if it comes up or to log a less verbose message for
it. Does that sound like it would help?
Also, do you have Tomcat fronted with an HTTP server like Apache or
Nginx? They sometimes have their own idea about socket lifecycle.
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_emailThe code in question is just writing out to the Tomcat request output
stream, and then the client goes away. I think we’ll need more help here
to investigate what’s wrong.
Can you quantify “went ugly and south” a bit more? It’s also possible
that performance tanks because we start spending a bunch of time
handling and logging this error.
AD wrote:
boxes so requests would queue up a bit but still finish. This test
logfiles filling up if the errors are legit. I am more concerned that
to try to keep up. It really doesn’t look like something we’re doing, otherTo unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_emailThis forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs