Forum: JRuby Error: Couldn't handle error: response committed

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-17 17:36
(Received via mailing list)
Starting to see a ton of these in my tomcat logs.  Not sure I
understand the origin of this error, but here is the backtrace.

http://pastie.org/391815

Charles, is this related to the other propogated caching issue ?

Adam

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
526d60de6472502bb570a9df2842b33b?d=identicon&s=25 Nick Sieger (Guest)
on 2009-02-17 20:05
(Received via mailing list)
On Tue, Feb 17, 2009 at 10:36 AM, AD <straightflush@gmail.com> wrote:
> Starting to see a ton of these in my tomcat logs.  Not sure I
> understand the origin of this error, but here is the backtrace.
>
> http://pastie.org/391815
>
> Charles, is this related to the other propogated caching issue ?

Not sure about the origin of the error. That message comes from code
in jruby-rack that catches an exception, and tries to rollback the
response so it can print an error message:

    private void handleException(Exception re, RackApplicationFactory
rackFactory,
            HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        if (response.isCommitted()) {
            servletContext.log("Error: Couldn't handle error: response
committed", re);
            return;
        }
        response.reset();
        .... // send an error response here
    }

Looks like something IO-related happened that prevented Tomcat from
allowing the response to be reset.

What version of JRuby are you using?

/Nick

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-18 01:12
(Received via mailing list)
jruby 1.1.6 and jruby-rack 0.9.3.

Could this be related to memcache issues ?



On Tue, Feb 17, 2009 at 1:56 PM, Nick Sieger <nicksieger@gmail.com>
wrote:
> response so it can print an error message:
>        response.reset();
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2009-02-18 04:52
(Received via mailing list)
AD wrote:
> Starting to see a ton of these in my tomcat logs.  Not sure I
> understand the origin of this error, but here is the backtrace.
>
> http://pastie.org/391815
>
> Charles, is this related to the other propogated caching issue ?

java.net.SocketException: Broken pipe looks like what happens when the
client shuts down the connection before the server is done sending.  At
any rate it doesn't seem like anything we've done would make that any
more likely than before...

Perhaps jruby-rack isn't handling the broken pipe gracefully? Or we're
not making it easy to handle gracefully?

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-18 04:56
(Received via mailing list)
Possible.  I am also seeing these with a similar error

ClientAbortException:  java.net.SocketException: Connection reset

So i guess it could be an issue of Tomcat closing the connection too
early

On Tue, Feb 17, 2009 at 10:51 PM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
> client shuts down the connection before the server is done sending.  At any
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-18 18:08
(Received via mailing list)
The bigger question here for me is, should we be disabling
"config.threadsafe!" until this issue is resolved?  We seem to get
more issues under high load in this scenario but i just want to be
sure.

On Tue, Feb 17, 2009 at 10:55 PM, AD <straightflush@gmail.com> wrote:
>>> Starting to see a ton of these in my tomcat logs.  Not sure I
>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2009-02-18 21:59
(Received via mailing list)
AD wrote:
> The bigger question here for me is, should we be disabling
> "config.threadsafe!" until this issue is resolved?  We seem to get
> more issues under high load in this scenario but i just want to be
> sure.

Well it's possible, but it seems unlikely. The individual requests
should be pretty well isolated. Is this error causing actual application
issues or just filling up logs? In either case it looks like perhaps
jruby-rack needs a better catch-all exception handler that knows to
ignore "broken pipe" if it comes up or to log a less verbose message for
it. Does that sound like it would help?

Also, do you have Tomcat fronted with an HTTP server like Apache or
Nginx? They sometimes have their own idea about socket lifecycle.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-19 03:37
(Received via mailing list)
The error seems to cause application issues under high load (we were
running a load test and things went haywire).  Not so concerned about
logfiles filling up if the errors are legit.  I am more concerned that
we moved backwards in performance by using Rails 2.2 and threadsafety
in the latest release, depending on how serious this error is.

We front Tomcat with apache using plain old ProxyPass to port 8080.
Keepalive is set to 1 (disabled) and maxthreads is set to around 600.
Machine is a quad core 8Gb RAM, Heap is set to 4Gb.  When loading 1000
or so connections things just went horribly wrong and saw all these
errors.

Adam

On Wed, Feb 18, 2009 at 3:58 PM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
> a better catch-all exception handler that knows to ignore "broken pipe" if
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2009-02-19 04:03
(Received via mailing list)
AD wrote:
> errors.
1000 concurrent connections? Were you able to handle that load under
pre-Rails 2.2? At that level it's possible Tomcat itself is culling
sockets to try to keep up. It really doesn't look like something we're
doing, other than making too much noise when the client cuts off.

Can you try gradually ramping up to see if there's a point where it
starts to fail?

To be honest it still seems like clients are dropping off early; maybe
the load is too high and the load tool is cutting them off?

- Charlie



---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
F15fdc7cb2e911b3808837f2be244add?d=identicon&s=25 AD (Guest)
on 2009-02-19 05:13
(Received via mailing list)
well its 1000 connections across 4 servers, each server with a
maxthread of about 500, and some sleep time put in (with JMeter).

I can try to ramp it up, but i also see a lot of those
"CacheDependant" Errors (we discussed on a previous thread fixed in
1.2) so just trying to sift through all of the noise and see if we
have a real problem.

We were able to handle that load to *some* degree, not really
concurrent b/c pre-2.2 was limited to max 8 runtimes per box *  4
boxes so requests would queue up a bit but still finish.  This test
just went ugly and south fast which is why I got concerned.  Let me
see what other data I can provide.

Adam

On Wed, Feb 18, 2009 at 10:03 PM, Charles Oliver Nutter
<charles.nutter@sun.com> wrote:
>> Machine is a quad core 8Gb RAM, Heap is set to 4Gb.  When loading 1000
>
>   http://xircles.codehaus.org/manage_email
>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2009-02-19 05:28
(Received via mailing list)
The code in question is just writing out to the Tomcat request output
stream, and then the client goes away. I think we'll need more help here
to investigate what's wrong.

Can you quantify "went ugly and south" a bit more? It's also possible
that performance tanks because we start spending a bunch of time
handling and logging this error.

- Charlie

AD wrote:
> boxes so requests would queue up a bit but still finish.  This test
>>> logfiles filling up if the errors are legit.  I am more concerned that
>> to try to keep up. It really doesn't look like something we're doing, other
>>
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.