Forum: JRuby warbler-rack timeout q / circuit breaker

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Mike H. (Guest)
on 2008-12-21 16:07
(Received via mailing list)
I just upgraded to latest warbler/rack (0.9.12/0.9.3). I'll do some
testing on it, but wanted to run something past you ...

On our last round of load testing we saw our app, TriSano go into a
predictable death spiral once it got to a certain point of load.

I'm hoping to come up with some form of a "circuit breaker" to avoid
this. What seems to happen is as threads build up and wait, the app
bleeds resources and can never recover. I realize there is a bug fixed
around this in 1.1.6 (threads / memory), but I still would like this
configured so that rather than allowing threads to build up, the circuit
breaker kicks in and we tell the user to try again later (and the thread
queue stays low).

I observed in rack 0.9.2 that the rack timeout didn't kick in the way I
expected. Under load I couldn't get it to reject requests. I configured
it to 5 seconds, but it wouldn't reject requests that clearly were
waiting this long.

My real question is: is Warbler timeout a good choice for a circuit
breaker? How have you observed it to work in cases like this?

Mike

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Nick S. (Guest)
on 2008-12-21 21:12
(Received via mailing list)
On Sun, Dec 21, 2008 at 8:07 AM, Mike H. <removed_email_address@domain.invalid>
wrote:
> rather than allowing threads to build up, the circuit breaker kicks in and
> we tell the user to try again later (and the thread queue stays low).
>
> I observed in rack 0.9.2 that the rack timeout didn't kick in the way I
> expected. Under load I couldn't get it to reject requests. I configured it
> to 5 seconds, but it wouldn't reject requests that clearly were waiting this
> long.
>
> My real question is: is Warbler timeout a good choice for a circuit breaker?
> How have you observed it to work in cases like this?

Please do another round of load testing if you can. There was a bug on
the runtime pooling code that prevented it from hitting the maximum
number of runtimes, and instead each new request would attempt to
create another runtime when one wasn't available. The new release of
jruby-rack (0.9.3) should fix that, and hopefully you should be able
to observe the timeout. Do report a bug in the new JRuby-Rack JIRA [1]
if you still can't get the timeout behavior to happen.

Also note you need at least two runtimes for pooling to be used.

/Nick

[1]: http://kenai.com/jira/browse/JRUBY_RACK

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Mike H. (Guest)
on 2008-12-22 05:27
(Received via mailing list)
Thanks Nick.

Will do. Glad I asked.

I'll let you know of anything interesting in the results.

Mike


Nick S. wrote:
>> in 1.1.6 (threads / memory), but I still would like this configured so that
>
> /Nick
>
> [1]: http://kenai.com/jira/browse/JRUBY_RACK
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
>     http://xircles.codehaus.org/manage_email
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
AD (Guest)
on 2008-12-22 16:49
(Received via mailing list)
Nick when you say new release of  0.9.3, is this something that is in
Trunk
or officially released as a new version?  Mike said he was using 0.9.3
so
just want to make sure I am not confused about versions here.

Adam
Nick S. (Guest)
on 2008-12-22 19:53
(Received via mailing list)
Adam, it's a new version released at the end of November. I suggested
Mike try again because he mentioned he had only recently upgraded to
the new version, but hadn't tested it yet. Right Mike?

/Nick

On Mon, Dec 22, 2008 at 8:49 AM, AD <removed_email_address@domain.invalid> 
wrote:
>> > I just upgraded to latest warbler/rack (0.9.12/0.9.3). I'll do some
>> > this
>> > this
>> jruby-rack (0.9.3) should fix that, and hopefully you should be able
>> To unsubscribe from this list, please visit:
>>
>>    http://xircles.codehaus.org/manage_email
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.