Mongrel timeout error

Server Mongrel
1

My mongrel cluster seems to have some weird issues. Anyone here with
some insight into whats going on?

Tue Jan 13 09:36:34 -0800 2009: Reaping 7 threads for slow workers
because of ‘shutdown’
Thread #<Thread:0x2aaaabb8adc8 sleep> is too old, killing.
Tue Jan 13 09:36:34 -0800 2009: Error calling Dispatcher.dispatch
#<Mongrel::TimeoutError: Mongrel timed out this thread: shutdown>Thread
#<Thread:0x2aaaabb8d230 sleep> is too old, killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Tue Jan 13 09:36:34 -0800 2009: Error calling
Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread #<Thread:0x2aaaabb8f648 sleep> is too old,
killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Tue Jan 13 09:36:34 -0800 2009: Error calling
Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread #<Thread:0x2aaaabb90b88 sleep> is too old,
killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Tue Jan 13 09:36:34 -0800 2009: Error calling
Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread #<Thread:0x2aaaabb91fb0 sleep> is too old,
killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Tue Jan 13 09:36:34 -0800 2009: Error calling
Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread #<Thread:0x2aaaabb933d8 sleep> is too old,
killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Tue Jan 13 09:36:34 -0800 2009: Error calling
Dispatcher.dispatch #<Mongrel::TimeoutError: Mongrel timed out this
thread: shutdown>Thread #<Thread:0x2aaaabb96100 sleep> is too old,
killing.

/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:221:in
process' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:158:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:inprocess_client’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:ininitialize’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:285:in
new' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
initialize' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:innew’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel.rb:268:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/…/lib/mongrel/configurator.rb:281:in
each' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in
run' /usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:inrun’
/usr/lib64/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:19:in `load’
/usr/bin/mongrel_rails:19Waiting for 3 requests to finish, could take
60.0 seconds.

** Daemonized, any open files are closed. Look at
tmp/pids/mongrel.8000.pid and log/mongrel.8000.log for info.
** Starting Mongrel listening at 0.0.0.0:8000
** Initiating groups for “mongrel”:“mongrel”.
** Changing group to “mongrel”.
** Changing user to “mongrel”.
** Starting Rails with production environment…

2

Recommended: Troubleshooting Ruby Processes: Leveraging System Tools when the Usual Ruby Tricks Stop Working (Digital Short Cut) | InformIT

Specifically sections 3 and 4, on lsof and strace, the latter of which
should be informative re: what the threads are doing when they’re not
responding to a shutdown, provided you can catch one “in the act”.

Does this happen -every- time you issue a shutdown, or only after the
cluster has been running for awhile?

This is otherwise stock mongrel behaviour; it’s just trying to comply
with a
shutdown request, and will forcibly kill threads that are unresponsive.

–

3

This is otherwise stock mongrel behaviour; it’s just trying to comply
with
a
shutdown request, and will forcibly kill threads that are unresponsive.

Yup, you have one active request that is blocked on something (most
likely
the db, or a web services call). The standard mogrel shutdown behviour
is
to issue a thread#kill for each thread in the queue. The thread at the
head
of the queue gets a few seconds grace before it gets a forcefull kill.

Cheers

DAve