Debugging endless loops


#1

Hi all,

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and
you don’t know how to reproduce it anywhere else. How would you debug
this?

Debugging these conditions in Java is trivial because of the “thread
dump” feature of JVM (where you send a kill -3 signal to a JVM process
and it spits out stack tracces of all threads to stdout). In any
environemnt where you can attach a debugger to a running app, it is not
heinously difficult either (although it may be problematic to attach a
debugger to a productiuon environment).

But what do I do if it’s Ruby?

Alex Verkhovsky


#2

Alexey V. wrote:

Hi all,

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and
you don’t know how to reproduce it anywhere else. How would you debug
this?

Debugging these conditions in Java is trivial because of the “thread
dump” feature of JVM (where you send a kill -3 signal to a JVM process
and it spits out stack tracces of all threads to stdout). In any
environemnt where you can attach a debugger to a running app, it is not
heinously difficult either (although it may be problematic to attach a
debugger to a productiuon environment).

But what do I do if it’s Ruby?

Can you do ruby -rdebug program.rb instead of running it normally?

Alex Verkhovsky

E


#3

Alexey V. wrote:

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and
you don’t know how to reproduce it anywhere else. How would you debug this?

But what do I do if it’s Ruby?

This might work and give you a remote debugging back door. I’m just not
sure if it will survive deadlocks:

require ‘breakpoint’
Breakpoint.activate_drb

Thread.new do
Thread.current.priority = 100
loop do
begin
breakpoint
rescue Exception
end
end
end

You can then connect to your application with breakpoint_client.

You’ll need the ruby-breakpoint library which is available from
http://ruby-breakpoint.rubyforge.org/


#4

Alexey V. wrote:

Hi, Florian,

Moin Alexey.

Remote ‘breakpointer’ might be a workable idea. But when I am in IRB,
how do I get stack traces for all currently running threads?

Ouch, I’m afraid this is yet another case where Ruby’s reflection isn’t
as useful as it could be…

You can get the backtrace via Thread#raise (set
thread.abort_on_exception to true first and catch the exception which
will have the backtrace) but it has the rather unfortunate side effect
of killing Ruby in the process.

I really think that Thread#caller would be a good idea…

Rest assured, I do know where to find ‘breakpoint’ library - I keep
telling about it to everyone pissed off or puzzled by the absence of a
decent debugger in the Ruby world.

And I hope I’ll be able to finally add a sensible GUI soon enough.
Thanks for the kind words.


#5

Florian GroÃ? wrote:

I really think that Thread#caller would be a good idea…

As well as a thread dump default signal handler on “kill -3”. In some
circumstances, this Java feature is just priceless.

Alex


#6

On Mon, 23 Jan 2006, Alexey V. wrote:

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and you
don’t know how to reproduce it anywhere else. How would you debug this?

Thread.abort_on_exception = true

John C. Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : removed_email_address@domain.invalid
New Zealand

Carter’s Clarification of Murphy’s Law.

“Things only ever go right so that they may go more spectacularly wrong
later.”

From this principle, all of life and physics may be deduced.


#7

Eero S. wrote:

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and
you don’t know how to reproduce it anywhere else. How would you debug
this?

Can you do ruby -rdebug program.rb instead of running it normally?

By the conditions of our imaginary problem, we only reproduce our
endless loop in the production environment, and -rdebug makes a typical
Ruby code two orders of magnitude slower, so this wouldn’t do.

Florian GroÃ? wrote:

This might work and give you a remote debugging back door. I’m just
not sure if it will survive deadlocks:
require ‘breakpoint’

You can then connect to your application with breakpoint_client.

Hi, Florian,
Remote ‘breakpointer’ might be a workable idea. But when I am in IRB,
how do I get stack traces for all currently running threads?
Rest assured, I do know where to find ‘breakpoint’ library - I keep
telling about it to everyone pissed off or puzzled by the absence of a
decent debugger in the Ruby world.

Alex


#8

On Mon, 23 Jan 2006, Alexey V. wrote:

John C. wrote:

Say, we have a Ruby app going into an endless loop (or perhaps a thread
deadlock) mode. Let us further assume that it happens in production and
you don’t know how to reproduce it anywhere else. How would you debug
this?

Thread.abort_on_exception = true

Neither endless loop, nor deadlock are exception conditions.

Yup, but if a thread dies and you don’t have a handler of last resort
nor
have set abort on exception, it just dies silently. ie. You may think
your
problem is a deadlock or infinite loop, but your actual problem may be a
needed thread has quietly died.

John C. Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : removed_email_address@domain.invalid
New Zealand

Carter’s Clarification of Murphy’s Law.

“Things only ever go right so that they may go more spectacularly wrong
later.”

From this principle, all of life and physics may be deduced.


#9

Alexey V. wrote:

environemnt where you can attach a debugger to a running app, it is
not heinously difficult either (although it may be problematic to
attach a debugger to a productiuon environment).

But what do I do if it’s Ruby?

You can write a singnal handler that outputs current thread states.
set_trace_func may help here also.

Kind regards

robert

#10

John C. wrote:

Say, we have a Ruby app going into an endless loop (or perhaps a
thread deadlock) mode. Let us further assume that it happens in
production and you don’t know how to reproduce it anywhere else. How
would you debug this?

Thread.abort_on_exception = true

Neither endless loop, nor deadlock are exception conditions.

Alex


#11

In message removed_email_address@domain.invalid, Alexey V.
removed_email_address@domain.invalid writes

But what do I do if it’s Ruby?

What about a flow tracer?

Ruby Bug Validator. http://www.softwareverify.com

Stephen