Segmentation fault in Mongrel when run with --debug

Server Mongrel
#1

I am having a problem with mongrel just stopping inside a
periodically_call_remote function so I ran mongrel with debug enabled
Mongrel seg faults after a few minutes

The mongrel filelog shows 5 files open when this happens. Same log files
that are open all the time.

Wed Dec 19 15:10:21 -0700 2007 FILES OPEN BEFORE REQUEST
/calibration/ackAdjustDistance

log/mongrel_debug/objects.log: 2
/home/scott/NetBeansProjects/webmonitor/config/…/log/development.log: 1
log/mongrel_debug/rails.log: 1
log/mongrel_debug/files.log: 1
log/mongrel_debug/threads.log: 1
log/mongrel_debug/access.log: 1

There are 2 threads running
Wed Dec 19 15:10:21 -0700 2007 REQUEST /calibration/ackAdjustDistance
0.0.0.0:3000 – THREADS: 2 -----
KEYS:
– #Thread:0xb6e10738: [:started_on]
– #Thread:0xb75194bc: [:started_on, :inspect_key]

Mongrel stops with this output to stdout.

127.0.0.1 - [Wed, 19 Dec 2007 22:10:17 GMT] “POST
/calibration/ackAdjustDistance HTTP/1.1”
127.0.0.1 - [Wed, 19 Dec 2007 22:10:19 GMT] “POST
/calibration/ackAdjustDistance HTTP/1.1”
127.0.0.1 - [Wed, 19 Dec 2007 22:10:21 GMT] “POST
/calibration/ackAdjustDistance HTTP/1.1”
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/…/lib/mongrel/debug.rb:128:
[BUG] Segmentation fault
ruby 1.8.6 (2007-09-24) [i686-linux]

the last part of the development log is

Processing CalibrationController#ackAdjustDistance (for 127.0.0.1 at
2007-12-19 15:10:23) [POST]
Session ID: 5f0b23a7ae94ac5648268113cd3b4560
Parameters: {“action”=>“ackAdjustDistance”,
“controller”=>“calibration”}
e[4;36;1mCalibration Load (0.000802)e[0m e[0;1mSELECT * FROM
configurations WHERE (Name = ‘CalibrationMessage’) LIMIT 1e[0m
DEPRECATION WARNING: @session is deprecated! Call session.respond_to?
instead of @session.respond_to?. Args: [:length] See
http://www.rubyonrails.org/deprecation for details. (called from process
at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/…/lib/mongrel/debug.rb:128)
Completed in 0.03658 (27 reqs/sec) | Rendering: 0.00144 (3%) | 200 OK
[http://localhost/calibration/ackAdjustDistance]
DEPRECATION WARNING: @session is deprecated! Call session.class instead
of @session.class. Args: [] See http://www.rubyonrails.org/deprecation
for details. (called from process at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/…/lib/mongrel/debug.rb:136)
DEPRECATION WARNING: @response is deprecated! Call response.respond_to?
instead of @response.respond_to?. Args: [:length] See
http://www.rubyonrails.org/deprecation for details. (called from process
at
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.2/bin/…/lib/mongrel/debug.rb:128)

both outputs refer to debug.rb line 128 which is

stolen from Robert K.

class Objects < GemPlugin::Plugin “/handlers”
include Mongrel::HttpHandlerPlugin

121 def process(request,response)
122 begin
123 stats = Hash.new(0)
124 lengths = {}
125 begin
126 ObjectSpace.each_object do |o|
127 begin
128 if o.respond_to? :length
129 len = o.length
130 lengths[o.class] ||= Mongrel::Stats.new(o.class)
131 lengths[o.class].sample(len)
132 end
133 rescue Object
134 end

anybody else seen this?

thanks,

Scott

#2

On Wed, 19 Dec 2007 23:51:55 +0100
Scott D. [email protected] wrote:

I am having a problem with mongrel just stopping inside a
periodically_call_remote function so I ran mongrel with debug enabled
Mongrel seg faults after a few minutes

The mongrel filelog shows 5 files open when this happens. Same log files
that are open all the time.

No idea what would cause it, since that’s a simple respond to, but
usually when you get segfaults in weird places it’s caused by some
other completely unknown library that doesn’t do ram properly.

What you should do is run your mongrel under gdb, when it blows up, gdb
will stop and you can then use the backtrace command to see what was
going on.

Apart from that, not sure what else.

–
Zed A. Shaw

#3

I replaced mongrel with webrick and the problem went away.

The offending page has been running for 45 minutes now, using mongrel I
always died between 5 and 10 minutes.

Guess I’ll use lighthttp or something…

Scott

#4

On Thu, 20 Dec 2007 18:42:47 +0100
Scott D. [email protected] wrote:

I replaced mongrel with webrick and the problem went away.

The offending page has been running for 45 minutes now, using mongrel I
always died between 5 and 10 minutes.

Guess I’ll use lighthttp or something…

Well, webrick might be hiding a problem that’s caused by speed like a
race condition or similar. When your app starts to get more traffic
you’ll probably hit it again.

Take a look at the extensions you’re using. I’m betting one of them
has a threading problem or isn’t written correctly.

And FYI, nginx, apache, and lighttpd will all work with webrick the
same way they work with Mongrel.

–
Zed A. Shaw

#5

Zed,

I think your right because webrick finally stopped after about 90
minutes.

It was locked up and when I killed it it complained of a deadlock.

I’m now thinking it may bein a call to the sqlite3_ruby gem.

I tried to run mongrel_rails under gdb but gdb complains its not the
correct file type.

Scott

Zed A. Shaw wrote:

On Thu, 20 Dec 2007 18:42:47 +0100
Scott D. [email protected] wrote:

I replaced mongrel with webrick and the problem went away.

The offending page has been running for 45 minutes now, using mongrel I
always died between 5 and 10 minutes.

Guess I’ll use lighthttp or something…

Well, webrick might be hiding a problem that’s caused by speed like a
race condition or similar. When your app starts to get more traffic
you’ll probably hit it again.

Take a look at the extensions you’re using. I’m betting one of them
has a threading problem or isn’t written correctly.

And FYI, nginx, apache, and lighttpd will all work with webrick the
same way they work with Mongrel.

–
Zed A. Shaw

#6

I thinking rails just can’t handle running periodically_call_remote with
an interval shorter than 1 second. eventually two calls are synced up
and it deadlocks. the faster I run periodically… the sooner the
deadlock.

If I run lighttpd as a load balancer with cluster would that alleviate
the problem? If that is the problem?

Scott

#7

On Dec 21, 2007, at 9:38 AM, Scott D. wrote:

I thinking rails just can’t handle running periodically_call_remote
with
an interval shorter than 1 second. eventually two calls are synced up
and it deadlocks. the faster I run periodically… the sooner the
deadlock.

Rails’s code is probably OK in this case – periodically_call_remote
isn’t inherently dangerous – it just writes some javascript that
periodically calls (via Prototype’s Ajax.Request) a controller /
action. If you use Firebug and make the same request in the console
(without using p_c_r), you’ll see the exact same problem after
several (hundred?) requests.

The problem is probably (almost certainly) in either code you’ve
written or in a library you’re using. gdb will help you out here.

Really, though, this is offtopic for Mongrel’s list – either the
Rails list or the Ruby list is a better match for these problems. The
Rails IRC channel might also be useful. Or contact me off-list, and
I’ll help if I can…

If I run lighttpd as a load balancer with cluster would that alleviate
the problem? If that is the problem?

Nope; it’ll just put off the inevitable. You need to find and fix the
deadlocking code. This can be challenging; race conditions are one of
the least fun parts of programming.

Cheers,
-Nate

#8

On Dec 20, 2007 3:51 PM, Scott D. [email protected] wrote:

Zed,

I think your right because webrick finally stopped after about 90
minutes.

It was locked up and when I killed it it complained of a deadlock.

I’m now thinking it may bein a call to the sqlite3_ruby gem.

Oh, sqlite3 gem!

I got a few segfaults with it, like unrolling transactions and other
weird stuff.

Still waiting for some tickets get fixed, but without luck.

I tried to run mongrel_rails under gdb but gdb complains its not the
correct file type.

you need to run gdb on “ruby”, not mongrel_rails (since mongrel_rails
is a ruby script).

gdb /path/to/ruby/ruby.exe /path/to/mongrel_rails param1 param2 param3

–
Luis L.
Multimedia systems

A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

#9

Nathan V. wrote:

Rails’s code is probably OK in this case – periodically_call_remote
isn’t inherently dangerous – it just writes some javascript that
periodically calls (via Prototype’s Ajax.Request) a controller /
action. If you use Firebug and make the same request in the console
(without using p_c_r), you’ll see the exact same problem after
several (hundred?) requests.

The problem is probably (almost certainly) in either code you’ve
written or in a library you’re using. gdb will help you out here.

I am making a find(… call which uses the sqlite3 gem

Really, though, this is offtopic for Mongrel’s list – either the
Rails list or the Ruby list is a better match for these problems. The
Rails IRC channel might also be useful. Or contact me off-list, and
I’ll help if I can…

thats true if its not a mongrel problem. I am pursuing a solution on
other forums.

thanks,

Scott

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs