Mongrel process spinning with CPU & MEM maxed out


#1

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID
COMMAND
12565 ettinger 39 0 1110m 1.1g 2680 R 99.9 54.4 21:53.77 1
/usr/bin/ruby1.8 /usr/bin/mongrel_rails start -C
/home/ettinger/directoryofdirect

CPU: 99.9%
Mem: 54.4%

What can I do to bring this back inline with an acceptable value?


Anthony E.
408-656-2473
http://anthony.ettinger.name


#2

Run monit on it and restart the fuckers when they aren’t playing nicely.

Here’s a sample of my old monit config that did just that:
http://pastie.org/287363

…OR switch to mod_rails. I just made the switch and I’m loving not
having to slap my mongrels around or deal with monit madness anymore.


seth - subimage llc

http://sublog.subimage.com

Cashboard - Estimates, invoices, and time tracking software - for free!
http://www.getcashboard.com

Substruct - Open source RoR e-commerce software.
http://code.google.com/p/substruct/


#3

Monit doesn’t really solve the problem though. I’ve also heard
mod_rails isn’t ready for primetime yet.

I’d like to know what is causing it to eat up memory and CPU usage so
badly…as it ran fine for the last few weeks up until the last few
releases.

I’ve read that it could be anything from a bad regex to low disk
space, gdb debugging rails didn’t give me much that a simpleton like
myself could understand:

http://weblog.jamisbuck.org/2006/9/22/inspecting-a-live-ruby-process
http://eigenclass.org/hiki.rb?ruby+live+process+introspection

On Tue, Oct 7, 2008 at 6:36 PM, seth b removed_email_address@domain.invalid wrote:

seth - subimage llc

CPU: 99.9%


Anthony E.
408-656-2473
http://anthony.ettinger.name


#4

Anthony,

What version of Ruby are you running? (ruby -v) I’ve seen this problem
quite a few times with 1.8.6 p114.


#5

1.8.5? Wow. That’s pretty old. I wouldn’t think that that would be the
problem, but it certainly couldn’t hurt to upgrade to something a bit
more recent. Unless you’re using rails 2.1 exclusively I’d suggest
1.8.6 p287 and see if that resolves your problem.


#6

$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-linux]

On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich
removed_email_address@domain.invalid wrote:

I’d like to know what is causing it to eat up memory and CPU usage so


On Tue, Oct 7, 2008 at 6:26 PM, Anthony E. removed_email_address@domain.invalid wrote:

What can I do to bring this back inline with an acceptable value?


Anthony E.
408-656-2473
http://anthony.ettinger.name


#7

Monit doesn’t really solve the problem though. I’ve also heard
mod_rails isn’t ready for primetime yet.

Yeah, Shopify and Cashboard are really not “primetime” Rails apps huh?
LOL.

I’d like to know what is causing it to eat up memory and CPU usage so
badly…as it ran fine for the last few weeks up until the last few
releases.

Could be a variety of things. Rmagick is a known culprit, but I’m sure
google will lend you many answers.


seth - subimage llc

http://sublog.subimage.com

Cashboard - Estimates, invoices, and time tracking software - for free!
http://www.getcashboard.com

Substruct - Open source RoR e-commerce software.
http://code.google.com/p/substruct/


#8

Thanks Alex, I’ll give it a shot.

On Fri, Oct 10, 2008 at 2:50 PM, Alex Malinovich
removed_email_address@domain.invalid wrote:

On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich

On Oct 8, 10:31 pm, “Anthony E.” removed_email_address@domain.invalid wrote:

having to slap my mongrels around or deal with monit madness anymore.
http://code.google.com/p/substruct/

Mem: 54.4%
408-656-2473http://anthony.ettinger.name


Anthony E.
408-656-2473http://anthony.ettinger.name


Anthony E.
408-656-2473
http://anthony.ettinger.name


#9

Hi Alex,

My ISP only has 1.8.5 and Rails 2.1.
I think there is a logic error somewhere in my code, prior to about a
week ago, the site was functioning fine.

If someone has any ideas on how to debug the 99% CPU usage, I’d be
interested in hearing.

Anthony

On Fri, Oct 10, 2008 at 2:50 PM, Alex Malinovich
removed_email_address@domain.invalid wrote:

On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich

On Oct 8, 10:31 pm, “Anthony E.” removed_email_address@domain.invalid wrote:

having to slap my mongrels around or deal with monit madness anymore.
http://code.google.com/p/substruct/

Mem: 54.4%
408-656-2473http://anthony.ettinger.name


Anthony E.
408-656-2473http://anthony.ettinger.name


Anthony E.
408-656-2473
http://anthony.ettinger.name


#10

On Sun, Oct 12, 2008 at 8:50 PM, Anthony E. removed_email_address@domain.invalid
wrote:

Hi Alex,

My ISP only has 1.8.5 and Rails 2.1.
I think there is a logic error somewhere in my code, prior to about a
week ago, the site was functioning fine.

If someone has any ideas on how to debug the 99% CPU usage, I’d be
interested in hearing.

Anthony,

Are you seeing similar results when running in development? Has much
changed in the application (code or visitor-wise) in the last week or
so that could be triggering this behavior?

You might consider looking at the performance tools from New Relic or
Five Runs, which can assist you in determining where some of the
bottlenecks in your code might be lingering.

Good luck!

Robby


Robby R.
Chief Evangelist, Partner

PLANET ARGON, LLC
design // development // hosting


http://www.robbyonrails.com/
aim: planetargon

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]


#11

On Tue, Oct 7, 2008 at 5:26 PM, Anthony E. removed_email_address@domain.invalid
wrote:

What can I do to bring this back inline with an acceptable value?

After switching hosting providers (as this was the default response I
got from many), the prod app still had the same issue and stopped
serving. Thanks to a couple of people on #rubyonrails channel, we
narrowed it down to a bad circular reference in the :include => […]
method I was using for the sortable_column_headers plugin.

Here’s the change I made, and the app loaded fine after that:

            @directories = Directory.paginate(  :page => 

params[:page],

                        :include => [{ :software => :directories

}, { :status => :directories }],
+
:include => [:software, :status],

You may be able to ascertain this by tailing the production.log and
then cutting and pasting the SQL query into the mysql monitor
manually, with the words "EXPLAIN " infront of it.
This could help explain where indexes could be useful, although that
helped it was not my problem.

My next task is to learn how to debug and profile my rails app and how
to copy production data in a test database so these issues surface as
soon as the code is written. For my case, the test database only had a
few rows, so the exponential growth in the query time was able to pass
through with only 5 data sets…on production it was 89 and tail spun
into a memory/cpu hog.


Anthony E.
408-656-2473
http://anthony.ettinger.name


#12

On Mon, Oct 13, 2008 at 7:13 AM, Robby R. removed_email_address@domain.invalid
wrote:

Are you seeing similar results when running in development? Has much
changed in the application (code or visitor-wise) in the last week or
so that could be triggering this behavior?

You might consider looking at the performance tools from New Relic or
Five Runs, which can assist you in determining where some of the
bottlenecks in your code might be lingering.

Good luck!

Its very odd, I have test.example.com (test) and example.com (prod) on
the same server, and the code has been deployed to both. However every
time I start the prod mongrel_rails server, each request hangs and
eats up CPU at 95% and takes 10 seconds for a request to complete.

test.example.com does not have this problem, it serves the pages in
under a second.

I originally thought it may be the DB, so I made prod (example.com)
use the test environment, and the issue continues.

I don’t know what else it could be, as they are seemingly identical
when both are running as “test” environment.