Forum: Rails deployment mongrel process spinning with CPU & MEM maxed out

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-08 03:26
(Received via mailing list)
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+   PPID
COMMAND
12565 ettinger  39   0 1110m 1.1g 2680 R 99.9 54.4  21:53.77     1
/usr/bin/ruby1.8 /usr/bin/mongrel_rails start -C
/home/ettinger/directoryofdirect



CPU: 99.9%
Mem: 54.4%

What can I do to bring this back inline with an acceptable value?


--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
Efe398b3b5436b0a8d4430090e8f86ea?d=identicon&s=25 seth b. (subimage)
on 2008-10-08 03:37
(Received via mailing list)
Run monit on it and restart the fuckers when they aren't playing nicely.

Here's a sample of my old monit config that did just that:
http://pastie.org/287363

...OR switch to mod_rails. I just made the switch and I'm loving not
having to slap my mongrels around or deal with monit madness anymore.

--------------------
seth - subimage llc
-----
http://sublog.subimage.com
-----
Cashboard - Estimates, invoices, and time tracking software - for free!
http://www.getcashboard.com
-----
Substruct - Open source RoR e-commerce software.
http://code.google.com/p/substruct/
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-09 07:32
(Received via mailing list)
Monit doesn't really solve the problem though. I've also heard
mod_rails isn't ready for primetime yet.

I'd like to know what is causing it to eat up memory and CPU usage so
badly...as it ran fine for the last few weeks up until the last few
releases.

I've read that it could be anything from a bad regex to low disk
space, gdb debugging rails didn't give me much that a simpleton like
myself could understand:

http://weblog.jamisbuck.org/2006/9/22/inspecting-a...
http://eigenclass.org/hiki.rb?ruby+live+process+in...



On Tue, Oct 7, 2008 at 6:36 PM, seth b <subimage@gmail.com> wrote:
> seth - subimage llc
>
>> CPU: 99.9%
>> >
>>
>
> >
>



--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
Ff803e4a435f1438fc928f41500b1ffa?d=identicon&s=25 Alex Malinovich (Guest)
on 2008-10-10 20:16
(Received via mailing list)
Anthony,

What version of Ruby are you running? (ruby -v) I've seen this problem
quite a few times with 1.8.6 p114.
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-10 22:46
(Received via mailing list)
$ ruby -v
ruby 1.8.5 (2006-08-25) [i386-linux]


On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich
<alex.malinovich@planetargon.com> wrote:
>> I'd like to know what is causing it to eat up memory and CPU usage so
>>
>> > --------------------
>> > On Tue, Oct 7, 2008 at 6:26 PM, Anthony Ettinger <ettin...@gmail.com> wrote:
>> >> What can I do to bring this back inline with an acceptable value?
>
--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
Ff803e4a435f1438fc928f41500b1ffa?d=identicon&s=25 Alex Malinovich (Guest)
on 2008-10-10 23:51
(Received via mailing list)
1.8.5? Wow. That's pretty old. I wouldn't think that that would be the
problem, but it certainly couldn't hurt to upgrade to something a bit
more recent. Unless you're using rails 2.1 exclusively I'd suggest
1.8.6 p287 and see if that resolves your problem.
Efe398b3b5436b0a8d4430090e8f86ea?d=identicon&s=25 seth b. (subimage)
on 2008-10-11 01:18
(Received via mailing list)
> Monit doesn't really solve the problem though. I've also heard
> mod_rails isn't ready for primetime yet.

Yeah, Shopify and Cashboard are really not "primetime" Rails apps huh?
LOL.

> I'd like to know what is causing it to eat up memory and CPU usage so
> badly...as it ran fine for the last few weeks up until the last few
> releases.

Could be a variety of things. Rmagick is a known culprit, but I'm sure
google will lend you many answers.

--------------------
seth - subimage llc
-----
http://sublog.subimage.com
-----
Cashboard - Estimates, invoices, and time tracking software - for free!
http://www.getcashboard.com
-----
Substruct - Open source RoR e-commerce software.
http://code.google.com/p/substruct/
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-11 01:34
(Received via mailing list)
Thanks Alex, I'll give it a shot.

On Fri, Oct 10, 2008 at 2:50 PM, Alex Malinovich
<alex.malinovich@planetargon.com> wrote:
>> On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich
>> > On Oct 8, 10:31 pm, "Anthony Ettinger" <ettin...@gmail.com> wrote:
>>
>> >> > having to slap my mongrels around or deal with monit madness anymore.
>> >> >http://code.google.com/p/substruct/
>> >> >> Mem: 54.4%
>> >> 408-656-2473http://anthony.ettinger.name
>>
>> --
>> Anthony Ettinger
>> 408-656-2473http://anthony.ettinger.name
> >
>



--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-13 05:51
(Received via mailing list)
Hi Alex,

My ISP only has 1.8.5 and Rails 2.1.
I think there is a logic error somewhere in my code, prior to about a
week ago, the site was functioning fine.

If someone has any ideas on how to debug the 99% CPU usage, I'd be
interested in hearing.

Anthony

On Fri, Oct 10, 2008 at 2:50 PM, Alex Malinovich
<alex.malinovich@planetargon.com> wrote:
>> On Fri, Oct 10, 2008 at 11:15 AM, Alex Malinovich
>> > On Oct 8, 10:31 pm, "Anthony Ettinger" <ettin...@gmail.com> wrote:
>>
>> >> > having to slap my mongrels around or deal with monit madness anymore.
>> >> >http://code.google.com/p/substruct/
>> >> >> Mem: 54.4%
>> >> 408-656-2473http://anthony.ettinger.name
>>
>> --
>> Anthony Ettinger
>> 408-656-2473http://anthony.ettinger.name
> >
>



--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
A82ba1167f4d4a8d1de63820e576a87f?d=identicon&s=25 Robby Russell (Guest)
on 2008-10-13 16:13
(Received via mailing list)
On Sun, Oct 12, 2008 at 8:50 PM, Anthony Ettinger <ettinger@gmail.com>
wrote:
>
> Hi Alex,
>
> My ISP only has 1.8.5 and Rails 2.1.
> I think there is a logic error somewhere in my code, prior to about a
> week ago, the site was functioning fine.
>
> If someone has any ideas on how to debug the 99% CPU usage, I'd be
> interested in hearing.

Anthony,

Are you seeing similar results when running in development? Has much
changed in the application (code or visitor-wise) in the last week or
so that could be triggering this behavior?

You might consider looking at the performance tools from New Relic or
Five Runs, which can assist you in determining where some of the
bottlenecks in your code might be lingering.

Good luck!

Robby


--
Robby Russell
Chief Evangelist, Partner

PLANET ARGON, LLC
design // development // hosting

http://www.planetargon.com/
http://www.robbyonrails.com/
aim: planetargon

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-10-13 19:46
(Received via mailing list)
On Mon, Oct 13, 2008 at 7:13 AM, Robby Russell <robby@planetargon.com>
wrote:
>
> Are you seeing similar results when running in development? Has much
> changed in the application (code or visitor-wise) in the last week or
> so that could be triggering this behavior?
>
> You might consider looking at the performance tools from New Relic or
> Five Runs, which can assist you in determining where some of the
> bottlenecks in your code might be lingering.
>
> Good luck!
>

Its very odd, I have test.example.com (test) and example.com (prod) on
the same server, and the code has been deployed to both. However every
time I start the prod mongrel_rails server, each request hangs and
eats up CPU at 95% and takes 10 seconds for a request to complete.

test.example.com does not have this problem, it serves the pages in
under a second.

I originally thought it may be the DB, so I made prod (example.com)
use the test environment, and the issue continues.

I don't know what else it could be, as they are seemingly identical
when both are running as "test" environment.
583d7d066b30d8d38df7d7c2e9e78045?d=identicon&s=25 Anthony Ettinger (Guest)
on 2008-11-02 20:30
(Received via mailing list)
On Tue, Oct 7, 2008 at 5:26 PM, Anthony Ettinger <ettinger@gmail.com>
wrote:
>
> What can I do to bring this back inline with an acceptable value?
>
>

After switching hosting providers (as this was the default response I
got from many), the prod app still had the same issue and stopped
serving. Thanks to a couple of people on #rubyonrails channel, we
narrowed it down to a bad circular reference in the :include => [...]
method I was using for the sortable_column_headers plugin.

Here's the change I made, and the app loaded fine after that:

                @directories = Directory.paginate(  :page =>
params[:page],
-
                            :include => [{ :software => :directories
}, { :status => :directories }],
+
                            :include => [:software, :status],


You may be able to ascertain this by tailing the production.log and
then cutting and pasting the SQL query into the mysql monitor
manually, with the words "EXPLAIN <sql query here>" infront of it.
This could help explain where indexes could be useful, although that
helped it was not my problem.

My next task is to learn how to debug and profile my rails app and how
to copy production data in a test database so these issues surface as
soon as the code is written. For my case, the test database only had a
few rows, so the exponential growth in the query time was able to pass
through with only 5 data sets...on production it was 89 and tail spun
into a memory/cpu hog.


--
Anthony Ettinger
408-656-2473
http://anthony.ettinger.name
This topic is locked and can not be replied to.