Memory Consumption issues

Rails
1

Greetings,

I’ve recently deployed an app that is having some serious memory
consumption issue. Dropping mongrels quite frequently when the
threshold is met.

Initially, we chalked this problem up to RMagick (I know, I know, but we
needed watermarking). We quickly optimized things so all images were
only processed once then neatly stored in their own folder. When an
image is requested, we confirm that it doesn’t exist before processing
it. Been tested, and this is working as it should.

According to some memory profiling we’ve done, there are a couple of key
pages that are creating large amounts of strings that are not being
reclaimed.

Has anyone ever experienced issues similar to this?

Cheers,
DJ

2

What is memory limit per process?

3

Dave -

On 1-Apr-08, at 2:49 PM, Dave J. wrote:

only processed once then neatly stored in their own folder. When an
Cheers,
DJ

I too am presently using monit to manage this until I can track down
the leak.

Our mongrel’s high water mark is 280 mb, and each of the 11 restart a
few times per day.

Rmagick is in place, but isolated, and likely not the source.

There’s an article somewhere (lost in my booksmarks) that helps track
down leaks. As I suspect my leaks are very specific, I have not
appealed to the general group, but perhaps this thread will turn out
something interesting.

Jodi

4

Thanx Dave -

On 1-Apr-08, at 5:10 PM, Dave J. wrote:

Memory leak profiling with Rails :: scottstuff.net

We’ve also been using bleak_house:
http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
yet still can’t nail down exactly whats going on here.

Jodi, what version of Rails are you using for you project?

Thanks,
Dave

We’re on 1.2.3, mysql 5, ubunutu 6.06 LTS, ruby 1.8.6 (i686).

I’m deploying a new server (throw money at the problem), but will be
getting to the leaks next week.

We’ve had the leaks since day 1, and although we have 1.8million plus
pages, controller actions are limited - so I should be able to
localize the area quickly. I’ll update this thread with anything
learned.

thank you again Dave for the references.

Jodi

5

On Apr 1, 2:49 pm, Dave J. [email protected] wrote:

Greetings,

I’ve recently deployed an app that is having some serious memory
consumption issue. Dropping mongrels quite frequently when the
threshold is met.

Initially, we chalked this problem up to RMagick (I know, I know, but we
needed watermarking).

If RMagick is not the source of leaks currently then this won’t help,
but I just do the following:

convert -resize 480x360 #{temp_file_path} #{file_path}

My image uploading is low volume, so this works fine for me. I think
it would work well even for much higher volumes, but I haven’t
benchmarked the difference between a command line invocation and
RMagick. It’s not likely to leak though.

If you track down the problem please post it.

6

Hey Jodi,

I believe our mongrels memory threshold is in the same ballpark. Here
are a couple of articles we’ve been looking at, following along with
these processes and using the memory analysis tool in the second has
revealed that the strings not getting reclaimed are blank…

http://codehappy.wordpress.com/2007/01/17/tracking-a-memory-leak-in-rails/

We’ve also been using bleak_house:
http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html
yet still can’t nail down exactly whats going on here.

Jodi, what version of Rails are you using for you project?

Thanks,
Dave

7

Greetings mem-leakers

On 1-Apr-08, at 8:13 PM, Brian A. wrote:

needed watermarking).

If you track down the problem please post it.

I don’t have any answers yet, a few clues, but have some tools to
suggest:

re: clues. this article looks promising:

http://blog.pluron.com/2008/01/ruby-on-rails-i.html

As for tools, I’m trying out Evan W.'s bleak-house:

http://blog.evanweaver.com/files/doc/fauna/bleak_house/files/README.html

as it patches ruby, you’ll want to use this on your staging server
(Evan says he uses the patched ruby in production)

this is the output from Bleak:

./homestars/shared/log $ bleak /tmp/bleak.3242.0.dump
632837 total objects
Final heap size 632837 filled, 732169 free
Displaying top 20 most common line/class pairs
510101 null:null:node
29450 null:null:String
5511 /usr/local/lib/ruby/gems/1.8/gems/memcached-0.8.1/lib/
memcached/memcached.rb:295:String
4692 (eval):1:node
3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:836:String
3442 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:
836:ActionController::Routing::DividerSegment
2084 null:null:Regexp
1902 /usr/local/lib/ruby/gems/1.8/specifications/
tzinfo-0.3.7.gemspec:12:String
1686 generated_code_(/var/www/apps/homestars/releases/
20080407140541/vendor/rails/actionpack/lib/action_controller/
routing.rb:388):1:node
1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:834:String
1491 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:
834:ActionController::Routing::StaticSegment
1287 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
activerecord/lib/…/…/activesupport/lib/active_support/deprecation.rb:
92:node
1106 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:
827:ActionController::Routing::DynamicSegment
1056 /usr/local/lib/ruby/gems/1.8/specifications/
rmagick-2.2.2.gemspec:13:String
1051 /usr/local/lib/ruby/1.8/yaml.rb:133:String
976 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
activerecord/lib/…/…/activesupport/lib/active_support/core_ext/
module/aliasing.rb:28:node
938 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
activerecord/lib/…/…/activesupport/lib/active_support/core_ext/
module/aliasing.rb:29:node
891 null:null:Class
845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:
945:ActionController::Routing::Route
845 /var/www/apps/homestars/releases/20080407140541/vendor/rails/
actionpack/lib/action_controller/routing.rb:865:Hash

(the above is from a single run of hitting a single url for 15
minutes. working out a test approach presently)

The above is after a 30 minute run - showing 5511 objects lost in
memached (funny enough also written by Evan - I’ve asked him about the
leak). The __ leaks are supposedly internal to ruby - boy, that’s a
load of lost objects -

other than tzinfo (which I’ve heard is a culprit), it seems to me that
routing is generating a load. Anybody have any info on routing and
leaks?

I’m at a bit of a loss when digging info ruby leaks/garbage
collection. If anybody has a good ref I’d appreciate it.

rails 1.2.6
ruby 1.8.6 (2008-03-03 patchlevel 901) [i686-linux]

thanx.
Jodi

8

Hey all,

We’ll we tracked down our memory issues. We had some complex join
queries going on and it appears that the AR objects created from said
queries were not getting reclaimed.

So if you are still having issues, and have some complex find_by_sql or
find statements using a lot/nested includes. Try breaking those up into
two queries… Vast improvement here after doing that.

Cheers,
Dave

9

Thanx Dave -

On 7-Apr-08, at 4:04 PM, Dave J. wrote:

into
two queries… Vast improvement here after doing that.

Cheers,
Dave

I’m interested in how your determined they weren’t be reclaimed? ie.
how did you zero in on the opportunity?

Jodi