RubyGems runtime memory use in Rails apps

This an open plea for someone with knowledge of RubyGems to explain why
RubyGems stays in memory (over 10MB) after a Rails application is
loaded. I
am unable to figure out why this would be necessary. I haven’t had any
response from Jim or Chad from RubyGems about it so I thought I would
throw
it open to the list to see if anyone could educate me or share thier
experiences with this.

There is a more detailed write up on my blog (
http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use).

Hello Steve,
Well, since you asked for help, can I suggest that you provide
pertinent details ? OS used, ruby version, and also the example code
that you used to profile would be nice :wink: I must say, that rubygems
hooks into the main process from what I can (hence the require) so I
would -expect- it to stay resident in memory. Which reminds me, what
memory are you profiling here ? VSZ ? RSZ ? Then again, in the day and
age of 1GB in my desktop machine, its probably highly easy to dismiss a
RSZ of 10MB :wink: Heck, almost all of the servers at work have >4GB. Thats
not the point, I know, but, there is no point in chasing rabbits down
holes if you don’t need to (also called the ‘premature optimisation’
problem).

Details details, the devil is in the details*
Regards
Stef

(* this should not be taken as an endorsement of satan nor that I have
-the- answer to your problems :wink:

Stef!!!

Thank you for answering at all! I am a leper on the list it feels
like
sometimes. I am profiling Typo, an app in need of some very “Mature
Optimization”. So no worries there. Locally I use OS X/PPC, Ruby
1.8.4.
However I am doing the real profiling against my www.stevelongdo.com
site on
TextDrive. It is also Ruby 1.8.4 and I think a Linux flavor not Solaris
10
yet anyway.

Typo and its external dependant version of Rails were trunk a couple
days
ago. TextDrive limits are based on RSZ going over 48MB. VSZ is almost
always lower than that when Ruby decides to allocate some more space and
send RSZ over the limit. I am not sure whether they are using Stefan
Kaes
suggested patched GC or if it is plain vanilla Ruby GC. I don’t for
sure
how to find that out on TextDrive.

At any rate there is no good reason for my personal Typo weblog to cross
into the 48MB zone. I’ll bet all of my content together is 512kB.

Outside of all that though I am trying to learn more about how RubyGems
actually does its business and whether it could do it in less than ~11MB
of
RAM. I am barreling into the code now since response to my “plea” for
help
went unanswered. I would expect that Shared Host providers would want
to
have somebody look at this because it should be an issue for all Rails
apps
not just Typo. Imagine if we figure out how to shave 10MB off of each
Rails
app out there? Shared Hosts become more viable, people send me presents
and
bow in my presence…

I’ve gotten a bit further in my research tonight take a look at my
latest
post:
http://www.stevelongdo.com/articles/2006/08/09/rubygems-single-multi-tons-and-memorysize

All hail Stef our dark underlord…Oh not affiliated with Satan, I see.
:stuck_out_tongue:

On 8/8/06, Stef T [email protected] wrote:

RSZ of 10MB :wink: Heck, almost all of the servers at work have >4GB. Thats
Steve L. wrote:

<
http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use

).

Thanks,
-Steve
http://www.stevelongdo.com

Hello Steve,
No no, your definitely not a pariah, and your emails are definitely
making it onto the list :wink: A crazy thought appears, perhaps woefully and
stupidly in my head, but, its 1AM EST here, so, forgive me for this,
but, why not simply use the design pattern of Singleton in
custom_require.rb if you believe this is the problem ?

There now follows a rather clunky 'human diff' .. please bear with 

me :slight_smile:

at line 24 (in the require method)
    @gempath_searcher ||= Gem::GemPathSearcher.new
should become
    @gempath_searcher = Gem::GemPathSearcher.instance

And then you want to change the module Gem part below to look

something like;

module Gem
  require 'singleton'
  #
  # GemPathSearcher has the capability to find loadable files inside
  # gems.  It generates data up front to speed up searches later.
  #
  class GemPathSearcher
        include Singleton

If you then put a 'puts @gempath_searcher.object_id' after the line

24 change, and run (say) your ruby ./script/server, you will see a huge
spew of lines like ;

stef@stef-desktop:~/work/myTest$ ruby ./script/server
-606820690
-606820690
-606820690

If you 'undo' those changes, and go back to the good old-fashioned

code, you will see something like;

stef@stef-desktop:~/work/myTest$ ruby ./script/server
-606681356
-606869086
-606681356
-606818976

Different object_id's means different object's .. so .. by using the

Singleton design pattern, your gemserver will now only be created
‘once’. Now, of course, wether or not this breaks things horrible OR if
this does indeed fix your memory leakage, you will have to let us all
know :slight_smile:

Either way, have a nice night, I am shuffling off to bed, for some

well earned R&R. Let me know how it all works out though please,
irregardless :wink:
Regards
Stef
(ps. I could never worship Satan, but only Cthulhu, mean to say, why
worship a lesser evil ;D
(pps. that -is- a joke for the humour impaired :wink:

I have posted an update on this at my blog:
http://www.stevelongdo.com/articles/2006/08/12/rubygems-memory-reduction

Basically RubyGems needs to be patched. Stef’s solution below works
perfectly, once you figure out how to set it up on a shared host! I
will be
following up on exactly how to do that tomorrow.

Looks like maybe Nicholas S. and the Rails core are going to be
doing
something different soon anyway.