Typo's memory footprint

I am hosting my blog on Textdrive and they have a 48 meg limit on
process usage. After that it just gets whacked. On a fresh restart
with Typo 4.0.0 its SIZE is 36016K and its RES is 33452K. A couple
hits on the main page tack on around 4 more megs and about 90% of the
time just editing an article will push it over the top. Pre 4.0.0 I
was running on trunk and did not experience this problem. Maybe I was
hovering at the limit and didn’t realize it. My install of Typo is
sitting behind lighttpd/fastcgi.

Does anybody on the list know if turning off or abstaining from the
use of certain features will help relieve how much memory is consumed?
In the spirit of Piers’ email to look at more mid/long term goals, I
think that memory consumption is definitely something that could be
looked at, at least to offer some way of working with a lighter memory
footprint.

I upgraded my install to Typo 4 when it came out on TextDrive and
haven’t noticed my site go down since. I’m a Linux newbie so how does
one tell how much memory is being used?

I’m also using FastCGI behind lighttpd.

“Eric Sendelbach” [email protected] writes:

use of certain features will help relieve how much memory is consumed?
In the spirit of Piers’ email to look at more mid/long term goals, I
think that memory consumption is definitely something that could be
looked at, at least to offer some way of working with a lighter memory
footprint.

I could be wrong (and I probably am, but we don’t have any
particualrly great tools for investigating this), but I think Scott’s
changes to the filter chain and caching have got rid of the big memory
leak, but the new, highly inclusive optimized SQL for getting content
which tends to pull in the world right at the start (reducing the hits
on the database, no bad thing) means that the process gets very big
rather quickly, but then doesn’t get a great deal bigger after that.

I don’t think it’s going to slim down very much I’m afraid.

On 01/08/06, Steve L. [email protected] wrote:

I have the same experience with TextDrive as Eric. I notice that memory use
jumps 10 MB just by entering the admin console.

I’ve found that removing references to Sparklines helps the memory use stay
under the threshold. I removed the sparklines_textfilter.rb and removed the
reference to vendor/sparklines in environment.rb. Also removing any
sidebars that you don’t use from the component directory helps.

Just the fact of a sidebar existing bumping memory usage seems broken
to me - is that a bug or just The Way Things Are?

More generally, how hard would it be to have knobs to disable unused
features?

I have the same experience with TextDrive as Eric. I notice that memory
use
jumps 10 MB just by entering the admin console.

I’ve found that removing references to Sparklines helps the memory use
stay
under the threshold. I removed the sparklines_textfilter.rb and removed
the reference to vendor/sparklines in environment.rb. Also removing
any
sidebars that you don’t use from the component directory helps.

It is a funtion of the sidebar and textfilter controller where it
dynamically loads all files in the directories into memory:

Dir["#{RAILS_ROOT}/components/plugins/sidebars/[_a-z]*.rb"].each do |f|
require_dependency f
end

So it is cool because when new plugins are added they automagically load
up,
but unfortunate when, like Sparklines, they have a large overhead
associated.

As for knobs, you could have something that wrote a more specific
regular
expression for the load instead of just [_a-z]*.rb…

“Steve L.” [email protected] writes:

As for knobs, you could have something that wrote a more specific regular
expression for the load instead of just [_a-z]*.rb…

This is something that could be helped by the mooted move of sidebar
plugins to vendor/plugins. Essentially you’d only need to install
those sidebars you use.

What needs to happen to transistion out of the current component model
into
the vendor/plugin model?

I have heard the new routing code allows plugins to register named
routes
when they are loaded into a parent app. This would allow for sidebars
that
could have more than one “content page”.

Eric Sendelbach wrote:

I am hosting my blog on Textdrive and they have a 48 meg limit on
process usage. After that it just gets whacked. On a fresh restart
with Typo 4.0.0 its SIZE is 36016K and its RES is 33452K.

I’m seeing size = 51592 and vsize = 63856 for each dispatch.cgi process
using lighttpd and fastcgi, so consider yourself lucky :slight_smile:

I actually had to move up to a higher level of hosting in order to keep
running typo, costing me significant cash. To make up for it, I’m now
running tracks on the same hosting, and I hope to run a wiki too as soon
as I can find or build one I consider suitable. (I’m not wild about
instiki.)

mathew

“Steve L.” [email protected] writes:

What needs to happen to transistion out of the current component model into
the vendor/plugin model?

Someone needs to work out if it can be done, document how its done and
move some of the offending plugins out.

I have heard the new routing code allows plugins to register named routes
when they are loaded into a parent app. This would allow for sidebars that
could have more than one “content page”.

I’m not entirely sure they’d be much use to sidebars tbh. Sidebar
controllers can already have more than one action/content page.

FYI, I have having a serious memory leak using typo 3.99.x (I think .4,
can’t remember). It was so bad I couldn’t even ssh into VPS account and
I
had to get the sysadmin to restart my “server”. Anyway, I just upgraded
to
4.00, couldn’t have been any easier. Taking at look at the memory
usage, it
looks like this in “ps aux”:

Right after starting:
root 3395 31.0 25.9 27732 24608 ? S 10:31 0:02
/usr/bin/ruby /usr/bin/mongrel_rails start

After first hit to home page:
root 3395 5.7 26.3 28120 24980 ? S 10:31 0:02
/usr/bin/ruby /usr/bin/mongrel_rails start

After first posted article:
root 3395 16.4 36.9 37900 34984 ? R 10:31 1:00
/usr/bin/ruby /usr/bin/mongrel_rails start
Seems to have leveled off at the last level. I’ll let you know if I
continue to have problems or if the memory leak is indeed solved with
4.00.
I’m using apache2 mod_proxy, mongrel and sqlite on a FC4 VPS host.

A few days later here’s where it’s at:

root 3395 0.3 53.3 53616 50524 ? S Aug02 9:37
/usr/bin/ruby /usr/bin/mongrel_rails start

Seems to be creeping up. Is this a memory leak or just normal? Is
anyone
else seeing this?

Almost a week later here’s what the memory usage looks like:

root 3395 0.1 58.0 76412 54980 ? S Aug02 14:07
/usr/bin/ruby /usr/bin/mongrel_rails start

So no one else is having problems with this? I gonna have to put in a
cron
job to restart this weekly, since it’s obviously leaking memory.

Yes, I am still having problems with this, but nobody seems to worry too
much about it. Unfortunately for me TextDrive would have killed the
process
a LONG time ago when it went over 48 megs. I suppose I could do the same
with a cron job and monitor memory usage, but it would have to restart
every
3 minutes or so and I don’t think I nor the nice folks at TextDrive
would
like that too much. :slight_smile:

Since TextDrive is supposed to be the premier RoR host and Typo the
premier
RoR blogging software, it only seems natural that the two should play
nice
together.

Paul B. wrote:

Almost a week later here’s what the memory usage looks like:

root 3395 0.1 58.0 76412 54980 ? S Aug02 14:07
/usr/bin/ruby /usr/bin/mongrel_rails start

So no one else is having problems with this? I gonna have to put in a
cron job to restart this weekly, since it’s obviously leaking memory.

One of my dispatch.cgis has gone up to 58128 70392, the other has gone
down to 49308 61572. I think the one that’s elevated is because I’m
logged in to tracks, however.

I don’t use Mongrel, maybe that’s the difference?

mathew

I’ve done a quick write up about this on my blog. I’ve also been doing
some
preliminary memory profiling. I’ll be writing more about that soon.

http://www.stevelongdo.com/articles/2006/08/04/typo-4-0-and-memory-reduction


Thanks,
-Steve
http://www.stevelongdo.com

Thanks. Memory leak debugging with Rails is difficult.
Unfortunately, slower leaks are even harder to find and harder to
verify as fixed, but we’ll find it and fix it.

Scott

I posted a follow up showing some more specfic memory hogs. I haven’t
been
able to get anyone involved with RubyGems to clue me into why it needs
so
much memory, yet. Anyone that reads this and can provide me some
insight as
the runtime shenanigans of RubyGems please do leave a comment on my site
or
feel free to send me an email directly mocking me :slight_smile:

The post:
http://www.stevelongdo.com/articles/2006/08/08/rubygems-dependency-management-and-memory-use


Thanks,
-Steve
http://www.stevelongdo.com

I’ll bring this up at tonight’s Seattle Ruby Hacking Night–I suspect
that someone there knows the internals of rubygems well enough to help
fix the problem.

Scott

Eric Sendelbach wrote:

Yes, I am still having problems with this, but nobody seems to worry
too much about it. Unfortunately for me TextDrive would have killed
the process a LONG time ago when it went over 48 megs. I suppose I
could do the same with a cron job and monitor memory usage, but it
would have to restart every 3 minutes or so and I don’t think I nor
the nice folks at TextDrive would like that too much. :slight_smile:

Present them with that choice! “Hey, would you rather I restart my Rails
process every few minutes to keep it under 48 MB, or would you rather
relax your memory limits a bit?” :slight_smile:

mathew