Fedora 8 and Mysql gem memory leak?

I was able to see a memory leak on my production server so I tried a
simple test on the dev server:

Fedora 8
Rails 1.2.6
MySQL 5
Gems:
mysql (2.7)
mongrel_cluster (1.0.5)
mongrel (1.1.5)

The mongrel is a cluster of 4.

app/controllers/memleak_controller.rb:

def mytest
@users = User.find(:all)
render(:text=>"DONE: "+Time.now.to_s)
end

The user model is very basic, no relationship.

class User < ActiveRecord::Base
end

coontrollers/application.rb is also stripped down to bare bones.

I then have a shell script that curl mytest (client/server on same
host).
The script is looped - so essentially a single thread request.
I watch the memory usage jumped to 3.1% per mongrel_rails instance (as
expected as it warms up).
Then it starts to crawl up: in less then 2 hours, each mongrel_rails
instance are taking up 5.6% of memory.
If I stop the shell/curl script and let the system stay idle. The memory
usage is never released.
If I restart the script/curl script, memory usage starts climbing from
where it left off. (garbage collections is not happening).

Is anyone aware of any issues?
Anything else I can do to make sure its not my code? (I’ve stripped down
as much as I can).

thanks.

So I updated ruby from:

1.86.114-1.fc8

to

1.86.230-3.fc8

The leak got WORSE!!! Leaking almost 3x as fast from the test above.

On Thu, 2008-07-03 at 21:16 +0200, RR YY wrote:

So I updated ruby from:

1.86.114-1.fc8

to

1.86.230-3.fc8

The leak got WORSE!!! Leaking almost 3x as fast from the test above.


1.86.230-4.fc8 should be installed if you update again

If that still causes issues, then remove all the ruby packages and
install the original 1.86.x that was included in Fedora-base before the
‘improvements’ but note that there were some security fixes rolled into
these updates.

Craig

Craig W. wrote:

On Thu, 2008-07-03 at 21:16 +0200, RR YY wrote:

So I updated ruby from:

1.86.114-1.fc8

to

1.86.230-3.fc8

The leak got WORSE!!! Leaking almost 3x as fast from the test above.


1.86.230-4.fc8 should be installed if you update again

If that still causes issues, then remove all the ruby packages and
install the original 1.86.x that was included in Fedora-base before the
‘improvements’ but note that there were some security fixes rolled into
these updates.

Craig

Does not look like 1.86.230-4.fc8 is out yet. At least not the fedora
mirror i am using

fedora.mirror.umoss.org

Where can I pick it up?

On Thu, 2008-07-03 at 22:43 +0200, RR YY wrote:

The leak got WORSE!!! Leaking almost 3x as fast from the test above.
Does not look like 1.86.230-4.fc8 is out yet. At least not the fedora
mirror i am using

fedora.mirror.umoss.org

Where can I pick it up?


they should have hit today but apparently not…the F9 rpms did but you
can always get them from koji…

http://koji.fedoraproject.org/koji/buildinfo?buildID=54357

Craig

Craig W. wrote:

On Thu, 2008-07-03 at 22:43 +0200, RR YY wrote:

The leak got WORSE!!! Leaking almost 3x as fast from the test above.
Does not look like 1.86.230-4.fc8 is out yet. At least not the fedora
mirror i am using

fedora.mirror.umoss.org

Where can I pick it up?


they should have hit today but apparently not…the F9 rpms did but you
can always get them from koji…

http://koji.fedoraproject.org/koji/buildinfo?buildID=54357

Craig

No good - looks like these recent ruby updates leeks memory like crazy
on Fedora 8.
I am going to use a diff fedora version and see if the leak is still
there.
(My code was originally on FC6 and ran with no leaks - so I suspect its
ruby - as the recent updates seems to validate this).

Have you tried compiling ruby from source? I have done that and do not
run into these problems. Generally, my problems were opposite in terms
of IDE installations etc. However, hand compiling the source seems to
have worked on both Ubuntu and Fedora (I am on 5, too lazy to upgrade).
I have not benchmarked anything for leaks though. One thing that
puzzles me is why do you have your Mongrels and Mongrel Clusters running
different versions?

Bharat R. wrote:

Have you tried compiling ruby from source? I have done that and do not
run into these problems. Generally, my problems were opposite in terms
of IDE installations etc. However, hand compiling the source seems to
have worked on both Ubuntu and Fedora (I am on 5, too lazy to upgrade).
I have not benchmarked anything for leaks though. One thing that
puzzles me is why do you have your Mongrels and Mongrel Clusters running
different versions?

I don’t believe its related to hand compile vs bin release.
The issue seems to be related to post FC6 (I’ve not tested FC7 - but the
leak is there for FC8). I am in the process of testing on FC6 - will
report on the results.

As for the mongrel vs mongrel cluster versions - I don’t think their
release version as in sync (someone correct me if i am wrong on this). I
use gem update and it picked up the latest version of each.

RR YY wrote:

Bharat R. wrote:

Have you tried compiling ruby from source? I have done that and do not
run into these problems. Generally, my problems were opposite in terms
of IDE installations etc. However, hand compiling the source seems to
have worked on both Ubuntu and Fedora (I am on 5, too lazy to upgrade).
I have not benchmarked anything for leaks though. One thing that
puzzles me is why do you have your Mongrels and Mongrel Clusters running
different versions?

I don’t believe its related to hand compile vs bin release.
The issue seems to be related to post FC6 (I’ve not tested FC7 - but the
leak is there for FC8). I am in the process of testing on FC6 - will
report on the results.

As for the mongrel vs mongrel cluster versions - I don’t think their
release version as in sync (someone correct me if i am wrong on this). I
use gem update and it picked up the latest version of each.

Heres what I found. For Ruby 1.8.6 OR 1.8.7 (built from source):

Recent Ruby changes does not play nice with Fedora 8. Memory usage is a
lot higher (for the same task on FC6 - its takes up a lot more memory)
AND it leaks.

On Fedora 6:

Memory usage is a lot smaller and no obvious memory leak is observed.

The tests were done using my setup described above.

Craig W. wrote:


get the source rpm from FC-6 and rebuild it on Fedora 8

Craig

not sure how that would help - since i build 1.8.7 from the same source
for both FC6 and FC8. The leak and usage on FC8 was still there.

On Sat, 2008-07-05 at 22:13 +0200, RR YY wrote:

I don’t believe its related to hand compile vs bin release.
Recent Ruby changes does not play nice with Fedora 8. Memory usage is a
lot higher (for the same task on FC6 - its takes up a lot more memory)
AND it leaks.

On Fedora 6:

Memory usage is a lot smaller and no obvious memory leak is observed.

The tests were done using my setup described above


get the source rpm from FC-6 and rebuild it on Fedora 8

Craig

On Sun, 2008-07-06 at 04:48 +0200, Bharat R. wrote:

It is quite possible that it may have nothing to do with Ruby version
but everything to do with the supporting libraries that are required to
build ruby. They are likely to be different versions too between the
releases of Fedora. Therefore what Craig is suggesting makes sense to
me, i.e., take the bin files or RPMs built for FC6 which you have no
problems with and deploy them in FC8. If you still experience problems
then it has something to do with the FC8 environment, but if you don’t
then you have squarely narrowed the problems down to libraries that were
used to build ruby either by you or the suppliers of rpms/bins for FC8.
Hope this helps.


It does seem that if he is building from source on both Fedora 6 and
Fedora 8 that the differences between them are the compiler and the
mysql-libs.

But I wonder whether he is actually re-compiling mongrel/mongrel-cluster
with each change of ruby as it does seem possible that might have some
impact.

Craig

Craig W. wrote:

On Sun, 2008-07-06 at 04:48 +0200, Bharat R. wrote:

It is quite possible that it may have nothing to do with Ruby version
but everything to do with the supporting libraries that are required to
build ruby. They are likely to be different versions too between the
releases of Fedora. Therefore what Craig is suggesting makes sense to
me, i.e., take the bin files or RPMs built for FC6 which you have no
problems with and deploy them in FC8. If you still experience problems
then it has something to do with the FC8 environment, but if you don’t
then you have squarely narrowed the problems down to libraries that were
used to build ruby either by you or the suppliers of rpms/bins for FC8.
Hope this helps.


It does seem that if he is building from source on both Fedora 6 and
Fedora 8 that the differences between them are the compiler and the
mysql-libs.

But I wonder whether he is actually re-compiling mongrel/mongrel-cluster
with each change of ruby as it does seem possible that might have some
impact.

Craig

I did not recompile mongrel/cluster. Just got the latest gem.
I am just going to revert back to FC6 - a lot less trouble and more
stable.

Thanks for the help.

It is quite possible that it may have nothing to do with Ruby version
but everything to do with the supporting libraries that are required to
build ruby. They are likely to be different versions too between the
releases of Fedora. Therefore what Craig is suggesting makes sense to
me, i.e., take the bin files or RPMs built for FC6 which you have no
problems with and deploy them in FC8. If you still experience problems
then it has something to do with the FC8 environment, but if you don’t
then you have squarely narrowed the problems down to libraries that were
used to build ruby either by you or the suppliers of rpms/bins for FC8.
Hope this helps.

RR YY wrote:

Craig W. wrote:

On Sun, 2008-07-06 at 04:48 +0200, Bharat R. wrote:

It is quite possible that it may have nothing to do with Ruby version
but everything to do with the supporting libraries that are required to
build ruby. They are likely to be different versions too between the
releases of Fedora. Therefore what Craig is suggesting makes sense to
me, i.e., take the bin files or RPMs built for FC6 which you have no
problems with and deploy them in FC8. If you still experience problems
then it has something to do with the FC8 environment, but if you don’t
then you have squarely narrowed the problems down to libraries that were
used to build ruby either by you or the suppliers of rpms/bins for FC8.
Hope this helps.


It does seem that if he is building from source on both Fedora 6 and
Fedora 8 that the differences between them are the compiler and the
mysql-libs.

But I wonder whether he is actually re-compiling mongrel/mongrel-cluster
with each change of ruby as it does seem possible that might have some
impact.

Craig

I did not recompile mongrel/cluster. Just got the latest gem.
I am just going to revert back to FC6 - a lot less trouble and more
stable.

Thanks for the help.

Just an update.
So I reverted back to 1.8.6.111-1.fc8 - this came with the default FC8
image that i am using and the leak seems to be gone (or at least not
noticeable or slower).

A few things I found (that may be related):

String handling had changed significantly with the new ruby updates. So
some of the plugins was behaving badly.
Here are some areas I noticed that will not work as expected with the
recent ruby updates:

i)
s = StringScanner.new(self.filename)
unless s.exist? /./ #need to use s.nil? instead with the new
update

ii)
From attachment_fu. This is a biggie. “my pic.jpg” will be “my_pic.jpg”
in old style BUT “my.jpg” in new ruby:

   def sanitize_filename(filename)
      returning filename.strip do |name|
        # NOTE: File.basename doesn't work right with Windows paths 

on Unix
# get only the filename, not the whole path
name.gsub! /^.*(\|/)/, ‘’

        # Finally, replace all non alphanumeric, underscore or 

periods with underscore
name.gsub! /[^\w.-]/, ‘_’
end
end

FYI:
Confirmed there are issues with Ruby:

http://www.rubyinside.com/june-2008-ruby-security-vulnerabilities-927.html

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs