# Mongrel and memory usage **URL:** \ **Category:** Mongrel\ **Created:** [November 5, 2007, 11:21am UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244 "2007-11-05T11:21:55Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 11:21am UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/1 "2007-11-05T11:21:55Z")
Hello, I’m running a Rails application which must sort and manipulate a lot of data which are loaded in memory. The Rails app runs on 2 mongrel processes. When I first load the app, both are 32Mb in memory. After some days, both are between 200Mb and 300Mb. My question is : is there some kind of garbage collector in Mongrel? I never see the two Mongrel processes memory footprint decrease. Is it normal? I use Mongrel 1.0.1 with Rails 1.2.3 on Debian. Best regards, Thomas. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:21pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/2 "2007-11-05T17:21:13Z")
On 11/5/07, Thomas B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I’m running a Rails application which must sort and manipulate a lot of data > which are loaded in memory. > The Rails app runs on 2 mongrel processes. > When I first load the app, both are 32Mb in memory. > After some days, both are between 200Mb and 300Mb. > > My question is : is there some kind of garbage collector in Mongrel? > I never see the two Mongrel processes memory footprint decrease. > Is it normal? Ruby is a garbage collected language. Ruby has a conservative mark and sweep garbage collector. Memory usage like that is probably not a Mongrel issue (unless you are generating _very_ large responses in your application). It’s likely an issue with your code. What version of Ruby are you using? Are you using any extensions? Kirk H. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:34pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/3 "2007-11-05T17:34:41Z")
On 11/5/07, Kirk H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > On 11/5/07, Thomas B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > I never see the two Mongrel processes memory footprint decrease. > > Is it normal? > > Ruby is a garbage collected language. Ruby has a conservative mark > > and sweep garbage collector. But Ruby processes never release memory back to the operating system. So, the fact that its RSS never goes down is normal. In normal circumstances, Mongrel should grow up to some point around 60-120 Mb and stay there. 300 Mb and growing is a sure sign you have a memory leak somewhere. – Alexey V. CruiseControl.rb [[http://cruisecontrolrb.thoughtworks.com](http://cruisecontrolrb.thoughtworks.com)] RubyWorks [[http://rubyworks.thoughtworks.com](http://rubyworks.thoughtworks.com)] ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:22pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/4 "2007-11-05T17:22:23Z")
We’re seeing that all the time with our Rails apps. I’m looking at four processes right now in the 700 to 900MB range. My first guess is that it’s something in Rails or our app. After all, that’s where most of the code is. You might try running requests through WEBrick on a test server to see if the leak still occurs. If so, then you know at least part of it is Rails. There’s always nightly restarts 😉 Not my choice on how to do things, but hey, it’ll have to hold till I can fix bigger things. What’s the Ruby GC like? Circular references a problem? ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:35pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/5 "2007-11-05T17:35:17Z")
On 11/5/07, Kirk H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > My question is : is there some kind of garbage collector in Mongrel? > > Kirk H. > > * * * > > Mongrel-users mailing list > removed\_email\_address@domain.invalid > [http://rubyforge.org/mailman/listinfo/mongrel-users](http://rubyforge.org/mailman/listinfo/mongrel-users) Hello Kirk, Thanks for your answer. I’m using ruby 1.8.5 (2006-08-25) [i486-linux]. The Rails app uses those plugins : - acts\_as\_taggable\_on\_steroids - attachment\_fu - exception\_notification - localization Which kink of issues with my code could use that much memory? If I load lots of records with Active Records, aren’t they “unloaded” at some times? Thanks in advance for your help. Thomas. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:48pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/6 "2007-11-05T17:48:09Z")
On 11/5/07, Thomas B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > If I load lots of records with Active Records, aren’t they “unloaded” at > some times? Does your code or any of those pluginx use Array#shift? There was a bug with Array#shift which still existed in 1.8.5 which basically left stuff inside the array data structure after a shift, so that those things didn’t get GCd when they should have. It’s a sneaky bug that can easily eat a lot of memory. Otherwise, can you start a test instance of your application, and then test it to see if there are certain actions which cause the memory growth. That would help you pinpoint where the likely problems are. Just use ab or httperf to send a large number of requests to specific urls in your app, and see how ram usage changes as you do that. Kirk H. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:51pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/7 "2007-11-05T17:51:40Z")
Hi guys, Along the lines of Thomas question, I’ve noticed that my mongrel rails processes start at around 50 MB, and creep up to around 100 MB (or a little over) pretty soon after being used. Is this something other folks are seeing (i.e. standard rails overhead), or does it sound specific to my app? Also, if anyone has any tips on finding memory leaks in mongrel, they’d be much appreciated. I’ve played with watching the ObjectSpace. Is this the best way? Kirk: thanks for the tip on Array.shift with Ruby 1.8.5. I’ll keep an eye out for this. Thanks, Pete ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 6:18pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/8 "2007-11-05T18:18:41Z")
On 11/5/07, Pete DeLaurentis [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hi guys, > > Along the lines of Thomas question, I’ve noticed that my mongrel > rails processes start at around 50 MB, and creep up to around 100 MB > (or a little over) pretty soon after being used. Is this something > other folks are seeing (i.e. standard rails overhead), or does it > sound specific to my app? There is probably a jump after the first request, then a slow creep upward for a bit, then it should stabilize. If it never stabilizes, then you have something in your code somewhere which is leaking. > Also, if anyone has any tips on finding memory leaks in mongrel, > they’d be much appreciated. I’ve played with watching the > ObjectSpace. Is this the best way? There are some tools that help, but yeah, mostly it’s by using objectspace and looking through your code. If the code uses an extension, it’s easy for an extension to have a leak that doesn’t show up so easily, though. I originally found the Array#shift leak by using valgrind on Ruby, since all of that is C code. > Kirk: thanks for the tip on Array.shift with Ruby 1.8.5. I’ll keep > an eye out for this. If this bites you, you can migrate to the most recent 1.8.6, or you can change your code to not use shift. Generally when shift is used, push is being used to stick things on one end of the array while shift pulls them off the front. Changing that to use unshift and pop gets around the problem. Kirk H. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 5:39pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/9 "2007-11-05T17:39:57Z")
On 11/5/07, Alexey V. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > In normal circumstances, Mongrel should grow up to some point around > [http://rubyforge.org/mailman/listinfo/mongrel-users](http://rubyforge.org/mailman/listinfo/mongrel-users) Hello Alexey, > 300 Mb and growing is a sure sign you have a memory leak somewhere. What would you suggest me to investigate? Thanks, Thomas. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 6:25pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/10 "2007-11-05T18:25:42Z")
On 11/5/07, Pete DeLaurentis [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Along the lines of Thomas question, I’ve noticed that my mongrel > rails processes start at around 50 MB, and creep up to around 100 MB Set --num-procs lower than default 1024, and it won’t be happening. – Alexey V. CruiseControl.rb [[http://cruisecontrolrb.thoughtworks.com](http://cruisecontrolrb.thoughtworks.com)] RubyWorks [[http://rubyworks.thoughtworks.com](http://rubyworks.thoughtworks.com)] ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 6:38pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/11 "2007-11-05T18:38:36Z")
On 11/5/07, Alexey V. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > On 11/5/07, Pete DeLaurentis [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > Along the lines of Thomas question, I’ve noticed that my mongrel > > rails processes start at around 50 MB, and creep up to around 100 MB > > Set --num-procs lower than default 1024, and it won’t be happening. It depends. One cause of that sort of creeping mem usage is having an app that sees large numbers of concurrent threads, as you know, but it’s not the only cause. If concurrent threads ARE a mem usage problem, one might try using evented\_mongrel out of the Swiftiply package. [http://swiftiply.swiftcore.org](http://swiftiply.swiftcore.org) Just run it in a test environment and see if it helps. For some apps, it makes a big difference in that thread related RAM creep. Kirk H. P.S. Yes, I WILL have the patch to fix it for Mongrel \> 1.0.1 today. The end of my week/weekend got very busy with things that don’t involve computer screens. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 7:55pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/12 "2007-11-05T19:55:21Z")
What is a good value for --num-procs for rails applications, since these are single threaded? Does it depend on how fast the application responds to users? Thanks, Pete ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 8:03pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/13 "2007-11-05T20:03:33Z")
On 11/5/07, Pete DeLaurentis [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > What is a good value for --num-procs for rails applications, since > these are single threaded? Does it depend on how fast the > application responds to users? It’s application specific. Your sweet spot is going to be big enough that you don’t experience capacity starvation during load bursts when you have temporary periods where more traffic is coming in than you are clearing, but small enough that you don’t waste resources. Experimentation will probably be required to find the best balance. If you try evented\_mongrel, you don’t need to worry about num\_procs. It’s irrelevant for the evented\_mongrel. Kirk H. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 9:49pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/14 "2007-11-05T21:49:00Z")
Which Image processor are you using for attachment\_fu? If you’re using RMagick, it is notorious for memory leaks. Look at mini\_magick or ImageScience as a replacement. == Will G. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 10:31pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/15 "2007-11-05T22:31:44Z")
If you’re using attachment\_fu and send\_file then mongrel is handling the sending of files. I had the same problem, spiking memory usage, until I switched to using x\_send\_file. It pushes the file downloads to apache, instead of mongrel. My memory usage has never spiked since… The XSendFile plugin [http://tn123.ath.cx/mod\_xsendfile/](http://tn123.ath.cx/mod_xsendfile/) Plugin to simplify using x-sendfile… [http://john.guen.in/past/2007/4/17/send\_files\_faster\_with\_xsendfile/](http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/) (e) ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 6, 2007, 4:35am UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/16 "2007-11-06T04:35:17Z")
Hi Kirk, I’m wondering if we’re being hit by this issue in our application. We generate a lot of thumbnails on the fly and use send\_file to transfer the data back to the browsers. Checking the rails docks for send\_file it indicates, that unless you use the option :stream =\> false, the file will be read into a 4096 byte buffer and streamed to the client. [Peak Obsession](http://api.rubyonrails.com/classes/ActionController/Streaming.html#M000093) Is this a bug in send\_file? Cheers Dave ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 6, 2007, 9:11am UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/17 "2007-11-06T09:11:46Z")
Hi Kirk, Does Mongrel need to be multi-threaded at all if you’re working with Rails applications? I use Lighttpd’s mod\_proxy\_core to distribute incoming requests between 8 mongrels. If mongrel A is working on another request, I want mongrel B to pick up the request right away. If all 8 mongrels are busy, I believe Lighty retries the cycle a few times. So, who needs threads? I’m guessing this is a naive question, but I’d appreciate it if you’d set me straight. Once I get a breather in my release schedule, I plan to look at a switch to evented mongrel. Performance benchmarks + community feedback looks very good. But I still need to get a better grasp on how it works + the differences from standard mongrel. Thanks, Pete ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 5, 2007, 10:39pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/18 "2007-11-05T22:39:57Z")
On 11/5/07, Matte E. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > If you’re using attachment\_fu and send\_file then mongrel is handling the > sending of files. I had the same problem, spiking memory usage, until I > switched to using x\_send\_file. It pushes the file downloads to apache, > instead of mongrel. My memory usage has never spiked since… This falls under the category of creating http responses. If you are using send\_file within Mongrel, then the response object that is created will contain all of the file contents. If your file is small to moderately sized, that’s no big deal, but if you start pushing large files around, it will have an impact on your RAM usage. Pushing huge files via send\_file necessarily implies huge RAM usage. Don’t do that. x\_send\_file is one way to avoid doing that. Kirk H. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 6, 2007, 12:30pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/19 "2007-11-06T12:30:14Z")
On 11/5/07, Steve M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > can easily eat a lot of memory. > > Thanks Kirk - I guess I’m totally OT at this point, but I hadn’t heard > > Mongrel-users mailing list > > removed\_email\_address@domain.invalid > > [http://rubyforge.org/mailman/listinfo/mongrel-users](http://rubyforge.org/mailman/listinfo/mongrel-users) Hello, Thanks everybody for all those informations. I’ll make some tests and I’ll keep you posted. I won’t have the time to run those tests this week, but I won’t forget to post the results on the list. Best, Thomas. ---
**Author:** ![tbaustert](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/ecae2f/32.png) [@tbaustert](https://www.ruby-forum.com/u/tbaustert)\ **Post date:** [November 6, 2007, 12:41pm UTC](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244/20 "2007-11-06T12:41:58Z")
On 11/5/07, Steve M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Thanks Kirk - I guess I’m totally OT at this point, but I hadn’t heard > about this bug before. From your description this is a specific problem > to the underlying C code implementing shift, which is not found in > related functions? So “array.slice!(0)” would be identical in function > to shift but not contain this leak? Yeah. It looked to me like whoever wrote the original array.c code just forgot something when writing the code, because it’s just #shift that has the problem. This bug was fixed, but not until 1.8.6. I know it is fixed as of at least the last couple of patch releases. I am unsure if it was fixed in the original 1.8.6 release, however. Kirk H. [Next page](https://www.ruby-forum.com/t/mongrel-and-memory-usage/119244.md?page=2)