I’m using Rails 3, Passenger, and Postgresql. Yesterday the rails
application was running very slow. So I rebooted the server. Less than
24
hours later, it is consuming even more memory than before and when
trying
to load the web page, it just hangs forever. I ran
passenger-memory-stats
to inspect the issue. And once again, I notice 2 rack processes using
over
2 gigabytes of private dirty RSS:
$ rvmsudo -E
/usr/local/rvm/gems/ruby-1.9.3-p327/bin/passenger-memory-stats
Use of uninitialized value $includedir in concatenation (.) or string at
(eval 9) line 1.
Use of uninitialized value $includedir in concatenation (.) or string at
(eval 9) line 1.
--------- Apache processes ----------
PID PPID VMSize Private Name
1079 1 145.6 MB 0.3 MB /usr/sbin/apache2 -k start
2163 1079 160.5 MB 15.2 MB /usr/sbin/apache2 -k start
2450 1079 145.9 MB 0.5 MB /usr/sbin/apache2 -k start
5638 1079 161.1 MB 15.8 MB /usr/sbin/apache2 -k start
6923 1079 145.8 MB 0.4 MB /usr/sbin/apache2 -k start
9829 1079 160.0 MB 14.9 MB /usr/sbin/apache2 -k start
11016 1079 160.5 MB 15.3 MB /usr/sbin/apache2 -k start
11097 1079 160.5 MB 15.3 MB /usr/sbin/apache2 -k start
11948 1079 145.8 MB 0.4 MB /usr/sbin/apache2 -k start
11961 1079 145.8 MB 0.3 MB /usr/sbin/apache2 -k start
11969 1079 145.8 MB 0.3 MB /usr/sbin/apache2 -k start
12460 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
12461 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
12462 1079 145.7 MB 0.2 MB /usr/sbin/apache2 -k start
12491 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
12533 1079 157.5 MB 14.4 MB /usr/sbin/apache2 -k start
12535 1079 159.8 MB 14.6 MB /usr/sbin/apache2 -k start
12536 1079 157.7 MB 14.3 MB /usr/sbin/apache2 -k start
12552 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
12717 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
12757 1079 150.1 MB 4.6 MB /usr/sbin/apache2 -k start
12990 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13015 1079 150.3 MB 5.2 MB /usr/sbin/apache2 -k start
13016 1079 159.8 MB 14.4 MB /usr/sbin/apache2 -k start
13045 1079 147.9 MB 4.3 MB /usr/sbin/apache2 -k start
13046 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13047 1079 152.4 MB 8.9 MB /usr/sbin/apache2 -k start
13288 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13324 1079 152.4 MB 8.9 MB /usr/sbin/apache2 -k start
13329 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13331 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13959 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13960 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
13961 1079 149.0 MB 3.8 MB /usr/sbin/apache2 -k start
14037 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14040 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14051 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14052 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14053 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14059 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14061 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14069 1079 148.7 MB 3.4 MB /usr/sbin/apache2 -k start
14089 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14127 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14132 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14149 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14481 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14548 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14552 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14553 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14554 1079 146.7 MB 3.4 MB /usr/sbin/apache2 -k start
14555 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14556 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14557 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14593 1079 148.8 MB 3.6 MB /usr/sbin/apache2 -k start
14594 1079 159.8 MB 14.4 MB /usr/sbin/apache2 -k start
14606 1079 146.7 MB 3.4 MB /usr/sbin/apache2 -k start
14792 1079 149.0 MB 3.8 MB /usr/sbin/apache2 -k start
14808 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14914 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14915 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14917 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14918 1079 145.9 MB 0.3 MB /usr/sbin/apache2 -k start
14920 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14927 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14997 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
14998 1079 159.7 MB 14.4 MB /usr/sbin/apache2 -k start
14999 1079 149.0 MB 3.8 MB /usr/sbin/apache2 -k start
15016 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15239 1079 145.9 MB 0.4 MB /usr/sbin/apache2 -k start
15244 1079 145.9 MB 0.3 MB /usr/sbin/apache2 -k start
15249 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15250 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15251 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15253 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15254 1079 154.5 MB 9.0 MB /usr/sbin/apache2 -k start
15255 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15256 1079 150.2 MB 4.6 MB /usr/sbin/apache2 -k start
15398 1079 145.8 MB 1.0 MB /usr/sbin/apache2 -k start
15405 1079 146.9 MB 3.7 MB /usr/sbin/apache2 -k start
15406 1079 160.0 MB 14.6 MB /usr/sbin/apache2 -k start
15407 1079 160.9 MB 15.4 MB /usr/sbin/apache2 -k start
15442 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15786 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
15906 1079 145.6 MB 0.2 MB /usr/sbin/apache2 -k start
Processes: 85
Total private dirty RSS: 282.41 MB
-------- Nginx processes --------
Processes: 0
Total private dirty RSS: 0.00 MB
------ Passenger processes ------
PID VMSize Private Name
1085 215.8 MB 0.3 MB PassengerWatchdog
1093 2021.7 MB 4.1 MB PassengerHelperAgent
1099 109.7 MB 6.7 MB Passenger spawn server
1102 159.2 MB 0.8 MB PassengerLoggingAgent
1816 563.2 MB 325.2 MB Rack:
/home/guarddog/public_html/guarddog.com/current
2593 541.4 MB 303.4 MB Rack:
/home/guarddog/public_html/guarddog.com/current
4794 354.3 MB 118.0 MB Rack:
/home/guarddog/public_html/guarddog.com/current
4829 455.9 MB 170.8 MB Rack:
/home/guarddog/public_html/guarddog.com/current
4925 1327.5 MB 1040.0 MB Rack:
/home/guarddog/public_html/guarddog.com/current
5421 1348.8 MB 1110.6 MB Rack:
/home/guarddog/public_html/guarddog.com/current
Processes: 10
Total private dirty RSS: 3079.98 MB
Do you notice how those two rack processes 4925 and 5421 are using an
incredible amount of memory, much more than anything else? Obviously,
this
is the source of the issue. But the question I have is why. It leaves me
virtually with no memory left on the machine:
$ cat meminfo
MemTotal: 6113156 kB
MemFree: 510920 kB
I looked at my server production log and noticed users were exporting
data
within a specific date range, say a week, which is probably 1 thousand
records of data. But still that shouldn’t cause so much memory
consumption.
Also emails are sent out quite frequently, but still I worked on other
rails applications, and I never saw email sending consume that much
memory.
Note I also create a new thread when sending emails:
def deliver_async(email)
Thread.new do
email.deliver
end
end
I am not sure the next step to resolve why those two rack processes are
consuming so much memory. A lot of the memory leak tools I see our
outdated
for rails 2 or older. THe rails guides mentions valgrind but that is for
testing isolated lines of code, and I am hoping for a quicker solution.
I
also read about “ObjectSpace.count_objects”. Is there an ideal place to
place this line of code in the application?