Question on memory usage, garbage collector, 'top' stats on linux

Rails
1

Hi all,
I am trying to tune and troubleshoot an application which runs a
background
(delayed_job) process involving the import and processing of large csv
files
into the db and then processing of that data. When I run the process
(here
Ubuntu 10.04) and monitor ‘top’ in the terminal, I see that ‘Mem: used’
starts at maybe 500000k and then ups through the process to upwards
towards
4gb (which is the system total memory). However, if I look at %MEM in
the
processes, I see that ruby is using maybe 10% of total memory. Also I do
not
see processes adding up %MEM that comes anywhere close to ‘Mem: used’ at
the
top summary.

So my questions are:
(1) do I need to be concerned that used memory is approaching max and
seems
to be caused by running my application (no other apps or processes are
running on the server that I have started).

(2) Does used memory (Mem: used) mean that it can be re-allocated, or
that
it is locked? (I believe on the stats on the mac differentiate more
between
active, inactive and used, but here on linux there is just ‘free’ and
‘used’, and I am assuming used contains both inactive and used
corresponding
to the mac buckets.

(3) Should I be looking at manually running the ruby garbage collector
manually to mitigate such excessive memory balooning (while the actual
current ruby processes are not using actively nearly so much memory)?

Example (linux) towards end of large background process run:
top - 13:24:52 up 5:19, 3 users, load average: 1.21, 1.42, 1.35
Tasks: 89 total, 2 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.3%us, 0.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 3994516k total, 3809772k used, 184744k free, 119424k buffers
Swap: 905208k total, 0k used, 905208k free, 3176008k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
6291 webappus 20 0 274m 151m 3276 S 0.3 3.9 8:27.12
ruby
6218 webappus 20 0 206m 82m 3544 R 98.9 2.1 0:48.81
ruby
6208 webappus 20 0 179m 59m 4788 S 0.0 1.5 0:07.50
ruby
6295 postgres 20 0 102m 32m 28m S 0.0 0.8 17:54.62
postgres
1034 postgres 20 0 98.7m 26m 25m S 0.0 0.7 0:23.67
postgres
843 mysql 20 0 174m 26m 6648 S 0.0 0.7 0:31.82
mysqld
6222 postgres 20 0 107m 19m 11m S 0.0 0.5 0:00.61
postgres
6158 root 20 0 42668 8684 2344 S 0.0 0.2 0:02.48
ruby
907 postgres 20 0 98.6m 6680 5528 S 0.0 0.2 0:13.14 postgres

2

On Jul 7, 3:59pm, DK [email protected] wrote:

It does include stuff that is free-able. In a nutshell, if the kernel
has read some pages then even after those pages are no longer in use
it will keep them around on the off chance that they’re used again
(see the large Swap: cached number).
You might also find the output of free -m useful.

It’s unlikely that you need to run the garbage collector more often -
if anything the out of the box configuration runs it too often.

Fred

3

On 7 July 2011 16:37, Colin L. [email protected] wrote:

towards

Thanks guys. What is interesting and seems to support Fredericks
statement
about the garbage collector is that when I did put in a ‘GC.start’ call
after each major step in the process it did not seem to free up any
memory,
in fact the memory stats looked a bit worse.

As far as cached, I see in top:

Mem: 3994516k total, 3830048k used, 164468k free, 119540k buffers
Swap: 905208k total, 0k used, 905208k free, 3207936k cached

But the ‘cached’ seems to fall under Swap. Do you think this is relating
to
Mem and not the Swap file? Would make sense as would leave ~600-700 mb
as
the actual memory being used which seems to correspond to what I see
being
taken by the actual processes.

4

DK <dk.kahn@…> writes:

As far as cached, I see in top:

Mem: 3994516k total, 3830048k used, 164468k free, 119540k buffers
Swap: 905208k total, 0k used, 905208k free, 3207936k cached

Being simplistic, I would not worry about it unless the swap file is
actually
being used (which would indicate you’re running out of actual RAM).

5

On 7 July 2011 15:59, DK [email protected] wrote:

Hi all,
I am trying to tune and troubleshoot an application which runs a background
(delayed_job) process involving the import and processing of large csv files
into the db and then processing of that data. When I run the process (here
Ubuntu 10.04) and monitor ‘top’ in the terminal, I see that ‘Mem: used’
starts at maybe 500000k and then ups through the process to upwards towards
4gb (which is the system total memory). However, if I look at %MEM in the
processes, I see that ruby is using maybe 10% of total memory. Also I do not
see processes adding up %MEM that comes anywhere close to ‘Mem: used’ at the
top summary.

My understanding is that the amount of memory that can be made
available if required is the sum of free, buffered and cached. Though
possibly this is a gross over simplification.

Colin