On Fri, 25 Aug 2006, Zed S. wrote:
Nope, I can’t agree with this because the ram goes up, the OS will kill it
eventually, and if I remove the guard the ram doesn’t do this.
that makes perfect sense. with the gaurd only one thread at a time can
be
initializing the huge array at once, it takes time, because of this the
number
of threads grows quite large - thus the large amount of memory consumed
as
they are added the the thread group. without the mutex the threads
simply
race right through their work - creating the array and quickly exiting.
so, when the threads can be created and die quickly the maximum memory
used by
the process simly never gets that big. with the mutex the maximum
memory is
larger simply because the threads take longer to run.
run this on your system:
harp:~ > cat a.rb
pid = Process.pid
a = []
new_array = lambda{|size| Array.new(size){ 42 }.map}
eat_memory = lambda{|n| n.times{ a << new_array[4242] }}
free_memory = lambda{a.clear and GC.start }
eat_memory[4242]
puts
puts "after malloc"
system "ps v #{ pid }"
free_memory[]
puts
puts "after free"
system "ps v #{ pid }"
harp:~ > ruby a.rb
after malloc
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
31264 pts/10 R+ 0:24 0 595 27164 26364 15.6 ruby a.rb
after free
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
31264 pts/10 S+ 0:24 0 595 27164 26368 15.6 ruby a.rb
it shows the sort of behaviour i’m talking about
And where are you getting your information that free doesn’t free
memory? I’d like to read that since all my years of C coding says that
is dead wrong.
http://groups.google.com/group/comp.unix.programmer/browse_frm/thread/44a5705312cc5df9/829197acdf6651ac?lnk=gst&q=memory+not+really+freed&rnum=4#829197acdf6651ac
http://groups.google.com/group/comp.unix.programmer/browse_frm/thread/23e7be26dd21434a/2f84b3dc080c7519?lnk=gst&q=memory+not+really+freed&rnum=9#2f84b3dc080c7519
the thing is that this has nothing to do with c and everything to do
with os
Advanced Memory Allocation | Linux Journal
"When a process needs memory, some room is created by moving the upper
bound of
the heap forward, using the brk() or sbrk() system calls. Because a
system call
is expensive in terms of CPU usage, a better strategy is to call brk()
to grab
a large chunk of memory and then split it as needed to get smaller
chunks. This
is exactly what malloc() does. It aggregates a lot of smaller malloc()
requests
into fewer large brk() calls. Doing so yields a significant performance
improvement. The malloc() call itself is much less expensive than brk(),
because it is a library call, not a system call. Symmetric behavior is
adopted
when memory is freed by the process. Memory blocks are not immediately
returned
to the system, which would require a new brk() call with a negative
argument.
Instead, the C library aggregates them until a sufficiently large,
contiguous
chunk can be freed at once.
For very large requests, malloc() uses the mmap() system call to find
addressable memory space. This process helps reduce the negative effects
of
memory fragmentation when large blocks of memory are freed but locked by
smaller, more recently allocated blocks lying between them and the end
of the
allocated space. In this case, in fact, had the block been allocated
with
brk(), it would have remained unusable by the system even if the process
freed
it."
Care to tell me how malloc/free would report 80M with Mutex but properly show
the ram go down when there is no-Mutex?
it’s a sort of race
And why Linux would kill processes if the ram get too high? Why whole
VPS servers crash? I mean if this ram was just “fake” reporting (which
is very hard to believe) then why are all these things happening?
all memory is fake. that’s why you can do this
harp:~ > free -b
total used free shared buffers
cached
Mem: 1051602944 1032327168 19275776 0 145215488
563167232
^^^^^^^^
harp:~ > ruby -e’ way_too_big = “42” * 19275776; p way_too_big.size
’
38551552
regarding the crashes - there are still limits whic something is
apparently
exceeding.
So please, point me at where in the specifications for malloc/free on
Linux it says that the memory reported will be high even though free and
malloc is called on 80M of ram slowly cycled out, and that linux will
still kill your process even though this ram is not really owned by the
process.
http://groups.google.com/group/comp.unix.programmer/search?group=comp.unix.programmer&q=memory+not+really+freed&qt_g=1&searchnow=Search+this+group
in any case, try running under electric fence, which will use a cleaner
malloc/free and show the ‘real’ behaviour. it really does seem fine.
regards.
-a
