How to reclaim memory without GC.start


#1

Hi - I have a memory and cpu-constrained embedded app. There’s a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush

GC.start - too expensive

end

Thanks, Dave


#2

On Jul 16, 2007, at 5:30 PM, removed_email_address@domain.invalid wrote:

GC.start - too expensive

end

Unfortunately, I don’t know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
‘=’ after buffer. It should read…

while buffer == file.read(512)

Note the double ‘=’.

G’luck, and PLEASE pop an email to the list if you figure out how to
accomplish it.
-------------------------------------------|
Nietzsche is my copilot


#3

On 7/16/07, Ari B. removed_email_address@domain.invalid wrote:

stream_host_connection.write buffer
stream_host_connection.flush

GC.start - too expensive

end

Unfortunately, I don’t know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
‘=’ after buffer. It should read…

while buffer == file.read(512)

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Todd


#4

removed_email_address@domain.invalid wrote:

GC.start - too expensive

end

Someone please correct me if I’m wrong, but…

You don’t need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration – it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you’re really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.


Travis W.

“Programming in Java is like dealing with your mom –
it’s kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend – it’s cold, unforgiving, and doesn’t tell
you what you’ve done wrong.”


#5

Travis D Warlick Jr wrote the following on 17.07.2007 01:31 :

stream_host_connection.write buffer
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you’re really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

Actually, as variable affectation is done by reference and the content
of buffer is originally duplicated by File#read memory usage is not
likely to change linearly with the value passed to File#read…

You can still play with different values passed to it and expect some
change of behaviour (ie: depending on the actual amount of memory
allocated for each read, you may or may not hit a sweat spot in the GC
heuristics).

My advise would be to do larger reads and GC.start less often:

read_amount = 65536
read_count = 0
loops_between_gcs = 100
while buffer = file.read(read_amount)
stream_host_connection.write buffer
stream_host_connection.flush
GC.start if ((read_count += 1) % loops_between_gcs) == 0
end

But note that if stream_host_connection.flush doesn’t actually wait
until the buffer isn’t referenced anymore anywhere in your code (or in
some other process sitting on the same system as a matter of fact),
calling GC.start probably won’t have any effect.

Lionel


#6

On Jul 16, 4:31 pm, Travis D Warlick Jr removed_email_address@domain.invalid
wrote:

stream_host_connection.flush
Deallocating it every time would actually increase CPU usage because you

“Programming in Java is like dealing with your mom –
it’s kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend – it’s cold, unforgiving, and doesn’t tell
you what you’ve done wrong.”

The object is being overwritten, but the old copies aren’t being
garbage collected. I know this because the memory does stay constant
if I call GC.start on every iteration.
I think I’ll go with a c extension for this small part of the code.

Thanks, Dave


#7

Travis D Warlick Jr wrote:

That should overwrite the contents of the buffer
Do you mean file.read(512, buffer) ?

---------------------------------------------------------------- IO#read
ios.read([length [, buffer]]) => string, buffer, or nil

  Reads at most length bytes from the I/O stream, or to the end of
  file if length is omitted or is nil. length must be a non-negative
  integer or nil. If the optional buffer argument is present, it
  must reference a String, which will receive the data.

  At end of file, it returns nil or "" depend on length. ios.read()
  and ios.read(nil) returns "". ios.read(positive-integer) returns
  nil.

     f = File.new("testfile")
     f.read(16)   #=> "This is line one"

#8

Lionel B. wrote:

Actually, as variable affectation is done by reference

I knew that… woops… beats head on keyboard

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer


Travis W.

“Programming in Java is like dealing with your mom –
it’s kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend – it’s cold, unforgiving, and doesn’t tell
you what you’ve done wrong.”


#9

Joel VanderWerf wrote:

Travis D Warlick Jr wrote:

Lionel B. wrote:

while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

while file.read(512, buffer)


Travis W.

“Programming in Java is like dealing with your mom –
it’s kind, forgiving, and gently chastising.
Programming in C++ is like dealing with a disgruntled
girlfriend – it’s cold, unforgiving, and doesn’t tell
you what you’ve done wrong.”


#10

On Jul 16, 2007, at 7:22 PM, Todd B. wrote:

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Yups. That is correct. My mistake, and thanks for pointing it out! At
least now I know I can do assignment in my ‘while’ statements. Right?

~ Ari
English is like a pseudo-random number generator - there are a
bajillion rules to it, but nobody cares.


#11

Travis D Warlick Jr wrote:

while file.read(512, buffer)

It’s not just prettier. With an assignment like

buffer[0…512] = file.read(512)

there is an intermediate string that instantly becomes garbage.


#12

On Jul 16, 11:30 pm, “removed_email_address@domain.invalid” removed_email_address@domain.invalid wrote:

file = File.new file_path
while buffer = file.read(512)
stream_host_connection.write buffer
stream_host_connection.flush

GC.start - too expensive

end

Note that IO#read supports a buffer argument:

buffer = “”
while file.read(512, buffer)

end

In that case it won’t create a new string object for each call. It
will just modify buffer instead.


#13

On Jul 17, 2:51 am, “Robert K.” removed_email_address@domain.invalid
wrote:

there is an intermediate string that instantly becomes garbage.
File.open file_path do |file|
Kind regards

robert

It works perfectly! Thanks everyone!


#14

2007/7/17, Joel VanderWerf removed_email_address@domain.invalid:

while file.read(512, buffer)

It’s not just prettier. With an assignment like

buffer[0…512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

There is a better way to do it:

buffer = “.” * 1024
while STDIN.read(1024, buffer)
print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
buffer = “.” * 512
while file.read(512, buffer)
stream_host_connection.write buffer
stream_host_connection.flush
end
end

The main point being here to reuse the buffer. No C extension needed.
See http://www.ruby-doc.org/core/classes/IO.html#M002295

Kind regards

robert


#15

buffer = “”
while file.read(512, buffer)

end

You could save one more line:

while file.read(512, buffer||="")

end

gegroet,
Erik V. - http://www.erikveen.dds.nl/


#16

On 8/4/07, Erik V. removed_email_address@domain.invalid wrote:

Thanks.
I think the idea was to do the allocation upfront instead of when
witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

gegroet,


#17

buffer = “.” * 1024
while STDIN.read(1024, buffer)
print buffer
end

What’s the point of initializing the buffer with 1024 characters,
instead of just initializing it with an empty string? I must be
missing something…

Thanks.

gegroet,
Erik V.


#18

2007/8/6, Logan C. removed_email_address@domain.invalid:

Thanks.

I think the idea was to do the allocation upfront instead of when witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

I believe you are right. I probably rather did it for documentary
reasons but in fact it’s not necessary. There would only be a small
but likely unnoticeable advantage if for some reason #read would not
read up to limit chars right away. Then there might be multiple
reallocations (i.e. until the buffer has reached the limit). But I
guess this is not very likely, at least not with file IO. Situation
might be different for sockets.

Kind regards

robert


#19

Finally! I’ve been waiting for someone to resurrect this thread.

Ok. I’ve decided that if a program consumes ALL of a computer’s
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what’s the best way to consume all of a computer’s RAM? I’ve been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?
---------------------------------------------------------------|
~Ari
“I don’t suffer from insanity. I enjoy every minute of it” --1337est
man alive


#20

2007/8/6, Ari B. removed_email_address@domain.invalid:

toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?

I’m not sure what you’re after. Consuming memory is easy. For example:

a=[];a << " "*1024 while true

Or, a bit more involved

Chain = Struct.new :data, :next
c = nil
c = Chain.new("."*1024, c) while true

Kind regards

robert