Forum: Ruby How to reclaim memory without GC.start

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
dtuttle1@gmail.com (Guest)
on 2007-07-16 23:32
(Received via mailing list)
Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end

Thanks, Dave
Ari Brown (Guest)
on 2007-07-17 01:10
(Received via mailing list)
On Jul 16, 2007, at 5:30 PM, dtuttle1@gmail.com wrote:

>   # GC.start - too expensive
> end

Unfortunately, I don't know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
'=' after buffer. It should read...
...
while buffer == file.read(512)
...

Note the double '='.

G'luck, and PLEASE pop an email to the list if you figure out how to
accomplish it.
-------------------------------------------|
Nietzsche is my copilot
Todd Benson (Guest)
on 2007-07-17 01:30
(Received via mailing list)
On 7/16/07, Ari Brown <ari@aribrown.com> wrote:
> >   stream_host_connection.write buffer
> >   stream_host_connection.flush
> >   # GC.start - too expensive
> > end
>
> Unfortunately, I don't know how to do it. If I did, I would include
> it in every single one of my programs. But you should have an extra
> '=' after buffer. It should read...
> ...
> while buffer == file.read(512)

No, Ari.  I think you misread the program.  There is an assignment to
buffer, not a comparison.  The while loop will exit when
file.read(512) returns nil.  I know why you thought it was a
comparison, though.

Todd
Travis D Warlick Jr (Guest)
on 2007-07-17 01:32
(Received via mailing list)
dtuttle1@gmail.com wrote:
>   # GC.start - too expensive
> end

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration.  However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."
Lionel Bouton (Guest)
on 2007-07-17 01:56
(Received via mailing list)
Travis D Warlick Jr wrote the following on 17.07.2007 01:31 :
>>   stream_host_connection.write buffer
> bytes (plus the static overhead for the object).
>
> Deallocating it every time would actually increase CPU usage because you
> would be deleting and recreating the buffer object at the beginning of
> every iteration.
>
> If you're really constrained on memory, you might want to consider
> decreasing the amount that you read at every iteration.  However,
> remember that you increase your CPU usage by the same factor that you
> decrease your memory usage in this case.
>

Actually, as variable affectation is done by reference and the *content*
of buffer is originally duplicated by File#read memory usage is not
likely to change linearly with the value passed to File#read...

You can still play with different values passed to it and expect some
change of behaviour (ie: depending on the actual amount of memory
allocated for each read, you may or may not hit a sweat spot in the GC
heuristics).

My advise would be to do larger reads and GC.start less often:

read_amount = 65536
read_count = 0
loops_between_gcs = 100
while buffer = file.read(read_amount)
  stream_host_connection.write buffer
  stream_host_connection.flush
  GC.start if ((read_count += 1) % loops_between_gcs) == 0
end


But note that if stream_host_connection.flush doesn't actually wait
until the buffer isn't referenced anymore anywhere in your code (or in
some other process sitting on the same system as a matter of fact),
calling GC.start probably won't have any effect.

Lionel
dtuttle1@gmail.com (Guest)
on 2007-07-17 02:07
(Received via mailing list)
On Jul 16, 4:31 pm, Travis D Warlick Jr <warli...@operissystems.com>
wrote:
> >   stream_host_connection.flush
> Deallocating it every time would actually increase CPU usage because you
>
>   "Programming in Java is like dealing with your mom --
>    it's kind, forgiving, and gently chastising.
>    Programming in C++ is like dealing with a disgruntled
>    girlfriend -- it's cold, unforgiving, and doesn't tell
>    you what you've done wrong."

The object is being overwritten, but the old copies aren't being
garbage collected. I know this because the memory does stay constant
if I call GC.start on every iteration.
I think I'll go with a c extension for this small part of the code.

Thanks, Dave
Travis D Warlick Jr (Guest)
on 2007-07-17 02:55
(Received via mailing list)
Lionel Bouton wrote:
> Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

    while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."
Joel VanderWerf (Guest)
on 2007-07-17 03:03
(Received via mailing list)
Travis D Warlick Jr wrote:
> That should overwrite the contents of the buffer
Do you mean file.read(512, buffer) ?


---------------------------------------------------------------- IO#read
      ios.read([length [, buffer]])    => string, buffer, or nil
------------------------------------------------------------------------
      Reads at most length bytes from the I/O stream, or to the end of
      file if length is omitted or is nil. length must be a non-negative
      integer or nil. If the optional buffer argument is present, it
      must reference a String, which will receive the data.

      At end of file, it returns nil or "" depend on length. ios.read()
      and ios.read(nil) returns "". ios.read(positive-integer) returns
      nil.

         f = File.new("testfile")
         f.read(16)   #=> "This is line one"
Travis D Warlick Jr (Guest)
on 2007-07-17 03:36
(Received via mailing list)
Joel VanderWerf wrote:
> Travis D Warlick Jr wrote:
>> Lionel Bouton wrote:
>>
>>     while buffer[0...512] = file.read(512)
>>
>
> Do you mean file.read(512, buffer) ?

Ahh, yes.  That would be an much prettier way to do it.

    while file.read(512, buffer)

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."
Florian Gross (Guest)
on 2007-07-17 03:37
(Received via mailing list)
On Jul 16, 11:30 pm, "dtutt...@gmail.com" <dtutt...@gmail.com> wrote:
> file = File.new file_path
> while buffer = file.read(512)
>   stream_host_connection.write buffer
>   stream_host_connection.flush
>   # GC.start - too expensive
> end

Note that IO#read supports a buffer argument:

buffer = ""
while file.read(512, buffer)
  ...
end

In that case it won't create a new string object for each call. It
will just modify buffer instead.
Ari Brown (Guest)
on 2007-07-17 03:42
(Received via mailing list)
On Jul 16, 2007, at 7:22 PM, Todd Benson wrote:
> No, Ari.  I think you misread the program.  There is an assignment to
> buffer, not a comparison.  The while loop will exit when
> file.read(512) returns nil.  I know why you thought it was a
> comparison, though.

Yups. That is correct. My mistake, and thanks for pointing it out! At
least now I know I can do assignment in my 'while' statements. Right?

~ Ari
English is like a pseudo-random number generator - there are a
bajillion rules to it, but nobody cares.
Joel VanderWerf (Guest)
on 2007-07-17 05:19
(Received via mailing list)
Travis D Warlick Jr wrote:
>     while file.read(512, buffer)
It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.
Robert Klemme (Guest)
on 2007-07-17 11:52
(Received via mailing list)
2007/7/17, Joel VanderWerf <vjoel@path.berkeley.edu>:
> >
> >     while file.read(512, buffer)
>
> It's not just prettier. With an assignment like
>
> buffer[0...512] = file.read(512)
>
> there is an intermediate string that instantly becomes garbage.

There is a better way to do it:

buffer = "." * 1024
while STDIN.read(1024, buffer)
  print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
  buffer = "." * 512
  while file.read(512, buffer)
    stream_host_connection.write buffer
    stream_host_connection.flush
  end
end

The main point being here to reuse the buffer.  No C extension needed.
 See http://www.ruby-doc.org/core/classes/IO.html#M002295

Kind regards

robert
dtuttle1@gmail.com (Guest)
on 2007-07-17 19:30
(Received via mailing list)
On Jul 17, 2:51 am, "Robert Klemme" <shortcut...@googlemail.com>
wrote:
>
> > there is an intermediate string that instantly becomes garbage.
> File.open file_path do |file|
> Kind regards
>
> robert

It works perfectly! Thanks everyone!
Erik Veenstra (Guest)
on 2007-08-04 13:18
(Received via mailing list)
> buffer = ""
> while file.read(512, buffer)
>   ...
> end

You could save one more line:

while file.read(512, buffer||="")
 ...
end

gegroet,
Erik V. - http://www.erikveen.dds.nl/
Erik Veenstra (Guest)
on 2007-08-04 13:22
(Received via mailing list)
> buffer = "." * 1024
> while STDIN.read(1024, buffer)
>   print buffer
> end

What's the point of initializing the buffer with 1024 characters,
instead of just initializing it with an empty string? I must be
missing something...

Thanks.

gegroet,
Erik V.
Logan Capaldo (Guest)
on 2007-08-06 02:58
(Received via mailing list)
On 8/4/07, Erik Veenstra <erikveen@gmail.com> wrote:
> Thanks.
I think the idea was to do the allocation upfront instead of when
witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

gegroet,
Ari Brown (Guest)
on 2007-08-06 03:42
(Received via mailing list)
Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?
---------------------------------------------------------------|
~Ari
"I don't suffer from insanity. I enjoy every minute of it" --1337est
man alive
Robert Klemme (Guest)
on 2007-08-06 09:40
(Received via mailing list)
2007/8/6, Logan Capaldo <logancapaldo@gmail.com>:
> >
> > Thanks.
>
>
> I think the idea was to do the allocation upfront instead of when witting to
> the string. But since that will only happen once, (on the first read) it
> seems unnecessary. But I could be wrong.

I believe you are right. I probably rather did it for documentary
reasons but in fact it's not necessary.  There would only be a small
but likely unnoticeable advantage if for some reason #read would not
read up to limit chars right away.  Then there might be multiple
reallocations (i.e. until the buffer has reached the limit).  But I
guess this is not very likely, at least not with file IO.  Situation
might be different for sockets.

Kind regards

robert
Robert Klemme (Guest)
on 2007-08-06 10:37
(Received via mailing list)
2007/8/6, Ari Brown <ari@aribrown.com>:
> toying with the idea of massive variable generation, but i think that
> might be a bit difficult.
>
> So what do you all think?

I'm not sure what you're after. Consuming memory is easy. For example:

a=[];a << " "*1024 while true

Or, a bit more involved

Chain = Struct.new :data, :next
c = nil
c = Chain.new("."*1024, c) while true

Kind regards

robert
Todd Benson (Guest)
on 2007-08-06 17:23
(Received via mailing list)
On 8/5/07, Ari Brown <ari@aribrown.com> wrote:
> toying with the idea of massive variable generation, but i think that
> might be a bit difficult.
>
> So what do you all think?

Why oh why are you asking for this, Ari?

Todd
Ari Brown (Guest)
on 2007-08-06 17:54
(Received via mailing list)
On Aug 6, 2007, at 11:22 AM, Todd Benson wrote:

>> So what's the best way to consume all of a computer's RAM? I've been
>> toying with the idea of massive variable generation, but i think that
>> might be a bit difficult.
>>
>> So what do you all think?
>
> Why oh why are you asking for this, Ari?

This can be of great use for computers with low memory - You run this
script, it uses up all available memory, and then it exits. Guess
what happens then? GC clears out all of the RAM the ruby script was
using. And since it was using up all available RAM, other programs
can then run better and faster.

You see? Its not malicious!


-------------------------------------------------------|
~ Ari
crap my sig won't fit
Lionel Bouton (Guest)
on 2007-08-06 18:00
(Received via mailing list)
Ari Brown wrote:
>
> This can be of great use for computers with low memory - You run this
> script, it uses up all available memory, and then it exits. Guess what
> happens then? GC clears out all of the RAM the ruby script was using.
> And since it was using up all available RAM, other programs can then
> run better and faster.

At this moment, other programs have been painfully pushed to swap, so
they have to get out of swap before doing anything. If you want to solve
a memory management problem, don't use a program, tune your kernel (or
swap the OS for another if it's unable to do memory management properly,
Ruby works on a large choice of OS...).

Lionel
Todd Benson (Guest)
on 2007-08-06 19:36
(Received via mailing list)
On 8/6/07, Ari Brown <ari@aribrown.com> wrote:
> >> I have done several scientific tests with this on World of Warcraft
> This can be of great use for computers with low memory - You run this
> script, it uses up all available memory, and then it exits. Guess
> what happens then? GC clears out all of the RAM the ruby script was
> using. And since it was using up all available RAM, other programs
> can then run better and faster.

OK.  I might be using up my credibility here, but ...

/memory/intelligence/

>
> You see? Its not malicious!

That's not  my intention either.  The only thing that comes to mind is
"shooting themselves in the foot"?

Point being: yes, it is useful to be destructive in order to be
productive.  But, do you really have to?

Todd
Ari Brown (Guest)
on 2007-08-06 21:57
(Received via mailing list)
On Aug 6, 2007, at 1:34 PM, Todd Benson wrote:
<snip>
>
> Point being: yes, it is useful to be destructive in order to be
> productive.  But, do you really have to?

Well, no, I don't have to, but it would be really nice to be able to
have a bunch of free memory so my computer doesn't stall during
another program.  Is there an alternative to soaking up the RAM and
then releasing it?

-------------------------------------------------------|
~ Ari
crap my sig won't fit
ara.t.howard (Guest)
on 2007-08-06 23:59
(Received via mailing list)
On Aug 6, 2007, at 1:55 PM, Ari Brown wrote:

> Well, no, I don't have to, but it would be really nice to be able
> to have a bunch of free memory so my computer doesn't stall during
> another program.  Is there an alternative to soaking up the RAM and
> then releasing it?

check out /dev/shm or mmap.  to allocate a big string just use

huge = 0.chr * (2 ** 32)

regards.

a @ http://drawohara.com/
John Joyce (Guest)
on 2007-08-07 18:10
(Received via mailing list)
On Aug 6, 2007, at 2:55 PM, Ari Brown wrote:

>>
> then releasing it?
>
> -------------------------------------------------------|
> ~ Ari
> crap my sig won't fit
>
>
It's just generally accepted that the Berkeley and MIT crowd have
figured out memory management well enough at the OS level. The best
way to free up additional memory is to run fewer apps at the same
time and tweak the OS to run less stuff in the background.

One aspect of memory management, however, is that there is just a
limit to how many apps can run smoothly at the same time on the same
machine. Even with lots of memory, you still run into limits on the
processor.
This topic is locked and can not be replied to.