# Simple garbage problem

**URL:** <https://www.ruby-forum.com/t/simple-garbage-problem/156743>\
**Category:** Ruby\
**Created:** [January 2, 2009, 9:17am UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743 "2009-01-02T09:17:53Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [January 2, 2009, 9:17am UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743/1 "2009-01-02T09:17:53Z")

</div>

When I run the program:

STDIN.each\_line { |line| line.split("\t") }

with a big input file, the process size grows rapidly, reaching 1G in  
less than 1 minute.  
If I let it go, it continues to grow at this rate until the whole  
system fails.  
This is ruby 1.8.6 on fedora 9.

Is this a known problem?  
Is there some way to work around it?

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [January 2, 2009, 9:31am UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743/2 "2009-01-02T09:31:23Z")

</div>

On Fri, Jan 2, 2009 at 10:10 AM, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Is this a known problem?  
> Is there some way to work around it?

I haven’t done much input processing in ruby but I’ve used this with no  
problems on large files:

while line = STDIN.gets  
…  
end

Just check that your file line breaks and the line breaks being used by  
gets/each\_line are the same.  
Also make sure you actually have line breaks (sometimes you just gotta  
check)

Andrew  
[http://ramblingsonrails.com](http://ramblingsonrails.com)

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [January 2, 2009, 11:27pm UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743/3 "2009-01-02T23:27:27Z")

</div>

> Is this a known problem?  
> Is there some way to work around it?

Which version of ruby are you using? Very old ones have some leaks.  
Besides that you may be able to apply some recent patches to GC [1] and  
see if they help.  
-=r  
[1] [http://www.ruby-forum.com/topic/170608](http://www.ruby-forum.com/topic/170608)

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [January 6, 2009, 11:13am UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743/4 "2009-01-06T11:13:37Z")

</div>

removed\_email\_address@domain.invalid wrote:

> Is this a known problem?  
> Is there some way to work around it?

Might be worth trying on JRuby. We’ve had no such reports, and the  
garbage collector is rock-solid.

- Charlie

---

<div class="post-metadata">

**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\
**Post date:** [January 4, 2009, 8:30pm UTC](https://www.ruby-forum.com/t/simple-garbage-problem/156743/5 "2009-01-04T20:30:52Z")

</div>

On 31.12.2008 17:28, removed\_email\_address@domain.invalid wrote:

> When I run the program:
> 
> STDIN.each\_line { |line| line.split("\t") }

Is it really only this line? How do you feed the big file to stdin?  
Does the big file actually _have_ lines? Because if not you would  
likely see that behavior because each\_line needs to read at least until  
it finds a line terminator.

> with a big input file, the process size grows rapidly, reaching 1G in  
> less than 1 minute.  
> If I let it go, it continues to grow at this rate until the whole  
> system fails.

What does this mean? Does the kernel panic? Or does the Ruby process  
terminate with an error?

> This is ruby 1.8.6 on fedora 9.

You do not accidentally have switched off GC completely, do you?

> Is this a known problem?  
> Is there some way to work around it?

1.8.6 is not really current any more. I would upgrade if possible - at  
least get the latest version of the package. For more advice we have a  
bit too little information, I am afraid.

When I try this with my cygwin version memory consumption stays at  
roughly 3MB:

robert@fussel ~  
$ perl -e ‘foreach $i (1…10000000) {print $i, “–\t–\t–\t–\n”;}’ \

> | ruby -e ‘STDIN.each\_line { |line| line.split("\t") }’

robert@fussel ~  
$ ruby -v  
ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]

robert@fussel ~  
$

But if I read a file that does not have lines behavior is as you  
describe, memory goes up and up

ruby -e ‘STDIN.each\_line { |line| line.split("\t") }’ \</dev/zero

Kind regards

robert