# Memory allocation failure: heavy files

**URL:** <https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969>\
**Category:** Ruby\
**Created:** [April 5, 2008, 9:58pm UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969 "2008-04-05T21:58:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dreamwave](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/7feea3/32.png) [@dreamwave](https://www.ruby-forum.com/u/dreamwave)\
**Post date:** [April 5, 2008, 9:58pm UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969/1 "2008-04-05T21:58:43Z")

</div>

Seems like any combo I try, eventually it lands me in a NoMemory error.  
I’ve tried:

#### 

file = File.open(datafile)

file.each{|data| @store \<\< data}  
file.close()

@large = @store.to\_s.split(" ")

#### 

File.open(datafile) do |file|  
data = file.gets.to\_s.split(" ")  
end

#### 

(this appears to be the most inefficient one)  
file = File.readlines(datafile)  
file.each{|data| @store \<\< data}  
@large = @store.to\_s.split(" ")

#### 

And this is with only a 24mb datafile, to which effect I had to split  
the file.  
The total size of the file is around 250-300mb.

Code inefficiency issue?

Thanks,

- Mac

---

<div class="post-metadata">

**Author:** ![dreamwave](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/7feea3/32.png) [@dreamwave](https://www.ruby-forum.com/u/dreamwave)\
**Post date:** [April 5, 2008, 10:08pm UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969/2 "2008-04-05T22:08:24Z")

</div>

On 5 Apr 2008, at 20:58, Michael L. wrote:

> @large = @store.to\_s.split(" ")
> 
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

You’ll want to buffer data reads, and do the splits on data that’s  
buffered properly and completely. Many of the operations there slurp  
or are never dropping data. If you have a _very_ long line, there may  
be more work required, too, also dependent on ram.

---

<div class="post-metadata">

**Author:** ![dreamwave](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/7feea3/32.png) [@dreamwave](https://www.ruby-forum.com/u/dreamwave)\
**Post date:** [April 6, 2008, 12:52am UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969/3 "2008-04-06T00:52:42Z")

</div>

Effect achieved via:

@store = []  
IO.foreach(“testpart.txt”){|x| @store \<\< x.split(" ").grep(/#{format}/)}

#format is a Regexp

- Mac

---

<div class="post-metadata">

**Author:** ![dreamwave](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/7feea3/32.png) [@dreamwave](https://www.ruby-forum.com/u/dreamwave)\
**Post date:** [April 6, 2008, 9:13pm UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969/4 "2008-04-06T21:13:11Z")

</div>

Robert K. wrote:

> On 06.04.2008 00:52, Michael L. wrote:
> 
> > Effect achieved via:
> > 
> > @store = []  
> > IO.foreach(“testpart.txt”){|x| @store \<\< x.split(" ").grep(/#{format}/)}
> > 
> > #format is a Regexp
> 
> If format is a regular expression why then don’t you do  
> “…grep(format)”?
> 
> Kind regards
> 
> robert

Just a habit, both ways work.

---

<div class="post-metadata">

**Author:** ![dreamwave](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/7feea3/32.png) [@dreamwave](https://www.ruby-forum.com/u/dreamwave)\
**Post date:** [April 6, 2008, 11:07am UTC](https://www.ruby-forum.com/t/memory-allocation-failure-heavy-files/133969/5 "2008-04-06T11:07:12Z")

</div>

On 06.04.2008 00:52, Michael L. wrote:

> Effect achieved via:
> 
> @store = []  
> IO.foreach(“testpart.txt”){|x| @store \<\< x.split(" ").grep(/#{format}/)}
> 
> #format is a Regexp

If format is a regular expression why then don’t you do  
“…grep(format)”?

Kind regards

robert