Memory allocation failure: heavy files

Ruby
1

Seems like any combo I try, eventually it lands me in a NoMemory error.
I’ve tried:

file = File.open(datafile)

file.each{|data| @store << data}
file.close()

@large = @store.to_s.split(" ")

File.open(datafile) do |file|
data = file.gets.to_s.split(" ")
end

(this appears to be the most inefficient one)
file = File.readlines(datafile)
file.each{|data| @store << data}
@large = @store.to_s.split(" ")

And this is with only a 24mb datafile, to which effect I had to split
the file.
The total size of the file is around 250-300mb.

Code inefficiency issue?

Thanks,

  • Mac
2

On 5 Apr 2008, at 20:58, Michael L. wrote:

@large = @store.to_s.split(" ")

Posted via http://www.ruby-forum.com/.

You’ll want to buffer data reads, and do the splits on data that’s
buffered properly and completely. Many of the operations there slurp
or are never dropping data. If you have a very long line, there may
be more work required, too, also dependent on ram.

3

Effect achieved via:

@store = []
IO.foreach(“testpart.txt”){|x| @store << x.split(" ").grep(/#{format}/)}

#format is a Regexp

  • Mac
4

Robert K. wrote:

On 06.04.2008 00:52, Michael L. wrote:

Effect achieved via:

@store = []
IO.foreach(“testpart.txt”){|x| @store << x.split(" ").grep(/#{format}/)}

#format is a Regexp

If format is a regular expression why then don’t you do
“…grep(format)”?

Kind regards

robert

Just a habit, both ways work.

5

On 06.04.2008 00:52, Michael L. wrote:

Effect achieved via:

@store = []
IO.foreach(“testpart.txt”){|x| @store << x.split(" ").grep(/#{format}/)}

#format is a Regexp

If format is a regular expression why then don’t you do
“…grep(format)”?

Kind regards

robert