Memory Issue while looping through CSV file


#1

I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I’m a little new with Ruby and Rails so if you can
tell me where I’m going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, “r”)

csv_file.each do | pattern_temp |

unless pattern_temp[0].nil?

	pattern_record = {
			:code => pattern_temp[0].delete("\""),
			:designer => pattern_temp[1],
			:name => pattern_temp[2],
			:designs => pattern_temp[3],
			:price => pattern_temp[4].delete("$"),
			:set => pattern_temp[5].delete("\"\r\n"),
			:set_price => pattern_temp[6].delete("$"),
			:themes => pattern_temp[7],
			:bundle_with_code => pattern_temp[8],
	                :matrix => pattern_temp[9],
			:description => pattern_temp[10],
			:active => pattern_temp[11],
			:panel => pattern_temp[12]
	}

puts "Processing #{pattern_record[:code]}..."
pattern = Pattern.find_by_pattern_code(pattern_record[:code]) || 

Pattern.new

if pattern_record[:active] == "Yes"
	  add_pattern(pattern, pattern_record) # function
	elsif pattern_record[:active] == "No" and !pattern.new_record?
  puts "Removing #{pattern_record[:code]}..."
  Pattern.destroy(pattern.id)
end

end

end


#2

Does anybody have any ideas on this below. Can I provide any more
information that would help someone?

Thanks in advance,
Nicholas

---------- Forwarded message ----------
Date: Feb 6, 2006 12:14 PM
Subject: Memory Issue while looping through CSV file
To: removed_email_address@domain.invalid

I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I’m a little new with Ruby and Rails so if you can
tell me where I’m going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, “r”)

csv_file.each do | pattern_temp |

unless pattern_temp[0].nil?

            pattern_record = {
                            :code => pattern_temp[0].delete("\""),
                            :designer => pattern_temp[1],
                            :name => pattern_temp[2],
                            :designs => pattern_temp[3],
                            :price => pattern_temp[4].delete("$"),
                            :set => 

pattern_temp[5].delete(""\r\n"),
:set_price =>
pattern_temp[6].delete("$"),
:themes => pattern_temp[7],
:bundle_with_code => pattern_temp[8],
:matrix => pattern_temp[9],
:description => pattern_temp[10],
:active => pattern_temp[11],
:panel => pattern_temp[12]
}

puts "Processing #{pattern_record[:code]}..."
pattern = Pattern.find_by_pattern_code(pattern_record[:code]) || 

Pattern.new

if pattern_record[:active] == "Yes"
              add_pattern(pattern, pattern_record) # function
            elsif pattern_record[:active] == "No" and 

!pattern.new_record?
puts “Removing #{pattern_record[:code]}…”
Pattern.destroy(pattern.id)
end
end

end


#3

Not a solution but a related plea for help from a new boy…

Can someone tell me please, if I have a csv file Foo stored in
public/csvfiles, how do I open this, read its rows and then present them
on an browser screen…
1: what “requires” do I need and where does it go? [require “csv” in
the CLASS file?]
2: what code opens and reads the file row by row, and where does this
go? [do I read the file row by row into an array within the controller?]
3: how do I then display the data on screen?

Sample code would be most welcomed, plus, if possible, a reference to
any further related reading on the web.

Many thanks.


#4

On Feb 10, 2006, at 2:31 PM, Les Bowker wrote:

3: how do I then display the data on screen?

Sample code would be most welcomed, plus, if possible, a reference to
any further related reading on the web.

Here’s a script I use to import from CSV that came from Excel
spreadsheets that came from MS Access.

http://www.rafb.net/paste/results/C1r7hF26.html


#5

From the command-line:
gem install fastercsv

At the bottom of config/environment.rb, add:
require_gem ‘fastercsv’

Let’s say you have a controller action called display_data.
Further, let’s say that you already know which filename you want. You
could also pass it as a parameter, etc, etc.

def display_data

@csv_rows will be an Array of rows from the CSV file. The rows

are Arrays of columns.

Further, let’s assume that the first row contains column names.

Read the CSV file from disk, and load it up into @csv_rows.

@csv_rows = FasterCSV.read(“public/csvfiles/example.csv”)
@columns = @csv_rows.shift # Push the top row out into @columns
end

Now, in display_data.rhtml:

<% @columns.each do |col| -%> <% end -%> <%= render :partial => 'csv_row', :collection => @csv_rows %>
<%= col.titleize -%>

In “_csv_row.rhtml”:

<% csv_row.each do |entry| -%> <%= h entry %> <% end -%>

There you go. The output will be an html table matching your CSV file.
The only non-trivial part is dealing with the fact that you’ve got
more than one collection of things to iterate over. (First the column
headings, then the rows and ‘cells’)

–Wilson.