Forum: Ruby on Rails Memory Issue while looping through CSV file

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Nicholas H. (Guest)
on 2006-02-06 19:14
(Received via mailing list)
I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I'm a little new with Ruby and Rails so if you can
tell me where I'm going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, "r")

csv_file.each do | pattern_temp |

  unless pattern_temp[0].nil?

		pattern_record = {
				:code => pattern_temp[0].delete("\""),
				:designer => pattern_temp[1],
				:name => pattern_temp[2],
				:designs => pattern_temp[3],
				:price => pattern_temp[4].delete("$"),
				:set => pattern_temp[5].delete("\"\r\n"),
				:set_price => pattern_temp[6].delete("$"),
				:themes => pattern_temp[7],
				:bundle_with_code => pattern_temp[8],
		                :matrix => pattern_temp[9],
				:description => pattern_temp[10],
				:active => pattern_temp[11],
				:panel => pattern_temp[12]
		}

    puts "Processing #{pattern_record[:code]}..."
    pattern = Pattern.find_by_pattern_code(pattern_record[:code]) ||
Pattern.new

    if pattern_record[:active] == "Yes"
		  add_pattern(pattern, pattern_record) # function
		elsif pattern_record[:active] == "No" and !pattern.new_record?
      puts "Removing #{pattern_record[:code]}..."
      Pattern.destroy(pattern.id)
    end
  end

end
Nicholas H. (Guest)
on 2006-02-07 05:48
(Received via mailing list)
Does anybody have any ideas on this below. Can I provide any more
information that would help someone?

Thanks in advance,
Nicholas

---------- Forwarded message ----------
Date: Feb 6, 2006 12:14 PM
Subject: Memory Issue while looping through CSV file
To: removed_email_address@domain.invalid


I have a site on a shared host at TextDrive. I have an import process
that imports a CSV file into the database using a ruby script executed
using script/runner. TextDrive has a memory limit which I am hitting
and the process gets killed. I get through to about 400 records before
it dies. I guess each time I instantate a new object for each record I
am using more memory but thought I was using reusing the same memory
allocated. Anyway, I'm a little new with Ruby and Rails so if you can
tell me where I'm going wrong that would be much appreciated. I have
included the code below. Thanks!

csv_file = CSV.open(CSV_PATH, "r")

csv_file.each do | pattern_temp |

  unless pattern_temp[0].nil?

                pattern_record = {
                                :code => pattern_temp[0].delete("\""),
                                :designer => pattern_temp[1],
                                :name => pattern_temp[2],
                                :designs => pattern_temp[3],
                                :price => pattern_temp[4].delete("$"),
                                :set =>
pattern_temp[5].delete("\"\r\n"),
                                :set_price =>
pattern_temp[6].delete("$"),
                                :themes => pattern_temp[7],
                                :bundle_with_code => pattern_temp[8],
                                :matrix => pattern_temp[9],
                                :description => pattern_temp[10],
                                :active => pattern_temp[11],
                                :panel => pattern_temp[12]
                }

    puts "Processing #{pattern_record[:code]}..."
    pattern = Pattern.find_by_pattern_code(pattern_record[:code]) ||
Pattern.new

    if pattern_record[:active] == "Yes"
                  add_pattern(pattern, pattern_record) # function
                elsif pattern_record[:active] == "No" and
!pattern.new_record?
      puts "Removing #{pattern_record[:code]}..."
      Pattern.destroy(pattern.id)
    end
  end

end
Les Bowker (Guest)
on 2006-02-11 00:31
(Received via mailing list)
Not a solution but a related plea for help from a new boy...

Can someone tell me please, if I have a csv file Foo stored in
public/csvfiles, how do I open this, read its rows and then present them
on an browser screen...
1: what "requires" do I need and where does it go?  [require "csv" in
the CLASS file?]
2: what code opens and reads the file row by row, and where does this
go? [do I read the file row by row into an array within the controller?]
3: how do I then display the data on screen?

Sample code would be most welcomed, plus, if possible, a reference to
any further related reading on the web.

Many thanks.
Matthew King (Guest)
on 2006-02-11 01:24
(Received via mailing list)
On Feb 10, 2006, at 2:31 PM, Les Bowker wrote:
> 3: how do I then display the data on screen?
>
> Sample code would be most welcomed, plus, if possible, a reference to
> any further related reading on the web.

Here's a script I use to import from CSV that came from Excel
spreadsheets that came from MS Access.

http://www.rafb.net/paste/results/C1r7hF26.html
Wilson B. (Guest)
on 2006-02-11 01:31
(Received via mailing list)
>From the command-line:
gem install fastercsv

At the bottom of config/environment.rb, add:
require_gem 'fastercsv'


Let's say you have a controller action called display_data.
Further, let's say that you already know which filename you want. You
could also pass it as a parameter, etc, etc.


def display_data
  # @csv_rows will be an Array of rows from the CSV file.  The rows
are Arrays of columns.
  # Further, let's assume that the first row contains column names.

  # Read the CSV file from disk, and load it up into @csv_rows.
  @csv_rows = FasterCSV.read("public/csvfiles/example.csv")
  @columns = @csv_rows.shift # Push the top row out into @columns
end

Now, in display_data.rhtml:

<table>
<thead>
  <% @columns.each do |col| -%>
  <th><%= col.titleize -%></th>
  <% end -%>
</thead>
<tbody>
  <%= render :partial => 'csv_row', :collection => @csv_rows %>
</tbody>
</table>

In "_csv_row.rhtml":
<tr>
<% csv_row.each do |entry| -%>
  <td><%= h entry %></td>
<% end -%>
</tr>

There you go. The output will be an html table matching your CSV file.
The only non-trivial part is dealing with the fact that you've got
more than one collection of things to iterate over. (First the column
headings, then the rows and 'cells')

--Wilson.
This topic is locked and can not be replied to.