CSV upload to database

Hello all,
I’m a newbie with Ruby on Rails. So far, 3 weeks and counting. I’m
really a PHP guy with a department that changed its development
platform. But enough of the chit chat.
My project consists of a “Project” database and periodic imports (new
data) to the database. This import file will be a .csv file and it’ll be
necessary to compare the entire database for similar titles and project
managers and then import the new data to the database. (Nice project for
a newbie, uh?)

I’ve set up the initial projects database including new, edit, show,
destroy.

I’ve also created an additional “import” model/view/controller. I’ve
also installed “fasterCSV” and “attachment_fu”.
I’ve read a couple “Upload” examples but they were mainly about
uploading images. I really need an example about uploading and reading
files to database.

What I’m thinking is I upload the csv data to a temporary hash then send
it to a grid for display. My supervisor wants a side by side comparison
of the csv and the current database.

Any help will be greatly appreciated.

Thank you.

JohnM

Welcome to Rails :slight_smile:

I understand you’re only looking for help on loading the CSV-files
into the import table using Rails? I’m no expert on this, because I
only did this using PHP a long time ago, but hey, any help is help,
right?

Using fasterCSV you’ll probably want to look at parsing the CSV file
line by line. So, for example, if you know every line has a name and a
value:

FasterCSV.foreach(“path/to/file.csv”) do |row|
Import.new(:name => row[0], :value => row[1]).save
end

You can then use Rails to compare every instance of the import table
to your target table.

This might bring up some performance issues though, because you’re
doing a MySQL INSERT query on every line. You can also look at
FasterCSV.read and get that array of arrays into your table some way
(I know MySQL can import an entire table within one query), but I’m
very tired and won’t be of much help there at this moment :wink:

Hope this helps!

Kind regards,
Jaap H.
w. http://www.relywebsolutions.nl

John M. wrote:

My project consists of a “Project” database and periodic imports (new
data) to the database. This import file will be a .csv file and it’ll be
necessary to compare the entire database for similar titles and project
managers and then import the new data to the database. (Nice project for
a newbie, uh?)

This is just a suggestion, but you might look into separating this
import out to a separate “daemon” process. Maybe just a simple Rack
application or possibly use Rails Metal. The goal, of course, being to
offload this “heavy” task as to not interfere with normal request
processing in your main Rails application.

I also think I would try to skip ActiveRecord altogether for this task.
Unless you gain significant benefit by going through ActiveRecord.

Just some things to consider. Good luck and welcome to the Rails
community.

Thank you for the reply.

I will do some reading on Metal and I see there’s a railscast on the
subject too.
It won’t be too “heavy” task as the file will be approx. 17-20 MB

Thanks again.

JohnM

Robert W. wrote:

John M. wrote:

My project consists of a “Project” database and periodic imports (new
data) to the database. This import file will be a .csv file and it’ll be
necessary to compare the entire database for similar titles and project
managers and then import the new data to the database. (Nice project for
a newbie, uh?)

This is just a suggestion, but you might look into separating this
import out to a separate “daemon” process. Maybe just a simple Rack
application or possibly use Rails Metal. The goal, of course, being to
offload this “heavy” task as to not interfere with normal request
processing in your main Rails application.

I also think I would try to skip ActiveRecord altogether for this task.
Unless you gain significant benefit by going through ActiveRecord.

Just some things to consider. Good luck and welcome to the Rails
community.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs