Forum: Ruby on Rails Best approach for importing data from a file

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
31fdda5f38fc6df4e193cfb7445ddc5a?d=identicon&s=25 RichardOnRails (Guest)
on 2008-10-24 04:24
(Received via mailing list)
Hi All,

I've got a basic Rails app started.  One wrinkle is that the user
wants ideally to be able to browse the files system (a la Windows
Explorer) to select a file and initiate the importation of data from
the file.

All files selected will be of a predetermined fixed format, e.g. CSV
with Name, Address, etc.

It seems to me that the browser is isolated from the file system,
albeit Rails' internals know how to get to it.  Is there a way to
implement the above-described functionality in a Rails app?  Can YAML
be used for such functionality?

The only approach I see right now is a Ruby database-updater using
perhaps Active Record with ar-extensions or Active Scaffold.

Any guidance would be most appreciated.

Best wishes,
Richard
B14575f0ca69f10938fdd67e7156e0e1?d=identicon&s=25 Craig Demyanovich (Guest)
on 2008-10-24 06:15
(Received via mailing list)
A Google search for

rails import csv

returns several promising results.

Regards,
Craig
31fdda5f38fc6df4e193cfb7445ddc5a?d=identicon&s=25 RichardOnRails (Guest)
on 2008-10-24 13:45
(Received via mailing list)
Hi All,

I've thought more about this and think I know how to proceed with a
Rails-integrated approach.

Thanks anyway if you've looked at the original post.

Best wishes,
Richard

On Oct 23, 10:23 pm, RichardOnRails
31fdda5f38fc6df4e193cfb7445ddc5a?d=identicon&s=25 RichardOnRails (Guest)
on 2008-10-24 13:47
(Received via mailing list)
Thanks, Craig.  That search suggestion does help.

Best wishes,
Richard

On Oct 24, 12:15 am, "Craig Demyanovich" <cdemyanov...@gmail.com>
Af2ce6689213fdb78913a9662b18da6b?d=identicon&s=25 Rick (Guest)
on 2008-10-27 18:57
(Received via mailing list)
Take a look at attachment_fu.

On Oct 23, 4:23 pm, RichardOnRails
31fdda5f38fc6df4e193cfb7445ddc5a?d=identicon&s=25 RichardOnRails (Guest)
on 2008-11-13 16:12
(Received via mailing list)
Hi Rick,

I've been busy, so I haven't checked for responses to my posts for a
week or so.  Sorry for the delay in this response.

Thanks for your response.  I just ordered Advanced Rails Recipes and
looked at the attachment_fu.rb from the book.  I'm too weak in Rails
to see how that's useful for my project.

I think I've concocted a solution to my main concern: how to access a
CVS file from the file-system from within a Rails app.  I now believe
I can:
1.  get the filenames housed in a specific directory from within a
controller using normal Ruby programming.
2.  display those filenames as a list in a view and allow the user to
select any one of then
3.  open the selected file and read its records, one by one
4.  extract the fields from each cvs-formatted record and update the
database with the contents of those fields.

Does that sound sensible to you?

Best wishes,
Richard
Af2ce6689213fdb78913a9662b18da6b?d=identicon&s=25 Rick (Guest)
on 2008-11-24 11:59
(Received via mailing list)
Hey Richard,

Sorry it's been so long, I seem to have dropped this thread.

1) My thought about attachment_fu is that it allows you to pull files
that are local to your "rails client" machine and bring them into the
rails application space.  One way it's used is to enable rails users
to post their own images onto a site for sharing with the community.
There is a browser included that works much like the usual file
browser to aid with locating / selecting file for inclusion.

I'm not sure where your csv files live right now but if they're not
somewhere below RAILS_ROOT + "public" then you'll need to figure out
how to get them there.  Attachment_fu can help here if the files are
user generated - field data and the like - that the user will upload
into the system.

2) The next thing is moving the data into your Model(s).  I'm not sure
how comfortable you are with the quality of the data but I guarantee
that, no matter what approach you take, you'll need to constantly work
on data validation.

You'll probably do best if you start with one model for each file
type.  That way you can use the model.rb file to validate, validate,
validate,... the data on it's way to the database.

3) Of course if it's your own data and you know it's always complete
and correct, you can take the migration file approach.  It's a little
more raw than what is generally considered socially acceptable in the
RoR community.  The real drawback is then specific database methods
leak in past the abstraction boundary.  Notice the code for postgresql
differs from that for mysql - and we haven't even looked at sqlite3 or
db2 or ...:-P

class CreateStateAbbrevs < ActiveRecord::Migration
  def self.up
    create_table :state_abbrevs do |t|
      t.string :name
      t.string :abbrev
    end
    csv_file = "#{RAILS_ROOT}/db/migrate/abbr_state.csv"
    fields = '(name, abbrev)'

    # PostgreSQL load from file...
    execute "COPY state_abbrevs #{fields} FROM '#{csv_file}' WITH CSV"

    # MySQL load from file...
    # execute "LOAD DATA INFILE '#{csv_file}' INTO TABLE state_abbrevs
FIELDS " +
    #   "TERMINATED BY ',' OPTIONALLY ENCLOSED BY \"\"\"\" " +
    #   "LINES TERMINATED BY '\n' " + fields

  end

  def self.down
    drop_table :state_abbrevs
  end
end

So many buttons and so little time,
Rick

On Nov 13, 5:11 am, RichardOnRails
This topic is locked and can not be replied to.