Importing Large Data Sets into Rails App

Rails
1

I am building an app for a client that will require importing large
amounts (~2GB) of zipcode-specific data from Excel files. I was just
wondering what you folks thought the best solution for importing the
data to the database would be? Currently I’m planning on converting
the Excel data to .csv and writing a script that will load from an
arbitrary number of .csv diles in a directory, but that will take
hours to complete. Is there a better way?

Thanks,

Michael B.
[email protected]

2

Well, you will not want to create an AR object for each row, because,
like you said, that would take hours and hours. You should use the
actual DB connection that AR gives you and do multiple inserts with
straight SQL. If you are using MySQL the best way to load that amount
of data would be to use the LOAD DATA directive. I’m sure most DB’s
have similar functionality for loading large amounts of data all at
once. If you are using MySQL read up on LOAD DATA because that is your
best option.

-Ben