# Migration: upload a large set of data via a separate sql fil

**URL:** <https://www.ruby-forum.com/t/migration-upload-a-large-set-of-data-via-a-separate-sql-fil/103963>\
**Category:** Rails\
**Created:** [May 12, 2007, 1:05am UTC](https://www.ruby-forum.com/t/migration-upload-a-large-set-of-data-via-a-separate-sql-fil/103963 "2007-05-12T01:05:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chuong\_Huynh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/3bc359/32.png) [@Chuong\_Huynh](https://www.ruby-forum.com/u/Chuong_Huynh)\
**Post date:** [May 12, 2007, 1:05am UTC](https://www.ruby-forum.com/t/migration-upload-a-large-set-of-data-via-a-separate-sql-fil/103963/1 "2007-05-12T01:05:39Z")

</div>

Hi all,

I need to upload a large set of data (the cvs file is about 70MB) into  
some look up tables. Using standard migration api would be tedious. I  
know that there’s a way to export data into YAML file and populate  
data using this file. However, I think this is still heavy.

Is there a way to upload data using a separate sql file in migration?

Thanks

---

<div class="post-metadata">

**Author:** ![Chuong\_Huynh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/3bc359/32.png) [@Chuong\_Huynh](https://www.ruby-forum.com/u/Chuong_Huynh)\
**Post date:** [May 12, 2007, 1:18am UTC](https://www.ruby-forum.com/t/migration-upload-a-large-set-of-data-via-a-separate-sql-fil/103963/2 "2007-05-12T01:18:40Z")

</div>

You can use the approach described here for loading a straight SQL  
file from migrations:

> **[Data Loaders in Migrations](http://revolutiononrails.blogspot.com/2007/02/data-loaders-in-migrations.html)**
>
> Two cases of data usage at RHG (as outlined in the DB Migrations With a Twist post) - migration consolidations and test data creation via ...

Val

---

<div class="post-metadata">

**Author:** ![Chuong\_Huynh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/3bc359/32.png) [@Chuong\_Huynh](https://www.ruby-forum.com/u/Chuong_Huynh)\
**Post date:** [May 12, 2007, 8:13pm UTC](https://www.ruby-forum.com/t/migration-upload-a-large-set-of-data-via-a-separate-sql-fil/103963/3 "2007-05-12T20:13:14Z")

</div>

On May 11, 2007, at 7:04 PM, Chuong Huynh wrote:

> Thanks  
> This is directly from one of my migrations:

# Get all that research data loaded!

def self.up  
say\_with\_time(“Create Tables and Indexes…”)  
{ create\_tables\_and\_indexes }  
say “Reset all models”; reset\_all\_models  
say\_with\_time(“Import all data…”)  
{ import\_data }  
end

The import\_data method used FasterCSV to load a 1.6Mb CSV file into  
three tables with ActiveRecord models. The thing to remember is that  
you can do _anything_ in your migrations. If the data were already  
in an SQL file, use the Revolution Health approach. If your data is  
in CSV, use FasterCSV. The “migration api” only deals directly with  
the schema changes.

Do what makes sense in your situation.

-Rob

Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com)  
removed\_email\_address@domain.invalid