# Newb: load lots of data into database **URL:** \ **Category:** Rails\ **Created:** [August 21, 2007, 6:13am UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591 "2007-08-21T06:13:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ktheory](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e480ec/32.png) [@ktheory](https://www.ruby-forum.com/u/ktheory)\ **Post date:** [August 21, 2007, 6:13am UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591/1 "2007-08-21T06:13:58Z")
Hello! I’m just starting to venture into Ruby on Rails, and I was wondering what the best way to populate my databases with exisiting data in the form of csv files that don’t map exactly to the columns (ie require some kind of manipulation/data cleansing). When I used PHP I would just write a little PHP or Perl script to load directly into a MySQL table, but I was wondering if RoR had a way of abstracting that and I was wondering what the corresponding Ruby code might look like… Thanks! ---
**Author:** ![ktheory](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e480ec/32.png) [@ktheory](https://www.ruby-forum.com/u/ktheory)\ **Post date:** [August 21, 2007, 6:18am UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591/2 "2007-08-21T06:18:10Z")
Rails doesn’t have a specific method for dealing with this, but you might write Ruby scripts to manipulate the data and insert the manipulated data into the DB. Depending on your data you may be able to use ActiveRecord in your scripts as well. -faisal ---
**Author:** ![ktheory](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e480ec/32.png) [@ktheory](https://www.ruby-forum.com/u/ktheory)\ **Post date:** [August 21, 2007, 8:16am UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591/3 "2007-08-21T08:16:17Z")
On Aug 20, 9:13 pm, Aaron S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > might look like… > > Thanks! LOAD DATA INFILE is pretty flexible. ISTR that it can take CSV’s directly, except that Excel would put in random double quotes around certain cells, and they needed to be cleaned out, if that’s where data’s from [http://dev.mysql.com/doc/refman/5.0/en/load-data.html](http://dev.mysql.com/doc/refman/5.0/en/load-data.html) ---
**Author:** ![ktheory](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e480ec/32.png) [@ktheory](https://www.ruby-forum.com/u/ktheory)\ **Post date:** [August 21, 2007, 3:30pm UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591/4 "2007-08-21T15:30:30Z")
If you can clean up the data, perhaps this CSV importer plugin might be useful: [http://www.stephenbartholomew.co.uk/2007/7/20/csv-importer-for-rails](http://www.stephenbartholomew.co.uk/2007/7/20/csv-importer-for-rails) Regards, Chris ---
**Author:** ![ktheory](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e480ec/32.png) [@ktheory](https://www.ruby-forum.com/u/ktheory)\ **Post date:** [August 21, 2007, 8:51pm UTC](https://www.ruby-forum.com/t/newb-load-lots-of-data-into-database/112591/5 "2007-08-21T20:51:40Z")
This is similar to a problem I had normalizing data for neural network training. Ruby is very good at handling csv data and you can use ActiveRecord outside of rails. My recommendation to you would be to write a script to read in the data (File and IO are the primary classes), using ActiveRecord. Using active record would allow you to use your object models to insert the data into the data base. If you have a lot of data, and going model by model would be too slow, you will want to look at the DBI interface. I don’t know of a good Ruby SQL loader. [http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/166724](http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/166724) [http://rails.aizatto.com/2007/05/21/activerecord-without-rails/](http://rails.aizatto.com/2007/05/21/activerecord-without-rails/)