# Handling Large Data Imports

**URL:** <https://www.ruby-forum.com/t/handling-large-data-imports/106138>\
**Category:** Deployment\
**Created:** [June 6, 2007, 8:16pm UTC](https://www.ruby-forum.com/t/handling-large-data-imports/106138 "2007-06-06T20:16:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sjames](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@sjames](https://www.ruby-forum.com/u/sjames)\
**Post date:** [June 6, 2007, 8:16pm UTC](https://www.ruby-forum.com/t/handling-large-data-imports/106138/1 "2007-06-06T20:16:23Z")

</div>

Greetings.

Have any of you guys dealt with importing large amounts of data on  
deploy? I have two distinct situations I need to deal with and I was  
hoping for some advice.

1. I have a fairly large CSV file of postal codes that seems to  
import very well with the MySQL import utility (mysqlimport) on  
development machines but very poorly when used remotely with  
Capistrano (i.e. during initial migration). How are you guys handling  
such data import on initial migration?

2. I’m going to have to move over a rather large amount of data from  
an older version of the site I’m updating (old site is written in ASP  
using a MS SQL database – it’s a mess). I know I can write scripts  
to massage data, but something I’m unsure of is how to actually _move_  
the data from the old servers to the new servers – neither of which  
I’ll have physical access to. Recommendations?

Thanks!

James H

---

<div class="post-metadata">

**Author:** ![sjames](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@sjames](https://www.ruby-forum.com/u/sjames)\
**Post date:** [June 6, 2007, 8:17pm UTC](https://www.ruby-forum.com/t/handling-large-data-imports/106138/2 "2007-06-06T20:17:23Z")

</div>

Can you give some performance #s on the remote side? Is that remote  
database under load during the import?

–

Jesse P.  
Blue Box Group, LLC

p. +1.800.613.4305 x801  
e. removed\_email\_address@domain.invalid

---

<div class="post-metadata">

**Author:** ![sjames](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@sjames](https://www.ruby-forum.com/u/sjames)\
**Post date:** [June 7, 2007, 8:43am UTC](https://www.ruby-forum.com/t/handling-large-data-imports/106138/3 "2007-06-07T08:43:26Z")

</div>

Hi James,

Different situations require different techniques.

if you are looking for ruby based dumping use fastercsv library

or plainly you can use mysqldump of the table or db itself from the  
development DB itself and upload and dump it into production server.

I have done it few times myself with a million+ records

these scripts do take significant amount of time and we do it only  
once in a while so I wont suggest CAP as script may timeout

regards  
Senthil