# Best way to convert a legacy database that's full of data?

**URL:** <https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497>\
**Category:** Rails\
**Created:** [May 30, 2007, 2:17pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497 "2007-05-30T14:17:34Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 30, 2007, 2:17pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/1 "2007-05-30T14:17:34Z")

</div>

I need to convert a legacy database - the usual non-Railish schema with  
several million rows - to a Rails-compatible schema.

Are there any tools / tutorials / books / blog-posts that would help me  
with that?

I need to solve problems like connecting to two databases at the same  
time, querying the old non-Railish schema and using the data to create  
Rails objects, etc.

Thanks in advance,  
Maurice B. Gladwell  
A recent Rails convert

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 30, 2007, 2:36pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/2 "2007-05-30T14:36:44Z")

</div>

Maurice,

Which DBMS ?

Each DBMS has different facilities available and different capabilities,  
especially when it comes to linking two databases together. If you are  
migrating everything, you’d probably be a lot easier putting it all into  
one  
database which will save you have to use “federated” support to join  
across  
databases.

You also have a chance to think whether you want to change DBMS.

Are you talking about modifying the structures ?

If so, the book “Refactoring Databases : Evolutionary Database Design”  
by  
Scott Ambler and Pramodkumar J Sadalage (Addison-Wesley, ISBN-10:  
0321293533,  
ISBN-13: 978-0321293534) is a great primer for the types of changes you  
will  
have to make.

The biggest challenge you are going to face is making modifications  
while not  
breaking existing applications (I assume you have existing apps).

If I can be any help then drop me a line : especially if you wish to  
consider  
DB2 as the DBMS of choice going forward. DB2 has good Rails support,  
and  
getting better by the day, and the free DB2 Express-C has no limit on  
database size, only on memory (4 gig) and processors (2 sockets, not  
cores).

HTH

Phil  
(removed\_email\_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 30, 2007, 2:42pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/3 "2007-05-30T14:42:46Z")

</div>

The Rails recipes from the Pragmatic Programmers by Chad Flower, has  
some tips on connecting to legacy databases.  
That should point you in the right direction, or at least give you  
some idea of what you’re up against

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 30, 2007, 3:17pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/4 "2007-05-30T15:17:00Z")

</div>

Hi Phil,

The DBMS is MySQL 5. It is serving a single, high-traffic web  
application.

My naive approach was much simpler than what you outlined - I was  
thinking about writing a conversion script, that will:

1. go over the legacy database, pulling data from it

2. use data pulled in #1 to instantiate ActiveRecord objects, which will  
be compatible with the new schema of the new database

3. save those ActiveRecord objects to the new database

Does that make sense? I thought it would be the most conventional  
approach, so there must be material describing such a procedure.

–  
M.

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 30, 2007, 3:18pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/5 "2007-05-30T15:18:50Z")

</div>

You might also want to look at the ETL (Extract, Transform, Load)  
approach found in active\_warehouse. If you can reduce your dependency  
to cyclical connectivity (say daily or hourly), you can simplify the  
process quite a bit. Or, if you could ETL the legacy data and just  
maintain connectivity for the sake of a few tables, you’ve reduced the  
runtime load and complexity of your system quite a bit.

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 31, 2007, 12:58pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/6 "2007-05-31T12:58:55Z")

</div>

How are you connecting to Access? I hope this isn’t too OT, but there  
are a lot of threads in this and other forums saying it’s a mess/not  
worth it.

Ron

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 31, 2007, 2:28pm UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/7 "2007-05-31T14:28:09Z")

</div>

Re connecting to multiple databases. Bookmarked this ‘how to’ earlier  
today anticipating a similar need:

[http://www.railsonwave.com/railsonwave/2006/12/11/multiple-database-handlng-with-rails](http://www.railsonwave.com/railsonwave/2006/12/11/multiple-database-handlng-with-rails)

On May 30, 8:17 am, Maurice G. \<rails-mailing-l…@andreas-

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [May 31, 2007, 11:36am UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/8 "2007-05-31T11:36:48Z")

</div>

Hi Maurice,

On May 30, 2:17 pm, Maurice G. \<rails-mailing-l…@andreas-  
[s.net](http://s.net)\> wrote:

> Does that make sense? I thought it would be the most conventional  
> approach, so there must be material describing such a procedure.

## This is exactly how we’re doing it for one of our clients. We’ve created AR objects for all the old tables, and then we’re doing a find on the old AR object, and looping over the result, to stick it into the new AR models. Something like this:

## class ImportBibData \< ActiveRecord::Base establish\_connection :import set\_table\_name “BibData” end

## Then doing something like this…

## i\_bib\_items = ImportBibItem.find\_all\_by\_doctype(“M”) i\_bib\_items.each do |i\_bib\_item| new\_object = NewObject.find\_or\_create\_by\_import\_id(i\_bib\_item.bibid) end

It’s not DRY at all, and it’s quite a bit of a mess - but it’s going  
to be more or less a one-off import once the client freezes their old  
(Access!!) DB and we start the import.

Hope that’s of any help.

Cheers,

JS

PS, my first post, so have mercy with me if I’m doing something wrong!  
PPS, Hi all!

---

<div class="post-metadata">

**Author:** ![maurice-gladwell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@maurice-gladwell](https://www.ruby-forum.com/u/maurice-gladwell)\
**Post date:** [June 1, 2007, 10:35am UTC](https://www.ruby-forum.com/t/best-way-to-convert-a-legacy-database-thats-full-of-data/105497/9 "2007-06-01T10:35:24Z")

</div>

On May 31, 11:57 am, paron [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> How are you connecting to Access? I hope this isn’t too OT, but there  
> are a lot of threads in this and other forums saying it’s a mess/not  
> worth it.

It is a mess - so we don’t! 😃 We use the mdb-tools package to extract  
the data as CSV, and the schema as SQL. We then convert the schema to  
MySQL compatible SQL, and dump it into MySQL. From there we run a  
mysqlimport on the CSV files to populate the DB with the old data.

It’s a mess for sure, but once it’s in the MySQL DB it’s much nicer to  
work with, and once it’s in the new structure we should be all happy  
smiling Railers…

/Jocke