Best way to convert a legacy database that's full of data?

Rails
1

I need to convert a legacy database - the usual non-Railish schema with
several million rows - to a Rails-compatible schema.

Are there any tools / tutorials / books / blog-posts that would help me
with that?

I need to solve problems like connecting to two databases at the same
time, querying the old non-Railish schema and using the data to create
Rails objects, etc.

Thanks in advance,
Maurice B. Gladwell
A recent Rails convert

2

Maurice,

Which DBMS ?

Each DBMS has different facilities available and different capabilities,
especially when it comes to linking two databases together. If you are
migrating everything, you’d probably be a lot easier putting it all into
one
database which will save you have to use “federated” support to join
across
databases.

You also have a chance to think whether you want to change DBMS.

Are you talking about modifying the structures ?

If so, the book “Refactoring Databases : Evolutionary Database Design”
by
Scott Ambler and Pramodkumar J Sadalage (Addison-Wesley, ISBN-10:
0321293533,
ISBN-13: 978-0321293534) is a great primer for the types of changes you
will
have to make.

The biggest challenge you are going to face is making modifications
while not
breaking existing applications (I assume you have existing apps).

If I can be any help then drop me a line : especially if you wish to
consider
DB2 as the DBMS of choice going forward. DB2 has good Rails support,
and
getting better by the day, and the free DB2 Express-C has no limit on
database size, only on memory (4 gig) and processors (2 sockets, not
cores).

HTH

Phil
([email protected])

3

The Rails recipes from the Pragmatic Programmers by Chad Flower, has
some tips on connecting to legacy databases.
That should point you in the right direction, or at least give you
some idea of what you’re up against

4

Hi Phil,

The DBMS is MySQL 5. It is serving a single, high-traffic web
application.

My naive approach was much simpler than what you outlined - I was
thinking about writing a conversion script, that will:

  1. go over the legacy database, pulling data from it

  2. use data pulled in #1 to instantiate ActiveRecord objects, which will
    be compatible with the new schema of the new database

  3. save those ActiveRecord objects to the new database

Does that make sense? I thought it would be the most conventional
approach, so there must be material describing such a procedure.

–
M.

5

You might also want to look at the ETL (Extract, Transform, Load)
approach found in active_warehouse. If you can reduce your dependency
to cyclical connectivity (say daily or hourly), you can simplify the
process quite a bit. Or, if you could ETL the legacy data and just
maintain connectivity for the sake of a few tables, you’ve reduced the
runtime load and complexity of your system quite a bit.

6

How are you connecting to Access? I hope this isn’t too OT, but there
are a lot of threads in this and other forums saying it’s a mess/not
worth it.

Ron

7

Re connecting to multiple databases. Bookmarked this ‘how to’ earlier
today anticipating a similar need:

http://www.railsonwave.com/railsonwave/2006/12/11/multiple-database-handlng-with-rails

On May 30, 8:17 am, Maurice G. <rails-mailing-l…@andreas-

8

Hi Maurice,

On May 30, 2:17 pm, Maurice G. <rails-mailing-l…@andreas-
s.net> wrote:

Does that make sense? I thought it would be the most conventional
approach, so there must be material describing such a procedure.

This is exactly how we’re doing it for one of our clients. We’ve
created AR objects for all the old tables, and then we’re doing a find
on the old AR object, and looping over the result, to stick it into
the new AR models.
Something like this:

class ImportBibData < ActiveRecord::Base
establish_connection :import
set_table_name “BibData”
end

Then doing something like this…

i_bib_items = ImportBibItem.find_all_by_doctype(“M”)
i_bib_items.each do |i_bib_item|
new_object =
NewObject.find_or_create_by_import_id(i_bib_item.bibid)
end

It’s not DRY at all, and it’s quite a bit of a mess - but it’s going
to be more or less a one-off import once the client freezes their old
(Access!!) DB and we start the import.

Hope that’s of any help.

Cheers,

JS

PS, my first post, so have mercy with me if I’m doing something wrong!
PPS, Hi all!

9

On May 31, 11:57 am, paron [email protected] wrote:

How are you connecting to Access? I hope this isn’t too OT, but there
are a lot of threads in this and other forums saying it’s a mess/not
worth it.

It is a mess - so we don’t! :smiley: We use the mdb-tools package to extract
the data as CSV, and the schema as SQL. We then convert the schema to
MySQL compatible SQL, and dump it into MySQL. From there we run a
mysqlimport on the CSV files to populate the DB with the old data.

It’s a mess for sure, but once it’s in the MySQL DB it’s much nicer to
work with, and once it’s in the new structure we should be all happy
smiling Railers…

/Jocke