Rails 2.1 and migrations

Rails Engines Development
1

Hi,

I thought I’d post here as the ticket relating to this issue[1] was
getting a little cluttered with comments.

I’m trying to stimulate some debate as to the best way to proceed with
regard to how engines handles migrations with Rails 2.1 and beyond as
I don’t think the current 2.1 release is ideal - it doesn’t handle
timestamped migrations in plugins.

Inspired by Azimux’s fork[2], I’ve made an attempt at a solution[3]
that ditches the plugin_schema_info table altogether and relies on the
Rails migrator to migrate plugins up and down. This works great if you
are starting with a fresh app and/or have timestamped migrations in
your plugins. It’s not so great if you are upgrading an existing app
to 2.1 or your plugins have sequential (and potentially clashing)
migration version numbers.

Now the questions are:

How bad would it be to break compatibility?
Can a rake task/automatic process (ala Rails 2.1) be created to offer
a painless upgrade?
Can we have a tag for a ‘legacy’ version, with the timestamp ready
version as head?

I don’t think there are any simple answers here, but I do feel
strongly that engines should keep parity with Rails and start using
timestamped migrations.

Sven’s fork[4] and Samuel’s patch[5] both offer alternative solutions
which use a plugin_schema_migrations table to track applied
migrations, but both suffer from the same problem - they don’t
distinguish between plugins when checking if migrations have been
applied so don’t behave correctly if plugins have migrations with
clashing version numbers.

My feeling is that going down the route Sven and Samuel have proposed
is probably the easiest way to maintain backwards compatibility while
supporting timestamped migrations. It would also be relatively trivial
to modify the migrator to automatically bootstrap the
plugin_schema_migrations table in the same way Rails does with
schema_info table. Personally though, I think a simplified engines
migration mechanism as in Azimux’s and my fork would be better in the
long term as it overrides less of Rails’ default behaviour.

Thoughts?

Tekin S.

[1]
http://engines.lighthouseapp.com/projects/10178/tickets/17-migrations-do-not-work-beyond-the-initial-migration
[2] GitHub - azimux/engines: The Rails Engines plugin
[3]http://github.com/tekin/engines/tree/master
[4] http://github.com/svenfuchs/engines/commits/master
[5] http://engines.lighthouseapp.com/attachments/32727/engines.diff

2

On 22/07/2008, at 10:30 AM, Tekin S. wrote:

Inspired by Azimux’s fork[2], I’ve made an attempt at a solution[3]
that ditches the plugin_schema_info table altogether and relies on
the Rails migrator to migrate plugins up and down. This works great
if you are starting with a fresh app and/or have timestamped
migrations in your plugins. It’s not so great if you are upgrading
an existing app to 2.1 or your plugins have sequential (and
potentially clashing) migration version numbers.

Thanks for your work. I’m sorry I have to have a negative response to
it, but I’m still thankful you took the time to work on it, because
the more we think about this problem, the better solution we could
come up with;

I don’t think this is a good idea for one fundamental reason - it is
logically broken. It may be very rare that two migrations end up with
the same timestamp, but it could happen. In this case, behavior is
undefined, which I think is very bad. Behavior in this case needs to
be defined.

Picture the following scenario. There are two separate developers
creating Engine A and Engine B. Both have the same time stamped
migration. Who’s responsibility is it for the time stamp to be
changed? Are we going to make all time stamp migrations unique, in the
entire world? Plugins exist in their own namespace - migrations like
these shouldn’t be treated the same - thus it makes sense for the
migration table to have a namespace (“plugin_name” column) associated
with it.

Now the questions are:

How bad would it be to break compatibility?

Very very bad, in my opinion. This is not an option in many cases. At
release, all previous engines would need fixing. Backwards
compatibility is a must for people to take any forward movement
seriously, in my opinion.

Can a rake task/automatic process (ala Rails 2.1) be created to
offer a painless upgrade?

Yes, it would be possible to code this. You can see the rake task in
my patch which does twiddling for the specific requirements of my patch.

Can we have a tag for a ‘legacy’ version, with the timestamp ready
version as head?

What about plugins that use both? An old engine of mine uses both the
old indexed format and timestamped format. Rails 2.1 officially
supports both methods AFAIK (feel free to correct me if wrong).

I don’t think there are any simple answers here, but I do feel
strongly that engines should keep parity with Rails and start using
timestamped migrations.

That’s true - there is no simple answer, however maintaining parity
may not be the right goal. I think fundamentally, the most important
thing is that it works, and that it works in a logical dependable way,
and that it doesn’t have unexpected behavior.

My feeling is that going down the route Sven and Samuel have
Thoughts?
My patch and Sven’s fork both achieve the same thing. I’m still
unclear on whether Sven has addressed fixing the old schema table or
not. My patch includes a rake task for this. This task can be used
when upgrading to rails 2.1. Having automatic magic in the migrator to
bootstrap the schema table may be a good idea, but it might be too
much magic - keeping the rake task as a task and maybe just logging an
error from the migrator with appropriate information might be a better
way to go.

I’m not sure why you think it would be better in the long term.
Plugins are inherently in their own namespace, therefore, in my
opinion, the schema table is just fine.

I think it would be good for you to summarize in what ways you think
that it is better in the long term, this way I can understand more
clearly what you are thinking.

I think from my point of view - relying on Rails core is sometimes
more difficult because it can be a moving target. If we can integrate
the functionality together between engines and rails to the point
where engines can be pulled along with rails, then we have achieved
something great. However, if we integrate the two and are constantly
playing catch up, then this is a difficult situation. At this point,
we are playing catch up to a major feature change - but as I see it,
this isn’t such a bad thing, because the change we are making is
actually a significant feature to engines in its own right - i.e. the
new time stamped migrations is a benefit to engines, not just playing
catch up.

In this regard, any new feature will require time and effort to
integrate cleanly. Even if engines relies on 90% of rails, it will
still require some kind of update when the core changes, for that 10%
which might not quite be right anymore. It is a fine line, but if
engines maintains its own independence where it makes sense, it might
actually end up being less highly coupled and require less
modification when rails changes. Does that make sense? More coupling =
more work generally, less coupling = less work generally. What you are
suggesting appears to be more coupling.

Kind regards,
Samuel

3

2008/7/22 Space Ship T. [email protected]:

methods AFAIK (feel free to correct me if wrong).
It might be worth pointing out that a recent discussion on rails-core
leads
me to believe that in Rails 2.<whatever_comes_next> there’ll be a config
option to let you choose which migration “uniquification” scheme you
want
your app to use for new migrations. So numeric’s aren’t going away.

Cheers

Muz

4

Thanks for your work. I’m sorry I have to have a negative response
to it, but I’m still thankful you took the time to work on it,
because the more we think about this problem, the better solution we
could come up with;

Don’t be sorry, I’m not even sure I agree with myself anymore! As you
say the more we think about it the better.

the entire world? Plugins exist in their own namespace - migrations
like these shouldn’t be treated the same - thus it makes sense for
the migration table to have a namespace (“plugin_name” column)
associated with it.

I agree, although this problem applies to migrations in Rails too! At
any rate, my understanding of engines is that it’s really been
designed for creating plugins for internal use across projects rather
than public consumption. But yes, potential clashes are bad.

What about plugins that use both? An old engine of mine uses both
the old indexed format and timestamped format. Rails 2.1 officially
supports both methods AFAIK (feel free to correct me if wrong).

This is the main reason I’m starting to feel my solution isn’t the way
to go.

My patch and Sven’s fork both achieve the same thing. I’m still
unclear on whether Sven has addressed fixing the old schema table or
not. My patch includes a rake task for this. This task can be used
when upgrading to rails 2.1. Having automatic magic in the migrator
to bootstrap the schema table may be a good idea, but it might be
too much magic - keeping the rake task as a task and maybe just
logging an error from the migrator with appropriate information
might be a better way to go.

I think I agree, but this is exactly the magic that Rails 2.1 performs
when you upgrade.

From my understanding, Sven’s fork only works if you’re starting from
fresh, there is no fixing of the old plugin schema table. One things
it does do though is restrict the filename length of generated plugin
migrations; useful as they can get very long with timestamps in them!

I’m not sure why you think it would be better in the long term.
Plugins are inherently in their own namespace, therefore, in my
opinion, the schema table is just fine.

Having slept on it, I’m not sure I do anymore! I think I was seduced
by the fact it used less code, overrode less of Rails’ functionality
and needed one less table to work.

I think from my point of view - relying on Rails core is sometimes
more difficult because it can be a moving target.

Indeed, as Murray’s post has just proved!!

In this regard, any new feature will require time and effort to
integrate cleanly. Even if engines relies on 90% of rails, it will
still require some kind of update when the core changes, for that
10% which might not quite be right anymore. It is a fine line, but
if engines maintains its own independence where it makes sense, it
might actually end up being less highly coupled and require less
modification when rails changes. Does that make sense? More coupling
= more work generally, less coupling = less work generally. What you
are suggesting appears to be more coupling.

I agree, although for arguments sake, it seems to me that engines has
always strove to be less code and have Rails do more of the work, as
happened when Rails added multiple view paths support. In a way, the
solution in my fork is doing something similar (multiple migration
path support!?) but may ultimately be a bit premature.

Tekin

5

On 22/07/2008, at 9:10 PM, Tekin S. wrote:

From my understanding, Sven’s fork only works if you’re starting
from fresh, there is no fixing of the old plugin schema table. One
things it does do though is restrict the filename length of
generated plugin migrations; useful as they can get very long with
timestamps in them!

In my patch is a rake task which will fix the old migrations table.
Please check it out. I’ve used it on several production sites with no
problems so far.

Kind regards,
Samuel

6

Thanks for all your work and thinking about this issue, guys, and most
of all your patches!

I too am unsure whether or not to follow Rails’ migration behaviour
too closely. I don’t have anything against timestamps, but equally I
feel that the problem they attempt to address is far less likely to be
encountered by plugin developers than application developers (and even
then it was relatively rare and could be solved by communication).

Agreed.

So, I’m hesitant to make significant changes at the moment, until
we’ve reached a better understanding of the implications (thanks for
starting the discussion by the way, Tekin!).

No worries!

Perhaps a useful way to approach this, for the moment and while Rails
itself settles down, is to consider what is currently broken with
plugin migrations. As I understand it, the main issue is that
migrations generated in an application cannot be moved into plugins
because they are named with timestamps. Is that correct?

Yes, partly. The problem is, now a lot of us now have plugins with
timestamped mirgations thanks to Sven’s fork and Samuel’s patch!!

Samuel’s/Sven’s approach gets my vote - update engines so it tracks
each applied migration and then offer a rake task (or even some magic)
that upgrades an existing plugin_schema_info table to a
plugin_schema_migrations table with applied migrations listed. This
would work with both timestamped and sequential migrations.

The only issue I know of with Samuel’s patch is that it calls
ActiveRecord::Migrator.migrate when migrating, which doesn’t scope the
applied migrations check by the plugin name, so it will not play
nicely if your plugins have migration numbers that clash.

7

The only issue I know of with Samuel’s patch is that it calls
ActiveRecord::Migrator.migrate when migrating, which doesn’t scope
the applied migrations check by the plugin name, so it will not play
nicely if your plugins have migration numbers that clash.

I’ll look into it. Thanks

8

On Tue, Jul 22, 2008 at 10:10 AM, Tekin S. [email protected]
wrote:

Thanks for your work. I’m sorry I have to have a negative response to it,
but I’m still thankful you took the time to work on it, because the more we
think about this problem, the better solution we could come up with;

Don’t be sorry, I’m not even sure I agree with myself anymore! As you say
the more we think about it the better.

Thanks for all your work and thinking about this issue, guys, and most
of all your patches!

I too am unsure whether or not to follow Rails’ migration behaviour
too closely. I don’t have anything against timestamps, but equally I
feel that the problem they attempt to address is far less likely to be
encountered by plugin developers than application developers (and even
then it was relatively rare and could be solved by communication).

So, I’m hesitant to make significant changes at the moment, until
we’ve reached a better understanding of the implications (thanks for
starting the discussion by the way, Tekin!).

Perhaps a useful way to approach this, for the moment and while Rails
itself settles down, is to consider what is currently broken with
plugin migrations. As I understand it, the main issue is that
migrations generated in an application cannot be moved into plugins
because they are named with timestamps. Is that correct?

9

Hey guys,

first of all please accept my apologies for ignoring your discussion
for so long. I’m up to my ears in other stuff and did not have any
time to review all of your work.

So although I most certainly won’t be able to actually contribute any
code I’d still like to throw my thoughts onto the table.

My fork was referenced a couple of times. I don’t feel strong about
this implemetation. It was just the easiest thing I found to work and
got it done. I’m sure I missed a couple of issues. (Although I’m
actually already using it.)

I feel strong about Engine migrations being as much aligned to Rails
migrations as possible. Not so much for technical reasons (clashes
aren’t that likely, although still can occur etc.) but more for
reasons like “framework usability” and education. I really don’t want
to wrapp my head around wondering how Engine migrations behave
different from Rails migrations on certain edge cases. And I don’t
want to explain it. It’s hard enough for me to remember how Rails
migrations work with certain conditions given. So, from my point of
view that would be the most important goal: having Engines migrations
transparently work exactly how Rails migrations work.

I personally don’t care too much about backwards compat in this case.
I’d implement things in a way that is “done the right way” for Rails
2.1+ and on top of that maybe have a rake task or whatever tool to
update migrations from old apps.

If we want to support numbered migrations (as Rails still does)
tracking migrations per plugin (in an extra table) is necessary.

On 22.07.2008, at 00:30, Tekin S. wrote:

Inspired by Azimux’s fork[2], I’ve made an attempt at a solution[3]
Can a rake task/automatic process (ala Rails 2.1) be created to
applied migrations, but both suffer from the same problem - they
engines migration mechanism as in Azimux’s and my fork would be
[2] GitHub - azimux/engines: The Rails Engines plugin
[3]http://github.com/tekin/engines/tree/master
[4] http://github.com/svenfuchs/engines/commits/master
[5] http://engines.lighthouseapp.com/attachments/32727/engines.diff

Engine-Developers mailing list
[email protected]
http://lists.rails-engines.org/listinfo.cgi/engine-developers-rails-engines.org

–
sven fuchs [email protected]
artweb design http://www.artweb-design.de
grünberger 65 + 49 (0) 30 - 47 98 69 96 (phone)
d-10245 berlin + 49 (0) 171 - 35 20 38 4 (mobile)

10

My fork was referenced a couple of times. I don’t feel strong about
this implemetation. It was just the easiest thing I found to work
and got it done. I’m sure I missed a couple of issues. (Although I’m
actually already using it.)

The only issue I’ve found was the one previously mentioned about
plugins having clashing migrations not being handled and also that
it’s a little behind James’ master (e.g. exception notification issue).

I personally don’t care too much about backwards compat in this
case. I’d implement things in a way that is “done the right way” for
Rails 2.1+ and on top of that maybe have a rake task or whatever
tool to update migrations from old apps.

With a few tweaks, yours/Samuel’s versions will work as Rails does AND
be backwards compatible, win-win?

If we want to support numbered migrations (as Rails still does)
tracking migrations per plugin (in an extra table) is necessary.

This is the bit that will need fixing, and then a rake task to upgrade
older projects.

Tekin

11

“the more we think about it the better.” very good!

12

Hey Tekin,

On 22.07.2008, at 13:31, Tekin S. wrote:

With a few tweaks, yours/Samuel’s versions will work as Rails does
AND be backwards compatible, win-win?

That’d be awesome.

If we want to support numbered migrations (as Rails still does)
tracking migrations per plugin (in an extra table) is necessary.
This is the bit that will need fixing, and then a rake task to
upgrade older projects.

To me that sounds like the way to go, definitely.

Thanks for your work on this!

–
sven fuchs [email protected]
artweb design http://www.artweb-design.de
grünberger 65 + 49 (0) 30 - 47 98 69 96 (phone)
d-10245 berlin + 49 (0) 171 - 35 20 38 4 (mobile)