Migrating Rails App to New Host/IP with No Downtime?

Hello Rails Community,

In the spirit of being agile with your Rails deployment and scaling,
can you recommend a process for moving your application to another
hosting provider where a new IP address will be assigned?

A wep app such as a Radiant CMS may be ok with a short downtime in the
early morning, but a paid SAS service where users call a Restful API
would not enjoy any downtime during a migration.

I’m in search of a solution that provides:

  1. Little to no downtime to customers.
  2. Considers Internet DNS propogation time for finding the new host
    IP.
  3. Considers database migration.

I’m not looking for a step-by-step, command line answer, only a higher-
level point of view that can point me in the proper direction.

Thank you.

-Jim

Thank you Hassan for your input. I image the old site would remain up
for a few days as current users my still have old DNS caches pointing
to it. I have not decided between MySQL or Postgres. It sounds like
DB Replication is a must in developing a good practice for a safe
migration to new servers?

-Jim

On Oct 12, 2:37 pm, Hassan S. [email protected]

  1. Considers Internet DNS propogation time for finding the new host
    IP.

With regard to this particular point, I can say that my experience in
doing exactly what you are asking about has shown that there are quite a
few ISPs that simply ignore TTLs below some minimum value. For example,
when I have done moves as you mention, we always start by cranking TTLs
down to 5 minutes, a week before the scheduled move date. At the move
time, we turn up the new site, put the old site into “maintenance mode”
(showing our standard “please excuse our dust while we do something
behind the scenes” screen), then edit the DNS entries to point to the
new IPs.

In theory, this should ensure no more than 5 minutes of unreachability
for anyone, and, no worries about DB replication, because the old site
is not accepting any new input.

In practice, I have watched in amusement and disgust as requests have
continued to appear on the old IPs for a week or more. My guess,
although unverified, is that some caching nameservers (probably
belonging to ISPs) simply ignore TTLs. If memory serves correctly, it
seemed like most of the requests were coming from Asia, so, maybe some
ISPs there impose some minimum TTL value to throttle trans-oceanic DNS
requests.

Don’t know for sure, but, that’s my experience, on more than one
occasion, so, thought I’d throw that out there.

Best,

Danny

On Mon, Oct 12, 2009 at 11:24 AM, JimCifarelli [email protected]
wrote:

In the spirit of being agile with your Rails deployment and scaling,
can you recommend a process for moving your application to another
hosting provider where a new IP address will be assigned?

I’m in search of a solution that provides:

  1. Little to no downtime to customers.
  2. Considers Internet DNS propogation time for finding the new host
    IP.
  3. Considers database migration.

I’d think #3 is the only question – you can certainly have both sites
running simultaneously, providing zero apparent downtime.

How you handle replication of data depends on the details of your
app, what DB you’re using, etc.

FWIW,

Hassan S. ------------------------ [email protected]
twitter: @hassan

On Mon, Oct 12, 2009 at 11:55 AM, JimCifarelli [email protected]
wrote:

I have not decided between MySQL or Postgres. It sounds like
DB Replication is a must in developing a good practice for a safe
migration to new servers?

I would think if there’s the possibility of users continuing to hit the
old system for some time – and I’ve seen just ridiculously long DNS
caching in the past – that you would need something like a circular
(master <-> master) replication to keep primary keys from getting
reused. But that may depend on your ratio of writes to reads, etc.

If this were my project, I would set up a comparable configuration
with pair of test machines/VMWare instances and experiment with
different DB replication types and timing.

HTH,

Hassan S. ------------------------ [email protected]
twitter: @hassan

Thank you Danny and Hassan. Your experience here is just what I was
seeking. Great advice.

-Jim

On Oct 12, 7:24 pm, JimCifarelli [email protected] wrote:
.

I’m in search of a solution that provides:

  1. Little to no downtime to customers.
  2. Considers Internet DNS propogation time for finding the new host
    IP.
  3. Considers database migration.

I’m not looking for a step-by-step, command line answer, only a higher-
level point of view that can point me in the proper direction.

You’ve had some good answers, here are some other ones:

  • register a new domain new.foo.com and point it at where your new
    hosting will be
  • setup database replication between your 2 sites
  • turn off mongrels on old site
  • stop replication
  • turn on the mongrels on new site
  • install a redirect on old.foo.com that redirects people to
    new.foo.com
  • update old.foo.com to point to new.foo.com

You can lose the database replication step at the expense of some time
by just dumping the db to a file and copying it over once you take
your old site down (or do stuff like directly copy the database’s
files) - how practical that is depends on the size of your data.

We’re doing something similar at work at the moment.

Option two is almost the same as option 1 except that instead of doing
a redirect from old.foo.com you actually proxy traffic between the two
(so unlike option one, users never see new.foo.com in their address
bar)

Fred

Thank you Fred for your response.

  1. I like the use of the “temporary” new.foo.com domain name and
    redirect.
    Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
    RestClient will handle 302 Redirects?
    I’m concerned that user’s “posts” to the application will fail when
    confronted with a redirect.

  2. Great idea on the transparent forward proxy. I’ll look into this.

  3. My end goals include phasing out the old server (and stop paying
    for it)…just keeping it running long enough to validate users
    ability to transition to the new server without issue. For a fair
    amount of time, I can also monitor for continued API use of the old
    server and perhaps warn valid user attempts of their DNS issues.

Thank you again Fred for all the time and quality answers you
contribute to this community.

-Jim

On Oct 13, 4:25 am, Frederick C. [email protected]

On Oct 13, 5:00 pm, JimCifarelli [email protected] wrote:

Thank you Fred for your response.

  1. I like the use of the “temporary” new.foo.com domain name and
    redirect.
    Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
    RestClient will handle 302 Redirects?
    I’m concerned that user’s “posts” to the application will fail when
    confronted with a redirect.

I’m not sure. It’s certainly possible that some of your api clients
wouldn’t understand them

Fred

On Tue, Oct 13, 2009 at 10:17 AM, Frederick C.
[email protected] wrote:

  1. I like the use of the “temporary” new.foo.com domain name and
    redirect.
    Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
    RestClient will handle 302 Redirects?
    I’m concerned that user’s “posts” to the application will fail when
    confronted with a redirect.

I’m not sure. It’s certainly possible that some of your api clients
wouldn’t understand them

Yeah, redirects work great with human-oriented browser clients, but
I’d be extremely surprised to see them work with an API client even
for a GET, and I’d bet not at all for POST data.

Using a proxy is probably a better idea if you want to avoid the DB
circular replication complexity.


Hassan S. ------------------------ [email protected]
twitter: @hassan

Thanks Fred and Hassan.

For anyone following this thread, here’ a tidy means for performing
the proxy with Apache:
http://blog.vacs.fr/index.php?post/2009/01/25/21-transparent-web-server-migration-with-apache-proxy

I’m not sure I’ll want to bring services down in order to dump/restore
the production db onto the new server, so I’m investigating RubyRep
for an easy method of db replication. They have a short screen cast
to watch if interested (http://www.rubyrep.org/)

So we’re down to just a few “easy” steps:

  1. A week ahead of the scheduled migration, throttle down the TTL for
    your DNS entry
  2. Get new server up and running with db/application/.etc (production
    ready)
  3. Install/Configure/Execute RubyRep on old server to replicate
    production data to the new server
  4. Restart old server’s apache with a transparent proxy configuration
    to new server’s IP
  5. Update DNS to point to new server’s IP address with restored 3600,
    7200 sec TTL
  6. Monitor old server’s use over a few days unil comfortable with
    shutting it down.

Possible Flaws:

  1. A user flip flops between servers and the replication to the new
    server hasn’t completed. Assuming that a user hits the new server
    with their DNS resolved to the new IP, I can’t see that they again hit
    the old server.
  2. RubyRep having any trouble or big latency working with a slave
    server outside it’s local network.

Thanks again.
-Jim

On Oct 13, 1:43 pm, Hassan S. [email protected]

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs