In the spirit of being agile with your Rails deployment and scaling,
can you recommend a process for moving your application to another
hosting provider where a new IP address will be assigned?
A wep app such as a Radiant CMS may be ok with a short downtime in the
early morning, but a paid SAS service where users call a Restful API
would not enjoy any downtime during a migration.
I’m in search of a solution that provides:
Little to no downtime to customers.
Considers Internet DNS propogation time for finding the new host
IP.
Considers database migration.
I’m not looking for a step-by-step, command line answer, only a higher-
level point of view that can point me in the proper direction.
Thank you Hassan for your input. I image the old site would remain up
for a few days as current users my still have old DNS caches pointing
to it. I have not decided between MySQL or Postgres. It sounds like
DB Replication is a must in developing a good practice for a safe
migration to new servers?
Considers Internet DNS propogation time for finding the new host
IP.
With regard to this particular point, I can say that my experience in
doing exactly what you are asking about has shown that there are quite a
few ISPs that simply ignore TTLs below some minimum value. For example,
when I have done moves as you mention, we always start by cranking TTLs
down to 5 minutes, a week before the scheduled move date. At the move
time, we turn up the new site, put the old site into “maintenance mode”
(showing our standard “please excuse our dust while we do something
behind the scenes” screen), then edit the DNS entries to point to the
new IPs.
In theory, this should ensure no more than 5 minutes of unreachability
for anyone, and, no worries about DB replication, because the old site
is not accepting any new input.
In practice, I have watched in amusement and disgust as requests have
continued to appear on the old IPs for a week or more. My guess,
although unverified, is that some caching nameservers (probably
belonging to ISPs) simply ignore TTLs. If memory serves correctly, it
seemed like most of the requests were coming from Asia, so, maybe some
ISPs there impose some minimum TTL value to throttle trans-oceanic DNS
requests.
Don’t know for sure, but, that’s my experience, on more than one
occasion, so, thought I’d throw that out there.
On Mon, Oct 12, 2009 at 11:24 AM, JimCifarelli [email protected]
wrote:
In the spirit of being agile with your Rails deployment and scaling,
can you recommend a process for moving your application to another
hosting provider where a new IP address will be assigned?
I’m in search of a solution that provides:
Little to no downtime to customers.
Considers Internet DNS propogation time for finding the new host
IP.
Considers database migration.
I’d think #3 is the only question – you can certainly have both sites
running simultaneously, providing zero apparent downtime.
How you handle replication of data depends on the details of your
app, what DB you’re using, etc.
On Mon, Oct 12, 2009 at 11:55 AM, JimCifarelli [email protected]
wrote:
I have not decided between MySQL or Postgres. It sounds like
DB Replication is a must in developing a good practice for a safe
migration to new servers?
I would think if there’s the possibility of users continuing to hit the
old system for some time – and I’ve seen just ridiculously long DNS
caching in the past – that you would need something like a circular
(master ↔ master) replication to keep primary keys from getting
reused. But that may depend on your ratio of writes to reads, etc.
If this were my project, I would set up a comparable configuration
with pair of test machines/VMWare instances and experiment with
different DB replication types and timing.
You can lose the database replication step at the expense of some time
by just dumping the db to a file and copying it over once you take
your old site down (or do stuff like directly copy the database’s
files) - how practical that is depends on the size of your data.
We’re doing something similar at work at the moment.
Option two is almost the same as option 1 except that instead of doing
a redirect from old.foo.com you actually proxy traffic between the two
(so unlike option one, users never see new.foo.com in their address
bar)
I like the use of the “temporary” new.foo.com domain name and
redirect.
Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
RestClient will handle 302 Redirects?
I’m concerned that user’s “posts” to the application will fail when
confronted with a redirect.
Great idea on the transparent forward proxy. I’ll look into this.
My end goals include phasing out the old server (and stop paying
for it)…just keeping it running long enough to validate users
ability to transition to the new server without issue. For a fair
amount of time, I can also monitor for continued API use of the old
server and perhaps warn valid user attempts of their DNS issues.
Thank you again Fred for all the time and quality answers you
contribute to this community.
I like the use of the “temporary” new.foo.com domain name and
redirect.
Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
RestClient will handle 302 Redirects?
I’m concerned that user’s “posts” to the application will fail when
confronted with a redirect.
I’m not sure. It’s certainly possible that some of your api clients
wouldn’t understand them
On Tue, Oct 13, 2009 at 10:17 AM, Frederick C. [email protected] wrote:
I like the use of the “temporary” new.foo.com domain name and
redirect.
Do you know how well Ruby libraries like Net::HTTP, ActiveResource, or
RestClient will handle 302 Redirects?
I’m concerned that user’s “posts” to the application will fail when
confronted with a redirect.
I’m not sure. It’s certainly possible that some of your api clients
wouldn’t understand them
Yeah, redirects work great with human-oriented browser clients, but
I’d be extremely surprised to see them work with an API client even
for a GET, and I’d bet not at all for POST data.
Using a proxy is probably a better idea if you want to avoid the DB
circular replication complexity.
For anyone following this thread, here’ a tidy means for performing
the proxy with Apache:
I’m not sure I’ll want to bring services down in order to dump/restore
the production db onto the new server, so I’m investigating RubyRep
for an easy method of db replication. They have a short screen cast
to watch if interested (http://www.rubyrep.org/)
So we’re down to just a few “easy” steps:
A week ahead of the scheduled migration, throttle down the TTL for
your DNS entry
Get new server up and running with db/application/.etc (production
ready)
Install/Configure/Execute RubyRep on old server to replicate
production data to the new server
Restart old server’s apache with a transparent proxy configuration
to new server’s IP
Update DNS to point to new server’s IP address with restored 3600,
7200 sec TTL
Monitor old server’s use over a few days unil comfortable with
shutting it down.
Possible Flaws:
A user flip flops between servers and the replication to the new
server hasn’t completed. Assuming that a user hits the new server
with their DNS resolved to the new IP, I can’t see that they again hit
the old server.
RubyRep having any trouble or big latency working with a slave
server outside it’s local network.