Best Practices; Deploying Updates

Hello,

So I am a running a few shopping carts using spree (open source rails
cart) and some of my clients site traffic is starting to rise.
Normally I do deploy updates late at night but some times I have to
deploy the asap, while customers are still on the site in the process
of ordering. Obviously when I deploy updates it takes the site
offline for about 15 seconds, which will cause all customers to leave
and never return. Is there a way to deploy an update without having
the site go offline?

I am running nginx and thin, and I deploy with just a simple bash
script. The only thing I was thinking of doing is to have multiple
instances of the app running and shut down each instance one at a
time, would that work?

Any advice in the right direction would be greatly appreciated

On Jan 28, 2010, at 11:52 AM, Dan P. wrote:

I am running nginx and thin, and I deploy with just a simple bash
script. The only thing I was thinking of doing is to have multiple
instances of the app running and shut down each instance one at a
time, would that work?

Any advice in the right direction would be greatly appreciated

Look into capistrano or one of the other deploy systems out there.
Capistrano will deploy the new code into a new directory (while your
site stays up), then move a symlink, and restart the app. So downtime
is very minimal. It’s not perfect in that if you run migrations
against the old code weird things could happen, but it’s still very
nice.

You may also want to look into Unicorn, which can give you no-downtime
deploys.

On Jan 28, 7:52 pm, Dan P. [email protected] wrote:

I am running nginx and thin, and I deploy with just a simple bash
script. The only thing I was thinking of doing is to have multiple
instances of the app running and shut down each instance one at a
time, would that work?

It can if your load balancer is smart enough to notice that is trying
to send a request to a dead thin instance and instead send it to
another one. (haproxy handles this pretty nicely, or something like
http://www.linuxvirtualserver.org/ to balance between several
servers).

you take your first server (or first set of thin instances) down,
upgrade them and then do the same on the remaining server/instances.

This is not always trouble free - for example your new code might
populate tables with data that the old instances still running don’t
understand, a migration might add a column that the old code won’t
populate properly etc - I typically only use this to roll out small
fixes outside of our normal maintenance period.

Fred

Regardless of the technology, this is what I would do:

Post maintenance notices on the site upfront. This way the customers
know what to expect and will not be disappointed if they try accessing
the site, while you are updating it, and if they encounter errors.

Communicating and talking to the customers helps.

If you were going to take only 15 seconds, say it will take 5 minutes.
When done, remove the update notice and say, “It is back. Thank you
for your patience” or something on those lines.

Surya Avantsa

Simone R. wrote:
[…]

Another nice-to-have is a minimal static html website to use if the
database connection is not working (if the connection fail just redirect
to the static pages)

FWIW, Capistrano can do this for you.

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]

suryava wrote:

Regardless of the technology, this is what I would do:

Post maintenance notices on the site upfront. This way the customers
know what to expect and will not be disappointed if they try accessing
the site, while you are updating it, and if they encounter errors.

Communicating and talking to the customers helps.

If you were going to take only 15 seconds, say it will take 5 minutes.
When done, remove the update notice and say, “It is back. Thank you
for your patience” or something on those lines.

Surya Avantsa

Hi, I agree with Surya,

also you can use more similar page for a graceful degradation on the
website if there are some problem connecting to other services (like a
credit card payment services) so your customers can create and save the
carts and will not be scared by an "OOPS your payment is lost " thing

Another nice-to-have is a minimal static html website to use if the
database connection is not working (if the connection fail just redirect
to the static pages)

Niels Meersschaert wrote:

I know of capistrano’s web:disable, but not an automatic switch if db
connection can’t be made. Any link for how to set that up?

No, I was thinking of web:disable. Sorry.

Niels

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]

I know of capistrano’s web:disable, but not an automatic switch if db
connection can’t be made. Any link for how to set that up?

Niels

Hi Dan,

Depends on the needs/functionality of your app, but the two things I
usually incorporate in my web apps to address production environment
upgrades/maintenance are:

  1. the ability to render a real-time system-wide message to all
    current users of the app; and

  2. the ability to toggle on/off read-only mode for the app in real-
    time.

To do this I’ll usually have a couple of before_filters in ./app/
controllers/application.rb: one that looks for a tmp text file that
contains the particular sys_wide msg that you want to display to all
current users of the app; and another tmp text file that indicates
whether the app should be put into read-only mode.

If the app sees that there’s a sys_wide msg to be displayed, then it
reads the contents of that sys_wide msg and renders it (via a partial
called from the app’s layouts, displayed prominently at the top of
every screen of the app). Similarly, if the app sees that the app
should be put into read-only mode, then the app displays another
sys_wide msg about the app being in read_only mode, and disallows all
requests other than GETs (usually redirecting POSTs, PUTs, … to the
prev refer’d GET req with a warning msg about being in read-only
mode).

So, my typical prod env upgrade/maintenance process is: put up a
sys_wide msg ahead of time to let current users know about the
upcoming/planned update (usually with the time that it’s planned to
occur and the est time that it should take to complete); then just
before the update, I’ll put the app in read-only mode (to disallow
data mods to the app during the update); then once the update is
completed, whether it required an app restart or not, I turn off read-
only mode, and put up a sys_wide msg about the update having been
completed (which is itself taken down after a brief period of time).

Jeff

Wow, thanks for all the replies I greatly appreciate it. Looks like I
got some reading to do, I have found some capistrano scripts that seem
pretty promising. And unicorn also seems really cool, github is using
them and I just read their article on how they use it and why it is
better.

I will have to play around with a few of them to see which one better
suits me. Thanks again!