Compiling Nginx on production. How to do it without down time

aris · October 12, 2012, 2:09am

Hi all,

I run vanilla builds of Nginx from the Nginx centos repo.

But recently Ive started to realize the need for additional modules in
Nginx like Nginx status, openresty, lua etc.

Now this will mean that my vanilla Nginx with

nginx version: nginx/1.0.12

built by gcc 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC)

TLS SNI support enabled

configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
–lock-path=/var/run/nginx.lock
–http-client-body-temp-path=/var/cache/nginx/client_temp
–http-proxy-temp-path=/var/cache/nginx/proxy_temp
–http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
–http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
–http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
–group=nginx
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_sub_module
–with-http_dav_module
–with-http_flv_module --with-http_mp4_module
–with-http_gzip_static_module --with-http_random_index_module
–with-http_secure_link_module --with-http_stub_status_module
–with-mail
–with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt=’-O2
-g’

will have to be recompiled.

How can I do it with minimal downtime on my prod machine?

Quintin_P · October 12, 2012, 2:27am

On Thu, Oct 11, 2012 at 05:08:55PM -0700, Quintin P. wrote:

Hi there,

How can I do it with minimal downtime on my prod machine?

Controlling nginx ?

f

Francis D. [email protected]

Quintin_P · October 12, 2012, 10:29pm

Thanks for this. This looks a bit complicated.

Id assume that make install will overwrite the executable and that will
ensure everything. Or should I just go ahead and do service nginx
restart

Quintin_P · October 12, 2012, 10:45pm

Check ‘nginx -V’, if match new version - just /etc/init.d/nginix upgrade
(if you’re on linux).

Quintin_P · October 12, 2012, 11:28pm

On Fri, Oct 12, 2012 at 01:29:14PM -0700, Quintin P. wrote:

Hi there,

Thanks for this. This looks a bit complicated.

I’d assume that “make install” will overwrite the executable and that will
ensure everything. Or should I just go ahead and do service nginx restart

“make install” will replace the binary. It won’t run the new binary.

After replacing the binary, you’ll want to do something like

kill -USR2 $(cat logs/nginx.pid)

And after testing that things are working as expected, then

kill -WINCH $(cat logs/nginx.pid.oldbin)

followed eventually by

kill -QUIT $(cat logs/nginx.pid.oldbin)

The “control.html” page has more details about how to handle problems.

“service nginx restart” probably doesn’t do that sequence, and so
probably
won’t be 0-downtime. But you can use your test system to find a sequence
that works well enough for you.

f

Francis D. [email protected]

Quintin_P · October 13, 2012, 6:51am

Since you are using CentOS, look into rebuilding the rpm against the src
rpm.

http://nginx.org/packages/centos/6/SRPMS/

I took the src rpm and enabled the modules that I need and added the
modules that I wanted from the addons. I then built the rpm and
created my own repo server inside my network to distribute it to my
internal servers. When you build the RPM and install it, it will
restart nginx but that should only be a few seconds of downtime.

Brent

Quintin_P · October 13, 2012, 2:26am

I don’t see how this can work.

Trying to replace the nginx executable with a new version of nginx,
while
nginx is running, will produce:

cp: cannot create regular file `nginx’: Text file busy

Or am I missing something?

This is standard behavior on any running executable in Linux.

Quintin_P · October 13, 2012, 2:11pm

On Saturday 13 October 2012 04:25:36 Bob S. wrote:

I don’t see how this can work.

Trying to replace the nginx executable with a new version of nginx, while
nginx is running, will produce:

cp: cannot create regular file `nginx’: Text file busy

Or am I missing something?

This is standard behavior on any running executable in Linux.

This is just a warning produced by “cp”, you can force it with -f flag.

wbr, Valentin V. Bartenev

–

Quintin_P · October 13, 2012, 1:44pm

On 12 Out 2012 22h29 CEST, [email protected] wrote:

Thanks for this. This looks a bit complicated.

You could just steal debian’s postinst script also:

gist.github.com

https://gist.github.com/perusio/3884272

upgrade_nginx.sh

#!/bin/sh

set -e

case "$1" in
  configure)
    if [ -z $2 ] && [ ! -e /etc/nginx/sites-enabled/default ] ; 
    then
      ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default
    fi

This file has been truncated. show original

of course there are debian specificities. Maybe you could adapt it or
see if there’s a similar thing on the centOS package.

— appa

Quintin_P · October 13, 2012, 2:52pm

On Fri, Oct 12, 2012 at 08:25:36PM -0400, Bob S. wrote:

Hi there,

I don’t see how this can work.

And yet, it does.

Trying to replace the nginx executable with a new version of nginx, while
nginx is running, will produce:

cp: cannot create regular file `nginx’: Text file busy

I’ve never seen that error message.

That’s not to say that it doesn’t happen; but that it is by no means
universal.

Or am I missing something?

It is possible to move, remove, rewrite, or replace a file, independent
of whether it is being used by another process.

This is standard behavior on any running executable in Linux.

I’d suggest that your Linux is not the same as my Linux.

“standard behavior” is that “file being used” does not block “file
being modified”.

All that said, if what “make install” does by default is not what you
want, you can always “cp” or “mv” the old executable; then “cp”, “mv”,
“cat”, or otherwise, put the new executable in place.

And then send the special nginx signals to ensure upgrade with zero
downtime.

(And it is possible that your system init script does exactly this.)

All the best,

f

Francis D. [email protected]

Quintin_P · October 13, 2012, 4:10pm

To add, if you require 0 downtime then you really need to reevaluate
your environment. Even in the linux world, things need to be
restarted or rebooted. If you cannot be down for more than a few
seconds to allow for a software update, then you need to look into
setting up a clustered/load balanced environment. I have 2 front end
load balancers with 4 backend web servers so to keep my environment up
and running. This allows me to run software updates and system
reboots with little to no downtime. At then end, it’s keeping the
perception that you are up and running to your customers.

Just my 2 cents

Brent

Quintin_P · October 14, 2012, 7:16pm

Some of the best advice yet. Anybody in serious production mode should
have no single points of failure.

Quintin_P · October 13, 2012, 2:22pm

On Saturday 13 October 2012 16:10:48 Valentin V. Bartenev wrote:

This is standard behavior on any running executable in Linux.

This is just a warning produced by “cp”, you can force it with -f flag.

But anyway, it would be a much better idea to “mv” or “rm” an old binary
instead
of overwrite it.

wbr, Valentin V. Bartenev

–