Forum: NGINX Compiling Nginx on production. How to do it without down time.

Hi all,

I run vanilla builds of Nginx from the Nginx centos repo.

But recently Ive started to realize the need for additional modules in
Nginx like Nginx status, openresty, lua etc.

Now this will mean that my vanilla Nginx with

nginx version: nginx/1.0.12

built by gcc 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC)

TLS SNI support enabled

configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf 
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx 
--group=nginx
--with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module 
--with-http_dav_module
--with-http_flv_module --with-http_mp4_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module 
--with-mail
--with-mail_ssl_module --with-file-aio --with-ipv6 --with-cc-opt='-O2 
-g'

will have to be recompiled.

How can I do it with minimal downtime on my prod machine?

On Thu, Oct 11, 2012 at 05:08:55PM -0700, Quintin Par wrote:

Hi there,

> How can I do it with minimal downtime on my prod machine?

http://nginx.org/en/docs/control.html ?

  f
--
Francis Daly        francis@daoine.org

Thanks for this. This looks a bit complicated.

Id assume that make install will overwrite the executable and that will
ensure everything. Or should I just go ahead and do service nginx 
restart

Check 'nginx -V', if match new version - just /etc/init.d/nginix upgrade
(if you're on linux).

On Fri, Oct 12, 2012 at 01:29:14PM -0700, Quintin Par wrote:

Hi there,

> Thanks for this. This looks a bit complicated.
>
> I’d assume that “make install” will overwrite the executable and that will
> ensure everything. Or should I just go ahead and do service nginx restart

"make install" will replace the binary. It won't run the new binary.

After replacing the binary, you'll want to do something like

  kill -USR2 $(cat logs/nginx.pid)

And after testing that things are working as expected, then

  kill -WINCH $(cat logs/nginx.pid.oldbin)

followed eventually by

  kill -QUIT $(cat logs/nginx.pid.oldbin)

The "control.html" page has more details about how to handle problems.

"service nginx restart" probably doesn't do that sequence, and so 
probably
won't be 0-downtime. But you can use your test system to find a sequence
that works well enough for you.

  f
--
Francis Daly        francis@daoine.org

I don't see how this can work.

Trying to replace the nginx executable with a new version of nginx, 
while
nginx is running,  will produce:

cp: cannot create regular file `nginx': Text file busy

Or am I missing something?

This is standard behavior on any running executable in Linux.

Since you are using CentOS, look into rebuilding the rpm against the src 
rpm.

http://nginx.org/packages/centos/6/SRPMS/

I took the src rpm and enabled the modules that I need and added the
modules that I wanted from the addons.  I then built the rpm and
created my own repo server inside my network to distribute it to my
internal servers.  When you build the RPM and install it, it will
restart nginx but that should only be a few seconds of downtime.

Brent

On 12 Out 2012 22h29 CEST, quintinpar@gmail.com wrote:

> Thanks for this. This looks a bit complicated.

You could just steal debian's postinst script also:

https://gist.github.com/3884272

of course there are debian specificities. Maybe you could adapt it or
see if there's a similar thing on the centOS package.

--- appa

On Saturday 13 October 2012 04:25:36 Bob S. wrote:
> I don't see how this can work.
>
> Trying to replace the nginx executable with a new version of nginx, while
> nginx is running,  will produce:
>
> cp: cannot create regular file `nginx': Text file busy
>
> Or am I missing something?
>
> This is standard behavior on any running executable in Linux.

This is just a warning produced by "cp", you can force it with -f flag.

 wbr, Valentin V. Bartenev

--
http://nginx.com/support.html

On Saturday 13 October 2012 16:10:48 Valentin V. Bartenev wrote:
> > This is standard behavior on any running executable in Linux.
>
> This is just a warning produced by "cp", you can force it with -f flag.
>

But anyway, it would be a much better idea to "mv" or "rm" an old binary 
instead
of overwrite it.

 wbr, Valentin V. Bartenev

--
http://nginx.com/support.html

On Fri, Oct 12, 2012 at 08:25:36PM -0400, Bob S. wrote:

Hi there,

> I don't see how this can work.

And yet, it does.

> Trying to replace the nginx executable with a new version of nginx, while
> nginx is running,  will produce:
>
> cp: cannot create regular file `nginx': Text file busy

I've never seen that error message.

That's not to say that it doesn't happen; but that it is by no means
universal.

> Or am I missing something?

It is possible to move, remove, rewrite, or replace a file, independent
of whether it is being used by another process.

> This is standard behavior on any running executable in Linux.

I'd suggest that your Linux is not the same as my Linux.

"standard behavior" is that "file being used" does not block "file
being modified".


All that said, if what "make install" does by default is not what you
want, you can always "cp" or "mv" the old executable; then "cp", "mv",
"cat", or otherwise, put the new executable in place.

And then send the special nginx signals to ensure upgrade with zero
downtime.

(And it is possible that your system init script does exactly this.)

All the best,

  f
--
Francis Daly        francis@daoine.org

To add, if you require 0 downtime then you really need to reevaluate
your environment.  Even in the linux world, things need to be
restarted or rebooted.  If you cannot be down for more than a few
seconds to allow for a software update, then you need to look into
setting up a clustered/load balanced environment.  I have 2 front end
load balancers with 4 backend web servers so to keep my environment up
and running.  This allows me to run software updates and system
reboots with little to no downtime.  At then end, it's keeping the
perception that you are up and running to your customers.

Just my 2 cents

Brent

Some of the best advice yet.  Anybody in serious production mode should
have no single points of failure.