Nginx reload problem

Hello:

I found that some nginx config option doesn’t take effect after
modification with reload, the following is a simple test case:

nginx version: nginx/1.2.7
nginx.conf:

worker_processes 1;
error_log logs/error.log info;

events {
worker_connections 1024;
}

http {
limit_req_zone $arg_a zone=testzone:64m rate=1r/s;

server {
    listen 80;

    location / {
        limit_req zone=testzone burst=2;
        alias /;
    }
}

}

I change $arg_a to $arg_b in the line ‘limit_req_zone $arg_a
zone=testzone:64m rate=1r/s;’ then reload nginx, but the change doesn’t
take effect, unless I stop nginx manually and start it again.

Is this an expected behavior ? or are there any other nginx config
options
that not compatible with reload operation?

Thanks!

Hello!

On Thu, Aug 15, 2013 at 11:07:30PM +0800, li zJay wrote:

    location / {

Is this an expected behavior ? or are there any other nginx config options
that not compatible with reload operation?

It’s expected behaviour. On such reload attempt, nginx will write
something like:

… [emerg] … limit_req “testzone” uses the “arg_b” variable while
previously it used the “arg_a” variable

into error log, explaining why it failed to load new
configuration.


Maxim D.
http://nginx.org/en/donation.html

Hi,

Is there any SAML2.0 module available for NGINX?

Regards,
John

I guess it would be nice if the doc warned about directives that need a
server restart to be reloaded.

Everyone supposes (as it seems obvious) that reloading Nginx is enough
to
apply configuration changes.

An interesting part of the question was the inquiry about the potential
existence other directives requiring a server restart rather than its
reload. Do you have intel on this Maxim?

B. R.

Hello!

On Fri, Aug 16, 2013 at 10:16:16PM -0400, B.R. wrote:

I guess it would be nice if the doc warned about directives that need a
server restart to be reloaded.

Everyone supposes (as it seems obvious) that reloading Nginx is enough to
apply configuration changes.

Reloading is enough. What is very wrong is to assume that sending
a HUP signal to nginx is enough for a reload. For various
reasons, ranging from configuration syntax errors to out of memory
problems, configuration reload might fail.

Quoting documentation:

… If this fails, it rolls back changes and continues to work
with old configuration. …

http://nginx.org/en/docs/control.html#reconfiguration

An interesting part of the question was the inquiry about the potential
existence other directives requiring a server restart rather than its
reload. Do you have intel on this Maxim?

There are no directives which require a server restart. But some
changes are not possible on the fly - e.g. you can’t change a
shared memory zone size. If you want to change it - you have to
create another shared memory zone, or use a binary upgrade which
doesn’t inherit shared memory zones and their contents (or use a
restart, which will obviously work as well).


Maxim D.
http://nginx.org/en/donation.html

Hello!

On Sat, Aug 17, 2013 at 12:07:22AM -0400, B.R. wrote:

server restart to be reloaded.

I saw on some init script on CentOS (probably with the packaged version of
that OS) that the configuration check was invoked automatically when the
service reload was called.
That would be a nice improvement to the init script shipped with the
official Nginx package (from nginx.org) to avoid a manual ‘nginx -t’ call
before the reload.

I don’t think that calling “nginx -t” as a mandatory step before
configuration reload is a good idea: nginx binary running and
nginx binary on disk might be different, and “nginx -t” result
might be incorrect because of this, in some cases rejecting valid
configurations.

Additionally, it does duplicate work by parsing/loading a
configuration which will be again parsed by a master process
during configuration reload. While in most cases it’s not
significant, I’ve seen configurations taking more than 1m to load
due to big geo module bases used.

​That reminds me that the init script shipped with Nginx doesn’t take
advantage of the 'hot binary switch’​

​to provide a ‘soft restart’ without downtime.
It clearly would be a nice alternative to the normal restart which is
basically a stop/start.​

There is the “upgrade” command in the init script shipped with
nginx.org linux packages.


Maxim D.
http://nginx.org/en/donation.html

Hello,

On Sat, Aug 17, 2013 at 7:37 AM, Maxim D. [email protected]
wrote:

during configuration reload. While in most cases it’s not
significant, I’ve seen configurations taking more than 1m to load
due to big geo module bases used.

​​In that case, the server admin has a problem, since he has no way to
test
the configuration other the calling ‘reload’ on the running instance and
check the logs for errors, hoping they are not already crawling under
production-related log messages…
One way or another, you test the configuration against an existing
binary
because you want to start or reload this binary with the conf. There is
no
point in having a running instance having already deleted its disk
binary
file: If you are in transition between 2​

​versions of Nginx, you shouldn’t also make big changes to the conf…
That’s a 2-steps procedures I’d say​: One thing at a time.

Testing conf is of course a duplicate of work, but that’s a safe
operation.
The command output will determine if your new configuration will work
without having to carefully watch logs with anxiety.

There is the “upgrade” command in the init script shipped with
nginx.org linux packages.

​​Ok, so could Li have used the ‘upgrade’ command insted of ‘reload’ to
reload the configuration and change the allocated memory?

​Thanks,​


B. R.

Hello Maxim! :o)

On Fri, Aug 16, 2013 at 11:16 PM, Maxim D. [email protected]
wrote:

Reloading is enough. What is very wrong is to assume that sending

​​Yup I knew that. I thought Nginx was able to re-arrange its memory
allocation for the new variable.
I didn’t know it was keeping the same fixed memory​, only replacing
existing values.

I saw on some init script on CentOS (probably with the packaged version
of
that OS) that the configuration check was invoked automatically when the
service reload was called.
That would be a nice improvement to the init script shipped with the
official Nginx package (from nginx.org) to avoid a manual ‘nginx -t’
call
before the reload.

​That reminds me that the init script shipped with Nginx doesn’t take
advantage of the 'hot binary switch’​

​to provide a ‘soft restart’ without downtime.
It clearly would be a nice alternative to the normal restart which is
basically a stop/start.​

Thanks for your answer and your involvement on the mailing list btw.
Your
team job interfacing with us makes Nginx a product with an added
reassuring
sensation of quality & comfort.

B. R.

Hello,

On Sun, Aug 18, 2013 at 3:14 PM, Maxim D. [email protected]
wrote:

​​OK ​I think I got it. ‘reload’ deals with a running instance while
‘upgrade’ starts a new one from the binary on disk, so it makes sense to
check the configuration against the binary when upgrading but not when
reloading in case the binary on disk changed in between.
The latter is a weirdest-case scenario (since you change the binary when
you want to upgrade something, which won’t result in a ‘reload’ call),
though possible…

You decision makes sense and is the safest. Thanks for your lights on
that.

Testing conf is of course a duplicate of work, but that’s a safe
operation.
The command output will determine if your new configuration will work
without having to carefully watch logs with anxiety.

As I already tried to explain, watching logs is required anyway.

​… if you had changes between the binary on disk and the one being
run.
Which is highly unlikely to happen as calling ‘reload’ on the current
process would mean applying the configuration made for the new binary to
the old running one (which needs to be replaced ASAP since it can’t
resist
to a server restart…)​. But yeah, in that weird case, you’ll watch the
logs.

There is the “upgrade” command in the init script shipped with
nginx.org linux packages.

​​Ok, so could Li have used the ‘upgrade’ command insted of ‘reload’ to
reload the configuration and change the allocated memory?

Yes.

​Thanks.​

Your input has been much appreciated, as always…

B. R.

Hello!

On Sun, Aug 18, 2013 at 05:29:11PM -0400, B.R. wrote:

[…]

process would mean applying the configuration made for the new binary to
the old running one (which needs to be replaced ASAP since it can’t resist
to a server restart…)​. But yeah, in that weird case, you’ll watch the
logs.

Quote from my first messages in this thread:

: What is very wrong is to assume that sending
: a HUP signal to nginx is enough for a reload. For various
: reasons, ranging from configuration syntax errors to out of memory
: problems, configuration reload might fail.

Even if the binary on disk matches one in memory, and
configuration syntax is ok - it’s always possible that system will
run out of memory or file descriptors (or will reach per-process
limits), or newly configured listening sockets will conflict with some
other services running (or previously configured sockets in case
of Linux, which doesn’t allow wildcard and non-wildcard sockets to
coexists), or something else will happen (including cases when
reload is not possible due to requested configuration changes, see
original question).

Questions “why nginx can’t reload configuration” appear in mailing
lists on a regular basis. In addition to this thread, this week
seen at least once in nginx-ru@, see [1]. Correct answer is
usually the same - “Try looking into error log”.

[1] релоад конфига


Maxim D.
http://nginx.org/en/donation.html

Hello!

On Sat, Aug 17, 2013 at 12:36:38PM -0400, B.R. wrote:

might be incorrect because of this, in some cases rejecting valid
the configuration other the calling ‘reload’ on the running instance and
check the logs for errors, hoping they are not already crawling under
production-related log messages…
One way or another, you test the configuration against an existing binary
because you want to start or reload this binary with the conf. There is no
point in having a running instance having already deleted its disk binary
file: If you are in transition between 2​

​versions of Nginx, you shouldn’t also make big changes to the conf…
That’s a 2-steps procedures I’d say​: One thing at a time.

Making any changes to the configuration isn’t something
significant: even without changes at all new binary on disk might
not consider an old configuration as a valid e.g. due to some
module not compiled in. And a reload might be required for
various external reasons.

I don’t say it’s a normal situation, but it’s possible, and
proposed change to init script will prevent init script from
working in such situation.

Testing conf is of course a duplicate of work, but that’s a safe operation.
The command output will determine if your new configuration will work
without having to carefully watch logs with anxiety.

As I already tried to explain, watching logs is required anyway.

There is the “upgrade” command in the init script shipped with
nginx.org linux packages.

​​Ok, so could Li have used the ‘upgrade’ command insted of ‘reload’ to
reload the configuration and change the allocated memory?

Yes.


Maxim D.
http://nginx.org/en/donation.html