Forum: NGINX nginx crash on reload -- how to detect?

Hello,

I'm currently running a self-built nginx 1.3.14 on a debian system and
use the attached (and also inlined) init.d script as /etc/init.d/nginx
for managing the service. It's taken unmodified from debian wheezy.

I somehow managed to get the nginx master process crashing (I have a few
third-party modules compiled in and that's not the issue to be discussed
right now) upon

$ service nginx reload

The problem is that in this case it just states "Reloading nginx
configuration: nginx." without me realizing that the master process
crashed and that the new config did not become activated. This is an
issue by itself -- I spent some time changing the config over and over
and doing reload attempts and was wondering why the heck the changes did
not take effect.

Another

$ service nginx stop
$ service nginx start

then revealed that something wrong is going on, because 'stop' did not
take action, 'start' spawned a new master process, but the listening
ports were still blocked by old workers.

I am wondering if there is a neat way to improve the service script in
order to make it realize when the nginx master unexpectedly dies in the
process of performing one of the service actions.

Cheers,

Jan-Philip



#!/bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/sbin/nginx
NAME=nginx
DESC=nginx

# Include nginx defaults if available
if [ -f /etc/default/nginx ]; then
  . /etc/default/nginx
fi

test -x $DAEMON || exit 0

set -e

. /lib/lsb/init-functions

test_nginx_config() {
  if $DAEMON -t $DAEMON_OPTS >/dev/null 2>&1; then
    return 0
  else
    $DAEMON -t $DAEMON_OPTS
    return $?
  fi
}

case "$1" in
  start)
    echo -n "Starting $DESC: "
    test_nginx_config
    # Check if the ULIMIT is set in /etc/default/nginx
    if [ -n "$ULIMIT" ]; then
      # Set the ulimits
      ulimit $ULIMIT
    fi
    start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;

  stop)
    echo -n "Stopping $DESC: "
    start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
        --exec $DAEMON || true
    echo "$NAME."
    ;;

  restart|force-reload)
    echo -n "Restarting $DESC: "
    start-stop-daemon --stop --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON || true
    sleep 1
    test_nginx_config
    # Check if the ULIMIT is set in /etc/default/nginx
    if [ -n "$ULIMIT" ]; then
      # Set the ulimits
      ulimit $ULIMIT
    fi
    start-stop-daemon --start --quiet --pidfile \
        /var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
    echo "$NAME."
    ;;

  reload)
    echo -n "Reloading $DESC configuration: "
    test_nginx_config
    start-stop-daemon --stop --signal HUP --quiet --pidfile
/var/run/$NAME.pid \
        --exec $DAEMON || true
    echo "$NAME."
    ;;

  configtest|testconfig)
    echo -n "Testing $DESC configuration: "
    if test_nginx_config; then
      echo "$NAME."
    else
      exit $?
    fi
    ;;

  status)
    status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || 
exit $?
    ;;
  *)
    echo "Usage: $NAME
{start|stop|restart|reload|force-reload|status|configtest}" >&2
    exit 1
    ;;
esac

exit 0

On Tue, Mar 12, 2013 at 05:39:27PM +0100, Jan-Philip Gehrcke wrote:

Hi there,

> I'm currently running a self-built nginx 1.3.14 on a debian system and
> use the attached (and also inlined) init.d script as /etc/init.d/nginx
> for managing the service. It's taken unmodified from debian wheezy.

This looks like a generic init script which has been slightly adapted
for nginx.

It introduces some extra levels between what you do (run "service") and
what you want to do (control nginx). These levels are frequently useful,
but in this case they do seem to hide the information you want.

> $ service nginx reload
>
> The problem is that in this case it just states "Reloading nginx
> configuration: nginx." without me realizing that the master process
> crashed and that the new config did not become activated.

> I am wondering if there is a neat way to improve the service script in
> order to make it realize when the nginx master unexpectedly dies in the
> process of performing one of the service actions.

Right now, you run "service" which runs this shell script which
(eventually) runs start-stop-daemon which runs nginx.

nginx can communicate using its return value, its stderr output, and
its log file.

The sequence you run seems to hide the return value and the stderr 
output
from you.

It's not clear to me exactly what happened in your failing case; but if
you can recreate it, what do you see when you run "nginx", "nginx -t",
"nginx -s reload", and "nginx -s stop"? Check the return value, stderr
output, and error.log after each one.

Is there enough information in that output to help you understand the
nature of the problem?

(I believe that a return value of non-0 means that something went wrong,
and the stderr output and/or error.log will indicate what that something
was. The output may also have information when the return value is 0.)

When you're happy that you understand the nginx output, then you can try
changing the init script to let that output get back to you somehow. 
That
might involve learning where the current script puts that information;
or changing the start-stop-daemon invocation line to make it available;
or replacing the entire script with something much simpler and with
fewer features.

Be aware that removing features will probably make this daemon not fit
in with the startup/shutdown of all others on your system. Only you can
decide how much that matters to you.

  f
--
Francis Daly        francis@daoine.org