Nginx does not re-open log files on SIGUSR1

Server NGINX
1

Hi there.

I think this is bug, I saw that after logrotate (which on the end
running SIGUSR1 on master process) my access log is empty. However if I
do SIGHUP it starts working. Looks like Nginx does not re-open logfiles
after USR1.

I was able reproduce it multiple times, nginx version 0.8.53, I am not
sure if it was working in older versions.

– Piotr.

2

Hi Piotr,

I think this is bug, I saw that after logrotate (which on the end
running SIGUSR1 on master process) my access log is empty. However if I
do SIGHUP it starts working. Looks like Nginx does not re-open logfiles
after USR1.

I was able reproduce it multiple times, nginx version 0.8.53, I am not
sure if it was working in older versions.

Are you sure that logrotate sends SIGUSR1?
Did you try sending this signal yourself?
How did you verify that logs did not re-open?

Everything works fine with nginx/0.8.53 on my system (OpenBSD):

$ nginx -V 2>&1 | head -1
nginx version: nginx/0.8.53

$ sudo rm -rf /var/log/nginx/*
$ sudo nginx

$ curl localhost >/dev/null 2>&1
$ ls -l /var/log/nginx/
total 8
-rw-r–r-- 1 root wheel 167 Jan 3 11:30 access.log
-rw-r–r-- 1 root wheel 775 Jan 3 11:30 error.log

$ sudo pkill -USR1 nginx

$ curl localhost >/dev/null 2>&1
$ ls -l /var/log/nginx/
total 8
-rw-r–r-- 1 _nginx wheel 334 Jan 3 11:31 access.log
-rw-r–r-- 1 _nginx wheel 1593 Jan 3 11:31 error.log

Best regards,
Piotr S. < [email protected] >

3

On 03.01.2011 13:38, Piotr S. wrote:

I think this is bug, I saw that after logrotate (which on the end
running SIGUSR1 on master process) my access log is empty. However if
I do SIGHUP it starts working. Looks like Nginx does not re-open
logfiles after USR1.

[…]

$ sudo pkill -USR1 nginx

pkill send USR1 signal to all nginx processes,
not only master, but also all worker processes
in random order - it may be first workers, and last master.

if worker processes does not have write permissions
to /var/log/nginx directory - worker processes can`t
create new log files, and can’t write to log files.

USR1 signal must be sent only to nginx master process
and only as postrotate script:

cat /etc/logrotate.d/nginx

/var/log/nginx/*log {
daily
rotate 9
missingok
notifempty
compress
sharedscripts
postrotate
[ ! -f /var/run/nginx.pid ] || kill -USR1 cat /var/run/nginx.pid
endscript
}

documentation: Управление nginx

–
Best regards,
Gena

4

Hi Piotr,

On 01/03/2011 12:38 PM, Piotr S. wrote:

Are you sure that logrotate sends SIGUSR1?
Did you try sending this signal yourself?
How did you verify that logs did not re-open?

Yes, I checked twice logrotate’s nginx rules and I tried send USR1 by
myself. To check, after rotate I did wc -l on my_vhost.access_log and it
was empty, I was hitting it with wget but until I did restart or reload,
log was empty.

I was able to ‘fix’ it, which is more like workaround than a real fix,
by adding permissions for nginx user to /var/log/nginx.

Before I had 700 root:root on /var/log/nginx because I am a little
paranoid and I saw no real reason to add workers there since master
process, running as root, is writting there.

After changing owner to nginx, nginx is able re-open logs after SIGUSR1.

Looks like rotated empty logs have root:root 600 perms, maybe it is the
problem? But again, I think master write there, not workers.

– Piotr.

5

Hi,

pkill send USR1 signal to all nginx processes,
not only master, but also all worker processes
in random order - it may be first workers, and last master.

$ curl localhost >/dev/null 2>&1
$ ls -l /var/log/nginx/
total 8
-rw-r–r-- 1 root wheel 167 Jan 3 12:09 access.log
-rw-r–r-- 1 root wheel 1202 Jan 3 12:12 error.log

$ sudo pkill -USR1 -f ^nginx:.*master.*process

$ curl localhost >/dev/null 2>&1
$ ls -l /var/log/nginx/
total 12
-rw-r–r-- 1 _nginx wheel 334 Jan 3 12:13 access.log
-rw-r–r-- 1 _nginx wheel 2738 Jan 3 12:13 error.log

Happy? Result is exactly the same…

if worker processes does not have write permissions
to /var/log/nginx directory - worker processes can`t
create new log files, and can’t write to log files.

Worker process doesn’t create log files, master process does.

Other than complaining about non-existing issue, you didn’t answer any
of my
questions…

Best regards,
Piotr S. < [email protected] >

6

Hi,

Before I had 700 root:root on /var/log/nginx because I am a little
paranoid and I saw no real reason to add workers there since master
process, running as root, is writting there.

You need at least 711, otherwise workers won’t be able to open files in
that
directory.

Best regards,
Piotr S. < [email protected] >

7

Hi,

you didn’t ask any questions.

I did… But I’ve realized just now that you’re not the OP :wink:

Best regards,
Piotr S. < [email protected] >

8

On 03.01.2011 14:18, Piotr S. wrote:

pkill send USR1 signal to all nginx processes,
not only master, but also all worker processes
in random order - it may be first workers, and last master.

if worker processes does not have write permissions
to /var/log/nginx directory - worker processes can`t
create new log files, and can’t write to log files.

Worker process doesn’t create log files, master process does.

if worker processes have write permissions to /var/log/nginx

  • it can create log files, see nginx sources for reference:
    ngx_process_cycle.c file, ngx_worker_process_cycle function.

you didn’t answer any of my questions…

you didn’t ask any questions.

P.S.

nginx 0.8.53 log files rotation work fine on my machines
with logrotate config mentioned in my previous message.

–
Best regards,
Gena

9

Is there a reason why every script or suggestion I see floating around
the net still does it “the hard way” (and I wonder, even on this
mailing list):

$ kill -USR1 cat /var/run/nginx.pid
$ pkill -USR1 -f ^nginx:.*master.*process

instead of simply using nginx’s built-in way to do this:

$ nginx -s reopen

After all, this is supported since 0.7.53.

It should also work fine if using multiple nginx instances, just pass
the path to the configuration file of the specific instance with -c.

See also CommandLine | NGINX or nginx -h:

-s signal : send signal to a master process: stop, quit, reopen, reload

10

On 03.01.2011 14:49, Piotr K. wrote:

I was able to ‘fix’ it, which is more like workaround than a real fix,
by adding permissions for nginx user to /var/log/nginx.

Before I had 700 root:root on /var/log/nginx because I am a little
paranoid and I saw no real reason to add workers there since master
process, running as root, is writting there.

After changing owner to nginx, nginx is able re-open logs after SIGUSR1.

master process running as root open/write files in /var/log/nginx

  • if nginx user have write permissions to this directory,
    700 nginx:nginx - such setup is vulnerable by symlink attack

better approach set permissions 750 root:nginx /var/log/nginx

or 750 root:www-logs /var/log/nginx and add user nginx to group www-logs

Looks like rotated empty logs have root:root 600 perms, maybe it is the
problem?

show your logrotate config for nginx log files.

But again, I think master write there, not workers.

nginx workers also write to log files.

–
Best regards,
Gena

11

Hi,

So nginx’ workers need exec permission on logdir?

Yes.

Exec on dir will allow only chdir there, why worker have to chdir there?

They don’t, man chmod.

Best regards,
Piotr S. < [email protected] >

12

On 01/03/2011 02:16 PM, Gena M. wrote:

master process running as root open/write files in /var/log/nginx

  • if nginx user have write permissions to this directory,
    700 nginx:nginx - such setup is vulnerable by symlink attack
    better approach set permissions 750 root:nginx /var/log/nginx

or 750 root:www-logs /var/log/nginx and add user nginx to group www-logs

Now when you mention it, if nginx worker have read perms there (as you
suggested above), then if user symlink to any log, he will be able fetch
it via nginx which is security hole.

nginx workers also write to log files.

In what cases? And direct or somehow ‘via master proicess’?

On 01/03/2011 01:54 PM, Piotr S. wrote:

You need at least 711, otherwise workers won’t be able to open
files in that directory.

So nginx’ workers need exec permission on logdir? Exec on dir will allow
only chdir there, why worker have to chdir there?

The only problem is that after SIGUSR1 nginx worker need access to
logs (shouldn’t), where on restart/reload nginx can handle it without
access to logs by workers, which as I said above, is [in my opinion]
security hole.

– Piotr

13

Hi John,

Is there a reason why every script or suggestion I see floating around
the net still does it “the hard way” (and I wonder, even on this
mailing list):

$ kill -USR1 cat /var/run/nginx.pid
$ pkill -USR1 -f ^nginx:.*master.*process

Yes, there is: consistency.

Pretty much every daemon supports “the standard way” for control
(graceful
restarts, termination, reloads, etc), so people that use other daemons
prefer to use the method that is consistent among all of them instead of
using daemon-specific one.

Best regards,
Piotr S. < [email protected] >

14

On 01/03/2011 03:09 PM, Piotr S. wrote:

Hi,

So nginx’ workers need exec permission on logdir?

Yes.

Any reason to? Nginx works for me flawless on each box with 700
root:root on /var/log/nginx, the only problem I found is SIGUSR1,
Whatever you agree with me or not, nginx shoudn’t need perms on its logs
dir, because it will allow users to use symlink to fetch logs.

– Piotr.

15

Hi,

Any reason to?

Yes, user requires “+x” permission to the directory in order to be able
to
open any file(s) inside it. Google/Bing/whatever for “unix permissions”,
this is as simple as it gets.

Nginx works for me flawless on each box with 700 root:root on
/var/log/nginx, the only problem I found is SIGUSR1, Whatever you agree
with me or not, nginx shoudn’t need perms on its logs dir, because it will
allow users to use symlink to fetch logs.

This is because:

  • on start and reload - master process opens log files before fork() and
    worker processes only inherit them,
  • on reopen - all processes need to open logs, so workers also need
    permission to open log files.

Best regards,
Piotr S. < [email protected] >

16

On 01/03/2011 03:25 PM, Piotr S. wrote:

Hi,

Any reason to?

Yes, user requires “+x” permission to the directory in order to be able
to open any file(s) inside it. Google/Bing/whatever for “unix
permissions”, this is as simple as it gets.

This is what I mean by ‘exec will allow only chdir there’. With X you
can access dir content and depends on files rights, you can read them
etc. Mental shortcut.

Nginx works for me flawless on each box with 700 root:root on
/var/log/nginx, the only problem I found is SIGUSR1, Whatever you
agree with me or not, nginx shoudn’t need perms on its logs dir,
because it will allow users to use symlink to fetch logs.

This is because:

  • on start and reload - master process opens log files before fork() and
    worker processes only inherit them,
  • on reopen - all processes need to open logs, so workers also need
    permission to open log files.

Well ok, I understand [now] why it is needed (perms that is). However
security issue still remains which in my opinion should be addressed as
bug and fixed, can you agree with me?

– Piotr.

17

Hi Piotr!

Pretty much every daemon supports “the standard way” for control
(graceful restarts, termination, reloads, etc), so people that use other
daemons prefer to use the method that is consistent among all of them
instead of using daemon-specific one.

I don’t argue that the most common set of signals and their similar
handlers in various daemons (HUP=reload, TERM=clean shutdown,
QUIT=graceful shutdown…) make life a lot easier for most
administrators.

But I’m not convinced that this is the case here. As soon as USR* and
whatever other custom signals are handled in a way that is special to
the specific daemon, there’s not really a point to “consistency” any
more. It is special already.

And IMHO this is exactly the time where a control CLI can shine,
especially since the available signals are limited and the possible
control knobs are not.

Apache has apachectl[1] or httpd’s “-k” option for quite some time now.
It is used by many distributions in init scripts and log rotation
scripts instead of hardcoding signals and paths to pid files everywhere.

Nginx has the “-s” option. For me, this appears to be cleaner and
simpler than to remember or always look up what custom signal X does to
this daemon. For the most basic set of control signals I agree with you,
but it still has the downside of having to find the pid first (and
hardcoding parameters of the daemon configuration into external
scripts).

[1] apachectl - Apache HTTP Server Control Interface - Apache HTTP Server Version 2.2

18

Hi,

I don’t argue that the most common set of signals and their similar
handlers in various daemons (HUP=reload, TERM=clean shutdown,
QUIT=graceful shutdown…) make life a lot easier for most administrators.

But I’m not convinced that this is the case here. As soon as USR* and
whatever other custom signals are handled in a way that is special to
the specific daemon, there’s not really a point to “consistency” any
more. It is special already.

USR1 is pretty much “standard” for log files re-open.

For the most basic set of control signals I agree with you,
but it still has the downside of having to find the pid first (and
hardcoding parameters of the daemon configuration into external scripts).

With “pkill” you don’t have to :wink:

Anyway, I’m not saying that sending signals is better than “nginx -s”,
I’m
just saying it’s more consistent and that’s why I’m using it.

Best regards,
Piotr S. < [email protected] >

19

Hi,

This is what I mean by ‘exec will allow only chdir there’. With X you can
access dir content and depends on files rights, you can read them etc.
Mental shortcut.

OK, but it has nothing to do with chdir :wink:

Well ok, I understand [now] why it is needed (perms that is). However
security issue still remains which in my opinion should be addressed as
bug and fixed, can you agree with me?

I would imagine that there are far more important files than nginx logs
that
your workers have access to :wink:

Best regards,
Piotr S. < [email protected] >

20

On 03.01.2011 15:56, John Feuerstein wrote:

kill -USR1 cat /var/run/nginx.pid

works faster then nginx -s reopen

and also

kill -USR1 cat /var/run/nginx.pid

has explicitly defined syntax and behavior.

It should also work fine if using multiple nginx instances, just pass
the path to the configuration file of the specific instance with -c.

in many cases nginx config can be very big, with thousands of includes.

nginx -s reopen feature was added for windows version of nginx, AFAIK.

–
Best regards,
Gena