Safe log rotation

Server NGINX
#1

Hi,

I’d like to check how nginx handles command from
http://wiki.nginx.org/LogRotation
kill -USR1 cat /var/run/nginx.pid

I’m using it to recreate log files during rotation.

My question if any loglines can be lost in case time interval between
physical log files rotation and USR1 is large enough, like seconds? Or
if
load is very high.

For example, while I was sending traffic to website with “ab -n 20 -c 10
http://test-s.mysite.com/static/99$i/logo/test.png > /dev/null 2>&1” in
the
loop, I executed on the web-server this

tail -n 5 /mnt/vg0-lv0/access.log
rm -rf /mnt/vg0-lv0/access.log
sleep 15; sudo kill -USR1 cat /var/run/nginx.pid
sleep 2
head -n 5 /mnt/vg0-lv0/dj-access.log

2014-10-01T21:16:10+00:00,1412198170.957,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2486547,1,201,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:10+00:00,1412198170.957,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2486548,1,240,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:10+00:00,1412198170.958,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2486549,1,279,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:10+00:00,1412198170.959,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2486550,1,318,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:10+00:00,1412198170.964,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2486551,1,279,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"

…

2014-10-01T21:16:27+00:00,1412198187.024,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2489854,1,162,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:27+00:00,1412198187.024,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2489855,1,162,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:27+00:00,1412198187.024,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2489856,1,162,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:27+00:00,1412198187.024,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2489857,1,240,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"
2014-10-01T21:16:27+00:00,1412198187.024,10.120.71.11,"-",“206.223.189.242”,test-s.mysite.com,2489858,1,279,"-","-","-",-,"-","-",“ApacheBench/2.3”,"-",HTTP/1.0,GET,http,/static/logo/test.png,"-",200,5676,5150,0.000,"-","-"

As you can see, there is a gap in timestamps sequence.
Is there a way to make sure all log-lines are going to be written on a
highly loaded web farm?

Thank you,
Roman Naumenko

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,253681,253681#msg-253681

#2

I think you should remove the line:

sleep 15

It doesnt’ anything useful, It just delay the restart, so during 15
seconds
the request will end been registered in the old log file

mv /mnt/vg0-lv0/access.log /mnt/vg0-lv0/access.log.OLD
sudo kill -USR1 cat /var/run/nginx.pid
sleep 2
head -n 5 /mnt/vg0-lv0/access.log

If I would do this in a interactive way, I’d prefer

mv /mnt/vg0-lv0/access.log /mnt/vg0-lv0/access.log.OLD
sudo kill -USR1 cat /var/run/nginx.pid
sleep 2
tail -f /mnt/vg0-lv0/access.log

because I see the log request in real time

#3

On 02 Oct 2014, at 01:30, jmobile [email protected] wrote:

load is very high.

For example, while I was sending traffic to website with “ab -n 20 -c 10
http://test-s.mysite.com/static/99$i/logo/test.png > /dev/null 2>&1” in the
loop, I executed on the web-server this

tail -n 5 /mnt/vg0-lv0/access.log
rm -rf /mnt/vg0-lv0/access.log
sleep 15; sudo kill -USR1 cat /var/run/nginx.pid
sleep 2
head -n 5 /mnt/vg0-lv0/dj-access.log

This is not log rotation but log removal. Try this:

mv /mnt/vg0-lv0/access.log /mnt/vg0-lv0/access.log.OLD
sleep 15;
sudo kill -USR1 cat /var/run/nginx.pid
sleep 2
head -n 5 /mnt/vg0-lv0/access.log

–
Igor S.
Join us for nginx.conf 2014, October 20-22, San Francisco.
Get 25% off with code NGINXUG: http://nginx.com/nginxconf/

#4

On 02 Oct 2014, at 11:08, oscaretu . [email protected] wrote:

I think you should remove the line:

sleep 15

It doesnt’ anything useful, It just delay the restart, so during 15 seconds the
request will end been registered in the old log file

Of course this sleep is not required, it is just to see that nothing is
lost during the sleep.

#5

Thanks Igor for pointing this out.

If file is renamed, would it guarantee that nginx continues writing to
it
without interruption?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,253681,253729#msg-253729

#6

On 03 Oct 2014, at 01:31, jmobile [email protected] wrote:

Thanks Igor for pointing this out.

If file is renamed, would it guarantee that nginx continues writing to it
without interruption?

This is guaranteed by OS.

–
Igor S.
Join us for nginx.conf 2014, October 20-22, San Francisco.
Get 25% off with code NGINXUG: http://nginx.com/nginxconf/