Forum: NGINX Webserver crashes sometimes - don't know why

Hello,

my nginx webserver crashes sometimes and I don't know why. It's not on 
every
day and not at the same time as the crash before. I just get sometimes a
message from my monitoring service that my http server isn't working
anymore. Then I restart nginx and everything is working fine again - 
till
the next mysterious crash.

So how I can find out why it crashes? I use the latest development 
version
of nginx. I have the problem since 1.3.8 or 1.3.9 (I guess).

Thanks for any help! :)

Greets from Austria,
Patrik / Patschi

So you are actually running1.3.10? What modules have you build in, and 
whats your exact configuration? What is the error logfile saying at 
crash time?

You may need to run a debug build of nginx to track this down further 
[1]...



[1] http://nginx.org/en/docs/debugging_log.html





----------------------------------------

Hi Lukas,

No, I currently have the latest development version from svn  of 
01.01.2013
- 1.3.11. (Revision 5001).
I only have one installed module: headers more

Have I add "debug" behind error.log? I currently have it without debug 
and I
don't see any information about the crash.

Thanks for your help :)

Greets,
Patschi

-----Ursprngliche Nachricht-----
Von: nginx-bounces@nginx.org [mailto:nginx-bounces@nginx.org] Im Auftrag 
von
Lukas Tribus
Gesendet: Montag, 07. Jnner 2013 10:05
An: nginx@nginx.org
Betreff: RE: Webserver crashes sometimes - don't know why


So you are actually running1.3.10? What modules have you build in, and
whats your exact configuration? What is the error logfile saying at 
crash
time?

You may need to run a debug build of nginx to track this down further 
[1]...



[1] http://nginx.org/en/docs/debugging_log.html





----------------------------------------
> isn't working anymore. Then I restart nginx and everything is working
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Well, actually you have to recompile the source-code and configure it 
with --with-debug AND enable the debug command behind error.log (see 
link [1] at the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are 
filesystem can cope with the amount of data logged.


[1] http://nginx.org/en/docs/debugging_log.html




----------------------------------------

I already built nginx with the --with-debug flag. Well, I have 10GB for 
my
/var/log/ partition. I hope that this is enough :)
I don't know when the next crash will be, so I have to wait. Thanks for 
the
help. I it crashes again I will answer back.


-----Ursprngliche Nachricht-----
Von: nginx-bounces@nginx.org [mailto:nginx-bounces@nginx.org] Im Auftrag 
von
Lukas Tribus
Gesendet: Montag, 07. Jnner 2013 10:37
An: nginx@nginx.org
Betreff: RE: AW: Webserver crashes sometimes - don't know why


Well, actually you have to recompile the source-code and configure it 
with
--with-debug AND enable the debug command behind error.log (see link [1] 
at
the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are
filesystem can cope with the amount of data logged.


[1] http://nginx.org/en/docs/debugging_log.html




----------------------------------------
> I only have one installed module: headers more
>
> Have I add "debug" behind error.log? I currently have it without debug
> and I don't see any information about the crash.
>
> Thanks for your help :)
>
> Greets,
> Patschi
>


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

Well, I would monitor the free space and the IO load carefully. If you 
have a lot of traffic this may become a problem.


----------------------------------------

I just found something interest in "dmesg" log:
[5294633.862284] __ratelimit: 20 callbacks suppressed
[5294633.862288] nginx[20568]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a1a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.659735] nginx[20569]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.818078] nginx[20571]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.819429] nginx[20581]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294635.313816] nginx[20589]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294635.402682] nginx[20590]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294682.926163] nginx[20596]: segfault at 4a ip 00000000004459df sp
00007fff0260a0f0 error 4 in nginx[400000+a3000]
[5294685.155117] nginx[20595]: segfault at 4a ip 00000000004459df sp
00007fff0260a280 error 4 in nginx[400000+a3000]
[5294686.158466] nginx[21276]: segfault at 4a ip 00000000004459df sp
00007fff0260a130 error 4 in nginx[400000+a3000]
[5294688.683947] nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]

Seems to be a error in libc...

-----Ursprngliche Nachricht-----
Von: nginx-bounces@nginx.org [mailto:nginx-bounces@nginx.org] Im Auftrag 
von
Lukas Tribus
Gesendet: Montag, 07. Jnner 2013 10:37
An: nginx@nginx.org
Betreff: RE: AW: Webserver crashes sometimes - don't know why


Well, actually you have to recompile the source-code and configure it 
with
--with-debug AND enable the debug command behind error.log (see link [1] 
at
the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are
filesystem can cope with the amount of data logged.


[1] http://nginx.org/en/docs/debugging_log.html




----------------------------------------
> I only have one installed module: headers more
>
> Have I add "debug" behind error.log? I currently have it without debug
> and I don't see any information about the crash.
>
> Thanks for your help :)
>
> Greets,
> Patschi
>


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

I doubt the dmesg output is enough for the developers to track down the 
bug.

Apart from the actual debug log (which is crucial), can you provide 
output of "nginx -V" and your configuration (remove confidential 
information like IP addresses or domain names if you need to, but leave 
the rest of it intact).

If you post all those informations, the developers should get a better 
picture of what happens here.


Regards,

Lukas

Hello!

On Mon, Jan 07, 2013 at 12:45:52PM +0100, Patrik Kernstock wrote:

> [5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
> 00007fff0260a130 error 4 in nginx[400000+a3000]
> [5294688.683947] nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
> 00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
> [5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41 sp
> 00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
>
> Seems to be a error in libc...

It's highly unlikely to be an error in libc, segfaults in libc
usually heppen when libc functions are called with incorrect
arguments.

You need to obtain coredump and provide a backtrace, see
http://wiki.nginx.org/Debugging for details, in paricular these two
sections:

http://wiki.nginx.org/Debugging#Core_dump
http://wiki.nginx.org/Debugging#Asking_for_help

Please note: it would be good idea to make sure you are able to
reproduce the problem without any 3rd party modules compiled in.

--
Maxim Dounin
http://nginx.com/support.html

Thanks for your help, but I don't really understand the part with 
"coredump"
and "backtrace"...

Thanks :)

-----Ursprngliche Nachricht-----
Von: nginx-bounces@nginx.org [mailto:nginx-bounces@nginx.org] Im Auftrag 
von
Maxim Dounin
Gesendet: Dienstag, 08. Jnner 2013 04:49
An: nginx@nginx.org
Betreff: Re: AW: Webserver crashes sometimes - don't know why

Hello!

On Mon, Jan 07, 2013 at 12:45:52PM +0100, Patrik Kernstock wrote:

> [5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
> 00007fff0260a130 error 4 in nginx[400000+a3000] [5294688.683947]
> nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
> 00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
> [5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41
> sp
> 00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
>
> Seems to be a error in libc...

It's highly unlikely to be an error in libc, segfaults in libc usually
heppen when libc functions are called with incorrect arguments.

You need to obtain coredump and provide a backtrace, see
http://wiki.nginx.org/Debugging for details, in paricular these two
sections:

http://wiki.nginx.org/Debugging#Core_dump
http://wiki.nginx.org/Debugging#Asking_for_help

Please note: it would be good idea to make sure you are able to 
reproduce
the problem without any 3rd party modules compiled in.

--
Maxim Dounin
http://nginx.com/support.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

> Thanks for your help, but I don't really understand the part with "coredump"
> and "backtrace"...

1. Recompile nginx with CFLAGS="-g -O0" (for debugging symbols and 
without compiler optimization). You can just prepend it to your 
./configure line.
Before:
./configure --with-debug --with-ipv6 --with-http_flv_module 
--with-http_mp4_module
After:
CFLAGS="-g -O0" ./configure --with-debug --with-ipv6 
--with-http_flv_module --with-http_mp4_module

2. compile nginx with "make" like you always do.

3. create a directory for the core files and make it readable from your 
workers. For example: mkdir /nginx-core-dumps/ && chmod a+w 
/nginx-core-dumps/

4. add this to your nginx configuration:
worker_rlimit_core 500M;
working_directory /nginx-core-dumps/;

5. (install and) start nginx and wait until it crashes. It should have 
created the core-dump in /nginx-core-dumps/.

6. (install and) start gdb: gdb nginx /nginx-core-dumps/nginx.core

7. within gdb, run the commands "bt" and "backtrace full", followed by a 
"quit".

8. Post the gdb output on this mailing list, the developers will analyze 
it then.


I hope I didn't missed anything, but I think this should be it. Example 
at [1].


[1] http://pastebin.com/raw.php?i=NPjdQcVu

Since weeks no crash. I don't know why the crash happened sometimes...
Working perfekt as usual :)
If a next crash happens, I know what to do - Thank you all!


_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx