Webserver crashes sometimes - don't know why

Patrik_Kernstock · January 7, 2013, 8:06am

Hello,

my nginx webserver crashes sometimes and I don’t know why. It’s not on
every
day and not at the same time as the crash before. I just get sometimes a
message from my monitoring service that my http server isn’t working
anymore. Then I restart nginx and everything is working fine again -
till
the next mysterious crash.

So how I can find out why it crashes? I use the latest development
version
of nginx. I have the problem since 1.3.8 or 1.3.9 (I guess).

Thanks for any help!

Greets from Austria,
Patrik / Patschi

Patrik_Kernstock · January 7, 2013, 10:05am

So you are actually running1.3.10? What modules have you build in, and
whats your exact configuration? What is the error logfile saying at
crash time?

You may need to run a debug build of nginx to track this down further
[1]…

[1] A debugging log

Patrik_Kernstock · January 7, 2013, 10:18am

Hi Lukas,

No, I currently have the latest development version from svn of
01.01.2013

1.3.11. (Revision 5001).
I only have one installed module: headers more

Have I add “debug” behind error.log? I currently have it without debug
and I
don’t see any information about the crash.

Thanks for your help

Greets,
Patschi

-----Ursprngliche Nachricht-----
Von: [email protected] [mailto:[email protected]] Im Auftrag
von
Lukas T.
Gesendet: Montag, 07. Jnner 2013 10:05
An: [email protected]
Betreff: RE: Webserver crashes sometimes - don’t know why

So you are actually running1.3.10? What modules have you build in, and
whats your exact configuration? What is the error logfile saying at
crash
time?

You may need to run a debug build of nginx to track this down further
[1]…

[1] A debugging log

isn’t working anymore. Then I restart nginx and everything is working

nginx mailing list
[email protected]
nginx Info Page

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

Patrik_Kernstock · January 7, 2013, 10:37am

Well, actually you have to recompile the source-code and configure it
with --with-debug AND enable the debug command behind error.log (see
link [1] at the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are
filesystem can cope with the amount of data logged.

[1] A debugging log

Patrik_Kernstock · January 7, 2013, 10:55am

Well, I would monitor the free space and the IO load carefully. If you
have a lot of traffic this may become a problem.

Patrik_Kernstock · January 7, 2013, 10:42am

I already built nginx with the --with-debug flag. Well, I have 10GB for
my
/var/log/ partition. I hope that this is enough
I don’t know when the next crash will be, so I have to wait. Thanks for
the
help. I it crashes again I will answer back.

-----Ursprngliche Nachricht-----
Von: [email protected] [mailto:[email protected]] Im Auftrag
von
Lukas T.
Gesendet: Montag, 07. Jnner 2013 10:37
An: [email protected]
Betreff: RE: AW: Webserver crashes sometimes - don’t know why

Well, actually you have to recompile the source-code and configure it
with
–with-debug AND enable the debug command behind error.log (see link [1]
at
the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are
filesystem can cope with the amount of data logged.

[1] A debugging log

I only have one installed module: headers more

Have I add “debug” behind error.log? I currently have it without debug
and I don’t see any information about the crash.

Thanks for your help

Greets,
Patschi

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

Patrik_Kernstock · January 7, 2013, 11:50pm

I doubt the dmesg output is enough for the developers to track down the
bug.

Apart from the actual debug log (which is crucial), can you provide
output of “nginx -V” and your configuration (remove confidential
information like IP addresses or domain names if you need to, but leave
the rest of it intact).

If you post all those informations, the developers should get a better
picture of what happens here.

Regards,

Lukas

Patrik_Kernstock · January 8, 2013, 4:49am

Hello!

On Mon, Jan 07, 2013 at 12:45:52PM +0100, Patrik Kernstock wrote:

[5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a130 error 4 in nginx[400000+a3000]
[5294688.683947] nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]

Seems to be a error in libc…

It’s highly unlikely to be an error in libc, segfaults in libc
usually heppen when libc functions are called with incorrect
arguments.

You need to obtain coredump and provide a backtrace, see
Debugging | NGINX for details, in paricular these two
sections:

Please note: it would be good idea to make sure you are able to
reproduce the problem without any 3rd party modules compiled in.

–
Maxim D.

Patrik_Kernstock · January 7, 2013, 12:46pm

I just found something interest in “dmesg” log:
[5294633.862284] __ratelimit: 20 callbacks suppressed
[5294633.862288] nginx[20568]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a1a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.659735] nginx[20569]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.818078] nginx[20571]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.819429] nginx[20581]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294635.313816] nginx[20589]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294635.402682] nginx[20590]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a0a8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294682.926163] nginx[20596]: segfault at 4a ip 00000000004459df sp
00007fff0260a0f0 error 4 in nginx[400000+a3000]
[5294685.155117] nginx[20595]: segfault at 4a ip 00000000004459df sp
00007fff0260a280 error 4 in nginx[400000+a3000]
[5294686.158466] nginx[21276]: segfault at 4a ip 00000000004459df sp
00007fff0260a130 error 4 in nginx[400000+a3000]
[5294688.683947] nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]

Seems to be a error in libc…

-----Ursprngliche Nachricht-----
Von: [email protected] [mailto:[email protected]] Im Auftrag
von
Lukas T.
Gesendet: Montag, 07. Jnner 2013 10:37
An: [email protected]
Betreff: RE: AW: Webserver crashes sometimes - don’t know why

Well, actually you have to recompile the source-code and configure it
with
–with-debug AND enable the debug command behind error.log (see link [1]
at
the bottom of the mail).

Obviously this will log a huge amounts of data, so make sure you are
filesystem can cope with the amount of data logged.

[1] A debugging log

I only have one installed module: headers more

Have I add “debug” behind error.log? I currently have it without debug
and I don’t see any information about the crash.

Thanks for your help

Greets,
Patschi

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

Patrik_Kernstock · January 9, 2013, 9:24pm

Thanks for your help, but I don’t really understand the part with “coredump”
and “backtrace”…

Recompile nginx with CFLAGS=“-g -O0” (for debugging symbols and
without compiler optimization). You can just prepend it to your
./configure line.
Before:
./configure --with-debug --with-ipv6 --with-http_flv_module
–with-http_mp4_module
After:
CFLAGS=“-g -O0” ./configure --with-debug --with-ipv6
–with-http_flv_module --with-http_mp4_module
compile nginx with “make” like you always do.
create a directory for the core files and make it readable from your
workers. For example: mkdir /nginx-core-dumps/ && chmod a+w
/nginx-core-dumps/
add this to your nginx configuration:
worker_rlimit_core 500M;
working_directory /nginx-core-dumps/;
(install and) start nginx and wait until it crashes. It should have
created the core-dump in /nginx-core-dumps/.
(install and) start gdb: gdb nginx /nginx-core-dumps/nginx.core
within gdb, run the commands “bt” and “backtrace full”, followed by a
“quit”.
Post the gdb output on this mailing list, the developers will analyze
it then.

I hope I didn’t missed anything, but I think this should be it. Example
at [1].

[1] http://pastebin.com/raw.php?i=NPjdQcVu

Patrik_Kernstock · January 9, 2013, 5:24am

Thanks for your help, but I don’t really understand the part with
“coredump”
and “backtrace”…

Thanks

-----Ursprngliche Nachricht-----
Von: [email protected] [mailto:[email protected]] Im Auftrag
von
Maxim D.
Gesendet: Dienstag, 08. Jnner 2013 04:49
An: [email protected]
Betreff: Re: AW: Webserver crashes sometimes - don’t know why

Hello!

On Mon, Jan 07, 2013 at 12:45:52PM +0100, Patrik Kernstock wrote:

[5294634.920149] nginx[20567]: segfault at aa ip 00007fdc5a44eb41 sp
00007fff0260a130 error 4 in nginx[400000+a3000] [5294688.683947]
nginx[21313]: segfault at 1 ip 00007fdc5a44eb41 sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]
[5294695.987059] nginx[21361]: segfault at 1193d ip 00007fdc5a44eb41
sp
00007fff0260a0c8 error 6 in libc-2.11.3.so[7fdc5a3cf000+159000]

Seems to be a error in libc…

It’s highly unlikely to be an error in libc, segfaults in libc usually
heppen when libc functions are called with incorrect arguments.

You need to obtain coredump and provide a backtrace, see
Debugging | NGINX for details, in paricular these two
sections:

Please note: it would be good idea to make sure you are able to
reproduce
the problem without any 3rd party modules compiled in.

–
Maxim D.

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

Patrik_Kernstock · January 26, 2013, 2:03am

Since weeks no crash. I don’t know why the crash happened sometimes…
Working perfekt as usual
If a next crash happens, I know what to do - Thank you all!

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx