# Weird crash over night with nginx

**URL:** <https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490>\
**Category:** NGINX\
**Created:** [March 21, 2010, 12:00pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490 "2010-03-21T12:00:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 21, 2010, 12:00pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/1 "2010-03-21T12:00:39Z")

</div>

I got some weird crash of nginx in the kernel logs. The problem is I  
cant find out why or what happened. It happened I suppose over night.

This is from the kernel log:

nginx[31486]: segfault at c4 ip 080aacb5 sp bfd79b60 error 4 in  
nginx[8048000+73000]  
nginx[31484]: segfault at c4 ip 080aacb5 sp bfd79b60 error 4 in  
nginx[8048000+73000]  
nginx[1528]: segfault at c4 ip 080aacb5 sp bfd79bc0 error 4 in  
nginx[8048000+73000]  
nginx[1568]: segfault at c4 ip 080aacb5 sp bfd79bc0 error 4 in  
nginx[8048000+73000]

and from error\_log:  
2010/03/21 09:42:35 [notice] 31483#0: signal 17 (SIGCHLD) received  
2010/03/21 09:42:35 [alert] 31483#0: worker process 31486 exited on  
signal 11  
2010/03/21 09:42:35 [notice] 31483#0: start worker process 1528  
2010/03/21 09:42:35 [notice] 31483#0: signal 29 (SIGIO) received  
2010/03/21 09:43:03 [notice] 31483#0: signal 17 (SIGCHLD) received  
2010/03/21 09:43:03 [alert] 31483#0: worker process 31484 exited on  
signal 11  
2010/03/21 09:43:03 [notice] 31483#0: start worker process 1568  
2010/03/21 09:43:03 [notice] 31483#0: signal 29 (SIGIO) received  
2010/03/21 09:43:09 [notice] 31483#0: signal 17 (SIGCHLD) received  
2010/03/21 09:43:09 [alert] 31483#0: worker process 1528 exited on  
signal 11  
2010/03/21 09:43:09 [notice] 31483#0: start worker process 1582  
2010/03/21 09:43:09 [notice] 31483#0: signal 29 (SIGIO) received  
2010/03/21 09:45:09 [notice] 31483#0: signal 17 (SIGCHLD) received  
2010/03/21 09:45:09 [alert] 31483#0: worker process 1568 exited on  
signal 11  
2010/03/21 09:45:09 [notice] 31483#0: start worker process 1757  
2010/03/21 09:45:09 [notice] 31483#0: signal 29 (SIGIO) received  
2010/03/21 12:00:13 [notice] 31483#0: signal 15 (SIGTERM) received,  
exiting

Does any one know what could be the issue?

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 22, 2010, 4:14pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/2 "2010-03-22T16:14:33Z")

</div>

And how am I suppose to do the backtrace, I mean I cant reproduce the  
error or you just need a simple backtrace?

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 21, 2010, 7:57pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/3 "2010-03-21T19:57:43Z")

</div>

Hello!

On Sun, Mar 21, 2010 at 12:00:39PM +0100, Robert G. wrote:

> I got some weird crash of nginx in the kernel logs. The problem is I  
> cant find out why or what happened. It happened I suppose over night.

[…]

> Does any one know what could be the issue?

If you want this to be investigated you may want to provide:

1. nginx -V output

2. config

3. backtrace as obtained from coredump via gdb

Maxim D.

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 25, 2010, 8:51am UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/4 "2010-03-25T08:51:17Z")

</div>

Sorry, I cant reproduce the backtrace, but here is the config and nginx  
-V:

System: Ubuntu Linux 9.10 Server with kernel: 2.6.33-server (custom  
kernel build)

nginx version: nginx/0.8.34  
TLS SNI support enabled  
configure arguments: --prefix=/applications/nginx  
–conf-path=/etc/nginx/nginx.conf --with-http\_ssl\_module  
–with-http\_realip\_module --with-http\_addition\_module  
–with-http\_flv\_module --with-http\_gzip\_static\_module  
–with-http\_sub\_module --http-log-path=/var/log/nginx/access\_log  
–with-http\_perl\_module --user=www-data --group=www-data  
–http-fastcgi-temp-path=/applications/nginx/tmp/fastcgi  
–http-client-body-temp-path=/applications/nginx/tmp/client  
–http-proxy-temp-path=/applications/nginx/tmp/proxy  
–pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error\_log  
–with-sha1=/usr/lib --with-md5=/usr/lib --with-file-aio

nginx config:

user www-data;  
worker\_processes 2;  
worker\_cpu\_affinity 0101 1010;  
pid /var/run/nginx.pid;  
error\_log /var/log/nginx/error\_log info;

events {  
worker\_connections 1024;  
use epoll;  
}

http {  
include mime.types;  
default\_type application/octet-stream;  
log\_format main '$remote\_addr - $remote\_user [$time\_local]  
“$request” ’  
'$status $body\_bytes\_sent  
“$http\_referer” ’  
‘"$http\_user\_agent"  
“$http\_x\_forwarded\_for”’;  
client\_body\_timeout 60;  
client\_header\_timeout 60;  
send\_timeout 60;  
server\_tokens off;  
server\_names\_hash\_bucket\_size 256;  
server\_names\_hash\_max\_size 512;  
aio on;  
directio 1m;  
output\_buffers 1 128k;  
sendfile off;  
tcp\_nopush off;  
tcp\_nodelay off;  
keepalive\_timeout 0;  
gzip on;  
gzip\_comp\_level 9;  
gzip\_buffers 16 8k;  
gzip\_http\_version 1.0;  
gzip\_min\_length 1024;  
gzip\_vary on;  
gzip\_proxied off;  
gzip\_disable msie6;  
gzip\_types text/plain text/css text/xml text/javascript  
application/x-javascript application/xml application/xml+rss;

include /etc/nginx/site-hosts/_;  
include /etc/nginx/site-users/_;  
include /etc/nginx/site-virtual/\*;

}

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 25, 2010, 10:37am UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/5 "2010-03-25T10:37:56Z")

</div>

Hello!

On Thu, Mar 25, 2010 at 08:51:17AM +0100, Robert G. wrote:

> –with-http\_realip\_module --with-http\_addition\_module

[…]

> include /etc/nginx/site-hosts/_;  
> include /etc/nginx/site-users/_;  
> include /etc/nginx/site-virtual/\*;

You may want to provide included files as well, they are parts of  
config. Or at least samples if they are identical.

Maxim D.

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 25, 2010, 7:41pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/6 "2010-03-25T19:41:34Z")

</div>

They are over 300 of them, you sure?

---

<div class="post-metadata">

**Author:** ![sovok](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/b5a626/32.png) [@sovok](https://www.ruby-forum.com/u/sovok)\
**Post date:** [March 22, 2010, 5:00pm UTC](https://www.ruby-forum.com/t/weird-crash-over-night-with-nginx/185490/7 "2010-03-22T17:00:53Z")

</div>

Hello!

On Mon, Mar 22, 2010 at 04:14:33PM +0100, Robert G. wrote:

> And how am I suppose to do the backtrace, I mean I cant reproduce the  
> error or you just need a simple backtrace?

You have to configure your system to dump cores, and once you’ll  
have coredump run

gdb /path/to/nginx /path/to/nginx.core

then in gdb:

bt

Depending on your OS procedure to enable core dumps is different,  
but it may be simplified using nginx’s own global direcitives  
worker\_rlimit\_core and working\_directory, e.g.:

```
worker_rlimit_core 500m;
working_directory /path/to/corefiles;

```

nginx must have write access to ‘/path/to/corefiles’ directory.

Note well: it’s good idea to make sure your nginx binary isn’t  
stripped (e.g. via file(1) command).

If you are unable to reproduce segmentation fault and therefore  
unable to obtain coredump - it’s still good idea to provide nginx  
-V output and your config. There is a chance that segmentation  
fault you’ve seen was already fixed or caused by known bad  
configuration.

In nginx 0.8.34 I’m currently aware of at least 4 possible  
segfaults: 2 caused by bugs (in fastcgi stderr handling and in  
subrequest loop handling; patches are available) and 2 caused by  
known bad configurations (error\_page 400 redirected to named  
location, “if” usage as outlined in IfIsEvil wiki page). Not even  
talking about older versions.

Maxim D.