Worker process exited on signal 7

Server NGINX
1

Hello everyone,

I started to use nginx insted of lighttpd for a few days ago. It
serves only static content (thumbnail pictures) and performs well, but
yesterday it stopped working, and I got the following messages on the
error.log.

2011/01/06 17:54:30 [alert] 7542#0: worker process 31959 exited on
signal 7

I didn’t found anything about this error message on the net. The error
logging level is “crit” because I don’t have enough storage to keep
all the “file not found” errors :slight_smile: (The logging goes to a tmpfs
partiton).
So I restarted the lighttpd on that server, and it works now, so I
didn’t think that it is a hardware error.
Here is th the output of nginx -V:

nginx version: nginx/0.8.54
built by gcc 4.3.3 (Ubuntu 4.3.3-5ubuntu4)
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx
–error-log-path=var/log/error.log --http-log-path=var/log/access.log
–http-client-body-temp-path=/opt/nginx/var/cache/client_body_temp
–http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
–http-fastcgi-temp-path=/opt/nginx/var/cache/fastcgi_temp
–without-http_uwsgi_module --without-http_scgi_module
–with-http_ssl_module --with-openssl=…/openssl-0.9.8k
–user=www-data --group=www-data --pid-
path=/opt/nginx/var/run/nginx.pid
–conf-path=/opt/nginx/etc/nginx.conf --with-http_stub_status_module

It is a x86_64 system.

At this machine the www data files are on two SSD-s, and one of those
SSD-s are bad, bacause not all ‘secors’ (hey what is the correct term
for sectors on SSD?) can be read. Can it cause this type of error?

Thanks,
Balazs

–
“Overhead, without any fuss, the stars were going out.”

2

Hello!

On Fri, Jan 07, 2011 at 10:35:31AM +0100, Balazs Pocze wrote:

Hello everyone,

I started to use nginx insted of lighttpd for a few days ago. It
serves only static content (thumbnail pictures) and performs well, but
yesterday it stopped working, and I got the following messages on the
error.log.

2011/01/06 17:54:30 [alert] 7542#0: worker process 31959 exited on signal 7

I didn’t found anything about this error message on the net.

It looks like signal 7 on linux x86 is SIGBUS (oh, linux has
architecture-dependant signal numbers… really intresting design
decision).

It usually generated on an attempt to access memory which doesn’t
exists, and may be generated on various hardware problems as well.
Under linux this includes uncorrectable memory errors as seen via
MCA and problems reading pages after page fault (e.g. swap disk
died, or mmap’ed file read failed).

The error
logging level is “crit” because I don’t have enough storage to keep
all the “file not found” errors :slight_smile: (The logging goes to a tmpfs
partiton).

Just a side note: you may want to use “log_not_found off;” instead.

–http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
SSD-s are bad, bacause not all ‘secors’ (hey what is the correct term
for sectors on SSD?) can be read. Can it cause this type of error?

Obviously it can if you also have swap on these disks and/or read
files via mmap(). While nginx doesn’t use mmap() to read files,
something like may be used internally by OS for e.g. sendfile(),
not sure.

This may also be a problem in nginx which causes it to access
nonexisting memory regions.

To further debug the problem it should be helpful to obtain
coredump and backtrace, as well as show full config. See
Debugging | NGINX for basic instructions.

Maxim D.

3

2011/1/7 Maxim D. [email protected]:

Under linux this includes uncorrectable memory errors as seen via
MCA and problems reading pages after page fault (e.g. swap disk
died, or mmap’ed file read failed).

Maybe yes. But I have no proof, and the problem is that: with lighttpd
i don’t have this errors. So I don’t know what to think about it.

The error
logging level is “crit” because I don’t have enough storage to keep
all the “file not found” errors :slight_smile: (The logging goes to a tmpfs
partiton).

Just a side note: you may want to use “log_not_found off;” instead.

It is not a good sollution for me, because I need the 404 errors. (for
fact, on this server I only need the 404 errors to get (re)generated
the thumbnail pictures, I don’t need these errors on error log, just
in access.log (and a feature will be welcome, if I can direct the 404
logs into one log file, and 200’s to an another.)

–http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
SSD-s are bad, bacause not all ‘secors’ (hey what is the correct term
To further debug the problem it should be helpful to obtain
coredump and backtrace, as well as show full config. See
Debugging | NGINX for basic instructions.

there’s no swap space on this machine.

–
“Overhead, without any fuss, the stars were going out.”

4

2011/1/10 Maxim D. [email protected]:

architecture-dependant signal numbers… really intresting design

This may be hardware-related error. No errors in lighttpd may
indicate it’s lucky (in case of memory errors) or using other
system interfaces (in case of paging-related problems).

yeah, it can be. I’ll put an strace process to one of the workers, and
look into the logs later.

in access.log (and a feature will be welcome, if I can direct the 404
logs into one log file, and 200’s to an another.)

“log_not_found off;” does exactly what you need - it just disables
logging “file not found” errors to error_log.

yep, I was wrong, you have right. thx.
my bad.

–http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
SSD-s are bad, bacause not all ‘secors’ (hey what is the correct term
To further debug the problem it should be helpful to obtain
coredump and backtrace, as well as show full config. See
Debugging | NGINX for basic instructions.

there’s no swap space on this machine.

It’s impossible to tell anything without further info. Please
follow provided instructions to obtain coredump.

yep I’ll do that.

thx.

–
“Overhead, without any fuss, the stars were going out.”

5

Hello!

On Mon, Jan 10, 2011 at 01:38:53PM +0100, Balazs Pocze wrote:

error.log.
exists, and may be generated on various hardware problems as well.
Under linux this includes uncorrectable memory errors as seen via
MCA and problems reading pages after page fault (e.g. swap disk
died, or mmap’ed file read failed).

Maybe yes. But I have no proof, and the problem is that: with lighttpd
i don’t have this errors. So I don’t know what to think about it.

This may be hardware-related error. No errors in lighttpd may
indicate it’s lucky (in case of memory errors) or using other
system interfaces (in case of paging-related problems).

in access.log (and a feature will be welcome, if I can direct the 404
logs into one log file, and 200’s to an another.)

“log_not_found off;” does exactly what you need - it just disables
logging “file not found” errors to error_log.

–http-proxy-temp-path=/opt/nginx/var/cache/proxy_temp
SSD-s are bad, bacause not all ‘secors’ (hey what is the correct term
To further debug the problem it should be helpful to obtain
coredump and backtrace, as well as show full config. See
Debugging | NGINX for basic instructions.

there’s no swap space on this machine.

It’s impossible to tell anything without further info. Please
follow provided instructions to obtain coredump.

Maxim D.