Nginx randomly crashes

Server NGINX
1

I’ve got a problem with Nginx running on Ubuntu 12.10.
I’m running it mainly as a reverse proxy and there is no high load on
the
machine.
It randomly crashes without any helpful log info (or at least I think
so).

Here’s the error log

2013/01/31 09:19:03 [debug] 15238#0: *10555 event timer del: 68:
1359620363778
2013/01/31 09:19:03 [debug] 15238#0: *10555 generic phase: 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 rewrite phase: 1
2013/01/31 09:19:03 [debug] 15238#0: 10555 http script regex: "^(.)"
2013/01/31 09:19:03 [notice] 15238#0: 10555 "^(.)" matches “/”,
client:
192.168.2.42, server: abc.def.com, request: “HEAD / HTTP/1.1”, host: "
abc.def.com"
2013/01/31 09:19:03 [debug] 15238#0: *10555 http script copy:
“https://abc.def.com”
2013/01/31 09:19:03 [debug] 15238#0: *10555 http script regex end
2013/01/31 09:19:03 [notice] 15238#0: *10555 rewritten redirect:
“https://abc.def.com”, client: 192.168.2.42, server: abc.def.com, reques
t: “HEAD / HTTP/1.1”, host: “abc.def.com”
2013/01/31 09:19:03 [debug] 15238#0: *10555 http finalize request: 301,
“/?”
a:1, c:1
2013/01/31 09:19:03 [debug] 15238#0: *10555 http special response: 301,
“/?”
2013/01/31 09:19:03 [debug] 15238#0: *10555 http set discard body
2013/01/31 09:19:03 [debug] 15238#0: *10555 xslt filter header
2013/01/31 09:19:03 [debug] 15238#0: *10555 HTTP/1.1 301 Moved
Permanently
Server: nginx/1.2.6
Date: Thu, 31 Jan 2013 08:19:03 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
Location: https://abc.def.com

2013/01/31 09:19:03 [debug] 15238#0: *10555 write new buf t:1 f:0
00000000027A0868, pos 00000000027A0868, size: 200 file: 0, size: 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 http write filter: l:1 f:0
s:200
2013/01/31 09:19:03 [debug] 15238#0: *10555 http write filter limit 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 writev: 200
2013/01/31 09:19:03 [debug] 15238#0: *10555 http write filter
0000000000000000
2013/01/31 09:19:03 [debug] 15238#0: *10555 http finalize request: 0,
“/?”
a:1, c:1
2013/01/31 09:19:03 [debug] 15238#0: *10555 set http keepalive handler
2013/01/31 09:19:03 [debug] 15238#0: *10555 http close request
2013/01/31 09:19:03 [debug] 15238#0: *10555 http log handler
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 000000000279FBA0,
unused:
360
2013/01/31 09:19:03 [debug] 15238#0: *10555 event timer add: 68:
30000:1359620373779
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 00000000027DE460
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 0000000002652060
2013/01/31 09:19:03 [debug] 15238#0: *10555 hc free: 0000000000000000 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 hc busy: 0000000000000000 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 tcp_nodelay
2013/01/31 09:19:03 [debug] 15238#0: *10555 reusable connection: 1
2013/01/31 09:19:03 [debug] 15238#0: *10555 post event 00000000025B5210
2013/01/31 09:19:03 [debug] 15238#0: *10555 delete posted event
00000000025B5210
2013/01/31 09:19:03 [debug] 15238#0: *10555 http keepalive handler
2013/01/31 09:19:03 [debug] 15238#0: *10555 malloc:
0000000002652060:8192
2013/01/31 09:19:03 [debug] 15238#0: *10555 recv: fd:68 -1 of 8192
2013/01/31 09:19:03 [debug] 15238#0: *10555 recv() not ready (11:
Resource
temporarily unavailable)
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 0000000002652060
2013/01/31 09:19:03 [debug] 15238#0: *10555 http keepalive handler
2013/01/31 09:19:03 [debug] 15238#0: *10555 malloc:
0000000002652060:8192
2013/01/31 09:19:03 [debug] 15238#0: *10555 recv: fd:68 0 of 8192
2013/01/31 09:19:03 [info] 15238#0: *10555 client 192.168.2.42 closed
keepalive connection
2013/01/31 09:19:03 [debug] 15238#0: *10555 close http connection: 68
2013/01/31 09:19:03 [debug] 15238#0: *10555 event timer del: 68:
1359620373779
2013/01/31 09:19:03 [debug] 15238#0: *10555 reusable connection: 0
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 0000000002652060
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 0000000000000000
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 000000000283EBF0,
unused:
8
2013/01/31 09:19:03 [debug] 15238#0: *10555 free: 00000000027F9CF0,
unused:
128
2013/01/31 09:24:49 [alert] 15237#0: worker process 15238 exited on
signal
9

At 09:24 I restarted the service because it was hanging.
This is some system info about tcp connections:

Thu Jan 31 09:18:01 CET 2013
14 ESTABLISHED
3 FIN_WAIT2
6 LISTEN
21 TIME_WAIT

Thu Jan 31 09:19:01 CET 2013
28 ESTABLISHED
6 LISTEN
12 TIME_WAIT

Thu Jan 31 09:20:01 CET 2013
13 CLOSE_WAIT
28 ESTABLISHED
6 LISTEN

Thu Jan 31 09:21:01 CET 2013
35 CLOSE_WAIT
36 ESTABLISHED
8 LISTEN

Thu Jan 31 09:22:01 CET 2013
46 CLOSE_WAIT
35 ESTABLISHED
7 LISTEN

Thu Jan 31 09:23:01 CET 2013
80 CLOSE_WAIT
33 ESTABLISHED
7 LISTEN

Thu Jan 31 09:24:01 CET 2013
128 CLOSE_WAIT
26 ESTABLISHED
7 LISTEN
8 SYN_RECV

Thu Jan 31 09:25:02 CET 2013
26 ESTABLISHED
1 FIN_WAIT1
7 LISTEN
4 TIME_WAIT

During that time I tried to connect with curl and this is the output:
roberto@t500:~> curl -I -L -k -m 30 http://abc.def.com

curl: (28) Operation timed out after 30001 milliseconds with 0 bytes
received

I upgraded Ubuntu package from 1.2.1 to 1.2.6 but no result. In
/var/log/syslog I have no info about this.

What can I do?

Posted at Nginx Forum:

2

Hello!

On Thu, Jan 31, 2013 at 03:48:31AM -0500, rg00 wrote:

I’ve got a problem with Nginx running on Ubuntu 12.10.
I’m running it mainly as a reverse proxy and there is no high load on the
machine.
It randomly crashes without any helpful log info (or at least I think so).

(Note: this looks like a hang, not a crash. There is a
difference.)

[…]

At 09:24 I restarted the service because it was hanging.
This is some system info about tcp connections:

What nginx -V shows? As a first step I would recommend trying to
reproduce the problem without any 3rd party modules/patches
compiled in. You may also find some helpful tips about debugging
here:

–
Maxim D.

3

This is the output of nginx -V:

nginx version: nginx/1.2.6
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-log-path=/var/log/nginx/access.log
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi
–lock-path=/var/lock/nginx.lock
–pid-path=/run/nginx.pid --with-pcre-jit --with-debug
–with-http_addition_module --with-http_dav_module
–with-http_geoip_module
–with-http_gzip_static_module --with-http_image_filter_module
–with-http_realip_module --with-http_stub_status_module
–with-http_ssl_module --with-http_sub_module --with-http_xslt_module
–with-ipv6 --with-sha1=/usr/include/openssl
–with-md5=/usr/include/openssl
–with-mail --with-mail_ssl_module
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-auth-pam
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-echo
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-upstream-fair
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-dav-ext-module

Posted at Nginx Forum:

4

I compiled 1.2.6 version and switched to it, here’s my new nginx -V:

nginx version: nginx/1.2.6
built by gcc 4.7.2 (Ubuntu/Linaro 4.7.2-2ubuntu1)
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx-1.2.6 --with-http_flv_module
–with-http_ssl_module --with-http_gzip_static_module
–add-module=/root/src/ngx_http_auth_pam_module-1.2 --with-debug

but same problem… I’m really
really
confused…

Posted at Nginx Forum:

5

Hello!

On Thu, Jan 31, 2013 at 06:38:27AM -0500, rg00 wrote:

–http-scgi-temp-path=/var/lib/nginx/scgi
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-upstream-fair
–add-module=/build/buildd/nginx-1.2.6/debian/modules/nginx-dav-ext-module

Well, basic suggestion is the same: try to reproduce the problem
without any 3rd party modules/patches compiled in.

If you use geoip module, please also make sure it’s database isn’t
corrupted (or just try without it), as MaxMind’s geoip library is
known to do bad things if database is corrupted.

–
Maxim D.

6

I really need ldap authentication, is there an alternative to that
module?

Posted at Nginx Forum:

7

On 02/01/2013 12:07 PM, rg00 wrote:

I really need ldap authentication, is there an alternative to that module?

A quick google gave:

Regards,
Patrick

8

On Friday 01 February 2013 13:45:05 rg00 wrote:

but same problem… I’m really really
confused…

It is very likely that the cause of your problem is the http_auth_pam
3rd-party
module. This module is known as being able to block worker processes
when it
used.

wbr, Valentin V. Bartenev

–

http://nginx.org/en/donation.html

9

On Friday 01 February 2013 15:11:58 Patrick L. wrote:

On 02/01/2013 12:07 PM, rg00 wrote:

I really need ldap authentication, is there an alternative to that
module?

A quick google gave:

GitHub - kvspb/nginx-auth-ldap: LDAP authentication module for nginx
Does nginx support authentification like LDAP or mysql?

Unfortunately, this module also uses blocking I/O operations. So, you
should be ready for poor performance and arbitrary hangs.

wbr, Valentin V. Bartenev

–

http://nginx.org/en/donation.html

10

Compiled nginx without auth_pam and using it as a reverse proxy. Now I’m
monitoring and hoping there are no more service crashes.
I’m using Apache to manage ldap authentication on certain locations.

Posted at Nginx Forum:

11

Hello!

On Fri, Feb 01, 2013 at 12:11:58PM +0100, Patrick L. wrote:

On 02/01/2013 12:07 PM, rg00 wrote:

I really need ldap authentication, is there an alternative to that module?

A quick google gave:

GitHub - kvspb/nginx-auth-ldap: LDAP authentication module for nginx

This one blocks as well.

There is no non-blocking API in PAM, hence correct nginx module to
use PAM for authentication is something impossible to write. And
LDAP client libraries out there are blocking too, so writing LDAP
authentication module isn’t something simple.

Does nginx support authentification like LDAP or mysql?

Yep, using X-Accel-Redirect is a good way to handle any
needed authentication/authorization.

Alternatively, one may use auth request module:

http://mdounin.ru/hg/ngx_http_auth_request_module

I wrote it once tired reading questions about PAM/LDAP/whatever
authentication modules and various blocking solutions people write
and try to use. Compared to X-Accel-Redirect it is simplier for
general use.

–
Maxim D.