We are using nginx version: 1.0.0 for pop-proxy. We have seen off-late
certain processes remaining in state whereby nginx becomes
non-responsive.
Below is the dump of ps showing the stuck process of nginx ==> R -->
4150:24.
i have shared the gdb dump and the backtrace of the stuck process.
38487 ? S 0:00 nginx: master process
/usr/local/nginx/sbin/nginx -c /etc/nginx/conf/nginx.conf
7044 ? R 4150:24 _ nginx: worker process
22408 ? S 0:03 /usr/local/apache3/sbin/httpd
22461 ? S 4:25 _ /usr/local/apache3/sbin/httpd
27699 ? D 0:00 | _
/home/httpd/dal/cgi-bin/dal/rmailajax.cgi
15853 ? S 0:06 _ /usr/local/apache3/sbin/httpd
17290 ? S 0:05 _ /usr/local/apache3/sbin/httpd
22796 ? S 0:04 /usr/sbin/httpd -DSSL
22801 ? S 0:29 _ /usr/sbin/httpd -DSSL
20605 ? S 4:05 _ /usr/sbin/httpd -DSSL
19391 ? S 2:33 _ /usr/sbin/httpd -DSSL
60469 ? S 0:59 _ /usr/sbin/httpd -DSSL
3437 ? S 0:35 _ /usr/sbin/httpd -DSSL
11200 ? S 0:30 _ /usr/sbin/httpd -DSSL
16597 ? S 0:19 _ /usr/sbin/httpd -DSSL
19556 ? S 0:09 _ /usr/sbin/httpd -DSSL
27218 ? S 0:00 _ /usr/sbin/httpd -DSSL
[root@xxx logs]# strace
[root@xxx logs]# gdb
GNU gdb Red Hat Linux (6.3.0.0-0.30.1rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for
details.
This GDB was configured as “i386-redhat-linux-gnu”.
(gdb) attach 7044
Attaching to process 7044
Reading symbols from /usr/local/nginx/sbin/nginx…done.
Using host libthread_db library “/lib/tls/libthread_db.so.1”.
Reading symbols from /lib/libcrypto.so.4…done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/tls/libc.so.6…done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2…done.
Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
Reading symbols from /usr/kerberos/lib/libkrb5.so.3…done.
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /usr/kerberos/lib/libcom_err.so.3…done.
Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
Reading symbols from /usr/kerberos/lib/libk5crypto.so.3…done.
Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
Reading symbols from /lib/libresolv.so.2…done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libdl.so.2…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1…done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/ld-linux.so.2…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2…done.
Loaded symbols for /lib/libnss_files.so.2
0x001dbb74 in _int_malloc () from /lib/tls/libc.so.6
(gdb) backtrace #0 0x001dbb74 in _int_malloc () from /lib/tls/libc.so.6 #1 0x001dc89c in _int_memalign () from /lib/tls/libc.so.6 #2 0x001db2c7 in memalign () from /lib/tls/libc.so.6 #3 0x001dd367 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805d614 in ngx_memalign (alignment=16, size=2746520, log=0x108)
at src/os/unix/ngx_alloc.c:56 #5 0x0804b603 in ngx_create_pool (size=256, log=0x93bd1a0) at
src/core/ngx_palloc.c:20 #6 0x0805be26 in ngx_event_accept (ev=0x93df338) at
src/event/ngx_event_accept.c:124 #7 0x0805b7f9 in ngx_event_process_posted (cycle=0x93bce28,
posted=0x8070e3c) at src/event/ngx_event_posted.c:39 #8 0x0805a8e1 in ngx_process_events_and_timers (cycle=0x93bce28) at
src/event/ngx_event.c:253 #9 0x0806051a in ngx_worker_process_cycle (cycle=0x93bce28, data=0x0)
at src/os/unix/ngx_process_cycle.c:800 #10 0x0805eb3c in ngx_spawn_process (cycle=0x93bce28, proc=0x8060465
<ngx_worker_process_cycle>, data=0x0, name=0x806b971 “worker process”,
respawn=17)
at src/os/unix/ngx_process.c:196 #11 0x0806023e in ngx_reap_children (cycle=0x93bce28) at
src/os/unix/ngx_process_cycle.c:617 #12 0x0805f8e1 in ngx_master_process_cycle (cycle=0x93bce28) at
src/os/unix/ngx_process_cycle.c:180 #13 0x08049fea in main (argc=3, argv=0x93bd5a4) at src/core/nginx.c:405
On Tue, Oct 18, 2011 at 12:31:49PM +0200, Sandesh K. wrote:
22801 ? S 0:29 _ /usr/sbin/httpd -DSSL
[root@xxx logs]# gdb
(gdb) attach 7044
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /lib/ld-linux.so.2…done.
at src/os/unix/ngx_alloc.c:56 #10 0x0805eb3c in ngx_spawn_process (cycle=0x93bce28, proc=0x8060465
<ngx_worker_process_cycle>, data=0x0, name=0x806b971 “worker process”,
respawn=17)
at src/os/unix/ngx_process.c:196 #11 0x0806023e in ngx_reap_children (cycle=0x93bce28) at
src/os/unix/ngx_process_cycle.c:617 #12 0x0805f8e1 in ngx_master_process_cycle (cycle=0x93bce28) at
src/os/unix/ngx_process_cycle.c:180 #13 0x08049fea in main (argc=3, argv=0x93bd5a4) at src/core/nginx.c:405
It seems that malloc() loops inside. This may be bug in nginx or glibc.
Could you show:
GNU gdb Red Hat Linux (6.3.0.0-0.30.1rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for
details.
This GDB was configured as “i386-redhat-linux-gnu”.
(gdb) attach 65233
Attaching to process 65233
Reading symbols from /usr/local/nginx/sbin/nginx…done.
Using host libthread_db library “/lib/tls/libthread_db.so.1”.
Reading symbols from /lib/tls/libpthread.so.0…done.
[Thread debugging using libthread_db enabled]
[New Thread -1218550432 (LWP 65233)]
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libcrypto.so.4…done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/tls/libc.so.6…done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2…done.
Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
Reading symbols from /usr/kerberos/lib/libkrb5.so.3…done.
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /usr/kerberos/lib/libcom_err.so.3…done.
Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
Reading symbols from /usr/kerberos/lib/libk5crypto.so.3…done.
Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
Reading symbols from /lib/libresolv.so.2…done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libdl.so.2…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1…done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2…done.
Loaded symbols for /lib/libnss_files.so.2
0x0031ebba in _int_malloc () from /lib/tls/libc.so.6
(gdb) backtrace #0 0x0031ebba in _int_malloc () from /lib/tls/libc.so.6 #1 0x0031f89c in _int_memalign () from /lib/tls/libc.so.6 #2 0x0031e2c7 in memalign () from /lib/tls/libc.so.6 #3 0x00320367 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805da38 in ngx_memalign (alignment=16, size=4069528, log=0x108) at
src/os/unix/ngx_alloc.c:56 #5 0x0804b917 in ngx_palloc_block (pool=0x8d0b9b0, size=52) at
src/core/ngx_palloc.c:184 #6 0x0804b892 in ngx_palloc (pool=0x8d0b9b0, size=52) at
src/core/ngx_palloc.c:138 #7 0x0804babf in ngx_pcalloc (pool=0x8d0b9b0, size=52) at
src/core/ngx_palloc.c:304 #8 0x0804d0b7 in ngx_create_temp_buf (pool=0x8d0b9b0, size=4096) at
src/core/ngx_buf.c:16 #9 0x08067508 in ngx_mail_proxy_init (s=0x8d0da20, peer=0x8d0c294) at
src/mail/ngx_mail_proxy_module.c:167 #10 0x080666e2 in ngx_mail_auth_http_process_headers (s=0x8d0da20,
ctx=0x8d12498) at src/mail/ngx_mail_auth_http_module.c:832 #11 0x0806606c in ngx_mail_auth_http_ignore_status_line (s=0x8d0da20,
ctx=0x8d12498) at src/mail/ngx_mail_auth_http_module.c:448 #12 0x08065f4b in ngx_mail_auth_http_read_handler (rev=0x3e2180) at
src/mail/ngx_mail_auth_http_module.c:331 #13 0x0806194e in ngx_rtsig_process_events (cycle=0x8cdb248,
timer=147634972, flags=1) at src/event/modules/ngx_rtsig_module.c:400 #14 0x0805ad0c in ngx_process_events_and_timers (cycle=0x8ccb228) at
src/event/ngx_event.c:245 #15 0x08060952 in ngx_worker_process_cycle (cycle=0x8ccb228, data=0x0)
at src/os/unix/ngx_process_cycle.c:801 #16 0x0805ef60 in ngx_spawn_process (cycle=0x8ccb228, proc=0x806089d
<ngx_worker_process_cycle>, data=0x0, name=0x806be29 “worker process”,
respawn=-3) at src/os/unix/ngx_process.c:196 #17 0x080601f6 in ngx_start_worker_processes (cycle=0x8ccb228, n=30,
type=-3) at src/os/unix/ngx_process_cycle.c:360 #18 0x0805fbe7 in ngx_master_process_cycle (cycle=0x8ccb228) at
src/os/unix/ngx_process_cycle.c:136 #19 0x0804a116 in main (argc=3, argv=0x8ccb9a4) at src/core/nginx.c:405
(gdb) quit
On Tue, Nov 22, 2011 at 06:51:56AM +0100, Sandesh K. wrote:
–without-mail_smtp_module
[…]
(gdb) backtrace #0 0x0031ebba in _int_malloc () from /lib/tls/libc.so.6 #1 0x0031f89c in _int_memalign () from /lib/tls/libc.so.6 #2 0x0031e2c7 in memalign () from /lib/tls/libc.so.6 #3 0x00320367 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805da38 in ngx_memalign (alignment=16, size=4069528, log=0x108) at
src/os/unix/ngx_alloc.c:56
The “size=4069528” looks really strange…
#5 0x0804b917 in ngx_palloc_block (pool=0x8d0b9b0, size=52) at
src/core/ngx_palloc.c:184
Could you please show
fr 5
p *pool
output from gdb?
#11 0x0806606c in ngx_mail_auth_http_ignore_status_line (s=0x8d0da20,
<ngx_worker_process_cycle>, data=0x0, name=0x806be29 “worker process”,
respawn=-3) at src/os/unix/ngx_process.c:196 #17 0x080601f6 in ngx_start_worker_processes (cycle=0x8ccb228, n=30,
type=-3) at src/os/unix/ngx_process_cycle.c:360 #18 0x0805fbe7 in ngx_master_process_cycle (cycle=0x8ccb228) at
src/os/unix/ngx_process_cycle.c:136 #19 0x0804a116 in main (argc=3, argv=0x8ccb9a4) at src/core/nginx.c:405
(gdb) quit
On Tue, Nov 22, 2011 at 06:51:56AM +0100, Sandesh K. wrote:
–without-mail_smtp_module
[…]
(gdb) backtrace #0 0x0031ebba in _int_malloc () from /lib/tls/libc.so.6 #1 0x0031f89c in _int_memalign () from /lib/tls/libc.so.6 #2 0x0031e2c7 in memalign () from /lib/tls/libc.so.6 #3 0x00320367 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805da38 in ngx_memalign (alignment=16, size=4069528, log=0x108) at
src/os/unix/ngx_alloc.c:56
The “size=4069528” looks really strange…
#5 0x0804b917 in ngx_palloc_block (pool=0x8d0b9b0, size=52) at
src/core/ngx_palloc.c:184
Could you please show
fr 5
p *pool
output from gdb?
#11 0x0806606c in ngx_mail_auth_http_ignore_status_line (s=0x8d0da20,
<ngx_worker_process_cycle>, data=0x0, name=0x806be29 “worker process”,
respawn=-3) at src/os/unix/ngx_process.c:196 #17 0x080601f6 in ngx_start_worker_processes (cycle=0x8ccb228, n=30,
type=-3) at src/os/unix/ngx_process_cycle.c:360 #18 0x0805fbe7 in ngx_master_process_cycle (cycle=0x8ccb228) at
src/os/unix/ngx_process_cycle.c:136 #19 0x0804a116 in main (argc=3, argv=0x8ccb9a4) at src/core/nginx.c:405
(gdb) quit
Maxim D.
Hello Maxim,
Please find the output.
43121 ? S 0:00 nginx: master process
/usr/local/nginx/sbin/nginx -c /etc/nginx/conf/nginx.conf
43123 ? S 1:58 _ nginx: worker process
43124 ? S 1:46 _ nginx: worker process
43127 ? S 2:02 _ nginx: worker process
43128 ? S 1:37 _ nginx: worker process
43130 ? S 1:50 _ nginx: worker process
43131 ? S 2:23 _ nginx: worker process
43132 ? S 2:44 _ nginx: worker process
43133 ? S 2:09 _ nginx: worker process
43134 ? S 1:47 _ nginx: worker process
43135 ? S 2:07 _ nginx: worker process
43129 ? R 866:11 _ nginx: worker process
43137 ? S 1:35 _ nginx: worker process
43138 ? S 1:28 _ nginx: worker process
43139 ? S 3:04 _ nginx: worker process
43140 ? S 1:42 _ nginx: worker process
43142 ? S 1:23 _ nginx: worker process
43143 ? S 2:24 _ nginx: worker process
7328 ? S 0:59 _ nginx: worker process
9444 ? S 0:46 _ nginx: worker process
57217 ? S 0:00 _ nginx: worker process
GNU gdb Red Hat Linux (6.3.0.0-1.132.EL3rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for
details.
This GDB was configured as “i386-redhat-linux-gnu”.
(gdb) attach 43129
Attaching to process 43129
Reading symbols from /usr/local/nginx/sbin/nginx…done.
Using host libthread_db library “/lib/tls/libthread_db.so.1”.
Reading symbols from /lib/tls/libpthread.so.0…done.
[Thread debugging using libthread_db enabled]
[New Thread -1218529952 (LWP 43129)]
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libcrypto.so.4…done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /lib/tls/libc.so.6…done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2…done.
Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
Reading symbols from /usr/kerberos/lib/libkrb5.so.3…done.
Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
Reading symbols from /usr/kerberos/lib/libcom_err.so.3…done.
Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
Reading symbols from /usr/kerberos/lib/libk5crypto.so.3…done.
Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
Reading symbols from /lib/libresolv.so.2…done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libdl.so.2…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libz.so.1…done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libnss_files.so.2…done.
Loaded symbols for /lib/libnss_files.so.2
0x00f36130 in _int_malloc () from /lib/tls/libc.so.6
(gdb) fr 5 #5 0x0804b917 in ngx_palloc_block (pool=0x90f58e0, size=128) at
src/core/ngx_palloc.c:184
184 m = ngx_memalign(NGX_POOL_ALIGNMENT, psize, pool->log);
(gdb) p *pool
$1 = {d = {last = 0x90f599c
“[email protected]\r\t\b\001”, end = 0x90f59e0
“\b\001”, next = 0x910cb20, failed = 0}, max = 216,
current = 0x90f58e0, chain = 0x0, large = 0x0, cleanup = 0x0, log =
0x90f5918}
(gdb) backtrace #0 0x00f36130 in _int_malloc () from /lib/tls/libc.so.6 #1 0x00f36b3c in _int_memalign () from /lib/tls/libc.so.6 #2 0x00f354f7 in memalign () from /lib/tls/libc.so.6 #3 0x00f37607 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805da3c in ngx_memalign (alignment=16, size=16751288, log=0x108)
at src/os/unix/ngx_alloc.c:56 #5 0x0804b917 in ngx_palloc_block (pool=0x90f58e0, size=128) at
src/core/ngx_palloc.c:184 #6 0x0804b892 in ngx_palloc (pool=0x90f58e0, size=128) at
src/core/ngx_palloc.c:138 #7 0x0804d0cf in ngx_create_temp_buf (pool=0x90f58e0, size=128) at
src/core/ngx_buf.c:21 #8 0x0806566a in ngx_mail_pop3_init_protocol (rev=0x8ebd4b4) at
src/mail/ngx_mail_pop3_handler.c:104 #9 0x0805bc4d in ngx_event_process_posted (cycle=0x8e951d8,
posted=0x8071ea0) at src/event/ngx_event_posted.c:39 #10 0x0805ad89 in ngx_process_events_and_timers (cycle=0x8e951d8) at
src/event/ngx_event.c:272 #11 0x08060956 in ngx_worker_process_cycle (cycle=0x8e951d8, data=0x0)
at src/os/unix/ngx_process_cycle.c:801 #12 0x0805ef64 in ngx_spawn_process (cycle=0x8e951d8, proc=0x80608a1
<ngx_worker_process_cycle>, data=0x0, name=0x806be29 “worker process”,
respawn=-3) at src/os/unix/ngx_process.c:196 #13 0x080601fa in ngx_start_worker_processes (cycle=0x8e951d8, n=20,
type=-3) at src/os/unix/ngx_process_cycle.c:360 #14 0x0805fbeb in ngx_master_process_cycle (cycle=0x8e951d8) at
src/os/unix/ngx_process_cycle.c:136 #15 0x0804a116 in main (argc=3, argv=0x8e95954) at src/core/nginx.c:405
On Wed, Nov 23, 2011 at 08:38:26AM +0100, Sandesh K. wrote:
#0 0x0031ebba in _int_malloc () from /lib/tls/libc.so.6
Could you please show
fr 5
p *pool
output from gdb?
[…]
(gdb) fr 5 #5 0x0804b917 in ngx_palloc_block (pool=0x90f58e0, size=128) at
src/core/ngx_palloc.c:184
184 m = ngx_memalign(NGX_POOL_ALIGNMENT, psize, pool->log);
(gdb) p *pool
$1 = {d = {last = 0x90f599c
“[email protected]\r\t\b\001”, end = 0x90f59e0
“\b\001”, next = 0x910cb20, failed = 0}, max = 216,
current = 0x90f58e0, chain = 0x0, large = 0x0, cleanup = 0x0, log =
0x90f5918}
Ok, it looks like pool is ok (pool->d.end - pool == 0x100).
Probably the cause of strange arguments is either corrupted stack
or some compiler optimizations.
(gdb) backtrace #0 0x00f36130 in _int_malloc () from /lib/tls/libc.so.6 #1 0x00f36b3c in _int_memalign () from /lib/tls/libc.so.6 #2 0x00f354f7 in memalign () from /lib/tls/libc.so.6 #3 0x00f37607 in posix_memalign () from /lib/tls/libc.so.6 #4 0x0805da3c in ngx_memalign (alignment=16, size=16751288, log=0x108)
at src/os/unix/ngx_alloc.c:56 #5 0x0804b917 in ngx_palloc_block (pool=0x90f58e0, size=128) at
src/core/ngx_palloc.c:184
For now I tend to think the problem is somewhere in (g)libc. You may
want to update it, and if it doesn’t help - recompile with debug
and dig into it further.
Maxim D.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.