Nginx segfault around ~600 requests per second (reverse proxy)

Server NGINX
1

Hello,

I have noticed that we seem to consistently get an nginx segault when
reaching around 600 requests per second (reverse proxy).

Here is the nginx build information:
$ nginx -V
nginx version: nginx/0.8.29
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
TLS SNI support disabled
configure arguments:
–add-module=…/agentzh-chunkin-nginx-module-cb610a5
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_sub_module
–with-http_dav_module --with-http_flv_module
–with-http_gzip_static_module --with-http_stub_status_module
–with-http_perl_module

Here is the segfault information:
Jan 17 12:51:31 util1 kernel: nginx[27888]: segfault at 0000000000000008
rip 0000000000429061 rsp 00007fffcc260500 error 6
Jan 24 16:21:33 util1 kernel: nginx[3835]: segfault at 0000000000000008
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:24 util1 kernel: nginx[8956]: segfault at 0000000000000008
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:26 util1 kernel: nginx[8982]: segfault at 0000000000000008
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:27 util1 kernel: nginx[9026]: segfault at 0000000000000008
rip 0000000000429061 rsp 00007fffcc260550 error 6

Does anyone know why this might be occurring, or any helpful information
we can provide to better track it down?

Thanks very much in advance.

J

Posted at Nginx Forum:

2

Hi,

yomuppet wrote:

Have you tried compiling without the chunkin module? AFAIK it’s still
experimental.

Here is the segfault information:
Jan 17 12:51:31 util1 kernel: nginx[27888]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260500 error 6
Jan 24 16:21:33 util1 kernel: nginx[3835]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:24 util1 kernel: nginx[8956]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:26 util1 kernel: nginx[8982]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:27 util1 kernel: nginx[9026]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6

Does anyone know why this might be occurring, or any helpful information we can provide to better track it down?

If compiling without the chunkin module doesn’t isolate the problem,
perform a backtrace.

Marcus.

3

On Tue, Jan 26, 2010 at 7:34 AM, yomuppet [email protected] wrote:

Does anyone know why this might be occurring, or any helpful information we can provide to better track it down?

It may be a bug in ngx_chunkin. I’ve just sent you a mail off-list
explaining how to produce a backtrace for me to investigate this issue
:slight_smile:

Thanks for reporting it!
-agentzh

4

Hello,

Sorry for the delayed reply, it has been taking longer to hit a segfault
(which is good), but still hitting some. Here is some backtrace output:

(gdb) bt
#0 0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050,
timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2 0x0000000000421f08 in ngx_process_events_and_timers
(cycle=0xc1e1050) at src/event/ngx_event.c:245
#3 0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050,
data=) at src/os/unix/ngx_process_cycle.c:795
#4 0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050,
proc=0x427678 , data=0x0, name=0x46f721 “worker process”, respawn=2)
at src/os/unix/ngx_process.c:196
#5 0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at
src/os/unix/ngx_process_cycle.c:612
#6 0x000000000040edfa in main (argc=22, argv=0xc1dff30) at
src/core/nginx.c:399
(gdb) bt
#0 0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050,
timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2 0x0000000000421f08 in ngx_process_events_and_timers
(cycle=0xc1e1050) at src/event/ngx_event.c:245
#3 0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050,
data=) at src/os/unix/ngx_process_cycle.c:795
#4 0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050,
proc=0x427678 , data=0x0, name=0x46f721 “worker process”, respawn=2)
at src/os/unix/ngx_process.c:196
#5 0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at
src/os/unix/ngx_process_cycle.c:612
#6 0x000000000040edfa in main (argc=22, argv=0xc1dff30) at
src/core/nginx.c:399
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768,
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131 iov->iov_len += (size_t) size;
(gdb) bt
#0 ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768,
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
#1 0x000000000041186a in ngx_chain_writer (data=, in=) at
src/core/ngx_output_chain.c:657
#2 0x0000000000412056 in ngx_output_chain (ctx=0xc7bd0b0, in=0x0) at
src/core/ngx_output_chain.c:205
#3 0x000000000044540c in ngx_http_upstream_send_request (r=0xca55610,
u=0xc7bd020) at src/http/ngx_http_upstream.c:1321
#4 0x0000000000445ef8 in ngx_http_upstream_send_request_handler (r=0x0,
u=0x0) at src/http/ngx_http_upstream.c:1422
#5 0x000000000044399f in ngx_http_upstream_handler (ev=) at
src/http/ngx_http_upstream.c:874
#6 0x00000000004220be in ngx_event_process_posted (cycle=, posted=0x0)
at src/event/ngx_event_posted.c:39
#7 0x0000000000421f80 in ngx_process_events_and_timers
(cycle=0xc1e1050) at src/event/ngx_event.c:272
#8 0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050,
data=) at src/os/unix/ngx_process_cycle.c:795
#9 0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050,
proc=0x427678 , data=0x0, name=0x46f721 “worker process”, respawn=2)
at src/os/unix/ngx_process.c:196
#10 0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at
src/os/unix/ngx_process_cycle.c:612
#11 0x000000000040edfa in main (argc=22, argv=0xc1dff30) at
src/core/nginx.c:399

(gdb) c
Continuing.
p iov
p prev
p cl->buf
p cl->buf->pos
up
detach

Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b21152bed40, in=0x48bfaf8,
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131 iov->iov_len += (size_t) size;
(gdb) p iov
$1 = (struct iovec *) 0x0
(gdb) p prev
$2 = (u_char *) 0x0
(gdb) p cl->buf
Cannot access memory at address 0x0
(gdb) p cl->buf->pos
Cannot access memory at address 0x0
(gdb) up
#1 0x000000000041186a in ngx_chain_writer (data=, in=) at
src/core/ngx_output_chain.c:657
657 ctx->out = c->send_chain(c, ctx->out, ctx->limit);
(gdb) detach
LND: Sending signal 11 to Thread 0x2b2114c6b750 (LWP 21815)
Detaching from program: /opt/nginx/sbin/nginx, process 21815

If anyone has further insight, would be much appreciated. Thanks.

J

Posted at Nginx Forum: