Forum: NGINX nginx segfault around ~600 requests per second (reverse proxy)

Hello,

I have noticed that we seem to consistently get an nginx segault when 
reaching around 600 requests per second (reverse proxy).

Here is the nginx build information:
$ nginx -V
nginx version: nginx/0.8.29
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-44)
TLS SNI support disabled
configure arguments: 
--add-module=../agentzh-chunkin-nginx-module-cb610a5 
--with-http_ssl_module --with-http_realip_module 
--with-http_addition_module --with-http_sub_module 
--with-http_dav_module --with-http_flv_module 
--with-http_gzip_static_module --with-http_stub_status_module 
--with-http_perl_module

Here is the segfault information:
Jan 17 12:51:31 util1 kernel: nginx[27888]: segfault at 0000000000000008 
rip 0000000000429061 rsp 00007fffcc260500 error 6
Jan 24 16:21:33 util1 kernel: nginx[3835]: segfault at 0000000000000008 
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:24 util1 kernel: nginx[8956]: segfault at 0000000000000008 
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:26 util1 kernel: nginx[8982]: segfault at 0000000000000008 
rip 0000000000429061 rsp 00007fffcc260550 error 6
Jan 24 16:22:27 util1 kernel: nginx[9026]: segfault at 0000000000000008 
rip 0000000000429061 rsp 00007fffcc260550 error 6

Does anyone know why this might be occurring, or any helpful information 
we can provide to better track it down?

Thanks very much in advance.

J

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,45615,45615#msg-45615

Hi,

yomuppet wrote:
>
>   
Have you tried compiling without the chunkin module?  AFAIK it's still
experimental.
> Here is the segfault information: 
> Jan 17 12:51:31 util1 kernel: nginx[27888]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260500 error 6
> Jan 24 16:21:33 util1 kernel: nginx[3835]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
> Jan 24 16:22:24 util1 kernel: nginx[8956]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
> Jan 24 16:22:26 util1 kernel: nginx[8982]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
> Jan 24 16:22:27 util1 kernel: nginx[9026]: segfault at 0000000000000008 rip 0000000000429061 rsp 00007fffcc260550 error 6
>
> Does anyone know why this might be occurring, or any helpful information we can provide to better track it down?
>   
If compiling without the chunkin module doesn't isolate the problem,
perform a backtrace.

Marcus.

On Tue, Jan 26, 2010 at 7:34 AM, yomuppet <nginx-forum@nginx.us> wrote:
> Does anyone know why this might be occurring, or any helpful information we can provide to better track it down?
>

It *may* be a bug in ngx_chunkin. I've just sent you a mail off-list
explaining how to produce a backtrace for me to investigate this issue
:)

Thanks for reporting it!
-agentzh

Hello,

Sorry for the delayed reply, it has been taking longer to hit a segfault 
(which is good), but still hitting some. Here is some backtrace output:



(gdb) bt
#0  0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050, 
timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2  0x0000000000421f08 in ngx_process_events_and_timers 
(cycle=0xc1e1050) at src/event/ngx_event.c:245
#3  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, 
data=) at src/os/unix/ngx_process_cycle.c:795
#4  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, 
proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#5  0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at 
src/os/unix/ngx_process_cycle.c:612
#6  0x000000000040edfa in main (argc=22, argv=0xc1dff30) at 
src/core/nginx.c:399
(gdb) bt
#0  0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050, 
timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2  0x0000000000421f08 in ngx_process_events_and_timers 
(cycle=0xc1e1050) at src/event/ngx_event.c:245
#3  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, 
data=) at src/os/unix/ngx_process_cycle.c:795
#4  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, 
proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#5  0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at 
src/os/unix/ngx_process_cycle.c:612
#6  0x000000000040edfa in main (argc=22, argv=0xc1dff30) at 
src/core/nginx.c:399
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768, 
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131                     iov->iov_len += (size_t) size;
(gdb) bt
#0  ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768, 
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
#1  0x000000000041186a in ngx_chain_writer (data=, in=) at 
src/core/ngx_output_chain.c:657
#2  0x0000000000412056 in ngx_output_chain (ctx=0xc7bd0b0, in=0x0) at 
src/core/ngx_output_chain.c:205
#3  0x000000000044540c in ngx_http_upstream_send_request (r=0xca55610, 
u=0xc7bd020) at src/http/ngx_http_upstream.c:1321
#4  0x0000000000445ef8 in ngx_http_upstream_send_request_handler (r=0x0, 
u=0x0) at src/http/ngx_http_upstream.c:1422
#5  0x000000000044399f in ngx_http_upstream_handler (ev=) at 
src/http/ngx_http_upstream.c:874
#6  0x00000000004220be in ngx_event_process_posted (cycle=, posted=0x0) 
at src/event/ngx_event_posted.c:39
#7  0x0000000000421f80 in ngx_process_events_and_timers 
(cycle=0xc1e1050) at src/event/ngx_event.c:272
#8  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, 
data=) at src/os/unix/ngx_process_cycle.c:795
#9  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, 
proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#10 0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at 
src/os/unix/ngx_process_cycle.c:612
#11 0x000000000040edfa in main (argc=22, argv=0xc1dff30) at 
src/core/nginx.c:399


(gdb) c
Continuing.
p iov
p prev
p cl->buf
p cl->buf->pos
up
detach

Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b21152bed40, in=0x48bfaf8, 
limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131                     iov->iov_len += (size_t) size;
(gdb) p iov
$1 = (struct iovec *) 0x0
(gdb) p prev
$2 = (u_char *) 0x0
(gdb) p cl->buf
Cannot access memory at address 0x0
(gdb) p cl->buf->pos
Cannot access memory at address 0x0
(gdb) up
#1  0x000000000041186a in ngx_chain_writer (data=, in=) at 
src/core/ngx_output_chain.c:657
657         ctx->out = c->send_chain(c, ctx->out, ctx->limit);
(gdb) detach
LND: Sending signal 11 to Thread 0x2b2114c6b750 (LWP 21815)
Detaching from program: /opt/nginx/sbin/nginx, process 21815




If anyone has further insight, would be much appreciated. Thanks.

J

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,45615,59278#msg-59278