Upstream keepalive - call for testing

I think is not your plugins problem. That is nginx core problem.
I using the original nginx/1.0.0 will continue to coredump.

but the dump position is not same.

gdb which nginx /var/www/ngx_coredump/core
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/
Reading symbols from /usr/sbin/nginx…done.
[New Thread 22042]

warning: Can’t read pathname for load map: 输入/输出错误.
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpcre.so.3
Reading symbols from /lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/local/lib/libprofiler.so.0…done.
Loaded symbols for /usr/local/lib/libprofiler.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/libunwind.so.7…(no debugging
symbols found)…done.
Loaded symbols for /usr/local/lib/libunwind.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6…(no
debugging symbols found)…done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault.
#0 ngx_http_upstream_handler (ev=) at
src/http/ngx_http_upstream.c:893
893 ctx = c->log->data;
(gdb) bt
#0 ngx_http_upstream_handler (ev=) at
src/http/ngx_http_upstream.c:893
#1 0x0000000000419979 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
#2 0x00000000004195ab in ngx_process_events_and_timers (cycle=0x855b40)
at src/event/ngx_event.c:261
#3 0x000000000041f1de in ngx_worker_process_cycle (cycle=0x855b40,
data=) at src/os/unix/ngx_process_cycle.c:800
#4 0x000000000041dbdb in ngx_spawn_process (cycle=0x855b40,
proc=0x41f10d <ngx_worker_process_cycle>, data=0x0, name=0x46275d
“worker process”, respawn=-4) at src/os/unix/ngx_process.c:196
#5 0x000000000041e876 in ngx_start_worker_processes (cycle=0x855b40,
n=4, type=-4) at src/os/unix/ngx_process_cycle.c:360
#6 0x000000000041fe86 in ngx_master_process_cycle (cycle=0x855b40) at
src/os/unix/ngx_process_cycle.c:249
#7 0x0000000000405e91 in main (argc=, argv=) at src/core/nginx.c:405
(gdb) backtrace full
#0 ngx_http_upstream_handler (ev=) at
src/http/ngx_http_upstream.c:893
c = 0x0
r = 0x1505440
ctx =
u = 0x0
#1 0x0000000000419979 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
ev = 0x7f6549bcd5a8
root = 0x7f65ef2ca4e0
sentinel = 0x687500
#2 0x00000000004195ab in ngx_process_events_and_timers (cycle=0x855b40)
at src/event/ngx_event.c:261
flags = 1
timer =
delta = 10
#3 0x000000000041f1de in ngx_worker_process_cycle (cycle=0x855b40,
data=) at src/os/unix/ngx_process_cycle.c:800
i =
c =
#4 0x000000000041dbdb in ngx_spawn_process (cycle=0x855b40,
proc=0x41f10d <ngx_worker_process_cycle>, data=0x0, name=0x46275d
“worker process”, respawn=-4) at src/os/unix/ngx_process.c:196
on = 1
pid = 0
s = 6
#5 0x000000000041e876 in ngx_start_worker_processes (cycle=0x855b40,
n=4, type=-4) at src/os/unix/ngx_process_cycle.c:360
i =
ch = {command = 1, pid = 22041, slot = 5, fd = 5}
#6 0x000000000041fe86 in ngx_master_process_cycle (cycle=0x855b40) at
src/os/unix/ngx_process_cycle.c:249
title =
p =
size =
i =
n =
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 8170904}, it_value =
{tv_sec = 0, tv_usec = 0}}
live = 1
delay = 0
ls =
—Type to continue, or q to quit—
ccf = 0x8562a0
#7 0x0000000000405e91 in main (argc=, argv=) at src/core/nginx.c:405
i =
log = 0x685320
cycle = 0x7cad80
init_cycle = {conf_ctx = 0x0, pool = 0x7ca240, log = 0x685320,
new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data
= 0x0, action = 0x0}, files = 0x0,
free_connections = 0x0, free_connection_n = 0,
reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts
= 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, pathes = {
elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0},
size = 0, nalloc = 0, pool = 0x0}, shared_memory = {
last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size
= 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections
= 0x0, read_events = 0x0, write_events = 0x0,
old_cycle = 0x0, conf_file = {len = 21, data = 0x45ef76
“/etc/nginx/nginx.conf”}, conf_param = {len = 0, data = 0x0},
conf_prefix = {len = 11, data = 0x45ef76 “/etc/nginx/nginx.conf”},
prefix = {len = 17, data = 0x45ef64 “/usr/local/nginx/”},
lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
ccf =

Posted at Nginx Forum:

I tried to disabled my script, but still cause a coredump.

here is my nginx.conf

user www-data;
worker_processes 4;

error_log /var/log/nginx/error.log ;
pid /var/run/nginx.pid;

worker_rlimit_core 8000M;
working_directory /var/www/ngx_coredump/;
debug_points abort;

worker_rlimit_nofile 131072;
events {
worker_connections 16384;
use epoll;
# multi_accept on;
}

http {
log_format logger
‘$http_host||$remote_addr||$msec||$status||$request_length||$bytes_sent||“$request”||“$http_referer”||“$http_user_agent”||$upstream_cache_status||$upstream_status’;

include       /etc/nginx/mime.types;

access_log  /var/log/nginx/access.log;


sendfile        on;

double size of page (get : getconf PAGESIZE)

client_header_buffer_size 64k;
#open_file_cache max=150000 inactive=20s;
#open_file_cache_valid 2s;
#open_file_cache_min_uses 1;
large_client_header_buffers 4 256k;

client_max_body_size 100m;
client_body_buffer_size 128k;
proxy_temp_path /dev/shm/proxy_temp;
proxy_connect_timeout 5s;
proxy_send_timeout 10;
proxy_read_timeout 10;
proxy_buffer_size 4k;
proxy_buffers 16 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_ignore_client_abort on;

fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
#first response buffer size
fastcgi_buffer_size 256K;
#continue response buffer size
fastcgi_buffers 32 256K;
#busy buffer
fastcgi_busy_buffers_size 2048k;
fastcgi_temp_file_write_size 2048k;

index index.html index.htm index.php;

#tcp_nopush: Sends the HTTP response headers in one packet.
tcp_nopush on;

#keepalive_timeout  0;
keepalive_timeout  5;

make it small for hard load server

#tcp_nodelay: Disables the Nagle buffering algorithm. Well, that
cleared that one up!

General web use does require a response from the client , tune it to

off
tcp_nodelay off;

server_tokens off;

gzip  off;
gzip_disable msie6;

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

}

=======================================================================================
sites-enabled/site.conf

proxy_cache_path /var/www/cdn_cache levels=1:2
keys_zone=cache_proxy:2048m max_size=4096m;

upstream backend {
server 10.0.0.1 weight=3 fail_timeout=2s;
keepalive 30;
}
limit_req_zone $binary_remote_addr zone=limit_cdn:32m rate=10r/s;
server {
listen 80;
server_name 10.0.0.2;

limit_req zone=limit_cdn burst=10;

add_header X-Via “$hostname”;
add_header X-Debug “$server_addr”;

root /var/www/cdn;
access_log /dev/shm/logger logger;

error_log /var/log/nginx/ct.error;
gzip on;
gzip_buffers 4 16k;
gzip_min_length 1k;
gzip_http_version 1.1;
gzip_comp_level 9;
gzip_types text/plain application/x-javascript text/css
application/xml;
gzip_vary on;
gzip_proxied any;

location ~ ^/purge(/.*) {
proxy_cache_purge cache_proxy $1$is_args$args;
}
location ~ .(swf)$
{
proxy_cache cache_proxy;
proxy_cache_valid 200 304 1h;
proxy_cache_key $uri;
add_header X-Cache “$upstream_cache_status from $hostname”;
expires 600s;
proxy_ssl_session_reuse on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection “keep-alive”;

proxy_pass http://backend;

}
location ~ ^/images/(.+)
{
proxy_cache cache_proxy;
proxy_cache_valid 200 304 8h;
proxy_cache_key $uri;
add_header X-Cache “$upstream_cache_status from $hostname”;
expires 1h;
proxy_ssl_session_reuse on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection “keep-alive”;

proxy_pass http://backend;

}
location ~ ^/sitemap/(.+)
{
proxy_cache cache_proxy;
proxy_cache_valid 200 304 8h;
proxy_cache_key $uri;
add_header X-Cache “$upstream_cache_status from $hostname”;
expires 8h;
proxy_ssl_session_reuse on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection “keep-alive”;

proxy_pass http://backend;

}
location ~ ^/search
{
proxy_cache cache_proxy;
proxy_cache_valid 200 304 600s;
proxy_cache_key $uri$is_args$args;
add_header X-Cache “$upstream_cache_status from $hostname”;
expires 600s;
proxy_ssl_session_reuse on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection “keep-alive”;

proxy_pass http://backend;

}
location / {

proxy_set_header X-Real-IP $remote_addr;
add_header X-Cache "MISS from $hostname";
proxy_set_header Connection "keep-alive";
proxy_buffering off;  #Direct send to client
proxy_ssl_session_reuse on;
proxy_pass http://backend;

}
rewrite “^/$” /index.htm;
}

Posted at Nginx Forum:

I am sorry, I just noticed that ngin 1.0.5 has fixed the cache cause
segmentation failure bug.
And I was test it at 1.0.0.

When I remove the following config, the server hasn’t dead again now.

proxy_ssl_session_reuse on;

I will continued to keep checking.

Posted at Nginx Forum:

I have patch and it won’t dead again, Now run 50 minutes, and handled
330000 request without problem.

the problem is not related to proxy_ssl_session_reuse

Maxim D. Wrote:

this problem.
b/src/event/ngx_event_pipe.c
if ((p->upstream_eof || p->upstream_error) &&
nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

Hello!

On Fri, Aug 26, 2011 at 12:17:01PM -0400, magicbear wrote:

It get the same result, I think may be cause by my upload script? I will
try to disable it to get result.
In another more high load server(without patch) using this script is no
this problem.

You mean you log rotation script you’ve posted earlier? It
shouldn’t cause problems, if it does - this means there is a bug
somewhere. And I think I’ve found the one you are seeing, please
try the following patch:

diff --git a/src/event/ngx_event_pipe.c b/src/event/ngx_event_pipe.c
— a/src/event/ngx_event_pipe.c
+++ b/src/event/ngx_event_pipe.c
@@ -414,6 +414,7 @@ ngx_event_pipe_read_upstream(ngx_event_p

 if (p->length == 0) {
     p->upstream_done = 1;
  •    p->read = 1;
    

    }

    if ((p->upstream_eof || p->upstream_error) && p->free_raw_bufs) {

You may also grab one from here:

http://mdounin.ru/files/patch-nginx-keepalive-cache.txt

Maxim D.

addition information:
the server may dead when processed about 20 million requests. Not very
high frequencies to dead.

2011/08/27 13:31:21 [alert] 30433#0: worker process 1944 exited on
signal 11 (core dumped)

2011/08/28 14:40:35 [alert] 30433#0: worker process 14510 exited on
signal 11 (core dumped)

Posted at Nginx Forum:

Hello!

On Sun, Aug 28, 2011 at 01:07:25PM -0400, magicbear wrote:

Hello, tested for 2 days, the segmentation failure frequence is
decreased, but sometime still will dead.

Here is the coredump information. It seems appear too at official
without patch version.

[…]

(gdb) bt
#0 0x000000000045bd99 in ngx_http_upstream_handler (ev=0x7f99438901e8)
at src/http/ngx_http_upstream.c:921
#1 0x0000000000427839 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149

Could you please test if the attached patch fixes things for you?

You may also grab it from here:
http://mdounin.ru/files/patch-nginx-ssl-shutdown.txt

Maxim D.

Hello, tested for 2 days, the segmentation failure frequence is
decreased, but sometime still will dead.

Here is the coredump information. It seems appear too at official
without patch version.

GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/
Reading symbols from /usr/sbin/nginx…done.
[New Thread 14510]

warning: Can’t read pathname for load map: 输入/输出错误.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpcre.so.3
Reading symbols from /lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/local/lib/libprofiler.so.0…done.
Loaded symbols for /usr/local/lib/libprofiler.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/libunwind.so.7…(no debugging
symbols found)…done.
Loaded symbols for /usr/local/lib/libunwind.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6…(no
debugging symbols found)…done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000045bd99 in ngx_http_upstream_handler (ev=0x7f99438901e8)
at src/http/ngx_http_upstream.c:921
921 u->write_event_handler(r, u);
(gdb) bt
#0 0x000000000045bd99 in ngx_http_upstream_handler (ev=0x7f99438901e8)
at src/http/ngx_http_upstream.c:921
#1 0x0000000000427839 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
#2 0x0000000000425a04 in ngx_process_events_and_timers
(cycle=0x12ba140) at src/event/ngx_event.c:261
#3 0x000000000043068c in ngx_worker_process_cycle (cycle=0x12ba140,
data=0x0) at src/os/unix/ngx_process_cycle.c:800
#4 0x000000000042d993 in ngx_spawn_process (cycle=0x12ba140,
proc=0x430509 <ngx_worker_process_cycle>, data=0x0, name=0x49c6a0
“worker process”, respawn=8) at src/os/unix/ngx_process.c:196
#5 0x0000000000430132 in ngx_reap_children (cycle=0x12ba140) at
src/os/unix/ngx_process_cycle.c:617
#6 0x000000000042f1da in ngx_master_process_cycle (cycle=0x12ba140) at
src/os/unix/ngx_process_cycle.c:180
#7 0x00000000004054b9 in main (argc=1, argv=0x7fff33e74ef8) at
src/core/nginx.c:405
(gdb) backtrace full
#0 0x000000000045bd99 in ngx_http_upstream_handler (ev=0x7f99438901e8)
at src/http/ngx_http_upstream.c:921
c = 0x7f9943a4f010
r = 0x11aee20
ctx = 0x11b01e0
u = 0x0
#1 0x0000000000427839 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
ev = 0x7f99438901e8
node = 0x7f9943890210
root = 0x7f99e91baf08
sentinel = 0x6c2f60
#2 0x0000000000425a04 in ngx_process_events_and_timers
(cycle=0x12ba140) at src/event/ngx_event.c:261
flags = 1
timer = 5
delta = 5
#3 0x000000000043068c in ngx_worker_process_cycle (cycle=0x12ba140,
data=0x0) at src/os/unix/ngx_process_cycle.c:800
i = 0
c = 0x7fff33e74ef0
#4 0x000000000042d993 in ngx_spawn_process (cycle=0x12ba140,
proc=0x430509 <ngx_worker_process_cycle>, data=0x0, name=0x49c6a0
“worker process”, respawn=8) at src/os/unix/ngx_process.c:196
on = 1
pid = 0
s = 8
#5 0x0000000000430132 in ngx_reap_children (cycle=0x12ba140) at
src/os/unix/ngx_process_cycle.c:617
i = 8
n = 13
live = 1
ch = {command = 2, pid = 1944, slot = 8, fd = -1}
ccf = 0x6c26be
#6 0x000000000042f1da in ngx_master_process_cycle (cycle=0x12ba140) at
src/os/unix/ngx_process_cycle.c:180
title = 0x1282de6 “”
p = 0x1282dfa “”
size = 21
i = 1
n = 11215018668083
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value =
{tv_sec = 0, tv_usec = 6}}
live = 1
delay = 0
ls = 0x0
ccf = 0x12ba8a0
#7 0x00000000004054b9 in main (argc=1, argv=0x7fff33e74ef8) at
src/core/nginx.c:405
i = 47
log = 0x6c0d60
cycle = 0x11a1d90
init_cycle = {conf_ctx = 0x0, pool = 0x11a1240, log = 0x6c0d60,
new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data
= 0x0, action = 0x0}, files = 0x0, free_connections = 0x0,
free_connection_n = 0,
—Type to continue, or q to quit—
reusable_connections_queue = {prev = 0x0, next = 0x0},
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
pathes = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
open_files = {
last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size
= 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts
= 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
connection_n = 0, files_n = 0, connections = 0x0, read_events
= 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data
= 0x49977e “/etc/nginx/nginx.conf”}, conf_param = {len = 0, data =
0x0},
conf_prefix = {len = 11, data = 0x49977e
“/etc/nginx/nginx.conf”}, prefix = {len = 17, data = 0x49976c
“/usr/local/nginx/”}, lock_file = {len = 0, data = 0x0}, hostname = {len
= 0, data = 0x0}}
ccf = 0x11a2c90

Posted at Nginx Forum:

OK, It works for 3 days with 40 million request haven’t coredump.
Thanks.

Maxim D. Wrote:

appear too at official

src/event/ngx_event_timer.c:149

User Maxim D. [email protected]

@@ -1205,6 +1205,7 @@
SSL_get_shutdown(c->ssl->connection);

  •    }
    

    }

    SSL_set_shutdown(c->ssl->connection, mode);


nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

I use at 2x100Mbps server (40% and 10% utilisation), when stable
enought, then I will continue to test at 1000Mbps server.

splitice Wrote:

haven’t coredump.

(gdb) bt

804454a04e22d32c064a3af8e7b24326c7fc2d63

This happens if there are some unsent alerts.
— a/src/event/ngx_event_openssl.c
} else {
+

nginx Info Page


nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

I havent had any crashes on 3x10mbit servers (90% utilisation) or
2x100mbit
servers (20% utilisation).

Seems very stable.

Hello , when run for 5 days, today have 2 segment failure.

2011/09/04 12:00:47 [alert] 32147#0: worker process 8496 exited on
signal 11 (core dumped)
2011/09/04 12:17:32 [alert] 8497#0: ignore long locked inactive cache
entry 57c7c6c612a651727b880a1c9a2c7d2d, count:1
2011/09/04 12:21:31 [alert] 32147#0: worker process 19185 exited on
signal 11 (core dumped)

Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault.
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8) at
src/http/ngx_http_upstream.c:915
915 ctx->current_request = r;
(gdb) bt
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8) at
src/http/ngx_http_upstream.c:915
#1 0x0000000000419e95 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
#2 0x0000000000419ac7 in ngx_process_events_and_timers
(cycle=0x21f1990) at src/event/ngx_event.c:261
#3 0x000000000041f76e in ngx_worker_process_cycle (cycle=0x21f1990,
data=) at src/os/unix/ngx_process_cycle.c:800
#4 0x000000000041e16b in ngx_spawn_process (cycle=0x21f1990,
proc=0x41f69d <ngx_worker_process_cycle>, data=0x0, name=0x463f3d
“worker process”, respawn=7) at src/os/unix/ngx_process.c:196
#5 0x000000000042008d in ngx_reap_children (cycle=0x21f1990) at
src/os/unix/ngx_process_cycle.c:617
#6 ngx_master_process_cycle (cycle=0x21f1990) at
src/os/unix/ngx_process_cycle.c:180
#7 0x0000000000406061 in main (argc=, argv=) at src/core/nginx.c:405

(gdb) bt full
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8) at
src/http/ngx_http_upstream.c:915
c = 0x23b4160
r = 0x22a2780
ctx = <error reading variable ctx (Cannot access memory at
address 0x20)>
u = 0x0
#1 0x0000000000419e95 in ngx_event_expire_timers () at
src/event/ngx_event_timer.c:149
ev = 0x7fc45735f8a8
root = 0x7fc4574ffa30
sentinel = 0x689560
#2 0x0000000000419ac7 in ngx_process_events_and_timers
(cycle=0x21f1990) at src/event/ngx_event.c:261
flags = 1
timer =
delta = 21
#3 0x000000000041f76e in ngx_worker_process_cycle (cycle=0x21f1990,
data=) at src/os/unix/ngx_process_cycle.c:800
i =
c =
#4 0x000000000041e16b in ngx_spawn_process (cycle=0x21f1990,
proc=0x41f69d <ngx_worker_process_cycle>, data=0x0, name=0x463f3d
“worker process”, respawn=7) at src/os/unix/ngx_process.c:196
on = 1
pid = 0
s = 7
#5 0x000000000042008d in ngx_reap_children (cycle=0x21f1990) at
src/os/unix/ngx_process_cycle.c:617
i =
live =
n =
ch = {command = 2, pid = 8496, slot = 7, fd = -1}
ccf =
#6 ngx_master_process_cycle (cycle=0x21f1990) at
src/os/unix/ngx_process_cycle.c:180
title =
p =
size =
i =
n =
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 35114632}, it_value
= {tv_sec = 0, tv_usec = 32146}}
live =
delay = 0
ls =
ccf = 0x21f2188
#7 0x0000000000406061 in main (argc=, argv=) at src/core/nginx.c:405
i =
log = 0x687380
cycle = 0x217ce70
init_cycle = {conf_ctx = 0x0, pool = 0x217c240, log = 0x687380,
new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data
= 0x0, action = 0x0}, files = 0x0,
free_connections = 0x0, free_connection_n = 0,
reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts
= 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
pathes = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool =
0x0}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next =
0x0}, size = 0, nalloc = 0, pool = 0x0},
shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0,
next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0,
files_n = 0, connections = 0x0, read_events = 0x0,
write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21,
data = 0x4606b6 “/etc/nginx/nginx.conf”}, conf_param = {len = 0, data =
0x0}, conf_prefix = {len = 11,
data = 0x4606b6 “/etc/nginx/nginx.conf”}, prefix = {len =
17, data = 0x4606a4 “/usr/local/nginx/”}, lock_file = {len = 0, data =
0x0}, hostname = {len = 0, data = 0x0}}
—Type to continue, or q to quit—
ccf =

Posted at Nginx Forum:

Hello!

On Sun, Sep 04, 2011 at 01:33:59PM -0400, magicbear wrote:

src/event/ngx_event_timer.c:149
#2 0x0000000000419ac7 in ngx_process_events_and_timers
(cycle=0x21f1990) at src/event/ngx_event.c:261

Hm, this looks identical to one you’ve reported previously. Do
you have patch from

http://mailman.nginx.org/pipermail/nginx/2011-August/028809.html

applied?

optimized out>) at src/core/nginx.c:405

(gdb) bt full
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8) at
src/http/ngx_http_upstream.c:915
c = 0x23b4160
r = 0x22a2780
ctx = <error reading variable ctx (Cannot access memory at
address 0x20)>
u = 0x0

Could you please show results of the following gdb commands:

fr 0
p *ev
p *((ngx_connection_t *) ev->data)

Maxim D.

Hello!

On Mon, Sep 05, 2011 at 02:31:54AM +0800, MagicBear wrote:

0,
sent = 0, log = 0x21f1770, pool = 0x237ff70, sockaddr = 0x0, socklen = 0,
addr_text = {len = 0, data = 0x0}, ssl = 0x0, local_sockaddr = 0x0,
buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 63713716,
requests = 0, buffered = 0, log_error = 1, single_connection = 0,
unexpected_eof = 0, timedout = 0, error = 0, destroyed = 0, idle = 0,
reusable = 0, close = 0, sendfile = 1, sndlowat = 0, tcp_nodelay = 0,
tcp_nopush = 0}

Could you please also show the following (in the same frame, i.e.
after “fr 0”):

p ngx_cycle->log
p *r
p *r->connection

Maxim D.

2011/09/04 12:00:47 [alert] 32147#0: worker process 8496 exited on
Program terminated with signal 11, Segmentation fault.

proc=0x41f69d <ngx_worker_process_cycle>, data=0x0, name=0x463f3d
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8) at
p *ev
p *((ngx_connection_t *) ev->data)

Maxim D.


nginx mailing list
[email protected]
nginx Info Page

(gdb) fr 0
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8)
at src/http/ngx_http_upstream.c:915
915 ctx->current_request = r;
(gdb) p *ev
$1 = {data = 0x7fc4576aa750, write = 1, accept = 0, instance = 1, active

1,
disabled = 0, ready = 1, oneshot = 0, complete = 0, eof = 0, error =
0,
timedout = 1, timer_set = 0, delayed = 0, read_discarded = 0,
unexpected_eof = 0, deferred_accept = 0, pending_eof = 0, posted_ready

0,
available = 0, handler = 0x43bf9c <ngx_http_upstream_handler>,
index = 3503345872, log = 0x21f1770, timer = {key = 0, left = 0x0,
right = 0x0, parent = 0x0, color = 0 ‘\000’, data = 0 ‘\000’},
closed = 0, channel = 0, resolver = 0, next = 0x0, prev = 0x0}
(gdb) p *((ngx_connection_t *) ev->data)
$2 = {data = 0x22a2780, read = 0x7fc4575008a8, write = 0x7fc45735f8a8,
fd = 66, recv = 0x41cc50 <ngx_unix_recv>, send = 0x41cf08
<ngx_unix_send>,
recv_chain = 0x41cd00 <ngx_readv_chain>,
send_chain = 0x421050 <ngx_linux_sendfile_chain>, listening = 0x0,
sent = 0, log = 0x21f1770, pool = 0x237ff70, sockaddr = 0x0, socklen =
0,
addr_text = {len = 0, data = 0x0}, ssl = 0x0, local_sockaddr = 0x0,
buffer = 0x0, queue = {prev = 0x0, next = 0x0}, number = 63713716,
requests = 0, buffered = 0, log_error = 1, single_connection = 0,
unexpected_eof = 0, timedout = 0, error = 0, destroyed = 0, idle = 0,
reusable = 0, close = 0, sendfile = 1, sndlowat = 0, tcp_nodelay = 0,
tcp_nopush = 0}

2011/9/5 Maxim D. [email protected]

(gdb) fr 0
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8)
at src/http/ngx_http_upstream.c:915
915 ctx->current_request = r;
(gdb) p ngx_cycle->log
$1 = (ngx_log_t *) 0x21f19a8
(gdb) p *r
$2 = {signature = 51686928, connection = 0x23b4160, ctx = 0x0,
main_conf = 0x0, srv_conf = 0x23e6dc8, loc_conf = 0x2216d58,
read_event_handler = 0x42d5dd <ngx_http_block_reading>,
write_event_handler = 0x42d2a6 <ngx_http_request_empty_handler>,
cache = 0x0, upstream = 0x0, upstream_states = 0x0, pool = 0x23e92d0,
header_in = 0x2309ea0, headers_in = {headers = {last = 0x22a27f0, part
= {
elts = 0x23e9ae8, nelts = 5, next = 0x0}, size = 48, nalloc =
20,
pool = 0x23e92d0}, host = 0x23e9b78, connection = 0x0,
if_modified_since = 0x0, if_unmodified_since = 0x0,
user_agent = 0x23e9b48, referer = 0x0, content_length = 0x0,
content_type = 0x0, range = 0x0, if_range = 0x0, transfer_encoding =
0x0,
expect = 0x0, accept_encoding = 0x0, via = 0x0, authorization = 0x0,
keep_alive = 0x0, x_forwarded_for = 0x0, user = {len = 0, data =
0x0},
passwd = {len = 0, data = 0x0}, cookies = {elts = 0x23e9ea8, nelts =
0,
size = 8, nalloc = 2, pool = 0x23e92d0}, server = {len = 14,
data = 0x23e9edb “static.loli.myconnectioner-ag\360\324,\002”},
content_length_n = -1, keep_alive_n = -1, connection_type = 1, msie

1,
msie6 = 1, opera = 0, gecko = 0, chrome = 0, safari = 0, konqueror =
0},
headers_out = {headers = {last = 0x22a2920, part = {elts = 0x23e9320,
nelts = 4, next = 0x0}, size = 48, nalloc = 20, pool =
0x23e92d0},
status = 200, status_line = {len = 0, data = 0x0}, server = 0x0,
date = 0x0, content_length = 0x0, content_encoding = 0x0, location =
0x0,
refresh = 0x0, last_modified = 0x0, content_range = 0x0,
—Type to continue, or q to quit—
accept_ranges = 0x23e93b0, www_authenticate = 0x0, expires =
0x23e9320,
etag = 0x0, override_charset = 0x0, content_type_len = 24,
content_type
= {
len = 24, data = 0x23e5bf0 “application/x-javascript”}, charset =
{
len = 0, data = 0x0},
content_type_lowcase = 0x23ea1e8
“application/x-javascripts\016>\002”,
content_type_hash = 15285649345721276481, cache_control = {
elts = 0x23ea188, nelts = 1, size = 8, nalloc = 1, pool =
0x23e92d0},
content_length_n = 19773, date_time = 0, last_modified_time =
1313998502},
request_body = 0x0, lingering_time = 1315110106, start_sec =
1315110076,
start_msec = 683, method = 2, http_version = 1001, request_line = {len

0,
data = 0x314ae20 “\360\f>\002”}, uri = {len = 15, data = 0x314ae24
“”},
args = {len = 0, data = 0x0}, exten = {len = 2, data = 0x314ae31 “”},
unparsed_uri = {len = 15, data = 0x314ae24 “”}, method_name = {len =
3,
data = 0x314ae20 “\360\f>\002”}, http_protocol = {len = 8,
data = 0x314ae34 “”}, out = 0x0, main = 0x22a2780, parent = 0x0,
postponed = 0x0, post_subrequest = 0x0, posted_requests = 0x0,
virtual_names = 0x2465c20, phase_handler = 11, content_handler = 0,
access_code = 0, variables = 0x23e9808, ncaptures = 4, captures =
0x23e9ff0,
captures_data = 0x314ae24 “”, limit_rate = 0, header_size = 339,
request_length = 183, err_status = 0, http_connection = 0x23b4228,
log_handler = 0x42da14 <ngx_http_log_error_handler>, cleanup = 0x0,
subrequests = 51, count = 0, blocked = 0, aio = 0, http_state = 2,
complex_uri = 0, quoted_uri = 0, plus_in_uri = 0, space_in_uri = 0,
—Type to continue, or q to quit—
invalid_header = 0, add_uri_to_alias = 0, valid_location = 1,
valid_unparsed_uri = 1, uri_changed = 0, uri_changes = 11,
request_body_in_single_buf = 0, request_body_in_file_only = 0,
request_body_in_persistent_file = 0, request_body_in_clean_file = 0,
request_body_file_group_access = 0, request_body_file_log_level = 5,
subrequest_in_memory = 0, waited = 0, cached = 0, gzip_tested = 1,
gzip_ok = 0, gzip_vary = 1, proxy = 0, bypass_cache = 0, no_cache = 0,
limit_zone_set = 0, limit_req_set = 0, pipeline = 0, plain_http = 0,
chunked = 0, header_only = 0, keepalive = 0, lingering_close = 0,
discard_body = 0, internal = 0, error_page = 0,
ignore_content_encoding =
0,
filter_finalize = 0, post_action = 0, request_complete = 1,
request_output = 1, header_sent = 1, expect_tested = 0, root_tested =
1,
done = 1, logged = 0, buffered = 0, main_filter_need_in_memory = 0,
filter_need_in_memory = 0, filter_need_temporary = 0, allow_ranges =
1,
stat_reading = 0, stat_writing = 1, state = 0,
header_hash = 2715320498552542, lowcase_index = 10,
lowcase_header = “connectionguage”, ‘\000’ <repeats 16 times>,
header_name_start = 0x314aed5 “\r\nible; MSIE 8.0; Windows NT 6.1;
Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET
CLR
3.0.30729; MALN; InfoPath.1; Media Center PC 6.0; Tablet PC 2.0;
.NET4.0C;
.NET4.0E)”,
header_name_end = 0x314aecc “”, header_start = 0x314aece “close”,
header_end = 0x314aed5 “\r\nible; MSIE 8.0; Windows NT 6.1;
Trident/4.0;
GTB7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
3.0.30729;
MALN; Info—Type to continue, or q to quit—
Path.1; Media Center PC 6.0; Tablet PC 2.0; .NET4.0C; .NET4.0E)”,
uri_start = 0x314ae24 “”, uri_end = 0x314ae33 “”, uri_ext = 0x314ae31
“”,
args_start = 0x0, request_start = 0x314ae20 “\360\f>\002”,
request_end = 0x314ae3c “”, method_end = 0x314ae22 “>\002”,
schema_start = 0x0, schema_end = 0x0, host_start = 0x0, host_end =
0x0,
port_start = 0x0, port_end = 0x0, http_minor = 1264, http_major = 0}
(gdb) p *r->connection
$3 = {data = 0x30, read = 0x111, write = 0x22a2770, fd = 36740720,
recv = 0x2309e80, send = 0, recv_chain = 0xb0, send_chain = 0x23b4170,
listening = 0x0, sent = 36740848, log = 0x0, pool = 0x23b41d0,
sockaddr = 0xf088e714b9a0002, socklen = 0, addr_text = {len = 4,
data = 0x21f1c18 “S”}, ssl = 0x3cc35a0, local_sockaddr = 0x42d957,
buffer = 0x23b4210, queue = {prev = 0x465bfc, next =
0x2e3234312e333131},
number = 3256446919138946616, requests = 140481256921360, buffered =
0,
log_error = 0, single_connection = 0, unexpected_eof = 0, timedout =
0,
error = 0, destroyed = 0, idle = 0, reusable = 0, close = 0, sendfile

0,
sndlowat = 0, tcp_nodelay = 0, tcp_nopush = 0}

2011/9/5 Maxim D. [email protected]

ok, I will try that at another dedicated server, thanks for your Helps.

MagicBear

Maxim D. Wrote:

(gdb) p ngx_cycle->log
improper handling
for details:

http://mailman.nginx.org/pipermail/nginx-devel/201
1-September/001147.html

Maxim D.


nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

Hello!

On Mon, Sep 05, 2011 at 11:42:31PM +0800, ビリビリⅤ wrote:

(gdb) fr 0
#0 ngx_http_upstream_handler (ev=0x7fc45735f8a8)
at src/http/ngx_http_upstream.c:915
915 ctx->current_request = r;
(gdb) p ngx_cycle->log
$1 = (ngx_log_t *) 0x21f19a8
(gdb) p *r
$2 = {signature = 51686928, connection = 0x23b4160, ctx = 0x0,

[…]

This looks like memory corruption, but unfortunately I don’t see
any traces of the real cause. My best quess is improper handling
of proxy_ignore_client_abort as fixed in 1.1.2. Please try 1.1.2
with patches from

http://nginx.org/patches/patch-nginx-keepalive-full-5.txt

It already includes upstream keepalive module, as well as all
other upstream-keepalive related fixes. See here for details:

http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001147.html

Maxim D.

Hello!

On Wed, Sep 07, 2011 at 04:33:38PM -0700, Matthieu T. wrote:

Thanks for the all-included patch from 1.1.2

It seems that keepalive connections aren’t working in combination with
proxy_buffering off; (c->read->ready is set to 1)
It worked before this last update, is this the intended behavior ?

Linux with epoll, right? Please try the following patch:

diff --git a/src/http/modules/ngx_http_upstream_keepalive_module.c
b/src/http/modules/ngx_http_upstream_keepalive_module.c
— a/src/http/modules/ngx_http_upstream_keepalive_module.c
+++ b/src/http/modules/ngx_http_upstream_keepalive_module.c
@@ -321,7 +321,6 @@ ngx_http_upstream_free_keepalive_peer(ng
if (kp->failed
|| c == NULL
|| c->read->eof

  •    || c->read->ready
       || c->read->error
       || c->read->timedout
       || c->write->error
    

@@ -382,6 +381,10 @@ ngx_http_upstream_free_keepalive_peer(ng
item->socklen = pc->socklen;
ngx_memcpy(&item->sockaddr, pc->sockaddr, pc->socklen);

  • if (c->read->ready) {
  •    ngx_http_upstream_keepalive_close_handler(c->read);
    
  • }

invalid:

 kp->original_free_peer(pc, kp->data, state);

diff --git a/src/http/ngx_http_upstream.c b/src/http/ngx_http_upstream.c
— a/src/http/ngx_http_upstream.c
+++ b/src/http/ngx_http_upstream.c
@@ -2142,7 +2142,7 @@ ngx_http_upstream_send_response(ngx_http
return;
}

  •        if (u->peer.connection->read->ready) {
    
  •        if (u->peer.connection->read->ready || u->length == 0) {
               ngx_http_upstream_process_non_buffered_upstream(r, u);
           }
       }
    

Second part fixes one more problem with handling of responses with
empty body over keepalive connections, found during testing with
various event methods.

Maxim D.

Hello!

On Thu, Sep 08, 2011 at 01:27:34PM +0400, Maxim D. wrote:

Linux with epoll, right? Please try the following patch:
Updated full patch may be grabbed here:

http://nginx.org/patches/patch-nginx-keepalive-full-6.txt

Maxim D.