I think I have found a small issue, if we’re using proxy_pass to talk to
an
origin that doesn’t support keep alives.
The origin will return a HTTP header “Connection: close”, and terminate
the
connection (TCP FIN).
We don’t take this into account, and assume there is a keep-alive
connection
available.
The next time the connection is used, it won’t be part of a valid TCP
stream, and the origin server will send a TCP RST.
This can be simulated with 2 nginx instances, one acting as a reverse
proxy
with keep alive connection. And the other using the directive
keepalive_timeout 0; (which will always terminate connections right
away).
The patches attached take into account the response of the origin, and
should fix this issue.
Oh and I havent been able to reproduce the crash, I tried for a while
but
gave up. if it happens again ill build with debugging and restart
howeaver
so far its been 36hours without issues (under a significant amount of
traffic)
On Fri, Aug 12, 2011 at 12:32:26PM -0700, Matthieu T. wrote:
Hi all,
I think I have found a small issue, if we’re using proxy_pass to talk to an
origin that doesn’t support keep alives.
The origin will return a HTTP header “Connection: close”, and terminate the
connection (TCP FIN).
We don’t take this into account, and assume there is a keep-alive connection
available.
The next time the connection is used, it won’t be part of a valid TCP
stream, and the origin server will send a TCP RST.
Yes, I’m aware of this, thank you. Actually, this is harmless:
upstream keepalive module should detect connection was closed
while keeping it, and even if it wasn’t able to do so - nginx will
re-try sending request if sending to cached connection fails.
This can be simulated with 2 nginx instances, one acting as a reverse proxy
with keep alive connection. And the other using the directive
keepalive_timeout 0; (which will always terminate connections right away).
The patches attached take into account the response of the origin, and
should fix this issue.
I’m planing to add similar patch, thanks.
Maxim D.
approximately
track it down (either to fix problem in keepalive patch or to
Could you please elaborate?
Also, if I was planning on having a lot of different connections using
the
upstream keepalive module.
Would it make sense to convert the queues into rbtrees for faster lookup
?
Is the peer selection algorithm guaranteed to never run at the same time
for
different workers? (i-e: creating race conditions in the keepalive
queue)
I see that the round robin code has a bunch of mutex locks all commented
out…
On the other hand nginx_http_upstream_check_module (healthchecks) uses
mutexes.
For faster lookups, I was thinking about a hashmap of queues, hashed on
sockaddr.
This is probably overkill for a small amount of keepalives connections
though.
I’ll send a patch if I get around to implement it.
On Fri, Aug 12, 2011 at 02:11:51PM -0700, Matthieu T. wrote:
Also, if I was planning on having a lot of different connections using the
upstream keepalive module.
Would it make sense to convert the queues into rbtrees for faster lookup ?
Yes, it may make sense if you are planning to keep lots of
connections to lots of different backends (you’ll still need
queues though, but that’s details).
Maxim D.
available.
with keep alive connection. And the other using the directive
was
Hello!
openvz, XEN and one vmware virtualised containers running debian
the
Hello!
I’m wondering what would be the easiest way,
Thoughts ?
On Wed, Aug 3, 2011 at 3:09 AM, David Yu <
wrote:
support to various backends (as with upstream
to
No, to keep backend connections alive you need
believe
that can be set in the global variable “errno” and
error
your
Nope not fastcgi multiplexing. Multiplexing over a
http
Keeping https connections will require additional support both in
nginx core and upstream keepalive module. You may try the
attached patches, also available here:
Hello, I found a bug, the stub_status module shows connection and
waiting is only increase without decrease. But in fact, system
connection haven’t such many.
Here is the trend chart http://m-b.cc/tmp/bugs.png
On Fri, Aug 26, 2011 at 03:08:10AM -0400, magicbear wrote:
Hello, I found a bug, the stub_status module shows connection and
waiting is only increase without decrease. But in fact, system
connection haven’t such many.
Here is the trend chart http://m-b.cc/tmp/bugs.png
What’s in error log?
Graph suggests there are worker processes dying for some reason,
there should be messages like “[alert] … worker process …
exited on signal …” in global error log.
If there are such messages, please follow [1] to obtain core dump
and provide backtrace. Some more details as outlined in [2] may
be also helpful.
P.S: I have a cron script that run every minute to upload log to my
analytics server, and every minute it will send a USR1 to nginx to
reload the log.
here is the script, may be help for find bug.
#!/bin/bash
if [ ! -e /dev/shm/logger ]; then exit 0; fi
if [ ! -s /dev/shm/logger ]; then exit 0; fi
gzip -c /dev/shm/logger | nc 10.0.0.1 989
rm -f /dev/shm/logger
touch /dev/shm/logger
chmod 0666 /dev/shm/logger
kill -USR1 cat /var/run/nginx.pid
magicbear Wrote:
nginx: nginx version: nginx/1.1.0
There is NO WARRANTY, to the extent permitted by
Cannot access memory at address 0x7f9558f332c8
fault.
) at src/core/ngx_string.c:253
d = <error reading variable d (Cannot
access memory at address 0x7ffffd349030)>
max_width = <error reading variable
access memory at address 0x7ffffd348fe0)>
curl: (52) Empty reply from server
attached patches, also available here:
s-module.txt
keepalive module.
connections.
+++ b/src/http/ngx_http_upstream.c
=====================================================
uname -a
Linux sunlight 2.6.38-10-server #46-Ubuntu SMP Tue Jun 28 16:31:00 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux
=====================================================
gdb which nginx core
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/…
Reading symbols from /usr/sbin/nginx…done.
BFD: Warning: /var/www/ngx_coredump/core is truncated: expected core
file size >= 2724024320, found: 2103255040.
[New Thread 6565]
Cannot access memory at address 0x7f9558f332c8
Cannot access memory at address 0x7f9558f332c8
Cannot access memory at address 0x7f9558f332c8
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Failed to read a valid object file image from memory.
Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault. #0 0x000000000040cff5 in ngx_vslprintf (buf=Cannot access memory at
address 0x7ffffd348fc8
) at src/core/ngx_string.c:253
253 while (*p && buf < last) {
(gdb) bt #0 0x000000000040cff5 in ngx_vslprintf (buf=Cannot access memory at
address 0x7ffffd348fc8
) at src/core/ngx_string.c:253
Cannot access memory at address 0x7ffffd349068
(gdb) backtrace full #0 0x000000000040cff5 in ngx_vslprintf (buf=Cannot access memory at
address 0x7ffffd348fc8
) at src/core/ngx_string.c:253
p = <error reading variable p (Cannot access memory at address
0x7ffffd349050)>
zero = <error reading variable zero (Cannot access memory at
address 0x7ffffd34905f)>
d = <error reading variable d (Cannot access memory at address
0x7ffffd349058)>
f = <error reading variable f (Cannot access memory at address
0x7ffffd349048)>
scale = <error reading variable scale (Cannot access memory at
address 0x7ffffd349040)>
len = <error reading variable len (Cannot access memory at
address 0x7ffffd348fe8)>
slen = <error reading variable slen (Cannot access memory at
address 0x7ffffd349038)>
i64 = <error reading variable i64 (Cannot access memory at
address 0x7ffffd349030)>
ui64 = <error reading variable ui64 (Cannot access memory at
address 0x7ffffd349028)>
ms = <error reading variable ms (Cannot access memory at address
0x7ffffd348fd8)>
width = <error reading variable width (Cannot access memory at
address 0x7ffffd349020)>
sign = <error reading variable sign (Cannot access memory at
address 0x7ffffd349018)>
hex = <error reading variable hex (Cannot access memory at
address 0x7ffffd349010)>
max_width = <error reading variable max_width (Cannot access
memory at address 0x7ffffd349008)>
frac_width = <error reading variable frac_width (Cannot access
memory at address 0x7ffffd349000)>
n = <error reading variable n (Cannot access memory at address
0x7ffffd348ff8)>
v = <error reading variable v (Cannot access memory at address
0x7ffffd348ff0)>
vv = <error reading variable vv (Cannot access memory at address
0x7ffffd348fe0)>
Cannot access memory at address 0x7ffffd349068
gdb which nginx core
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/…
Reading symbols from /usr/sbin/nginx…done.
[New Thread 10718]
warning: Can’t read pathname for load map: 输入/输出错误.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpcre.so.3
Reading symbols from /lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/local/lib/libprofiler.so.0…done.
Loaded symbols for /usr/local/lib/libprofiler.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/libunwind.so.7…(no debugging
symbols found)…done.
Loaded symbols for /usr/local/lib/libunwind.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6…(no
debugging symbols found)…done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault. #0 0x000000000040d661 in ngx_vslprintf (buf=0x7fff8f777bf5 “”,
last=0x7fff8f7783c0 “P\364z\363\346\177”, fmt=0x4a5f34 “s" failed”,
args=0x7fff8f777b90) at src/core/ngx_string.c:253
253 while (*p && buf < last) {
(gdb) bt #0 0x000000000040d661 in ngx_vslprintf (buf=0x7fff8f777bf5 “”,
last=0x7fff8f7783c0 “P\364z\363\346\177”, fmt=0x4a5f34 “s" failed”,
args=0x7fff8f777b90) at src/core/ngx_string.c:253 #1 0x000000000040760f in ngx_log_error_core (level=3, log=0x2cd4400,
err=14, fmt=0x4a5f2a “chmod() "%s" failed”) at src/core/ngx_log.c:119 #2 0x0000000000412bee in ngx_ext_rename_file (src=0x2518370,
to=0x25fe1e8, ext=0x7fff8f7785c0) at src/core/ngx_file.c:545 #3 0x00000000004755a7 in ngx_http_file_cache_update (r=0x280d420,
tf=0x2518368) at src/http/ngx_http_file_cache.c:808 #4 0x0000000000467958 in ngx_http_upstream_process_request
(r=0x280d420) at src/http/ngx_http_upstream.c:2697 #5 0x000000000046779d in ngx_http_upstream_process_upstream
(r=0x280d420, u=0x25fde90) at src/http/ngx_http_upstream.c:2645 #6 0x0000000000466cb2 in ngx_http_upstream_send_response (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:2330 #7 0x00000000004651f2 in ngx_http_upstream_process_header (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:1630 #8 0x0000000000463a90 in ngx_http_upstream_handler (ev=0x7fe796f2d6a0)
at src/http/ngx_http_upstream.c:924 #9 0x0000000000436596 in ngx_epoll_process_events (cycle=0x248ee50,
timer=13, flags=1) at src/event/modules/ngx_epoll_module.c:635 #10 0x0000000000427710 in ngx_process_events_and_timers
(cycle=0x248ee50) at src/event/ngx_event.c:245 #11 0x0000000000434506 in ngx_worker_process_cycle (cycle=0x248ee50,
data=0x0) at src/os/unix/ngx_process_cycle.c:800 #12 0x000000000043114c in ngx_spawn_process (cycle=0x248ee50,
proc=0x43434a <ngx_worker_process_cycle>, data=0x0, name=0x4a8f63
“worker process”, respawn=-3) at src/os/unix/ngx_process.c:196 #13 0x0000000000433318 in ngx_start_worker_processes (cycle=0x248ee50,
n=4, type=-3) at src/os/unix/ngx_process_cycle.c:360 #14 0x0000000000432996 in ngx_master_process_cycle (cycle=0x248ee50) at
src/os/unix/ngx_process_cycle.c:136 #15 0x0000000000405739 in main (argc=1, argv=0x7fff8f778d68) at
src/core/nginx.c:405
(gdb) backtrace full #0 0x000000000040d661 in ngx_vslprintf (buf=0x7fff8f777bf5 <Address
0x7fff8f777bf5 out of bounds>, last=0x7fff8f7783c0 <Address
0x7fff8f7783c0 out of bounds>, fmt=0x4a5f34 “s" failed”,
args=0x7fff8f777b90) at src/core/ngx_string.c:253
p = 0x0
zero = 32 ’ ’
d = 7
f = 6.8670184115474857e-320
scale = 2.9722451710386717e-316
len = 2147479551
slen = <error reading variable slen (Cannot access memory at
address 0x7fff8f777b38)>
i64 = <error reading variable i64 (Cannot access memory at
address 0x7fff8f777b30)>
ui64 = <error reading variable ui64 (Cannot access memory at
address 0x7fff8f777b28)>
ms = 0
width = <error reading variable width (Cannot access memory at
address 0x7fff8f777b20)>
sign = <error reading variable sign (Cannot access memory at
address 0x7fff8f777b18)>
hex = <error reading variable hex (Cannot access memory at
address 0x7fff8f777b10)>
max_width = <error reading variable max_width (Cannot access
memory at address 0x7fff8f777b08)>
frac_width = <error reading variable frac_width (Cannot access
memory at address 0x7fff8f777b00)>
n = 140629904076176
v = 0x2519f50
vv = 0x6c4f80 #1 0x000000000040760f in ngx_log_error_core (level=3, log=0x2cd4400,
err=14, fmt=0x4a5f2a “chmod() "%s" failed”) at src/core/ngx_log.c:119
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area =
0x7fff8f778490, reg_save_area = 0x7fff8f7783d0}}
p = 0x7fff8f777bec <Address 0x7fff8f777bec out of bounds>
last = 0x7fff8f7783c0 <Address 0x7fff8f7783c0 out of bounds>
msg = 0x7fff8f777bec <Address 0x7fff8f777bec out of bounds>
errstr = <error reading variable errstr (Cannot access memory at
address 0x7fff8f777bc0)> #2 0x0000000000412bee in ngx_ext_rename_file (src=0x2518370,
to=0x25fe1e8, ext=0x7fff8f7785c0) at src/core/ngx_file.c:545
name = <error reading variable name (Cannot access memory at
address 0x7fff8f7784e0)>
err = <error reading variable err (Cannot access memory at
address 0x7fff8f7784ec)>
cf = <error reading variable cf (Cannot access memory at address
0x7fff8f7784b0)> #3 0x00000000004755a7 in ngx_http_file_cache_update (r=0x280d420,
tf=0x2518368) at src/http/ngx_http_file_cache.c:808
fs_size = 0
rc = 4586997
uniq = <error reading variable uniq (Cannot access memory at
address 0x7fff8f7785f8)>
fi = <error reading variable fi (Cannot access memory at address
0x7fff8f778530)>
c = <error reading variable c (Cannot access memory at address
0x7fff8f7785f0)>
ext = <error reading variable ext (Cannot access memory at
address 0x7fff8f7785c0)>
cache = <error reading variable cache (Cannot access memory at
address 0x7fff8f7785e8)> #4 0x0000000000467958 in ngx_http_upstream_process_request
(r=0x280d420) at src/http/ngx_http_upstream.c:2697
del = 140632646669984
tf = 0x0
p = 0x2519d00
u = 0x25fde90 #5 0x000000000046779d in ngx_http_upstream_process_upstream
(r=0x280d420, u=0x25fde90) at src/http/ngx_http_upstream.c:2645
c = 0x7fe6f37b5390 #6 0x0000000000466cb2 in ngx_http_upstream_send_response (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:2330
tcp_nodelay = <error reading variable tcp_nodelay (Cannot access
memory at address 0x7fff8f7786fc)>
n = <error reading variable n (Cannot access memory at address
0x7fff8f7786d0)>
rc = <error reading variable rc (Cannot access memory at address
0x7fff8f7786e8)>
p = <error reading variable p (Cannot access memory at address
0x7fff8f7786c0)>
c = <error reading variable c (Cannot access memory at address
0x7fff8f7786e0)>
clcf = <error reading variable clcf (Cannot access memory at
address 0x7fff8f7786d8)> #7 0x00000000004651f2 in ngx_http_upstream_process_header (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:1630
n = 402
rc = 0
c = 0x7fe6f37b5390 #8 0x0000000000463a90 in ngx_http_upstream_handler (ev=0x7fe796f2d6a0)
at src/http/ngx_http_upstream.c:924
c = 0x7fe6f37af450
r = 0x280d420
ctx = 0x2cd4440
u = 0x25fde90 #9 0x0000000000436596 in ngx_epoll_process_events (cycle=0x248ee50,
timer=13, flags=1) at src/event/modules/ngx_epoll_module.c:635
events = 1
revents = 5
instance = 1
i = 0
level = 140735600363552
err = 0
rev = 0x7fe796f2d6a0
wev = 0x7fe6f37c57d0
queue = 0x6d1f80
c = 0x7fe6f37b5390 #10 0x0000000000427710 in ngx_process_events_and_timers
(cycle=0x248ee50) at src/event/ngx_event.c:245
flags = 1
timer = 13
delta = 1314359466746 #11 0x0000000000434506 in ngx_worker_process_cycle (cycle=0x248ee50,
data=0x0) at src/os/unix/ngx_process_cycle.c:800
i = 0
c = 0x7fff8f778d60 #12 0x000000000043114c in ngx_spawn_process (cycle=0x248ee50,
proc=0x43434a <ngx_worker_process_cycle>, data=0x0, name=0x4a8f63
“worker process”, respawn=-3) at src/os/unix/ngx_process.c:196
on = 1
pid = 0
s = 0 #13 0x0000000000433318 in ngx_start_worker_processes (cycle=0x248ee50,
n=4, type=-3) at src/os/unix/ngx_process_cycle.c:360
i = 0
ch = {command = 1, pid = 4399669, slot = 21, fd = 39214374} #14 0x0000000000432996 in ngx_master_process_cycle (cycle=0x248ee50) at
src/os/unix/ngx_process_cycle.c:136
title = 0x0
p = 0x0
size = 0
i = 0
n = 0
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value =
{tv_sec = 0, tv_usec = 0}}
live = 0
delay = 0
ls = 0x0
ccf = 0x0 #15 0x0000000000405739 in main (argc=1, argv=0x7fff8f778d68) at
src/core/nginx.c:405
i = 0
log = 0x0
cycle = 0x0
init_cycle = {conf_ctx = 0x0, pool = 0x248e240, log = 0x6cfd80,
new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data
= 0x0, action = 0x0}, files = 0x0, free_connections = 0x0,
free_connection_n = 0,
reusable_connections_queue = {prev = 0x0, next = 0x0},
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
pathes = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
open_files = {last = 0x0,
part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc
= 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts
= 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0,
files_n = 0,
connections = 0x0, read_events = 0x0, write_events = 0x0,
old_cycle = 0x0, conf_file = {len = 0, data = 0x0}, conf_param = {len =
0, data = 0x0}, conf_prefix = {len = 0, data = 0x0}, prefix = {len = 0,
data = 0x0},
lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data =
0x0}}
ccf = 0x0
(gdb)
On Fri, Aug 26, 2011 at 07:54:34AM -0400, magicbear wrote:
[…]
#4 0x0000000000467958 in ngx_http_upstream_process_request
(r=0x280d420) at src/http/ngx_http_upstream.c:2697 #5 0x000000000046779d in ngx_http_upstream_process_upstream
(r=0x280d420, u=0x25fde90) at src/http/ngx_http_upstream.c:2645 #6 0x0000000000466cb2 in ngx_http_upstream_send_response (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:2330 #7 0x00000000004651f2 in ngx_http_upstream_process_header (r=0x280d420,
u=0x25fde90) at src/http/ngx_http_upstream.c:1630 #8 0x0000000000463a90 in ngx_http_upstream_handler (ev=0x7fe796f2d6a0)
at src/http/ngx_http_upstream.c:924
Backtrace looks unrelated to the problem I was thinking about, but
may be indirectly caused by it due to memory corruption.
Please try the following patch for upstream keepalive module
instead of the one I’ve posted earlier:
It get the same result, I think may be cause by my upload script? I will
try to disable it to get result.
In another more high load server(without patch) using this script is no
this problem.
gdb which nginx core
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/…
Reading symbols from /usr/sbin/nginx…done.
[New Thread 17587]
warning: Can’t read pathname for load map: 输入/输出错误.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libpcre.so.3
Reading symbols from /lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/local/lib/libprofiler.so.0…done.
Loaded symbols for /usr/local/lib/libprofiler.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/libunwind.so.7…(no debugging
symbols found)…done.
Loaded symbols for /usr/local/lib/libunwind.so.7
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6…(no
debugging symbols found)…done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1…(no debugging
symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_dns.so.2
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2…(no
debugging symbols found)…done.
Loaded symbols for /lib/x86_64-linux-gnu/libresolv.so.2
Core was generated by `nginx:'.
Program terminated with signal 11, Segmentation fault. #0 0x000000000040cff5 in ngx_vslprintf (buf=0x7fff3babc4d5 “”,
last=0x7fff3babcca0 “\220\274\020\212\067\177”, fmt=0x49a2ac “s"
failed”, args=0x7fff3babc470) at src/core/ngx_string.c:253
253 while (*p && buf < last) {
(gdb) bt #0 0x000000000040cff5 in ngx_vslprintf (buf=0x7fff3babc4d5 “”,
last=0x7fff3babcca0 “\220\274\020\212\067\177”, fmt=0x49a2ac “s"
failed”, args=0x7fff3babc470) at src/core/ngx_string.c:253 #1 0x000000000040731f in ngx_log_error_core (level=3, log=0x288eec0,
err=14, fmt=0x49a2a2 “chmod() "%s" failed”) at src/core/ngx_log.c:119 #2 0x00000000004124ac in ngx_ext_rename_file (src=0x2146620,
to=0x2239458, ext=0x7fff3babcea0) at src/core/ngx_file.c:545 #3 0x000000000046c680 in ngx_http_file_cache_update (r=0x27a68d0,
tf=0x2146618) at src/http/ngx_http_file_cache.c:808 #4 0x000000000045f8ba in ngx_http_upstream_process_request
(r=0x27a68d0) at src/http/ngx_http_upstream.c:2697 #5 0x000000000045f6ff in ngx_http_upstream_process_upstream
(r=0x27a68d0, u=0x2239100) at src/http/ngx_http_upstream.c:2645 #6 0x000000000045ed87 in ngx_http_upstream_send_response (r=0x27a68d0,
u=0x2239100) at src/http/ngx_http_upstream.c:2330 #7 0x000000000045d38c in ngx_http_upstream_process_header (r=0x27a68d0,
u=0x2239100) at src/http/ngx_http_upstream.c:1630 #8 0x000000000045bdba in ngx_http_upstream_handler (ev=0x7f382f880ec0)
at src/http/ngx_http_upstream.c:924 #9 0x00000000004322a1 in ngx_epoll_process_events (cycle=0x1828d90,
timer=213, flags=1) at src/event/modules/ngx_epoll_module.c:635 #10 0x00000000004259b6 in ngx_process_events_and_timers
(cycle=0x1828d90) at src/event/ngx_event.c:245 #11 0x0000000000430680 in ngx_worker_process_cycle (cycle=0x1828d90,
data=0x0) at src/os/unix/ngx_process_cycle.c:800 #12 0x000000000042d987 in ngx_spawn_process (cycle=0x1828d90,
proc=0x4304fd <ngx_worker_process_cycle>, data=0x0, name=0x49c6a0
“worker process”, respawn=-4) at src/os/unix/ngx_process.c:196 #13 0x000000000042f891 in ngx_start_worker_processes (cycle=0x1828d90,
n=4, type=-4) at src/os/unix/ngx_process_cycle.c:360 #14 0x000000000042f46a in ngx_master_process_cycle (cycle=0x1828d90) at
src/os/unix/ngx_process_cycle.c:249 #15 0x00000000004054b9 in main (argc=1, argv=0x7fff3babd638) at
src/core/nginx.c:405
(gdb) backtrace full #0 0x000000000040cff5 in ngx_vslprintf (buf=0x7fff3babc4d5 “”,
last=0x7fff3babcca0 “\220\274\020\212\067\177”, fmt=0x49a2ac “s"
failed”, args=0x7fff3babc470) at src/core/ngx_string.c:253
p = 0x0
zero = 32 ’ ’
d = -1978670320
f = 0
scale = 0
len = 46
slen = 18446744073709551615
i64 = 0
ui64 = 0
ms = 0
width = 0
sign = 1
hex = 0
max_width = 0
frac_width = 0
n = 2147479551
v = 0x404e40
vv = 0x6b5f60 #1 0x000000000040731f in ngx_log_error_core (level=3, log=0x288eec0,
err=14, fmt=0x49a2a2 “chmod() "%s" failed”) at src/core/ngx_log.c:119
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area =
0x7fff3babcd70, reg_save_area = 0x7fff3babccb0}}
p = 0x7fff3babc4cc “chmod() "”
last = 0x7fff3babcca0 “\220\274\020\212\067\177”
msg = 0x7fff3babc4cc “chmod() "”
errstr = “2011/08/27 00:12:23 [crit] 17587#0: 161816 chmod()
"\000\000\000\020\273\020\212\006\000\000\000\344r\377\001\000\000\000\000
ʫ;\377\177\000\000@N@\000\000\000\000\000\377\357\377\177\000\000\000\000\320\362\257\001\000\000\000\000\220\223\017\212\067\177\000\000\v\vJ\000\000\000\000\000\260ɫ;\377\177\000\000\260ū;\377\177\000\000\220\223\017\212\067\177\000\000t\003\000\000\000\000\000\000pʫ;\377\177\000\000@N@\000\000\000\000\000[,C",
‘\000’ <repeats 13 times>“\220,
ʫ;\377\177\000\000@N@\000\000\000\000\000[,C\000\000\000\000\000\032\316l\002\000\000\000\000\262\000\000\000\000\000\000\000tw\001\002\000\000\000\000\255\001\000\000\000\000\000\000\270”… #2 0x00000000004124ac in ngx_ext_rename_file (src=0x2146620,
to=0x2239458, ext=0x7fff3babcea0) at src/core/ngx_file.c:545
name = 0x24531a0 “\a”
err = 0
cf = {size = 25934928, buf_size = 41576656, access =
140734194503312, time = 4245912, log = 0x7fff3babcdf0} #3 0x000000000046c680 in ngx_http_file_cache_update (r=0x27a68d0,
tf=0x2146618) at src/http/ngx_http_file_cache.c:808
fs_size = 0
rc = 4557437
uniq = 0
fi = {st_dev = 140734194503248, st_ino = 4557810, st_nlink =
140734194503312, st_mode = 26099352, st_uid = 0, st_gid = 1001115232,
__pad0 = 32767, st_rdev = 4656351, st_size = 140734194502976, st_blksize
= 1, st_blocks = 16,
st_atim = {tv_sec = 16, tv_nsec = 34900240}, st_mtim = {tv_sec
= 4678845, tv_nsec = 35885888}, st_ctim = {tv_sec = 38091144, tv_nsec =
3641}, __unused = {-2, 25566800, 47286217320}}
c = 0x2239450
ext = {access = 384, path_access = 384, time = -1, fd = 4365586,
create_path = 1, delete_file = 1, log = 0x288eec0}
cache = 0x18cb888 #4 0x000000000045f8ba in ngx_http_upstream_process_request
(r=0x27a68d0) at src/http/ngx_http_upstream.c:2697
del = 139879292341952
tf = 0x0
p = 0x2453988
u = 0x2239100 #5 0x000000000045f6ff in ngx_http_upstream_process_upstream
(r=0x27a68d0, u=0x2239100) at src/http/ngx_http_upstream.c:2645
c = 0x7f378a109290 #6 0x000000000045ed87 in ngx_http_upstream_send_response (r=0x27a68d0,
u=0x2239100) at src/http/ngx_http_upstream.c:2330
tcp_nodelay = 0
n = 140734194503664
rc = 0
p = 0x2453988
c = 0x7f378a10bc90
clcf = 0x18e3330 #7 0x000000000045d38c in ngx_http_upstream_process_header (r=0x27a68d0,
u=0x2239100) at src/http/ngx_http_upstream.c:1630
n = 396
rc = 0
c = 0x7f378a109290 #8 0x000000000045bdba in ngx_http_upstream_handler (ev=0x7f382f880ec0)
at src/http/ngx_http_upstream.c:924
c = 0x7f378a10bc90
r = 0x27a68d0
ctx = 0x288ef00
—Type to continue, or q to quit—
u = 0x2239100 #9 0x00000000004322a1 in ngx_epoll_process_events (cycle=0x1828d90,
timer=213, flags=1) at src/event/modules/ngx_epoll_module.c:635
events = 1
revents = 5
instance = 0
i = 0
level = 140734194503920
err = 0
rev = 0x7f382f880ec0
wev = 0x7f3789f553f8
queue = 0x6c2f60
c = 0x7f378a109290 #10 0x00000000004259b6 in ngx_process_events_and_timers
(cycle=0x1828d90) at src/event/ngx_event.c:245
flags = 1
timer = 213
delta = 1314375143673 #11 0x0000000000430680 in ngx_worker_process_cycle (cycle=0x1828d90,
data=0x0) at src/os/unix/ngx_process_cycle.c:800
i = 0
c = 0x7fff3babd630 #12 0x000000000042d987 in ngx_spawn_process (cycle=0x1828d90,
proc=0x4304fd <ngx_worker_process_cycle>, data=0x0, name=0x49c6a0
“worker process”, respawn=-4) at src/os/unix/ngx_process.c:196
on = 1
pid = 0
s = 1 #13 0x000000000042f891 in ngx_start_worker_processes (cycle=0x1828d90,
n=4, type=-4) at src/os/unix/ngx_process_cycle.c:360
i = 1
ch = {command = 1, pid = 17586, slot = 0, fd = 13} #14 0x000000000042f46a in ngx_master_process_cycle (cycle=0x1828d90) at
src/os/unix/ngx_process_cycle.c:249
title = 0x1904836 "\221”
p = 0x190484a “8748”
size = 21
i = 1
n = 11206445643057
sigio = 0
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value =
{tv_sec = 0, tv_usec = 6}}
live = 1
delay = 0
ls = 0x0
ccf = 0x18294f0 #15 0x00000000004054b9 in main (argc=1, argv=0x7fff3babd638) at
src/core/nginx.c:405
i = 47
log = 0x6c0d60
cycle = 0x1828d90
init_cycle = {conf_ctx = 0x0, pool = 0x1828240, log = 0x6c0d60,
new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0, data
= 0x0, action = 0x0}, files = 0x0, free_connections = 0x0,
free_connection_n = 0,
reusable_connections_queue = {prev = 0x0, next = 0x0},
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
pathes = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
open_files = {last = 0x0,
part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc
= 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts
= 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0,
files_n = 0,
connections = 0x0, read_events = 0x0, write_events = 0x0,
old_cycle = 0x0, conf_file = {len = 21, data = 0x49977e
“/etc/nginx/nginx.conf”}, conf_param = {len = 0, data = 0x0},
conf_prefix = {len = 11,
data = 0x49977e “/etc/nginx/nginx.conf”}, prefix = {len =
17, data = 0x49976c “/usr/local/nginx/”}, lock_file = {len = 0, data =
0x0}, hostname = {len = 0, data = 0x0}}
ccf = 0x1829c90
