# Nginx crash - possibly keepalives **URL:** \ **Category:** NGINX\ **Created:** [October 12, 2011, 4:54am UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731 "2011-10-12T04:54:17Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\ **Post date:** [October 12, 2011, 4:54am UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/1 "2011-10-12T04:54:17Z")
Hi all, After turning on keepalives, we’ve been seeing one crash pretty consistently. We’re running nginx 1.1.5 and here is the backtrace : (gdb) bt #0 ngx\_write\_chain\_to\_file (file=0x173dbd0, cl=0x0, offset=26779, pool=0x162b850) at src/os/unix/ngx\_files.c:162 #1 0x000000000040ea3e in ngx\_write\_chain\_to\_temp\_file (tf=0x173dbd0, chain=0x0) at src/core/ngx\_file.c:35 #2 0x000000000041c534 in ngx\_event\_pipe\_write\_chain\_to\_temp\_file (p=0x173d048) at src/event/ngx\_event\_pipe.c:771 #3 0x000000000041cd3d in ngx\_event\_pipe\_read\_upstream (p=0x173d048, do\_write=0) at src/event/ngx\_event\_pipe.c:234 #4 ngx\_event\_pipe (p=0x173d048, do\_write=0) at src/event/ngx\_event\_pipe.c:50 #5 0x000000000043e1b6 in ngx\_http\_upstream\_process\_upstream (r=0x1ae4bb0, u=0x1afb4f8) at src/http/ngx\_http\_upstream.c:2704 #6 0x000000000043e612 in ngx\_http\_upstream\_handler (ev=0x689b) at src/http/ngx\_http\_upstream.c:936 #7 0x000000000041b1f6 in ngx\_event\_process\_posted (cycle=, posted=0x0) at src/event/ngx\_event\_posted.c:39 #8 0x000000000041b0c6 in ngx\_process\_events\_and\_timers (cycle=0x151bbf0) at src/event/ngx\_event.c:272 #9 0x0000000000420bc3 in ngx\_worker\_process\_cycle (cycle=0x151bbf0, data=) at src/os/unix/ngx\_process\_cycle.c:801 #10 0x000000000041f4cc in ngx\_spawn\_process (cycle=0x151bbf0, proc=0x420aeb \, data=, name=0x470dc9 “worker process”, respawn=5) at src/os/unix/ngx\_process.c:196 #11 0x0000000000421359 in ngx\_reap\_children (cycle=0x151bbf0) at src/os/unix/ngx\_process\_cycle.c:617 #12 ngx\_master\_process\_cycle (cycle=0x151bbf0) at src/os/unix/ngx\_process\_cycle.c:180 #13 0x000000000040700a in main (argc=, argv=) at src/core/nginx.c:405 It seems that p-\>in is NULL in ngx\_event\_pipe\_write\_chain\_to\_temp\_file() but I can’t figure out why. It’s using the proxy module as an upstream. Let me know I could provide more info to help! Matthieu. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 12, 2011, 2:20pm UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/2 "2011-10-12T14:20:16Z")
Hello! On Tue, Oct 11, 2011 at 07:53:36PM -0700, Matthieu T. wrote: > chain=0x0) at src/core/ngx\_file.c:35 > #2 0x000000000041c534 in ngx\_event\_pipe\_write\_chain\_to\_temp\_file > (p=0x173d048) at src/event/ngx\_event\_pipe.c:771 Do you have any local patches here or it’s just some debug logging added? $ grep -n ngx\_write\_chain\_to\_temp\_file src/event/ngx\_event\_pipe.c 521: if (ngx\_write\_chain\_to\_temp\_file(p-\>temp\_file, &file) 765: if (ngx\_write\_chain\_to\_temp\_file(p-\>temp\_file, out) == NGX\_ERROR) { > #8 0x000000000041b0c6 in ngx\_process\_events\_and\_timers (cycle=0x151bbf0) at > #13 0x000000000040700a in main (argc=, argv=) at src/core/nginx.c:405 > > It seems that p-\>in is NULL in ngx\_event\_pipe\_write\_chain\_to\_temp\_file() but > I can’t figure out why. > It’s using the proxy module as an upstream. > > Let me know I could provide more info to help! Could you please show: fr 2 p \*p p p-\>bufs p \*p-\>temp\_file fr 5 p \*r p \*u p u-\>headers\_in I think I see what’s going on here, just to confirm. The underlying issue seems to be one identified in commit message here: [http://trac.nginx.org/nginx/changeset/4136/nginx](http://trac.nginx.org/nginx/changeset/4136/nginx) (all available buffers in the p-\>out chain) Setting proxy\_busy\_buffers to something lower compared to proxy\_buffers may help as a workaround. Maxim D. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 12, 2011, 7:23pm UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/3 "2011-10-12T19:23:06Z")
Hello! On Wed, Oct 12, 2011 at 09:32:41AM -0700, Matthieu T. wrote: > > > We’re running nginx 1.1.5 and here is the backtrace : > > > […] > length = 168, … } > Ok, it looks like I’m right and all buffers are in busy chain. > Likely this happens due to upstream sending response in many small > chunks. I’ll try to reproduce it here and provide proper fix. […] > > > > We’ll try that and see if that has a positive effect, thank you. > Also, would having u-\>buffering == 0 (proxy\_buffering off) help in this > scenario ? Yes, switching off proxy\_buffering should help. Switching off disk buffering (proxy\_cache off; proxy\_max\_temp\_file\_size 0;) should help as well. Maxim D. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 14, 2011, 9:44pm UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/4 "2011-10-14T21:44:01Z")
Hello! On Wed, Oct 12, 2011 at 09:22:41PM +0400, Maxim D. wrote: > > > > Hi all, > > > > > > > > After turning on keepalives, we’ve been seeing one crash pretty > > > > consistently. > > > > We’re running nginx 1.1.5 and here is the backtrace : […] > Ok, it looks like I’m right and all buffers are in busy chain. > Likely this happens due to upstream sending response in many small > chunks. > > I’ll try to reproduce it here and provide proper fix. Please try the attached patch. Maxim D. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 12, 2011, 6:40pm UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/5 "2011-10-12T18:40:31Z")
On Wed, Oct 12, 2011 at 5:20 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > (gdb) bt > > #0 ngx\_write\_chain\_to\_file (file=0x173dbd0, cl=0x0, offset=26779, > > pool=0x162b850) at src/os/unix/ngx\_files.c:162 > > #1 0x000000000040ea3e in ngx\_write\_chain\_to\_temp\_file (tf=0x173dbd0, > > chain=0x0) at src/core/ngx\_file.c:35 > > #2 0x000000000041c534 in ngx\_event\_pipe\_write\_chain\_to\_temp\_file > > (p=0x173d048) at src/event/ngx\_event\_pipe.c:771 > > Do you have any local patches here or it’s just some debug logging > added? There is one patch isn’t in this code path and shouldn’t have any effect here. > (r=0x1ae4bb0, > > > data=) at src/os/unix/ngx\_process\_cycle.c:801 > > p \*p > p p-\>bufs > p \*p-\>temp\_file > > fr 5 > p \*r > p \*u > p u-\>headers\_in # (gdb) fr 2 #2 0x000000000041c534 in ngx\_event\_pipe\_write\_chain\_to\_temp\_file (p=0x173d048) at src/event/ngx\_event\_pipe.c:771 771 src/event/ngx\_event\_pipe.c: No such file or directory. in src/event/ngx\_event\_pipe.c (gdb) p \*p $1 = {upstream = 0x7f3e2cbe1010, downstream = 0x7f3e2cbdd950, free\_raw\_bufs = 0x0, in = 0x0, last\_in = 0x173d060, out = 0x0, last\_out = 0x17b9bf8, free = 0x17b9b30, busy = 0x17b9ec0, input\_filter = 0x45c873 \, input\_ctx = 0x1ae4bb0, output\_filter = 0x42897f \, output\_ctx = 0x1ae4bb0, read = 0, cacheable = 1, single\_buf = 0, free\_bufs = 1, upstream\_done = 0, upstream\_error = 0, upstream\_eof = 0, upstream\_blocked = 0, downstream\_done = 0, downstream\_error = 0, cyclic\_temp\_file = 0, cache\_file\_maxed = 0, allocated = 9, bufs = {num = 9, size = 8192}, tag = 0x693400, busy\_size # 8192, read\_length = 27269, length = 168, max\_temp\_file\_size = 1048576, max\_cache\_file\_size = 0, temp\_file\_write\_size = 16384, read\_timeout = 120000, send\_timeout = 60000, send\_lowat = 0, pool = 0x162b850, log = 0x16530b0, preread\_bufs = 0x0, preread\_size = 12, buf\_to\_file = 0x0, temp\_file = 0x173dbd0, num = 61} (gdb) p p-\>bufs $2 = {num = 9, size = 8192} (gdb) p \*p-\>temp\_file $3 = {file = {fd = 308, name = {len = 46, data = 0x173dd88 “/usr/local/nginx-cache/storage/temp/0009520674”}, info = {st\_dev = 0, st\_ino = 0, st\_nlink = 0, st\_mode = 0, st\_uid = 0, st\_gid = 0, \_\_pad0 = 0, st\_rdev = 0, st\_size = 0, st\_blksize = 0, st\_blocks = 0, st\_atim = {tv\_sec = 0, tv\_nsec = 0}, st\_mtim = {tv\_sec = 0, tv\_nsec = 0}, st\_ctim = {tv\_sec = 0, tv\_nsec = 0}, \_\_unused = {0, 0, 0}}, offset 26779, sys\_offset = 26779, log = 0x16530b0, valid\_info = 0, directio = 0}, offset = 26779, path = 0x152efa0, pool = 0x162b850, warn = 0x0, access = 0, log\_level = 0, persistent = 1, clean = 0} # (gdb) fr 5 #5 0x000000000043e1b6 in ngx\_http\_upstream\_process\_upstream (r=0x1ae4bb0, u=0x1afb4f8) at src/http/ngx\_http\_upstream.c:2704 2704 src/http/ngx\_http\_upstream.c: No such file or directory. in src/http/ngx\_http\_upstream.c (gdb) p \*r $4 = {signature = 1347703880, connection = 0x7f3e2cbdd950, ctx = 0x162bc60, main\_conf = 0x151c9e0, srv\_conf = 0x1537c68, loc\_conf = 0x157d2c0, read\_event\_handler = 0x43e840 \, write\_event\_handler = 0x43edf5 \, cache = 0x173d188, upstream = 0x1afb4f8, upstream\_states = 0x173d658, pool = 0x162b850, header\_in = 0x15bff00, headers\_in = {headers = {last = 0x162c668, part = {elts = 0x1afa9b0, nelts = 20, next = 0x162c668}, size = 48, nalloc = 20, pool = 0x162b850}, host = 0x1afaa10, connection = 0x0, if\_modified\_since = 0x1afae30, if\_unmodified\_since = 0x0, user\_agent = 0x1afae00, referer = 0x0, content\_length = 0x0, content\_type = 0x0, range = 0x0, if\_range = 0x0, transfer\_encoding = 0x0, expect = 0x0, accept\_encoding = 0x0, via = 0x0, authorization = 0x0, keep\_alive = 0x0, x\_forwarded\_for = 0x0, x\_real\_ip = 0x1afaa40, user = {len = 0, data = 0x0}, passwd = {len = 0, data = 0x0}, cookies = {elts = 0x162c558, nelts = 0, size = 8, nalloc = 2, pool = 0x162b850}, server = {len = 13, data = 0x162c57d “sir-emeth.comx-real-ipconnect-via-httpscf-host-origin-ipcf-ipcountrycf-iptypecf-use-byccf-use-obcf-pref-obcf-pref-geoloccf-identify-botcf-cache-levelcf-force-miss-tscf-minify-modehost-spoofset-expires”…}, content\_length\_n = -1, keep\_alive\_n = -1, connection\_type = 0, msie = 0, msie6 = 0, opera = 0, gecko = 0, chrome = 0, safari = 0, konqueror = 0}, headers\_out = {headers = {last = 0x1ae4d58, part = {elts = 0x162b8a0, nelts = 2, next = 0x0}, size = 48, nalloc = 20, pool = 0x162b850}, status = 200, status\_line = {len = 6, data = 0x1afb981 “200 OK0 \321s\001”}, server = 0x0, date = 0x0, content\_length = 0x0, content\_encoding = 0x0, location = 0x0, refresh = 0x0, last\_modified = 0x0, content\_range = 0x0, accept\_ranges = 0x0, www\_authenticate = 0x0, expires = 0x0, etag = 0x0, override\_charset # 0x0, content\_type\_len = 9, content\_type = {len = 9, data = 0x173d9a8 “text/html”}, charset = {len = 0, data = 0x0}, content\_type\_lowcase = 0x0, content\_type\_hash = 0, cache\_control = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, content\_length\_n = -1, date\_time = 0, last\_modified\_time = -1}, request\_body = 0x173d158, lingering\_time = 0, start\_sec = 1318377836, start\_msec = 445, method = 2, http\_version = 1001, request\_line = {len = 116, data = 0x1822f70 “GET /bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”}, uri = {len = 12, data = 0x1afb392 “/@@se/bible/”}, args = {len = 95, data = 0x1822f7c “bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”}, exten = {len = 0, data = 0x0}, unparsed\_uri = {len = 103, data = 0x1822f74 “/bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”}, method\_name = {len = 3, data = 0x1822f70 “GET /bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”}, http\_protocol = { len = 8, data = 0x1822fdc “HTTP/1.1\r\nCF-IDTag”}, out = 0x17ba158, main = 0x1ae4bb0, parent = 0x0, postponed = 0x0, post\_subrequest = 0x0, posted\_requests = 0x0, virtual\_names = 0x0, phase\_handler = 11, content\_handler = 0x45960e \, access\_code = 0, variables = 0x162bdd8, ncaptures = 0, captures = 0x1afb130, captures\_data = 0x1822f74 “/bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”, limit\_rate = 0, header\_size = 271, request\_length = 736, err\_status = 0, http\_connection = 0x1653108, log\_handler = 0x42f4b2 \, cleanup # 0x173d6b8, subrequests = 51, count = 1, blocked = 0, aio = 0, http\_state = 2, complex\_uri = 0, quoted\_uri = 0, plus\_in\_uri = 0, space\_in\_uri = 0, invalid\_header = 0, add\_uri\_to\_alias = 0, valid\_location = 1, valid\_unparsed\_uri = 0, uri\_changed = 0, uri\_changes = 10, request\_body\_in\_single\_buf = 0, request\_body\_in\_file\_only = 0, request\_body\_in\_persistent\_file = 0, request\_body\_in\_clean\_file = 0, request\_body\_file\_group\_access = 0, request\_body\_file\_log\_level = 5, subrequest\_in\_memory = 0, waited = 0, cached = 0, gzip\_tested = 0, gzip\_ok = 0, gzip\_vary = 0, gzip\_force = 0, proxy = 0, bypass\_cache = 0, no\_cache = 0, limit\_zone\_set = 0, limit\_req\_set = 0, pipeline = 0, plain\_http = 0, chunked = 1, header\_only = 0, keepalive = 1, lingering\_close = 0, discard\_body = 0, internal = 1, error\_page = 0, ignore\_content\_encoding = 0, filter\_finalize = 0, post\_action = 0, request\_complete = 0, request\_output = 1, header\_sent = 1, expect\_tested = 0, root\_tested = 0, done = 0, logged = 0, buffered = 0, main\_filter\_need\_in\_memory = 0, filter\_need\_in\_memory = 0, filter\_need\_temporary = 0, allow\_ranges = 0, stat\_reading = 0, stat\_writing = 1, state = 0, header\_hash = 116750, lowcase\_index = 3, lowcase\_header = “viansfer-encoding”, ‘\000’ \, header\_name\_start = 0x1ae749c “33’\>\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\r\n1a\r\n\n \r\n6\r\n \r\n9\r\n \n\r\n”…, header\_name\_end = 0x1ae745c “‘20’ width=‘5’\>\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, "\n "…, header\_start = 0x1ae745e “0’ width=‘5’\>\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, "\n “…, header\_end = 0x1ae749c “33’\>\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\n”, ’ ’ \, “\r\n1a\r\n\n \r\n6\r\n \r\n9\r\n \n\r\n”…, uri\_start = 0x1822f74 “/bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”, uri\_end = 0x1822fdb " HTTP/1.1\r\nCF-IDTag”, uri\_ext = 0x0, args\_start = 0x1822f7c “bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”, request\_start = 0x1822f70 “GET /bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”, request\_end = 0x1822fe4 “\r\nCF-IDTag”, method\_end = 0x1822f72 “T /bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nCF-IDTag”, schema\_start = 0x0, schema\_end = 0x0, host\_start = 0x0, host\_end = 0x0, port\_start = 0x0, port\_end = 0x0, http\_minor = 1, http\_major = 1} (gdb) p \*u $5 = {read\_event\_handler = 0x43e15b \, write\_event\_handler = 0x43b866 \, peer = { connection = 0x7f3e2cbe1010, sockaddr = 0x1afb970, socklen = 16, name = 0x173d730, tries = 1, get = 0x465c0a \, free = 0x465954 \, data = 0x173d780, set\_session = 0x465262 \, save\_session = 0x465275 \, local = 0x157cf00, rcvbuf = 0, log = 0x16530b0, cached = 0, log\_error = 1}, pipe = 0x173d048, request\_bufs = 0x173d648, output = {buf = 0x0, in = 0x0, free = 0x0, busy = 0x0, sendfile = 1, directio = 0, unaligned = 0, need\_in\_memory = 0, need\_in\_temp = 0, alignment = 512, pool = 0x162b850, allocated = 0, bufs = {num = 1, size = 131072}, tag = 0x693400, output\_filter = 0x4096f0 \, filter\_ctx = 0x1afb5f0}, writer = {out = 0x0, last = 0x1afb5f0, connection = 0x7f3e2cbe1010, pool = 0x162b850, limit = 0}, conf = 0x157d858, headers\_in = {headers = {last = 0x1afb628, part = { elts = 0x173d7e0, nelts = 5, next = 0x0}, size = 48, nalloc = 8, pool = 0x162b850}, status\_n = 200, status\_line = {len = 6, data = 0x1afb981 “200 OK0 \321s\001”}, status = 0x0, date = 0x173d7e0, server = 0x173d810, connection = 0x0, expires = 0x0, etag = 0x0, x\_accel\_expires = 0x0, x\_accel\_redirect = 0x0, x\_accel\_limit\_rate = 0x0, content\_type = 0x173d840, content\_length = 0x0, last\_modified = 0x0, location = 0x0, accept\_ranges = 0x0, www\_authenticate = 0x0, transfer\_encoding = 0x173d870, content\_encoding = 0x0, content\_length\_n = -1, cache\_control = {elts # 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, connection\_close = 0, chunked = 1}, resolved = 0x173d000, buffer = { pos = 0x1ae7330 “…”, last = 0x1ae7330 “…”, file\_pos = 0, file\_last = 0, start = 0x1ae7330 “…”, end = 0x1ae9330 “”, tag = 0x693400, file = 0x0, shadow = 0x0, temporary = 1, memory = 0, mmap = 0, recycled = 1, in\_file = 0, flush = 0, sync = 0, last\_buf = 0, last\_in\_chain = 0, last\_shadow = 0, temp\_file = 0, num = 9}, length = -1, out\_bufs = 0x0, busy\_bufs = 0x0, free\_bufs = 0x0, input\_filter\_init = 0x4584f1 \, input\_filter = 0x459db3 \, input\_filter\_ctx = 0x1ae4bb0, create\_key = 0x45b21c \, max\_cache\_file\_size = 0, create\_request = 0x45a9bf \, reinit\_request = 0x458470 \, process\_header = 0x45a572 \, abort\_request = 0x45861a \, finalize\_request = 0x45861c \, rewrite\_redirect = 0, timeout = 0, state = 0x173d680, method = {len = 0, data = 0x0}, schema = {len = 7, data = 0x1afb8f0 “http:///bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly”}, uri = {len = 103, data = 0x173d4b4 "/bible/?bookSelect=2+Chronicles&chapterSelect=13&verseSelect=7&addedRadio=Brackets&vsRefRadio=VerseOnly HTTP/1.1\r\nHost: [sir-emeth.com](http://sir-emeth.com)\r\nConnection: Keep-Alive\r\nAccept-Encoding: gzip\r\nCF-Connecting-IP: "…}, cleanup # 0x173d6b8, bind\_hash = 3776832724, store = 0, cacheable = 1, no\_cache = 0, accel = 1, ssl = 0, cache\_status = 1, buffering = 1, keepalive = 0, keepalive\_status = 0, request\_sent = 1, header\_sent = 1} (gdb) p u-\>headers\_in $6 = {headers = {last = 0x1afb628, part = {elts = 0x173d7e0, nelts = 5, next = 0x0}, size = 48, nalloc = 8, pool = 0x162b850}, status\_n = 200, status\_line = {len = 6, data = 0x1afb981 “200 OK0 \321s\001”}, status = 0x0, date = 0x173d7e0, server = 0x173d810, connection = 0x0, expires = 0x0, etag = 0x0, x\_accel\_expires = 0x0, x\_accel\_redirect = 0x0, x\_accel\_limit\_rate = 0x0, content\_type = 0x173d840, content\_length = 0x0, last\_modified = 0x0, location = 0x0, accept\_ranges = 0x0, www\_authenticate = 0x0, transfer\_encoding = 0x173d870, content\_encoding = 0x0, content\_length\_n = -1, cache\_control = {elts = 0x0, nelts = 0, size 0, nalloc = 0, pool = 0x0}, connection\_close = 0, chunked = 1} Hope that this helps! > We’ll try that and see if that has a positive effect, thank you. Also, would having u-\>buffering == 0 (proxy\_buffering off) help in this scenario ? ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 19, 2011, 3:33am UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/6 "2011-10-19T03:33:40Z")
So in order to try to reproduce the problem locally, I made this debug module : > **[GitHub - mtourne/ngx\_massive\_chunker: Debug module to force nginx to output a...](https://github.com/mtourne/ngx_massive_chunker)** > > Debug module to force nginx to output a lot of small packets. - GitHub - mtourne/ngx\_massive\_chunker: Debug module to force nginx to output a lot of small packets. This is meant to have 2 nginx chained one to the other. nginx#1 has proxy\_pass to nginx#2 while using keepalives nginx#2 has mass\_chunk on, which for this extreme test will chunk a 15M file into 10 bytes chunk (in different tcp packets or not). I wasn’t able to reproduce the segfault, but when I’m using proxy\_buffering off it works fine. But, when I’m using proxy\_buffering and proxy\_caching (with sendfile on, and directio 2M). It seems like readv() returns 0, but finalize\_request returns NGX\_AGAIN. I’m not sure that test is relevant, and what I’m really benching here. And if it might not be the tcp layer directly? But maybe the failure should be explicit. The behavior was the same with or without the patch. I’ve also tried various combination of proxy buffers, sendfile, directio. Any thoughts ? Thank you! Matthieu. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 19, 2011, 7:50pm UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/7 "2011-10-19T19:50:35Z")
On Wed, Oct 19, 2011 at 2:24 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > nginx#2 has mass\_chunk on, which for this extreme test will chunk a 15M > > done carefully to avoid making some proxy buffers busy here, as > > The ngx\_http\_finalize\_request() is void, it doesn’t return anything. > > You may want to provide debug log to make it clear what are you > talking about. I’m not exactly sure what part of it is relevant, so instead of posting pages of log I provided steps to reproduce a configuration where nginx shows an erratic behavior, by not outputing the entire response while doing a proxy\_pass. I’ll be experimenting more, and I’ll let you know if I find the origin of this issue. M. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 19, 2011, 11:24am UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/8 "2011-10-19T11:24:46Z")
Hello! On Tue, Oct 18, 2011 at 06:32:53PM -0700, Matthieu T. wrote: > proxy\_buffering off it works fine. > To reproduce the segfault you have to: 1. Fill client socket buffers to make sure no buffers will be freed at (2), i.e. all buffers will become busy. This should be done carefully to avoid making some proxy buffers busy here, as this is likely to make (2) impossible. 2. Trigger “all buffers in the out chain” problem by sending multiple small chunks at once. Amount of data must be less than proxy\_busy\_buffers, but total input size (including chunked encoding overhead) must be big enough to consume all proxy\_buffers (including first one, proxy\_buffer\_size). 3. Send some more data to actually trigger segfault due to all buffers being busy. I’m able to reproduce it here (using specially crafted backend server and specially crafted client), and I’m not really intrested in another artificial reproduction. > But, when I’m using proxy\_buffering and proxy\_caching (with sendfile on, and > directio 2M). > It seems like readv() returns 0, but finalize\_request returns NGX\_AGAIN. ENOPARSE The ngx\_http\_finalize\_request() is void, it doesn’t return anything. You may want to provide debug log to make it clear what are you talking about. Maxim D. ---
**Author:** ![Matthieu\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/71c47a/32.png) [@Matthieu\_T](https://www.ruby-forum.com/u/Matthieu_T)\ **Post date:** [October 20, 2011, 12:12am UTC](https://www.ruby-forum.com/t/nginx-crash-possibly-keepalives/212731/9 "2011-10-20T00:12:46Z")
Hello! On Wed, Oct 19, 2011 at 10:49:26AM -0700, Matthieu T. wrote: > On Wed, Oct 19, 2011 at 2:24 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > Hello! > > > > On Tue, Oct 18, 2011 at 06:32:53PM -0700, Matthieu T. wrote: […] > > talking about. > > I’m not exactly sure what part of it is relevant, so instead of posting > pages of log I provided steps to reproduce a configuration where nginx shows > an erratic behavior, by not outputing the entire response while doing a > proxy\_pass. So the issue you are seeing is truncated response, right? I’m not able to reproduce it here (though I’ve tested under FreeBSD, not Linux). Do you see sendfile and/or directio have any influence? And the point still remains: please provide debug log. Obviously posting it here is a bad idea as it’s likely to be huge, though making gzipped version available for download would be really helpfull. > I’ll be experimenting more, and I’ll let you know if I find the origin of > this issue. Please note that testing with your module may be problematic, as there are at least the following issues with it: 1. “mass\_chunk\_multi\_packets on” expected to cause truncated responses as it aborts response on != NGX\_OK from next body filter (e.g. on NGX\_AGAIN if network isn’t as fast as reading from disk for some reason). 2. checksums will not match as buffers from previous filters are marked as free before actual data are sent, thus causing mostly arbitrary data to be sent (again, if network isn’t as fast as reading from disk for some reason). Maxim D.