Nginx large POST body under Solaris

Server NGINX
1

Hello Nginx-ers,

I had a problem with uploading large files to a FastCGI (Django)
process behind nginx. Nginx crashes when running under Solaris 10
(SunOS 5.11 snv_67 i86pc i386 i86pc) and nginx 0.5.35 compiled
without any special modules. I also tried nginx 0.6.29 with the same
problem.

The problem is that when I upload a small file, it works fine. But
when uploading a large file (> 1Mb) I got 413 request too large and
so I upped the client_max_body_size parameter in the config. When I
upload, I get a blank page back, with a message like this:

2008/03/25 08:03:15 [warn] 3176#0: *5 a client request body is
buffered to a temporary file /tmp/nginx/0/00/0000000000, client:
75.75.25.21, server: mojipage.com, request: “POST /account/avatar/?
t=1206428905 HTTP/1.1”, host: “mojipage.com”, referrer: “http://
mojipage.com/account/avatar/?t=1206428905”
2008/03/25 08:03:24 [notice] 3175#0: signal 18 (SIGCHLD) received
2008/03/25 08:03:24 [alert] 3175#0: worker process 3176 exited on
signal 11
2008/03/25 08:03:24 [notice] 3175#0: start worker process 3184

Here’s a debug log that I correspond to one of these attempts:

2008/03/25 07:29:21 [debug] 28590#0: *5 connect to unix:/home/moji/
web/floweb/floweb-fcgi.sock, fd:9 #6
2008/03/25 07:29:21 [debug] 28590#0: *5 devpoll add event: fd:9 ev:0001
2008/03/25 07:29:21 [debug] 28590#0: *5 devpoll fd:9 ev:0001 fl:0000
2008/03/25 07:29:21 [debug] 28590#0: *5 connected
2008/03/25 07:29:21 [debug] 28590#0: *5 http upstream connect: 0
2008/03/25 07:29:21 [debug] 28590#0: *5 http upstream send request
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:992
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:354
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:14
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 malloc: 080F0E00:4096
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:27809
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:15
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:27809
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:15
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer in: 080F0D4C
2008/03/25 07:29:21 [debug] 28590#0: *5 writev: 1408
2008/03/25 07:29:21 [notice] 28575#0: signal 18 (SIGCHLD) received
2008/03/25 07:29:21 [alert] 28575#0: worker process 28590 exited on
signal 11

Any pointers would be appreciated.

Thanks and best regards,

=wil

2

On Tue, Mar 25, 2008 at 03:45:46PM -0400, Wil Tan wrote:

so I upped the client_max_body_size parameter in the config. When I
2008/03/25 08:03:24 [notice] 3175#0: start worker process 3184
2008/03/25 07:29:21 [debug] 28590#0: *5 http upstream send request
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:8
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:32768
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer buf fl:0 s:15
2008/03/25 07:29:21 [debug] 28590#0: *5 chain writer in: 080F0D4C
2008/03/25 07:29:21 [debug] 28590#0: *5 writev: 1408
2008/03/25 07:29:21 [notice] 28575#0: signal 18 (SIGCHLD) received
2008/03/25 07:29:21 [alert] 28575#0: worker process 28590 exited on
signal 11

Any pointers would be appreciated.

You should enable core dumps for setuid processes (nginx calls setuid()
to change unprivileged user) and then run

gdb /path/to/nginx /path/to/core

bt

quit

3

Hello Igor,

Thank you for your quick response. I’ve installed gdb and enabled
global setuid core and reproduced the scenario for 0.6.29. Here’s the
backtrace:

[moji@d9b60282 /opt/local/src]$ sudo gdb /opt/local/nginx/sbin/nginx /
var/core/core.nginx.12427
GNU gdb 6.6
Copyright © 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for
details.
This GDB was configured as “i386-pc-solaris2.10”…
Reading symbols from /lib/libsocket.so.1…done.
Loaded symbols for /lib/libsocket.so.1
Reading symbols from /lib/libnsl.so.1…done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/librt.so.1…
warning: Lowest section in /lib/librt.so.1 is .dynamic at 00000074
done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libsendfile.so.1…done.
Loaded symbols for /lib/libsendfile.so.1
Reading symbols from /opt/local/lib/libpcre.so.0…done.
Loaded symbols for /usr/local/lib/libpcre.so.0
Reading symbols from /opt/local/lib/libz.so.1…done.
Loaded symbols for /usr/local/lib/libz.so.1
Reading symbols from /lib/libc.so.1…done.
Loaded symbols for /lib/libc.so.1
Reading symbols from /usr/sfw/lib/libgcc_s.so.1…done.
Loaded symbols for /usr/sfw/lib/libgcc_s.so.1
Reading symbols from /lib/ld.so.1…done.
Loaded symbols for /lib/ld.so.1
Core was generated by `/opt/local/nginx/sbin/nginx’.
Program terminated with signal 11, Segmentation fault.
#0 ngx_writev_chain (c=0x810e830, in=0x80d7d74, limit=2147479551)
at src/os/unix/ngx_writev_chain.c:92
92 iov->iov_len += size;
(gdb) where
#0 ngx_writev_chain (c=0x810e830, in=0x80d7d74, limit=2147479551)
at src/os/unix/ngx_writev_chain.c:92
#1 0x0807753e in ngx_solaris_sendfilev_chain (c=0x810e830,
in=0x80d7cfc, limit=0)
at src/os/unix/ngx_solaris_sendfilev_chain.c:64
#2 0x08062a05 in ngx_chain_writer (data=0x80ee7c8, in=0x0) at src/
core/ngx_output_chain.c:483
#3 0x08062518 in ngx_output_chain (ctx=0x80ee790, in=0xe) at src/
core/ngx_output_chain.c:217
#4 0x0808b859 in ngx_http_upstream_send_request (r=0x80f9798,
u=0x80ee75c)
at src/http/ngx_http_upstream.c:923
#5 0x0808b670 in ngx_http_upstream_connect (r=0x80f9798, u=0x80ee75c)
at src/http/ngx_http_upstream.c:766
#6 0x0808af77 in ngx_http_upstream_init (r=0x80f9798) at src/http/
ngx_http_upstream.c:442
#7 0x0808786a in ngx_http_do_read_client_request_body (r=0x80f9798)
at src/http/ngx_http_request_body.c:373
#8 0x08087495 in ngx_http_read_client_request_body_handler
(r=0x80f9798)
at src/http/ngx_http_request_body.c:246
#9 0x080818f8 in ngx_http_request_handler (ev=0x0) at src/http/
ngx_http_request.c:1566
#10 0x08070beb in ngx_event_process_posted (cycle=0x80d9da0,
posted=0x80d1d94)
at src/event/ngx_event_posted.c:39
#11 0x0806fb2a in ngx_process_events_and_timers (cycle=0x80d9da0) at
src/event/ngx_event.c:272
#12 0x08076b56 in ngx_worker_process_cycle (cycle=0x80d9da0, data=0x0)
at src/os/unix/ngx_process_cycle.c:767
#13 0x08074e11 in ngx_spawn_process (cycle=0x80d9da0, proc=0x8076a96
<ngx_worker_process_cycle>,
data=0x0, name=0x80a85b1 “worker process”, respawn=-3) at src/os/
unix/ngx_process.c:187
#14 0x0807636e in ngx_start_worker_processes (cycle=0x80d9da0, n=2,
type=-3)
at src/os/unix/ngx_process_cycle.c:332
#15 0x08076083 in ngx_master_process_cycle (cycle=0x80d9da0) at src/
os/unix/ngx_process_cycle.c:224
#16 0x0805f34d in main (argc=135037216, argv=0x8047d08) at src/core/
nginx.c:355
(gdb)

Thanks!

=wil

4

On Tue, Mar 25, 2008 at 06:42:05PM -0400, Wil Tan wrote:

GDB is free software, covered by the GNU General Public License, and
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libc.so.1…done.
(gdb) where
#0 ngx_writev_chain (c=0x810e830, in=0x80d7d74, limit=2147479551)
at src/os/unix/ngx_writev_chain.c:92

p iov
p *cl->buf
p prev_send
p send
p size

5

Hi Igor,

Thank you so much for your help. The variables that you asked for are
included at the bottom.

=wil

On Mar 26, 2008, at 2:46 AM, Igor S. wrote:

var/core/core.nginx.12427
Reading symbols from /lib/libsocket.so.1…done.
Loaded symbols for /usr/local/lib/libpcre.so.0
#0 ngx_writev_chain (c=0x810e830, in=0x80d7d74, limit=2147479551)
p size

(gdb) p iov
$1 = (struct iovec *) 0x0
(gdb) p *cl->buf
$2 = {pos = 0x0, last = 0x0, file_pos = 196608, file_last = 229376,
start = 0x0, end = 0x0,
tag = 0x0, file = 0x80db9b8, shadow = 0x0, temporary = 0, memory =
0, mmap = 0, recycled = 0,
in_file = 1, flush = 0, sync = 0, last_buf = 0, last_in_chain = 0,
last_shadow = 0,
temp_file = 0, zerocopy_busy = 0, num = 0}
(gdb) p prev_send
$3 = 1416
(gdb) p send
$4 = 1416
(gdb) p size
$5 = 0

6

That worked perfectly!
Thank you so much for taking the time to diagnose and fixing the issue!

=wil

7

On Wed, Mar 26, 2008 at 04:45:15AM -0400, Wil Tan wrote:

Thank you so much for your help. The variables that you asked for are
included at the bottom.

Try the attached patch.