We’ve been experiencing a weird issue with nginx used as a gzipping
proxy.
One of our internal services doesn’t support content-encoding and
because it
usually streams huge amounts of json data in response, we’ve decided to
put
it behind nginx. Unfortunately, about 5-10% of responses in this
configuration seem to just stall with the TCP connection perfectly fine,
but
no data going anywhere.
I’ve tried to diagnose the issue to check if it’s a problem with our
HTTP
libraries, VPN or network, but it does look like nginx is the culprit:
- The issue is reproducible with any HTTP client, including curl
–compress. - Packet capture on the server that’s hosting a client and the server
that’s hosting nginx show nothing interesting. There are no lost or
corrupted packets; the client receives the last packet sent by nginx,
ACKs
it and then ACKs keepalives. If the client is killed, it and nginx
exchange
FIN/FIN-ACKs just fine. - Packet capture on the loopback interface, however, shows that nginx
just
stops reading data at some point (I see a “full window” notice in
wireshark
and netstat shows an nginx worker sitting with a 512k-2M receive queue
with
an empty transmit queue).
Some technical details (I’ve masked our internal host names):
The server on which nginx is on is running a server build of Ubuntu;
uname
-a gives Linux 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26
21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux. This issue occurs on
nginx
from the Ubuntu repo (1.1.19-1ubuntu0.1) and Debian wheezy repo
(1.2.1-2.2).
nginx -V output:
nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-log-path=/var/log/nginx/access.log
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi
–lock-path=/var/lock/nginx.lock
–pid-path=/var/run/nginx.pid --with-pcre-jit --with-debug
–with-http_addition_module --with-http_dav_module
–with-http_geoip_module
–with-http_gzip_static_module --with-http_image_filter_module
–with-http_realip_module --with-http_stub_status_module
–with-http_ssl_module --with-http_sub_module --with-http_xslt_module
–with-ipv6 --with-sha1=/usr/include/openssl
–with-md5=/usr/include/openssl
–with-mail --with-mail_ssl_module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-auth-pam
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-echo
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upstream-fair
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-dav-ext-module
Basic nginx config is unchanged from Ubuntu-provided defaults; the
server
section looks like this:
server {
gzip on;
gzip_proxied any;
gzip_types application/json;
listen 80 default;
server_name <servername>;
access_log /var/log/nginx/<servername>-access.log;
location / {
proxy_pass http://127.0.0.1:8083;
proxy_buffering off;
}
}
You can see the nginx debug log here: http://pastebin.com/qDis2RS4 (I’ve
masked the request URL but it’s otherwise unchanged); it ends about 1
second
after the response stall.
Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,234275,234275#msg-234275