Response stalls with gzip and proxy

[aris]

We’ve been experiencing a weird issue with nginx used as a gzipping
proxy.
One of our internal services doesn’t support content-encoding and
because it
usually streams huge amounts of json data in response, we’ve decided to
put
it behind nginx. Unfortunately, about 5-10% of responses in this
configuration seem to just stall with the TCP connection perfectly fine,
but
no data going anywhere.

I’ve tried to diagnose the issue to check if it’s a problem with our
HTTP
libraries, VPN or network, but it does look like nginx is the culprit:

• The issue is reproducible with any HTTP client, including curl
  –compress.
• Packet capture on the server that’s hosting a client and the server
  that’s hosting nginx show nothing interesting. There are no lost or
  corrupted packets; the client receives the last packet sent by nginx,
  ACKs
  it and then ACKs keepalives. If the client is killed, it and nginx
  exchange
  FIN/FIN-ACKs just fine.
• Packet capture on the loopback interface, however, shows that nginx
  just
  stops reading data at some point (I see a “full window” notice in
  wireshark
  and netstat shows an nginx worker sitting with a 512k-2M receive queue
  with
  an empty transmit queue).

Some technical details (I’ve masked our internal host names):

The server on which nginx is on is running a server build of Ubuntu;
uname
-a gives Linux 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26
21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux. This issue occurs on
nginx
from the Ubuntu repo (1.1.19-1ubuntu0.1) and Debian wheezy repo
(1.2.1-2.2).

nginx -V output:
nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-log-path=/var/log/nginx/access.log
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi
–lock-path=/var/lock/nginx.lock
–pid-path=/var/run/nginx.pid --with-pcre-jit --with-debug
–with-http_addition_module --with-http_dav_module
–with-http_geoip_module
–with-http_gzip_static_module --with-http_image_filter_module
–with-http_realip_module --with-http_stub_status_module
–with-http_ssl_module --with-http_sub_module --with-http_xslt_module
–with-ipv6 --with-sha1=/usr/include/openssl
–with-md5=/usr/include/openssl
–with-mail --with-mail_ssl_module
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-auth-pam
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-echo
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upstream-fair
–add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-dav-ext-module

Basic nginx config is unchanged from Ubuntu-provided defaults; the
server
section looks like this:

server {
    gzip  on;
    gzip_proxied any;
    gzip_types application/json;

    listen 80 default;
    server_name <servername>;

    access_log  /var/log/nginx/<servername>-access.log;

    location / {
        proxy_pass http://127.0.0.1:8083;
        proxy_buffering off;
    }
}

You can see the nginx debug log here: nginx response stall - Pastebin.com (I’ve
masked the request URL but it’s otherwise unchanged); it ends about 1
second
after the response stall.

Posted at Nginx Forum:

Response stalls with gzip and proxy