Occasional misplaced gzip headers?

Server NGINX
1

Hi all,

I’ve been seeing rare malformed responses from nginx with a strange 26
byte header prepended to them.

$ od -c file.part | head
0000000 037 213 \b \0 \0 \0 \0 \0 \0 003 002 \0 \0 \0 377 377
0000020 003 \0 \0 \0 \0 \0 \0 \0 \0 \0 H T T P / 1
0000040 . 1 2 0 0 O K \r \n S e r v e

The first two bytes look suspiciously like a gzip header.

I saw a problem like this back in April '09 when I first deployed nginx
as a reverse proxy in front of Apache:

I upgraded versions at the time to 0.6.35, and the problem persisted.
Turns out, I had gzip on in apache, as well as in nginx. Disabling gzip
in apache seemed to correct the issue. We ran with that configuration,
with nginx on port 80, and apache behind it, on the same box for a year
without further issue.

Recently, we moved nginx to a separate box on the same LAN, keeping the
same version (0.6.35 – I know, it’s old, but it’s been stable).
Suddenly, we’re seeing very similar symptoms again. It doesn’t happen
reliably, and it doesn’t happen often. I had one user report 3
instances over the course of 15 minutes. After receiving a garbled
response, a second request for the same URL seems to succeed.

We still have one site going through nginx 0.6.35 on the same server as
apache without issue. Only the “remote” nginx installation is
malfunctioning.

I’m not seeing any errors in the apache or nginx error logs for these
requests, and I’m confident that apache is not gzipping in this context,
only nginx (the nginx access log shows a response size about 25% that of
apache, suggesting that apache’s bodies are infact uncompressed)

We’ve upgraded to 0.7.65, but I’m not really expecting that to resolve
the issue – if it does, I’ll certainly report back. Our sysop crawled
the nginx changelogs and much of the mailing list archives, and didn’t
find anything that sounded relevant.

So, has anyone seen anything like this before? Any leads would be
welcome.

Thanks!
Frank Farmer

2

We’re still seeing this on nginx 0.7.65, although the users who’ve been
reporting these issues claim they’re less frequent since the upgrade.

Again, here’s the first few bytes of the raw response as received by the
client, downloaded, and forwarded to us:

$ od -c postupgrade.dat | head
0000000 037 213 \b \0 \0 \0 \0 \0 \0 003 002 \0 \0 \0 377 377
0000020 003 \0 \0 \0 \0 \0 \0 \0 \0 \0 H T T P / 1
0000040 . 1 2 0 0 O K \r \n S e r v e
0000060 r : n g i n x / 0 . 7 . 6 5 \r
0000100 \n D a t e : M o n , 0 5 A

3

Hello!

On Fri, Apr 02, 2010 at 05:18:56PM -0700, Frank Farmer wrote:

The first two bytes look suspiciously like a gzip header.
Yep, looks like empty gzip file indeed. Such spurious bytes
before response may theoretically happen if subrequests are used
incorrectly (i.e.: do you use any 3rd party/your own modules?).

Could you please show nginx -V output and your config?

I saw a problem like this back in April '09 when I first deployed
nginx as a reverse proxy in front of Apache:

php - Strange http gzip issue - Stack Overflow

This is completely different problem and it’s in Apache, not
nginx.

[…]

Recently, we moved nginx to a separate box on the same LAN, keeping
the same version (0.6.35 – I know, it’s old, but it’s been stable).
Suddenly, we’re seeing very similar symptoms again. It doesn’t
happen reliably, and it doesn’t happen often. I had one user report
3 instances over the course of 15 minutes. After receiving a
garbled response, a second request for the same URL seems to
succeed.

If you are able to reproduce the problem it whould be helpfull to
look into problematic request/response debug log as well. See
here for details:

http://nginx.org/en/docs/debugging_log.html

Maxim D.

4

Thanks for the reply. Responses from our ops guy below:

Yep, looks like empty gzip file indeed. Such spurious bytes
before response may theoretically happen if subrequests are used
incorrectly (i.e.: do you use any 3rd party/your own modules?).

No.

Could you please show nginx -V output and your config?

nginx version: nginx/0.7.65
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-46)
TLS SNI support disabled
configure arguments: --user=nginx --group=nginx
–prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/tmp/client_body
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
–pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_sub_module
–with-http_dav_module --with-http_flv_module
–with-http_gzip_static_module --with-http_stub_status_module
–with-http_perl_module --with-mail --with-mail_ssl_module
–with-cc-opt=‘-O2 -g -m64 -mtune=generic’

nginx.conf:

worker_processes 1;

events {
worker_connections 1024;
}

http {
include /etc/nginx/conf.d/*.conf;
include mime.types;
include ipban.conf;
include /etc/nginx/proxy.conf;

default_type application/octet-stream;

log_format main '$host $remote_addr - $remote_user [$time_local]
“$request” ’
'$status $body_bytes_sent “$http_referer” ’
‘“$http_user_agent” “$http_x_forwarded_for” “$sent_http_content_type”’;

access_log /var/log/nginx/access.log main;

log_format statusonly ‘$status’;
access_log /var/log/nginxpub/status.log statusonly;

sendfile on;
tcp_nopush on;

keepalive_timeout 65;

gzip on;
gzip_proxied any;
gzip_types text/plain text/html text/css application/x-javascript
text/xml application/xml application/xml+rss text/javascript;

add_header P3P ‘policyref=“http://www.somesite.com/w3c/p3p.xml”, CP=“NOI
DSP COR CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT”’;
}

proxy.conf:

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 200m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 4 64k;
proxy_buffer_size 64k;
large_client_header_buffers 32 4k;

conf.d/example.conf

server {
listen 80;
server_name example.com http://example.com;

include /etc/nginx/extconf/example/nginxipban.conf;

location / {
proxy_pass http://varnish:6081;
}
}

If you are able to reproduce the problem it whould be helpfull to
look into problematic request/response debug log as well. See
here for details:

We’ll enable this log for one of the smaller sites we’ve been seeing
this on, and see if anything interesting comes up.

Thanks!
Frank

5

Hello!

On Mon, Apr 05, 2010 at 03:54:40PM -0700, Frank Farmer wrote:

Thanks for the reply. Responses from our ops guy below:

Yep, looks like empty gzip file indeed. Such spurious bytes
before response may theoretically happen if subrequests are used
incorrectly (i.e.: do you use any 3rd party/your own modules?).

No.

[…]

keepalive_timeout 65;

[…]

If you are able to reproduce the problem it whould be helpfull to
look into problematic request/response debug log as well. See
here for details:

We’ll enable this log for one of the smaller sites we’ve been seeing
this on, and see if anything interesting comes up.

Another possible reason include keepalive connection and backend
sending gzip with wrong Content-Length (i.e. 0 which corresponds
to original message length before gzipping instead of 26 after).

As nginx use HTTP/1.0 for backend connections and doesn’t normally
check length of upstream response it won’t notice. But client
connection will be screwed up and next response will be incorrect
from client’s point of view.

You may try to disable keepalive to check if it helps.

Anyway it whould be great to see connection debug log with problem
to say for sure.

Maxim D.