# Occasional misplaced gzip headers?

**URL:** <https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363>\
**Category:** NGINX\
**Created:** [April 5, 2010, 8:01pm UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363 "2010-04-05T20:01:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Frank\_Farmer](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/f0a364/32.png) [@Frank\_Farmer](https://www.ruby-forum.com/u/Frank_Farmer)\
**Post date:** [April 5, 2010, 8:01pm UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363/1 "2010-04-05T20:01:48Z")

</div>

Hi all,

I’ve been seeing rare malformed responses from nginx with a strange 26  
byte header prepended to them.

$ od -c file.part | head  
0000000 037 213 \b \0 \0 \0 \0 \0 \0 003 002 \0 \0 \0 377 377  
0000020 003 \0 \0 \0 \0 \0 \0 \0 \0 \0 H T T P / 1  
0000040 . 1 2 0 0 O K \r \n S e r v e

The first two bytes look suspiciously like a gzip header.

I saw a problem like this back in April '09 when I first deployed nginx  
as a reverse proxy in front of Apache:

> <https://stackoverflow.com/questions/736499/strange-http-gzip-issue>

I upgraded versions at the time to 0.6.35, and the problem persisted.  
Turns out, I had gzip on in apache, as well as in nginx. Disabling gzip  
in apache seemed to correct the issue. We ran with that configuration,  
with nginx on port 80, and apache behind it, on the same box for a year  
without further issue.

Recently, we moved nginx to a separate box on the same LAN, keeping the  
same version (0.6.35 – I know, it’s old, but it’s been stable).  
Suddenly, we’re seeing very similar symptoms again. It doesn’t happen  
reliably, and it doesn’t happen often. I had one user report 3  
instances over the course of 15 minutes. After receiving a garbled  
response, a second request for the same URL seems to succeed.

We still have one site going through nginx 0.6.35 on the same server as  
apache without issue. Only the “remote” nginx installation is  
malfunctioning.

I’m not seeing any errors in the apache or nginx error logs for these  
requests, and I’m confident that apache is not gzipping in this context,  
only nginx (the nginx access log shows a response size about 25% that of  
apache, suggesting that apache’s bodies are infact uncompressed)

We’ve upgraded to 0.7.65, but I’m not really expecting that to resolve  
the issue – if it does, I’ll certainly report back. Our sysop crawled  
the nginx changelogs and much of the mailing list archives, and didn’t  
find anything that sounded relevant.

So, has anyone seen anything like this before? Any leads would be  
welcome.

Thanks!  
Frank Farmer

---

<div class="post-metadata">

**Author:** ![Frank\_Farmer](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/f0a364/32.png) [@Frank\_Farmer](https://www.ruby-forum.com/u/Frank_Farmer)\
**Post date:** [April 5, 2010, 8:05pm UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363/2 "2010-04-05T20:05:18Z")

</div>

We’re still seeing this on nginx 0.7.65, although the users who’ve been  
reporting these issues claim they’re less frequent since the upgrade.

Again, here’s the first few bytes of the raw response as received by the  
client, downloaded, and forwarded to us:

$ od -c postupgrade.dat | head  
0000000 037 213 \b \0 \0 \0 \0 \0 \0 003 002 \0 \0 \0 377 377  
0000020 003 \0 \0 \0 \0 \0 \0 \0 \0 \0 H T T P / 1  
0000040 . 1 2 0 0 O K \r \n S e r v e  
0000060 r : n g i n x / 0 . 7 . 6 5 \r  
0000100 \n D a t e : M o n , 0 5 A

---

<div class="post-metadata">

**Author:** ![Frank\_Farmer](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/f0a364/32.png) [@Frank\_Farmer](https://www.ruby-forum.com/u/Frank_Farmer)\
**Post date:** [April 6, 2010, 12:28am UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363/3 "2010-04-06T00:28:41Z")

</div>

Hello!

On Fri, Apr 02, 2010 at 05:18:56PM -0700, Frank Farmer wrote:

> The first two bytes look suspiciously like a gzip header.  
> Yep, looks like empty gzip file indeed. Such spurious bytes  
> before response may theoretically happen if subrequests are used  
> incorrectly (i.e.: do you use any 3rd party/your own modules?).

Could you please show nginx -V output and your config?

> I saw a problem like this back in April '09 when I first deployed  
> nginx as a reverse proxy in front of Apache:
> 
> [http://stackoverflow.com/questions/736499/strange-http-gzip-issue](http://stackoverflow.com/questions/736499/strange-http-gzip-issue)

This is completely different problem and it’s in Apache, not  
nginx.

[…]

> Recently, we moved nginx to a separate box on the same LAN, keeping  
> the same version (0.6.35 – I know, it’s old, but it’s been stable).  
> Suddenly, we’re seeing very similar symptoms again. It doesn’t  
> happen reliably, and it doesn’t happen often. I had one user report  
> 3 instances over the course of 15 minutes. After receiving a  
> garbled response, a second request for the same URL seems to  
> succeed.

If you are able to reproduce the problem it whould be helpfull to  
look into problematic request/response debug log as well. See  
here for details:

> **[A debugging log](http://nginx.org/en/docs/debugging_log.html)**

Maxim D.

---

<div class="post-metadata">

**Author:** ![Frank\_Farmer](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/f0a364/32.png) [@Frank\_Farmer](https://www.ruby-forum.com/u/Frank_Farmer)\
**Post date:** [April 6, 2010, 12:56am UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363/4 "2010-04-06T00:56:11Z")

</div>

Thanks for the reply. Responses from our ops guy below:

```
Yep, looks like empty gzip file indeed. Such spurious bytes
before response may theoretically happen if subrequests are used
incorrectly (i.e.: do you use any 3rd party/your own modules?).

```

No.

```
Could you please show nginx -V output and your config?

```

nginx version: nginx/0.7.65  
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-46)  
TLS SNI support disabled  
configure arguments: --user=nginx --group=nginx  
–prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error.log  
–http-log-path=/var/log/nginx/access.log  
–http-client-body-temp-path=/var/lib/nginx/tmp/client\_body  
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy  
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi  
–pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx  
–with-http\_ssl\_module --with-http\_realip\_module  
–with-http\_addition\_module --with-http\_sub\_module  
–with-http\_dav\_module --with-http\_flv\_module  
–with-http\_gzip\_static\_module --with-http\_stub\_status\_module  
–with-http\_perl\_module --with-mail --with-mail\_ssl\_module  
–with-cc-opt=‘-O2 -g -m64 -mtune=generic’

nginx.conf:

worker\_processes 1;

events {  
worker\_connections 1024;  
}

http {  
include /etc/nginx/conf.d/\*.conf;  
include mime.types;  
include ipban.conf;  
include /etc/nginx/proxy.conf;

default\_type application/octet-stream;

log\_format main '$host $remote\_addr - $remote\_user [$time\_local]  
“$request” ’  
'$status $body\_bytes\_sent “$http\_referer” ’  
‘“$http\_user\_agent” “$http\_x\_forwarded\_for” “$sent\_http\_content\_type”’;

access\_log /var/log/nginx/access.log main;

log\_format statusonly ‘$status’;  
access\_log /var/log/nginxpub/status.log statusonly;

sendfile on;  
tcp\_nopush on;

keepalive\_timeout 65;

gzip on;  
gzip\_proxied any;  
gzip\_types text/plain text/html text/css application/x-javascript  
text/xml application/xml application/xml+rss text/javascript;

add\_header P3P ‘policyref=“[http://www.somesite.com/w3c/p3p.xml](http://www.somesite.com/w3c/p3p.xml)”, CP=“NOI  
DSP COR CURa ADMa DEVa TAIa OUR BUS IND UNI COM NAV INT”’;  
}

proxy.conf:

proxy\_redirect off;  
proxy\_set\_header Host $host;  
proxy\_set\_header X-Real-IP $remote\_addr;  
proxy\_set\_header X-Forwarded-For $proxy\_add\_x\_forwarded\_for;  
client\_max\_body\_size 200m;  
client\_body\_buffer\_size 128k;  
proxy\_connect\_timeout 90;  
proxy\_send\_timeout 90;  
proxy\_read\_timeout 90;  
proxy\_buffers 4 64k;  
proxy\_buffer\_size 64k;  
large\_client\_header\_buffers 32 4k;

conf.d/example.conf

server {  
listen 80;  
server\_name [example.com](http://example.com) [http://example.com](http://example.com);

include /etc/nginx/extconf/example/nginxipban.conf;

location / {  
proxy\_pass [http://varnish:6081](http://varnish:6081);  
}  
}

```
If you are able to reproduce the problem it whould be helpfull to
look into problematic request/response debug log as well. See
here for details:

```

We’ll enable this log for one of the smaller sites we’ve been seeing  
this on, and see if anything interesting comes up.

Thanks!  
Frank

---

<div class="post-metadata">

**Author:** ![Frank\_Farmer](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/f0a364/32.png) [@Frank\_Farmer](https://www.ruby-forum.com/u/Frank_Farmer)\
**Post date:** [April 6, 2010, 2:12am UTC](https://www.ruby-forum.com/t/occasional-misplaced-gzip-headers/186363/5 "2010-04-06T02:12:53Z")

</div>

Hello!

On Mon, Apr 05, 2010 at 03:54:40PM -0700, Frank Farmer wrote:

> Thanks for the reply. Responses from our ops guy below:
> 
> Yep, looks like empty gzip file indeed. Such spurious bytes  
> before response may theoretically happen if subrequests are used  
> incorrectly (i.e.: do you use any 3rd party/your own modules?).
> 
> No.

[…]

> keepalive\_timeout 65;

[…]

> If you are able to reproduce the problem it whould be helpfull to  
> look into problematic request/response debug log as well. See  
> here for details:
> 
> We’ll enable this log for one of the smaller sites we’ve been seeing  
> this on, and see if anything interesting comes up.

Another possible reason include keepalive connection and backend  
sending gzip with wrong Content-Length (i.e. 0 which corresponds  
to original message length before gzipping instead of 26 after).

As nginx use HTTP/1.0 for backend connections and doesn’t normally  
check length of upstream response it won’t notice. But client  
connection will be screwed up and next response will be incorrect  
from client’s point of view.

You may try to disable keepalive to check if it helps.

Anyway it whould be great to see connection debug log with problem  
to say for sure.

Maxim D.