# Nginx compression-at-edge (in front of a reverse proxy) compresses some content, not other?

**URL:** <https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705>\
**Category:** NGINX\
**Created:** [April 28, 2010, 3:13am UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705 "2010-04-28T03:13:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 3:13am UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/1 "2010-04-28T03:13:04Z")

</div>

I’ve an nginx + varnish + apache2 stack. Nginz server as redirector,  
ssl handshake, and gzip compression. Varnish serves as a reverse-proxy,  
and apache ‘just’ hosts apps & serves content.

I’m trying to get gzip compression behaving properly. I’m getting  
intermittent results – some content seems to be gzipped, some not.

I’m looking for some help figuring out WHERE the problem lies, and what  
to do to fix it.

In nginx conf, I’ve,

```auto
...
  http {
    ...
    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 9;
    gzip_proxied any;
    gzip_buffers 16 8k;
    gzip_min_length 0;
    gzip_types text/plain text/css text/xml text/javascript 
application/x-javascript;
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    gzip_vary on;

    upstream varnish {
      server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=15s;
    }

    server {
      listen x.y.z.w:443;
      ...
      location / {
        proxy_pass http://varnish;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For 
$proxy_add_x_forwarded_for;
        proxy_set_header X-Client-Verify SUCCESS;
        proxy_set_header X-SSL-Subject $ssl_client_s_dn;
        proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
      }
    }
    ...
```

In varnish config, I’ve a

```auto
  ...
  if (req.http.Accept-Encoding) {
    remove req.http.Accept-Encoding;
  }
  ...
```

clause, since the compression is to be done only at the nginx ‘edge’.

Apache’s compression (gzip, default, or otherwise), is completely  
disabled.

Atm, a check of my test site with YSlow complains that .js’s are NOT  
being compressed. Checking with LiveHTTPHeaders Firefox plugin, shows,  
e.g., the .js in question NOT being gzipped, but a .gif _is_, e.g.

```auto
...
  ----------------------------------------------------------

  https://my.site.com/main/apostrophePlugin/js/jquery.keycodes-0.2.js

  GET /main/apostrophePlugin/js/jquery.keycodes-0.2.js HTTP/1.1
  Host: my.site.com
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) 
Gecko/20100417
  Accept: */*
  Accept-Language: en-us,en;q=0.5
  Accept-Encoding: gzip,deflate
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  Keep-Alive: 115
  Connection: keep-alive
  Referer: https://my.site.com/main/
  Cookie: 
SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; 
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1
  If-Modified-Since: Tue, 27 Apr 2010 18:18:18 GMT
  If-None-Match: "362e4-1008-4853810064180"
  Authorization: Digest username="admin", realm="AUTH my.site.com", 
nonce="5GopBDmFBAA=99f7be8796e018dde459a07178393d235366ecd9", 
uri="/main/apostrophePlugin/js/jquery.keycodes-0.2.js", algorithm=MD5, 
response="b04cb995cd1f86a67197aab3b5a5dbc9", qop=auth, nc=000001c9, 
cnonce="9dbeaefee4d57b12"
  Cache-Control: max-age=0

  HTTP/1.1 304 Not Modified
  Server: nginx/0.8.35
  Date: Wed, 28 Apr 2010 00:39:48 GMT
  Connection: keep-alive
  Etag: "362e4-1008-4853beaefee80"
  Expires: Sat, 01 May 2010 00:39:48 GMT
  Cache-Control: max-age=259200
  Content-Length: 0
  X-Varnish: 940462008
  Age: 0
  Via: 1.1 varnish

  ----------------------------------------------------------

  https://my.site.com/apostrophePlugin/images/a-special-blank.gif

  GET /apostrophePlugin/images/a-special-blank.gif HTTP/1.1
  Host: my.site.com
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) 
Gecko/20100417
  Accept: image/png,image/*;q=0.8,*/*;q=0.5
  Accept-Language: en-us,en;q=0.5
  Accept-Encoding: gzip,deflate
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  Keep-Alive: 115
  Connection: keep-alive
  Referer: https://my.site.com/main/
  Cookie: 
SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; 
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1

  HTTP/1.1 404 Not Found
  Server: nginx/0.8.35
  Date: Wed, 28 Apr 2010 00:39:48 GMT
  Content-Type: text/html; charset=iso-8859-1
  Transfer-Encoding: chunked
  Connection: keep-alive
  Vary: Accept-Encoding, accept-language,accept-charset
  Content-Language: en
  X-Varnish: 940462009 940461993
  Age: 87
  Via: 1.1 varnish
  Content-Encoding: gzip
  ----------------------------------------------------------
  ...
```

where, iiuc, the presence/absence of “Content-Encoding: gzip” defines  
whether or not the item in question was succesfully gzipped.

Any ideas why/where the .js is not getting gzipped?

Thanks,

Ben

Posted at Nginx Forum:

> **[nginx compression-at-edge (in front of a reverse proxy) compresses some...](https://forum.nginx.org/read.php?2,80148,80148#msg-80148)**
>
> I've an nginx + varnish + apache2 stack. Nginz server as redirector, ssl handshake, and gzip compression. Varnish serves as a reverse-proxy, and apache 'just' hosts apps & serves content. I'm trying to get gzip compression behaving properly. I'm...

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 8:33am UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/2 "2010-04-28T08:33:13Z")

</div>

On Tue, Apr 27, 2010 at 09:12:13PM -0400, BenDJ wrote:

> http {  
> …  
> gzip on;  
> gzip\_http\_version 1.0;  
> gzip\_comp\_level 9;  
> gzip\_proxied any;  
> gzip\_buffers 16 8k;  
> gzip\_min\_length 0;  
> gzip\_types text/plain text/css text/xml text/javascript application/x-javascript;  
> gzip\_disable “MSIE [1-6].(?!.\*SV1)”;  
> gzip\_vary on;

BTW, in 0.7.12+ you may use  
gzip\_disable msie6;  
instead of  
gzip\_disable “MSIE [1-6].(?!.\*SV1)”;  
It’s faster.

Also, “gzip\_comp\_level 1” is enough. 9th level simply eats more CPU, but  
do not produce proportional cmpression ratio. Say, 1 decreases file 2  
times  
while 9 - only 2.5, but not 5 times.

–  
Igor S.  
[http://sysoev.ru/en/](http://sysoev.ru/en/)

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 8:26am UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/3 "2010-04-28T08:26:11Z")

</div>

On Tue, Apr 27, 2010 at 09:12:13PM -0400, BenDJ wrote:

> http {
> 
> ```
> proxy_set_header Host $host;
> 
> ```
> 
> clause, since the compression is to be done only at the nginx ‘edge’.

You may set on it on nginx side:  
proxy\_set\_header Accept-Encoding “”;

> ## Connection: keep-alive Server: nginx/0.8.35
> 
> Keep-Alive: 115  
> Vary: Accept-Encoding, accept-language,accept-charset  
> where, iiuc, the presence/absence of “Content-Encoding: gzip” defines whether or not the item in question was succesfully gzipped.
> 
> Any ideas why/where the .js is not getting gzipped?

In your examples .js response has 304 code without body, so here is  
nothing to gzip, and .gif response has 404 code and it is not image,  
but text/html, so it is gzipped.

BTW, why do you use varnish instead of nginx built-in proxy cache ?

–  
Igor S.  
[http://sysoev.ru/en/](http://sysoev.ru/en/)

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 5:47pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/4 "2010-04-28T17:47:04Z")

</div>

Hi Ben,

Your Apache/Varnish server returns:  
Content-Type: text/x-js  
for javascript files while nginx is configured to compress only those  
types:  
gzip\_types text/plain text/css text/xml text/javascript  
application/x-javascript;

My guess is that you should configure apache or varnish to return  
Content-Type: text/javascript  
instead.

Hope this helps,  
GFK’s

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 6:01pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/5 "2010-04-28T18:01:07Z")

</div>

Hi Guillaume,

I think we crossed messages in the mail …

On Wed, Apr 28, 2010 at 8:45 AM, Guillaume F. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Your Apache/Varnish server returns:  
> Content-Type: text/x-js  
> for javascript files while nginx is configured to compress only those types:  
> gzip\_types text/plain text/css text/xml text/javascript  
> application/x-javascript;
> 
> My guess is that you should configure apache or varnish to return  
> Content-Type: text/javascript  
> instead.

Yes, I’d discovered that. For now I added the type to nginx’s  
gzip-types, and it seems to work.

I’ll keep that there until I discover WHY my distro’s apache2 _ships_  
with that type definition in its mimetypes. I _suspect_ some compat  
with deprecated browser support, but it’s just as likely a mistake …  
just not clear.

In any case – thanks!

Ben

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 5:34pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/6 "2010-04-28T17:34:46Z")

</div>

Hi Igor,

On Tue, Apr 27, 2010 at 11:25 PM, Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > Any ideas why/where the .js is not getting gzipped?

> You may set on it on nginx side:  
> proxy\_set\_header Accept-Encoding “”;

Done. Thanks.

> In your examples .js response has 304 code without body, so here is  
> nothing to gzip, and .gif response has 404 code and it is not image,  
> but text/html, so it is gzipped.

Yup. Those 304s were a result of local caching in the client.  
Cleared that, and retried – getting 200s, as expected, BUT, still no  
js gzip.

Turns out that there was another issue, caught @ varnish list …  
namely, that the js is being served with Content-Type: text/x-js –  
which was NOT added to the nginx gzip-types.

The cause is,

grep js /etc/apache2/mime.types  
application/javascript js  
text/x-js js

defined in the standard distro-release of apache2. not mentioned at  
all in the rfc ([http://www.ietf.org/rfc/rfc4329.txt](http://www.ietf.org/rfc/rfc4329.txt)), but widely used,  
apparently, nonetheless. so, added it to gzip types, and tried again.

now, headers show BOTH js & css being compressed, as intended,

* * *

[https://my.site.com/apostrophePlugin/js/jquery.autogrow.js](https://my.site.com/apostrophePlugin/js/jquery.autogrow.js)

GET /apostrophePlugin/js/jquery.autogrow.js HTTP/1.1  
Host: [my.site.com](http://my.site.com)  
User-Agent: Mozilla/5.0 (X11; U; Linux x86\_64; en-US; rv:1.9.2.4)  
Gecko/20100417  
Accept: _/_  
Accept-Language: en-us,en;q=0.5  
Accept-Encoding: gzip,deflate  
Accept-Charset: ISO-8859-1,utf-8;q=0.7,\*;q=0.7  
Keep-Alive: 115  
Connection: keep-alive  
Referer: [https://my.site.com/](https://my.site.com/)  
Cookie: symfony=R%2Cq-BqTdX-ckfGCqHa2MnDsbJpd

HTTP/1.1 200 OK  
Server: nginx/0.8.35  
Date: Wed, 28 Apr 2010 15:09:34 GMT  
Content-Type: text/x-js  
Transfer-Encoding: chunked  
Connection: keep-alive  
Vary: Accept-Encoding  
Last-Modified: Wed, 28 Apr 2010 04:42:23 GMT  
Etag: “267f6-db6-48544a2d549c0”  
Cache-Control: max-age=259200  
Expires: Sat, 01 May 2010 15:09:34 GMT  
X-Varnish: 1289869751 1289869733  
Age: 283  
Via: 1.1 varnish  
Content-Encoding: gzip

* * *

[https://my.site.com/apostrophePlugin/css/a.css](https://my.site.com/apostrophePlugin/css/a.css)

GET /apostrophePlugin/css/a.css HTTP/1.1  
Host: [my.site.com](http://my.site.com)  
User-Agent: Mozilla/5.0 (X11; U; Linux x86\_64; en-US; rv:1.9.2.4)  
Gecko/20100417  
Accept: text/css,_/_;q=0.1  
Accept-Language: en-us,en;q=0.5  
Accept-Encoding: gzip,deflate  
Accept-Charset: ISO-8859-1,utf-8;q=0.7,\*;q=0.7  
Keep-Alive: 115  
Connection: keep-alive  
Referer: [https://my.site.com/](https://my.site.com/)  
Cookie: symfony=R%2Cq-BqTdX-ckfGCqHa2MnDsbJpd

HTTP/1.1 200 OK  
Server: nginx/0.8.35  
Date: Wed, 28 Apr 2010 15:09:34 GMT  
Content-Type: text/css  
Transfer-Encoding: chunked  
Connection: keep-alive  
Vary: Accept-Encoding  
Last-Modified: Wed, 28 Apr 2010 04:42:21 GMT  
Etag: “26233-14e26-48544a2b6c540”  
Cache-Control: max-age=259200  
Expires: Sat, 01 May 2010 15:09:34 GMT  
X-Varnish: 1289869748  
Age: 0  
Via: 1.1 varnish  
Content-Encoding: gzip

* * *

AND, the YSlow plugin now ‘grades’ the page an “A”.

> BTW, why do you use varnish instead of nginx built-in proxy cache ?

I’d originally had Pound in front of varnish; Nginx was a logical  
choice to replace it, as a first step.

As for using it as a proxy cache … Varnish provides all in-memory  
caching. I simply don’t know if nginx.

Also, I use Varnish to provide a content-based parallel CDN. Attempts  
to do it with nginx failed, and, it appeared others were having issue  
as well ([Nginx with Parallel CDN? | Drupal Groups](http://groups.drupal.org/node/48182)). The nginx + varnish  
solution works well.

> BTW, in 0.7.12+ you may use  
> gzip\_disable msie6;  
> instead of  
> gzip\_disable “MSIE [1-6].(?!.\*SV1)”;  
> It’s faster.

Missed that one completely. Thanks.

> Also, “gzip\_comp\_level 1” is enough. 9th level simply eats more CPU, but  
> do not produce proportional cmpression ratio. Say, 1 decreases file 2 times  
> while 9 - only 2.5, but not 5 times.

I didn’t realize that the difference was _that_ small …

Thanks for the help!

Ben

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 9:10pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/7 "2010-04-28T21:10:50Z")

</div>

On Wed, Apr 28, 2010 at 08:33:03AM -0700, Ben DJ wrote:

> > BTW, why do you use varnish instead of nginx built-in proxy cache ?
> 
> I’d originally had Pound in front of varnish; Nginx was a logical  
> choice to replace it, as a first step.
> 
> As for using it as a proxy cache … Varnish provides all in-memory  
> caching. I simply don’t know if nginx.

nginx uses file cache as persistent storage, however, once a response  
has been served from nginx cache, the response is cached in OS VM cache,  
and then nginx does not read from disk. If your hot content fits in  
a host memory it will be served from OS VM cache without disk access as  
in-memory cache. nginx usually uses sendfile() to send the cached  
responses,  
so only one copy of file present in the host memory.

Varnish mmap()s a large file and stores responses in the mmap()ed  
memory,  
i.e. in this file. However, since it mmap()s using MAP\_NOSYNC flags,  
so the stored data are not usually written to disk. The data will be  
flushed to disk if there is memory shortage. Varnish uses sendfile()  
too, so it also has a single copy of each response in the memory.  
Varnish does not store cached responses just in memory (without file  
mmap),  
because it can not use sendfile() in this case and then some responses  
will be storage in OS memory in several copies: one in Varnish memory  
cache  
and several in network buffers.

> Also, I use Varnish to provide a content-based parallel CDN. Attempts  
> to do it with nginx failed, and, it appeared others were having issue  
> as well ([Nginx with Parallel CDN? | Drupal Groups](http://groups.drupal.org/node/48182)). The nginx + varnish  
> solution works well.

I did not understand what that parallel CDN does and what problem you  
have encountered.

> > Also, “gzip\_comp\_level 1” is enough. 9th level simply eats more CPU, but  
> > do not produce proportional cmpression ratio. Say, 1 decreases file 2 times  
> > while 9 - only 2.5, but not 5 times.
> 
> I didn’t realize that the difference was _that_ small …

Try “gzip -1 some.js” and “gzip -9 some.js” to see the difference.  
Also you may try to gzip a large file to see time difference.

–  
Igor S.  
[http://sysoev.ru/en/](http://sysoev.ru/en/)

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 9:30pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/8 "2010-04-28T21:30:09Z")

</div>

Hi Igor,

> nginx uses file cache as persistent storage, however, once a response  
> (snip)

Thanks for the explanation. I’ll, eventually, get around to giving  
the capability a try in nginx. At the very least, the performance  
comparisons will be interesting …

> I did not understand what that parallel CDN does and what problem you  
> have encountered.

A good description is here, [Parallel | Drupal.org](http://drupal.org/project/parallel)

Basically, serving various content types in parallel from Varnish  
in-memory caches, in particular when memcache(d) is not an option due  
to low RAM constraints (on a VPS …).

The performance boost – admittedly under the right concurrent loads  
– can be dramatic.

> Try “gzip -1 some.js” and “gzip -9 some.js” to see the difference.  
> Also you may try to gzip a large file to see time difference.

Thanks!

Ben

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 9:54pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/9 "2010-04-28T21:54:44Z")

</div>

Igor,

On Wed, Apr 28, 2010 at 12:42 PM, Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I understand the idea, but do not see any issues with nginx.  
> Probably you need something like this:

I’ll give this, specifically, a try then. The issues others were  
having may be (a) Drupal specific (I’m migrating off …), &/or (b)  
old.

Thanks!

Ben

---

<div class="post-metadata">

**Author:** ![BenDJ](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/5fc32e/32.png) [@BenDJ](https://www.ruby-forum.com/u/BenDJ)\
**Post date:** [April 28, 2010, 9:43pm UTC](https://www.ruby-forum.com/t/nginx-compression-at-edge-in-front-of-a-reverse-proxy-compresses-some-content-not-other/187705/10 "2010-04-28T21:43:37Z")

</div>

On Wed, Apr 28, 2010 at 12:29:29PM -0700, Ben DJ wrote:

> – can be dramatic.  
> I understand the idea, but do not see any issues with nginx.  
> Probably you need something like this:

server {  
server\_name [example.com](http://example.com)  
[cdn1.example.com](http://cdn1.example.com)  
[cdn2.example.com](http://cdn2.example.com)  
[cdn3.example.com](http://cdn3.example.com)  
;

```
   location / {
       proxy_pass http://backend;
       ...
   }

   location ~ \.(js|css|gif|jpe?g|png)$ {
       proxy_pass http://backend;
       ...
       proxy_cache CACHE;
   }

```

The files [http://cdn1.example.com/some.js](http://cdn1.example.com/some.js) and  
[http://cdn2.example.com/some.js](http://cdn2.example.com/some.js)  
will be stored in nginx cache as single file with key  
[http://backend/some.js](http://backend/some.js).

–  
Igor S.  
[http://sysoev.ru/en/](http://sysoev.ru/en/)