Nginx compression-at-edge (in front of a reverse proxy) compresses some content, not other?

I’ve an nginx + varnish + apache2 stack. Nginz server as redirector,
ssl handshake, and gzip compression. Varnish serves as a reverse-proxy,
and apache ‘just’ hosts apps & serves content.

I’m trying to get gzip compression behaving properly. I’m getting
intermittent results – some content seems to be gzipped, some not.

I’m looking for some help figuring out WHERE the problem lies, and what
to do to fix it.

In nginx conf, I’ve,

...
  http {
    ...
    gzip              on;
    gzip_http_version 1.0;
    gzip_comp_level   9;
    gzip_proxied      any;
    gzip_buffers      16 8k;
    gzip_min_length   0;
    gzip_types text/plain text/css text/xml text/javascript 
application/x-javascript;
    gzip_disable "MSIE [1-6].(?!.*SV1)";
    gzip_vary         on;

    upstream varnish {
      server 127.0.0.1:8090 weight=10 max_fails=3 fail_timeout=15s;
    }

    server {
      listen                    x.y.z.w:443;
      ...
      location / {
        proxy_pass             http://varnish;
        proxy_redirect         off;
        proxy_set_header       Host $host;
        proxy_set_header       X-Real-IP $remote_addr;
        proxy_set_header       X-Forwarded-For 
$proxy_add_x_forwarded_for;
        proxy_set_header       X-Client-Verify SUCCESS;
        proxy_set_header       X-SSL-Subject $ssl_client_s_dn;
        proxy_set_header       X-SSL-Issuer  $ssl_client_i_dn;
      }
    }
    ...

In varnish config, I’ve a

  ...
  if (req.http.Accept-Encoding) {
    remove req.http.Accept-Encoding;
  }
  ...

clause, since the compression is to be done only at the nginx ‘edge’.

Apache’s compression (gzip, default, or otherwise), is completely
disabled.

Atm, a check of my test site with YSlow complains that .js’s are NOT
being compressed. Checking with LiveHTTPHeaders Firefox plugin, shows,
e.g., the .js in question NOT being gzipped, but a .gif is, e.g.

...
  ----------------------------------------------------------

  https://my.site.com/main/apostrophePlugin/js/jquery.keycodes-0.2.js



  GET /main/apostrophePlugin/js/jquery.keycodes-0.2.js HTTP/1.1
  Host: my.site.com
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) 
Gecko/20100417
  Accept: */*
  Accept-Language: en-us,en;q=0.5
  Accept-Encoding: gzip,deflate
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  Keep-Alive: 115
  Connection: keep-alive
  Referer: https://my.site.com/main/
  Cookie: 
SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; 
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1
  If-Modified-Since: Tue, 27 Apr 2010 18:18:18 GMT
  If-None-Match: "362e4-1008-4853810064180"
  Authorization: Digest username="admin", realm="AUTH my.site.com", 
nonce="5GopBDmFBAA=99f7be8796e018dde459a07178393d235366ecd9", 
uri="/main/apostrophePlugin/js/jquery.keycodes-0.2.js", algorithm=MD5, 
response="b04cb995cd1f86a67197aab3b5a5dbc9", qop=auth, nc=000001c9, 
cnonce="9dbeaefee4d57b12"
  Cache-Control: max-age=0



  HTTP/1.1 304 Not Modified
  Server: nginx/0.8.35
  Date: Wed, 28 Apr 2010 00:39:48 GMT
  Connection: keep-alive
  Etag: "362e4-1008-4853beaefee80"
  Expires: Sat, 01 May 2010 00:39:48 GMT
  Cache-Control: max-age=259200
  Content-Length: 0
  X-Varnish: 940462008
  Age: 0
  Via: 1.1 varnish

  ----------------------------------------------------------

  https://my.site.com/apostrophePlugin/images/a-special-blank.gif

  GET /apostrophePlugin/images/a-special-blank.gif HTTP/1.1
  Host: my.site.com
  User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) 
Gecko/20100417
  Accept: image/png,image/*;q=0.8,*/*;q=0.5
  Accept-Language: en-us,en;q=0.5
  Accept-Encoding: gzip,deflate
  Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  Keep-Alive: 115
  Connection: keep-alive
  Referer: https://my.site.com/main/
  Cookie: 
SESS6fa8cdc2d7064704bbda0c83e2c2588c=94889db68945e19ed6f666b7e00cdd36; 
symfony=3KOH8Qk0hV%2C%2CvXLi0PK5YmdenP1

  HTTP/1.1 404 Not Found
  Server: nginx/0.8.35
  Date: Wed, 28 Apr 2010 00:39:48 GMT
  Content-Type: text/html; charset=iso-8859-1
  Transfer-Encoding: chunked
  Connection: keep-alive
  Vary: Accept-Encoding, accept-language,accept-charset
  Content-Language: en
  X-Varnish: 940462009 940461993
  Age: 87
  Via: 1.1 varnish
  Content-Encoding: gzip
  ----------------------------------------------------------
  ...

where, iiuc, the presence/absence of “Content-Encoding: gzip” defines
whether or not the item in question was succesfully gzipped.

Any ideas why/where the .js is not getting gzipped?

Thanks,

Ben

Posted at Nginx Forum:

On Tue, Apr 27, 2010 at 09:12:13PM -0400, BenDJ wrote:

http {

gzip on;
gzip_http_version 1.0;
gzip_comp_level 9;
gzip_proxied any;
gzip_buffers 16 8k;
gzip_min_length 0;
gzip_types text/plain text/css text/xml text/javascript application/x-javascript;
gzip_disable “MSIE [1-6].(?!.*SV1)”;
gzip_vary on;

BTW, in 0.7.12+ you may use
gzip_disable msie6;
instead of
gzip_disable “MSIE [1-6].(?!.*SV1)”;
It’s faster.

Also, “gzip_comp_level 1” is enough. 9th level simply eats more CPU, but
do not produce proportional cmpression ratio. Say, 1 decreases file 2
times
while 9 - only 2.5, but not 5 times.


Igor S.
http://sysoev.ru/en/

On Tue, Apr 27, 2010 at 09:12:13PM -0400, BenDJ wrote:

http {

    proxy_set_header       Host $host;

clause, since the compression is to be done only at the nginx ‘edge’.

You may set on it on nginx side:
proxy_set_header Accept-Encoding “”;

Connection: keep-alive
Server: nginx/0.8.35

Keep-Alive: 115
Vary: Accept-Encoding, accept-language,accept-charset
where, iiuc, the presence/absence of “Content-Encoding: gzip” defines whether or not the item in question was succesfully gzipped.

Any ideas why/where the .js is not getting gzipped?

In your examples .js response has 304 code without body, so here is
nothing to gzip, and .gif response has 404 code and it is not image,
but text/html, so it is gzipped.

BTW, why do you use varnish instead of nginx built-in proxy cache ?


Igor S.
http://sysoev.ru/en/

Hi Ben,

Your Apache/Varnish server returns:
Content-Type: text/x-js
for javascript files while nginx is configured to compress only those
types:
gzip_types text/plain text/css text/xml text/javascript
application/x-javascript;

My guess is that you should configure apache or varnish to return
Content-Type: text/javascript
instead.

Hope this helps,
GFK’s

Hi Guillaume,

I think we crossed messages in the mail …

On Wed, Apr 28, 2010 at 8:45 AM, Guillaume F. [email protected]
wrote:

Your Apache/Varnish server returns:
Content-Type: text/x-js
for javascript files while nginx is configured to compress only those types:
gzip_types text/plain text/css text/xml text/javascript
application/x-javascript;

My guess is that you should configure apache or varnish to return
Content-Type: text/javascript
instead.

Yes, I’d discovered that. For now I added the type to nginx’s
gzip-types, and it seems to work.

I’ll keep that there until I discover WHY my distro’s apache2 ships
with that type definition in its mimetypes. I suspect some compat
with deprecated browser support, but it’s just as likely a mistake …
just not clear.

In any case – thanks!

Ben

Hi Igor,

On Tue, Apr 27, 2010 at 11:25 PM, Igor S. [email protected] wrote:

Any ideas why/where the .js is not getting gzipped?

You may set on it on nginx side:
proxy_set_header Accept-Encoding “”;

Done. Thanks.

In your examples .js response has 304 code without body, so here is
nothing to gzip, and .gif response has 404 code and it is not image,
but text/html, so it is gzipped.

Yup. Those 304s were a result of local caching in the client.
Cleared that, and retried – getting 200s, as expected, BUT, still no
js gzip.

Turns out that there was another issue, caught @ varnish list …
namely, that the js is being served with Content-Type: text/x-js –
which was NOT added to the nginx gzip-types.

The cause is,

grep js /etc/apache2/mime.types
application/javascript js
text/x-js js

defined in the standard distro-release of apache2. not mentioned at
all in the rfc (http://www.ietf.org/rfc/rfc4329.txt), but widely used,
apparently, nonetheless. so, added it to gzip types, and tried again.

now, headers show BOTH js & css being compressed, as intended,


https://my.site.com/apostrophePlugin/js/jquery.autogrow.js

GET /apostrophePlugin/js/jquery.autogrow.js HTTP/1.1
Host: my.site.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4)
Gecko/20100417
Accept: /
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: https://my.site.com/
Cookie: symfony=R%2Cq-BqTdX-ckfGCqHa2MnDsbJpd

HTTP/1.1 200 OK
Server: nginx/0.8.35
Date: Wed, 28 Apr 2010 15:09:34 GMT
Content-Type: text/x-js
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Wed, 28 Apr 2010 04:42:23 GMT
Etag: “267f6-db6-48544a2d549c0”
Cache-Control: max-age=259200
Expires: Sat, 01 May 2010 15:09:34 GMT
X-Varnish: 1289869751 1289869733
Age: 283
Via: 1.1 varnish
Content-Encoding: gzip


https://my.site.com/apostrophePlugin/css/a.css

GET /apostrophePlugin/css/a.css HTTP/1.1
Host: my.site.com
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4)
Gecko/20100417
Accept: text/css,/;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: https://my.site.com/
Cookie: symfony=R%2Cq-BqTdX-ckfGCqHa2MnDsbJpd

HTTP/1.1 200 OK
Server: nginx/0.8.35
Date: Wed, 28 Apr 2010 15:09:34 GMT
Content-Type: text/css
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Wed, 28 Apr 2010 04:42:21 GMT
Etag: “26233-14e26-48544a2b6c540”
Cache-Control: max-age=259200
Expires: Sat, 01 May 2010 15:09:34 GMT
X-Varnish: 1289869748
Age: 0
Via: 1.1 varnish
Content-Encoding: gzip


AND, the YSlow plugin now ‘grades’ the page an “A”.

BTW, why do you use varnish instead of nginx built-in proxy cache ?

I’d originally had Pound in front of varnish; Nginx was a logical
choice to replace it, as a first step.

As for using it as a proxy cache … Varnish provides all in-memory
caching. I simply don’t know if nginx.

Also, I use Varnish to provide a content-based parallel CDN. Attempts
to do it with nginx failed, and, it appeared others were having issue
as well (Nginx with Parallel CDN? | Drupal Groups). The nginx + varnish
solution works well.

BTW, in 0.7.12+ you may use
gzip_disable msie6;
instead of
gzip_disable “MSIE [1-6].(?!.*SV1)”;
It’s faster.

Missed that one completely. Thanks.

Also, “gzip_comp_level 1” is enough. 9th level simply eats more CPU, but
do not produce proportional cmpression ratio. Say, 1 decreases file 2 times
while 9 - only 2.5, but not 5 times.

I didn’t realize that the difference was that small …

Thanks for the help!

Ben

On Wed, Apr 28, 2010 at 08:33:03AM -0700, Ben DJ wrote:

BTW, why do you use varnish instead of nginx built-in proxy cache ?

I’d originally had Pound in front of varnish; Nginx was a logical
choice to replace it, as a first step.

As for using it as a proxy cache … Varnish provides all in-memory
caching. I simply don’t know if nginx.

nginx uses file cache as persistent storage, however, once a response
has been served from nginx cache, the response is cached in OS VM cache,
and then nginx does not read from disk. If your hot content fits in
a host memory it will be served from OS VM cache without disk access as
in-memory cache. nginx usually uses sendfile() to send the cached
responses,
so only one copy of file present in the host memory.

Varnish mmap()s a large file and stores responses in the mmap()ed
memory,
i.e. in this file. However, since it mmap()s using MAP_NOSYNC flags,
so the stored data are not usually written to disk. The data will be
flushed to disk if there is memory shortage. Varnish uses sendfile()
too, so it also has a single copy of each response in the memory.
Varnish does not store cached responses just in memory (without file
mmap),
because it can not use sendfile() in this case and then some responses
will be storage in OS memory in several copies: one in Varnish memory
cache
and several in network buffers.

Also, I use Varnish to provide a content-based parallel CDN. Attempts
to do it with nginx failed, and, it appeared others were having issue
as well (Nginx with Parallel CDN? | Drupal Groups). The nginx + varnish
solution works well.

I did not understand what that parallel CDN does and what problem you
have encountered.

Also, “gzip_comp_level 1” is enough. 9th level simply eats more CPU, but
do not produce proportional cmpression ratio. Say, 1 decreases file 2 times
while 9 - only 2.5, but not 5 times.

I didn’t realize that the difference was that small …

Try “gzip -1 some.js” and “gzip -9 some.js” to see the difference.
Also you may try to gzip a large file to see time difference.


Igor S.
http://sysoev.ru/en/

Hi Igor,

nginx uses file cache as persistent storage, however, once a response
(snip)

Thanks for the explanation. I’ll, eventually, get around to giving
the capability a try in nginx. At the very least, the performance
comparisons will be interesting …

I did not understand what that parallel CDN does and what problem you
have encountered.

A good description is here, Parallel | Drupal.org

Basically, serving various content types in parallel from Varnish
in-memory caches, in particular when memcache(d) is not an option due
to low RAM constraints (on a VPS …).

The performance boost – admittedly under the right concurrent loads
– can be dramatic.

Try “gzip -1 some.js” and “gzip -9 some.js” to see the difference.
Also you may try to gzip a large file to see time difference.

Thanks!

Ben

Igor,

On Wed, Apr 28, 2010 at 12:42 PM, Igor S. [email protected] wrote:

I understand the idea, but do not see any issues with nginx.
Probably you need something like this:

I’ll give this, specifically, a try then. The issues others were
having may be (a) Drupal specific (I’m migrating off …), &/or (b)
old.

Thanks!

Ben

On Wed, Apr 28, 2010 at 12:29:29PM -0700, Ben DJ wrote:

– can be dramatic.
I understand the idea, but do not see any issues with nginx.
Probably you need something like this:

server {
server_name example.com
cdn1.example.com
cdn2.example.com
cdn3.example.com
;

   location / {
       proxy_pass             http://backend;
       ...
   }

   location ~ \.(js|css|gif|jpe?g|png)$ {
       proxy_pass             http://backend;
       ...
       proxy_cache            CACHE;
   }

The files http://cdn1.example.com/some.js and
http://cdn2.example.com/some.js
will be stored in nginx cache as single file with key
http://backend/some.js.


Igor S.
http://sysoev.ru/en/