# Can proxy\_cache gzip cached content?

**URL:** <https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798>\
**Category:** NGINX\
**Created:** [February 14, 2012, 1:12pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798 "2012-02-14T13:12:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [February 14, 2012, 1:12pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/1 "2012-02-14T13:12:18Z")

</div>

Hello,

I’m looking at using Nginx as a reverse proxy to cache a few millions  
HTML  
pages coming from a backend server.

The cached content will very seldom (if at all) change so both  
proxy\_cache  
and proxy\_store could do, but all page URLs have a “/foo/$ID” pattern  
and  
IIUC with proxy\_store that would cause millions of files in the same  
directory, which the filesystem might not be ecstatic about. So for now  
I’m  
going with proxy\_cache and two levels of directories. All is going great  
in  
my preliminary tests.

Now, rather than caching uncompressed files and gzipping them before  
serving them most of the time, it would be great if cached content could  
be  
gzipped once (on disk) and served as such most of the time. This would  
decrease both disk space requirements (by 7-8 times) and processor load.

Is this doable? Patching/recompiling nginx as well as using Lua are fine  
with me. Serving gzipped content from the backend would in theory be  
possible though for other reasons better avoided.

Thanks for any insight!  
Massimiliano

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 14, 2012, 2:50pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/2 "2012-02-14T14:50:04Z")

</div>

Just make sure the “Accept-Encoding: gzip” is being passed to your  
back-end, and let the back end do the compression. We actually normalize  
the Accept-Encoding header as well with an if statement. Also use the  
value of the Accept-Encoding header in your proxy\_cache\_key. This allows  
non-cached responses for those clients that don’t support gzip (usually  
coming through an old, weird proxy). So you will get both compressed and  
uncompressed versions in your cache, but with our clients it’s like 99%  
compressed versions at any one time.

Example:

server {

#your server stuff here

#normalize all accept-encoding headers to just gzip  
set $myae “”;  
if ($http\_accept\_encoding ~\* gzip) {  
set $myae “gzip”;  
}

location / {  
proxy\_pass [http://backend](http://backend);  
#the following allows comressed responses from backend  
proxy\_set\_header Accept-Encoding $myae;

proxy\_cache zone1;  
proxy\_cache\_key “$request\_uri $myae”;  
proxy\_cache\_valid 5m;  
proxy\_cache\_use\_stale error updating;

}

}

Posted at Nginx Forum:

> **[Re: Can proxy\_cache gzip cached content?](https://forum.nginx.org/read.php?2,222382,222391#msg-222391)**
>
> Hello, I'm looking at using Nginx as a reverse proxy to cache a few millions HTML pages coming from a backend server. The cached content will very seldom (if at all) change so both proxy\_cache and proxy\_store could do, but all page URLs have a...

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 15, 2012, 3:56pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/3 "2012-02-15T15:56:01Z")

</div>

## bard Wrote:

> Thanks for the pointers. As I wrote, I’d rather  
> avoid gzipping in the  
> backend, but if that’s the only option so be it.

There’s no reason the “backend” for your caching layer cannot be another  
nginx server block running on a high port bound to localhost. This  
high-port server block could do gzip compression, and proxy-pass to the  
back end with “Accept-Encoding: identity”, so the back-end never has to  
do compression. The backend server will have to use “gzip\_http\_version  
1.0” and “gzip\_proxied any” to do compression because it is being  
proxied from the front-end.

There might be a moderate performance impact, but because you’re caching  
at the “frontmost” layer, the number of back-end hits should be small.

Also note there may be better options in the latest nginx versions, or  
by using the gunzip 3rd-party module:

> **[ngx\_http\_gunzip\_filter\_module: 27f057249155 README](http://mdounin.ru/hg/ngx_http_gunzip_filter_module/file/27f057249155/README)**

With the gunzip module, you can configure things so that you always  
cache compressed data, then only decompress it for the small number of  
clients that don’t support gzip compression.

–  
RPM

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,222382,222436#msg-222436](http://forum.nginx.org/read.php?2,222382,222436#msg-222436)

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 14, 2012, 9:03pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/4 "2012-02-14T21:03:19Z")

</div>

Thanks for the pointers. As I wrote, I’d rather avoid gzipping in the  
backend, but if that’s the only option so be it.

I was also concerned about caching gzipped content using the value of  
Accept-Encoding in the cache key and ending with many duplicates because  
of  
slightly different yet equivalent headers, but your suggestion to  
normalize  
it solves it nicely.

Cheers,  
Massimiliano

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 18, 2012, 9:43pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/5 "2012-02-18T21:43:58Z")

</div>

About this:

> ```
> location /proxied-stuff {
> proxy_set_header Accept-Encoding gzip;
> proxy_cache_key "$scheme$host$request_uri";
> proxy_cache_valid 2d;
> proxy_cache myapp_cache;
> proxy_pass http://127.0.0.1:85;
> }
> 
> ```

I was hoping that gunzip’ping for clients that don’t support compression  
would be as simple as adding the following inside the above block:

```
    if ($http_accept_encoding !~* gzip) {
            gunzip on;
    }

```

But when nginx configuration is reloaded, I get: “nginx: [emerg]  
“gunzip”  
directive is not allowed here”.

I suppose I could rewrite the request to an internal location, then  
within  
that location’s block re-set the proxy\_cache\_key accordingly. But  
perhaps  
there’s an easier way?

Massimiliano

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 17, 2012, 2:04pm UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/6 "2012-02-17T14:04:28Z")

</div>

On Wed, Feb 15, 2012 at 3:55 PM, rmalayter [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> There’s no reason the “backend” for your caching layer cannot be another  
> nginx server block running on a high port bound to localhost. This  
> high-port server block could do gzip compression, and proxy-pass to the  
> back end with “Accept-Encoding: identity”, so the back-end never has to  
> do compression. The backend server will have to use “gzip\_http\_version  
> 1.0” and “gzip\_proxied any” to do compression because it is being  
> proxied from the front-end.

Ah, good point. I tried to take this an extra step further by using a  
virtual host of the same server as “compression backend” and it appears  
to  
work nicely. Below is what I did so far, in case anyone is looking for  
the  
same and Google leads them here.

(Feels a bit like getting out of the door and back in through the window  
🙂  
but perhaps just like we have internal redirects it would be possible to  
use ngx\_lua to simulate an internal proxy and avoid the extra HTTP  
request.)

```
proxy_cache_path /var/lib/nginx/cache/myapp
    levels=1:2
    keys_zone=myapp_cache:10m
    max_size=1g
    inactive=2d;

log_format cache '***$time_local '
    '$upstream_cache_status '
    'Cache-Control: $upstream_http_cache_control '
    'Expires: $upstream_http_expires '
    '$host '
    '"$request" ($status) '
    '"$http_user_agent" '
    'Args: $args ';

access_log /var/log/nginx/cache.log cache;

upstream backend {
        server localhost:8002;
}

server { # this step only does compression
        listen 85;

        server_name myapp.local;
        include proxy_params;

        location / {
                gzip_http_version 1.0;
                proxy_set_header Accept-Encoding identity;
                proxy_pass http://backend;
        }
}

server {
        listen 80;

        server_name myapp.local;
        include proxy_params;

        location / {
                proxy_pass http://127.0.0.1:85;
        }

        location /similar-to {
                proxy_set_header Accept-Encoding gzip;
                proxy_cache_key "$scheme$host$request_uri";
                proxy_cache_valid 2d;
                proxy_cache myapp_cache;
                proxy_pass http://127.0.0.1:85;
        }
}

```

> Also note there may be better options in the latest nginx versions, or  
> by using the gunzip 3rd-party module:  
> [ngx\_http\_gunzip\_filter\_module: 27f057249155 README](http://mdounin.ru/hg/ngx_http_gunzip_filter_module/file/27f057249155/README)
> 
> With the gunzip module, you can configure things so that you always  
> cache compressed data, then only decompress it for the small number of  
> clients that don’t support gzip compression.

This looks perfect for having a gzip-only cache, which may not lead to  
save  
that much disk space but it certainly helps with mind space.

Cheers,  
Massimiliano

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 19, 2012, 11:03am UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/7 "2012-02-19T11:03:23Z")

</div>

On Sun, Feb 19, 2012 at 12:31 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Yes. The easier way is to just write
> 
> ```
> gunzip on;
> 
> ```
> 
> It will gunzip responses for clients which don’t support gzip (as per  
> Accept-Encoding and gzip\_http\_version/gzip\_proxied/gzip\_disabled,  
> i.e. the same checks as done for gzip and gzip\_static).

Thanks Max, I had come to the conclusion that it was always  
decompressing  
content but now I see I had been “testing” with curl -H  
‘Content-Encoding:  
gzip’. No wonder…

Cheers,  
Massimiliano

---

<div class="post-metadata">

**Author:** ![Massimiliano\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/839c29/32.png) [@Massimiliano\_M](https://www.ruby-forum.com/u/Massimiliano_M)\
**Post date:** [February 19, 2012, 12:31am UTC](https://www.ruby-forum.com/t/can-proxy-cache-gzip-cached-content/216798/8 "2012-02-19T00:31:50Z")

</div>

Hello!

On Sat, Feb 18, 2012 at 09:43:11PM +0100, Massimiliano M. wrote:

> > 
> 
> I suppose I could rewrite the request to an internal location, then within  
> that location’s block re-set the proxy\_cache\_key accordingly. But perhaps  
> there’s an easier way?

Yes. The easier way is to just write

```
 gunzip on;

```

It will gunzip responses for clients which don’t support gzip (as per  
Accept-Encoding and gzip\_http\_version/gzip\_proxied/gzip\_disabled,  
i.e. the same checks as done for gzip and gzip\_static).

Maxim D.