Cache gzipped content with proxy_pass

Hi.

AFAIK nginx only stores plain (uncompressed) content when acting as a
caching proxy server. With gzip=on, content for gzip-accepting clients
will be compressed by the server for every request (?).

Wouldn’t it be better to have two cache files like this:

/data/nginx/cache/c29b7f54b2df7773722d382f4809d65029c
/data/nginx/cache/c29b7f54b2df7773722d382f4809d65029c.gz

Another solution would be to add the internal variable gzip_ok to the
config variable space. Then one could add this variable to
proxy_cache_key:

proxy_cache_key “…:$gzip_ok:…”

Currently I’ve added http_accept_enconding to the cache key, but since
there are hundreds of different permutations for the value this is not
an optimal solution.

Chris

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,120314,120314#msg-120314

Hello!

On Mon, Aug 16, 2010 at 08:56:47AM -0400, chrisl wrote:

AFAIK nginx only stores plain (uncompressed) content when acting as a
caching proxy server.

Not really. nginx caches what upstream server sent to it. If
response was compressed by upstream server - it will cache
compressed response.

With gzip=on, content for gzip-accepting clients
will be compressed by the server for every request (?).

Yes - unless upstream server already returned precompressed
content.

Wouldn’t it be better to have two cache files like this:

/data/nginx/cache/c29b7f54b2df7773722d382f4809d65029c
/data/nginx/cache/c29b7f54b2df7773722d382f4809d65029c.gz

You may want to follow this thread:

http://nginx.org/pipermail/nginx/2010-March/019471.html

It is believed that separate gzip cache is better idea.

Another solution would be to add the internal variable gzip_ok to the
config variable space. Then one could add this variable to
proxy_cache_key:

proxy_cache_key “…:$gzip_ok:…”

Currently I’ve added http_accept_enconding to the cache key, but since
there are hundreds of different permutations for the value this is not
an optimal solution.

The problem is that nginx itself and upstream server may have
different ideas on what may be compressed. And you need
upstream’s one here, not nginx’s one. Though probably it’s
a good idea to expose nginx’s one anyway.

Maxim D.

Maxim,

thank you for your answer. I’ve patched nginx 0.7.67 to export gzip_ok
to config file variable space (see attachement). I’m not a C guy, but it
seems to work :slight_smile: I’ currently using it like this:

location /json-data/ {
set $ae “”;
if ($gzip_ok) {
set $ae “gzip”;
}
proxy_pass http://10.2.3.4/json-upstream/;
proxy_cache_key “$gzip_ok:$uri”;
proxy_set_header User-Agent “nginx”;
proxy_set_header Accept-Encoding $ae;
[ … ]
}

I get cache keys like this:

grep -a ^KEY /var/spool/nginx/*

/var/spool/nginx/0c341794ed57bf03d72fb9a207f43d0e:KEY:
:/json-data/123/45/
/var/spool/nginx/17961a45b339647f5e8f239dde713428:KEY:
OK:/json-data/123/45/

By hiding the client User-Agent and Accept-Encoding, I deactivate
upstream gzip detection logic. For this specific task this patch/config
works perfectly. I more general approach would be nice, though.

Chris

— src/http/ngx_http_variables.c.orig 2010-08-21 12:17:42.000000000
+0200
+++ src/http/ngx_http_variables.c 2010-08-21 12:19:07.000000000
+0200
@@ -66,6 +66,10 @@
ngx_http_variable_value_t *v, uintptr_t data);
static ngx_int_t
ngx_http_variable_request_completion(ngx_http_request_t *r,
ngx_http_variable_value_t *v, uintptr_t data);
+#if (NGX_HTTP_GZIP)
+static ngx_int_t ngx_http_variable_gzip_ok(ngx_http_request_t *r,

  • ngx_http_variable_value_t *v, uintptr_t data);
    +#endif
    static ngx_int_t ngx_http_variable_request_body(ngx_http_request_t *r,
    ngx_http_variable_value_t *v, uintptr_t data);
    static ngx_int_t ngx_http_variable_request_body_file(ngx_http_request_t
    *r,
    @@ -123,6 +127,9 @@
    #if (NGX_HTTP_GZIP)
    { ngx_string(“http_via”), NULL, ngx_http_variable_header,
    offsetof(ngx_http_request_t, headers_in.via), 0, 0 },
  • { ngx_string(“gzip_ok”), NULL, ngx_http_variable_gzip_ok,
  •  0, 0, 0 },
    

#endif

#if (NGX_HTTP_PROXY || NGX_HTTP_REALIP)
@@ -1538,6 +1545,41 @@
return NGX_OK;
}

+#if (NGX_HTTP_GZIP)
+static ngx_int_t
+ngx_http_variable_gzip_ok(ngx_http_request_t *r,

  • ngx_http_variable_value_t *v, uintptr_t data)
    +{
  • ngx_int_t rc;
  • if (!r->gzip_tested) {
  •    rc = ngx_http_gzip_ok(r);
    
  •    if (rc != NGX_OK && rc != NGX_DECLINED) {
    
  •        v->not_found = 1;
    
  •        return NGX_OK;
    
  •    }
    
  • }
  • if (r->gzip_ok) {
  •    v->len = 2;
    
  •    v->valid = 1;
    
  •    v->no_cacheable = 0;
    
  •    v->not_found = 0;
    
  •    v->data = (u_char *) "OK";
    
  •    return NGX_OK;
    
  • }
  • v->len = 0;
  • v->valid = 1;
  • v->no_cacheable = 0;
  • v->not_found = 0;
  • v->data = (u_char *) “”;
  • return NGX_OK;
    +}
    +#endif

static ngx_int_t
ngx_http_variable_request_body(ngx_http_request_t *r,

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,120314,122064#msg-122064

I’ve been following this discussion and I’m anxiously waiting for some
resolution. I think Maxim’s approach is right: implementing a
gzip_cache filter that could be used for caching any content. This
would allow for much more flexibility than just using it for
proxy_pass/proxy_cache.

This could effectively eliminate the gzip_static filter. If you have
plain HTML served and “gzip on;” plus something like “gzip_cache on;”
all of your plain text content would be served using the cached gzip
content. Similarly, you would be able to use this for proxy_pass,
proxy_store, fastcgi_pass, etc. It wouldn’t matter since the caching of
the gzip’d content would be stored in the gzip_cache filter and
negotiated directly with the client.

There’s a lot of chatter about using Varnish or Lighty or <insert
another web/cache server here> to give Nginx pre-gzipped data in a proxy
response. This sounds absolutely horrible. The bottom line is that if
Nginx is the webserver that is dealing with the client directly, it
should then be Nginx’s responsibility to serve the appropriate version
(gzip’d or plain text) content based on the client’s request headers.
That being said, Nginx should be managing it’s own cache of filtered
items.

The same goes for the image_filter. Currently, I use the image_filter
for a project and I’m bummed out that Nginx is scaling the same image
over and over each time it is requested. An image_filter_cache would be
another great feature.

Am I on the right track?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,120314,201370#msg-201370

You can cache the image filter using either lua or through a proxy pass
to
yourself.

But yes I agree with your points.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs