Forum: NGINX Nginx unexpectedly returning 304 for content cached from proxy

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
F53376936223195fa43f0eb8c5fd4ed2?d=identicon&s=25 Antonio L. (Guest)
on 2009-06-08 11:52
(Received via mailing list)
Hello!
We recently started using Nginx 0.7.59 to replace Squid on a couple of
our
static file caching servers (Debian 5.0). However, nginx is returning
"HTTP
304 Not Modified" responses for some—not all—of the cached images, even
if
the browser doesn't send an If-modified-since header -- causing lots of
broken images when viewing the site in browser.

As soon as I turn off the "proxy_cache" directive and let the requests
be
proxied to the upstream server, bypassing nginx's cache, no more
unexpected
304's are served, and the broken images disappear.

Do you have any idea of what might be causing this problem?

My nginx.conf file is below:

###
worker_processes  3;

error_log  logs/error.log;

events {
    worker_connections  10240;
    use epoll;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  off;

    sendfile        on;
    keepalive_timeout  10;

    gzip  on;

    upstream backend {
        server  xxx.xxx.xxx.xxx;
    }

    proxy_buffer_size 16k;
    proxy_buffers 8 16k;
    proxy_busy_buffers_size 32k;
    proxy_intercept_errors on;
    proxy_cache_path /var/www/cache levels=1:2
            keys_zone=zone_one:10m inactive=24h;

    server {
        listen       80;
        server_name  localhost;

        location / {
            proxy_cache zone_one;
            proxy_cache_valid 200 7d;
            proxy_pass http://backend;
        }
    }
}
###

I added the "proxy_cache_valid" directive to only cache 200 responses,
but
that didn't seem to change anything (even after deleting the cache).

This is pretty much what I'm seeing:

$ curl -I
http://www.myserver.com/thumbs/125x100/10081/m88_1...
HTTP/1.1 304 Not Modified
Server: nginx/0.7.59
Date: Mon, 08 Jun 2009 09:26:05 GMT
Connection: keep-alive
Last-Modified: Mon, 27 Apr 2009 07:10:40 GMT
Expires: Wed, 08 Jul 2009 08:19:13 GMT
Cache-Control: max-age=2592000

If I disable the proxy_cache (leaving just proxy_pass in location /):

$ curl -I
http://www.myserver.com/thumbs/125x100/10081/m88_1...
HTTP/1.1 200 OK
Server: nginx/0.7.59
Date: Mon, 08 Jun 2009 09:28:17 GMT
Content-Type: image/jpeg
Connection: keep-alive
Content-Length: 3869
Last-Modified: Mon, 27 Apr 2009 07:10:40 GMT
Vary: Accept-Encoding
Expires: Wed, 08 Jul 2009 09:32:33 GMT
Cache-Control: max-age=2592000
Accept-Ranges: bytes

I'm not quite sure where to start debugging this.  Does anyone have any
advice?

Thanks,
Antonio
4047860c1dcc8eb044b647ba4ecda617?d=identicon&s=25 merlin corey (Guest)
on 2009-06-08 12:36
(Received via mailing list)
Please don't double post to the list and the forum.

Thanks
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-06-08 12:37
(Received via mailing list)
On Mon, Jun 08, 2009 at 01:39:12PM +0400, Antonio L. wrote:

>
>     worker_connections  10240;
>     keepalive_timeout  10;
>     proxy_intercept_errors on;
>             proxy_pass http://backend;
>         }
>     }
> }
> ###

This is a bug. I will try to fix in today 0.8.1.
As quick workaround, you may set any proxy_set_header directive in
cached location, e.g.:

         location / {
             proxy_cache zone_one;
             proxy_cache_valid 200 7d;
             proxy_pass http://backend;
+            proxy_set_header  Host  backend;
         }
F53376936223195fa43f0eb8c5fd4ed2?d=identicon&s=25 Antonio L. (Guest)
on 2009-06-08 14:10
(Received via mailing list)
Thank you! Adding the proxy_set_header directive worked (after clearing
the
cache).
I apologize for the double post.

-a
2974d09ac2541e892966b762aad84943?d=identicon&s=25 uqtdettr (Guest)
on 2009-09-22 00:18
(Received via mailing list)
I'm getting similar problems with 304 messages being incorrectly
returned with nginx 0.8.14:


> HEAD /css/rich.css HTTP/1.1
> User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8i zlib/1.2.3 
libssh2/0.15-CVS
> Host: prod-login.uqconnect.net
> Accept: */*
>
< HTTP/1.1 304 Not Modified
< Server: nginx/0.8.14
< Date: Mon, 21 Sep 2009 04:52:33 GMT
< Connection: keep-alive
< ETag: "189ad-1b68-4727c27516700"
< Expires: Mon, 21 Sep 2009 04:52:34 GMT
< Cache-Control: max-age=5


instead of:

> HEAD /css/rich.css HTTP/1.1
> User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8i zlib/1.2.3 
libssh2/0.15-CVS
> Host: prod-login.uqconnect.net
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/0.8.14
< Date: Mon, 21 Sep 2009 04:52:32 GMT
< Content-Type: text/css
< Connection: keep-alive
< Last-Modified: Tue, 01 Sep 2009 03:57:45 GMT
< ETag: "20c55-1b68-4727c2723a040"
< Content-Length: 7016
< Cache-Control: max-age=5
< Expires: Mon, 21 Sep 2009 04:52:33 GMT
< Accept-Ranges: bytes


This only happens occasionally. Certainly less than 10% of requests.
Unfortunately, failing to load CSS is rather spectacular...

The relevant portion of my config is:

gzip on;
gzip_proxied any;
# Need Vary on encoding though, otherwise caches may get confused
gzip_vary on;

error_log       /var/log/nginx/error.log info;

proxy_temp_path /var/tmp/nginx/tmp;
# Should not be used, but useful for ensuring "cache" dir is created
proxy_cache_path  /var/tmp/nginx/cache            levels=1:2
keys_zone=core:1m;
# Actual caches start here
proxy_cache_path  /var/tmp/nginx/cache/itmp-prod  levels=1:2
keys_zone=itmp-prod:1m;
proxy_cache_path  /var/tmp/nginx/cache/itmp-test  levels=1:2
keys_zone=itmp-test:1m;

proxy_cache_valid  200 302  5m;
proxy_cache_valid  301 1h;
proxy_cache_valid  404 30s;

upstream apache {
        server apache.localhost:8080;
        server itmp1-gpn.soe.uq.edu.au:8080 backup;
        server itmp1-prentice.soe.uq.edu.au:8080 backup;
        server itmp2-prentice.soe.uq.edu.au:8080 backup;
}

server {
        listen 80;
        server_name prod-login.uqconnect.net;

        proxy_cache itmp-prod;

        location / {
                # Proxy to apache backends
                proxy_pass http://apache;
                # Set header to be what we requested
                proxy_set_header        Host    $host;
                # Need this for snooping with tcpdump (turns off
upstream compression)
                proxy_set_header        Accept-Encoding  "";
                # Set real IP header (needed for portal client IP
detection)
                proxy_set_header  X-Real-IP  $remote_addr;
        }

}

server {
        listen 80;
        server_name test-login.uqconnect.net;
        server_name dev-login.uqconnect.net;

        proxy_cache itmp-test;

        location / {
                # Proxy to apache backends
                proxy_pass http://apache;
                # Set header to be what we requested
                proxy_set_header        Host    $host;
                # Need this for snooping with tcpdump (turns off
upstream compression)
                proxy_set_header        Accept-Encoding  "";
                # Set real IP header (needed for portal client IP
detection)
                proxy_set_header  X-Real-IP  $remote_addr;
        }

}


Any ideas on a work-around or further debugging details I should
provide?

Thank you,

Tim

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2703,8049#msg-8049
2974d09ac2541e892966b762aad84943?d=identicon&s=25 uqtdettr (Guest)
on 2009-10-27 02:54
(Received via mailing list)
I believe I have now located cause of this problem. It appears that
"If-None-Match" is being passed back to the Apache backend. (For
reference: If-None-Match
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14....)

Unfortunately, the 0.8.1 fix doesn't apply to this header:


*) Bugfix: the "If-Modified-Since", "If-Range", etc. client request
       header lines were passed to backend while caching if no
       "proxy_set_header" directive was used with any parameters.


As a result, the Apache backend can still return a 304, which nginx
happily caches. This is obviously not correct behavior for HTTP/1.1
(RFC2616):


If a 304 response indicates an entity not currently cached, then the
cache MUST disregard the response and repeat the request without the
conditional.


As a workaround, I'm removing this header for proxying:

proxy_set_header If-None-Match "";


However, this causes requests using only If-None-Match to return 200 OK:


> GET /css/rich.css HTTP/1.1
> User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8k zlib/1.2.3 
libssh2/0.15-CVS
> Host: prod-login.uqconnect.net:444
> Accept: */*
> If-None-Match: "48892-1dfa-47682392e3c00"
>
< HTTP/1.1 200 OK
< Server: nginx/0.8.17
< Date: Tue, 27 Oct 2009 01:28:41 GMT
< Content-Type: text/css
< Connection: keep-alive
< Last-Modified: Thu, 22 Oct 2009 08:53:04 GMT
< ETag: "48892-1dfa-47682392e3c00"
< Accept-Ranges: bytes
< Content-Length: 7674
< Cache-Control: max-age=5
< Expires: Tue, 27 Oct 2009 01:28:46 GMT


This is fine for now, though not ideal. RFC2616 only states that a
server SHOULD return a 304, not that it must. (
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.... ).

I don't think the work-around should be required though. While the proxy
module is limited to HTTP/1.0 (and RFC1945 doesn't include
"If-None-Match"), nginx should still comply with RFC2616 for its own
replies without extra configuration. This bug is particularly nasty
because the current stripping of "If-Modified-Since" means that 304 will
be returned in cases where the original preconditions would have
generated a 200 OK.

Any chance of a bug fix?

Thank you,

Tim

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2703,17110#msg-17110
2974d09ac2541e892966b762aad84943?d=identicon&s=25 arnoldguo (Guest)
on 2010-01-22 23:40
(Received via mailing list)
the similar issue:
if use default setting,If-Modified-Since header is not fw to back end,
when the cache file expired the cache server will get the file again
from the backend server even if the file is not changed,it's not good
for a high traffic load site.
is use proxy_set_header If-Modified-Since $http_if_modified_since,then
will get  unexpectedly returning 304 .
cache server cache 304 is not good i think....
Maybe the nginx should see wether the cache file exist first:
if exist then fw the if-Modified-Since header to backend,if the file has
been changed, get file from the backend and overwrite the cache file,if
not changes,return 304,then remain the cache file.
if no exist,then get rid of the  If-Modified-Since header to get a new
file from the backend

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,2703,44200#msg-44200
This topic is locked and can not be replied to.