Bug while using "proxy_cache_use_stale updating"

Hi everybody,

I come up with a strange bug. I’m using nginx as a proxy cache to
apache.

I’m using the following conf (only useful line):
proxy_buffering on;
proxy_buffers 4 8k;
proxy_cache_use_stale updating error timeout invalid_header http_500
http_502 http_503 http_504;
proxy_cache_key $scheme$host$request_uri;
proxy_send_timeout 5;
proxy_read_timeout 120;
proxy_connect_timeout 5;
proxy_hide_header X-Powered-By;
proxy_cache_path /LIBRE/nginx/cache levels=1:2
keys_zone=cache.www:10m inactive=1d max_size=4g;
proxy_ignore_headers Expires Cache-Control;

upstream backend.www {
server 10.26.49.71:8004;
server 10.26.49.71:8005;
server 10.26.49.71:8006;
server 10.26.49.71:8007;
server 10.26.49.71:8011;
server 10.26.49.71:8012;
server 10.26.49.71:8015;
server 10.26.49.71:8016;
}

server {
proxy_cache cache.www;
location / {
proxy_cache_valid 200 302 1m;
gzip on;
proxy_pass http://backend.www;
}

Until now everything was working great. But a bug come up to me.

It seems that for an unknown reason, an “update cache” request didn’t
go well. And it stays as is (state: UPDATING). So every next request
are served from cache because for nginx there is still a request to
the backend running for this request. So even hours after having
removing a page in the backend, nginx is still serving the old and
valid (not 404) content.

Here is what I have in the access log.

17:19:26 /pid21838/test.html 200 MISS
17:20:32 /pid21838/test.html 200 EXPIRED <-- more than one minute
later, the backend is requested to update the cache
17:20:49 /pid21838/test.html 200 HIT <-- less than one minute later,
the cache is served
17:20:49 /pid21838/test.html 200 HIT <-- the same
17:21:40 /pid21838/test.html 404 EXPIRED <-- The cache is expired,
request backend. The backend answers with a 404. But the cache does
not seem to be updated
17:21:42 /pid21838/test.html 200 UPDATING <-- all the next requests
are served from stale cache.
17:22:09 /pid21838/test.html 200 UPDATING
17:23:06 /pid21838/test.html 200 UPDATING
17:27:00 /pid21838/test.html 200 UPDATING
17:32:16 /pid21838/test.html 200 UPDATING
17:41:20 /pid21838/test.html 200 UPDATING
17:44:06 /pid21838/test.html 200 UPDATING
17:45:09 /pid21838/test.html 200 UPDATING
17:47:01 /pid21838/test.html 200 UPDATING
17:52:06 /pid21838/test.html 200 UPDATING
17:52:08 /pid21838/test.html 200 UPDATING
19:12:08 /pid21838/test.html 200 UPDATING
19:20:59 /pid21838/test.html 200 UPDATING
19:34:36 /pid21838/test.html 200 UPDATING
19:58:00 /pid21838/test.html 200 UPDATING
20:11:55 /pid21838/test.html 200 UPDATING <-- 3 hours later: still the
same

I can reproduce the problem:

1- create a page on the backend (say /test.html)
2- request the frontend for /test.html --> MISS 200
3- request the frontend for /test.html before it expires --> HIT 200
4- request the frontend for /test.html after it expires --> EXPIRED 200
5- remove /test.html from the backend
6- request the frontend for /test.html after it expires --> EXPIRED 404
6- request the frontend for /test.html any time you want --> UPDATING
200 (the old /test.html is still served)

I’m using nginx 0.8.4.

I didn’t have time to test with newer releases.

Hope it helps

++ Jerome

Hi Jerome, I may be hitting a similar issue with 0.8.10. Backend is also
an apache box. I don’t see the MISS / HIT / UPDATING / EXPIRED status
strings in my log. What log_format are you using to log your accesses?
The default?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,5225,5243#msg-5243

2009/8/27 icqheretic [email protected]:

Hi Jerome, I may be hitting a similar issue with 0.8.10.

I confirm that the bug is having a similar issue with 0.8.10. I’ve
just tested it.

By the way, the bug does not appear if I add 404 page to
proxy_cache_valid directive:

proxy_cache_valid 200 302 404 30s;

An other solution is to remove the “updating” option for the
“proxy_cache_use_stale” directive.

I’m looking into the code to try to find a solution.

++ jerome

2009/8/27 icqheretic [email protected]:

Hi Jerome, I may be hitting a similar issue with 0.8.10. Backend is also an apache box. I don’t see the MISS / HIT / UPDATING / EXPIRED status strings in my log. What log_format are you using to log your accesses? The default?

I’m not using default log_format. You can add the
$upstream_cache_status variable, it should log the info you want.

++ Jerome

On Fri, Aug 28, 2009 at 03:57:04PM +0200, J?r?me Loyet wrote:

šproxy_cache_valid 200 302 404 30s;
+++ src/http/ngx_http_upstream.c 2009-08-28 15:54:43.000000000 +0200
@@ -2574,6 +2574,8 @@
} else if (p->upstream_error) {
ngx_http_file_cache_free(r, u->pipe->temp_file);
}

  •    } else {
    
  •        ngx_http_file_cache_free(r, u->pipe->temp_file);
       }
    

#endif

No, this is not correct fix. I will look this.

Le 28 août 2009 13:59, Jérôme Loyet[email protected] a écrit :

An other solution is to remove the “updating” option for the
“proxy_cache_use_stale” directive.

I’m looking into the code to try to find a solution.

After several trials, I made a patch. This seems to correct the issue.
Hope no other bugs are implied buy this diff:

— src/http/ngx_http_upstream.c.orig 2009-08-23 18:10:39.000000000
+0200
+++ src/http/ngx_http_upstream.c 2009-08-28 15:54:43.000000000
+0200
@@ -2574,6 +2574,8 @@
} else if (p->upstream_error) {
ngx_http_file_cache_free(r, u->pipe->temp_file);
}

  •    } else {
    
  •        ngx_http_file_cache_free(r, u->pipe->temp_file);
       }
    

#endif

On Thu, Aug 27, 2009 at 11:06:46AM +0200, J?r?me Loyet wrote:

proxy_send_timeout 5;
server 10.26.49.71:8004;
proxy_cache cache.www;
are served from cache because for nginx there is still a request to
the cache is served
17:41:20 /pid21838/test.html 200 UPDATING

200 (the old /test.html is still served)

I’m using nginx 0.8.4.

I didn’t have time to test with newer releases.

Hope it helps

The patch.

Thx igor, I’ll test that asap

2009/9/1 Igor S. [email protected]:

I was wondering if anyone else is still having this problem of Nginx
getting stuck in the UPDATING state after upgrading to the fixed version
of Nginx.

I saw this using Nginx 0.7.62 and I upgraded to 0.8.24 because I saw it
had been fixed, but recently saw the problem again.

Here is my configuration:

proxy_cache_path /mnt/nginx_cache levels=1:2
                 keys_zone=one:10m
                 inactive=7d max_size=10g;
proxy_cache_path /mnt/nginx_cache_media levels=1:2
                 keys_zone=MEDIA:10m
                 inactive=7d max_size=5g;
proxy_temp_path /var/www/nginx_temp;

server {
    listen 80;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_pass_header Set-Cookie;
    client_max_body_size 8m;

    # capture cookies for use in the cache key
    set $wordpress_logged_in "";
    set $comment_author_email "";
    set $comment_author "";
    if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
        set $wordpress_logged_in wordpress_logged_in_$1;
    }
    if ($http_cookie ~* "comment_author_email_[^=]*=([^;]+)(;|$)") {
        set $comment_author_email comment_author_email_$1;
    }
    if ($http_cookie ~* "comment_author_[^=]*=([^;]+)(;|$)") {
        set $comment_author comment_author_$1;
    }
    set $my_cache_key 

$scheme$host$uri$is_args$args$wordpress_logged_in$comment_author_email$comment_author;

    # proxy cache config for media files
    location ~* \.(gif|jpg|jpeg|png|js|css)$ {
        proxy_pass http://backend;
        proxy_cache MEDIA;
        proxy_cache_key $my_cache_key;

        proxy_cache_valid  200 302 304 30d;
        proxy_cache_valid  301 1h;
        proxy_cache_valid  any 1m;
        proxy_cache_use_stale error timeout invalid_header updating 

http_500 http_502 http_503 http_504 http_404;
expires 30d;
}

    # proxy cache config for all other files
    location / {
        proxy_pass http://backend;
        proxy_cache one;
        proxy_cache_key $my_cache_key;

        proxy_cache_valid  200 302 304 10m;
        proxy_cache_valid  301 1h;
        proxy_cache_valid  any 1m;
        proxy_cache_use_stale error timeout invalid_header updating 

http_500 http_502 http_503 http_504 http_404;
}

    # Don't use the cache for the following locations
    location /wp-admin {
        proxy_read_timeout 180;
        proxy_pass http://backend;
    }
    location /wp-login.php { proxy_pass http://backend; }
}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,5225,30772#msg-30772

I have 0.8.54, and I just hit this issue, which resulted in the RSS feed
not updating for a full day. Due to the way my server is configured,
because of my cookies, I was getting straight to the feed bypassing the
cache and only found the problem when using incognito mode without
cookies. A server restart seemed to fix the problem for now.

Igor, is this fixed in some version >0.8.54?

Thanks,
Artem

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,5225,218347#msg-218347

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs