Hi everybody,
I come up with a strange bug. I’m using nginx as a proxy cache to
apache.
I’m using the following conf (only useful line):
proxy_buffering on;
proxy_buffers 4 8k;
proxy_cache_use_stale updating error timeout invalid_header http_500
http_502 http_503 http_504;
proxy_cache_key $scheme$host$request_uri;
proxy_send_timeout 5;
proxy_read_timeout 120;
proxy_connect_timeout 5;
proxy_hide_header X-Powered-By;
proxy_cache_path /LIBRE/nginx/cache levels=1:2
keys_zone=cache.www:10m inactive=1d max_size=4g;
proxy_ignore_headers Expires Cache-Control;
…
upstream backend.www {
server 10.26.49.71:8004;
server 10.26.49.71:8005;
server 10.26.49.71:8006;
server 10.26.49.71:8007;
server 10.26.49.71:8011;
server 10.26.49.71:8012;
server 10.26.49.71:8015;
server 10.26.49.71:8016;
}
server {
proxy_cache cache.www;
location / {
proxy_cache_valid 200 302 1m;
gzip on;
proxy_pass http://backend.www;
}
…
Until now everything was working great. But a bug come up to me.
It seems that for an unknown reason, an “update cache” request didn’t
go well. And it stays as is (state: UPDATING). So every next request
are served from cache because for nginx there is still a request to
the backend running for this request. So even hours after having
removing a page in the backend, nginx is still serving the old and
valid (not 404) content.
Here is what I have in the access log.
17:19:26 /pid21838/test.html 200 MISS
17:20:32 /pid21838/test.html 200 EXPIRED ← more than one minute
later, the backend is requested to update the cache
17:20:49 /pid21838/test.html 200 HIT ← less than one minute later,
the cache is served
17:20:49 /pid21838/test.html 200 HIT ← the same
17:21:40 /pid21838/test.html 404 EXPIRED ← The cache is expired,
request backend. The backend answers with a 404. But the cache does
not seem to be updated
17:21:42 /pid21838/test.html 200 UPDATING ← all the next requests
are served from stale cache.
17:22:09 /pid21838/test.html 200 UPDATING
17:23:06 /pid21838/test.html 200 UPDATING
17:27:00 /pid21838/test.html 200 UPDATING
17:32:16 /pid21838/test.html 200 UPDATING
17:41:20 /pid21838/test.html 200 UPDATING
17:44:06 /pid21838/test.html 200 UPDATING
17:45:09 /pid21838/test.html 200 UPDATING
17:47:01 /pid21838/test.html 200 UPDATING
17:52:06 /pid21838/test.html 200 UPDATING
17:52:08 /pid21838/test.html 200 UPDATING
19:12:08 /pid21838/test.html 200 UPDATING
19:20:59 /pid21838/test.html 200 UPDATING
19:34:36 /pid21838/test.html 200 UPDATING
19:58:00 /pid21838/test.html 200 UPDATING
20:11:55 /pid21838/test.html 200 UPDATING ← 3 hours later: still the
same
I can reproduce the problem:
1- create a page on the backend (say /test.html)
2- request the frontend for /test.html → MISS 200
3- request the frontend for /test.html before it expires → HIT 200
4- request the frontend for /test.html after it expires → EXPIRED 200
5- remove /test.html from the backend
6- request the frontend for /test.html after it expires → EXPIRED 404
6- request the frontend for /test.html any time you want → UPDATING
200 (the old /test.html is still served)
I’m using nginx 0.8.4.
I didn’t have time to test with newer releases.
Hope it helps
++ Jerome
Hi Jerome, I may be hitting a similar issue with 0.8.10. Backend is also
an apache box. I don’t see the MISS / HIT / UPDATING / EXPIRED status
strings in my log. What log_format are you using to log your accesses?
The default?
Posted at Nginx Forum:
2009/8/27 icqheretic [email protected]:
Hi Jerome, I may be hitting a similar issue with 0.8.10.
I confirm that the bug is having a similar issue with 0.8.10. I’ve
just tested it.
By the way, the bug does not appear if I add 404 page to
proxy_cache_valid directive:
proxy_cache_valid 200 302 404 30s;
An other solution is to remove the “updating” option for the
“proxy_cache_use_stale” directive.
I’m looking into the code to try to find a solution.
++ jerome
2009/8/27 icqheretic [email protected]:
Hi Jerome, I may be hitting a similar issue with 0.8.10. Backend is also an apache box. I don’t see the MISS / HIT / UPDATING / EXPIRED status strings in my log. What log_format are you using to log your accesses? The default?
I’m not using default log_format. You can add the
$upstream_cache_status variable, it should log the info you want.
++ Jerome
On Fri, Aug 28, 2009 at 03:57:04PM +0200, J?r?me Loyet wrote:
šproxy_cache_valid 200 302 404 30s;
+++ src/http/ngx_http_upstream.c 2009-08-28 15:54:43.000000000 +0200
@@ -2574,6 +2574,8 @@
} else if (p->upstream_error) {
ngx_http_file_cache_free(r, u->pipe->temp_file);
}
#endif
No, this is not correct fix. I will look this.
Le 28 août 2009 13:59, Jérôme Loyet[email protected] a écrit :
An other solution is to remove the “updating” option for the
“proxy_cache_use_stale” directive.
I’m looking into the code to try to find a solution.
After several trials, I made a patch. This seems to correct the issue.
Hope no other bugs are implied buy this diff:
— src/http/ngx_http_upstream.c.orig 2009-08-23 18:10:39.000000000
+0200
+++ src/http/ngx_http_upstream.c 2009-08-28 15:54:43.000000000
+0200
@@ -2574,6 +2574,8 @@
} else if (p->upstream_error) {
ngx_http_file_cache_free(r, u->pipe->temp_file);
}
#endif
On Thu, Aug 27, 2009 at 11:06:46AM +0200, J?r?me Loyet wrote:
proxy_send_timeout 5;
server 10.26.49.71:8004;
proxy_cache cache.www;
are served from cache because for nginx there is still a request to
the cache is served
17:41:20 /pid21838/test.html 200 UPDATING
200 (the old /test.html is still served)
I’m using nginx 0.8.4.
I didn’t have time to test with newer releases.
Hope it helps
The patch.
Thx igor, I’ll test that asap
2009/9/1 Igor S. [email protected]:
I was wondering if anyone else is still having this problem of Nginx
getting stuck in the UPDATING state after upgrading to the fixed version
of Nginx.
I saw this using Nginx 0.7.62 and I upgraded to 0.8.24 because I saw it
had been fixed, but recently saw the problem again.
Here is my configuration:
proxy_cache_path /mnt/nginx_cache levels=1:2
keys_zone=one:10m
inactive=7d max_size=10g;
proxy_cache_path /mnt/nginx_cache_media levels=1:2
keys_zone=MEDIA:10m
inactive=7d max_size=5g;
proxy_temp_path /var/www/nginx_temp;
server {
listen 80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header Set-Cookie;
client_max_body_size 8m;
# capture cookies for use in the cache key
set $wordpress_logged_in "";
set $comment_author_email "";
set $comment_author "";
if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
set $wordpress_logged_in wordpress_logged_in_$1;
}
if ($http_cookie ~* "comment_author_email_[^=]*=([^;]+)(;|$)") {
set $comment_author_email comment_author_email_$1;
}
if ($http_cookie ~* "comment_author_[^=]*=([^;]+)(;|$)") {
set $comment_author comment_author_$1;
}
set $my_cache_key
$scheme$host$uri$is_args$args$wordpress_logged_in$comment_author_email$comment_author;
# proxy cache config for media files
location ~* \.(gif|jpg|jpeg|png|js|css)$ {
proxy_pass http://backend;
proxy_cache MEDIA;
proxy_cache_key $my_cache_key;
proxy_cache_valid 200 302 304 30d;
proxy_cache_valid 301 1h;
proxy_cache_valid any 1m;
proxy_cache_use_stale error timeout invalid_header updating
http_500 http_502 http_503 http_504 http_404;
expires 30d;
}
# proxy cache config for all other files
location / {
proxy_pass http://backend;
proxy_cache one;
proxy_cache_key $my_cache_key;
proxy_cache_valid 200 302 304 10m;
proxy_cache_valid 301 1h;
proxy_cache_valid any 1m;
proxy_cache_use_stale error timeout invalid_header updating
http_500 http_502 http_503 http_504 http_404;
}
# Don't use the cache for the following locations
location /wp-admin {
proxy_read_timeout 180;
proxy_pass http://backend;
}
location /wp-login.php { proxy_pass http://backend; }
}
Posted at Nginx Forum:
I have 0.8.54, and I just hit this issue, which resulted in the RSS feed
not updating for a full day. Due to the way my server is configured,
because of my cookies, I was getting straight to the feed bypassing the
cache and only found the problem when using incognito mode without
cookies. A server restart seemed to fix the problem for now.
Igor, is this fixed in some version >0.8.54?
Thanks,
Artem
Posted at Nginx Forum: