Proxy_ignore_headers doesn't seem to work

Hello,

I’ve problem similar as in this thread:
http://mailman.nginx.org/pipermail/nginx/2010-May/020349.html

nginx is ignoring my proxy_cache_valid value, using instead expire time
specified by upstream server.

proxy_ignore_headers X-Accel-Expires Expires Cache-Control
Set-Cookie;
proxy_cache_valid 200 302 30h;
proxy_cache_valid any 1h;

headers from file stored on cache:


KEY: httpXYZ.com/tiles/1.0.0/osm/15/5240/12662.jpg
HTTP/1.0 200 OK
Expires: Wed, 14 Dec 2011 16:33:27 GMT
Last-Modified: Tue, 13 Dec 2011 06:16:41 GMT
Cache-Control: max-age=600
Content-Type: image/jpeg
Content-Length: 27946
Server: Mapnik2/0.8.0
Date: Wed, 14 Dec 2011 16:23:27 GMT
Connection: close


from logs:


14/Dec/2011:10:02:11 -0600 HIT Cache-Control:max-age=600 Expires:Wed, 14
Dec 2011 16:09:55 GMT upstream_server:- upstream_status:- (…)
14/Dec/2011:10:22:55 -0600 MISS Cache-Control:max-age=600 Expires:Wed,
14 Dec 2011 16:32:55 GMT upstream_server(…)


I’ve tried to add also Date and Last-Modified to proxy_ignore_headers,
adding same headers to proxy_hide_header, but without any luck.

nginx/0.8.54 from source on RHEL5.7, and nginx/1.0.10 from official deb
package on Ubuntu10.04.

Does anybody have any idea what could be wrong?

Thanks!

Piotr

Posted at Nginx Forum:

Hello!

On Wed, Dec 14, 2011 at 11:34:15AM -0500, piotr wrote:

proxy_cache_valid 200 302 30h;
Content-Length: 27946
14 Dec 2011 16:32:55 GMT upstream_server(…)


I’ve tried to add also Date and Last-Modified to proxy_ignore_headers,
adding same headers to proxy_hide_header, but without any luck.

nginx/0.8.54 from source on RHEL5.7, and nginx/1.0.10 from official deb
package on Ubuntu10.04.

Does anybody have any idea what could be wrong?

Try “nginx -t”. If it doesn’t help, please post the full config
you’re testing with.

Maxim D.

Hello!

On Sat, Dec 17, 2011 at 10:19:29AM -0500, piotr wrote:

I’ve checked both servers for config errors before, but without any
clues.

[…]

Accept-Encoding: gzip, deflate
Connection: keep-alive
Note:
Expires == Date + Cache-Control max-age.

Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7
Date: Fri, 16 Dec 2011 14:08:43 GMT
Content-Type: image/jpeg
Connection: keep-alive
Expires: Wed, 14 Dec 2011 14:59:21 GMT
Last-Modified: Tue, 13 Dec 2011 06:16:41 GMT
Cache-Control: max-age=600
Content-Length: 18419

Note:
Expires < Date + Cache-Control max-age.

The difference is about 2 days, so it looks like
proxy_ignore_headers works as it should.

I suppose previously you’ve looked at files cached without
proxy_ignore_headers in config. Note that changing config isn’t
enough to get longer cache valid times for already cached
responses, since validity time is calculated while obtaining
response from a backend. A response have to be recached with the
new settings.

A side note: you may want to hide or change Expires header to
make sure clients will be able to cache responses. This may be
done either with proxy_hide_header or with “expires” directive.

Maxim D.

Maxim D. Wrote:

Try “nginx -t”. If it doesn’t help, please post
the full config
you’re testing with.

Hi,

Maxim, thank you for your response!

I’ve checked both servers for config errors before, but without any
clues.

nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Current configuration:

***** cat /etc/nginx/nginx.conf

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local]

“$request” ’
'$status $body_bytes_sent “$http_referer” ’
‘“$http_user_agent” “$http_x_forwarded_for”’;

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  5;

#gzip  on;

include /etc/nginx/conf.d/*.conf;

}

***** cat /etc/nginx/conf.d/proxy.conf

log_format cache '$remote_addr $time_local ’
'$upstream_cache_status ’
'proxy_host:$proxy_host ’
'Cache-Control:$upstream_http_cache_control ’
'Expires:$upstream_http_expires ’
'upstream_server:$upstream_addr ’
'upstream_status:$upstream_status ’
'“$request” ($status) ’
'“$http_user_agent” ';

proxy_cache_path /home/mpqcache levels=1:2:2
keys_zone=mpqcache:30000m;

upstream mpq_cluster {
#ip_hash;
server otile1.mqcdn.com:80;
server otile2.mqcdn.com:80;
server otile3.mqcdn.com:80;
server otile4.mqcdn.com:80;
}

server {

listen  80;
server_name mpq1.XXX.com mpq2.XXX.com;

access_log /var/log/nginx/mpq1.XXX.com_access.log cache;
error_log /var/log/nginx/mpq1.XXX.com_error.log info;

location / {
  #proxy_pass http://mpq_cluster;
  proxy_pass http://otile1.mqcdn.com;
  proxy_set_header Host otile1.mqcdn.com;
  proxy_ignore_headers X-Accel-Expires Expires Cache-Control

Set-Cookie;
proxy_cache mpqcache;
proxy_cache_key “$scheme$host$request_uri”;
proxy_cache_valid 200 302 600h;
proxy_cache_valid any 90m;
}

} # server

***** sample headers from upstream server:

http://otile3.mqcdn.com/tiles/1.0.0/osm/15/5240/12661.jpg

GET /tiles/1.0.0/osm/15/5240/12661.jpg HTTP/1.1
Host: otile3.mqcdn.com
User-Agent: Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:8.0) Gecko/20100101
Firefox/8.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en,en-us;q=0.7,pl;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
If-Modified-Since: Tue, 13 Dec 2011 06:16:41 GMT

HTTP/1.1 304 Not Modified
Content-Type: image/jpeg
Expires: Fri, 16 Dec 2011 14:16:45 GMT
Last-Modified: Tue, 13 Dec 2011 06:16:41 GMT
Cache-Control: max-age=600
Date: Fri, 16 Dec 2011 14:06:45 GMT
Connection: keep-alive

***** sample headers from nginx proxy
http://mpq1.XXX.com/tiles/1.0.0/osm/15/5240/12661.jpg

GET /tiles/1.0.0/osm/15/5240/12661.jpg HTTP/1.1
Host: mpq1.XXX.com
User-Agent: Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:8.0) Gecko/20100101
Firefox/8.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en,en-us;q=0.7,pl;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Cookie: bblastvisit=1323941196; bblastactivity=0;
__qca=P0-811271075-1323941200190;
__utma=26892847.397077741.1323941200.1323964564.1324035857.4;
__utmz=26892847.1323941200.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
bbuserid=33280; bbpassword=be7f8ba39107f78fb42020ad76ca953b;
__utmc=26892847; bbsessionhash=0d7cdd74637b9f503c4f33d7ad1f8394

HTTP/1.1 200 OK
Server: nginx/1.0.10
Date: Fri, 16 Dec 2011 14:08:43 GMT
Content-Type: image/jpeg
Connection: keep-alive
Expires: Wed, 14 Dec 2011 14:59:21 GMT
Last-Modified: Tue, 13 Dec 2011 06:16:41 GMT
Cache-Control: max-age=600
Content-Length: 18419

Thanks!

Regards,
P.

Posted at Nginx Forum:

On Wed, Jan 04, 2012 at 03:12:10PM -0500, piotr wrote:

Hi there,

I’m trying to hide those headers, you can see it in config posted
before:

proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;

No, that’s proxy_ignore_headers, not proxy_hide_header.

proxy_cache_valid 200 302 600h;
proxy_cache_valid any 90m;

But as you can see, you can find expire, cache-control headers in proxy
repsonse, and also in headers of stored cache files on disk. With values
from upstream server, not those set in config.

What comes from upstream is stored on disk. That’s expected.

proxy_ignore_headers determines which of upstream headers or
proxy_cache_valid is used by nginx to decide for how long nginx will
cache the response.

Separate from that, you can use “proxy_hide_header” to tell nginx not to
send some headers that came from upstream, to the client.

Separate from that (mostly), you can use “expires” to tell nginx how to
set Expires: and Cache-Control: headers in the response to the client.

“(mostly)” is there because nginx will only send a single Expires
header,
so if you use “expires” to set one, then the one from upstream will not
go to the client, even if it isn’t in “proxy_hide_header”.

I’m not sure now those times from my previous response, but whatever i
do (delete cached files from disk, restart nginx, wait 10 minutes to
expire files cached on disk), nginx is always returning in response
cache-control and expire in 600 sec - taking this value from upstream
server, not from the config file.

That is as expected, since you have no other value in the config file
to send in the response, and you have nothing in the config file to tell
nginx not to send what came from upstream.

Also, file is stored on disk for 600s,
after this time i can see in logs that nginx is downloading again file
from upstream server. Nginx should cache files for 600h in current
config, but i’ve also tried before other values, like 6h or 6d etc.

That observation, I don’t understand.

You can examine the first few bytes of the on-disk cache file to see
when nginx thinks that it will become stale:

od -t dL $cache_file | head -n 1

The second and fourth numbers there are valid-until- and fetched-at-
timestamps, respectively. So if they are only 600 apart, then something
unexpected (to me) is happening.

When I use configuration like

proxy_cache_valid 200 302 600h;
proxy_cache_valid any 90m;

in 1.1.11, I see them being the expected 2160000 (seconds, = 600 hours)
apart for a http 200 response.

Good luck,

f

Francis D. [email protected]

On Wed, Jan 04, 2012 at 10:26:05PM +0000, Francis D. wrote:

On Wed, Jan 04, 2012 at 03:12:10PM -0500, piotr wrote:

Hi there,

One more thing on the final “600s”…

Also, file is stored on disk for 600s,
after this time i can see in logs that nginx is downloading again file
from upstream server. Nginx should cache files for 600h in current
config, but i’ve also tried before other values, like 6h or 6d etc.

That observation, I don’t understand.

You don’t specify “inactive” in your proxy_cache_path directive, so it
defaults to 10 minutes.

If you see one new request per url every 11 minutes, then it is expected
that nginx will re-fetch from upstream each time.

If you see a new request every 9 minutes, then nginx should not need to
re-fetch from upstream.

Does that explain what you see?

f

Francis D. [email protected]

Maxim,

Sorry for the late response.

I’m trying to hide those headers, you can see it in config posted
before:

proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
proxy_cache_valid 200 302 600h;
proxy_cache_valid any 90m;

But as you can see, you can find expire, cache-control headers in proxy
repsonse, and also in headers of stored cache files on disk. With values
from upstream server, not those set in config.

I’m not sure now those times from my previous response, but whatever i
do (delete cached files from disk, restart nginx, wait 10 minutes to
expire files cached on disk), nginx is always returning in response
cache-control and expire in 600 sec - taking this value from upstream
server, not from the config file. Also, file is stored on disk for 600s,
after this time i can see in logs that nginx is downloading again file
from upstream server. Nginx should cache files for 600h in current
config, but i’ve also tried before other values, like 6h or 6d etc.

http://mpq1.XXX.com/tiles/1.0.0/osm/12/1035/1519.jpg

GET /tiles/1.0.0/osm/12/1035/1519.jpg HTTP/1.1
Host: mpq1.XXX.com
User-Agent: Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1)
Gecko/20100101 Firefox/9.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en,en-us;q=0.7,pl;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-2,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Cookie: bblastvisit=1325618979; bblastactivity=0;
__utma=26892847.843027784.1325618982.1325618982.1325703951.2;
__utmz=26892847.1325618982.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);
__qca=P0-367095175-1325618982208;
bbsessionhash=02acddda9354e88dd04bab92358aeca6; __utmc=26892847

HTTP/1.1 200 OK
Server: nginx/1.0.11
Date: Wed, 04 Jan 2012 19:52:01 GMT
Content-Type: image/jpeg
Connection: keep-alive
Content-Length: 8010
Cache-Control: max-age=600
Last-Modified: Wed, 04 Jan 2012 02:18:15 GMT
Expires: Wed, 04 Jan 2012 20:02:01 GMT

Regards,
Piotr

Posted at Nginx Forum:

Maxim,

Yes, i’ve missed this “inactive” param - unfortunately expire time from
upstream server was the same as default inactive value, with was
misleading for me.

May I suggest to add additional info about this inactive param to
Module ngx_http_proxy_module page?

Thank you for your help!

Regards,
p.

Posted at Nginx Forum:

Hello!

On Wed, Jan 04, 2012 at 03:12:10PM -0500, piotr wrote:

But as you can see, you can find expire, cache-control headers in proxy
repsonse, and also in headers of stored cache files on disk. With values
from upstream server, not those set in config.

You don’t hide the headers, and don’t set headers in config. The
proxy_ignore_headers instruct nginx to ignore headers in question
(i.e. don’t set/limit cache validity time as stored in cache
metadata based on these headers). The proxy_cache_valid
directives set cache validity times. None of these directives
hide or modify headers in question.

To hide headers, you have to use the proxy_hide_header directive
(as already suggested in previous message), see here for details:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_hide_header

To explicitly set Cache-Control/Expires headers, use the “expires”
directive, see here for details:

http://wiki.nginx.org/HttpHeadersModule#expires

(no official docs in English yet)

I’m not sure now those times from my previous response, but whatever i
do (delete cached files from disk, restart nginx, wait 10 minutes to
expire files cached on disk), nginx is always returning in response
cache-control and expire in 600 sec - taking this value from upstream
server, not from the config file. Also, file is stored on disk for 600s,
after this time i can see in logs that nginx is downloading again file
from upstream server. Nginx should cache files for 600h in current
config, but i’ve also tried before other values, like 6h or 6d etc.

Do you request the file in question with interval larger than 600
seconds, without any intermediate requests? It is expected to be
removed from cache as inactive one in such test, see desciption of
the “inactive” parameter of the “proxy_cache_path” directive here:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path

Try requesting a file once per minute to keep it active.

Maxim D.