Using $upstream* variables inside map directive

Server NGINX
1

Hello,
I’m trying to avoid caching of small responses from upstreams using map:
map $upstream_http_content_length $dontcache {
default 0;
~^\d\d$ 1;
~^\d$ 1;
}

Unfortunatelly, nginx seems to ignore $upstream* variables at the map
processing stage, hence variables like $upstream_http_content_length or
$upstream_response_length stay empty when map directive is processed
(this
can be observed in debug log as “http map started” message). In case I
use
non-upstream related variables, a map works as expected.

Question: is there any way to use $upstream* vars inside the map
directive,
or maybe someone can offer alternative way to detect small upstream
response
in order to bypass cache?

Thank you.

Posted at Nginx Forum:

2

On 7 May 2014 06:38, Kirill K. [email protected] wrote:

$upstream_response_length stay empty when map directive is processed (this
can be observed in debug log as “http map started” message). In case I use
non-upstream related variables, a map works as expected.

Question: is there any way to use $upstream* vars inside the map directive,
or maybe someone can offer alternative way to detect small upstream response
in order to bypass cache?

I don’t explicitly know how to achieve what you’re trying to, but I
seem to recall mention on this list that a map’s value gets stuck
(per-request) the first time it’s evaluated. I might be
misremembering, but this does ring a bell.

So - is your map somehow being evaluated /before/ the upstream vars
are available? Does your config perhaps cause it to be evaluated when
the initial request arrives, to see if the response should be served
from cache; then the request is proxy_pass’d, then after receiving a
response the caching bypass config is examined but to no avail as the
map has “stuck” with the initially set value?

Sorry I can’t be more specific - I’m sure others can help more
definitively!

J

3

Probably that’s the case, and I’m not sure if there’s a way to use map
inside upstream {…} or other context apart from http {…}, which
makes
your theory sound correct.
What confuses me most: I googled a bit, and using map w/
$upstream_response_length is the most common way offered to avoid
caching of
small (or zero-sized) responses, yet it just does not work in a real
life
scenario…

Posted at Nginx Forum:

4

Thanks, Ruslan,
Thing is, I tried to “debug” whether $dontcache is being set at all by
exposing it via response headers (along with content-length), and it
shows
that $upstream_response_length is ignored by map completely, i.e. no
matter
where I use $dontcache, it will never get any value different from
default
(i.e. 0). Even though $upstream_response_length is validated correctly
(and
can be exposed in headers), the map directive just ignores it.

Posted at Nginx Forum:

5

On Wed, May 07, 2014 at 08:53:56AM -0400, Kirill K. wrote:

Thanks, Ruslan,
Thing is, I tried to “debug” whether $dontcache is being set at all by
exposing it via response headers (along with content-length), and it shows
that $upstream_response_length is ignored by map completely, i.e. no matter
where I use $dontcache, it will never get any value different from default
(i.e. 0). Even though $upstream_response_length is validated correctly (and
can be exposed in headers), the map directive just ignores it.

I tested that your map works when used ONLY in proxy_no_cache,
and it indeed DTRT: responses with content length less than
100 aren’t cached.

Here’s the config snippet:

: http {
: proxy_cache_path proxy_cache keys_zone=proxy_cache:10m;
:
: map $upstream_http_content_length $dontcache {
: default 0;
: ~^\d\d$ 1;
: ~^\d$ 1;
: }
:
: server {
: listen 8000;
:
: location / {
: proxy_pass http://127.0.0.1:8001;
: proxy_cache proxy_cache;
: proxy_cache_valid 300m;
: proxy_no_cache $dontcache;
:
: add_header X-DontCache $dontcache;
: }
: }
:
: server {
: listen 8001;
:
: return 200 “ok”;
: }
: }

$ curl -i http://127.0.0.1:8000/test
HTTP/1.1 200 OK
Server: nginx/1.7.1
Date: Wed, 07 May 2014 14:34:19 GMT
Content-Type: text/plain
Content-Length: 2
Connection: keep-alive
X-DontCache: 1

ok

And there will be no file in “proxy_cache” directory.

6

On Wed, May 07, 2014 at 01:38:04AM -0400, Kirill K. wrote:

$upstream_response_length stay empty when map directive is processed (this
can be observed in debug log as “http map started” message). In case I use
non-upstream related variables, a map works as expected.

Question: is there any way to use $upstream* vars inside the map directive,
or maybe someone can offer alternative way to detect small upstream response
in order to bypass cache?

If you use $dontcache with proxy_cache_bypass, then it’s expected
behavior. At the time proxy_cache_bypass is evaluated, there’s no
response yet, so the $upstream_http_* do not exist.

If you try to use $dontcache with proxy_no_cache ONLY, it’ll work,
because the latter is evaluated after obtaining a response.

If you use it both with proxy_cache_bypass and proxy_no_cache,
please realize that using it with proxy_cache_bypass makes no
sense, and then the fact that “map” creates the so-called
cacheable variables plays its role.

I have a patch for “map” that makes map variables “volatile”.
If you absolutely need such a “map” behavior, I can send it
to you for testing, but better limit the use of $upstream_http_*
to only proxy_no_cache.

7

Ruslan, you’re a hero!
I just commented the following line in my existing config
#proxy_cache_bypass $dontcache;
and everything works now!

I won’t be able to comprehend such nginx’s behaviour w/o your help,
greatly
appreciated.

Posted at Nginx Forum: