404 fallback only hitting 2/3 upstream sets

I currently have a setup where I have 3 upstream sets (directives?), and
it will fallback in a chain. So if it gets a 404 from setA, it’ll go to
setB. If setB also has a 404, then setC. The config is:

upstream setA {
server ipA:port
}

upstream setB {
server ipB:port
}

upstream setC {
server ipC:port
}

server {
    listen       80;
    server_name  localhost;


  #SetAHandler
  location /
  {
    access_log  logs/access.log  main;

    error_page 404 502 503 504 = @SetBHandler;
    proxy_intercept_errors on;
    proxy_pass http://setA;
    proxy_set_header Host $host;
    proxy_connect_timeout 60s;
    proxy_next_upstream error timeout http_500 http_502 http_503

http_504 http_404;

  #  proxy_cache nginx_cache;
  #  proxy_cache_valid  200 304 12h;
  #  expires      1d;

  #  break;
  }


  location @SetBHandler
  {
    access_log  logs/access.log  main;
    proxy_intercept_errors on;

    error_page 404 502 503 504 = @SetCHandler;
    proxy_pass http://setB;
    proxy_set_header Host $host;
    proxy_connect_timeout 60s;
    proxy_next_upstream error timeout http_500 http_502 http_503

http_504 http_404;

  #  proxy_cache nginx_cache;
  #  proxy_cache_valid  200 304 12h;
  #  expires      1d;

  #  break;
  }

  location @SetCHandler
  {
    access_log  logs/access.log  main;
    proxy_pass http://setC;
    proxy_set_header Host $host;
    proxy_connect_timeout 60s;
    proxy_next_upstream error timeout http_500 http_502 http_503

http_504 http_404;
#Last proxy, return whatever it gives
proxy_intercept_errors off;

  #  proxy_cache nginx_cache;
  #  proxy_cache_valid  200 304 12h;
  #  expires      1d;

  #  break;
  }

The problem is, it only seems to check setA and setB, and bypasses setC
completely. I tested it by putting a file on the third IP in setC that
wasn’t in setA or setB. Whenever I hit Nginx, it always returns 404,
when the expected behavior is for it to return the file at ipC.

I have it logging th $upstream_addr, and everytime it 404s, the logs
always show:

:

Is there some directive from the HttpProxyModule that limits it to 2 by
default?

Posted at Nginx Forum:

I think i got it. I had to add the following directive under the server
{ } heading:

recursive_error_pages on;

I’m checking the access logs, and I’m seeing the third one now (ipA :
ipB : ipC), and it’s also returning the file correctly. I hope this is
the right way.

Posted at Nginx Forum:

Hello!

On Tue, Aug 07, 2012 at 01:15:45PM -0400, helliax wrote:

I think i got it. I had to add the following directive under the server
{ } heading:

recursive_error_pages on;

I’m checking the access logs, and I’m seeing the third one now (ipA :
ipB : ipC), and it’s also returning the file correctly. I hope this is
the right way.

It is, more or less. You should be careful though, as
recursive_error_pages set to on might easily result in infinite
loop (nginx will break it eventually, but nevertheless). I would
recommend limiting recursive_error_pages scope to minimum
possible.

In your case, it should be enough to set recursive_error_pages to
on in “location /”.

Maxim D.

Thanks for heads up, Maxim. In what cases would it go into an infinite
loop if I left it outside? I had thought that “location /” would catch
all requests, and setting

proxy_intercept_errors off

at the last handler would break the chain by returning whatever upstream
setC returned?

Posted at Nginx Forum:

That makes sense. Thanks so much for all your help.

Posted at Nginx Forum:

Hello!

On Wed, Aug 08, 2012 at 12:35:41PM -0400, helliax wrote:

Thanks for heads up, Maxim. In what cases would it go into an infinite
loop if I left it outside? I had thought that “location /” would catch
all requests, and setting

proxy_intercept_errors off

at the last handler would break the chain by returning whatever upstream
setC returned?

Even with proxy_intercept_errors set to off errors might happen -
e.g. 502 will be generated if nginx won’t be able to connect to
upstream for some reason.

In any case I don’t think that your config creates a loop (unless
you have some error_pages defined at http level). I’ve mostly
wrote about recursive_error_pages in general.

Maxim D.