# How to make nginx loadbalancer give 404 when all upstream servers are down

**URL:** <https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988>\
**Category:** NGINX\
**Created:** [August 13, 2015, 11:13am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988 "2015-08-13T11:13:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rafa\_F](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/958977/32.png) [@Rafa\_F](https://www.ruby-forum.com/u/Rafa_F)\
**Post date:** [August 13, 2015, 11:13am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/1 "2015-08-13T11:13:46Z")

</div>

Hi folks,

I have been stuck this issue for a long time now. Searches could not  
solve  
my issue hence posting here.  
Please help.

I am using nginx as a load balancer and ngnix.conf looks like:

worker\_processes 4;

events { worker\_connections 1024; }

http {  
upstream ab\_backend {  
server :7000;  
server :7000 backup;  
}

```
server {
        listen 8000;

        location / {
                proxy_set_header X-Forwarded-For

```

$proxy\_add\_x\_forwarded\_for;  
proxy\_pass http://ab\_backend;  
}  
}  
}  
IP1 and IP2 are simple httpd docker containers. From another machine I  
am  
running ab tool. What I am observing is: When IP1 is up the http it  
responds  
to request from ab. Bring IP1 and backup IP2 starts to respond to al  
http  
requests.

starnge thing is when both IP1, IP2 are down the nginx server itself  
takes  
it up and responds to the http requests from ab. Is there a way to make  
nginx not behave this way?

what I am looking for is way to force http requests to be answered by  
only  
the upstream servers if none of them are reachable requests should get  
errors instead of giving out non bona fide responses.

I am testing using ab and httperf tool and saw the bahavior in both  
cases.  
Even when no upstreams are up test reports no Errors or dropped  
requests.  
Please help 🙂

Posted at Nginx Forum:

> **[how to make nginx loadbalancer give 404 when all upstream servers are down](https://forum.nginx.org/read.php?2,260980,260980#msg-260980)**
>
> Hi folks, I have been stuck this issue for a long time now. Searches could not solve my issue hence posting here. Please help. I am using nginx as a load balancer and ngnix.conf looks like: worker\_processes 4; events { worker\_connect

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 13, 2015, 1:12pm UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/2 "2015-08-13T13:12:56Z")

</div>

On Thu, Aug 13, 2015 at 05:12:57AM -0400, karkunpavan wrote:

Hi there,

> starnge thing is when both IP1, IP2 are down the nginx server itself takes  
> it up and responds to the http requests from ab. Is there a way to make  
> nginx not behave this way?

That does seem strange to me.

What output do you get from a request like

curl -i [http://your-nginx:8000/](http://your-nginx:8000/)

when one “upstream” is active; and when no “upstream” is active?

What does your nginx access\_log say for those two requests?

## f

Francis D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 14, 2015, 8:06am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/3 "2015-08-14T08:06:53Z")

</div>

Hi Fracis,

Thanks for taking a look. With it curl behaves as expected, but with  
tools  
like ab and httperf, I am not seeing bad gateway responses when both the  
upstream servers are down. Read ahead for more details.

Below is info using ab request and curl request.

ab:

When one upstream server is up:

nginx console:  
\<\<client\_IP\>\> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45  
“-”  
“ApacheBench/2.3”  
\<\<client\_IP\>\> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45  
“-”  
“ApacheBench/2.3”  
upstream httpd server console:  
\<\<nginx\_IP\>\> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45  
\<\<nginx\_IP\>\> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45

When both upstream servers are down:

2015/08/14 13:43:45 [error] 5#5: \*14 connect() failed (111: Connection  
refused) while connecting to upstream, client: \<\<client\_IP\>\>, server: ,  
request: “GET / HTTP/1.0”, upstream: “http://\<upstream\_1\_IP\>\>:/”,  
host: “\<nginx\_IP:”  
2015/08/14 13:43:45 [error] 5#5: \*14 connect() failed (111: Connection  
refused) while connecting to upstream, client: \<\<client\_IP\>\>, server: ,  
request: “GET / HTTP/1.0”, upstream: “http://\<upstream\_2\_IP\>:/”,  
host:  
“\<nginx\_IP:”  
2015/08/14 13:43:45 [error] 5#5: \*14 no live upstreams while connecting  
to  
upstream, client: \<\<client\_IP\>\>, server: , request: “GET / HTTP/1.0”,  
upstream: “http://ab\_backend/”, host: “\<nginx\_IP:”  
\<\<client\_IP\>\> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172  
“-”  
“ApacheBench/2.3”  
\<\<client\_IP\>\> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172  
“-”  
“ApacheBench/2.3”

as a result in bith cases ab just shows all successful responses:

# ab -n 1 -c 1 http://\<nginx\_IP\>:/

This is ApacheBench, Version 2.3 \<$Revision: 655654 $\>  
Copyright 1996 Adam Twiss, Zeus Technology Ltd, [http://www.zeustech.net/](http://www.zeustech.net/)  
Licensed to The Apache Software Foundation, [http://www.apache.org/](http://www.apache.org/)  
Benchmarking \<\<nginx\_IP\>\> (be patient)…done  
Server Software: nginx/1.9.3  
Server Hostname: 10.107.53.127  
Server Port: 9000  
Document Path: /  
Document Length: 45 bytes  
Concurrency Level: 1  
Time taken for tests: 0.040 seconds  
Complete requests: 1  
Failed requests: 0  
Write errors: 0  
Keep-Alive requests: 1  
Total transferred: 285 bytes  
HTML transferred: 45 bytes  
Requests per second: 24.85 [#/sec] (mean)  
Time per request: 40.234 [ms] (mean)  
Time per request: 40.234 [ms] (mean, across all concurrent  
requests)  
Transfer rate: 6.92 [Kbytes/sec] received

Connection Times (ms)  
min mean[+/-sd] median max  
Connect: 2 2 0.0 2 2  
Processing: 39 39 0.0 39 39  
Waiting: 38 38 0.0 38 38  
Total: 40 40 0.0 40 40

curl:

when one upstream is up:  
\<\<client\_IP\>\> - - [14/Aug/2015:13:58:37 +0000] “GET / HTTP/1.1” 200 45  
“-”  
“curl/7.19.7 (x86\_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0  
zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

When both upstream servers are down:  
2015/08/14 14:00:37 [error] 8#8: \*26 connect() failed (111: Connection  
refused) while connecting to upstream, client: \<\<client\_IP\>\>, server: ,  
request: “GET / HTTP/1.1”, upstream: “http://\<upstream\_1\_IP\>:/”,  
host:  
“\<nginx\_IP\>:”  
2015/08/14 14:00:37 [error] 8#8: \*26 connect() failed (111: Connection  
refused) while connecting to upstream, client: \<\<client\_IP\>\>, server: ,  
request: “GET / HTTP/1.1”, upstream: “http://\<upstream\_2\_IP\>:/”,  
host:  
“\<nginx\_IP\>:”  
\<\<client\_IP\>\> - - [14/Aug/2015:14:00:37 +0000] “GET / HTTP/1.1” 502 172  
“-”  
“curl/7.19.7 (x86\_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0  
zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

and curl gets a “502 Bad Gateway” which is right.

Any suggestions what I might be doing wrong?

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,260980,260998#msg-260998](http://forum.nginx.org/read.php?2,260980,260998#msg-260998)

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 14, 2015, 9:12am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/4 "2015-08-14T09:12:54Z")

</div>

## karkunpavan Wrote:

> \<\<client\_IP\>\> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502  
> 172 “-” “ApacheBench/2.3”  
> \<\<client\_IP\>\> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502  
> 172 “-” “ApacheBench/2.3”  
> […]  
> \<\<client\_IP\>\> - - [14/Aug/2015:14:00:37 +0000] “GET / HTTP/1.1” 502  
> 172 “-” “curl/7.19.7 (x86\_64-redhat-linux-gnu) libcurl/7.19.7  
> NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

I see a 502 for both tools, might be an ‘ab’ thing.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,260980,261000#msg-261000](http://forum.nginx.org/read.php?2,260980,261000#msg-261000)

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 14, 2015, 9:46am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/5 "2015-08-14T09:46:12Z")

</div>

Indeed. It’s right there I did not even see it. Thanks itpp2012 and  
Francis  
Daly.

since ab and httperf are behaving unexpectedly, I think best way is to  
parse  
the nginx logs to find out dropped requests.

Thanks a lot 🙂 🙂

Posted at Nginx Forum:

> **[Re: how to make nginx loadbalancer give 404 when all upstream servers are down](https://forum.nginx.org/read.php?2,260980,261001#msg-261001)**
>
> Hi folks, I have been stuck this issue for a long time now. Searches could not solve my issue hence posting here. Please help. I am using nginx as a load balancer and ngnix.conf looks like: worker\_processes 4; events { worker\_connect

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 14, 2015, 9:54am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/6 "2015-08-14T09:54:58Z")

</div>

On Fri, Aug 14, 2015 at 02:06:29AM -0400, karkunpavan wrote:

Hi there,

> Thanks for taking a look. With it curl behaves as expected, but with tools  
> like ab and httperf, I am not seeing bad gateway responses when both the  
> upstream servers are down.

> ab:
> 
> When one upstream server is up:
> 
> nginx console:  
> \<\<client\_IP\>\> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45 “-”  
> “ApacheBench/2.3”

Response status 200, content size 45.

> When both upstream servers are down:
> 
> \<\<client\_IP\>\> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172 “-”  
> “ApacheBench/2.3”

Response status 502, content size 172.

> as a result in bith cases ab just shows all successful responses:
> 
> # ab -n 1 -c 1 http://\<nginx\_IP\>:/

> Server Port: 9000  
> Complete requests: 1  
> Failed requests: 0  
> HTML transferred: 45 bytes

One request, 45 bytes of content. That is the “success” case.

If the “fail” case shows the same, while the nginx logs show something  
else, then your testing tool is wrong.

> Any suggestions what I might be doing wrong?

ab is talking to port 9000.

Your previous config showed nginx listening on port 8000.

Might that be related?

## f

Francis D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![karkunpavan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e79b87/32.png) [@karkunpavan](https://www.ruby-forum.com/u/karkunpavan)\
**Post date:** [August 14, 2015, 10:18am UTC](https://www.ruby-forum.com/t/how-to-make-nginx-loadbalancer-give-404-when-all-upstream-servers-are-down/242988/7 "2015-08-14T10:18:20Z")

</div>

nginx is listening on 8000. port 9000 is mapped to 8000 on the docker so  
I  
think it’s not related.

Posted at Nginx Forum:

> **[Re: how to make nginx loadbalancer give 404 when all upstream servers are down](https://forum.nginx.org/read.php?2,260980,261004#msg-261004)**
>
> Hi folks, I have been stuck this issue for a long time now. Searches could not solve my issue hence posting here. Please help. I am using nginx as a load balancer and ngnix.conf looks like: worker\_processes 4; events { worker\_connect