# Nginx upstream problem

**URL:** <https://www.ruby-forum.com/t/nginx-upstream-problem/186894>\
**Category:** NGINX\
**Created:** [April 13, 2010, 10:29pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894 "2010-04-13T22:29:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 13, 2010, 10:29pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/1 "2010-04-13T22:29:28Z")

</div>

Hi,

I have several problems when using nginx as a load balancer.

Nginx 0.8.35  
CentOS 5.4 64bit  
Kernel 2.6.28.10

Configs:  
/etc/nginx/nginx.conf

user nobody;  
worker\_processes 32;  
worker\_rlimit\_nofile 10240;

error\_log /var/log/nginx/error.log;

events {  
use epoll;  
accept\_mutex off;  
worker\_connections 8192;  
}

http {  
include mime.types;  
default\_type application/octet-stream;

```
sendfile on;
tcp_nodelay on;
access_log off;
server_tokens off;
client_max_body_size 10m;

keepalive_timeout 65;

include /etc/nginx/conf.d/*.conf;

```

}

/etc/nginx/conf.d/upstrea-nonssl.conf

upstream cloud {  
server apache3 max\_fails=1 fail\_timeout=5;  
server apache2 max\_fails=1 fail\_timeout=5;  
server apache1 max\_fails=1 fail\_timeout=5;  
}

server {  
listen 1.1.1.1:80;  
location / {  
proxy\_set\_header Host $host;  
proxy\_set\_header X-Real-IP $remote\_addr;  
proxy\_set\_header X-Forwarded-For $proxy\_add\_x\_forwarded\_for;  
proxy\_read\_timeout 30;  
proxy\_next\_upstream error timeout;  
proxy\_pass [http://cloud](http://cloud);  
}  
}  
server {  
listen 1.1.1.2:80;  
location / {  
proxy\_set\_header Host $host;  
proxy\_set\_header X-Real-IP $remote\_addr;  
proxy\_set\_header X-Forwarded-For $proxy\_add\_x\_forwarded\_for;  
proxy\_read\_timeout 30;  
proxy\_next\_upstream error timeout;  
proxy\_pass [http://cloud](http://cloud);  
}  
}

Problem 1:  
Regardless of the order of the apache servers in the upstream, apache3  
most of the time gets the lowest traffic. Sometimes it gets the same,  
sometimes it gets very few traffic.

Problem 2:  
If I take down one apache server(halt or /etc/init.d/network stop) then  
I can see from the error log of nginx that it is still sending traffic  
to that apache server and I see time out or no route to host errors.  
When browsing the website I get waiting pages all the time which proves  
it tries to send to the apache server that is down.

Problem 3:  
If I use the nginx fair module then the server that is listed last in  
the upstream, gets the highest amount of traffic.

Question 1:  
How do I make nginx split the traffic equally between the 3 apache  
servers?

Question 2:  
How do I make nginx not send traffic to an apache server that is down?

Please help

Thanks

Posted at Nginx Forum:

> **[nginx upstream problem](https://forum.nginx.org/read.php?2,74663,74663#msg-74663)**
>
> Hi, I have several problems when using nginx as a load balancer. Nginx 0.8.35 CentOS 5.4 64bit Kernel 2.6.28.10 Configs: /etc/nginx/nginx.conf \[code\] user nobody; worker\_processes 32; worker\_rlimit\_nofile

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 12:06am UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/2 "2010-04-14T00:06:39Z")

</div>

Hello!

On Tue, Apr 13, 2010 at 04:28:35PM -0400, izghitu wrote:

[…]

> worker\_processes 32;

This looks a bit too many.

[…]

> ```
> include /etc/nginx/conf.d/*.conf;
> 
> ```

Do you have only one file there? Note that upstream{} blocks are  
all defined in http section, and having two with identical names  
are likely to cause confusion.

[…]

> upstream cloud {  
> server apache3 max\_fails=1 fail\_timeout=5;  
> server apache2 max\_fails=1 fail\_timeout=5;  
> server apache1 max\_fails=1 fail\_timeout=5;  
> }

[…]

> ```
> proxy_read_timeout 30;
> proxy_next_upstream error timeout;
> proxy_pass http://cloud;
> 
> ```

[…]

> Problem 1:  
> Regardless of the order of the apache servers in the upstream,  
> apache3 most of the time gets the lowest traffic. Sometimes it  
> gets the same, sometimes it gets very few traffic.

Does these servers (apache1, apache2, apache3) actually resolve  
to distinct ip addresses?

> Problem 2:  
> If I take down one apache server(halt or /etc/init.d/network  
> stop) then I can see from the error log of nginx that it is  
> still sending traffic to that apache server and I see time out  
> or no route to host errors. When browsing the website I get  
> waiting pages all the time which proves it tries to send to the  
> apache server that is down.

nginx marks server as “down” after “max\_fails” failures in  
“fail\_timeout” seconds, and re-enables it again after  
“fail\_timeout” seconds.

As it takes 60 seconds (default proxy\_connect\_timeout) to detect  
that server is down, and you switch it off only for 5 seconds -  
it’s somewhat expected that nginx will try to use dead server most  
of the time.

Also note that statuses of upstream servers are tracked  
independantly by each worker (so with 32 workers you are likely to  
see dead server used all the time by different workers).

You may tune nginx to behave a bit better by setting bigger  
fail\_timeout and/or smaller proxy\_connect\_timeout.

[…]

Maxim D.

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 12:19am UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/3 "2010-04-14T00:19:32Z")

</div>

Hi,

> > ```
> > include /etc/nginx/conf.d/*.conf;
> > 
> > ```
> 
> Do you have only one file there? Note that  
> upstream{} blocks are  
> all defined in http section, and having two with  
> identical names  
> are likely to cause confusion.

I have 2 upstreams, one for http and one for https, both have different  
names

> Does these servers (apache1, apache2, apache3)  
> actually resolve  
> to distinct ip addresses?  
> Yes, they do resolve to distinct IP addresses. I used those names to  
> hide the IPs.

> 

> it’s somewhat expected that nginx will try to use  
> You may tune nginx to behave a bit better by  
> setting bigger  
> fail\_timeout and/or smaller proxy\_connect\_timeout.

Thanks for the info. I will try that and see how it goes.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,74663,74698#msg-74698](http://forum.nginx.org/read.php?2,74663,74698#msg-74698)

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 6:28pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/4 "2010-04-14T18:28:24Z")

</div>

Hi,

Ok, the failover problem is solved after I applied Maxim’s suggestions.

Now the current problem is that nginx does not load balance equally  
between servers.

Under heavy load(300 clients) I killed 2 apache servers so apache3 was  
taking all the load. I put back online apache2 and now for some reason  
apache3 was getting few hits and most of the load was going to apache2.  
When I put back online apache1, it got most of the load, apache2 taking  
less load and apache3 almost no load at all.

Why the strange behavior? How do I make it load balance equally between  
the 3 apache servers? All the apache servers have identical hardware and  
software configuration.

Please help

Posted at Nginx Forum:

> **[Re: nginx upstream problem](https://forum.nginx.org/read.php?2,74663,74884#msg-74884)**
>
> Hi, I have several problems when using nginx as a load balancer. Nginx 0.8.35 CentOS 5.4 64bit Kernel 2.6.28.10 Configs: /etc/nginx/nginx.conf \[code\] user nobody; worker\_processes 32; worker\_rlimit\_nofile

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 5:50pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/5 "2010-04-14T17:50:56Z")

</div>

Hi,

I’ve changed the weight of the 3 apache servers as follows:  
apache3 weight=1  
apache2 weight=2  
apache1 weight=3

When I ran the load testing(300 clients) this time, apache3 was still  
getting the lowest load, apache2 the highest and apache1 was getting  
less then apache2 but much more then apache3

Any ideas?

Posted at Nginx Forum:

> **[Re: nginx upstream problem](https://forum.nginx.org/read.php?2,74663,74925#msg-74925)**
>
> Hi, I have several problems when using nginx as a load balancer. Nginx 0.8.35 CentOS 5.4 64bit Kernel 2.6.28.10 Configs: /etc/nginx/nginx.conf \[code\] user nobody; worker\_processes 32; worker\_rlimit\_nofile

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 7:21pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/6 "2010-04-14T19:21:41Z")

</div>

Hello!

On Wed, Apr 14, 2010 at 12:27:33PM -0400, izghitu wrote:

> now for some reason apache3 was getting few hits and most of the  
> load was going to apache2. When I put back online apache1, it  
> got most of the load, apache2 taking less load and apache3  
> almost no load at all.
> 
> Why the strange behavior? How do I make it load balance equally  
> between the 3 apache servers? All the apache servers have  
> identical hardware and software configuration.

Please do the following:

1. make sure you compiled nginx without third party modules and  
patches, and show nginx -V output;

2. show your config;

3. show access logs with $upstream\_addr logged to make it clear  
that requests are indeed distributed unequally (note that with  
with 32 workers and 3 backends you have to provide at least 32 \* 3  
lines).

Maxim D.

---

<div class="post-metadata">

**Author:** ![izghitu](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/cc9497/32.png) [@izghitu](https://www.ruby-forum.com/u/izghitu)\
**Post date:** [April 14, 2010, 6:59pm UTC](https://www.ruby-forum.com/t/nginx-upstream-problem/186894/7 "2010-04-14T18:59:26Z")

</div>

> How do I make it load balance equally between the 3 apache servers? All the apache servers have identical hardware and software  
> configuration.

Maybe try the upstream fair balancer?

[http://nginx.localdomain.pl/wiki/UpstreamFair](http://nginx.localdomain.pl/wiki/UpstreamFair)

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

“The main feature of upstream\_fair is that it knows how many requests  
each backend is processing (a backend is simply one of the  
servers, among which the load balancer has to make its choice). Thus it  
can make a more informed scheduling decision and avoid  
sending further requests to already busy backends.”

.. as simple roundrobin doesnt always give a consistent result in a  
given period of time.

rr