# Achieving strong client -\> upstrem\_server affinity

**URL:** <https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506>\
**Category:** NGINX\
**Created:** [August 30, 2012, 8:28pm UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506 "2012-08-30T20:28:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\
**Post date:** [August 30, 2012, 8:28pm UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/1 "2012-08-30T20:28:23Z")

</div>

Hi,

I am using Nginx 1.1.19 on Ubuntu 12.04 (LTS) server.

Nginx is used to load balance traffic between two instances of the  
Punjab XMPP-BOSH server.

Below is the relevant part from my nginx configuration :

upstream chat\_bosh {  
ip\_hash;  
server 10.98.29.135:5280;  
server 10.98.29.135:5281;  
}

server {  
…  
…  
location /http-bind {  
proxy\_next\_upstream off;  
proxy\_pass http://chat\_bosh;  
expires off;  
}  
}

I am using ip\_hash to make sure that a client will always be served by  
the same upstream server. This is essential.  
I am using “proxy\_next\_upstream off;” to prevent a request being tried  
on multiple upstream servers, because such requests will invariably  
fail. I realize that this will cost me redundancy and fallback in case  
a particular upstream server goes down, but that isn’t useful in this  
particular case. I plan to handle it separately via monitoring and  
alerts.

Anomalies :

1. Despite ip\_hash being specified the request from a particular  
client IP sometimes (close to 7% of requests) get routed to a second  
upstream server
2. Despute proxy\_next\_upstream off; some requests (about 5%) are tried  
over multiple upstream servers.

What could be causing these and how do I go about fixing these?

Here are two sets of log line captures which depict the above  
mentioned problems.

[http://pastebin.com/vnEHQBxK](http://pastebin.com/vnEHQBxK) - upstrem\_next\_server on; (i.e. default  
value)  
[http://pastebin.com/vvPBsPgT](http://pastebin.com/vvPBsPgT) - - upstrem\_next\_server off; (as specified  
above)

These log lines were created with this command : tail -F  
/var/log/nginx/access.log | grep “POST /http-bind” | awk ‘{print $1  
“|” $3 “|” $8 “|” $14}’

$1 = $remote\_addr  
$2 = $upstream\_addr  
$3 = $msec  
$4 = $status

To see the ip\_hash anamoly search for “|404” and look at the adjacent  
lines. The same $remote\_addr will be forwarded to two different  
upstream servers.  
To see the upstream\_next\_server off; anamoly search for “HTTP” -  
Because of my brittle awk statement the status is replace by the  
string “HTTP/1.1” when upstream\_addr has multiple addresses.

–  
Regards,  
Srirang G Doddihal  
Brahmana.

The LIGHT shows the way.  
The WISE see it.  
The BRAVE walk it.  
The PERSISTENT endure and complete it.

I want to do it all ALONE.

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [August 30, 2012, 11:23pm UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/2 "2012-08-30T23:23:04Z")

</div>

> I am using ip\_hash to make sure that a client will always be served by  
> the same upstream server. This is essential.

> 1. Despite ip\_hash being specified the request from a particular  
> client IP sometimes (close to 7% of requests) get routed to a second  
> upstream server

> 1. Despute proxy\_next\_upstream off; some requests (about 5%) are tried  
> over multiple upstream servers.

> how do I go about fixing these?

With a little help from perl 🙂  
Check out this example: [2124034’s gists · GitHub](https://gist.github.com/2124034)  
It decides which upstream to use by hashing $r-\>uri, but you can  
replace it with $r-\>remote\_addr

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [August 31, 2012, 9:59am UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/3 "2012-08-31T09:59:19Z")

</div>

Hi,

On Fri, Aug 31, 2012 at 2:52 AM, Alexandr G. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> > how do I go about fixing these?
> 
> With a little help from perl 🙂  
> Check out this example: [2124034’s gists · GitHub](https://gist.github.com/2124034)  
> It decides which upstream to use by hashing $r-\>uri, but you can  
> replace it with $r-\>remote\_addr

Perl.. ouch.!!. 🙂

I was hoping it wouldn’t come to that.

So does that mean ip\_hash works on a best effort basis and doesn’t  
always ensure that requests from a particular remote\_addr go the same  
upstream server?

> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

–  
Regards,  
Srirang G Doddihal  
Brahmana.

The LIGHT shows the way.  
The WISE see it.  
The BRAVE walk it.  
The PERSISTENT endure and complete it.

I want to do it all ALONE.

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [August 31, 2012, 10:55am UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/4 "2012-08-31T10:55:25Z")

</div>

Hello!

On Thu, Aug 30, 2012 at 11:58:02PM +0530, Srirang Doddihal wrote:

> ```
> ip_hash;
> expires off;
> 
> ```
> 
> alerts.  
> In addition to “proxy\_next\_upstream off” you should at least  
> specify max\_fails=0 for upstream servers. Else a server might  
> be considered down and request from a client will be re-hashed to  
> another one.

Note that docs might be a bit misleading here as they say one  
should refer to proxy\_next\_upstream setting to see what is  
considered to be server failure. This isn’t exactly true: if  
upstream server fails to return correct http answer (i.e. on  
error, timeout, invalid\_header in proxy\_next\_upstream terms) the  
failure is always counted. What can be considered to be failure  
or not is valid http responses, i.e. http\_500 and so on.

> Anomalies :
> 
> 1. Despite ip\_hash being specified the request from a particular  
> client IP sometimes (close to 7% of requests) get routed to a second  
> upstream server

This is correct as long as upstream servers fail and you don’t use  
max\_fails=0. See above.

> 1. Despute proxy\_next\_upstream off; some requests (about 5%) are tried  
> over multiple upstream servers.

This is strange, and you may want to provide more info, see  
[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Debugging).

I would suggest this is likely some configuration error though  
(requests are handled in another location, without  
proxy\_next\_upstream set to off?).

Maxim D.

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [August 31, 2012, 4:29pm UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/5 "2012-08-31T16:29:10Z")

</div>

Hello Maxim,

Thank you very much for the detailed explanation. Things are much  
clearer now.

On Fri, Aug 31, 2012 at 2:25 PM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> > …  
> > on multiple upstream servers, because such requests will invariably  
> > fail. I realize that this will cost me redundancy and fallback in case  
> > a particular upstream server goes down, but that isn’t useful in this  
> > particular case. I plan to handle it separately via monitoring and  
> > alerts.
> 
> In addition to “proxy\_next\_upstream off” you should at least  
> specify max\_fails=0 for upstream servers. Else a server might  
> be considered down and request from a client will be re-hashed to  
> another one.

Got it. How about the following scenario :

Request - 1] “client-1” is forwarded to “server-1”.  
Request - 2] “server-1” does not respond properly and hence is  
considered down. “client-1” gets an error message  
Request - 3] “client-1” is now hashed to “server-2” and is forwarded  
to “server-2”  
Request - 4] Now will “client-1” continue to be forwarded to  
“server-2” or will it be come back to “server-1”?

i.e Whether the re-hash is permanent or a temporary?

> Note that docs might be a bit misleading here as they say one  
> should refer to proxy\_next\_upstream setting to see what is  
> considered to be server failure. This isn’t exactly true: if  
> upstream server fails to return correct http answer (i.e. on  
> error, timeout, invalid\_header in proxy\_next\_upstream terms) the  
> failure is always counted.

Understood till here.

> What can be considered to be failure  
> or not is valid http responses, i.e. http\_500 and so on.

This was confusing. Are you saying that any only HTTP 1xx, 2xx or 3xx  
responses from the upstream server will not count towards failure  
count and any 4xx or 5xx responses will be considered as failures?

> > over multiple upstream servers.
> 
> This is strange, and you may want to provide more info, see  
> [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Debugging).

I will try to get a debug log. Currently I am using the Ubuntu  
package. I probably will have do a custom build for this.

> I would suggest this is likely some configuration error though  
> (requests are handled in another location, without  
> proxy\_next\_upstream set to off?).

All requests to the concerned upstream servers are sent from only one  
location and that location has proxy\_next\_upstream set to off.

I am setting up a test environment to isolate this issue. I will get  
back with more details a little later.  
Is there anything specific that you want to capture?

> Maxim D.
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

–  
Regards,  
Srirang G Doddihal  
Brahmana.

The LIGHT shows the way.  
The WISE see it.  
The BRAVE walk it.  
The PERSISTENT endure and complete it.

I want to do it all ALONE.

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [September 2, 2012, 8:57pm UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/6 "2012-09-02T20:57:45Z")

</div>

Hi Maxim,

Thanks for the explanation. I understand “upstream” and “proxy” much  
better now.

[…]

> > All requests to the concerned upstream servers are sent from only one  
> > location and that location has proxy\_next\_upstream set to off.
> 
> Anything like “keepalive” ommitted in the upstream block snippet  
> posted for clarity?

Nope. The “upstream” and “location /http-bind” blocks are posted  
verbatim. Nothing is removed or changed in those two blocks.

> The only case in the code which allows request to be passed to  
> multiple upstream servers with proxy\_next\_upstream set to off is a  
> failed request in a cached connection (this case isn’t quite  
> correct either, but it’s (a) known issue and (b) rather uncommon  
> under normal conditions).

Since I am not using “keepalive” for my upstream servers I am guessing  
we can safely rule this out.

Lets see if something comes out in the test setup.

> Once you’ll set a test environment you may also want to check if  
> the problem is still reproduceable with latest nginx. As  
> far as I understand Ubuntu currently ships nginx 1.1.19, which  
> isn’t as old as it can be keeping in mind Ubuntu is Debian  
> derivative ;), but there were several important changes in  
> upstream infrastructure since then. While I don’t recall anything  
> which might affect “proxy\_next\_upstream off” case, it’s always  
> good idea to test latest version.

Cool. Will try out the latest version too.

> Maxim D.
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

–  
Regards,  
Srirang G Doddihal  
Brahmana.

The LIGHT shows the way.  
The WISE see it.  
The BRAVE walk it.  
The PERSISTENT endure and complete it.

I want to do it all ALONE.

---

<div class="post-metadata">

**Author:** ![Srirang\_Doddihal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/f4b2a3/32.png) [@Srirang\_Doddihal](https://www.ruby-forum.com/u/Srirang_Doddihal)\
**Post date:** [September 1, 2012, 2:20am UTC](https://www.ruby-forum.com/t/achieving-strong-client-upstrem-server-affinity/222506/7 "2012-09-01T02:20:26Z")

</div>

Hello!

On Fri, Aug 31, 2012 at 07:58:52PM +0530, Srirang Doddihal wrote:

[…]

> Request - 3] “client-1” is now hashed to “server-2” and is forwarded  
> to “server-2”  
> Request - 4] Now will “client-1” continue to be forwarded to  
> “server-2” or will it be come back to “server-1”?
> 
> i.e Whether the re-hash is permanent or a temporary?

The only information kept is upstream server status. As soon as  
it will be considered alive again (after fail\_timeout) - client-1  
will be again forwarded to server-1.

> > or not is valid http responses, i.e. http\_500 and so on.
> 
> This was confusing. Are you saying that any only HTTP 1xx, 2xx or 3xx  
> responses from the upstream server will not count towards failure  
> count and any 4xx or 5xx responses will be considered as failures?

No. The proxy\_next\_upstream directive have the following valid  
values ([Module ngx\_http\_proxy\_module](http://nginx.org/r/proxy_next_upstream)):

: error  
: timeout  
: invalid\_header  
: http\_500  
: http\_502  
: http\_503  
: http\_504  
: http\_404  
: off

By default valid http response regardless of it’s status code is  
just a valid http response and it isn’t counted as a failure.

But if you’ll write “http\_500” in proxy\_next\_upstream, then  
(perfectly valid and in many cases expected) HTTP response with  
status code 500 will be counted as a server failure. (And nginx  
will thow away the response and will try ask another upstream  
server for a response.)

The same applies to http\_502, http\_503, http\_504.

[…]

> All requests to the concerned upstream servers are sent from only one  
> location and that location has proxy\_next\_upstream set to off.

Anything like “keepalive” ommitted in the upstream block snippet  
posted for clarity?

The only case in the code which allows request to be passed to  
multiple upstream servers with proxy\_next\_upstream set to off is a  
failed request in a cached connection (this case isn’t quite  
correct either, but it’s (a) known issue and (b) rather uncommon  
under normal conditions).

> I am setting up a test environment to isolate this issue. I will get  
> back with more details a little later.  
> Is there anything specific that you want to capture?

Nothing special, just nginx -V (and it’s better to make sure there  
are no 3rd party modules/patches), a minimal full config to  
reproduce the problem, and a debug log. Much like the link above  
suggests.

Once you’ll set a test environment you may also want to check if  
the problem is still reproduceable with latest nginx. As  
far as I understand Ubuntu currently ships nginx 1.1.19, which  
isn’t as old as it can be keeping in mind Ubuntu is Debian  
derivative ;), but there were several important changes in  
upstream infrastructure since then. While I don’t recall anything  
which might affect “proxy\_next\_upstream off” case, it’s always  
good idea to test latest version.

Maxim D.