Achieving strong client -> upstrem_server affinity

Hi,

I am using Nginx 1.1.19 on Ubuntu 12.04 (LTS) server.

Nginx is used to load balance traffic between two instances of the
Punjab XMPP-BOSH server.

Below is the relevant part from my nginx configuration :

upstream chat_bosh {
ip_hash;
server 10.98.29.135:5280;
server 10.98.29.135:5281;
}

server {


location /http-bind {
proxy_next_upstream off;
proxy_pass http://chat_bosh;
expires off;
}
}

I am using ip_hash to make sure that a client will always be served by
the same upstream server. This is essential.
I am using “proxy_next_upstream off;” to prevent a request being tried
on multiple upstream servers, because such requests will invariably
fail. I realize that this will cost me redundancy and fallback in case
a particular upstream server goes down, but that isn’t useful in this
particular case. I plan to handle it separately via monitoring and
alerts.

Anomalies :

  1. Despite ip_hash being specified the request from a particular
    client IP sometimes (close to 7% of requests) get routed to a second
    upstream server
  2. Despute proxy_next_upstream off; some requests (about 5%) are tried
    over multiple upstream servers.

What could be causing these and how do I go about fixing these?

Here are two sets of log line captures which depict the above
mentioned problems.

http://pastebin.com/vnEHQBxK - upstrem_next_server on; (i.e. default
value)
http://pastebin.com/vvPBsPgT - - upstrem_next_server off; (as specified
above)

These log lines were created with this command : tail -F
/var/log/nginx/access.log | grep “POST /http-bind” | awk ‘{print $1
“|” $3 “|” $8 “|” $14}’

$1 = $remote_addr
$2 = $upstream_addr
$3 = $msec
$4 = $status

To see the ip_hash anamoly search for “|404” and look at the adjacent
lines. The same $remote_addr will be forwarded to two different
upstream servers.
To see the upstream_next_server off; anamoly search for “HTTP” -
Because of my brittle awk statement the status is replace by the
string “HTTP/1.1” when upstream_addr has multiple addresses.


Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.

I am using ip_hash to make sure that a client will always be served by
the same upstream server. This is essential.

  1. Despite ip_hash being specified the request from a particular
    client IP sometimes (close to 7% of requests) get routed to a second
    upstream server
  1. Despute proxy_next_upstream off; some requests (about 5%) are tried
    over multiple upstream servers.

how do I go about fixing these?

With a little help from perl :slight_smile:
Check out this example: 2124034’s gists · GitHub
It decides which upstream to use by hashing $r->uri, but you can
replace it with $r->remote_addr

Hi,

On Fri, Aug 31, 2012 at 2:52 AM, Alexandr G. [email protected]
wrote:

how do I go about fixing these?

With a little help from perl :slight_smile:
Check out this example: 2124034’s gists · GitHub
It decides which upstream to use by hashing $r->uri, but you can
replace it with $r->remote_addr

Perl… ouch.!!. :slight_smile:

I was hoping it wouldn’t come to that.

So does that mean ip_hash works on a best effort basis and doesn’t
always ensure that requests from a particular remote_addr go the same
upstream server?


nginx mailing list
[email protected]
nginx Info Page


Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.

Hello!

On Thu, Aug 30, 2012 at 11:58:02PM +0530, Srirang Doddihal wrote:

    ip_hash;
            expires off;

alerts.
In addition to “proxy_next_upstream off” you should at least
specify max_fails=0 for upstream servers. Else a server might
be considered down and request from a client will be re-hashed to
another one.

Note that docs might be a bit misleading here as they say one
should refer to proxy_next_upstream setting to see what is
considered to be server failure. This isn’t exactly true: if
upstream server fails to return correct http answer (i.e. on
error, timeout, invalid_header in proxy_next_upstream terms) the
failure is always counted. What can be considered to be failure
or not is valid http responses, i.e. http_500 and so on.

Anomalies :

  1. Despite ip_hash being specified the request from a particular
    client IP sometimes (close to 7% of requests) get routed to a second
    upstream server

This is correct as long as upstream servers fail and you don’t use
max_fails=0. See above.

  1. Despute proxy_next_upstream off; some requests (about 5%) are tried
    over multiple upstream servers.

This is strange, and you may want to provide more info, see
Debugging | NGINX.

I would suggest this is likely some configuration error though
(requests are handled in another location, without
proxy_next_upstream set to off?).

Maxim D.

Hello Maxim,

Thank you very much for the detailed explanation. Things are much
clearer now.

On Fri, Aug 31, 2012 at 2:25 PM, Maxim D. [email protected]
wrote:


on multiple upstream servers, because such requests will invariably
fail. I realize that this will cost me redundancy and fallback in case
a particular upstream server goes down, but that isn’t useful in this
particular case. I plan to handle it separately via monitoring and
alerts.

In addition to “proxy_next_upstream off” you should at least
specify max_fails=0 for upstream servers. Else a server might
be considered down and request from a client will be re-hashed to
another one.

Got it. How about the following scenario :

Request - 1] “client-1” is forwarded to “server-1”.
Request - 2] “server-1” does not respond properly and hence is
considered down. “client-1” gets an error message
Request - 3] “client-1” is now hashed to “server-2” and is forwarded
to “server-2”
Request - 4] Now will “client-1” continue to be forwarded to
“server-2” or will it be come back to “server-1”?

i.e Whether the re-hash is permanent or a temporary?

Note that docs might be a bit misleading here as they say one
should refer to proxy_next_upstream setting to see what is
considered to be server failure. This isn’t exactly true: if
upstream server fails to return correct http answer (i.e. on
error, timeout, invalid_header in proxy_next_upstream terms) the
failure is always counted.

Understood till here.

What can be considered to be failure
or not is valid http responses, i.e. http_500 and so on.

This was confusing. Are you saying that any only HTTP 1xx, 2xx or 3xx
responses from the upstream server will not count towards failure
count and any 4xx or 5xx responses will be considered as failures?

over multiple upstream servers.

This is strange, and you may want to provide more info, see
Debugging | NGINX.

I will try to get a debug log. Currently I am using the Ubuntu
package. I probably will have do a custom build for this.

I would suggest this is likely some configuration error though
(requests are handled in another location, without
proxy_next_upstream set to off?).

All requests to the concerned upstream servers are sent from only one
location and that location has proxy_next_upstream set to off.

I am setting up a test environment to isolate this issue. I will get
back with more details a little later.
Is there anything specific that you want to capture?

Maxim D.


nginx mailing list
[email protected]
nginx Info Page


Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.

Hi Maxim,

Thanks for the explanation. I understand “upstream” and “proxy” much
better now.

[…]

All requests to the concerned upstream servers are sent from only one
location and that location has proxy_next_upstream set to off.

Anything like “keepalive” ommitted in the upstream block snippet
posted for clarity?

Nope. The “upstream” and “location /http-bind” blocks are posted
verbatim. Nothing is removed or changed in those two blocks.

The only case in the code which allows request to be passed to
multiple upstream servers with proxy_next_upstream set to off is a
failed request in a cached connection (this case isn’t quite
correct either, but it’s (a) known issue and (b) rather uncommon
under normal conditions).

Since I am not using “keepalive” for my upstream servers I am guessing
we can safely rule this out.

Lets see if something comes out in the test setup.

Once you’ll set a test environment you may also want to check if
the problem is still reproduceable with latest nginx. As
far as I understand Ubuntu currently ships nginx 1.1.19, which
isn’t as old as it can be keeping in mind Ubuntu is Debian
derivative ;), but there were several important changes in
upstream infrastructure since then. While I don’t recall anything
which might affect “proxy_next_upstream off” case, it’s always
good idea to test latest version.

Cool. Will try out the latest version too.

Maxim D.


nginx mailing list
[email protected]
nginx Info Page


Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.

Hello!

On Fri, Aug 31, 2012 at 07:58:52PM +0530, Srirang Doddihal wrote:

[…]

Request - 3] “client-1” is now hashed to “server-2” and is forwarded
to “server-2”
Request - 4] Now will “client-1” continue to be forwarded to
“server-2” or will it be come back to “server-1”?

i.e Whether the re-hash is permanent or a temporary?

The only information kept is upstream server status. As soon as
it will be considered alive again (after fail_timeout) - client-1
will be again forwarded to server-1.

or not is valid http responses, i.e. http_500 and so on.

This was confusing. Are you saying that any only HTTP 1xx, 2xx or 3xx
responses from the upstream server will not count towards failure
count and any 4xx or 5xx responses will be considered as failures?

No. The proxy_next_upstream directive have the following valid
values (Module ngx_http_proxy_module):

: error
: timeout
: invalid_header
: http_500
: http_502
: http_503
: http_504
: http_404
: off

By default valid http response regardless of it’s status code is
just a valid http response and it isn’t counted as a failure.

But if you’ll write “http_500” in proxy_next_upstream, then
(perfectly valid and in many cases expected) HTTP response with
status code 500 will be counted as a server failure. (And nginx
will thow away the response and will try ask another upstream
server for a response.)

The same applies to http_502, http_503, http_504.

[…]

All requests to the concerned upstream servers are sent from only one
location and that location has proxy_next_upstream set to off.

Anything like “keepalive” ommitted in the upstream block snippet
posted for clarity?

The only case in the code which allows request to be passed to
multiple upstream servers with proxy_next_upstream set to off is a
failed request in a cached connection (this case isn’t quite
correct either, but it’s (a) known issue and (b) rather uncommon
under normal conditions).

I am setting up a test environment to isolate this issue. I will get
back with more details a little later.
Is there anything specific that you want to capture?

Nothing special, just nginx -V (and it’s better to make sure there
are no 3rd party modules/patches), a minimal full config to
reproduce the problem, and a debug log. Much like the link above
suggests.

Once you’ll set a test environment you may also want to check if
the problem is still reproduceable with latest nginx. As
far as I understand Ubuntu currently ships nginx 1.1.19, which
isn’t as old as it can be keeping in mind Ubuntu is Debian
derivative ;), but there were several important changes in
upstream infrastructure since then. While I don’t recall anything
which might affect “proxy_next_upstream off” case, it’s always
good idea to test latest version.

Maxim D.