Best way to handle dynamic upstream backend?

Hi,

I use nginx as a reverse proxy in front of about 20 apaches. Now in my
backend, currently defined as

upstream backend_all {
server 1.1.1.1;
server 1.1.1.2;

server 1.1.1.20;
}

i would like to include dynamic information about the health of the
backend nodes.
If a backend server is down, it should be marked as down automatically.
Or if a server is specifically busy, it should get less used as a
backend.

I could implement a scrip which checks the backend and then generates
the upstream definition accordingly. I would then have to reload nginx.
(Would that generate timeouts for my users? - reload takes a few (aprox
10) seconds as there are a lot of websites.). This seems not like the
best solution to me, as I feel that nginx might have build in solution
for this (?).

Also I have not only one nginx server, but four, so think I need to take
care that these four distribute evenly to the backend. For this I was
thinking of ip_hash.

Any suggestions on how to handle this?

Isaac

Hello!

On Fri, Sep 21, 2012 at 04:08:52PM +0200, Isaac H. wrote:

}

i would like to include dynamic information about the health of the
backend nodes.
If a backend server is down, it should be marked as down
automatically. Or if a server is specifically busy, it should get
less used as a backend.

Normal nginx aproach is to mark backends down atomatically based
on max_fails/fail_timeout, see here for detail:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server

It is proven to work well for handling server outages and gives
system administrators enough time to either revive dead
server or remove it from configuration.

To atomatically accomodate for unequal server load/speed (due to
external factors) I would recommend using least_conn balancer, see
here for details:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#least_conn

It should also help to better detect outages when requests start
taking a long time by routing less requests to a faulty server
before nginx will actually able to see timeouts.

I could implement a scrip which checks the backend and then
generates the upstream definition accordingly. I would then have to
reload nginx. (Would that generate timeouts for my users? - reload
takes a few (aprox 10) seconds as there are a lot of websites.).

Configuration reload is seamless - old worker processes continue
to serve clients while master process reads new configuration and
spawns new worker processes.

It consumes resources though, as it spawns additional set of
worker processes, and it takes some time for old workers to
shutdown as they have to serve requests which are already in
progress. Doing configuration reloads at high frequency might not
be a good idea, especially if there are long running requests.

Summing up the above - it’s ok to do reloads as long as they are
initiated by hand and/or relatively rare, but you may need some
safeguards for automatic scripts to make sure they aren’t run too
frequently.

This seems not like the best solution to me, as I feel that nginx
might have build in solution for this (?).

See above.

Also I have not only one nginx server, but four, so think I need to
take care that these four distribute evenly to the backend. For this
I was thinking of ip_hash.

As long as each frontend distribute requests evenly to all backend
servers - resulting backend load should be equal. There is no
need for ip_hash.

Maxim D.

Thank you Maxim for your advice!

Isaac

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs