Dynamic upstream configuration

I’m new to this mailing list, but have used nginx for simple websites
for years. Now, I am planning a more complicated website which will
require multiple upstream servers and more dynamic reconfiguration
events.

I see that there are several third party health check modules available,
but I would like to use only “officially” supported nginx modules. I
would like to configure/reconfigure my upstream servers dynamically
using a health check module (or upgraded upstream module).

Rather than editing the nginx conf file and reloading for upstream
reconfiguration, I’d to do it through health checks to each upstream
server. Initially, my nginx conf file would define all possible backend
servers and mark them as DOWN:

upstream backend {
ip_hash;
server backend1.example.com:8000 DOWN;
server backend1.example.com:8001 DOWN;
server backend2.example.com:8000 DOWN;
server backend2.example.com:8001 DOWN;
. . .
server backendN.example.com:8000 DOWN;
server backendN.example.com:8001 DOWN;
}

Then, I would like to have nginx do health check to each downed backend
once every 5 minutes (or configurable interval). If the health check
times out, the backend stays down. If the health check returns “UP”,
the server is marked up. If check returns “STARTING”, the health check
will be made again in a minute (or configurable interval). If check
returns “STOPPING” or fails a configurable number of checks (like
implemented now), the server is marked down. An UPed server should be
checked frequently (every couple seconds) for a status change and a
DOWNed server should be checked less frequently.

As requested in my first post to this list, I also want the server to
have a MAXCONN attribute and “first” load balancing. I would like the
health checks to also be able to dynamically change the MAXCONN setting
which would be checked before sending new requests to the backend. This
way, the backend can “throttle” itself, by only allowing a few
connections on startup to warm up the server and gradually increase the
potential number of requests that it can handle. The load balancing
algorithms would need to be adjusted to honor the dynamic MAXCONN
attribute when deciding to send a request to the upstream server.

I only need HTTP health checks supported and the status UP, STARTING,
STOPPING, and DOWN could be http status codes instead of text.

I like using health checks in this manner since the backend server has
input into when and how many requests it can server. And, I don’t have
to have a backend server edit the load balancer’s configuration file and
reload nginx when I want to take the server offline for maintenance or
to deploy a new version of the backend app (new app can start listening
on port 8001 for requests while old app can mark the port 8000 server
DOWN and finish handling current requests).