# Introducing backend healthchecking plugin

**URL:** <https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240>\
**Category:** NGINX\
**Created:** [February 27, 2010, 2:26am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240 "2010-02-27T02:26:26Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [February 27, 2010, 2:26am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/1 "2010-02-27T02:26:26Z")

</div>

I’ve written a plugin that can health check nginx backends, which  
everyone is free to use. This is similar to the healthchecking features  
that varnish and haproxy support. Here’s a sample config[1] just to  
give you an idea, that uses the upstream\_hash plugin. You can get the  
code here [2] and an example of how to patch upstream\_hash here [3].  
The plugin is actually an optional feature that other upstream plugins,  
like upstream fair or iphash for example, can plug into and use. To use  
it, their code needs to be modified to also check the health of the  
backend.

This plugin is super beta, so please be careful. Feedback/patches  
welcome.

[1]  
upstream test\_upstreams {  
server localhost:11114;  
server localhost:11115;

```
hash $filename;
hash_again 10;
healthcheck_enabled;
healthcheck_delay 1000;
healthcheck_timeout 1000;
healthcheck_failcount 1;
healthcheck_expected 'I_AM_ALIVE';
healthcheck_send "GET /health HTTP/1.1" 'Host: www.mysite.com' 

```

‘Connection: close’;  
}

[2]

> **[GitHub - cep21/healthcheck\_nginx\_upstreams: Health checks upstreams for nginx](https://github.com/cep21/healthcheck_nginx_upstreams)**
>
> Health checks upstreams for nginx. Contribute to cep21/healthcheck\_nginx\_upstreams development by creating an account on GitHub.

[3]

> **[GitHub - cep21/nginx\_upstream\_hash at support\_http\_healthchecks](https://github.com/cep21/nginx_upstream_hash/tree/support_http_healthchecks)**
>
> support\_http\_healthchecks

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 1, 2010, 11:49am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/2 "2010-03-01T11:49:19Z")

</div>

On Feb 26, Jack Lindamood wrote:

> This plugin is super beta, so please be careful. Feedback/patches  
> welcome.

I have been wanting to write something similar from a long time, so  
thanks for getting started.

Does the health check compete with the existing logic to mark an  
upstream as up/down? Here is a scenario:

The real traffic goes to this upstream url “/service/login”. My health  
check url is configured as “/hc”. Now /hc is always available by  
/service/login is thowing up a lot of errors like timeouts, 500 etc.  
etc for a given upstream server. What will the status be eventually  
marked as?

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 8:58am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/3 "2010-03-02T08:58:18Z")

</div>

It’s up to the logic in your upstream. The module just runs the health  
check. Your upstream module will have to be changed to query the status  
of the healthcheck and decide if it should continue anyways, or pick  
another upstream server.

Posted at Nginx Forum:

> **[Re: Introducing backend healthchecking plugin](https://forum.nginx.org/read.php?2,57810,58927#msg-58927)**
>
> I've written a plugin that can health check nginx backends, which everyone is free to use. This is similar to the healthchecking features that varnish and haproxy support. Here's a sample config\[1\] just to give you an idea, that uses the...

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 9:35am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/4 "2010-03-02T09:35:11Z")

</div>

> Did you look at ngx\_supervisord[1]?

Do you mean use only ngx\_supervisord? From what I can tell, it requires  
a separate daemon running on the machine that does the healthcheck,  
which then sends a call to nginx via supervisord to turn servers off or  
on. Is that correct? The goal was to build the checking into nginx so  
that we don’t have to monitor another process, making the system more  
stable. If nginx is running, then you’re guaranteed the health checks  
are running.

Or did you mean call into ngx\_supervisord when a healthcheck fails? I’m  
happy to integrate other APIs if you have some ideas.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,57810,58940#msg-58940](http://forum.nginx.org/read.php?2,57810,58940#msg-58940)

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 10:09am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/5 "2010-03-02T10:09:08Z")

</div>

On Tue, Mar 02, 2010 at 03:34:36AM -0500, cep221 wrote:

> > Did you look at ngx\_supervisord[1]?
> 
> Do you mean use only ngx\_supervisord? From what I can tell, it requires a separate daemon running on the machine that does the healthcheck, which then sends a call to nginx via supervisord to turn servers off or on. Is that correct? The goal was to build the checking into nginx so that we don’t have to monitor another process, making the system more stable. If nginx is running, then you’re guaranteed the health checks are running.
> 
> Or did you mean call into ngx\_supervisord when a healthcheck fails? I’m happy to integrate other APIs if you have some ideas.

I mean using the same API so that patching a balancer for  
ngx\_supervisord support would automatically make it support your  
healthcheck module. This means that healthcheck would get passed a  
callback which should be executed with specified parameters when it  
detects a failed backend.

See:

> <https://github.com/FRiCKLE/ngx_supervisord/blob/master/patches/ngx_http_upstream_round_robin.patch>

(grep for ngx\_http\_upstream\_backend\_monitor).

I don’t expect you to use ngx\_supervisord itself (or supervisord), that  
would indeed be quite pointless 🙂

@Piotr: if the health check plugin adapts to ngx\_supervisord API, I  
guess the API spec could use s/\_supervisord//ig, no? A bunch of #defines  
should suffice.

Best regards,  
Grzegorz N.

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 11:21am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/6 "2010-03-02T11:21:45Z")

</div>

> Do you mean use only ngx\_supervisord? From what I can tell, it requires a  
> separate daemon running on the machine that does the healthcheck, which  
> then sends a call to nginx via supervisord to turn servers off or on. Is  
> that correct?

No, it’s not 😛

Neither ngx\_supervisord nor supervisord do any healt-checking.  
ngx\_supervisord communicates with supervisord (process manager) to  
dynamically start or stop backend servers, depending on the load. Since  
version 1.3 it also supports supervisord-less configuration which  
disables  
all communication with supervisord daemon (so it basically takes backend  
servers out of rotation without the need to reload nginx).

> Or did you mean call into ngx\_supervisord when a healthcheck fails? I’m  
> happy to integrate other APIs if you have some ideas.

Yeah, that’s probably what Grzegorz meant. You would just need to call  
ngx\_supervisord\_execute(uscf, NGX\_SUPERVISORD\_CMD\_STOP, backend\_number,  
NULL) and then all ngx\_supervisord-aware load balancers (upstream\_fair,  
round\_robin & ip\_hash) would automagically stop using failed backend  
until  
you would execute NGX\_SUPERVISORD\_CMD\_START.

Full API spec is available at:

> <https://github.com/FRiCKLE/ngx_supervisord/blob/master/patches/ngx_http_upstream_round_robin.patch>

At the moment one would need to specify “supervisord none;” in order to  
enable supervisord-less configuration, because there is no such call in  
API,  
but I could add this in next release if you would like to use it.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 11:24am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/7 "2010-03-02T11:24:40Z")

</div>

> @Piotr: if the health check plugin adapts to ngx\_supervisord API, I  
> guess the API spec could use s/\_supervisord//ig, no? A bunch of #defines  
> should suffice.

Sure, why not 😉

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 9:06am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/8 "2010-03-02T09:06:30Z")

</div>

Hi,

(piggybacking on Arvind’s mail because I don’t seem to have the original  
post).

On Mon, Mar 01, 2010 at 04:18:32PM +0530, Arvind Jayaprakash wrote:

> > This plugin is super beta, so please be careful. Feedback/patches  
> > welcome.

Did you look at ngx\_supervisord[1]? It serves a roughly similar purpose  
(more direct interaction of load balancers with the outside world) and  
it also requires patches to the load balancer. So maybe we could kill  
two birds with one stone and use the same API.

[1] [FRiCKLE Labs / nginx / ngx\_supervisord](http://labs.frickle.com/nginx_ngx_supervisord/)

Best regards,  
Grzegorz N.

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 7:27pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/9 "2010-03-02T19:27:37Z")

</div>

On Tue, Mar 2, 2010 at 10:17 AM, Arvind Jayaprakash  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> (2) In addition, the health-check module provides an out of band health  
> check mechanism wherein, it periodically polls a specific url and uses  
> the HTTP status/body to determine if an upstream needs to be marked as  
> up or down

+1

This has been something in my mind that will help advance nginx  
further as a load balancing option.

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 7:18pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/10 "2010-03-02T19:18:15Z")

</div>

On Mar 02, Piotr S. wrote:

> enable supervisord-less configuration, because there is no such call in API,  
> but I could add this in next release if you would like to use it.

This sonuds like what I needed. So let me rephrase the problem in its  
entirity:

(1) Under normal circumstances, nginx would use proxy\_next\_upstream in  
conjunction with max\_fails and fail\_timeout (for the rr module) to  
declare an upstream as up or down. This is an inline check since it is  
monitoring real traffic.

(2) In addition, the health-check module provides an out of band health  
check mechanism wherein, it periodically polls a specific url and uses  
the HTTP status/body to determine if an upstream needs to be marked as  
up or down

Both these styles have their own benefits.

The first style keeps track of the health by looking at the response of  
actual requests. This is important since a health check url does not  
automatically indicate the health of your real application.

The second style is needed in cases where we plan to do some maintanence  
activity on an upstream server and want to proactively not send traffic  
to it. Typical example is when you want to push new software, check with  
a couple of requests and see if your app is behaving well and if all  
looks fine, direct traffic to it.

In the absence of priority between the two styles of checking, we could  
end up with a flapping upstream status. The logical priority seems to be  
that #2 wins over #1. So, if the health check url says an upstream  
server is down, no traffic should be sent it way and the health status  
evaluation based on style #1 should be ignored. If the health check  
deems and upstream to be up, then the outcome of #1 is the final status.

So where do we get these features from:

- #1 is provided by the stock upstream modules
- #2 is provided by the health check module
- the ngx\_supervisord module seems have the hooks that will let us  
achieve the prioritization once the health-check module uses this  
feature

To get all of this running, we would need 2 patches on the upstream  
module; one for supervisord and the other for health-check and the  
health-check module itself will have to invoke ngx\_supervisord\_execute  
to mark an upstream as up or down.

I will not have time before this weekend to get started on merging  
these; so if someone gets down to doing it earlier, thanks 🙂

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 2, 2010, 11:11pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/11 "2010-03-02T23:11:15Z")

</div>

> The second style is needed in cases where we plan to do some maintanence  
> activity on an upstream server and want to proactively not send traffic  
> to it. Typical example is when you want to push new software, check with  
> a couple of requests and see if your app is behaving well and if all  
> looks fine, direct traffic to it.

Actually, you don’t need health-checks for that, ngx\_supervisord  
provides  
that functionality out-of-the-box. If you plan on taking servers down  
for  
maintenance, you can use supervisord\_stop and supervisord\_start  
handlers.  
Please take a look at example configuration #2, because it does exactly  
that:  
[http://labs.frickle.com/nginx\_ngx\_supervisord/README](http://labs.frickle.com/nginx_ngx_supervisord/README)

> In the absence of priority between the two styles of checking, we could  
> end up with a flapping upstream status. The logical priority seems to be  
> that #2 wins over #1. So, if the health check url says an upstream  
> server is down, no traffic should be sent it way and the health status  
> evaluation based on style #1 should be ignored. If the health check  
> deems and upstream to be up, then the outcome of #1 is the final status.

In-line checks are considered failures. When you take down server with  
ngx\_supervisord, it’s marked down (same way as if you would add “down”  
in  
your configuration and reload nginx), so it takes priority.

> I will not have time before this weekend to get started on merging  
> these; so if someone gets down to doing it earlier, thanks 🙂

I’ll try to find some time today, can’t promise anything though.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 3, 2010, 3:48am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/12 "2010-03-03T03:48:44Z")

</div>

On Mar 02, Piotr S. wrote:

> that:  
> [http://labs.frickle.com/nginx\_ngx\_supervisord/README](http://labs.frickle.com/nginx_ngx_supervisord/README)

The only philosophical objection I have to this style is having  
something to do on the LB servers to change and upstream status. I  
wanted a facility wherein something can be done on the upstream server  
(rm the health check file) to remove it out of service.

I come from a world wherein any sort of access on the LB servers is  
tightly controlled and the people managing the upstreams (application  
servers) can plan a maintanence without ever involving the LB folks.

Am also not a fan of the supervisord\_start/supervisord\_stop directives  
since managing the security aspects of it becomes a hassle.

I am however a fan of supervisord\_inherit\_backend\_status which is why I  
wanted to integrate it with the health-check plugin 🙂

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 8, 2010, 10:16pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/13 "2010-03-08T22:16:25Z")

</div>

> I’ll try to find some time today, can’t promise anything though.

It took a little more time than expected, but I just pushed modified  
version  
of health-check module into my temporary repository:

> **[GitHub - PiotrSikora/healthcheck\_nginx\_upstreams: Health checks upstreams for nginx](https://github.com/PiotrSikora/healthcheck_nginx_upstreams/)**
>
> Health checks upstreams for nginx

It communicates with ngx\_supervisord via its API, which means that  
servers  
are taken out of the rotation when healthcheck fails.

DISCLAIMER: Health-check module doesn’t work for me at all (checks  
always  
time out and because of that servers are taken out of the rotation by  
ngx\_supervisord), but if it works for you then this modified version  
should  
work as well.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 9, 2010, 6:42pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/14 "2010-03-09T18:42:42Z")

</div>

> DISCLAIMER: Health-check module doesn’t work for me at all

To redeem the module, I checked over your config and it was missing the  
“Host:” header in your healthcheck\_send. Try using something similar to  
the sample config:

```
healthcheck_send "GET /health HTTP/1.1" 'Host: www.ahost.com' 

```

‘Connection: close’;

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,57810,61920#msg-61920](http://forum.nginx.org/read.php?2,57810,61920#msg-61920)

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 3, 2010, 5:09am UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/15 "2010-03-03T05:09:32Z")

</div>

> since managing the security aspects of it becomes a hassle.
> 
> I am however a fan of supervisord\_inherit\_backend\_status which is why I  
> wanted to integrate it with the health-check plugin 🙂

I can see your point.

The question which arises now is:  
Should health-check or any other ngx\_supervisord-aware load balancer be  
able  
to “enable” backend server which was administratively taken out of the  
rotation with “server A.B.C.D down;” in nginx.conf?

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![Jack\_Lindamood](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/45deac/32.png) [@Jack\_Lindamood](https://www.ruby-forum.com/u/Jack_Lindamood)\
**Post date:** [March 9, 2010, 7:48pm UTC](https://www.ruby-forum.com/t/introducing-backend-healthchecking-plugin/184240/16 "2010-03-09T19:48:19Z")

</div>

Hello Jack,

> To redeem the module, I checked over your config and it was missing the  
> “Host:” header in your healthcheck\_send. Try using something similar to  
> the sample config:
> 
> healthcheck\_send “GET /health HTTP/1.1” ‘Host: [www.ahost.com](http://www.ahost.com)’  
> ‘Connection: close’;

No, it wasn’t. Missing header wouldn’t yield timeout. But just for the  
sake  
of it, I tested it with the above line and it didn’t help.

Like I said yesterday, I don’t really have time to fully investigate  
this  
right now, but I’ll try to narrow down / fix the problem over the  
weekend.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>