# Nginx-sticky & nginx\_http\_upstream\_check modules not working together

**URL:** <https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962>\
**Category:** NGINX\
**Created:** [June 26, 2012, 4:15pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962 "2012-06-26T16:15:31Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\
**Post date:** [June 26, 2012, 4:15pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/1 "2012-06-26T16:15:31Z")

</div>

Hi,

We use nginx-sticky-module for session persistence and we planned to use  
nginx\_http\_upstream\_check\_module for to check the upstream service  
health.

Scenario: We expect nginx\_http\_upstream\_check\_module to check the  
service (return 2xx|3xx) and take it out from the upstream pool if the  
“service” is not responsive. And nginx-sticky should respect the  
decision taken by nginx\_http\_upstream and start new session to available  
upstream server. However, the upstream server is up with the service  
port; only the content not being served.

The problem in this scenario, Nginx continue to serve the failed  
upstream server until we shutdown the upstream service port. Note that  
http\_upstream module status page indicates the respective service is  
down upon I glitch the service.

I think nginx-sticky got the previous cookie and found the peer match  
which is the failed server and continue to maintain the previous  
session. If I shutdown the server/port, then sticky tries available  
peers and create a new session with available upstream server.

What we need is nginx-sticky module to respect the decision taken by the  
nginx\_http\_upstream module as the priority is “service” availability not  
just the “server and port” availability. We run jboss on upstream  
servers and the probability is very high the service is malfunction but  
the jboss server is up. These two modules are working fine  
individually.

Here is the configuration I have tried. We really need your support on  
possible configuration options to get this two modules work together.

upstream cluster {  
sticky;  
server 192.168.0.61:8081;  
server 192.168.0.61:8082;  
check interval=3000 rise=2 fall=5 timeout=1000 type=http;  
check\_http\_send “GET /keepalive.html HTTP/1.0\r\n\r\n”;  
check\_http\_expect\_alive http\_2xx http\_3xx;  
}

server {  
listen 80;  
server\_name localhost;

```
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;

```

location /stat {  
check\_status;  
}

location /test {  
proxy\_pass [http://cluster/service1/test.html](http://cluster/service1/test.html);  
error\_log /var/log/nginx/error.log debug;  
}

Here is my nginx version:

nginx version: nginx/1.2.1-1  
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)  
TLS SNI support disabled  
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error.log  
–http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid  
–lock-path=/var/run/nginx.lock  
–http-client-body-temp-path=/var/cache/nginx/client\_temp  
–http-proxy-temp-path=/var/cache/nginx/proxy\_temp  
–http-fastcgi-temp-path=/var/cache/nginx/fastcgi\_temp  
–http-uwsgi-temp-path=/var/cache/nginx/uwsgi\_temp  
–http-scgi-temp-path=/var/cache/nginx/scgi\_temp --user=nginx  
–group=nginx --with-http\_ssl\_module --with-http\_realip\_module  
–with-http\_addition\_module --with-http\_sub\_module  
–with-http\_dav\_module --with-http\_flv\_module --with-http\_mp4\_module  
–with-http\_gzip\_static\_module --with-http\_random\_index\_module  
–with-http\_secure\_link\_module --with-http\_stub\_status\_module  
–with-mail\_ssl\_module --with-file-aio --with-debug --with-cc-opt=‘-O2  
-g -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables’  
–without-http\_uwsgi\_module --without-http\_scgi\_module  
–without-mail\_pop3\_module --without-mail\_imap\_module  
–without-mail\_smtp\_module  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/simpl-ngx\_devel\_kit-24202b4  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-set-misc-nginx-module-87d0ab2  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/mikewest-nginx-static-etags-25bfaf9  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/nginx-sticky-module-1.0  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-memc-nginx-module-8befc56  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-srcache-nginx-module-8df221e  
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/yaoweibin-nginx\_upstream\_check\_module-dfee401

uname -rop  
2.6.18-194.el5PAE i686 GNU/Linux

cat /etc/redhat-release  
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

debug logs:  
service down, service port still up - [http://pastebin.com/S9wJFkmR](http://pastebin.com/S9wJFkmR)  
service down, service port down- [http://pastebin.com/HSjMMs75](http://pastebin.com/HSjMMs75)

Thanks in advance.

Posted at Nginx Forum:

> **[nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,227937#msg-227937)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [June 26, 2012, 4:26pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/2 "2012-06-26T16:26:38Z")

</div>

2012/6/26 n1xman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

> decision taken by nginx\_http\_upstream and start new session to available  
> session. If I shutdown the server/port, then sticky tries available  
> possible configuration options to get this two modules work together.  
> server {  
> location /test {  
> –conf-path=/etc/nginx/nginx.conf  
> –with-http\_dav\_module --with-http\_flv\_module --with-http\_mp4\_module

–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1

> service down, service port down- [http://pastebin.com/HSjMMs75](http://pastebin.com/HSjMMs75)
> 
> Thanks in advance.

Hi,

can you please open a bug report about the sticky module  
([Google Code Archive - Long-term storage for Google Code Project Hosting.](http://code.google.com/p/nginx-sticky-module/)). I’ll look into it (no  
ETA as i’m away for vacations these days).

++ Jerome

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [June 26, 2012, 10:00pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/3 "2012-06-26T22:00:24Z")

</div>

Thanks Jerome,

We will hold the nginx\_http\_upstream\_check module until resolve this.  
This module is really important for us as we use multiple Cometd  
instances successfully. The only issue is sometimes the Cometd service  
is not functional but the jboss port is listing.

[http://code.google.com/p/nginx-sticky-module/issues/detail?id=22](http://code.google.com/p/nginx-sticky-module/issues/detail?id=22)

Thanks in advance

Posted at Nginx Forum:

> **[Re: nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,227959#msg-227959)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [June 28, 2012, 4:48am UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/4 "2012-06-28T04:48:11Z")

</div>

Hi,

I’m the author of the upstream check module. The upstream check only  
support official round robin and ip hash module.

The sticky upstream module need an extra patch to work with this check  
module. See the fair module patch as an example  
([nginx\_upstream\_check\_module/upstream\_fair.patch at master · yaoweibin/nginx\_upstream\_check\_module · GitHub](https://github.com/yaoweibin/nginx_upstream_check_module/blob/master/upstream_fair.patch)).

2012/6/27 n1xman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [June 30, 2012, 12:36pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/5 "2012-06-30T12:36:54Z")

</div>

Hi 姚伟斌,

Thanks for the update and it is noted. However, why we would like to use  
sticky instead of ip\_hash or upstream\_fair as we want to make sure  
persistence browser (using cookie) instead of client connecting IP as  
sometimes they are behind proxies.

Posted at Nginx Forum:

> **[Re: nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,228138#msg-228138)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [July 22, 2012, 8:18am UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/6 "2012-07-22T08:18:19Z")

</div>

Done, I have added the nginx-sticky-module.patch  
([nginx\_upstream\_check\_module/nginx-sticky-module.patch at master · yaoweibin/nginx\_upstream\_check\_module · GitHub](https://github.com/yaoweibin/nginx_upstream_check_module/blob/master/nginx-sticky-module.patch))  
for your. You can enable it like this:

$ svn checkout [http://nginx-sticky-module.googlecode.com/svn/trunk/](http://nginx-sticky-module.googlecode.com/svn/trunk/)  
nginx-sticky-module  
$ cd nginx-sticky-module  
$ patch -p0 \<  
/path/to/nginx\_http\_upstream\_check\_module/nginx-sticky-module.patch  
$ cd /path/to/nginx-1.0.14  
$ ./configure --add-module=/path/to/nginx\_http\_upstream\_check\_module  
–add-module=/path/to/nginx-sticky-module  
$ make  
$ make install

Note that, the nginx-sticky-module also needs the original check.patch.

See the README file for detail.

Thanks.

2012/7/20 jjk77 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [July 20, 2012, 4:39pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/7 "2012-07-20T16:39:25Z")

</div>

So, does anybody did a patch of the sticky module?

@姚伟斌: Any chance you provide a patch?

best regrads  
Jens

Posted at Nginx Forum:

> **[Re: nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,228788#msg-228788)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [August 9, 2012, 10:24am UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/8 "2012-08-09T10:24:48Z")

</div>

Hi 姚伟斌,

We have tested it on test environment and it is working now; and thanks  
to you! 🙂

However, in our test environment we also wanted to test  
nginx\_tcp\_proxy\_module for WebSocket. We have noticed upstream server  
health check and status monitor bundle to the nginx\_tcp\_proxy\_module.

Can you tell if we compile nginx\_tcp\_proxy\_module, we can drop the  
nginx\_http\_upstream\_check\_module; and we still patch the  
nginx-sticky-module with  
nginx\_http\_upstream\_check\_module/nginx-sticky-module.patch and use  
nginx\_tcp\_proxy\_module for upstream health check also…?

BTW: We have tried to compile both nginx\_tcp\_proxy\_module and  
nginx\_http\_upstream\_check\_module together, following error fires during  
the make process.

[hirantha@abmx-test nginx-1.2.1]$ patch -p1 \<  
./contrib/yaoweibin-nginx\_upstream\_check\_module-be97c70/check\_1.2.1.patch  
patching file src/http/modules/ngx\_http\_upstream\_ip\_hash\_module.c  
patching file src/http/ngx\_http\_upstream\_round\_robin.c  
patching file src/http/ngx\_http\_upstream\_round\_robin.h  
[hirantha@abmx-test nginx-1.2.1]$ patch -p1 \<  
./contrib/yaoweibin-nginx\_tcp\_proxy\_module-a40c99a/tcp.patch  
patching file src/core/ngx\_log.c  
Hunk #1 succeeded at 67 (offset 1 line).  
patching file src/core/ngx\_log.h  
Hunk #1 succeeded at 30 (offset 1 line).  
patching file src/event/ngx\_event\_connect.h  
Hunk #1 succeeded at 33 (offset 1 line).  
[hirantha@abmx-test nginx-1.2.1]$ ./configure  
–add-module=./contrib/yaoweibin-nginx\_tcp\_proxy\_module-a40c99a  
–add-module=./contrib/yaoweibin-nginx\_upstream\_check\_module-be97c70  
checking for OS

- Linux 2.6.18-194.el5PAE i686  
checking for C compiler … found  
…  
checking for openat(), fstatat() … found  
configuring additional modules  
adding module in ./contrib/yaoweibin-nginx\_tcp\_proxy\_module-a40c99a  
checking for nginx\_tcp\_module … found
- ngx\_tcp\_module was configured  
adding module in  
./contrib/yaoweibin-nginx\_upstream\_check\_module-be97c70  
checking for ngx\_http\_upstream\_check\_module … found
- ngx\_http\_upstream\_check\_module was configured  
checking for PCRE library … found  
checking for PCRE JIT support … not found  
checking for OpenSSL library … found  
checking for zlib library … found  
creating objs/Makefile

[hirantha@abmx-test nginx-1.2.1]$make  
…  
…  
objs/addon/modules/ngx\_tcp\_ssl\_module.o \

objs/addon/yaoweibin-nginx\_upstream\_check\_module-be97c70/ngx\_http\_upstream\_check\_module.o  
\

objs/addon/yaoweibin-nginx\_upstream\_check\_module-be97c70/ngx\_http\_upstream\_check\_handler.o  
  
objs/ngx\_modules.o   
-lpthread -lcrypt -lpcre -lssl -lcrypto -ldl -lz  
objs/addon/yaoweibin-nginx\_upstream\_check\_module-be97c70/ngx\_http\_upstream\_check\_handler.o:(.rodata+0x40):  
multiple definition of `sslv3_client_hello_pkt' objs/addon/yaoweibin-nginx_tcp_proxy_module-a40c99a/ngx_tcp_upstream_check.o:(.rodata+0x0): first defined here collect2: ld returned 1 exit status make[1]: *** [objs/nginx] Error 1 make[1]: Leaving directory `/usr/local/hirantha/nginx-1.2.1’  
make: \*\*\* [build] Error 2

Thanks in advance.

Posted at Nginx Forum:

> **[Re: nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,229498#msg-229498)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [August 13, 2012, 5:53am UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/9 "2012-08-13T05:53:36Z")

</div>

Hi 姚伟斌,

With the latest code, I was managed to compile both modules together  
without any issue.

Thanks for the modules and great support. 🙂

Posted at Nginx Forum:

> **[Re: nginx-sticky & nginx\_http\_upstream\_check modules not working together](https://forum.nginx.org/read.php?2,227937,229615#msg-229615)**
>
> Hi, We use nginx-sticky-module for session persistence and we planned to use nginx\_http\_upstream\_check\_module for to check the upstream service health. Scenario: We expect nginx\_http\_upstream\_check\_module to check the service (return 2xx|3xx) and...

---

<div class="post-metadata">

**Author:** ![n1xman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/13edae/32.png) [@n1xman](https://www.ruby-forum.com/u/n1xman)\
**Post date:** [August 9, 2012, 2:30pm UTC](https://www.ruby-forum.com/t/nginx-sticky-nginx-http-upstream-check-modules-not-working-together/220962/10 "2012-08-09T14:30:50Z")

</div>

Hi,

Try the latest code, you can use these modules both.

Thanks.

2012/8/9 n1xman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):
