Forum: NGINX nginx-sticky & nginx_http_upstream_check modules not working together

Hi,

We use nginx-sticky-module for session persistence and we planned to use
nginx_http_upstream_check_module for to check the upstream service
health.

Scenario: We expect nginx_http_upstream_check_module to check the
service (return 2xx|3xx) and take it out from the upstream pool if the
"service" is not responsive. And nginx-sticky should respect the
decision taken by nginx_http_upstream and start new session to available
upstream server. However, the upstream server is up with the service
port; only the content not being served.

The problem in this scenario, Nginx continue to serve the failed
upstream server until we shutdown the upstream service port. Note that
http_upstream module status page indicates the respective service is
down upon I glitch the service.

I think nginx-sticky got the previous cookie and found the peer match
which is the failed server and continue to maintain the previous
session. If I shutdown the server/port, then sticky tries available
peers and create a new session with available upstream server.

What we need is nginx-sticky module to respect the decision taken by the
nginx_http_upstream module as the priority is "service" availability not
just the "server and port" availability. We run jboss on upstream
servers and the probability is very high the service is malfunction but
the jboss server is up. These two modules are working fine
individually.

Here is the configuration I have tried. We really need your support on
possible configuration options to get this two modules work together.

upstream cluster {
  sticky;
  server 192.168.0.61:8081;
  server 192.168.0.61:8082;
        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_http_send "GET /keepalive.html HTTP/1.0\r\n\r\n";
        check_http_expect_alive http_2xx http_3xx;
}

server {
    listen       80;
    server_name  localhost;

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

location /stat {
    check_status;
}

 location /test {
        proxy_pass http://cluster/service1/test.html;
        error_log /var/log/nginx/error.log debug;
        }

Here is my nginx version:

nginx version: nginx/1.2.1-1
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
TLS SNI support disabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-mail_ssl_module --with-file-aio --with-debug --with-cc-opt='-O2
-g -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables'
--without-http_uwsgi_module --without-http_scgi_module
--without-mail_pop3_module --without-mail_imap_module
--without-mail_smtp_module
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/simpl-ngx_devel_kit-24202b4
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-set-misc-nginx-module-87d0ab2
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/mikewest-nginx-static-etags-25bfaf9
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/nginx-sticky-module-1.0
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-memc-nginx-module-8befc56
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-srcache-nginx-module-8df221e
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/yaoweibin-nginx_upstream_check_module-dfee401

uname -rop
2.6.18-194.el5PAE i686 GNU/Linux

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

debug logs:
service down, service port still up - http://pastebin.com/S9wJFkmR
service down, service port down- http://pastebin.com/HSjMMs75

Thanks in advance.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,227937#msg-227937

2012/6/26 n1xman <nginx-forum@nginx.us>
> decision taken by nginx_http_upstream and start new session to available
> session. If I shutdown the server/port, then sticky tries available
> possible configuration options to get this two modules work together.
> server {
> location /test {
> --conf-path=/etc/nginx/nginx.conf
> --with-http_dav_module --with-http_flv_module --with-http_mp4_module
> 
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1
>
> service down, service port down- http://pastebin.com/HSjMMs75
>
> Thanks in advance.

Hi,

can you please open a bug report about the sticky module
(http://code.google.com/p/nginx-sticky-module/). I'll look into it (no
ETA as i'm away for vacations these days).

++ Jerome

Thanks Jerome,

We will hold the nginx_http_upstream_check module until resolve this.
This module is really important for us as we use multiple Cometd
instances successfully. The only issue is sometimes the Cometd service
is not functional but the jboss port is listing.

http://code.google.com/p/nginx-sticky-module/issue...

Thanks in advance

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,227959#msg-227959

Hi,

I'm the author of the upstream check module.  The upstream check only
support official round robin and ip hash module.

The sticky upstream module need an extra patch to work with this check
module. See the fair module patch as an example
(https://github.com/yaoweibin/nginx_upstream_check_...).


2012/6/27 n1xman <nginx-forum@nginx.us>:

Hi 姚伟斌,

Thanks for the update and it is noted. However, why we would like to use
sticky instead of ip_hash or upstream_fair as we want to make sure
persistence browser (using cookie) instead of client connecting IP as
sometimes they are behind proxies.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,228138#msg-228138

So, does anybody did a patch of the sticky module?

@姚伟斌: Any chance you provide a patch?

best regrads
  Jens

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,228788#msg-228788

Done, I have added the nginx-sticky-module.patch
(https://github.com/yaoweibin/nginx_upstream_check_...)
for your. You can enable it like this:

$ svn checkout http://nginx-sticky-module.googlecode.com/svn/trunk/
nginx-sticky-module
$ cd nginx-sticky-module
$ patch -p0 < 
/path/to/nginx_http_upstream_check_module/nginx-sticky-module.patch
$ cd /path/to/nginx-1.0.14
$ ./configure --add-module=/path/to/nginx_http_upstream_check_module
--add-module=/path/to/nginx-sticky-module
$ make
$ make install

 Note that, the nginx-sticky-module also needs the original check.patch.

See the README file for detail.

Thanks.

2012/7/20 jjk77 <nginx-forum@nginx.us>:

Hi 姚伟斌,

We have tested it on test environment and it is working now; and thanks
to you! :)

However, in our test environment we also wanted to test
nginx_tcp_proxy_module for WebSocket. We have noticed upstream server
health check and  status monitor bundle to the  nginx_tcp_proxy_module.

Can you tell if we compile nginx_tcp_proxy_module, we can drop the
nginx_http_upstream_check_module; and we still patch the
nginx-sticky-module with
nginx_http_upstream_check_module/nginx-sticky-module.patch and use
nginx_tcp_proxy_module for upstream health check also..?

BTW: We have tried to compile both nginx_tcp_proxy_module and
nginx_http_upstream_check_module together, following error fires during
the make process.

[hirantha@abmx-test nginx-1.2.1]$ patch -p1 <
./contrib/yaoweibin-nginx_upstream_check_module-be97c70/check_1.2.1.patch
patching file src/http/modules/ngx_http_upstream_ip_hash_module.c
patching file src/http/ngx_http_upstream_round_robin.c
patching file src/http/ngx_http_upstream_round_robin.h
[hirantha@abmx-test nginx-1.2.1]$ patch -p1 <
./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a/tcp.patch
patching file src/core/ngx_log.c
Hunk #1 succeeded at 67 (offset 1 line).
patching file src/core/ngx_log.h
Hunk #1 succeeded at 30 (offset 1 line).
patching file src/event/ngx_event_connect.h
Hunk #1 succeeded at 33 (offset 1 line).
[hirantha@abmx-test nginx-1.2.1]$ ./configure
--add-module=./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a
--add-module=./contrib/yaoweibin-nginx_upstream_check_module-be97c70
checking for OS
 + Linux 2.6.18-194.el5PAE i686
checking for C compiler ... found
..
checking for openat(), fstatat() ... found
configuring additional modules
adding module in ./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a
checking for nginx_tcp_module ... found
 + ngx_tcp_module was configured
adding module in
./contrib/yaoweibin-nginx_upstream_check_module-be97c70
checking for ngx_http_upstream_check_module ... found
 + ngx_http_upstream_check_module was configured
checking for PCRE library ... found
checking for PCRE JIT support ... not found
checking for OpenSSL library ... found
checking for zlib library ... found
creating objs/Makefile

[hirantha@abmx-test nginx-1.2.1]$make
...
...
        objs/addon/modules/ngx_tcp_ssl_module.o \

objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_module.o
\

objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_handler.o
\
        objs/ngx_modules.o \
        -lpthread -lcrypt -lpcre -lssl -lcrypto -ldl -lz
objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_handler.o:(.rodata+0x40):
multiple definition of `sslv3_client_hello_pkt'
objs/addon/yaoweibin-nginx_tcp_proxy_module-a40c99a/ngx_tcp_upstream_check.o:(.rodata+0x0):
first defined here
collect2: ld returned 1 exit status
make[1]: *** [objs/nginx] Error 1
make[1]: Leaving directory `/usr/local/hirantha/nginx-1.2.1'
make: *** [build] Error 2

Thanks in advance.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,229498#msg-229498

Hi,

Try the latest code, you can use these modules both.

Thanks.

2012/8/9 n1xman <nginx-forum@nginx.us>:

Hi 姚伟斌,

With the latest code, I was managed to compile both modules together
without any issue.

Thanks for the modules and great support. :)

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,227937,229615#msg-229615