# Nginx load balancing Tomcat servers

**URL:** <https://www.ruby-forum.com/t/nginx-load-balancing-tomcat-servers/217408>\
**Category:** NGINX\
**Created:** [February 29, 2012, 11:39pm UTC](https://www.ruby-forum.com/t/nginx-load-balancing-tomcat-servers/217408 "2012-02-29T23:39:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [February 29, 2012, 11:39pm UTC](https://www.ruby-forum.com/t/nginx-load-balancing-tomcat-servers/217408/1 "2012-02-29T23:39:41Z")

</div>

Hello,

What is the right way to setup a true load balancing of a few Tomcat  
servers?  
By word “true” I understand taking into account at least CPU load of  
those  
nodes, not just round robining.  
I’ve looked at 3rd party modules, but doesn’t look there is the one.

Thank you,  
Ruslan

---

<div class="post-metadata">

**Author:** ![Ruslan\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/48db29/32.png) [@Ruslan\_D](https://www.ruby-forum.com/u/Ruslan_D)\
**Post date:** [March 2, 2012, 7:49pm UTC](https://www.ruby-forum.com/t/nginx-load-balancing-tomcat-servers/217408/2 "2012-03-02T19:49:46Z")

</div>

01 марта 2012, 02:39 от Ruslan D. : \> \> What is the right way to  
setup a true load balancing of a few Tomcat \> servers? \> By word “true”  
I understand taking into account at least CPU load of those \> nodes, not  
just round robining. \> I’ve looked at 3rd party modules, but doesn’t  
look there is the one. Hello, You may want to try my Lua based load  
balancer, it picks the least loaded backend server based on the latest  
load averages that it collects by polling backend servers using parallel  
subrequests. It’s not optimized because I use it mainly for  
benchmarking, but if you spend some time tuning the parameters, you  
should be able to achieve better than standard load balancing  
performance. The following frontend server load balancer requires  
Agentzh’s Lua module: [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpLuaModule) # # NGiNXYZ  
Load Balancer - Frontend Server # Copyright (C) Max # http { # Set up a  
cache zone for storing backend server load averages. proxy\_cache\_path  
/var/tmp/nginx/load\_average\_cache levels=1  
keys\_zone=load\_average\_cache:64k inactive=5s; server { location / { set  
$backend\_server “”; rewrite\_by\_lua ’ --[[ Only for debugging. local  
start\_time = ngx.now() --]] --[[ Replace these entries with your backend  
servers. ]] local backend\_servers = { “192.168.2.1:8080”,  
“192.168.2.2:8080”, “192.168.2.3:8080” } local fallback\_backend\_server =  
“192.168.2.1:8080” local subrequests = {} for i, server in  
ipairs(backend\_servers) do table.insert(subrequests, {  
“/poll\_load\_average/”..server, { method = ngx.HTTP\_GET } }) end local  
responses = { ngx.location.capture\_multi(subrequests) } local  
min\_load\_average = 1024 local min\_index = 0 for i, response in  
ipairs(responses) do local body\_length = #response.body if (body\_length

> 0 and body\_length \< 8) then local load\_average =  
> tonumber(response.body) --[[ Only for debugging. ngx.log(ngx.DEBUG,  
> "backend server ", backend\_servers[i], " load average: ", load\_average)  
> –]] if (load\_average and load\_average \< min\_load\_average) then  
> min\_load\_average = load\_average min\_index = i end end end if (min\_index  
> 0) then ngx.var.backend\_server = backend\_servers[min\_index] else  
> ngx.var.backend\_server = fallback\_backend\_server; end --[[ Only for  
> debugging. ngx.log(ngx.DEBUG, "least loaded backend server ",  
> ngx.var.backend\_server) local total\_time = ngx.now() - start\_time  
> ngx.log(ngx.DEBUG, "poll\_load\_average subrequest completion time: ",  
> total\_time) --]] '; # The least loaded backend server has been  
> determined and # assigned to the $backend\_server variable inside the #  
> rewrite\_by\_lua block. proxy\_pass http://$backend\_server; } location ~  
> ^/poll\_load\_average/(.+)$ { # This is the subrequest location block used  
> by the # rewrite\_by\_lua block to poll backend servers for # the load  
> average. internal; proxy\_pass\_request\_headers off;  
> proxy\_pass\_request\_body off; # Prevent subrequests from taking more time  
> than we allow # them to. If a backend server is too slow to respond, it

# will be skipped automatically. proxy\_connect\_timeout 30ms;

proxy\_send\_timeout 20ms; proxy\_read\_timeout 60ms; # Do not cache load  
averages and do not read them from cache #proxy\_no\_cache 1;  
#proxy\_cache\_bypass 1; # Cache load averages and read them from cache  
proxy\_cache load\_average\_cache; # $1 = backend\_server:port  
proxy\_cache\_key $1; # Poll the backend server for its latest load  
average. proxy\_pass http://$1/load\_average/; } } } You should first  
replace the entries in the backend\_servers array and the  
fallback\_backend\_server string with your backend server address:port  
values. Then you may want to turn debugging on by adding “]]” at the end  
of every "–[[ Only for debugging. " line in order to see what the Lua  
code does: 1) A subrequest for the /poll\_average\_load/ location is  
created for each of the backend servers. 2) All subrequests are  
initiated and executed simultaneously in parallel, so the total time /  
latency equals the completion time of the slowest request.  
ngx.location.capture\_multi() waits for all the subrequests to complete  
and captures their responses (status, header and body). In order to  
prevent a slow subrequest / backend server from delaying the execution  
of the original request, proxy\__timeout directives are used in the  
/poll\_load\_average/ location to skip any backend servers that are too  
slow to respond in time. 3) Subrequest response bodies are validated  
(they may contain only the load average and must be shorter than 8  
characters), and the backend server with the lowest load average is then  
assigned to the $backend\_server nginx variable that had been created  
before the rewrite\_by\_lua block. If none of the backend servers return a  
valid load average, the backend you assigned to fallback\_backend\_server  
is used. 4) The original request is passed on to the least loaded  
backend server by using the proxy\_pass directive. Note that caching can  
be used inside the /poll\_load\_average/ subrequest location block to  
reduce the overhead of constantly polling all of the backend servers,  
but load averages should typically not be cached for more than a few  
seconds. Cache expiration time for load averages should be set  
individually on each of the backend servers in order to optimize  
performance. You should also experiment with the proxy_\_timeout values  
to find optimal values for your setup. The matching backend server load  
balancer configuration for this setup should be configured like this: #

# NGiNXYZ Load Balancer - Backend Server # Copyright (C) Max # http {

Set up a limit request zone to prevent too frequent # load average  
requests. limit\_req\_zone $binary\_remote\_addr  
zone=load\_average\_limit\_request\_zone:256k rate=10r/s; # Extract the last  
minute load average from the /usr/bin/uptime # output and assign it to  
$load\_average. # # The first alarm call sets up a 2 second timer and the  
2nd call # cancels it. If the Perl subroutine is not completed within #  
2 seconds, a return is forced to prevent Perl from holding up # further  
processing. perl\_set $load\_average ’ sub { eval { local $SIG{ALRM} = sub  
{ return “”; }; alarm 2; `/usr/bin/uptime` =~ /averages: (\d+.\d+)/;  
alarm 0; return $1; } } '; server { location /load\_average/ { # Prevent  
this load average from being cached for more # than 2 seconds by the  
frontend server. expires 2s; # Prevent too frequent load average  
requests. limit\_req\_log\_level error; limit\_req  
zone=load\_average\_limit\_request\_zone nodelay; # Return $load\_average  
using Agentzh’s Echo module echo $load\_average; # Note that using  
“return 200 $load\_average;” instead of # “echo $load\_average;” to return  
$load\_average makes # any limit\_request directive useless inside this #  
location block because it prevents the limit\_request # triggered error  
code 503 from being returned. #return 200 $load\_average; } } } The  
backend servers would require the following additional modules for this  
configuration to work: Embedded Perl module  
[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/EmbeddedPerlModule) Agentzh’s Echo module (optional

- see the “return 200” comment above)  
[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpEchoModule) My comments are extensive, so  
there’s nothing to add, except that this could be done using Apache and  
a CGI script that does something like this to extract the load average  
from the uptime output: uptime | awk -F’: |,’ ‘{print $5}’ You could  
also set up a shell script based netcat server to answer load average  
requests, but you’d have to set up additional firewall rules to prevent  
too frequent requests from degrading performance. You could also extend  
the load balancer to include other performance criteria, such as iostat,  
netstat values etc. The Stub Status module statistics could also be  
included: [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpStubStatusModule) Note that if you  
decide to cache load averages on the frontend server, you should always  
make sure that your backend servers always set an appropriate expiration  
time (by using the “Cache-Control: max-age=$x” header, for example) in  
the response (typically, it should be no longer than a few seconds, but  
that really depends on your setup). I hope that helps, any benchmarks  
and comments will be appreciated. Max

---

<div class="post-metadata">

**Author:** ![Ruslan\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/48db29/32.png) [@Ruslan\_D](https://www.ruby-forum.com/u/Ruslan_D)\
**Post date:** [March 2, 2012, 7:56pm UTC](https://www.ruby-forum.com/t/nginx-load-balancing-tomcat-servers/217408/3 "2012-03-02T19:56:33Z")

</div>

Apologies, [mail.ru](http://mail.ru) mangles my posts sometimes, unfortunately  
there is no option to turn HTML off.

01 марта 2012, 02:39 от Ruslan D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> What is the right way to setup a true load balancing of a few Tomcat  
> servers?  
> By word “true” I understand taking into account at least CPU load of those  
> nodes, not just round robining.  
> I’ve looked at 3rd party modules, but doesn’t look there is the one.

Hello,

You may want to try my Lua based load balancer, it picks the least  
loaded  
backend server based on the latest load averages that it collects by  
polling backend servers using parallel subrequests. It’s not optimized  
because I use it mainly for benchmarking, but if you spend some time  
tuning the parameters, you should be able to achieve better than  
standard load balancing performance.

The following frontend server load balancer requires Agentzh’s Lua  
module:

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

# 

# NGiNXYZ Load Balancer - Frontend Server

# Copyright (C) Max \<nginxyz (at) users dot berlios dot de\>

# 

http {

```
# Set up a cache zone for storing backend server load averages.
proxy_cache_path /var/tmp/nginx/load_average_cache
                 levels=1
                 keys_zone=load_average_cache:64k inactive=5s;

server {

    location / {

        set $backend_server "";

        rewrite_by_lua '

            --[[ Only for debugging.
            local start_time = ngx.now()
            --]]

            --[[Replace these entries with your backend servers.]]
            local backend_servers = { "192.168.2.1:8080",
                                      "192.168.2.2:8080",
                                      "192.168.2.3:8080" }

            local fallback_backend_server = "192.168.2.1:8080"

            local subrequests = {}

            for i, server in ipairs(backend_servers) do
                table.insert(subrequests,
                             { "/poll_load_average/"..server,
                               { method = ngx.HTTP_GET } })
            end

            local responses = { 

```

ngx.location.capture\_multi(subrequests) }

```
            local min_load_average = 1024
            local min_index = 0
            for i, response in ipairs(responses) do

                local body_length = #response.body
                if (body_length > 0 and body_length < 8) then
                    local load_average = tonumber(response.body)

                    --[[ Only for debugging.
                    ngx.log(ngx.DEBUG, "backend server ",
                                       backend_servers[i],
                                       " load average: ", 

```

load\_average)  
–]]

```
                    if (load_average
                        and load_average < min_load_average) then

                        min_load_average = load_average
                        min_index = i
                    end
                end
            end

            if (min_index > 0) then
                ngx.var.backend_server = backend_servers[min_index]
            else
                ngx.var.backend_server = fallback_backend_server;
            end

            --[[ Only for debugging.
            ngx.log(ngx.DEBUG, "least loaded backend server ",
                               ngx.var.backend_server)

            local total_time = ngx.now() - start_time

            ngx.log(ngx.DEBUG,
                    "poll_load_average subrequest completion time: 

```

",  
total\_time)  
–]]  
';

```
        # The least loaded backend server has been determined and
        # assigned to the $backend_server variable inside the
        # rewrite_by_lua block.

        proxy_pass http://$backend_server;
    }

    location ~ ^/poll_load_average/(.+)$ {

        # This is the subrequest location block used by the
        # rewrite_by_lua block to poll backend servers for
        # the load average.

        internal;

        proxy_pass_request_headers off;
        proxy_pass_request_body off;

        # Prevent subrequests from taking more time than we allow
        # them to. If a backend server is too slow to respond, it
        # will be skipped automatically.
        proxy_connect_timeout 30ms;
        proxy_send_timeout 20ms;
        proxy_read_timeout 60ms;

        # Do not cache load averages and do not read them from cache
        #proxy_no_cache 1;
        #proxy_cache_bypass 1;

        # Cache load averages and read them from cache
        proxy_cache load_average_cache;
        # $1 = backend_server:port
        proxy_cache_key $1;

        # Poll the backend server for its latest load average.
        proxy_pass http://$1/load_average/;
    }
}

```

}

You should first replace the entries in the backend\_servers array  
and the fallback\_backend\_server string with your backend server  
address:port values.

Then you may want to turn debugging on by adding “]]” at the end of  
every "–[[ Only for debugging. " line in order to see what the  
Lua code does:

1. A subrequest for the /poll\_average\_load/ location is  
created for each of the backend servers.

2. All subrequests are initiated and executed simultaneously in  
parallel, so the total time / latency equals the completion  
time of the slowest request. ngx.location.capture\_multi()  
waits for all the subrequests to complete and captures  
their responses (status, header and body). In order to  
prevent a slow subrequest / backend server from delaying  
the execution of the original request, proxy\_\*\_timeout  
directives are used in the /poll\_load\_average/ location to  
skip any backend servers that are too slow to respond in time.

3. Subrequest response bodies are validated (they may contain  
only the load average and must be shorter than 8 characters),  
and the backend server with the lowest load average is then  
assigned to the $backend\_server nginx variable that had been  
created before the rewrite\_by\_lua block. If none of the backend  
servers return a valid load average, the backend you assigned to  
fallback\_backend\_server is used.

4. The original request is passed on to the least loaded backend  
server by using the proxy\_pass directive.

Note that caching can be used inside the /poll\_load\_average/  
subrequest location block to reduce the overhead of constantly  
polling all of the backend servers, but load averages should  
typically not be cached for more than a few seconds. Cache  
expiration time for load averages should be set individually  
on each of the backend servers in order to optimize performance.

You should also experiment with the proxy\_\*\_timeout values to  
find optimal values for your setup.

The matching backend server load balancer configuration for this  
setup should be configured like this:

# 

# NGiNXYZ Load Balancer - Backend Server

# Copyright (C) Max \<nginxyz (at) users dot berlios dot de\>

# 

http {

```
# Set up a limit request zone to prevent too frequent
# load average requests.
limit_req_zone $binary_remote_addr
               zone=load_average_limit_request_zone:256k rate=10r/s;

# Extract the last minute load average from the /usr/bin/uptime
# output and assign it to $load_average.
#
# The first alarm call sets up a 2 second timer and the 2nd call
# cancels it. If the Perl subroutine is not completed within
# 2 seconds, a return is forced to prevent Perl from holding up
# further processing.
perl_set $load_average '
    sub {
        eval {
            local $SIG{ALRM} = sub { return ""; };
            alarm 2;
            `/usr/bin/uptime` =~ /averages: (\d+\.\d+)/;
            alarm 0;
            return $1;
        }
    }
';

server {

    location /load_average/ {

        # Prevent this load average from being cached for more
        # than 2 seconds by the frontend server.
        expires 2s;

        # Prevent too frequent load average requests.
        limit_req_log_level error;
        limit_req zone=load_average_limit_request_zone nodelay;

        # Return $load_average using Agentzh's Echo module
        echo $load_average;

        # Note that using "return 200 $load_average;" instead of
        # "echo $load_average;" to return $load_average makes
        # any limit_request directive useless inside this
        # location block because it prevents the limit_request
        # triggered error code 503 from being returned.
        #return 200 $load_average;
    }
}

```

}

The backend servers would require the following additional modules  
for this configuration to work:

Embedded Perl module

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

Agentzh’s Echo module (optional - see the “return 200” comment above)

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

My comments are extensive, so there’s nothing to add, except that  
this could be done using Apache and a CGI script that does something  
like this to extract the load average from the uptime output:

uptime | awk -F’: |,’ ‘{print $5}’

You could also set up a shell script based netcat server to answer  
load average requests, but you’d have to set up additional firewall  
rules to prevent too frequent requests from degrading performance.

You could also extend the load balancer to include other performance  
criteria, such as iostat, netstat values etc.

The Stub Status module statistics could also be included:

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

Note that if you decide to cache load averages on the frontend  
server, you should always make sure that your backend  
servers always set an appropriate expiration time (by using  
the “Cache-Control: max-age=$x” header, for example) in the  
response (typically, it should be no longer than a few  
seconds, but that really depends on your setup).

I hope that helps, any benchmarks and comments will be appreciated.

Max