Debugging nginx Load Balancer


#1

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun


#2

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

  1. nginx should be build with ./configure --with-debug

  2. add ip to debug requests to nginx.conf:

events {
debug_connection 10.10.1.1;
}


#3

Thanks for the response!

nginx was built using --with-debug, here is my complete CLI:

./configure --prefix=/usr/local/nginx --sbin-path=/usr/sbin --with-debug
–with-http_ssl_module

nginx and GlassFish gem are all running on the same box for my
experiment, i.e. 127.0.0.1. So I updated the existing events entry as:

events {
worker_connections 1024;
debug_connection 127.0.0.1;
}

Saw no new messages in access.log and lots of messages in error.log but
no information on the port being servicing the request.

Am I missing something ?

-Arun

Anton Y. wrote:

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

  1. nginx should be build with ./configure --with-debug

  2. add ip to debug requests to nginx.conf:

events {
debug_connection 10.10.1.1;
}


#4

May be I misunderstood your question, but if you want to know upstream
port, you may add to
access_log format variable $upstream_addr

http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_addr
http://wiki.nginx.org/NginxHttpLogModule
Ok, that took care of logging.

I created a cluster of 3 backend servers and the access.log now shows

127.0.0.1 - [127.0.0.1:3002] - [29/Apr/2009:11:03:43 -0700] GET /runlogs
HTTP/1.1 “200” 621 “-” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1” “-”
127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:11:03:43 -0700] GET
/stylesheets/scaffold.css?1240977992 HTTP/1.1 “200” 889
http://localhost/runlogs” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1” “-”
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:11:03:43 -0700] GET
/favicon.ico HTTP/1.1 “200” 0 “http://localhost/runlogs” “Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1
(KHTML, like Gecko) Version/3.2.1 Safari/525.27.1” “-”

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3
?

-Arun


#5

Arun G. wrote:

/favicon.ico HTTP/1.1 “200” 0 “http://localhost/runlogs” “Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1
(KHTML, like Gecko) Version/3.2.1 Safari/525.27.1” “-”

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3

It seems to be 3 different requests and they was served by different
upstreams because of load
balancing upstreams.


#6

Arun G. wrote:

events {
worker_connections 1024;
debug_connection 127.0.0.1;
}

Saw no new messages in access.log and lots of messages in error.log but
no information on the port being servicing the request.

This should be in error.log

May be I misunderstood your question, but if you want to know upstream
port, you may add to
access_log format variable $upstream_addr

http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_addr
http://wiki.nginx.org/NginxHttpLogModule


#7

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3

It seems to be 3 different requests and they was served by different
upstreams because of load
balancing upstreams.
Interesting …

If I curl the page then I see only one log entry but invoking the page
from the browser still shows 3 log entries.

Any idea why 3 requests are made from the browser ?

-Arun


#8

Any idea why 3 requests are made from the browser ?

-Arun

It looks to me like three requests were made, because it’s getting
three different resources:

GET /runlogs
GET /stylesheets/scaffold.css?1240977992
GET /favicon.ico

Probably because the HTML at /runlogs refers to the stylesheet and
icon, and the browser goes and gets them to display them. No?


#9

Ok that explains part of it!

curl (wget on Mac) is indeed using single request. I set
keepalive_timeout to 0 but still see 3 requests being made.

-Arun

Cliff W. wrote:

On Wed, 2009-04-29 at 20:42 +0200, Arun G. wrote:

Any idea why 3 requests are made from the browser ?

Most browsers will use several separate connections per domain for
downloading separate elements (Firefox defaults to 8 for normal
connections, 2 for persistent connections). Your keepalive settings for
Nginx can affect this number.

I believe wget can only use one under all circumstances.

Cliff


#10

It looks to me like three requests were made, because it’s getting
three different resources:

GET /runlogs
GET /stylesheets/scaffold.css?1240977992
GET /favicon.ico

Probably because the HTML at /runlogs refers to the stylesheet and
icon, and the browser goes and gets them to display them. No?
Ok, that explains it!

I guess now static file caching needs to be enabled which will reduce
the trip to back-end servers, right ?

-Arun


#11

You need a block that will cause nginx to serve static files directly.
Something like

location ~* ^.+\.(css|js)$ {
    root /var/www/foo;
    access_log off;
    expires max;
    break;
}

#12

On Wed, 2009-04-29 at 20:42 +0200, Arun G. wrote:

Any idea why 3 requests are made from the browser ?

Most browsers will use several separate connections per domain for
downloading separate elements (Firefox defaults to 8 for normal
connections, 2 for persistent connections). Your keepalive settings for
Nginx can affect this number.

I believe wget can only use one under all circumstances.

Cliff


#13

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun

you should switch over to a load master 2500 because the load master
2500 is an essential component of high-availability, clustering and
fault tolerance, all of which provide the infrastructure for reliable
Internet sites and corporate intranets.

http://www.kemptechnologies.com/load-balancer-2500.shtml


#14

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun

you should switch over to a load master 2500 because the load master
2500 is an essential component of high-availability, clustering and
fault tolerance, all of which provide the infrastructure for reliable
Internet sites and corporate intranets.

http://www.kemptechnologies.com/load-balancer-2500.shtml