Debugging nginx Load Balancer

arung · April 29, 2009, 7:44am

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun

arung · April 29, 2009, 9:11am

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

nginx should be build with ./configure --with-debug
add ip to debug requests to nginx.conf:

events {
debug_connection 10.10.1.1;
}

arung · April 29, 2009, 3:47pm

Thanks for the response!

nginx was built using --with-debug, here is my complete CLI:

./configure --prefix=/usr/local/nginx --sbin-path=/usr/sbin --with-debug
–with-http_ssl_module

nginx and GlassFish gem are all running on the same box for my
experiment, i.e. 127.0.0.1. So I updated the existing events entry as:

events {
worker_connections 1024;
debug_connection 127.0.0.1;
}

Saw no new messages in access.log and lots of messages in error.log but
no information on the port being servicing the request.

Am I missing something ?

-Arun

Anton Y. wrote:

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

nginx should be build with ./configure --with-debug

add ip to debug requests to nginx.conf:

events {
debug_connection 10.10.1.1;
}

arung · April 29, 2009, 8:07pm

May be I misunderstood your question, but if you want to know upstream
port, you may add to
access_log format variable $upstream_addr

Module ngx_http_upstream_module
Module ngx_http_log_module
Ok, that took care of logging.

I created a cluster of 3 backend servers and the access.log now shows
…

127.0.0.1 - [127.0.0.1:3002] - [29/Apr/2009:11:03:43 -0700] GET /runlogs
HTTP/1.1 “200” 621 “-” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1” “-”
127.0.0.1 - [127.0.0.1:3000] - [29/Apr/2009:11:03:43 -0700] GET
/stylesheets/scaffold.css?1240977992 HTTP/1.1 “200” 889
“http://localhost/runlogs” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1” “-”
127.0.0.1 - [127.0.0.1:3001] - [29/Apr/2009:11:03:43 -0700] GET
/favicon.ico HTTP/1.1 “200” 0 “http://localhost/runlogs” “Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1
(KHTML, like Gecko) Version/3.2.1 Safari/525.27.1” “-”

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3
?

-Arun

arung · April 29, 2009, 8:34pm

Arun G. wrote:

/favicon.ico HTTP/1.1 “200” 0 “http://localhost/runlogs” “Mozilla/5.0
(Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1
(KHTML, like Gecko) Version/3.2.1 Safari/525.27.1” “-”

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3

It seems to be 3 different requests and they was served by different
upstreams because of load
balancing upstreams.

arung · April 29, 2009, 4:49pm

Arun G. wrote:

events {
worker_connections 1024;
debug_connection 127.0.0.1;
}

Saw no new messages in access.log and lots of messages in error.log but
no information on the port being servicing the request.

This should be in error.log

May be I misunderstood your question, but if you want to know upstream
port, you may add to
access_log format variable $upstream_addr

http://wiki.nginx.org/NginxHttpUpstreamModule#.24upstream_addr
http://wiki.nginx.org/NginxHttpLogModule

arung · April 29, 2009, 8:42pm

The port is shown as the second parameter in each log line, that’s good!
But I expected the request to be served by only one instance, why all 3

It seems to be 3 different requests and they was served by different
upstreams because of load
balancing upstreams.
Interesting …

If I curl the page then I see only one log entry but invoking the page
from the browser still shows 3 log entries.

Any idea why 3 requests are made from the browser ?

-Arun

arung · April 29, 2009, 9:38pm

Any idea why 3 requests are made from the browser ?

-Arun

It looks to me like three requests were made, because it’s getting
three different resources:

GET /runlogs
GET /stylesheets/scaffold.css?1240977992
GET /favicon.ico

Probably because the HTML at /runlogs refers to the stylesheet and
icon, and the browser goes and gets them to display them. No?

arung · April 29, 2009, 9:53pm

Ok that explains part of it!

curl (wget on Mac) is indeed using single request. I set
keepalive_timeout to 0 but still see 3 requests being made.

-Arun

Cliff W. wrote:

On Wed, 2009-04-29 at 20:42 +0200, Arun G. wrote:

Any idea why 3 requests are made from the browser ?

Most browsers will use several separate connections per domain for
downloading separate elements (Firefox defaults to 8 for normal
connections, 2 for persistent connections). Your keepalive settings for
Nginx can affect this number.

I believe wget can only use one under all circumstances.

Cliff

arung · April 29, 2009, 9:55pm

It looks to me like three requests were made, because it’s getting
three different resources:
GET /runlogs
GET /stylesheets/scaffold.css?1240977992
GET /favicon.ico
Probably because the HTML at /runlogs refers to the stylesheet and
icon, and the browser goes and gets them to display them. No?
Ok, that explains it!

I guess now static file caching needs to be enabled which will reduce
the trip to back-end servers, right ?

-Arun

arung · April 30, 2009, 12:32am

You need a block that will cause nginx to serve static files directly.
Something like

location ~* ^.+\.(css|js)$ {
    root /var/www/foo;
    access_log off;
    expires max;
    break;
}

arung · April 29, 2009, 9:01pm

On Wed, 2009-04-29 at 20:42 +0200, Arun G. wrote:

Any idea why 3 requests are made from the browser ?

Most browsers will use several separate connections per domain for
downloading separate elements (Firefox defaults to 8 for normal
connections, 2 for persistent connections). Your keepalive settings for
Nginx can affect this number.

I believe wget can only use one under all circumstances.

Cliff

arung · May 15, 2009, 11:18pm

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun

you should switch over to a load master 2500 because the load master
2500 is an essential component of high-availability, clustering and
fault tolerance, all of which provide the infrastructure for reliable
Internet sites and corporate intranets.

http://www.kemptechnologies.com/load-balancer-2500.shtml

arung · May 15, 2009, 11:19pm

Arun G. wrote:

I’m using nginx 0.6.36 for a load balancing a cluster of GlassFish Gem.

Is there a debug level or statement that can be added to nginx.conf that
will log the instance serving the request ?

-Arun

you should switch over to a load master 2500 because the load master
2500 is an essential component of high-availability, clustering and
fault tolerance, all of which provide the infrastructure for reliable
Internet sites and corporate intranets.

http://www.kemptechnologies.com/load-balancer-2500.shtml