Kill -HUP failing (0.6.32)

Hi,

I have an nginx setup as a reverse proxy into a large number of Rails
and
Java applications. As it’s a very dynamic system, nginx incudes a whole
directory of conf files, one for each application.

nginx works really well, but now that we have about 320 different config
files nginx stopped responding to kill -HUP for reloading the conf
files. It
just ignores the request and nothing happens. My test shows that at 247
files it stop responding to HUP. Up to that number it’s all ok and HUP
works
perfectly. The files are all pretty much the same, with this format:

Definition of the different compute instances we proxy into

upstream redmine_nginx {
server 10.252.179.143:5150;
}

vhost definition for this application

server {
listen 80;
client_max_body_size 10M;
client_body_buffer_size 128k;
server_name redmine.morphexchange.com ; # Add the redirected domains
here
(like: www.gadgets.com that is redirected to gadgets.mex.com)
set $original_uri $uri;
access_log /mnt/nginx/logs/redmine/access.log main;
error_log /mnt/nginx/logs/redmine/error.log warn;
location ~* ^./(themes|stylesheets|javascripts|images)(/.$|$) {
proxy_pass http://redmine_nginx;
proxy_set_header X-Real-IP $http_x_source_address;
proxy_set_header Host $host;
}
location / {
proxy_set_header X-Real-IP $http_x_source_address;
proxy_intercept_errors on;
proxy_set_header Host $host;
error_page 400 401 403 404 414 500 501 502 503 504 505 506 507 508
509
510 = @rails_redmine;
if ($request_method = POST ) {
proxy_pass http://redmine_nginx;
break;
}
if ($is_args = “?” ) {
proxy_pass http://redmine_nginx;
break;
}

}
location @rails_redmine {
rewrite (.*) $original_uri break;
proxy_pass http://redmine_nginx;
proxy_set_header X-Real-IP $http_x_source_address;
proxy_set_header Host $host;
}
}

I also tried to increase the server hash size and bucket size but it
didn’t
help.

Looking at the code, there are a lot of conditions that might result in
HUP
failing, but those are not logged. Any idea what might be causing this?
The
same config with less files has no problem with the HUP signal.

Regards,

Guy N…

On Fri, Oct 10, 2008 at 05:48:42AM -0700, Guy N. wrote:

server_name redmine.morphexchange.com ; # Add the redirected domains here
proxy_set_header X-Real-IP $http_x_source_address;
break;

I also tried to increase the server hash size and bucket size but it didn’t
help.

Looking at the code, there are a lot of conditions that might result in HUP
failing, but those are not logged. Any idea what might be causing this? The
same config with less files has no problem with the HUP signal.

What does “nginx -t” show ?

(Responding in behalf of Guy. :slight_smile: )

Hi Igor.

“nginx -t” shows:

2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf syntax is ok
2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf was tested successfully

Kenneth

Is it the number of config files, or the number of server {} blocks
causing the issue? If the former, a quick fix might be to cat all the
files into one which nginx uses to read. If the latter… its not my
area of expertise.

Guy N. wrote:

it’s all ok and HUP works perfectly. The files are all pretty much the
listen 80;
proxy_pass http://redmine_nginx;
proxy_pass http://redmine_nginx;
proxy_pass http://redmine_nginx;
causing this? The same config with less files has no problem with the
Morph Labs, Inc.
[email protected] mailto:[email protected]
www.morphexchange.com http://www.morphexchange.com

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]


Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

Hi Igor.

I was tailing the error_log file when I issued the -HUP, and there’s
no new entry added in the log file after I gave the signal to the
master process. :frowning:

Kenneth

On Fri, Oct 10, 2008 at 11:31:14PM +0800, Kenneth Demanawa wrote:

Hi Igor.

I was tailing the error_log file when I issued the -HUP, and there’s
no new entry added in the log file after I gave the signal to the
master process. :frowning:

If you set error_log to notice level, do you see messages like these:

2008/10/10 18:30:14 [notice] 43492#0: signal 1 (SIGHUP) received,
reconfiguring
2008/10/10 18:30:14 [notice] 43492#0: reconfiguring

On Fri, Oct 10, 2008 at 10:47:03PM +0800, Kenneth Demanawa wrote:

(Responding in behalf of Guy. :slight_smile: )

Hi Igor.

“nginx -t” shows:

2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf syntax is ok
2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf was tested successfully

What is in error_log after -HUP ?

On Sat, Oct 11, 2008 at 12:14:14AM +0800, Kenneth Demanawa wrote:

Hi Igor.

here’s what the error log file says:

2008/10/10 16:11:14 [emerg] 29967#0: open()
“/var/nginx/logs/xxxxx/error.log” failed (24: Too many open files)

So, this is the answer. You need to increase file numbers.

Hi Igor.

here’s what the error log file says:

2008/10/10 16:11:14 [emerg] 29967#0: open()
“/var/nginx/logs/xxxxx/error.log” failed (24: Too many open files)

Kenneth

Hi,

We increased the file limit (using ulimit -S -H -n 4096) and it worked
like
a charm!

Thanks,

Guy.