Forum: NGINX kill -HUP failing (0.6.32)

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
A21b515e2b9c5040e87c298a407a8df9?d=identicon&s=25 Guy Naor (Guest)
on 2008-10-10 14:55
(Received via mailing list)
Hi,

I have an nginx setup as a reverse proxy into a large number of Rails
and
Java applications. As it's a very dynamic system, nginx incudes a whole
directory of conf files, one for each application.

nginx works really well, but now that we have about 320 different config
files nginx stopped responding to kill -HUP for reloading the conf
files. It
just ignores the request and nothing happens. My test shows that at 247
files it stop responding to HUP. Up to that number it's all ok and HUP
works
perfectly. The files are all pretty much the same, with this format:


# Definition of the different compute instances we proxy into
upstream redmine_nginx {
  server 10.252.179.143:5150;
}

# vhost definition for this application
server {
  listen 80;
  client_max_body_size 10M;
  client_body_buffer_size 128k;
  server_name redmine.morphexchange.com ; # Add the redirected domains
here
(like: www.gadgets.com that is redirected to gadgets.mex.com)
  set $original_uri $uri;
  access_log /mnt/nginx/logs/redmine/access.log main;
  error_log  /mnt/nginx/logs/redmine/error.log warn;
  location ~* ^.*/(themes|stylesheets|javascripts|images)(/.*$|$) {
    proxy_pass        http://redmine_nginx;
    proxy_set_header  X-Real-IP  $http_x_source_address;
    proxy_set_header Host $host;
  }
  location / {
    proxy_set_header  X-Real-IP  $http_x_source_address;
    proxy_intercept_errors  on;
    proxy_set_header Host $host;
    error_page  400 401 403 404 414 500 501 502 503 504 505 506 507 508
509
510  =  @rails_redmine;
    if ($request_method = POST ) {
      proxy_pass        http://redmine_nginx;
     break;
    }
    if ($is_args = "?" ) {
      proxy_pass        http://redmine_nginx;
     break;
    }

  }
  location @rails_redmine {
    rewrite (.*) $original_uri break;
    proxy_pass        http://redmine_nginx;
    proxy_set_header  X-Real-IP  $http_x_source_address;
    proxy_set_header Host $host;
   }
}

I also tried to increase the server hash size and bucket size but it
didn't
help.

Looking at the code, there are a lot of conditions that might result in
HUP
failing, but those are not logged. Any idea what might be causing this?
The
same config with less files has no problem with the HUP signal.


Regards,

Guy Naor.
4eac1f7d1ff86ab17e7a9dbe6af0b44b?d=identicon&s=25 Phillip B Oldham (Guest)
on 2008-10-10 15:04
(Received via mailing list)
Attachment: phill.vcf (273 Bytes)
Is it the number of config files, or the number of server {} blocks
causing the issue? If the former, a quick fix might be to cat all the
files into one which nginx uses to read. If the latter... its not my
area of expertise.

Guy Naor wrote:
> it's all ok and HUP works perfectly. The files are all pretty much the
>   listen 80;
>     proxy_pass        http://redmine_nginx;
>       proxy_pass        http://redmine_nginx;
>     proxy_pass        http://redmine_nginx;
> causing this? The same config with less files has no problem with the
> Morph Labs, Inc.
> guy@mor.ph <mailto:guy@mor.ph>
> www.morphexchange.com <http://www.morphexchange.com>

--

*Phillip B Oldham*
The Activity People
phill@theactivitypeople.co.uk <mailto:phill@theactivitypeople.co.uk>

------------------------------------------------------------------------

*Policies*

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2008-10-10 15:09
(Received via mailing list)
On Fri, Oct 10, 2008 at 05:48:42AM -0700, Guy Naor wrote:

>
>   server_name redmine.morphexchange.com ; # Add the redirected domains here
>     proxy_set_header  X-Real-IP  $http_x_source_address;
>      break;
>
> I also tried to increase the server hash size and bucket size but it didn't
> help.
>
> Looking at the code, there are a lot of conditions that might result in HUP
> failing, but those are not logged. Any idea what might be causing this? The
> same config with less files has no problem with the HUP signal.

What does "nginx -t" show ?
410529be456752e9c98f19934aec8d62?d=identicon&s=25 Kenneth Demanawa (Guest)
on 2008-10-10 17:01
(Received via mailing list)
(Responding in behalf of Guy. :) )

Hi Igor.

"nginx -t" shows:

2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf syntax is ok
2008/10/10 14:43:32 [info] 23171#0: the configuration file
./nginx.conf was tested successfully

Kenneth
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2008-10-10 17:14
(Received via mailing list)
On Fri, Oct 10, 2008 at 10:47:03PM +0800, Kenneth Demanawa wrote:

> (Responding in behalf of Guy. :) )
>
> Hi Igor.
>
> "nginx -t" shows:
>
> 2008/10/10 14:43:32 [info] 23171#0: the configuration file
> ./nginx.conf syntax is ok
> 2008/10/10 14:43:32 [info] 23171#0: the configuration file
> ./nginx.conf was tested successfully

What is in error_log after -HUP ?
410529be456752e9c98f19934aec8d62?d=identicon&s=25 Kenneth Demanawa (Guest)
on 2008-10-10 17:38
(Received via mailing list)
Hi Igor.

I was tailing the error_log file when I issued the -HUP, and there's
no new entry added in the log file after I gave the signal to the
master process. :(

Kenneth
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2008-10-10 17:46
(Received via mailing list)
On Fri, Oct 10, 2008 at 11:31:14PM +0800, Kenneth Demanawa wrote:

> Hi Igor.
>
> I was tailing the error_log file when I issued the -HUP, and there's
> no new entry added in the log file after I gave the signal to the
> master process. :(

If you set error_log to notice level, do you see messages like these:

2008/10/10 18:30:14 [notice] 43492#0: signal 1 (SIGHUP) received,
reconfiguring
2008/10/10 18:30:14 [notice] 43492#0: reconfiguring
410529be456752e9c98f19934aec8d62?d=identicon&s=25 Kenneth Demanawa (Guest)
on 2008-10-10 18:24
(Received via mailing list)
Hi Igor.

here's what the error log file says:

2008/10/10 16:11:14 [emerg] 29967#0: open()
"/var/nginx/logs/xxxxx/error.log" failed (24: Too many open files)

Kenneth
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2008-10-10 18:32
(Received via mailing list)
On Sat, Oct 11, 2008 at 12:14:14AM +0800, Kenneth Demanawa wrote:

> Hi Igor.
>
> here's what the error log file says:
>
> 2008/10/10 16:11:14 [emerg] 29967#0: open()
> "/var/nginx/logs/xxxxx/error.log" failed (24: Too many open files)

So, this is the answer. You need to increase file numbers.
A21b515e2b9c5040e87c298a407a8df9?d=identicon&s=25 Guy Naor (Guest)
on 2008-10-10 21:50
(Received via mailing list)
Hi,

We increased the file limit (using ulimit -S -H -n 4096) and it worked
like
a charm!

Thanks,

Guy.
This topic is locked and can not be replied to.