High CPU when upstream server down

Nginx goes to nearly 100% CPU when I take down my upstream servers
(gunicorn). Has anybody heard of this as an issue?

$ nginx -v
nginx version: nginx/0.7.67
$ uname -a
Linux ip-10-242-226-66 2.6.35-27-virtual #48-Ubuntu SMP Tue Feb 22
23:09:12 UTC 2011 i686 GNU/Linux

Summary of top:

top - 20:10:17 up 1 day, 29 min, 4 users, load average: 1.11, 0.83,
0.56
Tasks: 79 total, 2 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.4%us, 36.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
56.7%st
Mem: 1705708k total, 442308k used, 1263400k free, 91240k
buffers
Swap: 917500k total, 0k used, 917500k free, 249920k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

30663 www-data 20 0 5492 1944 916 R 99.9 0.1 5:26.32 nginx

1 root      20   0  2848 1652 1216 S  0.0  0.1   0:00.47 init

2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

3 root      20   0     0    0    0 S  0.0  0.0   0:03.74 ksoftirqd/0

4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

5 root      RT   0     0    0    0 S  0.0  0.0   0:00.20 watchdog/0

6 root      20   0     0    0    0 S  0.0  0.0   0:01.91 events/0

7 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cpuset

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,181397,181397#msg-181397

On Tue, Mar 8, 2011 at 2:10 PM, adamn [email protected] wrote:

Nginx goes to nearly 100% CPU when I take down my upstream servers
(gunicorn). Has anybody heard of this as an issue?

Configuration file? I would suspect something causing an endless loop
of redirects or something like that. I did similar to myself in a QA
environment a while back with an ill-concieved “catch-all” dynamic
error_page.

RPM

Using the same config file - everything works fine and this is not
reproducible on 0.8.54. Here is my config. Keep in mind that the
upstream and server section are in reality 8 upstreams and servers -
most of which are exactly the same with the exception of the server_name
and the unix socket filename.

user www-data;
worker_processes 1;

error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
# multi_accept on;
}

http {
include /etc/nginx/mime.types;

access_log  /var/log/nginx/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;
tcp_nodelay        on;

gzip  on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

upstream dev.example.com {
    server unix:/tmp/gunicorn_dev.example.com.sock;
    # For a TCP configuration:
    #server 127.0.0.1:8000;
}
server {
    listen 80;
    client_max_body_size 4G;
    server_name dev.example.com;

    keepalive_timeout 5;

    # path for static files
    root /var/www/dev-example-env/example/media;

    location = /favicon.ico {
        alias /var/www/dev-example-env/example/media/favicon.ico;
    }

    location = /robots.txt {

        alias /var/www/dev-example-env/example/media/no_robots.txt;

    }

    location /img {
        alias /var/www/dev-example-env/example/media/img/;
    }

    location /admin_media {
        alias

/var/www/dev-example-env/lib/python2.6/site-packages/django/contrib/admin/media/;
}

    location / {

            auth_basic  "Restricted";
            auth_basic_user_file

“/var/www/dev-example-env/example/.htpasswd”;

        proxy_set_header X-Forwarded-For

$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://dev.example.com;
            break;
        }
    }



    error_page 500 502 503 504 /500.html;
    location = /500.html {
        alias /var/www/dev-example-env/example/templates/500.html;
    }
}

}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,181397,181406#msg-181406

On Tue, Mar 8, 2011 at 2:35 PM, adamn [email protected] wrote:

       if (!-f $request_filename) {
           proxy_pass http://dev.example.com;
           break;
       }

As they say, http://wiki.nginx.org/IfIsEvil
I’ve never seen proxy_pass work correcly inside an if block; the fact
that it works for you on 0.8.x seems to be happy coincidence.

Try creating a separate location (maybe regex based if necessary) and
put your proxy_pass statement in there.


RPM

Just got rid of the if entirely. That logic was totally unnecessary so
now it’s just:

            proxy_pass http://dev.example.com;

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,181397,181459#msg-181459

Heh - All I did was copy the docs :slight_smile:

http://gunicorn.org/deploy.html

I’ll work out a solution and post it back.

Cheers,
Adam

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,181397,181435#msg-181435

On Tue, Mar 8, 2011 at 4:02 PM, adamn [email protected] wrote:

Heh - All I did was copy the docs :slight_smile:

http://gunicorn.org/deploy.html

I’ll work out a solution and post it back.

As far as I can tell, that site is in no way affiliated with Igor or
any of the other nginx developers. The authoritative site for nginx is
nginx.org (and the links from there).


RPM

That is true. Unfortunately there is no documentation on the nginx site
regarding gunicorn.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,181397,181484#msg-181484

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs