High CPU when upstream server down

Server NGINX
1

Nginx goes to nearly 100% CPU when I take down my upstream servers
(gunicorn). Has anybody heard of this as an issue?

$ nginx -v
nginx version: nginx/0.7.67
$ uname -a
Linux ip-10-242-226-66 2.6.35-27-virtual #48-Ubuntu SMP Tue Feb 22
23:09:12 UTC 2011 i686 GNU/Linux

Summary of top:

top - 20:10:17 up 1 day, 29 min, 4 users, load average: 1.11, 0.83,
0.56
Tasks: 79 total, 2 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.4%us, 36.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
56.7%st
Mem: 1705708k total, 442308k used, 1263400k free, 91240k
buffers
Swap: 917500k total, 0k used, 917500k free, 249920k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

30663 www-data 20 0 5492 1944 916 R 99.9 0.1 5:26.32 nginx

1 root      20   0  2848 1652 1216 S  0.0  0.1   0:00.47 init

2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

3 root      20   0     0    0    0 S  0.0  0.0   0:03.74 ksoftirqd/0

4 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

5 root      RT   0     0    0    0 S  0.0  0.0   0:00.20 watchdog/0

6 root      20   0     0    0    0 S  0.0  0.0   0:01.91 events/0

7 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cpuset

Posted at Nginx Forum:

2

On Tue, Mar 8, 2011 at 2:10 PM, adamn [email protected] wrote:

Nginx goes to nearly 100% CPU when I take down my upstream servers
(gunicorn). Has anybody heard of this as an issue?

Configuration file? I would suspect something causing an endless loop
of redirects or something like that. I did similar to myself in a QA
environment a while back with an ill-concieved “catch-all” dynamic
error_page.

RPM

3

Using the same config file - everything works fine and this is not
reproducible on 0.8.54. Here is my config. Keep in mind that the
upstream and server section are in reality 8 upstreams and servers -
most of which are exactly the same with the exception of the server_name
and the unix socket filename.

user www-data;
worker_processes 1;

error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
# multi_accept on;
}

http {
include /etc/nginx/mime.types;

access_log  /var/log/nginx/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;
tcp_nodelay        on;

gzip  on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

upstream dev.example.com {
    server unix:/tmp/gunicorn_dev.example.com.sock;
    # For a TCP configuration:
    #server 127.0.0.1:8000;
}
server {
    listen 80;
    client_max_body_size 4G;
    server_name dev.example.com;

    keepalive_timeout 5;

    # path for static files
    root /var/www/dev-example-env/example/media;

    location = /favicon.ico {
        alias /var/www/dev-example-env/example/media/favicon.ico;
    }

    location = /robots.txt {

        alias /var/www/dev-example-env/example/media/no_robots.txt;

    }

    location /img {
        alias /var/www/dev-example-env/example/media/img/;
    }

    location /admin_media {
        alias

/var/www/dev-example-env/lib/python2.6/site-packages/django/contrib/admin/media/;
}

    location / {

            auth_basic  "Restricted";
            auth_basic_user_file

“/var/www/dev-example-env/example/.htpasswd”;

        proxy_set_header X-Forwarded-For

$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://dev.example.com;
            break;
        }
    }



    error_page 500 502 503 504 /500.html;
    location = /500.html {
        alias /var/www/dev-example-env/example/templates/500.html;
    }
}

}

Posted at Nginx Forum:

4

On Tue, Mar 8, 2011 at 2:35 PM, adamn [email protected] wrote:

       if (!-f $request_filename) {
           proxy_pass http://dev.example.com;
           break;
       }

As they say, If is Evil… when used in location context | NGINX
I’ve never seen proxy_pass work correcly inside an if block; the fact
that it works for you on 0.8.x seems to be happy coincidence.

Try creating a separate location (maybe regex based if necessary) and
put your proxy_pass statement in there.

–
RPM

5

Just got rid of the if entirely. That logic was totally unnecessary so
now it’s just:

            proxy_pass http://dev.example.com;

Posted at Nginx Forum:

6

Heh - All I did was copy the docs :slight_smile:

http://gunicorn.org/deploy.html

I’ll work out a solution and post it back.

Cheers,
Adam

Posted at Nginx Forum:

7

On Tue, Mar 8, 2011 at 4:02 PM, adamn [email protected] wrote:

Heh - All I did was copy the docs :slight_smile:

Green Unicorn - Deployment

I’ll work out a solution and post it back.

As far as I can tell, that site is in no way affiliated with Igor or
any of the other nginx developers. The authoritative site for nginx is
nginx.org (and the links from there).

–
RPM

8

That is true. Unfortunately there is no documentation on the nginx site
regarding gunicorn.

Posted at Nginx Forum: