# Nginx 100% cpu and it may be a nginx bug **URL:** \ **Category:** NGINX\ **Created:** [November 22, 2011, 11:38am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121 "2011-11-22T11:38:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\ **Post date:** [November 22, 2011, 11:38am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/1 "2011-11-22T11:38:40Z")
The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer for the first time, in some cases it returns NGX\_OK while pc-\>tries equals to 1. When in ngx\_http\_upstream\_process\_header, nginx process may enter the following: 1524 if (n == NGX\_ERROR || n == 0) { 1525 ngx\_http\_upstream\_next(r, u, NGX\_HTTP\_UPSTREAM\_FT\_ERROR); 1526 return; 1527 } then enter ngx\_http\_upstream\_next function In the ngx\_http\_upstream\_next,nginx process will call ngx\_http\_upstream\_free\_round\_robin\_peer 2796 if (ft\_type != NGX\_HTTP\_UPSTREAM\_FT\_NOLIVE) { 2797 u-\>peer.free(&u-\>peer, u-\>peer.data, state); 2798 } we now enter ngx\_http\_upstream\_free\_round\_robin\_peer. Pay attention to the following codes in the function: if (pc-\>tries) { pc-\>tries–; } After executing the above, pc-\>tries changes from 1 to 0. then the process returns to ngx\_http\_upstream\_next 2892 ngx\_http\_upstream\_connect(r, u); 2893 } we enter ngx\_http\_upstream\_connect here: this function will call 1081 rc = ngx\_event\_connect\_peer(&u-\>peer); … It finally enter ngx\_http\_upstream\_get\_round\_robin\_peer again. And then the process will loop for a long time which is unpredictable. Posted at Nginx Forum: > **[nginx 100% cpu and it may be a nginx bug](https://forum.nginx.org/read.php?2,218736,218736#msg-218736)** > > The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer... ---
**Author:** ![wangbin579](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/bc79bd/32.png) [@wangbin579](https://www.ruby-forum.com/u/wangbin579)\ **Post date:** [November 22, 2011, 11:41am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/2 "2011-11-22T11:41:46Z")
we find all processes of nginx are all nearly cpu 100%. we used tcpcopy to find this problem. Posted at Nginx Forum: > **[Re: nginx 100% cpu and it may be a nginx bug](https://forum.nginx.org/read.php?2,218736,218737#msg-218737)** > > The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer... ---
**Author:** ![wangbin579](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/bc79bd/32.png) [@wangbin579](https://www.ruby-forum.com/u/wangbin579)\ **Post date:** [November 22, 2011, 11:44am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/3 "2011-11-22T11:44:52Z")
we use gcore to generate a core file. the backtrace is as the following: #0 0x000000000046ae0b in ngx\_http\_upstream\_get\_round\_robin\_peer (pc=0x1b822678, data=0x1b826e10) at src/http/ngx\_http\_upstream\_round\_robin.c:482 #1 0x00000000004a6a60 in ngx\_http\_upstream\_get\_keepalive\_peer (pc=0x1b822678, data=0x1b826de0) at …/ngx\_http\_upstream\_keepalive-c6396fef9295/ngx\_http\_upstream\_keepalive\_module.c:227 #2 0x0000000000429fab in ngx\_event\_connect\_peer (pc=0x1b822678) at src/event/ngx\_event\_connect.c:24 #3 0x000000000046219c in ngx\_http\_upstream\_connect (r=0x1b821230, u=0x1b822668) at src/http/ngx\_http\_upstream.c:1103 #4 0x0000000000465f61 in ngx\_http\_upstream\_next (r=0x1b821230, u=0x1b822668, ft\_type=2) at src/http/ngx\_http\_upstream.c:2900 #5 0x00000000004631c0 in ngx\_http\_upstream\_process\_header (r=0x1b821230, u=0x1b822668) at src/http/ngx\_http\_upstream.c:1547 #6 0x0000000000462d8c in ngx\_http\_upstream\_send\_request (r=0x1b821230, u=0x1b822668) at src/http/ngx\_http\_upstream.c:1414 #7 0x00000000004625de in ngx\_http\_upstream\_connect (r=0x1b821230, u=0x1b822668) at src/http/ngx\_http\_upstream.c:1204 #8 0x000000000046129d in ngx\_http\_upstream\_init\_request (r=0x1b821230) at src/http/ngx\_http\_upstream.c:631 #9 0x0000000000460a6d in ngx\_http\_upstream\_init (r=0x1b821230) at src/http/ngx\_http\_upstream.c:432 #10 0x000000000045782b in ngx\_http\_read\_client\_request\_body (r=0x1b821230, post\_handler=0x460989 \) at src/http/ngx\_http\_request\_body.c:154 #11 0x00000000004a0875 in ngx\_hmux\_handler (r=0x1b821230) at …/hmux/ngx\_hmux\_module.c:673 #12 0x000000000043ffd1 in ngx\_http\_core\_content\_phase (r=0x1b821230, ph=0x1b5d4f88) at src/http/ngx\_http\_core\_module.c:1365 #13 0x000000000043ebf5 in ngx\_http\_core\_run\_phases (r=0x1b821230) at src/http/ngx\_http\_core\_module.c:861 #14 0x000000000043eb71 in ngx\_http\_handler (r=0x1b821230) at src/http/ngx\_http\_core\_module.c:844 #15 0x000000000044c9a1 in ngx\_http\_process\_request (r=0x1b821230) at src/http/ngx\_http\_request.c:1665 #16 0x000000000044b34d in ngx\_http\_process\_request\_headers (rev=0x2b3f10a3f4f0) at src/http/ngx\_http\_request.c:1111 #17 0x000000000044ab26 in ngx\_http\_process\_request\_line (rev=0x2b3f10a3f4f0) at src/http/ngx\_http\_request.c:911 #18 0x0000000000449d7d in ngx\_http\_init\_request (rev=0x2b3f10a3f4f0) at src/http/ngx\_http\_request.c:518 #19 0x0000000000434ea7 in ngx\_epoll\_process\_events (cycle=0x1b5ae090, timer=1352, flags=1) at src/event/modules/ngx\_epoll\_module.c:678 #20 0x0000000000426480 in ngx\_process\_events\_and\_timers (cycle=0x1b5ae090) at src/event/ngx\_event.c:245 —Type to continue, or q to quit— #21 0x0000000000432d0c in ngx\_worker\_process\_cycle (cycle=0x1b5ae090, data=0x0) at src/os/unix/ngx\_process\_cycle.c:801 #22 0x000000000042faad in ngx\_spawn\_process (cycle=0x1b5ae090, proc=0x432b60 \, data=0x0, name=0x57450e “worker process”, respawn=-3) at src/os/unix/ngx\_process.c:196 #23 0x0000000000431c12 in ngx\_start\_worker\_processes (cycle=0x1b5ae090, n=8, type=-3) at src/os/unix/ngx\_process\_cycle.c:360 #24 0x00000000004312fe in ngx\_master\_process\_cycle (cycle=0x1b5ae090) at src/os/unix/ngx\_process\_cycle.c:136 #25 0x0000000000403c8c in main (argc=1, argv=0x7fffba823048) at src/core/nginx.c:405 (gdb) info lo rrp = 0x1b826e10 now = 1321860194 m = 8 rc = 461502464 i = 18446744061305903636 n = 0 c = 0x40 peer = 0x1b5cf2c0 peers = 0x1 (gdb) p \*rrp $5 = {peers = 0x1b5cf230, current = 3, tried = 0x1b826e28, data = 4} (gdb) p rrp-\>peers-\>peer[0] $6 = {sockaddr = 0x1b5c63b8, socklen = 16, name = {len = 19, data = 0x1b5c63c8 “xxx.xxx.xxx.xxx:36901server”}, current\_weight = 0, weight = 1, fails = 1, accessed = 1321860193, max\_fails = 1, fail\_timeout = 10, down = 0, ssl\_session = 0x0} (gdb) p rrp-\>peers-\>peer[1] $7 = {sockaddr = 0x1b5c6418, socklen = 16, name = {len = 19, data = 0x1b5c6428 “xxx.xxx.xxx.xxx:36902server”}, current\_weight = 0, weight = 1, fails = 1, accessed = 1321860194, max\_fails = 1, fail\_timeout = 10, down = 0, ssl\_session = 0x0} (gdb) p rrp-\>peers-\>peer[2] $8 = {sockaddr = 0x1b5c6478, socklen = 16, name = {len = 19, data = 0x1b5c6488 “xxx.xxx.xxx.xxx:36903server”}, current\_weight = 0, weight = 1, fails = 1, accessed = 1321860194, max\_fails = 1, fail\_timeout = 10, down = 0, ssl\_session = 0x0} (gdb) p rrp-\>peers-\>peer[3] $9 = {sockaddr = 0x1b5c64d8, socklen = 16, name = {len = 19, data = 0x1b5c64e8 “xxx.xxx.xxx.xxx:36904keepalive”}, current\_weight = 0, weight = 1, fails = 1, accessed = 1321860193, max\_fails = 1, fail\_timeout = 10, down = 0, ssl\_session = 0x0} Posted at Nginx Forum: > **[Re: nginx 100% cpu and it may be a nginx bug](https://forum.nginx.org/read.php?2,218736,218738#msg-218738)** > > The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer... ---
**Author:** ![wangbin579](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/bc79bd/32.png) [@wangbin579](https://www.ruby-forum.com/u/wangbin579)\ **Post date:** [November 22, 2011, 11:46am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/4 "2011-11-22T11:46:07Z")
nginx-0.8.53 has the same problem Posted at Nginx Forum: > **[Re: nginx 100% cpu and it may be a nginx bug](https://forum.nginx.org/read.php?2,218736,218740#msg-218740)** > > The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer... ---
**Author:** ![wangbin579](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/bc79bd/32.png) [@wangbin579](https://www.ruby-forum.com/u/wangbin579)\ **Post date:** [November 22, 2011, 12:02pm UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/5 "2011-11-22T12:02:50Z")
Hello! On Tue, Nov 22, 2011 at 05:38:11AM -0500, wangbin579 wrote: > 1524 if (n == NGX\_ERROR || n == 0) { > 2796 if (ft\_type != NGX\_HTTP\_UPSTREAM\_FT\_NOLIVE) { > > After executing the above, pc-\>tries changes from 1 to 0. > > then the process returns to ngx\_http\_upstream\_next > > 2892 ngx\_http\_upstream\_connect(r, u); > 2893 } The ngx\_http\_upstream\_next() will finalize request if tries == 0. You are probably facing the problem with keepalive backend connections in 1.0.x (where they aren’t officially supported). It is fixed with this changeset while introducing official keepalive backend connections support in 1.1.4: [http://trac.nginx.org/nginx/changeset/4116/nginx](http://trac.nginx.org/nginx/changeset/4116/nginx) Maxim D. ---
**Author:** ![wangbin579](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/bc79bd/32.png) [@wangbin579](https://www.ruby-forum.com/u/wangbin579)\ **Post date:** [November 22, 2011, 11:45am UTC](https://www.ruby-forum.com/t/nginx-100-cpu-and-it-may-be-a-nginx-bug/214121/6 "2011-11-22T11:45:29Z")
[root@208 sbin]# /home/nginx/sbin/nginx -V nginx: nginx version: nginx/1.0.10 nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48) nginx: TLS SNI support enabled nginx: configure arguments: --prefix=/home/nginx --user=nginx –group=nginx --without-select\_module --without-poll\_module –with-http\_ssl\_module --with-http\_realip\_module –with-http\_gzip\_static\_module --with-http\_stub\_status\_module –without-http\_fastcgi\_module --without-http\_memcached\_module –without-http\_empty\_gif\_module --with-pcre=…/pcre-7.9 –with-openssl=…/openssl-0.9.8r --add-module=…/hmux –add-module=…/ngx\_http\_upstream\_keepalive-c6396fef9295 --with-debug part of nginx conf file： events { worker\_connections 8192; use epoll; epoll\_events 4096; accept\_mutex off; } … upstream yyy { server xxx.xxx.xxx.xxx:36901; server xxx.xxx.xxx.xxx:36902; server xxx.xxx.xxx.xxx:36903; server xxx.xxx.xxx.xxx:36904; ``` keepalive 1024; } ``` Posted at Nginx Forum: > **[Re: nginx 100% cpu and it may be a nginx bug](https://forum.nginx.org/read.php?2,218736,218739#msg-218739)** > > The function ngx\_http\_upstream\_get\_round\_robin\_peer in nginx will loop for a long period of times when pc-\>tries equals to 0. Well then, under what conditions does pc-\>tries equal to zero? When executing ngx\_http\_upstream\_get\_round\_robin\_peer...