High CPU Usage and NGINX Hangup

Server NGINX
#1

I am seeing my NGINX server randomly hangup. Multiple worker processes
utilizing 100% CPU. I see the hangups in strace, but not quite sure what
would be causing the issue.

0.000081 futex(0x7f563c893068, FUTEX_WAIT, 0, NULL) = ? ERESTARTSYS (To
be
restarted)
92.011974 — SIGHUP (Hangup) @ 0 (0) —
0.000069 rt_sigreturn(0x934b9b) = -1 EINTR (Interrupted system
call)
0.000056 futex(0x7f563c893068, FUTEX_WAIT, 0, NULL) = 0
154.792082 futex(0x7f563c893068, FUTEX_WAKE, 1) = 0

0.000036 recvmsg(32, 0x7fff134e28b0, 0) = -1 EAGAIN (Resource
temporarily
unavailable)
0.000842 epoll_wait(33, {{EPOLLIN, {u32=404296080,
u64=140007748211088}}}, 512, 54019) = 1
49.323010 recvmsg(32, {msg_name(0)=NULL,
msg_iov(1)=[{"\2\0\0\0\0\0\0\0\261m\0\0\0\0\0\0\5\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0",
32}], msg_controllen=0, msg_flags=0}, 0) = 32
0.000129 close(23) = 0
0.002116 recvmsg(32, 0x7fff134e28b0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
0.000837 epoll_wait(33, {{EPOLLIN, {u32=404296080,
u64=140007748211088}}}, 512, 4695) = 1

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,248055,248055#msg-248055

#2

I am now seeing a bunch of epoll_waits in the strace. Multiple master
processes spawn.

4519 0.000000 epoll_wait(143, {}, 512, 500) = 0
4519 0.500833 epoll_wait(143, {}, 512, 500) = 0
4519 0.500785 epoll_wait(143, {}, 512, 500) = 0
4519 0.500756 epoll_wait(143, {}, 512, 500) = 0
4519 0.500742 epoll_wait(143, {}, 512, 500) = 0
4519 0.500760 epoll_wait(143, {}, 512, 500) = 0
4519 0.500815 epoll_wait(143, {}, 512, 500) = 0
4519 0.500792 epoll_wait(143, {}, 512, 500) = 0
4519 0.500727 epoll_wait(143, {}, 512, 500) = 0
4519 0.500760 epoll_wait(143, {}, 512, 500) = 0
4519 0.500754 epoll_wait(143, {}, 512, 500) = 0
4519 0.500702 epoll_wait(143, {}, 512, 500) = 0
4519 0.500742 epoll_wait(143, {}, 512, 500) = 0
4519 0.500751 epoll_wait(143, {}, 512, 500) = 0
4519 0.500786 epoll_wait(143, {}, 512, 500) = 0
4519 0.500764 epoll_wait(143, {}, 512, 500) = 0
4519 0.500743 epoll_wait(143, {}, 512, 500) = 0

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,248055,248119#msg-248119

#3

On Mar 4, 2014, at 9:07 , loki wrote:

154.792082 futex(0x7f563c893068, FUTEX_WAKE, 1) = 0
temporarily unavailable)
0.000837 epoll_wait(33, {{EPOLLIN, {u32=404296080,
u64=140007748211088}}}, 512, 4695) = 1

nginx does not use mutexes. It seems you run 3rd-party modules.

–
Igor S.
http://nginx.com

#4

Hello!

On Mon, Mar 3, 2014 at 9:07 PM, loki wrote:

I am seeing my NGINX server randomly hangup. Multiple worker processes
utilizing 100% CPU.

100% CPU occupancy usually happen on the userland code, so strace is
usually not help by providing info on the syscall level.

It’ll be ideal if you can take a C-land on-CPU flame graph for your
worker processes spining at 100% CPU time, which will give the whole
picture about how CPU time is distributed among all the code paths:

https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt

We’ve been using this to analyse and optimize CPU hogs in our
production environment, and not just for NGINX but for everything that
we can get proper backtraces :slight_smile:

Best regards,
-agentzh