100% CPU nginx master process

Server NGINX
1

We are have issue that nginx master process is at 100% CPU and it stop
responding to any request. The CPU utilization for all the workers are
low.

I am also seeing multiple nginx master process running at the same time
when
server stop responding, all of them are at 100% CPU. For my understand,
there should only be one master process. What will cause multiple master
process running?

The problem is very intermittent. It will go away by itself.

Any kind of help will be appreciated.

Thanks

Posted at Nginx Forum:

2

Hello!

On Sun, Mar 2, 2014 at 2:40 PM, badtzhou wrote:

We are have issue that nginx master process is at 100% CPU and it stop
responding to any request. The CPU utilization for all the workers are low.

If I were you, I would just take a C-land on-CPU flame graph for the
process with high CPU usage by sampling for a few or a dozen seconds:

https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt

The graph would tell us the complete story about all the hot spots
during the sampling window.

Regards,
-agentzh

3

We seeing lot of empty requests from nginx process trace. Is this normal
or
some kind of Dos attack?

0.000048 recvfrom(254, “”, 1024, 0, NULL, NULL) = 0
0.000033 close(254) = 0
0.000054 recvfrom(600, “”, 1024, 0, NULL, NULL) = 0
0.000029 close(600) = 0
0.000040 recvfrom(147, “”, 1024, 0, NULL, NULL) = 0
0.000029 close(147) = 0
0.000066 recvfrom(644, “”, 1024, 0, NULL, NULL) = 0
0.000029 close(644) = 0
0.000642 recvfrom(519, “”, 1024, 0, NULL, NULL) = 0
0.000040 close(519) = 0
0.000047 recvfrom(361, “”, 1024, 0, NULL, NULL) = 0
0.000031 close(361) = 0
0.000047 recvfrom(248, “”, 1024, 0, NULL, NULL) = 0
0.000036 close(248) = 0
0.000044 recvfrom(691, “”, 1024, 0, NULL, NULL) = 0
0.000029 close(691) = 0
0.000046 recvfrom(561, “”, 1024, 0, NULL, NULL) = 0
0.000031 close(561) = 0
0.000046 recvfrom(270, “”, 1024, 0, NULL, NULL) = 0
0.000031 close(270) = 0
0.000049 recvfrom(723, “”, 1024, 0, NULL, NULL) = 0
0.000030 close(723) = 0
0.000046 recvfrom(362, “”, 1024, 0, NULL, NULL) = 0
0.000032 close(362) = 0

Posted at Nginx Forum:

4

Hello!

On Sun, Mar 02, 2014 at 05:40:09PM -0500, badtzhou wrote:

We are have issue that nginx master process is at 100% CPU and it stop
responding to any request. The CPU utilization for all the workers are low.

I am also seeing multiple nginx master process running at the same time when
server stop responding, all of them are at 100% CPU. For my understand,
there should only be one master process. What will cause multiple master
process running?

Multiple master processes is not something expected to happen
during normal operation. Multiple master processes may be present
during binary upgrade, but this isn’t something expected to happen
automatically. It might also be new worker processes just spawn -
before they were able to change name and credentials, I’ve seen
something like this on systems with a LDAP-based user database
used when LDAP server was unresponsive.

Note well that master processes doesn’t handle client connections.
Master process only manages worker processes and a configuration.
If you see nginx not responding - there should be another reason
for this. Looking into nginx error log might be a good idea, as
well as examining your system’s logs.

–
Maxim D.
http://nginx.org/