Nginx 100% cpu usage

lalyos · July 11, 2012, 1:01pm

Hi,

im using nginx 0.8.54 in front of a tomcat. its running since 1 week
now, and with just a few open sessions, nginx is using 100% cpu.

strace is telling that ‘munmap’ is responsible for 94%. so looks like
some sort of memory problem,
but top shows that 400mb is free from the 4gb total memory.

more detailed logs (strace sample/top/nginx acces/error logs)

gist.github.com

https://gist.github.com/lalyos/3089367

nginx -V

nginx version: nginx/0.8.54
built by gcc 4.4.3 (Ubuntu 4.4.3-4ubuntu5) 
TLS SNI support enabled
configure arguments: --prefix=/home/xxx/cloudfoundry/.deployments/devbox/deploy/nginx/nginx-0.8.54 --with-pcre=../pcre-8.12 --add-module=../nginx_upload_module-2.2.0 --add-module=../agentzh-headers-more-nginx-module-5fac223 --add-module=../simpl-ngx_devel_kit-bc97eea --add-module=../chaoslawful-lua-nginx-module-4d92cb1 --with-http_ssl_module

nginx_acces.log

api.vcap.me - [11/Jul/2012:12:31:26 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
fake.acme.com - [11/Jul/2012:12:31:27 +0200] "POST /teamreviews/doexport HTTP/1.1" 200 33191 "https://fake.acme.com/teamreviews/export" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11" 10.0.13.19 response_time:0.334
api.vcap.me - [11/Jul/2012:12:31:52 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:32:09 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:32:20 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:32:26 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:32:52 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:33:09 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:33:20 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008
api.vcap.me - [11/Jul/2012:12:33:26 +0200] "POST /services/v1/offerings HTTP/1.1" 200 306 "-" "EventMachine HttpClient" 127.0.0.1 response_time:0.008

This file has been truncated. show original

nginx_error.log

2012/07/11 12:42:26 [debug] 13011#0: *689939 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:42:53 [debug] 13011#0: *689942 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:43:09 [debug] 13011#0: *689945 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:43:20 [debug] 13011#0: *689948 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:43:26 [debug] 13011#0: *689951 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:43:53 [debug] 13011#0: *689954 route api.vcap.me to 10.0.11.180:9022
2012/07/11 12:44:05 [info] 13011#0: *689957 client closed prematurely connection while SSL handshaking, client: 10.0.14.122, server: _
2012/07/11 12:44:05 [debug] 13011#0: *689958 req sticks to backend session:TRNhUoO3UoV468Lx53NyifTZipDyxg6euAWhFZsf4ieb4vImBtzV5eNnp20+Ft9zZ/YtqgnAWMP0hyJKQsn+aA==
2012/07/11 12:44:05 [debug] 13011#0: *689958 route fake.acme.com to 10.0.11.180:45683
2012/07/11 12:44:05 [notice] 13011#0: *689961 "^/(.*)" matches "/member/filledquestionaries", client: 10.0.14.122, server: _, request: "GET /member/filledquestionaries HTTP/1.1", host: "fake.acme.com"

This file has been truncated. show original

There are more than three files. show original

note this is a vmware virtual machine.

any tips?

Posted at Nginx Forum:

lalyos · July 11, 2012, 2:11pm

any tips?

Upgrade.
0.8.54 is like 2 years old already and there have been few bugs relating
to
cpu hogging.

rr

lalyos · July 30, 2012, 10:00am

Hi Reinis,

I was using nginx as part of the CloudFoundry framework, so first i
tried to use the nginx version they provided.
but i couldn’t figure it out, so I just did an upgrade to 1.1.5 (that
was the latest i could use with the required plugins)

cpu hogging is gone,
thanks for the tip!

cheers,

lalyos

Posted at Nginx Forum: