Hi,
Recently we added a ‘thread_pool’ directive to our main configuration. A
few hours later we saw a huge increase in the connections_writing stat
as
reported by stub_status module. This number reached ± 3800 and is stuck
there since. The server in question is operating normally, but this is
very
strange.
Any hints on what this could be?
Some info:
-
Here is a graph of the stats reported, for a server with thread_pool
and
another without: http://imgur.com/a/lF2EL -
I don`t have older data anymore, but the jump from <100 to ± 3800
connections_writing happened in two sharp jumps. The first one following
a
reload; -
The machines’ hardware and software are identical except for the
thread_pool directive in their nginx.conf. They live in two different
data
centers; -
Both machines are performing normally. Nothing unusual in CPU or RAM
usage. Nginx performance is about the same. -
Reloading Nginx with ‘nginx -s reload’ does nothing. Restarting the
process brings connections_writing down.
Debug stuff:
mallmann# uname -a
Linux xxx 3.8.13-98.5.2.el6uek.x86_64 #2 SMP Tue Nov 3 18:32:04 PST 2015
x86_64 x86_64 x86_64 GNU/Linux
mallmann# nginx -V
nginx version: nginx/1.8.0
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx
–sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/tmp/client_body
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
–http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi
–http-scgi-temp-path=/var/lib/nginx/tmp/scgi
–pid-path=/var/run/nginx.pid
–lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx
–with-ipv6
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_xslt_module
–with-http_image_filter_module --with-http_geoip_module
–with-http_sub_module --with-http_flv_module --with-http_mp4_module
–with-http_gunzip_module --with-http_gzip_static_module
–with-http_random_index_module --with-http_secure_link_module
–with-http_degradation_module --with-http_stub_status_module
–with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre
–with-google_perftools_module
–add-module=/builddir/build/BUILD/nginx-1.8.0/headers-more-nginx-module-0.25
–add-module=/builddir/build/BUILD/nginx-1.8.0/ngx_http_bytes_filter_module
–add-module=/builddir/build/BUILD/nginx-1.8.0/echo-nginx-module-0.55
–with-threads --with-debug --with-cc-opt=‘-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
–param=ssp-buffer-size=4 -m64 -mtune=generic’ --with-ld-opt=’ -Wl,-E’
Affected server:
mallmann# lsof -n -u nginx | awk ‘{print $5}’ | sort | uniq -c | sort
-nr
4172 REG
2140 IPv4
100 unix
30 CHR
20 DIR
20 0000
3 sock
1 TYPE
mallmann# curl http://127.0.0.1/status
Active connections: 5924
server accepts handled requests
5864099 5864099 15527178
Reading: 0 Writing: 3883 Waiting: 2040
Normal server:
mallmann# lsof -n -u nginx | awk ‘{print $5}’ | sort | uniq -c | sort
-nr
4454 REG
1967 IPv4
100 unix
30 CHR
20 DIR
20 0000
1 unknown
1 TYPE
1 sock
mallmann# curl http://127.0.0.1/status
Active connections: 2096
server accepts handled requests
1136132 1136132 3464904
Reading: 0 Writing: 107 Waiting: 1989