# High number connections-writing stuck

**URL:** <https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676>\
**Category:** NGINX\
**Created:** [December 18, 2015, 6:56pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676 "2015-12-18T18:56:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rafa\_F](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/958977/32.png) [@Rafa\_F](https://www.ruby-forum.com/u/Rafa_F)\
**Post date:** [December 18, 2015, 6:56pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/1 "2015-12-18T18:56:05Z")

</div>

Hi,

Recently we added a ‘thread\_pool’ directive to our main configuration. A  
few hours later we saw a huge increase in the connections\_writing stat  
as  
reported by stub\_status module. This number reached ± 3800 and is stuck  
there since. The server in question is operating normally, but this is  
very  
strange.

Any hints on what this could be?

Some info:

- Here is a graph of the stats reported, for a server with thread\_pool  
and  
another without: [http://imgur.com/a/lF2EL](http://imgur.com/a/lF2EL)

- I don`t have older data anymore, but the jump from \<100 to ± 3800  
connections\_writing happened in two sharp jumps. The first one following  
a  
reload;

- The machines’ hardware and software are identical except for the  
thread\_pool directive in their nginx.conf. They live in two different  
data  
centers;

- Both machines are performing normally. Nothing unusual in CPU or RAM  
usage. Nginx performance is about the same.

- Reloading Nginx with ‘nginx -s reload’ does nothing. Restarting the  
process brings connections\_writing down.

Debug stuff:

mallmann# uname -a  
Linux xxx 3.8.13-98.5.2.el6uek.x86\_64 #2 SMP Tue Nov 3 18:32:04 PST 2015  
x86\_64 x86\_64 x86\_64 GNU/Linux

mallmann# nginx -V  
nginx version: nginx/1.8.0  
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)  
built with OpenSSL 1.0.1e-fips 11 Feb 2013  
TLS SNI support enabled  
configure arguments: --prefix=/usr/share/nginx  
–sbin-path=/usr/sbin/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error.log  
–http-log-path=/var/log/nginx/access.log  
–http-client-body-temp-path=/var/lib/nginx/tmp/client\_body  
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy  
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi  
–http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi  
–http-scgi-temp-path=/var/lib/nginx/tmp/scgi  
–pid-path=/var/run/nginx.pid  
–lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx  
–with-ipv6  
–with-http\_ssl\_module --with-http\_realip\_module  
–with-http\_addition\_module --with-http\_xslt\_module  
–with-http\_image\_filter\_module --with-http\_geoip\_module  
–with-http\_sub\_module --with-http\_flv\_module --with-http\_mp4\_module  
–with-http\_gunzip\_module --with-http\_gzip\_static\_module  
–with-http\_random\_index\_module --with-http\_secure\_link\_module  
–with-http\_degradation\_module --with-http\_stub\_status\_module  
–with-http\_perl\_module --with-mail --with-mail\_ssl\_module --with-pcre  
–with-google\_perftools\_module  
–add-module=/builddir/build/BUILD/nginx-1.8.0/headers-more-nginx-module-0.25  
–add-module=/builddir/build/BUILD/nginx-1.8.0/ngx\_http\_bytes\_filter\_module  
–add-module=/builddir/build/BUILD/nginx-1.8.0/echo-nginx-module-0.55  
–with-threads --with-debug --with-cc-opt=‘-O2 -g -pipe -Wall  
-Wp,-D\_FORTIFY\_SOURCE=2 -fexceptions -fstack-protector  
–param=ssp-buffer-size=4 -m64 -mtune=generic’ --with-ld-opt=’ -Wl,-E’

Affected server:

mallmann# lsof -n -u nginx | awk ‘{print $5}’ | sort | uniq -c | sort  
-nr  
4172 REG  
2140 IPv4  
100 unix  
30 CHR  
20 DIR  
20 0000  
3 sock  
1 TYPE  
mallmann# curl [http://127.0.0.1/status](http://127.0.0.1/status)  
Active connections: 5924  
server accepts handled requests  
5864099 5864099 15527178  
Reading: 0 Writing: 3883 Waiting: 2040

Normal server:

mallmann# lsof -n -u nginx | awk ‘{print $5}’ | sort | uniq -c | sort  
-nr  
4454 REG  
1967 IPv4  
100 unix  
30 CHR  
20 DIR  
20 0000  
1 unknown  
1 TYPE  
1 sock  
mallmann# curl [http://127.0.0.1/status](http://127.0.0.1/status)  
Active connections: 2096  
server accepts handled requests  
1136132 1136132 3464904  
Reading: 0 Writing: 107 Waiting: 1989

---

<div class="post-metadata">

**Author:** ![Marcelo\_MD](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Marcelo\_MD](https://www.ruby-forum.com/u/Marcelo_MD)\
**Post date:** [December 19, 2015, 6:09pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/2 "2015-12-19T18:09:51Z")

</div>

From what you provided, I do not see anything unexpected…

Enabling threads usage with thread\_pool and aio threads enables  
asynchronous file reading/writing.  
The connections\_writing stat indicates… the number of connections to  
which the server is writing a response.

Since you are working asynchronously, if the request takes much less  
time  
to process than the response to be sent, it is normal this numbers  
increases, until reaching a balance point.

You saw this number increasing when you reloaded the configuration -\>  
shows  
the new configuration has been applied immediately  
You saw this number drop on restart -\> restarting kills the current  
processes and the attached connections  
Nothing to see on CPU & RAM -\> they are probably not the bottleneck, and  
one nginx strength is to have a minimal fingerprint there, compared to  
other Web server alternatives. Have a look at I/O wait to check if the  
worker processes might be having a hard time getting content they seek  
from  
the disk. Less often, it might also be a congestion on backend  
communication if content is being read from some.

To me, it is just a matter of raton time to process request/time to  
serve  
answer. Multithreading means a worker might be answering several  
connections at the same time.  
If I am right about nginx’ design, events might be processed in any  
order,  
but each worker still only does one thing at a time and onky switch task  
when the current one is finished/waiting on something else.  
Multithreading  
changes the whole behavior.

## Hope to help,

_B. R._

---

<div class="post-metadata">

**Author:** ![Marcelo\_MD](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Marcelo\_MD](https://www.ruby-forum.com/u/Marcelo_MD)\
**Post date:** [January 18, 2016, 7:34pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/3 "2016-01-18T19:34:36Z")

</div>

Hi,

Just getting back to you. Sorry about the delay.

This behaviour was caused by a custom patch in a module. The patch was  
ending the requests without finalizing them, leaking connections.

Eventually Nginx just exploded =) Nothing like production workload to  
stress things out.

Thanks a lot.

On Sun, Dec 20, 2015 at 11:04 AM, Valentin V. Bartenev [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![Marcelo\_MD](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Marcelo\_MD](https://www.ruby-forum.com/u/Marcelo_MD)\
**Post date:** [December 20, 2015, 2:05pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/4 "2015-12-20T14:05:37Z")

</div>

On Friday 18 December 2015 15:55:47 Marcelo MD wrote:

> thread\_pool directive in their nginx.conf. They live in two different data  
> centers;
> 
> - Both machines are performing normally. Nothing unusual in CPU or RAM  
> usage. Nginx performance is about the same.
> 
> - Reloading Nginx with ‘nginx -s reload’ does nothing. Restarting the  
> process brings connections\_writing down.  
> [..]

As I understand from your message everything works well. You should  
also  
check the error\_log, if it doesn’t have anything suspicious then there  
is  
nothing to worry about.

The increased number of writing connections can be explained by  
increased  
concurrency. Now nginx processing cycle doesn’t block on disk and can  
accept more connections at the same time. All the connections that were  
waiting in listen socket before are waiting now in thread pool.

wbr, Valentin V. Bartenev

---

<div class="post-metadata">

**Author:** ![Marcelo\_MD](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Marcelo\_MD](https://www.ruby-forum.com/u/Marcelo_MD)\
**Post date:** [January 19, 2016, 9:49am UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/5 "2016-01-19T09:49:22Z")

</div>

Testingin production?  
Well you are not the only one… far from that.

## There are other means to create ‘realistic’ traffic on benches, provided you are thinking those tests well, though.

_B. R._

---

<div class="post-metadata">

**Author:** ![Marcelo\_MD](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Marcelo\_MD](https://www.ruby-forum.com/u/Marcelo_MD)\
**Post date:** [January 19, 2016, 2:29pm UTC](https://www.ruby-forum.com/t/high-number-connections-writing-stuck/243676/6 "2016-01-19T14:29:47Z")

</div>

=)

That module passed all our tests, both in dev and staging environments.  
We  
were deploying in batches and got it on the first one. We also rolled  
back  
before any incidents.

Not perfect but pretty safe, I think.

Maybe we need to be more ‘creative’ with our testing.