# Dropped connections on our Nginx **URL:** \ **Category:** NGINX\ **Created:** [June 20, 2012, 4:53am UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783 "2012-06-20T04:53:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\ **Post date:** [June 20, 2012, 4:53am UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/1 "2012-06-20T04:53:46Z")
Hi list I have a problem with dropped connections on an Nginx cluster that has up to 100k requests per minute per Nginx instance. It seems that in around 1 of 10`000 requests that are sent to our Nginx, the TCP connection just gets reset by the server. At first I was guessing that some values in the /etc/sysctl.conf are maybe causing this problem, because we have modified multiple TCP related values there. But after resetting all of them to the default, the connection resets still kept happening. I am guessing the problem must be related to Nginx and not to a kernel setting because in our traffic only around 25% of all requests are POSTs and the rest are GETs, but more than 90% of the requests where the problem appears are POSTs. I don’t think that the kernel can be aware if a request is POST or GET. The problem happens on many different URLs, mostly ones where we POST to, so it does not seem to be related to any rewrite rules. I have tcpdumped the problem and I can see that the request was sent correctly by the client. But after the request was received by the Nginx, it only sends back a packet with the ACK and FIN flags set. So the connection gets killed and most of the browsers display some empty pages or “zero sized reply” errors. The fact that the FIN is sent by the server makes me assume that the problem cannot be related to network hardware. Also we have this problem on all Nginx instances inside that cluster, so I don’t think its related to broken networking hardware. When the problem happens, i see statements like this one in the access log. As you can see the Nginx says HTTP status code and length both are 0: - - [20/Jun/2012:04:13:23 +0200] “POST /userProfile/rateResult?userId=&\_csrf\_token=7e23ef60c67800c4765d32b0536fc536&rate=5 HTTP/1.1” 0 0 “” “Mozilla/5.0 (X11; U; Linux x86\_64; en-US; rv:1.9.1.6) Gecko/20091216 Mandriva Linux/1.9.1.6-0.1mdv2010.0 (2010.0) Firefox/3.5.6” What i also find very interesting is that the problem can happen at any time, so it does not seem to be related to the load or number of requests on the Nginx. In the morning hours we have less than 5% of the traffic of the evening hours, and still I sometimes see this problem appearing in the morning. My Nginx config is very long, so its too long to post it here. So I only post the parts which i think might be important, without all the rewrite rules: user wwwrun www; worker\_processes 64; worker\_rlimit\_nofile 524288; events { worker\_connections 32768; use epoll; multi\_accept on; } http { sendfile on; tcp\_nopush on; keepalive\_requests 0; recursive\_error\_pages on; large\_client\_header\_buffers 4 16k; What I also found via tcpdump is that on the requests where this problem appears, the Nginx receives the incoming request and then sends the correct request to the FastCGI backend and also receives the correct answer from the backend, but before the answer from the backend comes back (less than 300ms), it already resets the client’s connection. Just in case this matters anyhow, this is my sysctl.conf: net.ipv4.icmp\_echo\_ignore\_broadcasts = 1 net.ipv4.conf.all.rp\_filter = 1 fs.inotify.max\_user\_watches = 65536 net.ipv4.conf.default.promote\_secondaries = 1 net.ipv4.conf.all.promote\_secondaries = 1 net.ipv4.ip\_forward = 0 net.ipv4.conf.lo.arp\_ignore = 1 net.ipv4.conf.lo.arp\_announce = 2 net.ipv4.conf.all.arp\_ignore = 1 net.ipv4.conf.all.arp\_announce = 2 net.netfilter.nf\_conntrack\_max = 262144 net.nf\_conntrack\_max = 262144 net.ipv4.tcp\_max\_syn\_backlog = 30000 net.ipv4.tcp\_max\_tw\_buckets = 2000000 net.core.netdev\_max\_backlog = 50000 net.ipv4.tcp\_tw\_reuse = 0 net.ipv4.tcp\_tw\_recycle = 1 net.ipv4.tcp\_fin\_timeout = 3 net.ipv4.tcp\_keepalive\_time = 120 net.core.wmem\_max = 8388608 net.core.rmem\_max = 8388608 net.ipv4.tcp\_rmem = 4096 87380 8388608 net.ipv4.tcp\_wmem = 4096 87380 8388608 net.core.somaxconn = 1024 kernel.pid\_max = 65536 net.ipv4.conf.all.log\_martians = 0 net.ipv4.conf.default.log\_martians = 0 net.ipv4.conf.lo.log\_martians = 0 net.ipv4.conf.eth0.log\_martians = 0 net.ipv4.conf.eth1.log\_martians = 0 Our operating system is SuSE Linux Enterprise 11.0. The Nginx configure params are the following: nginx version: nginx/1.2.1 built by gcc 4.3.2 [gcc-4\_3-branch revision 141291] (SUSE Linux) configure arguments: --prefix=/usr/local/nginx-1.2.1 –error-log-path=/var/log/nginx/error.log –http-log-path=/var/log/nginx/access.log –with-http\_stub\_status\_module --without-http\_autoindex\_module –without-http\_geo\_module --without-http\_map\_module –without-http\_referer\_module --without-http\_limit\_conn\_module –without-http\_empty\_gif\_module --without-mail\_pop3\_module –without-mail\_imap\_module --without-mail\_smtp\_module –with-http\_geoip\_module --with-pcre=/usr/local/src/nginx/pcre-8.30 –add-module=3rd/agentzh-nginx-eval-module-4eb2a02 –add-module=3rd/ngx\_http\_log\_request\_speed –add-module=3rd/replay-ngx\_http\_generate\_secure\_download\_links-4c1a46a –add-module=3rd/agentzh-memc-nginx-module-8befc56 –add-module=3rd/agentzh-echo-nginx-module-080c0a1 –add-module=3rd/replay-ngx\_http\_php\_memcache\_standard\_balancer-4f7dcba –add-module=3rd/masterzen-nginx-upload-progress-module-a788dea –add-module=3rd/replay-ngx\_http\_php\_session-30f69b3 –add-module=3rd/simpl-ngx\_devel\_kit-24202b4 –add-module=3rd/chaoslawful-lua-nginx-module-c5be5ff –add-module=3rd/replay-ngx\_http\_lower\_upper\_case-44958e0 –add-module=3rd/gnosek-nginx-upstream-fair-a18b409 In the dmesg I cannot see anything suspicious, there are no segfaults or related networking messages. I have already tried setting the Nginx error log to some high log level, but I didn’t see anything related to my problem, even at times when I saw that the problem is happening. Now I don’t really know what else to check anymore… I would be really glad if somebody had some ideas? Thanks for help, Mauro ---
**Author:** ![Mauro\_Stettler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Mauro\_Stettler](https://www.ruby-forum.com/u/Mauro_Stettler)\ **Post date:** [June 20, 2012, 6:55am UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/2 "2012-06-20T06:55:56Z")
whats ur fs ulimit set to? Sent from my iPhone ---
**Author:** ![Mauro\_Stettler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Mauro\_Stettler](https://www.ruby-forum.com/u/Mauro_Stettler)\ **Post date:** [June 20, 2012, 7:09am UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/3 "2012-06-20T07:09:22Z")
its set to a high value: rsid-a-57:/etc/nginx # ulimit -n 524288 i think if the problem would be the fs ulimit, i should see some error related to that no more files can be opened. but there is no such error. ---
**Author:** ![Mauro\_Stettler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Mauro\_Stettler](https://www.ruby-forum.com/u/Mauro_Stettler)\ **Post date:** [June 20, 2012, 1:58pm UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/4 "2012-06-20T13:58:09Z")
In the meantime I found that there is a log entry when this problem happens: 2012/06/20 13:14:33 [alert] 4340#0: \*23199 http request count is zero while sending to client, client: , server: , request: “POST /userProfile/updateVisitCountAndVisitHistory/user\_id//gender/w/nickname/geilpopperrin/membership/normal/hash/bb7e2305b293458dc8e9043486a6a455 HTTP/1.1”, upstream: “fastcgi://10.20.0.159:9000”, host: “”, referrer: “” If I understand that error right, it means that when Nginx tried to build the reply to the client, it somehow lost the context to which request this is associated to? As far as I see the error is outputted from this part of the source: src/http/ngx\_http\_request.c: 2983 if (r-\>count == 0) { 2984 ngx\_log\_error(NGX\_LOG\_ALERT, c-\>log, 0, “http request count is zero”); 2985 } What would be a possible reason why the r-\>count is 0? Do I have to assume that there is a bug in one of the 3th party modules which are used? Mauro On Wed, Jun 20, 2012 at 1:08 PM, Mauro Stettler ---
**Author:** ![Mauro\_Stettler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Mauro\_Stettler](https://www.ruby-forum.com/u/Mauro_Stettler)\ **Post date:** [June 20, 2012, 2:56pm UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/5 "2012-06-20T14:56:01Z")
Hello! On Wed, Jun 20, 2012 at 07:57:13PM +0800, Mauro Stettler wrote: > request this is associated to? As far as I see the error is outputted > from this part of the source: > > src/http/ngx\_http\_request.c: > 2983 if (r-\>count == 0) { > 2984 ngx\_log\_error(NGX\_LOG\_ALERT, c-\>log, 0, “http request > count is zero”); > 2985 } > > What would be a possible reason why the r-\>count is 0? The message means that nginx detected request reference count being incorrect. This in turn means there is a bug somewhere. > Do I have to assume that there is a bug in one of the 3th party > modules which are used? Likely yes. Maxim D. ---
**Author:** ![Mauro\_Stettler](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Mauro\_Stettler](https://www.ruby-forum.com/u/Mauro_Stettler)\ **Post date:** [June 20, 2012, 2:49pm UTC](https://www.ruby-forum.com/t/dropped-connections-on-our-nginx/220783/6 "2012-06-20T14:49:00Z")
Hello! On Wed, Jun 20, 2012 at 10:52:36AM +0800, Mauro Stettler wrote: […] > When the problem happens, i see statements like this one in the access > log. As you can see the Nginx says HTTP status code and length both > are 0: > - - [20/Jun/2012:04:13:23 +0200] "POST /userProfile/rateResult?userId=&\_csrf\_token=7e23ef60c67800c4765d32b0536fc536&rate=5 > HTTP/1.1" 0 0 “” “Mozilla/5.0 (X11; U; Linux x86\_64; en-US; > rv:1.9.1.6) Gecko/20091216 Mandriva Linux/1.9.1.6-0.1mdv2010.0 > (2010.0) Firefox/3.5.6” The status 0 suggests there is a major problem which caused request to be terminated abnormally before status was set. This might happen if there is lack of some resources (e.g. no memory) - this should have an entry in error log though. Alternatively, this might happen due to some bug, which doesn’t seems unrealistic specially given the number of 3rd party modules you use. […] > Now I don’t really know what else to check anymore… I would be > really glad if somebody had some ideas? I would recommend to: 1. Try to reproduce the problem without 3rd party modules/patches. 2. Try to obtain debug log of a problematic request. See here for more details: > **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)** > > ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com Maxim D.