# TIME\_WAITs and sustained connections was: Re: Memcached module -- unix domain socket support? (too m **URL:** \ **Category:** NGINX\ **Created:** [August 8, 2008, 8:06am UTC](https://www.ruby-forum.com/t/time-waits-and-sustained-connections-was-re-memcached-module-unix-domain-socket-support-too-m/145342 "2008-08-08T08:06:36Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Kon\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e99b99/32.png) [@Kon\_W](https://www.ruby-forum.com/u/Kon_W)\ **Post date:** [August 8, 2008, 8:06am UTC](https://www.ruby-forum.com/t/time-waits-and-sustained-connections-was-re-memcached-module-unix-domain-socket-support-too-m/145342/1 "2008-08-08T08:06:36Z")
On Thu, Aug 7, 2008 at 11:03 AM, Chavelle V. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > definitly not an expert but when I read your post, it’s the idea that I > have had. From what I read ncache works by caching items from disk into memory. The problem I have is that the media is changing every 250ms. Anyway an update, I have tweaked the kernel IPV4 parameters and even though I have at times 23,000 TIME\_WAITs, they are now being recycled. The server is sustaining about 50Mbits of traffic going through memcached-\>nginx. My connection count hovers around 220 when using nginx\_stats module. Is this the norm? I am wondering if I have an artificial bottleneck somewhere? Looking more into the unix domain sockets I am not sure if this would be the most pertinent solution to the problem. I noticed there was some talk on persistent connections to upstream servers – is this planned for memcached (I am using this model on the app server side to feed memcached and it is working great)? Cheers Kon ---
**Author:** ![Kon\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e99b99/32.png) [@Kon\_W](https://www.ruby-forum.com/u/Kon_W)\ **Post date:** [August 8, 2008, 10:50am UTC](https://www.ruby-forum.com/t/time-waits-and-sustained-connections-was-re-memcached-module-unix-domain-socket-support-too-m/145342/2 "2008-08-08T10:50:33Z")
what is the operating system? what do you mean tweaked(details pls.)? regards, lix ---
**Author:** ![Kon\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e99b99/32.png) [@Kon\_W](https://www.ruby-forum.com/u/Kon_W)\ **Post date:** [August 10, 2008, 9:16pm UTC](https://www.ruby-forum.com/t/time-waits-and-sustained-connections-was-re-memcached-module-unix-domain-socket-support-too-m/145342/3 "2008-08-10T21:16:38Z")
> what is the operating system? > what do you mean tweaked(details pls.)? I am seeing the same behavior on Centos 4.5 with nginx 0.6.32 and memcached 1.2.6. After running load tests using apache bench, I see a high number of TIME\_WAIT connections from netstat. The load tests hits nginx which fetches static response from memcached. $ uname -a Linux [geocode1.admin.zvents.com](http://geocode1.admin.zvents.com) 2.6.9-55.ELsmp #1 SMP Wed May 2 14:04:42 EDT 2007 x86\_64 x86\_64 x86\_64 GNU/Linux $ cat /etc/redhat-release CentOS release 4.5 (Final) $ netstat -an |grep -c TIME\_WAIT 6 $ ab -c 4 -n 20000 ‘[http://localhost/geocode?address=94131](http://localhost/geocode?address=94131)’ $ netstat -an |grep -c TIME\_WAIT 34444 If I run the test again, I see a number of errors in error.log that look like this: 2008/01/31 20:49:30 [crit] 24806#0: \*70538 connect() to 127.0.0.1:11211 failed (99: Cannot assign requested address) while connecting to upstream, client: 192.168.200.10, server: [www.xxxxxxxxxx.com](http://www.xxxxxxxxxx.com), request: “GET /geocode?address=94131 HTTP/1.0”, upstream: “memcached://127.0.0.1:11211”, host: “[www.xxxxxxxxxx.com](http://www.xxxxxxxxxx.com)” I believe this is a consequence of having too many sockets in TIME\_WAIT. Finally, I think the original poster was referring to the following kernel parameter. net.ipv4.ip\_local\_port\_range = 1024 65000 ---
**Author:** ![Kon\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e99b99/32.png) [@Kon\_W](https://www.ruby-forum.com/u/Kon_W)\ **Post date:** [August 11, 2008, 7:57pm UTC](https://www.ruby-forum.com/t/time-waits-and-sustained-connections-was-re-memcached-module-unix-domain-socket-support-too-m/145342/4 "2008-08-11T19:57:17Z")
On Sun, Aug 10, 2008 at 12:09 PM, Tyler K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > what is the operating system? > > what do you mean tweaked(details pls.)? > net.ipv4.ip\_local\_port\_range = 1024 65000 Indeed. Also: net.ipv4.ip\_local\_port\_range = 10000 65535 (since I have memcached and other services on port 9000-10000) net.ipv4.tcp\_tw\_recycle = 1 (fast recycle of T\_W’s) net.ipv4.tcp\_tw\_reuse = 1 (reuse T\_W’s when ‘safe’ to do so) I would advise anyone to read [http://www.ietf.org/rfc/rfc1337.txt](http://www.ietf.org/rfc/rfc1337.txt) before committing these to production. It would seem that a persistent backend connection or domain socket support for memcached would resolve this issue. In my tests I have also noticed that connecting and disconnecting from memcached has a performance impact on serving content. When I use this mechanism on my application backend my video serving FPS maxes out at 2-3 updates per second. With a persistent connection I can go upwards of 10fps. It would seem likely that the non-persistent nginx\<-\>memcached connectivity would also cause a bottleneck under high connection rates. Cheers Kon