# One mongrel with \*lots\* of close\_wait tcp connections

**URL:** <https://www.ruby-forum.com/t/one-mongrel-with-lots-of-close-wait-tcp-connections/109816>\
**Category:** Rails\
**Created:** [July 19, 2007, 10:14am UTC](https://www.ruby-forum.com/t/one-mongrel-with-lots-of-close-wait-tcp-connections/109816 "2007-07-19T10:14:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mike-moscow](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/53a042/32.png) [@mike-moscow](https://www.ruby-forum.com/u/mike-moscow)\
**Post date:** [July 19, 2007, 10:14am UTC](https://www.ruby-forum.com/t/one-mongrel-with-lots-of-close-wait-tcp-connections/109816/1 "2007-07-19T10:14:53Z")

</div>

- cross posted to the mongrel mailing list\*

Hi, I’m running into a strange issue where one mongrel will sometimes  
develop hundreds of CLOSE\_WAIT TCP connections, mostly to apache (I  
think –  
see sample lsof output below). I haven’t had a chance to get the  
mongrel  
with this behavior into USR1 debug mode yet. I wrote a little loop  
below that will catch it next time.  
This issue occurs a couple times a day on average at seemingly random  
times. The  
problem goes away within a minute or two, probably after a restart of  
the  
mongrel.

I’m probably doing something crazy to cause this behavior, but I’m  
having  
trouble figuring out exactly what the problem is. It probably has to  
do  
with the fact that my mongrels get files off of amazon s3 for some  
requests. We do HTTPClient.get(url) for some s3 urls. I’m setting  
up  
dnsmasq now, by the way, but it’s not up yet.

My next steps are to get the mongrel into USR1 debugging mode and to  
see  
what actions are causing the problem, and to install dnsmasq and  
cacti. I  
think I’ve got a good guess which action is responsible – it’s  
probably the  
one that gets the files from s3, but I’ll make sure.

If you have any thoughts or other ideas, please let me know. Thanks a  
ton  
for your help!

Some sample output from lsof:

lsof -i -P | grep CLOSE\_ | grep mongrel

CLOSE\_WAIT --mysite  
mongrel\_r 831 root 6u IPv4 95162945 TCP  
localhost.localdomain  
:8011-\>localhost.localdomain:59311 (CLOSE\_WAIT)  
mongrel\_r 831 root 9u IPv4 95161753 TCP  
mysite.com:49269-\>[xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80)\<http://xxx-xxx-xxx-  
[xxx.amazon.com/](http://xxx.amazon.com/)\>(CLOSE\_WAIT)  
mongrel\_r 831 root 11u IPv4 95162093 TCP [mysite.com](http://mysite.com):  
49339-\>  
[xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80) \<[http://xxx-xxx-xxx-xxx.amazon.com/](http://xxx-xxx-xxx-xxx.amazon.com/)

> (CLOSE\_WAIT)  
> mongrel\_r 831 root 14u IPv4 95162202 TCP [mysite.com](http://mysite.com):  
> 49373-\>  
> [xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80) \<[http://xxx-xxx-xxx-xxx.amazon.com/](http://xxx-xxx-xxx-xxx.amazon.com/)  
> (CLOSE\_WAIT)  
> mongrel\_r 831 root 15u IPv4 95162229 TCP [mysite.com](http://mysite.com):  
> 49380-\>  
> [xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80) \<[http://xxx-xxx-xxx-xxx.amazon.com/](http://xxx-xxx-xxx-xxx.amazon.com/)  
> (CLOSE\_WAIT)  
> mongrel\_r 831 root 16u IPv4 95162319 TCP  
> mysite.com:49399-\>[xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80)\<http://xxx-xxx-xxx-  
> [xxx.amazon.com/](http://xxx.amazon.com/)\>(CLOSE\_WAIT)  
> mongrel\_r 831 root 17u IPv4 95162477 TCP [mysite.com](http://mysite.com):  
> 49436-\>  
> [xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80) \<[http://xxx-xxx-xxx-xxx.amazon.com/](http://xxx-xxx-xxx-xxx.amazon.com/)  
> (CLOSE\_WAIT)  
> mongrel\_r 831 root 19u IPv4 95163082 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59348 (CLOSE\_WAIT)  
> mongrel\_r 831 root 20u IPv4 95163221 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain :59387 (CLOSE\_WAIT)  
> mongrel\_r 831 root 21u IPv4 95163360 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59426 (CLOSE\_WAIT)  
> mongrel\_r 831 root 22u IPv4 95161592 TCP [mysite.com](http://mysite.com):  
> 49227 →   
> [xxx-xxx-xxx-xxx.amazon.com:80](http://xxx-xxx-xxx-xxx.amazon.com:80) \<[http://xxx-xxx-xxx-xxx.amazon.com/](http://xxx-xxx-xxx-xxx.amazon.com/)  
> (CLOSE\_WAIT)  
> mongrel\_r 831 root 23u IPv4 95163507 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain :59463 (CLOSE\_WAIT)  
> mongrel\_r 831 root 24u IPv4 95163675 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59495 (CLOSE\_WAIT)  
> mongrel\_r 831 root 25u IPv4 95164041 TCP  
> localhost.localdomain:8011-\>  
> localhost.localdomain:59586 (CLOSE\_WAIT)  
> mongrel\_r 831 root 26u IPv4 95164181 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59618 (CLOSE\_WAIT)  
> mongrel\_r 831 root 27u IPv4 95164293 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59641 (CLOSE\_WAIT)  
> mongrel\_r 831 root 28u IPv4 95164441 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59670 (CLOSE\_WAIT)  
> mongrel\_r 831 root 29u IPv4 95164607 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59705 (CLOSE\_WAIT)  
> mongrel\_r 831 root 30u IPv4 95164748 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59746 (CLOSE\_WAIT)  
> mongrel\_r 831 root 31u IPv4 95164895 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59786 (CLOSE\_WAIT)  
> mongrel\_r 831 root 32u IPv4 95165064 TCP  
> localhost.localdomain  
> :8011-\>localhost.localdomain:59830 (CLOSE\_WAIT)

etc. this goes on for 700 lines, where the mongrel on port 8011 has  
roughly  
700 CLOSE\_WAIT TCP connections to the 30-60k port range (to apache, I  
believe). All of these close\_waits are for the mongrel on port 8011,  
in  
this case. Also, any ideas what’s going on with the close\_wait  
connections to amazon s3?

lsof -i -P | grep CLOSE\_ | grep mongrel | wc -l  
703

netstat | grep 56586 # an example port  
tcp 1 0 localhost.localdomain:8011 localhost.localdomain:  
56586  
CLOSE\_WAIT  
tcp 0 0 localhost.localdomain :56586 localhost.localdomain:  
8011  
FIN\_WAIT2  
getnameinfo failed  
getnameinfo failed

#background loop to set the bad mongrel to debug mode during the  
close\_wait period  
def debug\_mongrel\_loop  
sleep (60) until (`lsof -i -P | grep CLOSE_WAIT | grep mongrel | wc -l`).to\_i \> 100  
`killall -USR1 mongrel_rails`  
AdminMailer.deliver\_mongrel\_debug\_mode\_turned\_on # optional  
email alert

```
# sleep 2 minutes, and then undo the debug mode.
sleep(120)
`killall -USR1 mongrel_rails`

```

end