# TCP connection problems (FIN\_WAIT2/LAST\_ACK) when upstream server is keepalived

**URL:** <https://www.ruby-forum.com/t/tcp-connection-problems-fin-wait2-last-ack-when-upstream-server-is-keepalived/193360>\
**Category:** NGINX\
**Created:** [August 16, 2010, 3:15pm UTC](https://www.ruby-forum.com/t/tcp-connection-problems-fin-wait2-last-ack-when-upstream-server-is-keepalived/193360 "2010-08-16T15:15:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dwass](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/71c47a/32.png) [@dwass](https://www.ruby-forum.com/u/dwass)\
**Post date:** [August 16, 2010, 3:15pm UTC](https://www.ruby-forum.com/t/tcp-connection-problems-fin-wait2-last-ack-when-upstream-server-is-keepalived/193360/1 "2010-08-16T15:15:43Z")

</div>

Have a strange problem. We have a configuration (this is only a part of  
the configuration) that has problems if the upstream server is a  
load-balanced server using ipvs/keepalived.

Here is the relevant part of the configuration:

upstream fb\_server {  
server foobar:8080;  
}  
server {  
listen 80;  
server\_name [f.b.com](http://f.b.com);  
location /fb/ {  
proxy\_pass http://fb\_server;  
}  
}

The hostname “foobar” resolves to an IP address that is load-balanced  
over 2 physical servers (“foo” and “bar”) using ipvs/keepalived.

What we see is that on the 2 upstream servers “foo” and “bar” there are  
a lot of connections in FIN\_WAIT2 state and on the machine where nginx  
is running we see an increasing number of connections in LAST\_ACK state.  
This would imply that nginx is not managing the upstream connection  
shutdown properly.

In order to prove that the problem is related to the load-balancing, if  
we change the upstream server to either “foo” or “bar” the problem goes  
away.

Now I realize that we can also use nginx to do the load-balancing and  
not use ipvs/keepalived, but I’d like to know why this doesn’t work  
properly in this configuration. It is also probably a useful thing for  
the nginx authors and users to know as well.

If anyone has any idea why this is behaving like this I’d love to hear  
from you.

Posted at Nginx Forum:

> **[TCP connection problems (FIN\_WAIT2/LAST\_ACK) when upstream server is keepalived](https://forum.nginx.org/read.php?2,120319,120319#msg-120319)**
>
> Have a strange problem. We have a configuration (this is only a part of the configuration) that has problems if the upstream server is a load-balanced server using ipvs/keepalived. Here is the relevant part of the configuration: upstream fb\_server {...

---

<div class="post-metadata">

**Author:** ![dwass](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/71c47a/32.png) [@dwass](https://www.ruby-forum.com/u/dwass)\
**Post date:** [August 16, 2010, 4:01pm UTC](https://www.ruby-forum.com/t/tcp-connection-problems-fin-wait2-last-ack-when-upstream-server-is-keepalived/193360/2 "2010-08-16T16:01:58Z")

</div>

This isn’t a problem with nginx, it is likely a configuration issue with  
your ipvs software. Removing ipvs from the configuration shows that. You  
might want to look at how your LB is timing out tcp sessions. In your  
case  
it shows that the connection close (fin) from your nginx server isn’t  
making  
it to your realservers “foo” and “bar”. Capture a http request at all 3  
points in the chain to determine what is happening.

Sridhar

---

<div class="post-metadata">

**Author:** ![dwass](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/71c47a/32.png) [@dwass](https://www.ruby-forum.com/u/dwass)\
**Post date:** [August 18, 2010, 9:53am UTC](https://www.ruby-forum.com/t/tcp-connection-problems-fin-wait2-last-ack-when-upstream-server-is-keepalived/193360/3 "2010-08-18T09:53:43Z")

</div>

Yes, it turns out that it isn’t a problem with nginx, it is a problem  
with the iptables configuration on the load-balancer. It isn’t a timeout  
problem though. The iptables configuration was causing the FIN packets  
to be dropped so that they weren’t passed on to the realservers. This  
post explains the problem (and possible solution) nicely:

[http://www.mail-archive.com/removed\_email\_address@domain.invalid/msg02103.html](http://www.mail-archive.com/removed_email_address@domain.invalid/msg02103.html)

-DWass

## sridhar basam Wrote:

> points in the chain to determine what is
> 
> > load-balanced server using ipvs/keepalived.  
> > proxy\_pass http://fb\_server;  
> > a lot of connections in FIN\_WAIT2 state and on  
> > “bar” the problem goes  
> > If anyone has any idea why this is behaving like  
> > this I’d love to hear  
> > from you.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,120319,120916#msg-120916](http://forum.nginx.org/read.php?2,120319,120916#msg-120916)