# Performance plateau when keepalives enabled

**URL:** <https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787>\
**Category:** NGINX\
**Created:** [March 8, 2010, 5:31pm UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787 "2010-03-08T17:31:21Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 8, 2010, 5:31pm UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/1 "2010-03-08T17:31:21Z")

</div>

Hi,

I am benchmarking nginx 0.7.65 with httperf simulating a typical user  
visit which lasts max 10 seconds including pause time (i.e. hit page,  
grab inline elements such as images/css/js, pause, hit another page,  
etc.) I am serving only static html and images/css/javascript.

httperf is running from two separate client machines both within 10ms  
ping from the webserver. While httperf is running, I am also browsing  
the site on my workstation (200ms from the webserver) to see how it  
feels for a user.

When I run the benchmark with keepalive\_timeout 0, nginx is very very  
fast and accepts new connections instantly. Concurrent connections  
never goes above 5 since the benchmark clients are so close to the  
server. However,

When I run the benchmark with keepalive\_timeout 65, nginx is very slow  
(5-20 seconds delay) to accept new connections as soon as it reaches  
1500 concurrent connections. It plateaus here: it completes all the  
requests but it never exceeds approximately 1500 concurrent connections.

Is this normal? Everywhere I read tells me to leave keepalive on and it  
shouldn’t make any difference in nginx. So is this a keepalive problem,  
or a problem handling \>1500 concurrent connections? What should I look  
at to fix it?

The box is 97% idle and it is not waiting on disk.

OS is Linux 2.6 with fs.file-max=65535 also before starting nginx,  
ulimit -n 65535 is called (from /etc/init.d/nginx)

There are no errors at all in the nginx error.log. Not for file  
descriptors, or anything. I watched error.log at “info” logging level  
and the only messages at all are the keepalive connection closing  
messages, which I believe are normal.

The nginx.conf has (amongst other things):

worker\_processes 4; # it’s quad core machine  
worker\_rlimit\_nofile 6144;  
events {  
worker\_connections 3072;  
use epoll;  
}

http {  
sendfile on;  
tcp\_push on;  
tcp\_nodelay on;  
}

Thanks  
Steve

Posted at Nginx Forum:

> **[Performance plateau when keepalives enabled](https://forum.nginx.org/read.php?2,61482,61482#msg-61482)**
>
> Hi, I am benchmarking nginx 0.7.65 with httperf simulating a typical user visit which lasts max 10 seconds including pause time (i.e. hit page, grab inline elements such as images/css/js, pause, hit another page, etc.) I am serving only static html...

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 8, 2010, 11:06pm UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/2 "2010-03-08T23:06:10Z")

</div>

Hello!

On Mon, Mar 08, 2010 at 11:30:52AM -0500, powderhound wrote:

[…]

> When I run the benchmark with keepalive\_timeout 65, nginx is  
> very slow (5-20 seconds delay) to accept new connections as soon  
> as it reaches 1500 concurrent connections. It plateaus here: it  
> completes all the requests but it never exceeds approximately  
> 1500 concurrent connections.
> 
> Is this normal? Everywhere I read tells me to leave keepalive  
> on and it shouldn’t make any difference in nginx. So is this a  
> keepalive problem, or a problem handling \>1500 concurrent  
> connections? What should I look at to fix it?

Most likely you hit some network-related limit of your OS or some  
firewall inbetween. Check if firewalls have enough number of  
state entries, your OS isn’t limited by number of sockets, network  
buffers and so on.

Maxim D.

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 8, 2010, 11:21pm UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/3 "2010-03-08T23:21:58Z")

</div>

> When I run the benchmark with keepalive\_timeout 65, nginx is  
> very slow (5-20 seconds delay) to accept new connections as soon  
> as it reaches 1500 concurrent connections. It plateaus here: it  
> completes all the requests but it never exceeds approximately  
> 1500 concurrent connections.

Most likely you hit some network-related limit of your OS or some  
firewall inbetween. Check if firewalls have enough number of  
state entries, your OS isn’t limited by number of sockets, network  
buffers and so on.

Thank you for replying. I should have mentioned, I have tested  
with firewall completely disabled and the following sysctls which  
I believe should be sufficient.

net.core.rmem\_max = 16777216  
net.core.rmem\_default = 16777216  
net.core.wmem\_max = 16777216  
net.core.wmem\_default = 16777216  
net.core.netdev\_max\_backlog = 262144  
net.core.somaxconn = 262144  
net.ipv4.tcp\_syncookies = 1  
net.ipv4.tcp\_max\_orphans = 262144  
net.ipv4.tcp\_max\_syn\_backlog = 262144  
net.ipv4.tcp\_synack\_retries = 2  
net.ipv4.tcp\_syn\_retries = 2

Is there anything else I can try?

Thanks!  
Steve

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,61482,61624#msg-61624](http://forum.nginx.org/read.php?2,61482,61624#msg-61624)

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 12:08am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/4 "2010-03-09T00:08:13Z")

</div>

Yes

fs.file-max = 65535

James

On Mon, Mar 8, 2010 at 2:21 PM, powderhound [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> buffers and so on.  
> net.core.netdev\_max\_backlog = 262144  
> Steve
> 
> Posted at Nginx Forum:  
> [http://forum.nginx.org/read.php?2,61482,61624#msg-61624](http://forum.nginx.org/read.php?2,61482,61624#msg-61624)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)

–  
[http://www.miami-criminallaw.com](http://www.miami-criminallaw.com)

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 12:15am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/5 "2010-03-09T00:15:50Z")

</div>

James Matthews wrote:

> > Is there anything else I can try?
> 
> fs.file-max = 65535

Thanks for replying but see original post I already  
have this:

> OS is Linux 2.6 with fs.file-max=65535 also before  
> starting nginx, ulimit -n 65535 is called (from  
> /etc/init.d/nginx)

Any other ideas? Is there any more information I  
should post about my environment?

Steve

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,61482,61640#msg-61640](http://forum.nginx.org/read.php?2,61482,61640#msg-61640)

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 12:34am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/6 "2010-03-09T00:34:34Z")

</div>

Try these

net.core.netdev\_max\_backlog = 5000  
net.ipv4.tcp\_window\_scaling = 1

On Mon, Mar 8, 2010 at 3:15 PM, powderhound [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> > starting nginx, ulimit -n 65535 is called (from  
> > /etc/init.d/nginx)
> 
> Any other ideas? Is there any more information I  
> should post about my environment?
> 
> Steve
> 
> Posted at Nginx Forum:  
> [http://forum.nginx.org/read.php?2,61482,61640#msg-61640](http://forum.nginx.org/read.php?2,61482,61640#msg-61640)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)

–

> **[Jewelry | Outstanding](https://www.outstanding.com/jewelry)**

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 3:36am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/7 "2010-03-09T03:36:37Z")

</div>

More information:

With keepalive\_timeout 65, this is what /nginx\_status looks like, and  
new connections take 5-20 seconds to start delivering content.

Active connections: 1552  
server accepts handled requests  
2940 2940 38965  
Reading: 0 Writing: 271 Waiting: 1281

With keepalive\_timeout 0, this is what /nginx\_status looks like  
with the same workload, new connections start receiving  
content immediately.

Active connections: 2  
server accepts handled requests  
5797 5797 5796  
Reading: 1 Writing: 1 Waiting: 0

Can anyone suggest what to look at next? Is there some  
extra logging I can turn on that might shed light on the situation?

Steve

Posted at Nginx Forum:

> **[Re: Performance plateau when keepalives enabled](https://forum.nginx.org/read.php?2,61482,61704#msg-61704)**
>
> Hi, I am benchmarking nginx 0.7.65 with httperf simulating a typical user visit which lasts max 10 seconds including pause time (i.e. hit page, grab inline elements such as images/css/js, pause, hit another page, etc.) I am serving only static html...

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 12:42am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/8 "2010-03-09T00:42:46Z")

</div>

James Matthews wrote:

> net.core.netdev\_max\_backlog = 5000  
> net.ipv4.tcp\_window\_scaling = 1

Thanks again! Except I already have window\_scaling = 1 although I  
neglected to say. I had backlog at 262144, I just tried with 5000 and  
no change.

Are there any nginx.conf settings that may affect this or is it  
likely tcp/ip stack?

Steve

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,61482,61647#msg-61647](http://forum.nginx.org/read.php?2,61482,61647#msg-61647)

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 9, 2010, 5:05am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/9 "2010-03-09T05:05:03Z")

</div>

Again more information:

When my benchmarks aren’t running, I can receive via wget a  
single file at 50megabyte/s from one of my benchmark clients.  
Note this is not part of my nginx benchmark, I am just showing  
my nginx server can receive 50mbyte/s throughput no problem.

Now, when my benchmark clients are running httperf against  
the server, there is a max of 8megabyte/s throughput and  
there is approx. 1500 concurrent connections open:

# cat /proc/net/sockstat

TCP: inuse 1466 orphan 26 tw 0 alloc 1467 mem 12544  
UDP: inuse 3  
RAW: inuse 0  
FRAG: inuse 0 memory 0

This is when nginx (or the OS) becomes slow to start  
serving new connections.

I tried just now, downloading the test file to my server while the  
benchmark is running. This is the file I can usually receive at  
50megabyte/s. I straced wget and it first hangs in poll()  
trying to lookup the hostname. It retries a few times and after  
about 5-10 seconds it begins downloading but only at 13  
kilobyte/s.

So, it’s not a problem of throughput saturation. I don’t believe  
it’s because of any limits like file descriptors, somaxconn or  
rmem/wmem, as I have maxed those (and tried them at  
smaller values too.) CPU is 97-98% idle the entire time and  
there is no waiting on disk.

I honestly have no idea what can be causing this! Any advice  
would be very much appreciated.

Steve

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,61482,61726#msg-61726](http://forum.nginx.org/read.php?2,61482,61726#msg-61726)

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 21, 2010, 9:25am UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/10 "2010-03-21T09:25:00Z")

</div>

On Mon, Mar 8, 2010 at 10:00 PM, powderhound [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> OS is Linux 2.6 with fs.file-max=65535 also before starting nginx, ulimit  
> -n 65535 is called (from /etc/init.d/nginx)

> worker\_rlimit\_nofile 6144;

Why are you limiting fds to 6144?

---

<div class="post-metadata">

**Author:** ![powderhound](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/838e76/32.png) [@powderhound](https://www.ruby-forum.com/u/powderhound)\
**Post date:** [March 11, 2010, 2:25pm UTC](https://www.ruby-forum.com/t/performance-plateau-when-keepalives-enabled/184787/11 "2010-03-11T14:25:30Z")

</div>

На понеделник 08 март 2010 17:30:52 powderhound написа:

> ```
> tcp_nodelay on;
> 
> ```
> 
> }

try increasing the backlog in your server section

Momchil