Performance plateau when keepalives enabled

Server NGINX
1

Hi,

I am benchmarking nginx 0.7.65 with httperf simulating a typical user
visit which lasts max 10 seconds including pause time (i.e. hit page,
grab inline elements such as images/css/js, pause, hit another page,
etc.) I am serving only static html and images/css/javascript.

httperf is running from two separate client machines both within 10ms
ping from the webserver. While httperf is running, I am also browsing
the site on my workstation (200ms from the webserver) to see how it
feels for a user.

When I run the benchmark with keepalive_timeout 0, nginx is very very
fast and accepts new connections instantly. Concurrent connections
never goes above 5 since the benchmark clients are so close to the
server. However,

When I run the benchmark with keepalive_timeout 65, nginx is very slow
(5-20 seconds delay) to accept new connections as soon as it reaches
1500 concurrent connections. It plateaus here: it completes all the
requests but it never exceeds approximately 1500 concurrent connections.

Is this normal? Everywhere I read tells me to leave keepalive on and it
shouldn’t make any difference in nginx. So is this a keepalive problem,
or a problem handling >1500 concurrent connections? What should I look
at to fix it?

The box is 97% idle and it is not waiting on disk.

OS is Linux 2.6 with fs.file-max=65535 also before starting nginx,
ulimit -n 65535 is called (from /etc/init.d/nginx)

There are no errors at all in the nginx error.log. Not for file
descriptors, or anything. I watched error.log at “info” logging level
and the only messages at all are the keepalive connection closing
messages, which I believe are normal.

The nginx.conf has (amongst other things):

worker_processes 4; # it’s quad core machine
worker_rlimit_nofile 6144;
events {
worker_connections 3072;
use epoll;
}

http {
sendfile on;
tcp_push on;
tcp_nodelay on;
}

Thanks
Steve

Posted at Nginx Forum:

2

Hello!

On Mon, Mar 08, 2010 at 11:30:52AM -0500, powderhound wrote:

[…]

When I run the benchmark with keepalive_timeout 65, nginx is
very slow (5-20 seconds delay) to accept new connections as soon
as it reaches 1500 concurrent connections. It plateaus here: it
completes all the requests but it never exceeds approximately
1500 concurrent connections.

Is this normal? Everywhere I read tells me to leave keepalive
on and it shouldn’t make any difference in nginx. So is this a
keepalive problem, or a problem handling >1500 concurrent
connections? What should I look at to fix it?

Most likely you hit some network-related limit of your OS or some
firewall inbetween. Check if firewalls have enough number of
state entries, your OS isn’t limited by number of sockets, network
buffers and so on.

Maxim D.

3

When I run the benchmark with keepalive_timeout 65, nginx is
very slow (5-20 seconds delay) to accept new connections as soon
as it reaches 1500 concurrent connections. It plateaus here: it
completes all the requests but it never exceeds approximately
1500 concurrent connections.

Most likely you hit some network-related limit of your OS or some
firewall inbetween. Check if firewalls have enough number of
state entries, your OS isn’t limited by number of sockets, network
buffers and so on.

Thank you for replying. I should have mentioned, I have tested
with firewall completely disabled and the following sysctls which
I believe should be sufficient.

net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_max = 16777216
net.core.wmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

Is there anything else I can try?

Thanks!
Steve

Posted at Nginx Forum:

4

Yes

fs.file-max = 65535

James

On Mon, Mar 8, 2010 at 2:21 PM, powderhound [email protected]
wrote:

buffers and so on.
net.core.netdev_max_backlog = 262144
Steve

Posted at Nginx Forum:
Re: Performance plateau when keepalives enabled

nginx mailing list
[email protected]
nginx Info Page

–
http://www.miami-criminallaw.com

5

James Matthews wrote:

Is there anything else I can try?

fs.file-max = 65535

Thanks for replying but see original post I already
have this:

OS is Linux 2.6 with fs.file-max=65535 also before
starting nginx, ulimit -n 65535 is called (from
/etc/init.d/nginx)

Any other ideas? Is there any more information I
should post about my environment?

Steve

Posted at Nginx Forum:

6

Try these

net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1

On Mon, Mar 8, 2010 at 3:15 PM, powderhound [email protected]
wrote:

starting nginx, ulimit -n 65535 is called (from
/etc/init.d/nginx)

Any other ideas? Is there any more information I
should post about my environment?

Steve

Posted at Nginx Forum:
Re: Performance plateau when keepalives enabled

nginx mailing list
[email protected]
nginx Info Page

–
http://www.goldwatches.com

7

More information:

With keepalive_timeout 65, this is what /nginx_status looks like, and
new connections take 5-20 seconds to start delivering content.

Active connections: 1552
server accepts handled requests
2940 2940 38965
Reading: 0 Writing: 271 Waiting: 1281

With keepalive_timeout 0, this is what /nginx_status looks like
with the same workload, new connections start receiving
content immediately.

Active connections: 2
server accepts handled requests
5797 5797 5796
Reading: 1 Writing: 1 Waiting: 0

Can anyone suggest what to look at next? Is there some
extra logging I can turn on that might shed light on the situation?

Steve

Posted at Nginx Forum:

8

James Matthews wrote:

net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1

Thanks again! Except I already have window_scaling = 1 although I
neglected to say. I had backlog at 262144, I just tried with 5000 and
no change.

Are there any nginx.conf settings that may affect this or is it
likely tcp/ip stack?

Steve

Posted at Nginx Forum:

9

Again more information:

When my benchmarks aren’t running, I can receive via wget a
single file at 50megabyte/s from one of my benchmark clients.
Note this is not part of my nginx benchmark, I am just showing
my nginx server can receive 50mbyte/s throughput no problem.

Now, when my benchmark clients are running httperf against
the server, there is a max of 8megabyte/s throughput and
there is approx. 1500 concurrent connections open:

cat /proc/net/sockstat

TCP: inuse 1466 orphan 26 tw 0 alloc 1467 mem 12544
UDP: inuse 3
RAW: inuse 0
FRAG: inuse 0 memory 0

This is when nginx (or the OS) becomes slow to start
serving new connections.

I tried just now, downloading the test file to my server while the
benchmark is running. This is the file I can usually receive at
50megabyte/s. I straced wget and it first hangs in poll()
trying to lookup the hostname. It retries a few times and after
about 5-10 seconds it begins downloading but only at 13
kilobyte/s.

So, it’s not a problem of throughput saturation. I don’t believe
it’s because of any limits like file descriptors, somaxconn or
rmem/wmem, as I have maxed those (and tried them at
smaller values too.) CPU is 97-98% idle the entire time and
there is no waiting on disk.

I honestly have no idea what can be causing this! Any advice
would be very much appreciated.

Steve

Posted at Nginx Forum:

10

On Mon, Mar 8, 2010 at 10:00 PM, powderhound [email protected]
wrote:

OS is Linux 2.6 with fs.file-max=65535 also before starting nginx, ulimit
-n 65535 is called (from /etc/init.d/nginx)

worker_rlimit_nofile 6144;

Why are you limiting fds to 6144?

11

На понеделник 08 март 2010 17:30:52 powderhound написа:

tcp_nodelay on;

}

try increasing the backlog in your server section

Momchil