Forum: NGINX Performance plateau when keepalives enabled

Hi,

I am benchmarking nginx 0.7.65 with httperf simulating a typical user 
visit which lasts max 10 seconds including pause time (i.e. hit page, 
grab inline elements such as images/css/js, pause, hit another page, 
etc.)  I am serving only static html and images/css/javascript.

httperf is running from two separate client machines both within 10ms 
ping from the webserver.  While httperf is running, I am also browsing 
the site on my workstation (200ms from the webserver) to see how it 
feels for a user.

When I run the benchmark with keepalive_timeout 0, nginx is very very 
fast and accepts new connections instantly.  Concurrent connections 
never goes above 5 since the benchmark clients are so close to the 
server.  However,

When I run the benchmark with keepalive_timeout 65, nginx is very slow 
(5-20 seconds delay) to accept new connections as soon as it reaches 
1500 concurrent connections.  It plateaus here: it completes all the 
requests but it never exceeds approximately 1500 concurrent connections.

Is this normal?  Everywhere I read tells me to leave keepalive on and it 
shouldn't make any difference in nginx.  So is this a keepalive problem, 
or a problem handling >1500 concurrent connections?  What should I look 
at to fix it?

The box is 97% idle and it is not waiting on disk.

OS is Linux 2.6 with fs.file-max=65535 also before starting nginx, 
ulimit -n 65535 is called (from /etc/init.d/nginx)

There are no errors at all in the nginx error.log.  Not for file 
descriptors, or anything.  I watched error.log at "info" logging level 
and the only messages at all are the keepalive connection closing 
messages, which I believe are normal.

The nginx.conf has (amongst other things):

worker_processes 4;  # it's quad core machine
worker_rlimit_nofile 6144;
events {
    worker_connections 3072;
    use epoll;
}

http {
    sendfile on;
    tcp_push on;
    tcp_nodelay on;
}

Thanks
Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61482#msg-61482

Hello!

On Mon, Mar 08, 2010 at 11:30:52AM -0500, powderhound wrote:

[...]

> When I run the benchmark with keepalive_timeout 65, nginx is 
> very slow (5-20 seconds delay) to accept new connections as soon 
> as it reaches 1500 concurrent connections.  It plateaus here: it 
> completes all the requests but it never exceeds approximately 
> 1500 concurrent connections.
> 
> Is this normal?  Everywhere I read tells me to leave keepalive 
> on and it shouldn't make any difference in nginx.  So is this a 
> keepalive problem, or a problem handling >1500 concurrent 
> connections?  What should I look at to fix it?

Most likely you hit some network-related limit of your OS or some
firewall inbetween.  Check if firewalls have enough number of
state entries, your OS isn't limited by number of sockets, network
buffers and so on.

Maxim Dounin

> When I run the benchmark with keepalive_timeout 65, nginx is
> very slow (5-20 seconds delay) to accept new connections as soon
> as it reaches 1500 concurrent connections. It plateaus here: it
> completes all the requests but it never exceeds approximately
> 1500 concurrent connections.

Most likely you hit some network-related limit of your OS or some
firewall inbetween. Check if firewalls have enough number of
state entries, your OS isn't limited by number of sockets, network
buffers and so on.


Thank you for replying.  I should have mentioned, I have tested
with firewall completely disabled and the following sysctls which
I believe should be sufficient.

net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_max = 16777216
net.core.wmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

Is there anything else I can try?

Thanks!
Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61624#msg-61624

Yes

fs.file-max = 65535

James


On Mon, Mar 8, 2010 at 2:21 PM, powderhound <nginx-forum@nginx.us> 
wrote:
> buffers and so on.
> net.core.netdev_max_backlog = 262144
> Steve
>
> Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,61482,61624#msg-61624
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://nginx.org/mailman/listinfo/nginx



--
http://www.miami-criminallaw.com

James Matthews wrote:
>> Is there anything else I can try?
>
> fs.file-max = 65535

Thanks for replying but see original post I already
have this:

> OS is Linux 2.6 with fs.file-max=65535 also before
> starting nginx, ulimit -n 65535 is called (from
> /etc/init.d/nginx)

Any other ideas?  Is there any more information I
should post about my environment?

Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61640#msg-61640

Try these

net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1



On Mon, Mar 8, 2010 at 3:15 PM, powderhound <nginx-forum@nginx.us> 
wrote:
> > starting nginx, ulimit -n 65535 is called (from
> > /etc/init.d/nginx)
>
> Any other ideas?  Is there any more information I
> should post about my environment?
>
> Steve
>
> Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,61482,61640#msg-61640
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://nginx.org/mailman/listinfo/nginx



--
http://www.goldwatches.com

James Matthews wrote:
> net.core.netdev_max_backlog = 5000
> net.ipv4.tcp_window_scaling = 1

Thanks again!  Except I already have window_scaling = 1 although I
neglected to say.  I had backlog at 262144, I just tried with 5000 and
no change.

Are there any nginx.conf settings that may affect this or is it
likely tcp/ip stack?

Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61647#msg-61647

More information:

With keepalive_timeout 65, this is what /nginx_status looks like, and
new connections take 5-20 seconds to start delivering content.

Active connections: 1552
server accepts handled requests
 2940 2940 38965
Reading: 0 Writing: 271 Waiting: 1281

With keepalive_timeout 0, this is what /nginx_status looks like
with the same workload, new connections start receiving
content immediately.

Active connections: 2
server accepts handled requests
 5797 5797 5796
Reading: 1 Writing: 1 Waiting: 0

Can anyone suggest what to look at next?  Is there some
extra logging I can turn on that might shed light on the situation?

Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61704#msg-61704

Again more information:

When my benchmarks aren't running, I can receive via wget a
single file at 50megabyte/s from one of my benchmark clients.
Note this is not part of my nginx benchmark, I am just showing
my nginx server can receive 50mbyte/s throughput no problem.

Now, when my benchmark clients are running httperf against
the server, there is a max of 8megabyte/s throughput and
there is approx. 1500 concurrent connections open:

# cat /proc/net/sockstat
TCP: inuse 1466 orphan 26 tw 0 alloc 1467 mem 12544
UDP: inuse 3
RAW: inuse 0
FRAG: inuse 0 memory 0

This is when nginx (or the OS) becomes slow to start
serving new connections.

I tried just now, downloading the test file to my server while the
benchmark is running.  This is the file I can usually receive at
50megabyte/s.  I straced wget and it first hangs in poll()
trying to lookup the hostname.  It retries a few times and after
about 5-10 seconds it begins downloading but only at 13
kilobyte/s.

So, it's not a problem of throughput saturation.  I don't believe
it's because of any limits like file descriptors, somaxconn or
rmem/wmem, as I have maxed those (and tried them at
smaller values too.)  CPU is 97-98% idle the entire time and
there is no waiting on disk.

I honestly have no idea what can be causing this!  Any advice
would be very much appreciated.

Steve

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,61482,61726#msg-61726

На понеделник 08 март 2010 17:30:52 powderhound написа:
>     tcp_nodelay on;
> }

try increasing the backlog in your server section

Momchil

On Mon, Mar 8, 2010 at 10:00 PM, powderhound <nginx-forum@nginx.us> 
wrote:

> OS is Linux 2.6 with fs.file-max=65535 also before starting nginx, ulimit
> -n 65535 is called (from /etc/init.d/nginx)
>


> worker_rlimit_nofile 6144;
>
>
Why are you limiting fds to 6144?