Forum: NGINX Handling 500k concurrent connections on Linux

I was wondering if anyone had some tips/guidelines for scaling Nginx on
Linux to >500k concurrent connections. Playing with the
nginx_http_push_stream module in streaming mode. Noticing periodic slow
accept and/or response headers. I've scoured the Internet
looking/learning ways to tune Nginx/Linux but I think I've exhausted my
abilities.

Any help would be appreciated.

Hardware
Dual Nehalem 5520
24G RAM
Intel 82576 (igb)
Ubuntu 12.04.1 (3.2.0-31-generic x86_64)

Thank You,

John W

John,

On Oct 9, 2012, at 11:10 PM, John Watson wrote:

> Dual Nehalem 5520
> 24G RAM
> Intel 82576 (igb)
> Ubuntu 12.04.1 (3.2.0-31-generic x86_64)
>
> Thank You,
>
> John W

I'd assume you've already checked/fixed the following, right?

1) Error logs - anything wrong seen in there?

2) http://nginx.org/en/docs/ngx_core_module.html#multi_accept and 
http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try 
it on/off?

3) file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && 
ulimit, worker_rlimit_nofile)

4) sysctl net.ipv4.ip_local_port_range (if you're aiming at proxying all 
those connections to upstreams)

Additional information about what's happening in all those 500k 
connections might be helpful, as well as the relevant configuration 
section :)

Hope this helps


--
AA @ nginx
http://nginx.com/support.html

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

1) Error logs are clean (except for some 404s)

2) nginx.conf and sysctl.conf: 
https://gist.github.com/0b3b52050254e273ff11

Set TX/RX descriptors to 4096/4096 (maximum):
    ethtool -G eth1 tx 4096 rx 4096

Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC

Don't know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.

Some graphs on the network interface for past couple days:
https://www.dropbox.com/s/0bl304ulhqp6a4n/push_str...

Thank you,

John W

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
By increasing that to a much larger number I haven't noticed slow
accepts/response headers. Also for reference, backlog on a listen
socket is silently limited to net.core.somaxconn (which defaults to
128) so make sure to increase that and other necessary tunings as well.

On Tuesday 16 October 2012 11:30:29 John Watson wrote:
> After a bit more digging I discovered that Nginx sets the backlog on the
> listen socket to only 511 (at least on Linux), not the -1 in the docs.
> [...]

Docs: "By default, backlog is set to -1 on FreeBSD, and to 511 on other
platforms." @ http://nginx.org/r/listen

 wbr, Valentin V. Bartenev

--
http://nginx.com/support.html
http://nginx.org/en/donation.html