Handling 500k concurrent connections on Linux

Server NGINX
1

I was wondering if anyone had some tips/guidelines for scaling Nginx on
Linux to >500k concurrent connections. Playing with the
nginx_http_push_stream module in streaming mode. Noticing periodic slow
accept and/or response headers. I’ve scoured the Internet
looking/learning ways to tune Nginx/Linux but I think I’ve exhausted my
abilities.

Any help would be appreciated.

Hardware
Dual Nehalem 5520
24G RAM
Intel 82576 (igb)
Ubuntu 12.04.1 (3.2.0-31-generic x86_64)

Thank You,

John W

2

John,

On Oct 9, 2012, at 11:10 PM, John W. wrote:

Dual Nehalem 5520
24G RAM
Intel 82576 (igb)
Ubuntu 12.04.1 (3.2.0-31-generic x86_64)

Thank You,

John W

I’d assume you’ve already checked/fixed the following, right?

  1. Error logs - anything wrong seen in there?

  2. Core functionality and
    Core functionality - did you try
    it on/off?

  3. file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx &&
    ulimit, worker_rlimit_nofile)

  4. sysctl net.ipv4.ip_local_port_range (if you’re aiming at proxying all
    those connections to upstreams)

Additional information about what’s happening in all those 500k
connections might be helpful, as well as the relevant configuration
section :slight_smile:

Hope this helps

–
AA @ nginx

3

nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

4

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
By increasing that to a much larger number I haven’t noticed slow
accepts/response headers. Also for reference, backlog on a listen
socket is silently limited to net.core.somaxconn (which defaults to
128) so make sure to increase that and other necessary tunings as well.

5

On Tuesday 16 October 2012 11:30:29 John W. wrote:

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
[…]

Docs: “By default, backlog is set to -1 on FreeBSD, and to 511 on other
platforms.” @ Module ngx_http_core_module

wbr, Valentin V. Bartenev

–

http://nginx.org/en/donation.html

6

  1. Error logs are clean (except for some 404s)

  2. nginx.conf and sysctl.conf:
    nginx-push-stream-module configuration for >1MM concurrent subscribers · GitHub

Set TX/RX descriptors to 4096/4096 (maximum):
ethtool -G eth1 tx 4096 rx 4096

Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC

Don’t know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.

Some graphs on the network interface for past couple days:

Thank you,

John W