Handling 500k concurrent connections on Linux

I was wondering if anyone had some tips/guidelines for scaling Nginx on
Linux to >500k concurrent connections. Playing with the
nginx_http_push_stream module in streaming mode. Noticing periodic slow
accept and/or response headers. I’ve scoured the Internet
looking/learning ways to tune Nginx/Linux but I think I’ve exhausted my
abilities.

Any help would be appreciated.

Hardware
Dual Nehalem 5520
24G RAM
Intel 82576 (igb)
Ubuntu 12.04.1 (3.2.0-31-generic x86_64)

Thank You,

John W

John,

On Oct 9, 2012, at 11:10 PM, John W. wrote:

Dual Nehalem 5520
24G RAM
Intel 82576 (igb)
Ubuntu 12.04.1 (3.2.0-31-generic x86_64)

Thank You,

John W

I’d assume you’ve already checked/fixed the following, right?

  1. Error logs - anything wrong seen in there?

  2. http://nginx.org/en/docs/ngx_core_module.html#multi_accept and
    http://nginx.org/en/docs/ngx_core_module.html#accept_mutex - did you try
    it on/off?

  3. file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx &&
    ulimit, worker_rlimit_nofile)

  4. sysctl net.ipv4.ip_local_port_range (if you’re aiming at proxying all
    those connections to upstreams)

Additional information about what’s happening in all those 500k
connections might be helpful, as well as the relevant configuration
section :slight_smile:

Hope this helps


AA @ nginx
http://nginx.com/support.html


nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
By increasing that to a much larger number I haven’t noticed slow
accepts/response headers. Also for reference, backlog on a listen
socket is silently limited to net.core.somaxconn (which defaults to
128) so make sure to increase that and other necessary tunings as well.

On Tuesday 16 October 2012 11:30:29 John W. wrote:

After a bit more digging I discovered that Nginx sets the backlog on the
listen socket to only 511 (at least on Linux), not the -1 in the docs.
[…]

Docs: “By default, backlog is set to -1 on FreeBSD, and to 511 on other
platforms.” @ http://nginx.org/r/listen

wbr, Valentin V. Bartenev


http://nginx.com/support.html
http://nginx.org/en/donation.html

  1. Error logs are clean (except for some 404s)

  2. nginx.conf and sysctl.conf:
    https://gist.github.com/0b3b52050254e273ff11

Set TX/RX descriptors to 4096/4096 (maximum):
ethtool -G eth1 tx 4096 rx 4096

Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC

Don’t know exact amount, but a good majority of the connections are
sitting idle for 90s before being closed.

Some graphs on the network interface for past couple days:
https://www.dropbox.com/s/0bl304ulhqp6a4n/push_stream_network.png

Thank you,

John W

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs