# Handling 500k concurrent connections on Linux **URL:** \ **Category:** NGINX\ **Created:** [October 9, 2012, 9:10pm UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568 "2012-10-09T21:10:41Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![aris](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/258eb7/32.png) [@aris](https://www.ruby-forum.com/u/aris)\ **Post date:** [October 9, 2012, 9:10pm UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/1 "2012-10-09T21:10:41Z")
I was wondering if anyone had some tips/guidelines for scaling Nginx on Linux to \>500k concurrent connections. Playing with the nginx\_http\_push\_stream module in streaming mode. Noticing periodic slow accept and/or response headers. I’ve scoured the Internet looking/learning ways to tune Nginx/Linux but I think I’ve exhausted my abilities. Any help would be appreciated. Hardware Dual Nehalem 5520 24G RAM Intel 82576 (igb) Ubuntu 12.04.1 (3.2.0-31-generic x86\_64) Thank You, John W ---
**Author:** ![princenamor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/c0e974/32.png) [@princenamor](https://www.ruby-forum.com/u/princenamor)\ **Post date:** [October 10, 2012, 11:05am UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/2 "2012-10-10T11:05:34Z")
John, On Oct 9, 2012, at 11:10 PM, John W. wrote: > Dual Nehalem 5520 > 24G RAM > Intel 82576 (igb) > Ubuntu 12.04.1 (3.2.0-31-generic x86\_64) > > Thank You, > > John W I’d assume you’ve already checked/fixed the following, right? 1. Error logs - anything wrong seen in there? 2. [Core functionality](http://nginx.org/en/docs/ngx_core_module.html#multi_accept) and [Core functionality](http://nginx.org/en/docs/ngx_core_module.html#accept_mutex) - did you try it on/off? 3. file descriptors limits (cat /proc/sys/fs/file-max, sudo - nginx && ulimit, worker\_rlimit\_nofile) 4. sysctl net.ipv4.ip\_local\_port\_range (if you’re aiming at proxying all those connections to upstreams) Additional information about what’s happening in all those 500k connections might be helpful, as well as the relevant configuration section 🙂 Hope this helps – AA @ nginx > **[Technical Support for NGINX and NGINX Plus Software](https://www.nginx.com/support/)** > > Get the help you need with support directly from the authors and maintainers of NGINX Open Source and NGINX Plus. ---
**Author:** ![princenamor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/c0e974/32.png) [@princenamor](https://www.ruby-forum.com/u/princenamor)\ **Post date:** [October 10, 2012, 12:06pm UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/3 "2012-10-10T12:06:56Z")
* * * nginx mailing list removed\_email\_address@domain.invalid [http://mailman.nginx.org/mailman/listinfo/nginx](http://mailman.nginx.org/mailman/listinfo/nginx) ---
**Author:** ![princenamor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/c0e974/32.png) [@princenamor](https://www.ruby-forum.com/u/princenamor)\ **Post date:** [October 16, 2012, 9:31am UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/4 "2012-10-16T09:31:14Z")
After a bit more digging I discovered that Nginx sets the backlog on the listen socket to only 511 (at least on Linux), not the -1 in the docs. By increasing that to a much larger number I haven’t noticed slow accepts/response headers. Also for reference, backlog on a listen socket is silently limited to net.core.somaxconn (which defaults to 128) so make sure to increase that and other necessary tunings as well. ---
**Author:** ![princenamor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/c0e974/32.png) [@princenamor](https://www.ruby-forum.com/u/princenamor)\ **Post date:** [October 16, 2012, 10:56am UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/5 "2012-10-16T10:56:01Z")
On Tuesday 16 October 2012 11:30:29 John W. wrote: > After a bit more digging I discovered that Nginx sets the backlog on the > listen socket to only 511 (at least on Linux), not the -1 in the docs. > […] Docs: “By default, backlog is set to -1 on FreeBSD, and to 511 on other platforms.” @ [Module ngx\_http\_core\_module](http://nginx.org/r/listen) wbr, Valentin V. Bartenev – > **[Technical Support for NGINX and NGINX Plus Software](https://www.nginx.com/support/)** > > Get the help you need with support directly from the authors and maintainers of NGINX Open Source and NGINX Plus. [http://nginx.org/en/donation.html](http://nginx.org/en/donation.html) ---
**Author:** ![princenamor](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/c0e974/32.png) [@princenamor](https://www.ruby-forum.com/u/princenamor)\ **Post date:** [October 10, 2012, 10:12pm UTC](https://www.ruby-forum.com/t/handling-500k-concurrent-connections-on-linux/223568/6 "2012-10-10T22:12:31Z")
1. Error logs are clean (except for some 404s) 2. nginx.conf and sysctl.conf: [nginx-push-stream-module configuration for \>1MM concurrent subscribers · GitHub](https://gist.github.com/0b3b52050254e273ff11) Set TX/RX descriptors to 4096/4096 (maximum): ethtool -G eth1 tx 4096 rx 4096 Disabled irqbalanced and pinned IRQs to CPU0-7 for NIC Don’t know exact amount, but a good majority of the connections are sitting idle for 90s before being closed. Some graphs on the network interface for past couple days: [![](https://dl.dropboxusercontent.com/s/0bl304ulhqp6a4n/push_stream_network.png) ](https://dl.dropboxusercontent.com/s/0bl304ulhqp6a4n/push_stream_network.png) Thank you, John W