# Nginx serving large files - performance issues with more than ~800-1000 connections **URL:** \ **Category:** NGINX\ **Created:** [May 24, 2012, 9:29am UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990 "2012-05-24T09:29:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [May 24, 2012, 9:29am UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/1 "2012-05-24T09:29:53Z")
Hi, I have a cluster of 10 nginx 1.2.0 servers, on Linux. They primarily serve large files. Whenever the number of ESTABLISHED connections to nginx is above 800-1000, the things get very slow. I.e. it can take a minute or more before nginx starts serving such a connection; then, the file is served very slow (started from a server in the same rack): wget -O /dev/null [http://server/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat](http://server/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat) I’ve tried different tuning parameters (nginx, sysctl etc.), but they don’t seem to change much. The only thing which helps is starting one more nginx instance, on a different port. Then, this second instance serves the files just fine. I.e. with the number of established connections above 800-1000, this one is slow: PORT=80; wget -O /dev/null [http://server](http://server):$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat The second instance running on port 82 will reply fast and serve files fast: PORT=82; wget -O /dev/null [http://server](http://server):$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat Does it suggest nginx issues? Because the second nginx instance serves the files fine. Or maybe some system / sysctl parameters? – Tomasz Chmielewski [http://www.ptraveler.com](http://www.ptraveler.com) ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [May 24, 2012, 12:26pm UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/2 "2012-05-24T12:26:20Z")
Hello! On Thu, May 24, 2012 at 02:28:56PM +0700, Tomasz Chmielewski wrote: > wget -O /dev/null > the number of established connections above 800-1000, this one > [http://server](http://server):$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat > > Does it suggest nginx issues? Because the second nginx instance > serves the files fine. > > Or maybe some system / sysctl parameters? It suggests you are disk-bound and all nginx workers are busy waiting for I/O operations. Try looking here for basic optimization steps: > **[All workers in 'D' state using sendfile](http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html)** Maxim D. ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [July 6, 2012, 4:32pm UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/3 "2012-07-06T16:32:49Z")
On 05/24/2012 06:25 PM, Maxim D. wrote: > > Does it suggest nginx issues? Because the second nginx instance > > serves the files fine. > > > > Or maybe some system / sysctl parameters? > > It suggests you are disk-bound and all nginx workers are busy > waiting for I/O operations. Try looking here for basic > optimization steps: > > [All workers in 'D' state using sendfile](http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html) I’ve tried to follow these recommendations, but don’t really see any improvement. The systems are not disk bound (see below). Even if I try to fetch a file which is stored in tmpfs, it is slow - 20, 30 secs, even more, like here: [root@da1 ~]# time curl ca3/404/404.html curl: (7) couldn’t connect to host real 1m3.204s user 0m0.000s sys 0m0.000s I see it only when the number of established connections to nginx is around 700 and more (most serving large files, so the connections are long-lived). Disk load: root@ca2:~# iostat -k 1 (…) avg-cpu: %user %nice %system %iowait %steal %idle 21.23 0.00 25.69 2.41 0.00 50.66 Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn sda 0.00 0.00 0.00 0 0 sdb 26.00 1240.00 0.00 1240 0 sdc 23.00 2324.00 0.00 2324 0 sdd 12.00 916.00 0.00 916 0 sde 9.00 532.00 0.00 532 0 sdf 9.00 532.00 0.00 532 0 sdg 9.00 532.00 0.00 532 0 sdh 22.00 2196.00 0.00 2196 0 sdi 32.00 1044.00 172.00 1044 172 sdj 5.00 20.00 0.00 20 0 sdk 13.00 1044.00 0.00 1044 0 sdl 17.00 1556.00 0.00 1556 0 sdm 18.00 940.00 0.00 940 0 root@ca2:~# iostat -x 1 avg-cpu: %user %nice %system %iowait %steal %idle 21.81 0.00 29.46 4.46 0.00 44.26 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 26.00 6.00 2.00 1536.00 224.00 220.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 18.00 0.00 2656.00 0.00 147.56 0.04 2.22 1.11 2.00 sdd 0.00 33.00 4.00 3.00 1024.00 288.00 187.43 0.00 0.00 0.00 0.00 sde 0.00 26.00 4.00 2.00 768.00 224.00 165.33 0.00 0.00 0.00 0.00 sdf 0.00 23.00 1.00 2.00 192.00 200.00 130.67 0.02 6.67 6.67 2.00 sdg 0.00 27.00 6.00 2.00 560.00 232.00 99.00 0.09 11.25 2.50 2.00 sdh 0.00 39.00 28.00 19.00 2392.00 464.00 60.77 0.24 5.11 0.85 4.00 sdi 0.00 0.00 8.00 0.00 1792.00 0.00 224.00 0.02 2.50 1.25 1.00 sdj 0.00 25.00 0.00 2.00 0.00 216.00 108.00 0.00 0.00 0.00 0.00 sdk 0.00 0.00 8.00 0.00 2048.00 0.00 256.00 0.03 3.75 2.50 2.00 sdl 0.00 28.00 14.00 2.00 1152.00 240.00 87.00 0.00 0.00 0.00 0.00 sdm 0.00 60.00 4.00 3.00 1024.00 504.00 218.29 0.01 1.43 1.43 1.00 – Tomasz Chmielewski > **[Perpetual Traveler | Slow Travel Guides to Asia and Europe](https://ptraveler.com/)** > > Honest, practical travel guides from Dasza and Tomek, who spent years living month by month in cities across Asia and Europe. ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [July 6, 2012, 6:18pm UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/4 "2012-07-06T18:18:38Z")
Hello! On Fri, Jul 06, 2012 at 10:32:08PM +0800, Tomasz Chmielewski wrote: > > [All workers in 'D' state using sendfile](http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html) > > I’ve tried to follow these recommendations, but don’t really see any > improvement. > > The systems are not disk bound (see below). If your system isn’t disk bound these recommendations aren’t likely to help, indeed. You have to find what’s bottleneck in your case first. On the other hand, the fact that second nginx instance running on different port doesn’t have problems (as indicated in your original message) suggests there is some blocking which makes first instance slow. Try looking into what nginx processes do when this happens. Something like ps -eopid,user,args,wchan | grep nginx should show where it blocks. > I see it only when the number of established connections to > nginx is around 700 and more (most serving large files, so the > connections are long-lived). Actually, this is expected behaviour for disk blocking: nginx worker processes are busy waiting for disk I/O and can’t accept and handle new connections in a timely manner. It doesn’t matter where requested file resides as the pause is a result of nginx worker processes being blocked due to other requests. […] Maxim D. ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [July 7, 2012, 4:54am UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/5 "2012-07-07T04:54:56Z")
On 07/07/2012 01:17 AM, Tomasz Chmielewski wrote: > > ps -eopid,user,args,wchan | grep nginx > > > > should show where it blocks. > > OK, it’s the time of the day when the traffic drops (Asia based > deployment), below ~600 connections, so I no longer see this “blocking”. > Will follow up tomorrow. OK, the traffic is back, I’m finally able to run “ps -eopid,user,args,wchan”. I see 16 nginx processes in “wait\_answer\_interruptible” state (which matches the worker\_processes setting on this server). root@ca2:~# ps aux|grep D USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1900 0.0 0.0 7608 844 pts/1 S+ 02:48 0:00 grep D root@ca2:~# ps aux|grep D USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1902 0.0 0.0 7608 848 pts/1 S+ 02:48 0:00 grep D root@ca2:~# netstat -tpna | grep -c ESTABLI 637 root@ca2:~# ps aux|grep D USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1906 0.0 0.0 7608 844 pts/1 S+ 02:48 0:00 grep D root@ca2:~# ps -eopid,user,args,wchan | grep nginx 1908 root grep nginx pipe\_wait 25101 root nginx: master process /usr/ rt\_sigsuspend 29353 www-data nginx: worker process wait\_answer\_interruptible 29354 www-data nginx: worker process wait\_answer\_interruptible 29355 www-data nginx: worker process wait\_answer\_interruptible 29356 www-data nginx: worker process wait\_answer\_interruptible 29357 www-data nginx: worker process wait\_answer\_interruptible 29358 www-data nginx: worker process wait\_answer\_interruptible 29359 www-data nginx: worker process wait\_answer\_interruptible 29360 www-data nginx: worker process wait\_answer\_interruptible 29361 www-data nginx: worker process wait\_answer\_interruptible 29362 www-data nginx: worker process wait\_answer\_interruptible 29363 www-data nginx: worker process wait\_answer\_interruptible 29364 www-data nginx: worker process wait\_answer\_interruptible 29365 www-data nginx: worker process wait\_answer\_interruptible 29366 www-data nginx: worker process wait\_answer\_interruptible 29367 www-data nginx: worker process wait\_answer\_interruptible 29368 www-data nginx: worker process wait\_answer\_interruptible – Tomasz Chmielewski [http://www.ptravever.com](http://www.ptravever.com) ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [July 6, 2012, 7:17pm UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/6 "2012-07-06T19:17:43Z")
On 07/07/2012 12:17 AM, Maxim D. wrote: > should show where it blocks. > OK, it’s the time of the day when the traffic drops (Asia based > deployment), below ~600 connections, so I no longer see this “blocking”. > Will follow up tomorrow. > Actually, this is expected behaviour for disk blocking: nginx > worker processes are busy waiting for disk I/O and can’t accept > and handle new connections in a timely manner. It doesn’t matter > where requested file resides as the pause is a result of nginx > worker processes being blocked due to other requests. OK, so, what would be the conclusion? Start hundreds of worker processes? Say, 512 or even 1024 instead of 32 I’m running now? Note that I don’t see any blocked processes with: # ps aux | grep D Rarely, there will be 1-3 nginx processes there, but they disappear very fast (i.e. immediately after I run the next ps, they are no longer there). – Tomasz Chmielewski > **[Perpetual Traveler | Slow Travel Guides to Asia and Europe](https://ptraveler.com/)** > > Honest, practical travel guides from Dasza and Tomek, who spent years living month by month in cities across Asia and Europe. ---
**Author:** ![Tomasz\_Chmielewski](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f05b48/32.png) [@Tomasz\_Chmielewski](https://www.ruby-forum.com/u/Tomasz_Chmielewski)\ **Post date:** [July 7, 2012, 12:17pm UTC](https://www.ruby-forum.com/t/nginx-serving-large-files-performance-issues-with-more-than-800-1000-connections/219990/7 "2012-07-07T12:17:46Z")
Hello! On Sat, Jul 07, 2012 at 10:54:14AM +0800, Tomasz Chmielewski wrote: > > > > > “wait\_answer\_interruptible” state (which matches the > worker\_processes setting on this server). Trivial grep though Linux sources suggests this is blocking on fuse filesystem accesses. So after all it’s disk blocking, at least from nginx point of view. All basic optimization steps still apply, though correct optimization of your filesystem probably much more important here. (Just in case, correct optimization for network filesystems like nfs is to avoid their use with nginx and use proxying instead.) Maxim D.