Nginx serving large files - performance issues with more than ~800-1000 connections

Server NGINX
1

Hi,

I have a cluster of 10 nginx 1.2.0 servers, on Linux. They primarily
serve large files.

Whenever the number of ESTABLISHED connections to nginx is above
800-1000, the things get very slow.

I.e. it can take a minute or more before nginx starts serving such a
connection; then, the file is served very slow (started from a server in
the same rack):

wget -O /dev/null
http://server/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat

I’ve tried different tuning parameters (nginx, sysctl etc.), but they
don’t seem to change much.

The only thing which helps is starting one more nginx instance, on a
different port.

Then, this second instance serves the files just fine. I.e. with the
number of established connections above 800-1000, this one is slow:

PORT=80; wget -O /dev/null
http://server:$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat

The second instance running on port 82 will reply fast and serve files
fast:

PORT=82; wget -O /dev/null
http://server:$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat

Does it suggest nginx issues? Because the second nginx instance serves
the files fine.

Or maybe some system / sysctl parameters?

–
Tomasz Chmielewski
http://www.ptraveler.com

2

Hello!

On Thu, May 24, 2012 at 02:28:56PM +0700, Tomasz Chmielewski wrote:

wget -O /dev/null
the number of established connections above 800-1000, this one
http://server:$PORT/content/7a35859b7d91ca48fef7a3e2a9bc6fc8.dat

Does it suggest nginx issues? Because the second nginx instance
serves the files fine.

Or maybe some system / sysctl parameters?

It suggests you are disk-bound and all nginx workers are busy
waiting for I/O operations. Try looking here for basic
optimization steps:

http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html

Maxim D.

3

On 05/24/2012 06:25 PM, Maxim D. wrote:

Does it suggest nginx issues? Because the second nginx instance
serves the files fine.

Or maybe some system / sysctl parameters?

It suggests you are disk-bound and all nginx workers are busy
waiting for I/O operations. Try looking here for basic
optimization steps:

All workers in 'D' state using sendfile

I’ve tried to follow these recommendations, but don’t really see any
improvement.

The systems are not disk bound (see below).

Even if I try to fetch a file which is stored in tmpfs, it is slow - 20,
30 secs, even more, like here:

[root@da1 ~]# time curl ca3/404/404.html
curl: (7) couldn’t connect to host

real 1m3.204s
user 0m0.000s
sys 0m0.000s

I see it only when the number of established connections to nginx is
around 700 and more (most serving large files, so the connections are
long-lived).

Disk load:

root@ca2:~# iostat -k 1
(…)

avg-cpu: %user %nice %system %iowait %steal %idle
21.23 0.00 25.69 2.41 0.00 50.66

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 26.00 1240.00 0.00 1240 0
sdc 23.00 2324.00 0.00 2324 0
sdd 12.00 916.00 0.00 916 0
sde 9.00 532.00 0.00 532 0
sdf 9.00 532.00 0.00 532 0
sdg 9.00 532.00 0.00 532 0
sdh 22.00 2196.00 0.00 2196 0
sdi 32.00 1044.00 172.00 1044 172
sdj 5.00 20.00 0.00 20 0
sdk 13.00 1044.00 0.00 1044 0
sdl 17.00 1556.00 0.00 1556 0
sdm 18.00 940.00 0.00 940 0

root@ca2:~# iostat -x 1

avg-cpu: %user %nice %system %iowait %steal %idle
21.81 0.00 29.46 4.46 0.00 44.26

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00
sdb 0.00 26.00 6.00 2.00 1536.00 224.00
220.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 18.00 0.00 2656.00 0.00
147.56 0.04 2.22 1.11 2.00
sdd 0.00 33.00 4.00 3.00 1024.00 288.00
187.43 0.00 0.00 0.00 0.00
sde 0.00 26.00 4.00 2.00 768.00 224.00
165.33 0.00 0.00 0.00 0.00
sdf 0.00 23.00 1.00 2.00 192.00 200.00
130.67 0.02 6.67 6.67 2.00
sdg 0.00 27.00 6.00 2.00 560.00 232.00
99.00 0.09 11.25 2.50 2.00
sdh 0.00 39.00 28.00 19.00 2392.00 464.00
60.77 0.24 5.11 0.85 4.00
sdi 0.00 0.00 8.00 0.00 1792.00 0.00
224.00 0.02 2.50 1.25 1.00
sdj 0.00 25.00 0.00 2.00 0.00 216.00
108.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 8.00 0.00 2048.00 0.00
256.00 0.03 3.75 2.50 2.00
sdl 0.00 28.00 14.00 2.00 1152.00 240.00
87.00 0.00 0.00 0.00 0.00
sdm 0.00 60.00 4.00 3.00 1024.00 504.00
218.29 0.01 1.43 1.43 1.00

–
Tomasz Chmielewski
http://www.ptraveler.com

4

Hello!

On Fri, Jul 06, 2012 at 10:32:08PM +0800, Tomasz Chmielewski wrote:

All workers in 'D' state using sendfile

I’ve tried to follow these recommendations, but don’t really see any
improvement.

The systems are not disk bound (see below).

If your system isn’t disk bound these recommendations aren’t
likely to help, indeed. You have to find what’s bottleneck in
your case first.

On the other hand, the fact that second nginx instance running on
different port doesn’t have problems (as indicated in your
original message) suggests there is some blocking which makes
first instance slow.

Try looking into what nginx processes do when this happens.
Something like

ps -eopid,user,args,wchan | grep nginx

should show where it blocks.

I see it only when the number of established connections to
nginx is around 700 and more (most serving large files, so the
connections are long-lived).

Actually, this is expected behaviour for disk blocking: nginx
worker processes are busy waiting for disk I/O and can’t accept
and handle new connections in a timely manner. It doesn’t matter
where requested file resides as the pause is a result of nginx
worker processes being blocked due to other requests.

[…]

Maxim D.

5

On 07/07/2012 01:17 AM, Tomasz Chmielewski wrote:

ps -eopid,user,args,wchan | grep nginx

should show where it blocks.

OK, it’s the time of the day when the traffic drops (Asia based
deployment), below ~600 connections, so I no longer see this “blocking”.
Will follow up tomorrow.

OK, the traffic is back, I’m finally able to run “ps
-eopid,user,args,wchan”. I see 16 nginx processes in
“wait_answer_interruptible” state (which matches the worker_processes
setting on this server).

root@ca2:~# ps aux|grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1900 0.0 0.0 7608 844 pts/1 S+ 02:48 0:00 grep D

root@ca2:~# ps aux|grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1902 0.0 0.0 7608 848 pts/1 S+ 02:48 0:00 grep D

root@ca2:~# netstat -tpna | grep -c ESTABLI
637

root@ca2:~# ps aux|grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1906 0.0 0.0 7608 844 pts/1 S+ 02:48 0:00 grep D

root@ca2:~# ps -eopid,user,args,wchan | grep nginx
1908 root grep nginx pipe_wait
25101 root nginx: master process /usr/ rt_sigsuspend
29353 www-data nginx: worker process wait_answer_interruptible
29354 www-data nginx: worker process wait_answer_interruptible
29355 www-data nginx: worker process wait_answer_interruptible
29356 www-data nginx: worker process wait_answer_interruptible
29357 www-data nginx: worker process wait_answer_interruptible
29358 www-data nginx: worker process wait_answer_interruptible
29359 www-data nginx: worker process wait_answer_interruptible
29360 www-data nginx: worker process wait_answer_interruptible
29361 www-data nginx: worker process wait_answer_interruptible
29362 www-data nginx: worker process wait_answer_interruptible
29363 www-data nginx: worker process wait_answer_interruptible
29364 www-data nginx: worker process wait_answer_interruptible
29365 www-data nginx: worker process wait_answer_interruptible
29366 www-data nginx: worker process wait_answer_interruptible
29367 www-data nginx: worker process wait_answer_interruptible
29368 www-data nginx: worker process wait_answer_interruptible

–
Tomasz Chmielewski
http://www.ptravever.com

6

On 07/07/2012 12:17 AM, Maxim D. wrote:

should show where it blocks.
OK, it’s the time of the day when the traffic drops (Asia based
deployment), below ~600 connections, so I no longer see this “blocking”.
Will follow up tomorrow.

Actually, this is expected behaviour for disk blocking: nginx
worker processes are busy waiting for disk I/O and can’t accept
and handle new connections in a timely manner. It doesn’t matter
where requested file resides as the pause is a result of nginx
worker processes being blocked due to other requests.

OK, so, what would be the conclusion? Start hundreds of worker
processes?
Say, 512 or even 1024 instead of 32 I’m running now?

Note that I don’t see any blocked processes with:

ps aux | grep D

Rarely, there will be 1-3 nginx processes there, but they disappear very
fast (i.e. immediately after I run the next ps, they are no longer
there).

–
Tomasz Chmielewski
http://www.ptraveler.com

7

Hello!

On Sat, Jul 07, 2012 at 10:54:14AM +0800, Tomasz Chmielewski wrote:

“wait_answer_interruptible” state (which matches the
worker_processes setting on this server).

Trivial grep though Linux sources suggests this is blocking on
fuse filesystem accesses. So after all it’s disk blocking, at
least from nginx point of view.

All basic optimization steps still apply, though correct
optimization of your filesystem probably much more important here.

(Just in case, correct optimization for network filesystems like
nfs is to avoid their use with nginx and use proxying instead.)

Maxim D.