Nginx worker processors D state and high I/O utilization!

Server NGINX
1

Hello,

  We're using nginx-1.2.8 to serve large static files for video

streaming. However all nginx worker_processes are in “D” state and HDD
I/O
utilization is 99%.

[root@DNTX010 ~]# ps aux |grep nginx
root 3046 0.0 0.0 20272 688 ? Ss 20:39 0:00 nginx:
master process nginx
nginx 3047 3.2 0.9 94480 74808 ? D 20:39 0:03 nginx:
worker process
nginx 3048 1.4 0.3 52104 31388 ? D 20:39 0:01 nginx:
worker process
nginx 3049 0.2 0.1 33156 12156 ? S 20:39 0:00 nginx:
worker process
nginx 3050 0.1 0.1 29968 8844 ? D 20:39 0:00 nginx:
worker process
nginx 3051 0.2 0.1 30332 10076 ? D 20:39 0:00 nginx:
worker process
nginx 3052 2.7 0.8 91788 69504 ? D 20:39 0:02 nginx:
worker process
nginx 3053 0.3 0.0 25632 5384 ? D 20:39 0:00 nginx:
worker process
nginx 3054 0.2 0.1 36032 15852 ? D 20:39 0:00 nginx:
worker process
nginx 3055 0.4 0.2 37592 17396 ? D 20:39 0:00 nginx:
worker process
nginx 3056 0.2 0.1 32580 11028 ? S 20:39 0:00 nginx:
worker process
nginx 3057 0.3 0.2 39288 19116 ? D 20:39 0:00 nginx:
worker process
nginx 3058 0.3 0.2 41764 19744 ? D 20:39 0:00 nginx:
worker process
nginx 3059 0.3 0.1 31124 10480 ? D 20:39 0:00 nginx:
worker process
nginx 3060 1.0 0.3 52736 31776 ? D 20:39 0:01 nginx:
worker process
nginx 3061 1.1 0.3 51920 29956 ? D 20:39 0:01 nginx:
worker process
nginx 3062 1.6 0.4 58808 35548 ? D 20:39 0:01 nginx:
worker process

[root@DNTX010 ~]# iostat -x -d 3
Linux 2.6.32-358.6.2.el6.x86_64 (DNTX010.local) 07/16/2013
x86_64 (8 CPU)

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz
avgqu-sz await svctm %util
sda 30.28 177.37 260.32 2.96 38169.26 1442.70
150.46 2.29 8.70 3.52 92.78

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz
avgqu-sz await svctm %util
sda 4.33 0.00 544.00 0.00 34376.00 0.00
63.19 43.83 75.25 1.84 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz
avgqu-sz await svctm %util
sda 9.00 6.33 547.67 0.67 34637.33 56.00
63.27 48.01 86.20 1.82 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.67 568.00 2.33 36024.00 29.33
63.21 54.98 101.10 1.75 100.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
avgrq-sz
avgqu-sz await svctm %util
sda 0.00 4.33 560.33 1.33 35712.00 45.33
63.66 37.20 65.06 1.78 100.00

Nginx.conf :

http {
include mime.types;
default_type application/octet-stream;
client_body_buffer_size 128K;
sendfile_max_chunk 128k;
client_max_body_size 800m;
client_header_buffer_size 256k;
large_client_header_buffers 4 256k;
output_buffers 1 512k;
server_tokens off; #Conceals nginx version
#access_log logs/access.log main;
access_log off;
error_log warn;
sendfile on;

aio on;

directio 512k;

ignore_invalid_headers on;
client_header_timeout  3m;
client_body_timeout 3m;
send_timeout     3m;
keepalive_timeout  0;
reset_timedout_connection on;

}

We’ve also tried enabling aio directive but nothing changed. Help will
be
highly appreciated.

Thanks
Shahzaib

2

Your disk maxes out at 20MB/sec read?

On Tue, 2013-07-16 at 09:59 +0500, shahzaib shahzaib wrote:

nginx 3047 3.2 0.9 94480 74808 ? D 20:39 0:03
nginx: worker process
nginx 3058 0.3 0.2 41764 19744 ? D 20:39 0:00

avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.67 568.00 2.33 36024.00 29.33

#access_log  logs/access.log  main;
client_body_timeout 3m;

Thanks
Shahzaib

nginx mailing list
[email protected]
nginx Info Page

–
Steve H. BSc(Hons) MNZCS

Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa

3

We’re using 4XSata HDD 7200 rpm and yes it is hardly crossing 20MB/sec.
Could you please guide me a bit about the Max speed of SATA read?

Thanks for prompt reply.

4

How are you using them?? Raid 10?

On the offchance, is

relevant?

Cheers,

Steve

On Tue, 2013-07-16 at 10:10 +0500, shahzaib shahzaib wrote:

[email protected] wrote:
> I/O utilization is 99%.
> nginx: worker process
0:02
> nginx 3056 0.2 0.1 32580 11028 ? S 20:39
> nginx: worker process
>
> 150.46 2.29 8.70 3.52 92.78
> avgrq-sz avgqu-sz await svctm %util
>
>
> #access_log logs/access.log main;
> client_body_timeout 3m;
>
–

nginx mailing list
[email protected]
nginx Info Page

–
Steve H. BSc(Hons) MNZCS

Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa

5

Yes we’re using raid-10.

6

What should be my next step ? Should i buy SAS Drive with hard-raid-10 ?

7

On Jul 16, 2013, at 8:59 AM, shahzaib shahzaib [email protected]
wrote:

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util

You are likely hitting the IOPS limit. 550r/s is quite enough to
saturate
4xSATA 7200 in RAID10. There are reads of 64 of something per request
in average. I’d first look at what does something designated as “rsec/s”
exactly mean in linux terms here.

–
Sergey Kandaurov
[email protected]

8

Hi,

can you tell use the HDD modell you are currently using? And are you
using real dedicated servers or some kind of virtualization?

Current SATA disks (Seagate ST1000DM-003 for example) are able to
provide an avg speed of 190MB/s (keep in mind: It’s their avg speed.
So when the disk is full, the speed will drop).

When you reach the disk limit, you would add more spindles (e.g. add
more disks). Do your own math. Calculate how many IOPS one user needs
(if you cannot calculate IOPS, just calculate in MB/s for the
beginning). For example:

Your videos have an avg bit rate of ~8000kbit/s (HD videos).
To serve one stream, you would need at least 1 MB/s disk speed.

So using a disk with avg speed of ~120 MB/s will allow you to serve
~100 concurrent requests (we keep a 20% buffer, to be sure). Keep in
mind that in this example, the disk is dedicated to serve your videos
only. If you run your web, mail and streaming server on the same
disk… :>

BTW: Serving ~100 concurrent HD streams would utilize one 1 Gbit uplink.

Now, doubling the spindles (=add one more disk, RAID 0) would allow
you to serve 200 concurrent requests (you want to go with RAID 10, to
add redundancy).

Doing the calculation in MB/s is not as accurate as doing the
calculation in IOPS, but it’s better than nothing.

Yes, you should buy real SAS server disks. Don’t ever use green SATA
disks in servers :slight_smile:

But when your current disks only provide 30MB/s make sure if it is
their limit or if there is another problem (for example the alignment
problem Sergey mentioned). That’s why I ask you for the modell in the
beginning.

–
Regards,
Igor

9

On Tue, Jul 16, 2013 at 5:41 PM, Igor Sverkos
[email protected] wrote:

Yes, you should buy real SAS server disks. Don’t ever use green SATA
disks in servers :slight_smile:

Or SSDs for good IOPS :slight_smile:

–
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org