Forum: NGINX Nginx worker processors D state and high I/O utilization!

13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2013-07-16 07:00
(Received via mailing list)
Hello,

      We're using nginx-1.2.8 to serve large static files for video
streaming. However all nginx worker_processes are in "D" state and HDD
I/O
utilization is 99%.

[root@DNTX010 ~]# ps aux |grep nginx
root      3046  0.0  0.0  20272   688 ?        Ss   20:39   0:00 nginx:
master process nginx
nginx     3047  3.2  0.9  94480 74808 ?        D    20:39   0:03 nginx:
worker process
nginx     3048  1.4  0.3  52104 31388 ?        D    20:39   0:01 nginx:
worker process
nginx     3049  0.2  0.1  33156 12156 ?        S    20:39   0:00 nginx:
worker process
nginx     3050  0.1  0.1  29968  8844 ?        D    20:39   0:00 nginx:
worker process
nginx     3051  0.2  0.1  30332 10076 ?        D    20:39   0:00 nginx:
worker process
nginx     3052  2.7  0.8  91788 69504 ?        D    20:39   0:02 nginx:
worker process
nginx     3053  0.3  0.0  25632  5384 ?        D    20:39   0:00 nginx:
worker process
nginx     3054  0.2  0.1  36032 15852 ?        D    20:39   0:00 nginx:
worker process
nginx     3055  0.4  0.2  37592 17396 ?        D    20:39   0:00 nginx:
worker process
nginx     3056  0.2  0.1  32580 11028 ?        S    20:39   0:00 nginx:
worker process
nginx     3057  0.3  0.2  39288 19116 ?        D    20:39   0:00 nginx:
worker process
nginx     3058  0.3  0.2  41764 19744 ?        D    20:39   0:00 nginx:
worker process
nginx     3059  0.3  0.1  31124 10480 ?        D    20:39   0:00 nginx:
worker process
nginx     3060  1.0  0.3  52736 31776 ?        D    20:39   0:01 nginx:
worker process
nginx     3061  1.1  0.3  51920 29956 ?        D    20:39   0:01 nginx:
worker process
nginx     3062  1.6  0.4  58808 35548 ?        D    20:39   0:01 nginx:
worker process


[root@DNTX010 ~]# iostat -x -d 3
Linux 2.6.32-358.6.2.el6.x86_64 (DNTX010.local)         07/16/2013
_x86_64_        (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz
avgqu-sz   await  svctm  %util
sda              30.28   177.37  260.32    2.96 38169.26  1442.70
150.46     2.29    8.70   3.52  92.78

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz
avgqu-sz   await  svctm  %util
sda               4.33     0.00  544.00    0.00 34376.00     0.00
63.19    43.83   75.25   1.84 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz
avgqu-sz   await  svctm  %util
sda               9.00     6.33  547.67    0.67 34637.33    56.00
63.27    48.01   86.20   1.82 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     0.67  568.00    2.33 36024.00    29.33
63.21    54.98  101.10   1.75 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
avgrq-sz
avgqu-sz   await  svctm  %util
sda               0.00     4.33  560.33    1.33 35712.00    45.33
63.66    37.20   65.06   1.78 100.00


Nginx.conf :

http {
    include       mime.types;
    default_type  application/octet-stream;
    client_body_buffer_size 128K;
    sendfile_max_chunk 128k;
    client_max_body_size 800m;
    client_header_buffer_size 256k;
    large_client_header_buffers 4 256k;
    output_buffers 1 512k;
    server_tokens off; #Conceals nginx version
    #access_log  logs/access.log  main;
    access_log off;
    error_log warn;
    sendfile        on;
#    aio on;
 #   directio 512k;
    ignore_invalid_headers on;
    client_header_timeout  3m;
    client_body_timeout 3m;
    send_timeout     3m;
    keepalive_timeout  0;
    reset_timedout_connection on;
}

We've also tried enabling aio directive but nothing changed. Help will
be
highly appreciated.

Thanks
Shahzaib
671d9faabfe3d3382be736b93fbfa1d5?d=identicon&s=25 Steve Holdoway (Guest)
on 2013-07-16 07:07
(Received via mailing list)
Your disk maxes out at 20MB/sec read?

On Tue, 2013-07-16 at 09:59 +0500, shahzaib shahzaib wrote:
> nginx     3047  3.2  0.9  94480 74808 ?        D    20:39   0:03
> nginx: worker process
> nginx     3058  0.3  0.2  41764 19744 ?        D    20:39   0:00
>
> avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.67  568.00    2.33 36024.00    29.33
>
>     #access_log  logs/access.log  main;
>     client_body_timeout 3m;
> Thanks
> Shahzaib
>
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Steve Holdoway BSc(Hons) MNZCS
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2013-07-16 07:11
(Received via mailing list)
We're using 4XSata HDD 7200 rpm and yes it is hardly crossing 20MB/sec.
Could you please guide me a bit about the Max speed of SATA read?

Thanks for prompt reply.
671d9faabfe3d3382be736b93fbfa1d5?d=identicon&s=25 Steve Holdoway (Guest)
on 2013-07-16 07:18
(Received via mailing list)
How are you using them?? Raid 10?

On the offchance, is
http://www.greengecko.co.nz/content/western-digita...
 relevant?

Cheers,

Steve

On Tue, 2013-07-16 at 10:10 +0500, shahzaib shahzaib wrote:
> <steve@greengecko.co.nz> wrote:
>         > I/O utilization is 99%.
>         > nginx: worker process
>         0:02
>         > nginx     3056  0.2  0.1  32580 11028 ?        S    20:39
>         > nginx: worker process
>         >
>         > 150.46     2.29    8.70   3.52  92.78
>         > avgrq-sz avgqu-sz   await  svctm  %util
>         >
>         >
>         >     #access_log  logs/access.log  main;
>         >     client_body_timeout 3m;
>         >
>         --
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

--
Steve Holdoway BSc(Hons) MNZCS
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2013-07-16 07:25
(Received via mailing list)
Yes we're using raid-10.
5cede995240b2c8749c2e41d13cd33ce?d=identicon&s=25 Sergey Kandaurov (Guest)
on 2013-07-16 09:34
(Received via mailing list)
On Jul 16, 2013, at 8:59 AM, shahzaib shahzaib <shahzaib.cb@gmail.com>
wrote:
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz
avgqu-sz   await  svctm  %util
>
You are likely hitting the IOPS limit. 550r/s is quite enough to
saturate
4xSATA 7200 in RAID10.  There are reads of 64 of something per request
in average. I'd first look at what does something designated as "rsec/s"
exactly mean in linux terms here.

--
Sergey Kandaurov
pluknet@nginx.com
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2013-07-16 09:47
(Received via mailing list)
What should be my next step ? Should i buy SAS Drive with hard-raid-10 ?
Bb9c2912739ece4181267a3866c0ce76?d=identicon&s=25 Igor Sverkos (Guest)
on 2013-07-16 10:42
(Received via mailing list)
Hi,

can you tell use the HDD modell you are currently using? And are you
using real dedicated servers or some kind of virtualization?

Current SATA disks (Seagate ST1000DM-003 for example) are able to
provide an avg speed of 190MB/s (keep in mind: It's their avg speed.
So when the disk is full, the speed will drop).

When you reach the disk limit, you would add more spindles (e.g. add
more disks). Do your own math. Calculate how many IOPS one user needs
(if you cannot calculate IOPS, just calculate in MB/s for the
beginning). For example:

Your videos have an avg bit rate of ~8000kbit/s (HD videos).
To serve one stream, you would need at least 1 MB/s disk speed.

So using a disk with avg speed of ~120 MB/s will allow you to serve
~100 concurrent requests (we keep a 20% buffer, to be sure). Keep in
mind that in this example, the disk is dedicated to serve your videos
only. If you run your web, mail and streaming server on the same
disk... :>

BTW: Serving ~100 concurrent HD streams would utilize one 1 Gbit uplink.

Now, doubling the spindles (=add one more disk, RAID 0) would allow
you to serve 200 concurrent requests (you want to go with RAID 10, to
add redundancy).

Doing the calculation in MB/s is not as accurate as doing the
calculation in IOPS, but it's better than nothing.

Yes, you should buy real SAS server disks. Don't ever use green SATA
disks in servers :)

But when your current disks only provide 30MB/s make sure if it is
their limit or if there is another problem (for example the alignment
problem Sergey mentioned). That's why I ask you for the modell in the
beginning.


--
Regards,
Igor
B459184a1a76aadb382353b20988ccfa?d=identicon&s=25 edogawaconan (Guest)
on 2013-07-16 11:07
(Received via mailing list)
On Tue, Jul 16, 2013 at 5:41 PM, Igor Sverkos
<igor.sverkos@googlemail.com> wrote:
>
> Yes, you should buy real SAS server disks. Don't ever use green SATA
> disks in servers :)
>

Or SSDs for good IOPS :)

--
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.