Forum: NGINX HDD util is 100% - aio questions

Hello

Right now nginx manages to put hdds in the server to high util rate.

I try to run Nginx 1.2.3 with aio support to deliver mp4 videos with the
streaming module.
I compiled the server with aio and it starts fine.
In config I set it like this

    sendfile        off;
    output_buffers 1 2m;
    #sndbuf=32K;
    aio on;
    directio 512;

I read that sendfile should be off, but it won't send video unless I 
turn
it on.
In this case does aio work at all? How can I tell, before I wait a week 
and
see that maybe HDD util is not 100% all the time anymore :P

AIO will not work with the streaming module:

http://nginx.org/en/docs/http/ngx_http_core_module.html#aio

On Linux,directiocan only be used for reading blocks that are aligned on 
512-byte boundaries (or 4K for XFS). Reading of unaligned files end is 
still made in blocking mode. The same holds true for byte range 
requests, and for FLV requests not from the beginning of a file: reading 
of unaligned data at the beginning and end of a file will be blocking. 
There is no need to turn offsendfileexplicitly as it is turned off 
automatically whendirectiois used.



What is your exact configuration? What OS do you use, what load and what 
disk and RAM configuration do you have?





________________________________

On Monday 28 January 2013 10:53:52 Cristian Rusu wrote:
> Hello
>
> Right now nginx manages to put hdds in the server to high util rate.
>
> I try to run Nginx 1.2.3 with aio support to deliver mp4 videos with the
> streaming module.
> I compiled the server with aio and it starts fine.
> In config I set it like this
[...]
>     directio 512;
>

So, you effectively switched off the page cache for any response longer
than 512 bytes.

> I read that sendfile should be off, but it won't send video unless I turn
> it on.

No, it should not for Linux.

> In this case does aio work at all? How can I tell, before I wait a week and
> see that maybe HDD util is not 100% all the time anymore :P
>

It seems, and you have almost all the data read directly from drive,
which is resulted in 100% disk utilization.

  wbr, Valentin V. Bartenev

--
http://nginx.com/support.html
http://nginx.org/en/donation.html

I have Centos Linux with RAM 64GB, 18TB RAID 10 HDDs,
CPU and load is practically 0, everything is in HDDs that hold the 
server
back

Should I remove directio?
The server is mainly for streaming large video files or for direct 
download.

Any particular setting I shuld make to nginx in this case to lower util?

Thank you

---------------------------------------------------------------
Cristian Rusu
Web Developement & Electronic Publishing

======
Crilance.com
Crilance.blogspot.com

Yes, remove directio and aio, and let the pagecache handle load. Monitor 
the performance and load carefully and report back the results.



________________________________

Pagecache?

---------------------------------------------------------------
Cristian Rusu
Web Developement & Electronic Publishing

======
Crilance.com
Crilance.blogspot.com

On 4 February 2013 07:00, Cristian Rusu <crirus@gmail.com> wrote:
> Pagecache?

Yes, page cache. http://bit.ly/12mJ61j

I read alreayd, pagecache is a plugin for Magento?
---------------------------------------------------------------
Cristian Rusu
Web Developement & Electronic Publishing

======
Crilance.com
Crilance.blogspot.com


On Mon, Feb 4, 2013 at 2:34 PM, Jonathan Matthews

No, read this (first hit for page cache @Google):

http://en.wikipedia.org/wiki/Page_cache




________________________________

64 GB of RAM might not be sufficient for keeping a significant part of
his video data in memory. Hence, depending on the number of concurrent
users and the average size of the videos Cristian wants to stream it is
entirely possible that caching videos in memory does not help at all. In
this case, he needs proper disk I/O settings.

Yes, I'm aware, thats why I told him to monitor the box carefully. 
However, async IO is not so easy to accomplish under linux, and since he 
is also using the streaming module, things can get complicated.

I wonder if switching to FreeBSD would be a better idea to avoid the 
linux AIO limitations (as in the nginx documentation).

Anyway, he needs to do some serious testing/thinking/spending time with 
it. Nobody will come up with the perfect configuration for him resolving 
all the issues.



----------------------------------------

Hello

Well, we don't have a single box, we have a few setups with large slow 
HDDs
and a couple of edge servers running on 1.5TB RAID SSD for actual 
streaming.

Right now it's stable(70% max util) as we managed to write a caching 
code
so basically the slow HDDs mainly feed the edges and only a few users at
first.

The edge servers are way faster and cope with 10Gbit bandwidth so far.

I just hoped that aio is a solution for less strain in HDDs util%.

As for page cache, that is pretty useless as web-mysql boxes does just 
fine
in serving up to 2500 connections per second.

The main issue is around getting large videos in and out of HDDs fast.

---------------------------------------------------------------
Cristian Rusu
Web Developement & Electronic Publishing

======
Crilance.com
Crilance.blogspot.com