HDD util is 100% - aio questions

Server NGINX
1

Hello

Right now nginx manages to put hdds in the server to high util rate.

I try to run Nginx 1.2.3 with aio support to deliver mp4 videos with the
streaming module.
I compiled the server with aio and it starts fine.
In config I set it like this

sendfile        off;
output_buffers 1 2m;
#sndbuf=32K;
aio on;
directio 512;

I read that sendfile should be off, but it won’t send video unless I
turn
it on.
In this case does aio work at all? How can I tell, before I wait a week
and
see that maybe HDD util is not 100% all the time anymore :stuck_out_tongue:

2

AIO will not work with the streaming module:

http://nginx.org/en/docs/http/ngx_http_core_module.html#aio

On Linux,directiocan only be used for reading blocks that are aligned on
512-byte boundaries (or 4K for XFS). Reading of unaligned files end is
still made in blocking mode. The same holds true for byte range
requests, and for FLV requests not from the beginning of a file: reading
of unaligned data at the beginning and end of a file will be blocking.
There is no need to turn offsendfileexplicitly as it is turned off
automatically whendirectiois used.

What is your exact configuration? What OS do you use, what load and what
disk and RAM configuration do you have?

3

On Monday 28 January 2013 10:53:52 Cristian R. wrote:

Hello

Right now nginx manages to put hdds in the server to high util rate.

I try to run Nginx 1.2.3 with aio support to deliver mp4 videos with the
streaming module.
I compiled the server with aio and it starts fine.
In config I set it like this
[…]
directio 512;

So, you effectively switched off the page cache for any response longer
than 512 bytes.

I read that sendfile should be off, but it won’t send video unless I turn
it on.

No, it should not for Linux.

In this case does aio work at all? How can I tell, before I wait a week and
see that maybe HDD util is not 100% all the time anymore :stuck_out_tongue:

It seems, and you have almost all the data read directly from drive,
which is resulted in 100% disk utilization.

wbr, Valentin V. Bartenev

–

http://nginx.org/en/donation.html

4

I have Centos Linux with RAM 64GB, 18TB RAID 10 HDDs,
CPU and load is practically 0, everything is in HDDs that hold the
server
back

Should I remove directio?
The server is mainly for streaming large video files or for direct
download.

Any particular setting I shuld make to nginx in this case to lower util?

Thank you

Cristian R.
Web Developement & Electronic Publishing

======
Crilance.com

5

Yes, remove directio and aio, and let the pagecache handle load. Monitor
the performance and load carefully and report back the results.

6

Pagecache?

Cristian R.
Web Developement & Electronic Publishing

======
Crilance.com

7

I read alreayd, pagecache is a plugin for Magento?

Cristian R.
Web Developement & Electronic Publishing

======
Crilance.com

On Mon, Feb 4, 2013 at 2:34 PM, Jonathan M.

8

On 4 February 2013 07:00, Cristian R. [email protected] wrote:

Pagecache?

Yes, page cache. LMGTFY - Let Me Google That For You

9

No, read this (first hit for page cache @Google):

10

Yes, I’m aware, thats why I told him to monitor the box carefully.
However, async IO is not so easy to accomplish under linux, and since he
is also using the streaming module, things can get complicated.

I wonder if switching to FreeBSD would be a better idea to avoid the
linux AIO limitations (as in the nginx documentation).

Anyway, he needs to do some serious testing/thinking/spending time with
it. Nobody will come up with the perfect configuration for him resolving
all the issues.

11

64 GB of RAM might not be sufficient for keeping a significant part of
his video data in memory. Hence, depending on the number of concurrent
users and the average size of the videos Cristian wants to stream it is
entirely possible that caching videos in memory does not help at all. In
this case, he needs proper disk I/O settings.

12

Hello

Well, we don’t have a single box, we have a few setups with large slow
HDDs
and a couple of edge servers running on 1.5TB RAID SSD for actual
streaming.

Right now it’s stable(70% max util) as we managed to write a caching
code
so basically the slow HDDs mainly feed the edges and only a few users at
first.

The edge servers are way faster and cope with 10Gbit bandwidth so far.

I just hoped that aio is a solution for less strain in HDDs util%.

As for page cache, that is pretty useless as web-mysql boxes does just
fine
in serving up to 2500 connections per second.

The main issue is around getting large videos in and out of HDDs fast.

Cristian R.
Web Developement & Electronic Publishing

======
Crilance.com