Hi
A lot of talk about AIO recently - I actually didn’t know it existed
before, so I am trying it on freebsd8
sysctl -a | grep aio
kern.features.aio: 1
vfs.aio.max_buf_aio: 16
vfs.aio.max_aio_queue_per_proc: 256
vfs.aio.max_aio_per_proc: 32
vfs.aio.unloadable: 0
vfs.aio.aiod_lifetime: 30000
vfs.aio.aiod_timeout: 10000
vfs.aio.num_buf_aio: 0
vfs.aio.num_queue_count: 0
vfs.aio.max_aio_queue: 1024
vfs.aio.target_aio_procs: 4
vfs.aio.num_aio_procs: 4
vfs.aio.max_aio_procs: 32
Could anyone comment what these mean and how to go about tweaking for
best performance (should any be increased?)
Particularly… vfs.aio.num_queue_count - does this monitor the amount of
io waiting?
On Thu, May 05, 2011 at 05:05:37AM +0000, Richard K. wrote:
vfs.aio.aiod_timeout: 10000
vfs.aio.num_buf_aio: 0
vfs.aio.num_queue_count: 0
vfs.aio.max_aio_queue: 1024
vfs.aio.target_aio_procs: 4
vfs.aio.num_aio_procs: 4
vfs.aio.max_aio_procs: 32
Could anyone comment what these mean and how to go about tweaking for best
performance (should any be increased?)
$ sysctl -d vfs.aio
vfs.aio: Async IO management
vfs.aio.max_buf_aio: Maximum buf aio requests per process (stored in the
process)
vfs.aio.max_aio_queue_per_proc: Maximum queued aio requests per process
(stored in the process)
vfs.aio.max_aio_per_proc: Maximum active aio requests per process
(stored in the process)
vfs.aio.unloadable: Allow unload of aio (not recommended)
vfs.aio.aiod_lifetime: Maximum lifetime for idle aiod
vfs.aio.aiod_timeout: Timeout value for synchronous aio operations
vfs.aio.num_buf_aio: Number of aio requests presently handled by the buf
subsystem
vfs.aio.num_queue_count: Number of queued aio requests
vfs.aio.max_aio_queue: Maximum number of aio requests to queue, globally
vfs.aio.target_aio_procs: Preferred number of ready kernel threads for
async IO
vfs.aio.num_aio_procs: Number of presently active kernel threads for
async IO
vfs.aio.max_aio_procs: Maximum number of kernel threads to use for
handling async IO
There are two types AIO operations inside FreeBSD kernel - raw file
AIO (buf aio) and usual file AIO that uses in-kernel threads/processes.
nginx uses only usual file AIO. Relevant sysctls are:
vfs.aio.max_aio_queue: 1024
vfs.aio.max_aio_queue_per_proc: 256
vfs.aio.max_aio_per_proc: 32
vfs.aio.num_queue_count: 0
vfs.aio.max_aio_procs: 32
vfs.aio.target_aio_procs: 4
vfs.aio.num_aio_procs: 4
vfs.aio.aiod_lifetime: 30000
vfs.aio.aiod_timeout: 10000
You may want to increase
vfs.aio.max_aio_procs
vfs.aio.max_aio_queue
vfs.aio.max_aio_queue_per_proc
vfs.aio.max_aio_per_proc
Particularly… vfs.aio.num_queue_count - does this monitor the amount of io
waiting?
Yes.
–
Igor S.
Is aio used while reading files from proxy_cache?
I test aio on a file every 1s (it should be in ram buffers, so no disk
io needed)
Usually it takes 1ms (or lower), but sometimes it spikes to over a
second if disks are really busy
The max_aio_queue doesn’t go above 3-4 during test
Any ideas the cause of spikes?
On Fri, May 06, 2011 at 09:37:12AM +0000, Richard K. wrote:
Is aio used while reading files from proxy_cache?
Yes, it works. For FreeBSD I recommend this settings:
sendfile on;
aio sendfile;
tcp_nopush on;
read_ahead 512k;
I test aio on a file every 1s (it should be in ram buffers, so no disk io
needed)
Usually it takes 1ms (or lower), but sometimes it spikes to over a second if
disks are really busy
The max_aio_queue doesn’t go above 3-4 during test
Any ideas the cause of spikes?
May be locks on VFS level. You can try this:
open_file_cache max=5000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 3;
–
Igor S.
On Fri, May 06, 2011 at 12:52:51PM +0000, Richard K. wrote:
Hi
Many thanks for your help, it is much appreciated.
I am still getting these pauses. They happen every 30 seconds on the dot.
What could be running/locking every 30 seconds? System disk (ad4) incurs usage
at this point, and this should not be being used (ad5 and ad6 are the cache disks)
(note that I increased the valid/inactive times you exampled to 300s to rule
that out)
Could you run “top -S” and “systat -v 1” at time near to these peaks ?
–
Igor S.
Hi
Many thanks for your help, it is much appreciated.
I am still getting these pauses. They happen every 30 seconds on the
dot.
What could be running/locking every 30 seconds? System disk (ad4) incurs
usage at this point, and this should not be being used (ad5 and ad6 are
the cache disks)
(note that I increased the valid/inactive times you exampled to 300s to
rule that out)
Hi Igor,
Nothing stands out when looking at those stats (other than the disk
%busy going from 50/60 straight up to 100). I.e. nothing is grabbing
100% cpu or something - there is barely 2% cpu usage and appears to drop
during the 2-3 second lock period…
Note that during my test, nginx is rebuilding the proxy_cache index
(1,000,000’s of files that it has to read from disk to build index after
a restart) - does something happen with the cache manager process every
30 seconds during rebuild?
Hello!
On Mon, May 09, 2011 at 09:33:18AM +0000, Richard K. wrote:
Running nginx on freebsd 8.2 with 16 workers and aio on, when the server
encounters heavy writes, all processes are in D state
Aio queue is way below system/process limit
A request for small file (said file will be in fs buffer/cache) every second
varies wildly from 1ms to over 1s
From man ps:
D Marks a process in disk (or other short term, uninter-ruptible) wait.
What’s causing the lock?
You may want to look at WCHAN instead, it should contain more
information.
Please note that AIO interface only allows to do async reads, but
open()/stat()/… are still blocking. This may be a problem if
you’ve run out of vnodes (sysctl kern.maxvnodes) or there are
other reasons NAMEI lookups can’t be effectively cached by OS.
Maxim D.
Running nginx on freebsd 8.2 with 16 workers and aio on, when the server
encounters heavy writes, all processes are in D state
Aio queue is way below system/process limit
A request for small file (said file will be in fs buffer/cache) every
second varies wildly from 1ms to over 1s
From man ps:
D Marks a process in disk (or other short term, uninter-ruptible)
wait.
What’s causing the lock?
Many thanks
Richard