# Aio info **URL:** \ **Category:** NGINX\ **Created:** [May 5, 2011, 7:06am UTC](https://www.ruby-forum.com/t/aio-info/206581 "2011-05-05T07:06:27Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [May 5, 2011, 7:06am UTC](https://www.ruby-forum.com/t/aio-info/206581/1 "2011-05-05T07:06:27Z")
Hi A lot of talk about AIO recently - I actually didn’t know it existed before, so I am trying it on freebsd8 # sysctl -a | grep aio kern.features.aio: 1 vfs.aio.max\_buf\_aio: 16 vfs.aio.max\_aio\_queue\_per\_proc: 256 vfs.aio.max\_aio\_per\_proc: 32 vfs.aio.unloadable: 0 vfs.aio.aiod\_lifetime: 30000 vfs.aio.aiod\_timeout: 10000 vfs.aio.num\_buf\_aio: 0 vfs.aio.num\_queue\_count: 0 vfs.aio.max\_aio\_queue: 1024 vfs.aio.target\_aio\_procs: 4 vfs.aio.num\_aio\_procs: 4 vfs.aio.max\_aio\_procs: 32 Could anyone comment what these mean and how to go about tweaking for best performance (should any be increased?) Particularly… vfs.aio.num\_queue\_count - does this monitor the amount of io waiting? ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 5, 2011, 9:12am UTC](https://www.ruby-forum.com/t/aio-info/206581/2 "2011-05-05T09:12:29Z")
On Thu, May 05, 2011 at 05:05:37AM +0000, Richard K. wrote: > vfs.aio.aiod\_timeout: 10000 > vfs.aio.num\_buf\_aio: 0 > vfs.aio.num\_queue\_count: 0 > vfs.aio.max\_aio\_queue: 1024 > vfs.aio.target\_aio\_procs: 4 > vfs.aio.num\_aio\_procs: 4 > vfs.aio.max\_aio\_procs: 32 > > Could anyone comment what these mean and how to go about tweaking for best > performance (should any be increased?) $ sysctl -d vfs.aio vfs.aio: Async IO management vfs.aio.max\_buf\_aio: Maximum buf aio requests per process (stored in the process) vfs.aio.max\_aio\_queue\_per\_proc: Maximum queued aio requests per process (stored in the process) vfs.aio.max\_aio\_per\_proc: Maximum active aio requests per process (stored in the process) vfs.aio.unloadable: Allow unload of aio (not recommended) vfs.aio.aiod\_lifetime: Maximum lifetime for idle aiod vfs.aio.aiod\_timeout: Timeout value for synchronous aio operations vfs.aio.num\_buf\_aio: Number of aio requests presently handled by the buf subsystem vfs.aio.num\_queue\_count: Number of queued aio requests vfs.aio.max\_aio\_queue: Maximum number of aio requests to queue, globally vfs.aio.target\_aio\_procs: Preferred number of ready kernel threads for async IO vfs.aio.num\_aio\_procs: Number of presently active kernel threads for async IO vfs.aio.max\_aio\_procs: Maximum number of kernel threads to use for handling async IO There are two types AIO operations inside FreeBSD kernel - raw file AIO (buf aio) and usual file AIO that uses in-kernel threads/processes. nginx uses only usual file AIO. Relevant sysctls are: vfs.aio.max\_aio\_queue: 1024 vfs.aio.max\_aio\_queue\_per\_proc: 256 vfs.aio.max\_aio\_per\_proc: 32 vfs.aio.num\_queue\_count: 0 vfs.aio.max\_aio\_procs: 32 vfs.aio.target\_aio\_procs: 4 vfs.aio.num\_aio\_procs: 4 vfs.aio.aiod\_lifetime: 30000 vfs.aio.aiod\_timeout: 10000 You may want to increase vfs.aio.max\_aio\_procs vfs.aio.max\_aio\_queue vfs.aio.max\_aio\_queue\_per\_proc vfs.aio.max\_aio\_per\_proc > Particularly.. vfs.aio.num\_queue\_count - does this monitor the amount of io > waiting? Yes. – Igor S. ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 6, 2011, 11:37am UTC](https://www.ruby-forum.com/t/aio-info/206581/3 "2011-05-06T11:37:55Z")
Is aio used while reading files from proxy\_cache? I test aio on a file every 1s (it should be in ram buffers, so no disk io needed) Usually it takes 1ms (or lower), but sometimes it spikes to over a second if disks are really busy The max\_aio\_queue doesn’t go above 3-4 during test Any ideas the cause of spikes? ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 6, 2011, 12:50pm UTC](https://www.ruby-forum.com/t/aio-info/206581/4 "2011-05-06T12:50:50Z")
On Fri, May 06, 2011 at 09:37:12AM +0000, Richard K. wrote: > Is aio used while reading files from proxy\_cache? Yes, it works. For FreeBSD I recommend this settings: ``` sendfile on; aio sendfile; tcp_nopush on; read_ahead 512k; ``` > I test aio on a file every 1s (it should be in ram buffers, so no disk io > needed) > Usually it takes 1ms (or lower), but sometimes it spikes to over a second if > disks are really busy > The max\_aio\_queue doesn’t go above 3-4 during test > Any ideas the cause of spikes? May be locks on VFS level. You can try this: ``` open_file_cache max=5000 inactive=20s; open_file_cache_valid 30s; open_file_cache_min_uses 3; ``` – Igor S. ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 6, 2011, 3:00pm UTC](https://www.ruby-forum.com/t/aio-info/206581/5 "2011-05-06T15:00:43Z")
On Fri, May 06, 2011 at 12:52:51PM +0000, Richard K. wrote: > Hi > Many thanks for your help, it is much appreciated. > I am still getting these pauses. They happen every 30 seconds on the dot. > What could be running/locking every 30 seconds? System disk (ad4) incurs usage > at this point, and this should not be being used (ad5 and ad6 are the cache disks) > (note that I increased the valid/inactive times you exampled to 300s to rule > that out) Could you run “top -S” and “systat -v 1” at time near to these peaks ? – Igor S. ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 6, 2011, 2:53pm UTC](https://www.ruby-forum.com/t/aio-info/206581/6 "2011-05-06T14:53:17Z")
Hi Many thanks for your help, it is much appreciated. I am still getting these pauses. They happen every 30 seconds on the dot. What could be running/locking every 30 seconds? System disk (ad4) incurs usage at this point, and this should not be being used (ad5 and ad6 are the cache disks) (note that I increased the valid/inactive times you exampled to 300s to rule that out) ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 6, 2011, 3:50pm UTC](https://www.ruby-forum.com/t/aio-info/206581/7 "2011-05-06T15:50:46Z")
Hi Igor, Nothing stands out when looking at those stats (other than the disk %busy going from 50/60 straight up to 100). I.e. nothing is grabbing 100% cpu or something - there is barely 2% cpu usage and appears to drop during the 2-3 second lock period… Note that during my test, nginx is rebuilding the proxy\_cache index (1,000,000’s of files that it has to read from disk to build index after a restart) - does something happen with the cache manager process every 30 seconds during rebuild? ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 9, 2011, 1:22pm UTC](https://www.ruby-forum.com/t/aio-info/206581/8 "2011-05-09T13:22:36Z")
Hello! On Mon, May 09, 2011 at 09:33:18AM +0000, Richard K. wrote: > Running nginx on freebsd 8.2 with 16 workers and aio on, when the server > encounters heavy writes, all processes are in D state > Aio queue is way below system/process limit > A request for small file (said file will be in fs buffer/cache) every second > varies wildly from 1ms to over 1s > > > From man ps: > > D Marks a process in disk (or other short term, uninter-ruptible) wait. > > What’s causing the lock? You may want to look at WCHAN instead, it should contain more information. Please note that AIO interface only allows to do async reads, but open()/stat()/… are still blocking. This may be a problem if you’ve run out of vnodes (sysctl kern.maxvnodes) or there are other reasons NAMEI lookups can’t be effectively cached by OS. Maxim D. ---
**Author:** ![rkoskela](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/ce73a5/32.png) [@rkoskela](https://www.ruby-forum.com/u/rkoskela)\ **Post date:** [May 9, 2011, 11:33am UTC](https://www.ruby-forum.com/t/aio-info/206581/9 "2011-05-09T11:33:42Z")
Running nginx on freebsd 8.2 with 16 workers and aio on, when the server encounters heavy writes, all processes are in D state Aio queue is way below system/process limit A request for small file (said file will be in fs buffer/cache) every second varies wildly from 1ms to over 1s > From man ps: > D Marks a process in disk (or other short term, uninter-ruptible) > wait. What’s causing the lock? Many thanks Richard