# Is sendfile\_max\_chunk needed when using aio on FreeBSD 9? **URL:** \ **Category:** NGINX\ **Created:** [February 8, 2012, 1:50pm UTC](https://www.ruby-forum.com/t/is-sendfile-max-chunk-needed-when-using-aio-on-freebsd-9/216581 "2012-02-08T13:50:53Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [February 8, 2012, 1:50pm UTC](https://www.ruby-forum.com/t/is-sendfile-max-chunk-needed-when-using-aio-on-freebsd-9/216581/1 "2012-02-08T13:50:53Z")
Hi all, Following Igor’s advice [1], I’m using the following configuration for file handling on FreeBSD 9.0 amd64: sendfile on; aio sendfile; tcp\_nopush on; read\_ahead 256K; My understanding of this setup is that sendfile, which is a synchronous operation, is restricted to sending only the bytes that are already buffered in memory. Once the data has to be read from disk, sendfile returns and nginx issues a 1-byte aio\_read operation to buffer an additional 256 KB of data. The question is whether it is beneficial to use sendfile\_max\_chunk option is this configuration as well? Since sendfile is guaranteed to return as soon as it runs out of buffered data, is there any real advantage to further restricting how much it can send in a single call? By the way, is tcp\_nopush needed here just to make sure that the first packet, which contains headers, doesn’t get sent out without any data in it? I think this would also prevent transmission of partial packets when sendfile runs out of data to send and nginx has to wait for the aio\_read to finish. Wouldn’t it be better in this case to send the packet without waiting for disk I/O? - Max [1] [aio info](http://mailman.nginx.org/pipermail/nginx/2011-May/026651.html) ---
**Author:** ![xamdam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d78d45/32.png) [@xamdam](https://www.ruby-forum.com/u/xamdam)\ **Post date:** [February 8, 2012, 2:46pm UTC](https://www.ruby-forum.com/t/is-sendfile-max-chunk-needed-when-using-aio-on-freebsd-9/216581/2 "2012-02-08T14:46:57Z")
Hello! On Wed, Feb 08, 2012 at 07:49:29AM -0500, Maxim K. wrote: > My understanding of this setup is that sendfile, which is a > synchronous operation, is restricted to sending only the bytes that > are already buffered in memory. Once the data has to be read from > disk, sendfile returns and nginx issues a 1-byte aio\_read operation to > buffer an additional 256 KB of data. > > The question is whether it is beneficial to use sendfile\_max\_chunk > option is this configuration as well? Since sendfile is guaranteed to > return as soon as it runs out of buffered data, is there any real > advantage to further restricting how much it can send in a single > call? It may make sense as in exreame conditions (i.e. if sendfile(NODISKIO) fails to send anything right after aio preread) nginx will fallback to normal sendfile() (without SF\_NODISKIO). On the other hand, if the above happens - it means you have problem anyway. > By the way, is tcp\_nopush needed here just to make sure that the first > packet, which contains headers, doesn’t get sent out without any data > in it? I think this would also prevent transmission of partial packets > when sendfile runs out of data to send and nginx has to wait for the > aio\_read to finish. Wouldn’t it be better in this case to send the > packet without waiting for disk I/O? The tcp\_nopush is expected to prevent transmission of incomplete packets. I see no problem here. Maxim D. ---
**Author:** ![xamdam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d78d45/32.png) [@xamdam](https://www.ruby-forum.com/u/xamdam)\ **Post date:** [February 8, 2012, 3:49pm UTC](https://www.ruby-forum.com/t/is-sendfile-max-chunk-needed-when-using-aio-on-freebsd-9/216581/3 "2012-02-08T15:49:18Z")
On Wed, Feb 8, 2012 at 8:46 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > aio sendfile; > > option is this configuration as well? Since sendfile is guaranteed to > > return as soon as it runs out of buffered data, is there any real > > advantage to further restricting how much it can send in a single > > call? > > It may make sense as in exreame conditions (i.e. if > sendfile(NODISKIO) fails to send anything right after aio preread) > nginx will fallback to normal sendfile() (without SF\_NODISKIO). > On the other hand, if the above happens - it means you have > problem anyway. I see. So there shouldn’t be any harm in specifying something like ‘sendfile\_max\_chunk 512K’, since that limit would almost never come into play. Would I see anything in the log files when the fallback to the normal sendfile happens? > > By the way, is tcp\_nopush needed here just to make sure that the first > > packet, which contains headers, doesn’t get sent out without any data > > in it? I think this would also prevent transmission of partial packets > > when sendfile runs out of data to send and nginx has to wait for the > > aio\_read to finish. Wouldn’t it be better in this case to send the > > packet without waiting for disk I/O? > > The tcp\_nopush is expected to prevent transmission of incomplete > packets. I see no problem here. The way I was thinking about it is that if the system has a packet that is half-full from the last sendfile call, and is now going to spend some number of ms buffering the next chunk, then for interactivity and throughput reasons it may be better to send the packet now. It would consume a bit more bandwidth for the TCP/IP headers, but as long as the entire packet is sent before the aio\_read call is finished, you win on the throughput front. This might be completely insignificant, I’m not sure. - Max ---
**Author:** ![xamdam](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d78d45/32.png) [@xamdam](https://www.ruby-forum.com/u/xamdam)\ **Post date:** [February 8, 2012, 4:34pm UTC](https://www.ruby-forum.com/t/is-sendfile-max-chunk-needed-when-using-aio-on-freebsd-9/216581/4 "2012-02-08T16:34:41Z")
Hello! On Wed, Feb 08, 2012 at 09:48:24AM -0500, Maxim K. wrote: > > > sendfile on; > > > The question is whether it is beneficial to use sendfile\_max\_chunk > > I see. So there shouldn’t be any harm in specifying something like > ‘sendfile\_max\_chunk 512K’, since that limit would almost never come > into play. > > Would I see anything in the log files when the fallback to the normal > sendfile happens? There will be alert about “sendfile(…) returned busy again”. > The way I was thinking about it is that if the system has a packet > that is half-full from the last sendfile call, and is now going to > spend some number of ms buffering the next chunk, then for > interactivity and throughput reasons it may be better to send the > packet now. > > It would consume a bit more bandwidth for the TCP/IP headers, but as > long as the entire packet is sent before the aio\_read call is > finished, you win on the throughput front. This might be completely > insignificant, I’m not sure. If you don’t care about pps, you’ll probably won’t set tcp\_nopush at all. Maxim D.