Possible cached file corruption with aio_write enabled


I’ve recently upgraded one of the nginx servers within a caching
(proxy_cache module) cluster from 1.8.1 to 1.10 and soon after I noticed
unusually high number of various errors only on that server which I
eventually pin pointed to a mismatch between the actual cached file size
disk and size reported in file metadata (either content-length or
else). Apparently, cached files on 1.10 are cut short and miss up to few
hundred kilobytes at their ends (for files over 100 MB in total size).
For a
few files I checked, the larger file was, more content it missed at the
Eventually I found the culprit to be aio_write which I enabled the same
I upgraded nginx. Disabling it and removing all already cached files
resolved the problem.

Relevant directives:

thread_pool default threads=4 max_queue=65536;
sendfile on;
aio            threads=default;
aio_write   on;
# output_buffers are used if sendfile is not used
# output_buffers 8 512k;
read_ahead  1;

proxy_cache_path  /cache  ...  use_temp_path=on;
proxy_buffering                 on;
proxy_buffers           32 64k;
proxy_busy_buffers_size         128k;
proxy_temp_file_write_size      128k;

OS is Linux 4.1.


Posted at Nginx Forum: