Hello,
I’m seeing odd disk usage statistics for my cache. The cache is
primarily of small files (images, javascript, and css) and sits on a
128GB SSD using ext2 file system mounted with “noatime” option. The
cache is all that the the drive is being used for at this time.
The attached graphs were generated by Munin using df. The data from du
correlates closely with df when run manually.
As you can see, periodically, with no set interval, disk usage rises
gradually, and then drops precipitously. I don’t see anything out of the
ordinary in the error log. The cache serves about 2.8-3 million requests
per day, up from ~2.2-2.4 million one month ago.
The cache config is:
fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m
inactive=1d max_size=80g;
where the SSD is mounted as /falcon.
On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim O. wrote:
As you can see, periodically, with no set interval, disk usage rises
gradually, and then drops precipitously. I don’t see anything out of the
ordinary in the error log. The cache serves about 2.8-3 million requests
per day, up from ~2.2-2.4 million one month ago.
The cache config is:
fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m
inactive=1d max_size=80g;
where the SSD is mounted as /falcon.
As I understand /falcon is used solely for nginx cache ?
Could you monitor inode usage also ?
Did you restart nginx in the times of the drops or not ?
Currently I have no idea why it may be.
On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim O. wrote:
The attached graphs were generated by Munin using df. The data from du
inactive=1d max_size=80g;
Did you restart nginx in the times of the drops or not ?
No.
Currently I have no idea why it may be.
I saw the similar pattern, when nginx is restarted (not reHUPed) at the
cliff
growing start point: then cache loader loads many old inactive cache
items
and sets close inactivity timers for the items. Therefore cache size
grows
lineary and at some time all these inactive items are purged almost
simultaneously causing precipitous drop.
Have you restarted nginx at the growing start points ?
On Tue, Oct 06, 2009 at 03:56:46PM +0400, Igor S. wrote:
primarily of small files (images, javascript, and css) and sits on a
I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
growing start point: then cache loader loads many old inactive cache items
and sets close inactivity timers for the items. Therefore cache size grows
lineary and at some time all these inactive items are purged almost
simultaneously causing precipitous drop.
Have you restarted nginx at the growing start points ?
This may be not a restart, but an online upgrade too.
On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor S. wrote:
cache is all that the the drive is being used for at this time.
and sets close inactivity timers for the items. Therefore cache size grows
lineary and at some time all these inactive items are purged almost
simultaneously causing precipitous drop.
Have you restarted nginx at the growing start points ?
A note: the lineary growth period is equal to an “inactive=” parameter
in
fastcgi_cache_path.
Igor S. wrote:
ordinary in the error log. The cache serves about 2.8-3 million requests
lineary and at some time all these inactive items are purged almost
simultaneously causing precipitous drop.
Have you restarted nginx at the growing start points ?
This may be not a restart, but an online upgrade too.
I had error log level set for “crit” so I can’t be sure. The binary was
built on September 29 @ 12:39 which was after the the climb started but
it’s possible there was a restart earlier. I generally don’t restart
except when upgrading or rebooting but it is possible.
I will set error logging to “notice” and upgrade the binary today and
watch what happens.