We setup some nginx instances a few months ago in order to serve
up/cache
video content. Recently, we’ve noticed that there is severe cache
overrun
– on the order of 1TB + per cache directory.
It seems like proxy_cache_path is entirely ignoring the max_size value
specified in the config.
The actual cache size on-disk is ~1.1-1.4TB rather than the expected
320GB.
I’m at a loss after having asked for days on end in the nginx channel
on
freenode.
Did I pooch the config? Is the nginx cache loader unable to keep up with
the
incoming data rate? Is the max_size value arbitrarily ignored for some
reason?
The actual cache size on-disk is ~1.1-1.4TB rather than the expected 320GB.
I’m at a loss after having asked for days on end in the nginx channel on
freenode.
Did I pooch the config? Is the nginx cache loader unable to keep up with the
incoming data rate? Is the max_size value arbitrarily ignored for some
reason?
Is cache manager process running? What’s in logs? What “nginx -V”
shows?
The cache manager process is, indeed, running. I even did a brief strace
on
it to be sure it was actually working. There is nothing in the logs that
indicates an issue. In fact, the only thing in the logs is expected http
requests.
On Mon, Apr 08, 2013 at 12:28:46PM -0400, gwinans wrote:
TLS SNI support enabled
–with-http_realip_module --with-http_addition_module
–with-ld-opt=-Wl,-E
Have you tried without the cache purge module compiled in?
On Mon, Apr 08, 2013 at 01:16:34PM -0400, gwinans wrote:
Maxim D.
I have not – this is actually an RPM-build from the Atomic Repo. Wouldn’t
removing said purge module disable the ability for it to clear items? Or
does this module have some other purpose?
Purge module is a 3rd party module to allow selective removal of
cache items via HTTP requests. It is not required for normal
operation of nginx cache.
(On the other hand, it may iterfere with normal cache operations
and may cause problems you observe, much like any other 3rd party
module. That’s why we usually ask people to reproduce the issue
without 3rd party modules/patches as one of first investigation
steps.)
(On the other hand, it may iterfere with normal cache operations
and may cause problems you observe, much like any other 3rd party
module. That’s why we usually ask people to reproduce the issue
without 3rd party modules/patches as one of first investigation
steps.)
–
Maxim D.
Got it. I’ve made a build minus that module and have rolled it to
production. Will keep an eye on it and see how it works out – if at
all.
Have you tried without the cache purge module compiled in?
–
Maxim D.
I have not – this is actually an RPM-build from the Atomic Repo.
Wouldn’t
removing said purge module disable the ability for it to clear items? Or
does this module have some other purpose?
So, I’ve run without this third party cache purge module since I
mentioned a
custom build.
The server is still chewing through disk space and IO remains pegged to
the
wall (mix of the cache loader and new cache items). The cache loader
seems
unable to keep up – I’ve never seen it actually delete anything.
Through
watching strace, it’s just loading items. Does it not work on the queue
until all items are loaded?
Is there anything I can do to try to alleviate this problem?
On Tue, Apr 09, 2013 at 05:24:12PM -0400, gwinans wrote:
So, I’ve run without this third party cache purge module since I mentioned a
custom build.
The server is still chewing through disk space and IO remains pegged to the
wall (mix of the cache loader and new cache items). The cache loader seems
unable to keep up – I’ve never seen it actually delete anything. Through
watching strace, it’s just loading items. Does it not work on the queue
until all items are loaded?
Cache loader never deletes anything. It loads information about
the cache from disk. If it’s running, it means that information
about the cache wasn’t yet loaded and nginx can’t maintain
configured max_size as it don’t know current cache size (yet).
In contrast, cache manager maintains max_size and deletes
inactive cache items. It will not be able to maintain max_size
till the cache is loaded, see above.
Is there anything I can do to try to alleviate this problem?
If you have problems with IO, most trivial solution to try is to
reduce IO with proxy_cache_min_uses, see Module ngx_http_proxy_module.