Nginx fastcgi_cache not freeing deleted items causing excessive disk usage

Hello,

I’m using nginx 1.10.0 and the nginx_fastcgi_cache option. I’ve noticed
that
with a high amount of requests per second (I’m not sure when it occurs
exactly, but we have ~2500RPS with about 2Gbit/s of outgoing traffic
responses) there is an issue with the fastcgi_cache files not being
freed.
lsof shows a huge amount of files as (deleted) but the space it not
being
freed. Eventually the entire partition fills up like this.

The keys_zone and max_size are both set to 500m
The fastcgi_cache_valid is set to 1m

Even when the server is idle for a while (30+ minutes) the space is not
being freed. Obviously restarting nginx manually immediately frees the
space.

Is this a bug or is there a setting I can use to perform some sort of
garbage collection?

Cheers,
Niels

Posted at Nginx Forum:

Hello!

On Sun, Jun 05, 2016 at 11:32:35AM -0400, b4a456fb-1402 wrote:

The fastcgi_cache_valid is set to 1m

Even when the server is idle for a while (30+ minutes) the space is not
being freed. Obviously restarting nginx manually immediately frees the
space.

Is this a bug or is there a setting I can use to perform some sort of
garbage collection?

The fact that a file is shown as “deleted” indicates that it was
deleted but is still open. This may happen due to legitimate
reasons - e.g., the file was deleted, but is current being served
to a client and hence it is still open. But if you see no changes
with 30+ minutes without load, there is likely a problem
somewhere. You may try debugging it further.

Some basic things to consider:

  • Make sure you are not using open_file_cache with some insane
    parameters. If you use open_file_cache, comment it out and check
    if you are still able to reproduce the problem.

  • Check if “nginx -V” shows any 3rd party modules. If it does,
    try reproducing the problem without them.

  • When leaving the server idle, wait to make sure all client
    connections are actually closed on socket level by clients -
    that is, no connections in the ESTABLISHED state. Check lsof
    when this happens.


Maxim D.
http://nginx.org/