Reverse proxy and too many temp files

hi

we are using nginx as a reverse proxy in front of some webservers with
media files.
evrerithing works great beside when we start nginx new with an empty
cache.

in this situation allot of files in proxy_temp_path are create consuming
quite fast all our diskspace, in the same time proxy_cache_path is only
very slowly populated.

i see two parameters who might help:
proxy_cache_use_stale
proxy_max_temp_file_size

a quick test with *use_stale was not succesfull, but maybe something
else was wrong
and for proxy_max_temp_file_size=0 i’m not shure if it is the right
thing todo

what we would like to achieve is to uses as much of the diskspace for
caching. but on a restart with clean cache, not filling up the temp
folder which does not count to proxy_cache_path max_size.

or should we just reduce the workers (16) to provent from having to many
workers creating the same temp file for the same origin file?

cheers
andi

Posted at Nginx Forum:

On Wed, Oct 27, 2010 at 4:16 PM, roadfox [email protected] wrote:

caching. but on a restart with clean cache, not filling up the temp
folder which does not count to proxy_cache_path max_size.

or should we just reduce the workers (16) to provent from having to many
workers creating the same temp file for the same origin file?

You probably need to increase the number and/or size of proxy_buffers
and perhaps your gzip_buffers. The defaults are way too small and
trade low memory usage for disk I/O (which is a really bad trade-off
most of the time). I have “proxy_buffers 16 32k;” and don’t see temp
responses on disk at all with my applications (some of which send up
to 512KB uncompressed pages.

RPM

[quote]You probably need to increase the number and/or size of
proxy_buffers[/quote]

Our nginx reverse proxy is only serving media files with sizes between
100KB and 2MB.
As DiskI/O is a bottleneck we will try to increase proxy_buffers and see
what happens.

Posted at Nginx Forum:

You mentioned gzip buffers, I was of the impression that gzip was all in
memory?

Thanks all for your replies!

So we go out and buy some more disks. I forgot to mention but the temp
file folder grows to more than 60GB.
And it must be fast disks as I/O is huge.

Will increasing proxy_buffers help in our case to reduce disk i/o?

  • during cache population
  • if cache is populatet
    And i was wrong above, average file size served is 140MB, largest around
    2GB.

regards
Andi

Posted at Nginx Forum:

Hello!

On Wed, Oct 27, 2010 at 05:16:02PM -0400, roadfox wrote:

i see two parameters who might help:
proxy_cache_use_stale
proxy_max_temp_file_size

a quick test with *use_stale was not succesfull, but maybe something
else was wrong

proxy_cache_use_stale only make sense if you already have data in
cache.

and for proxy_max_temp_file_size=0 i’m not shure if it is the right
thing todo

proxy_max_temp_file_size can’t (and doesn’t) limit writing to disk
responses expected to be saved to cache as nginx needs them on
disk to save to cache.

what we would like to achieve is to uses as much of the diskspace for
caching. but on a restart with clean cache, not filling up the temp
folder which does not count to proxy_cache_path max_size.

or should we just reduce the workers (16) to provent from having to many
workers creating the same temp file for the same origin file?

No, reducing number of workers isn’t likely to help as each worker
may handle many connections.

Solution to this problem is so called “busy locks” which are not
yet implemented.

Right now there is no good solution. Some possibilities include:
using limit_zone to limit number of requests (though this will
limit serving from cache as well), warming up cache before
startup, adding more disk space.

Maxim D.

A large number off buffers would help 250 10mb although that’s a huge
amount of memory so im not so sure!

Sent from my iPod

Hello!

On Fri, Oct 29, 2010 at 02:32:13AM +1100, Splitice wrote:

A large number off buffers would help 250 10mb although that’s
a huge amount of memory so im not so sure!

No, it won’t. Responses subject to cache will go to disk anyway.

Maxim D.

p.s. Please do not top-post. Thank you.