I’ve noticed that multiple (as great as 8 or more) parallel redundant
streams and corresponding temp_files are opened reading the same file
from a reverse proxy backend into nginx, upon even a single request by
an up-stream client, if not already cached (or stored in a static
proxy’ed file) local to nginx.
This seems extremely wasteful of bandwidth between nginx and
corresponding reverse proxy backends; does anyone know why this is
occurring and how to limit this behavior?
(For example, upon receiving a request for example small 250MB mp4 video
podcast video file, it’s not uncommon for 8 parallel streams to be
opened, each sourcing (and competing for bandwidth) a corresponding
temp_file, where the upstream client appears to being feed by the most
complete stream/temp_file; but even upon the compete file being fully
transferred to the upstream client,the remaining streams remain active
until they too have finished their transfers, and then closed, and their
corresponding temp_files deleted. All resulting in 2GB of data being
transferred when only 250MB needed be, not to mention that the transfer
took nearly 8x longer to complete, so unless there were concerns about
the integrity of the connection, it seems like a huge waste of
resources?)
Thanks, any insight/assistance would be appreciated.
Hello!
On Tue, Jun 24, 2014 at 02:49:57PM -0400, Paul Schlie wrote:
I’ve noticed that multiple (as great as 8 or more) parallel
redundant streams and corresponding temp_files are opened
reading the same file from a reverse proxy backend into nginx,
upon even a single request by an up-stream client, if not
already cached (or stored in a static proxy’ed file) local to
nginx.
This seems extremely wasteful of bandwidth between nginx and
corresponding reverse proxy backends; does anyone know why this
is occurring and how to limit this behavior?
http://nginx.org/r/proxy_cache_lock
–
Maxim D.
http://nginx.org/
Thank you; however it appears to have no effect on reverse proxy_store’d
static files?
(Which seems odd, if it actually works for cached files; as both are
first read into temp_files, being the root of the problem.)
Any idea on how to prevent multiple redundant streams and corresponding
temp_files being created when reading/updating a reverse proxy’d static
file from the backend?
(Out of curiosity, why would anyone ever want many multiple redundant
streams/temp_files ever opened by default?)
Hello!
On Tue, Jun 24, 2014 at 07:51:04PM -0400, Paul Schlie wrote:
Thank you; however it appears to have no effect on reverse proxy_store’d static
files?
Yes, it’s part of the cache machinery. The proxy_store
functionality is dumb and just provides a way to store responses
received, nothing more.
(Which seems odd, if it actually works for cached files; as both
are first read into temp_files, being the root of the problem.)
See above (and below).
Any idea on how to prevent multiple redundant streams and
corresponding temp_files being created when reading/updating a
reverse proxy’d static file from the backend?
You may try to do so using limit_conn, and may be error_page and
limit_req to introduce some delay. But unlikely it will be a
good / maintainable / easy to write solution.
(Out of curiosity, why would anyone ever want many multiple
redundant streams/temp_files ever opened by default?)
You never know if responses are going to be the same. The part
which knows (or, rather, tries to) is called “cache”, and has
lots of directives to control it.
–
Maxim D.
http://nginx.org/
Again thank you. However … (below)
On Jun 24, 2014, at 8:30 PM, Maxim D. [email protected] wrote:
Hello!
On Tue, Jun 24, 2014 at 07:51:04PM -0400, Paul Schlie wrote:
Thank you; however it appears to have no effect on reverse proxy_store’d static
files?
Yes, it’s part of the cache machinery. The proxy_store
functionality is dumb and just provides a way to store responses
received, nothing more.
- There should be no difference between how reverse proxy’d files are
accessed and first stored into corresponding temp_files (and below).
You may try to do so using limit_conn, and may be error_page and
limit_req to introduce some delay. But unlikely it will be a
good / maintainable / easy to write solution.
- Please consider implementing by default that no more streams than may
become necessary if a previously opened stream appears to have died
(timed out), as otherwise only more bandwidth and thereby delay will
most likely result to complete the request. Further as there should be
no difference between how reverse proxy read-streams and corresponding
temp_files are created, regardless of whether they may be subsequently
stored as either symbolically-named static files, or hash-named cache
files; this behavior should be common to both.
(Out of curiosity, why would anyone ever want many multiple
redundant streams/temp_files ever opened by default?)
You never know if responses are going to be the same. The part
which knows (or, rather, tries to) is called “cache”, and has
lots of directives to control it.
- If they’re not “the same” then the tcp protocol stack has failed,
which is nothing to do with ngiinx.
(unless a backend server is frequently dropping connections, it’s
counterproductive to open multiple redundant streams; as doing so by
default will only likely result in higher-bandwidth and thereby slower
response completion.)
Hi, Upon further testing, it appears the problem exists even with
proxy_cache’d files with “proxy_cache_lock on”.
(Please consider this a serious bug, which I’m surprised hasn’t been
detected before; verified on recently released 1.7.2)
Is there any possible solution for this problem?
As although proxy_cache_lock may inhibit the creation of multiple
proxy_cache files, it has seemingly no effect on the creation of
multiple proxy_temp files, being the true root of the problem which the
description of proxy_cache_lock claims to solve (as all proxy_cache
files are first proxy_temp files, so unless proxy_cache_lock can
properly prevent the creation of multiple redundant proxy_temp file
streams, it can seemingly not have the effect it claims to)?
(Further, as temp_file’s are used to commonly source all reverse proxy’d
reads, regardless of whether they’re using a cache hashed naming scheme
for proxy_cache files, or a symbolic naming scheme for reverse proxy’d
static files; it would be nice if the fix were applicable to both.)
Hello!
On Mon, Jun 30, 2014 at 09:14:06PM -0400, Paul Schlie wrote:
(Seemingly, it may be beneficial to simply replace the
sequentially numbered temp_file scheme with hash-named scheme,
where if cached, the file is simply retained for some period of
time and/or other condition, and which may be optionally
symbolically aliased using their uri path and thereby
respectively logically accessed as a local static file, or
deleted upon no longer being needed and not being cached; and
thereby kill multiple birds with one stone per-se?)
Sorry for not following your discussion with yourself, but it looks
you didn’t understand what was explained earlier:
[…]
(Out of curiosity, why would anyone ever want many multiple
redundant streams/temp_files ever opened by default?)
You never know if responses are going to be the same. The part
which knows (or, rather, tries to) is called “cache”, and has
lots of directives to control it.
- If they’re not “the same” then the tcp protocol stack has failed, which is
nothing to do with ngiinx.
In http, responses are not guaranteed to be the same. Each
response can be unique, and you can’t assume responses have to be
identical even if their URLs match.
–
Maxim D.
http://nginx.org/
(Seemingly, it may be beneficial to simply replace the sequentially
numbered temp_file scheme with hash-named scheme, where if cached, the
file is simply retained for some period of time and/or other condition,
and which may be optionally symbolically aliased using their uri path
and thereby respectively logically accessed as a local static file, or
deleted upon no longer being needed and not being cached; and thereby
kill multiple birds with one stone per-se?)
Regarding:
In http, responses are not guaranteed to be the same. Each
response can be unique, and you can’t assume responses have to be
identical even if their URLs match.
Yes, but potentially unique does not imply that upon the first valid ok
or valid
partial response that it will likely be productive to continue to open
further such
channels unless no longer responsive, as doing so will most likely be
counter
productive, only wasting limited resources by establishing redundant
channels;
being seemingly why proxy_cache_lock was introduced, as you initially
suggested.
Hello!
On Mon, Jun 30, 2014 at 11:10:52PM -0400, Paul Schlie wrote:
being seemingly why proxy_cache_lock was introduced, as you initially suggested.
Again: responses are not guaranteed to be the same, and unless
you are using cache (and hence proxy_cache_key and various header
checks to ensure responses are at least interchangeable), the only
thing you can do is to proxy requests one by one.
If you are using cache, then there is proxy_cache_key to identify
a resource requested, and proxy_cache_lock to prevent multiple
parallel requests to populate the same cache node (and
“proxy_cache_use_stale updating” to prevent multiple requests when
updating a cache node).
In theory, cache code can be improved (compared to what we
currently have) to introduce sending of a response being loaded
into a cache to multiple clients. I.e., stop waiting for a cache
lock once we’ve got the response headers, and stream the response
body being load to all clients waited for it. This should/can
help when loading large files into a cache, when waiting with
proxy_cache_lock for a complete response isn’t cheap. In
practice, introducing such a code isn’t cheap either, and it’s not
about using other names for temporary files.
–
Maxim D.
http://nginx.org/
Hello!
On Tue, Jul 01, 2014 at 08:44:47AM -0400, Paul Schlie wrote:
completed transfer as you’ve noted?
Your reading of the code is incorrect.
A node in shared memory is created on a request start, and this is
enough for proxy_cache_lock to work. On the request completion,
the temporary file is placed into the cache directory, and the
node is updated to reflect that the cache file exists and can be
used.
–
Maxim D.
http://nginx.org/
Then how could multiple streams and corresponding temp_files ever be
created upon successive requests for the same $uri with “proxy_cache_key
$uri” and “proxy_cache_lock on”; if all subsequent requests are locked
to the same cache_node created by the first request even prior to its
completion?
You’ve previously noted:
In theory, cache code can be improved (compared to what we
currently have) to introduce sending of a response being loaded
into a cache to multiple clients. I.e., stop waiting for a cache
lock once we’ve got the response headers, and stream the response
body being load to all clients waited for it. This should/can
help when loading large files into a cache, when waiting with
proxy_cache_lock for a complete response isn’t cheap. In
practice, introducing such a code isn’t cheap either, and it’s not
about using other names for temporary files.
Being what I apparently incorrectly understood proxy_cache_lock to
actually do.
So if not the above, what does proxy_cache_lock actually do upon receipt
of subsequent requests for the same $uri?
As it appears a downstream response is not cached until first completely
read into a temp_file (which for a large file may require 100’s if not
1,000’s of MB be transferred), there appears to be no “cache node
formed” which to “lock” or serve “stale” responses from, and thereby
until the first “cache node” is useably created, proxy_cache_lock has
nothing to lock requests to?
The code does not appear to be forming a “cache node” using the
designated cache_key until the requested downstream element has
completed transfer as you’ve noted?
For the scheme to work, a lockable cache_node would need to formed
immediately upon the first unique cache_key request, and not wait until
the transfer of the requested item being stored into a temp_file is
complete; as otherwise multiple redundant active streams between nginx
and a backend server may be formed, each most likely transferring the
same information needlessly; being what proxy_cache_lock was seemingly
introduced to prevent (but it doesn’t)?
Hello!
On Tue, Jul 01, 2014 at 10:15:47AM -0400, Paul Schlie wrote:
Then how could multiple streams and corresponding temp_files
ever be created upon successive requests for the same $uri with
“proxy_cache_key $uri” and “proxy_cache_lock on”; if all
subsequent requests are locked to the same cache_node created by
the first request even prior to its completion?
Quoting documentation, Alphabetical index of directives
: When enabled, only one request at a time will be allowed to
: populate a new cache element identified according to the
: proxy_cache_key directive by passing a request to a proxied
: server. Other requests of the same cache element will either wait
: for a response to appear in the cache or the cache lock for this
: element to be released, up to the time set by the
: proxy_cache_lock_timeout directive.
So, there are at least two cases “prior to its completion” which
are explicitly documented:
-
If the cache lock is released - this happens, e.g., if the
response isn’t cacheable according to the response headers.
-
If proxy_cache_lock_timeout expires.
–
Maxim D.
http://nginx.org/
Thank you for your patience.
I mistakenly thought the 5 second default value associated with
proxy_cache_lock_timeout was the maximum delay allowed between
successive responses from the backend server is satisfaction of the
reverse proxy request being cached prior to the cache lock being
released, not the maximum delay for the response to be completely
received and cached as it appears to actually be.
Now that I understand, please consider setting the default value much
higher, or more ideally set in proportion to the size of the item being
cached and possibly some measure of the activity of the stream; as in
most circumstances, redundant streams should never be opened, as it will
tend to only make matters worse.
Thank you.
Lastly, is there any way to try to get proxy_store to work in
combination with proxy_cache, possibly by enabling the completed
temp_file to be saved as a proxy_store file within its uri logical path
hierarchy, and the cache_file descriptor aliased to it, or visa versa?
(As it’s often nice to be able to view/access cached files within their
natural uri hierarchy, being virtually impossible if stored using their
corresponding hashed names alone; and not lose the benefit of being able
to lock multiple pending requests to the same cache_node being fetched
so as to minimize otherwise redundant down-stream requests prior to the
file being cached.)