Forum: NGINX request body and client_body_buffer_size

Ee9a642d40e35659c66ea7f3539ddadc?d=identicon&s=25 Jeroen Ooms (Guest)
on 2013-09-14 02:57
(Received via mailing list)
Is it correct that when $content_length > client_body_buffer_size,
then $request_body == "" ? If so this would be worth documenting at
request_body.

I am using:

    proxy_cache_methods POST;
    proxy_cache_key "$request_method$request_uri$request_body";

Which works for small requests, but for large requests clients got
very strange results due to $request_body being empty and hence
getting false cache hits for completely different form posts.

Is there something available like $body_hash that can be used as a
caching key even for large request bodies? Or alternatively, how
would I configure nginx to not cache requests when content_length
is larger than client_body_buffer_size?
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-09-14 15:49
(Received via mailing list)
Hello!

On Fri, Sep 13, 2013 at 05:56:51PM -0700, Jeroen Ooms wrote:

> Is it correct that when $content_length > client_body_buffer_size,
> then $request_body == "" ? If so this would be worth documenting at
> request_body.

Yes, it's intended behaviour.  If a request body is larger than
client_body_buffer_size, it's written to disk and not available in
memory, hence no $request_body.  The limitation is more or less
obvious, and it's also explicitly documented here in
the $r->request_body() method documentation:

http://nginx.org/en/docs/http/ngx_http_perl_module...

It might worth adding some short reference into $request_body
variable description though.

> caching key even for large request bodies? Or alternatively, how
> would I configure nginx to not cache requests when content_length
> is larger than client_body_buffer_size?

The

    proxy_no_cache $request_body_file;

should do the trick, see http://nginx.org/r/proxy_no_cache.

--
Maxim Dounin
http://nginx.org/en/donation.html
Ee9a642d40e35659c66ea7f3539ddadc?d=identicon&s=25 Jeroen Ooms (Guest)
on 2013-09-14 20:16
(Received via mailing list)
@ Maxim Dounin

Thanks! This is very helpful. I have also set:

client_body_buffer_size 1m;

Could this setting have any side effects? I am not expecting too many
large POST request. From what I read, client_body_buffer_size is
actually the maximum amount of memory allocated. Does this mean that
for small requests (e.g. without a body) there is no additional
overhead introduced by raising this value?
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-09-14 21:24
(Received via mailing list)
Hello!

On Sat, Sep 14, 2013 at 11:15:39AM -0700, Jeroen Ooms wrote:

> overhead introduced by raising this value?
Yes, it's not allocated if there is no request body, and only
needed buffer is allocated if a request body is known to be
smaller.  On the other hand, it can be used as a DoS vector if an
attacker is allowed to open many connections but you can't afford
them all to allocate client_body_buffer_size buffer.

Additionally, using such a big $request_body in proxy_cache_key
implies various overheads.  In particular, proxy_buffer_size should
be set big enough to be able to contain cache header with a key.
Not even talking about reading/writing cache files with such keys.

--
Maxim Dounin
http://nginx.org/en/donation.html
Ee9a642d40e35659c66ea7f3539ddadc?d=identicon&s=25 Jeroen Ooms (Guest)
on 2013-09-26 02:42
(Received via mailing list)
One more question regarding this:

> The
>    proxy_no_cache $request_body_file;
> should do the trick, see http://nginx.org/r/proxy_no_cache.

I tried this and get a warning:
   nginx: [warn] "proxy_no_cache" functionality has been changed in
0.8.46, now it should be used together with "proxy_cache_bypass"

Do I just need to add an additional line:
   proxy_cache_bypass $request_body_file;

It is not clear to me how proxy_cache_bypass is different from
proxy_no_cache.
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-09-26 14:38
(Received via mailing list)
Hello!

On Wed, Sep 25, 2013 at 08:41:44PM -0400, Jeroen Ooms wrote:

> Do I just need to add an additional line:
>    proxy_cache_bypass $request_body_file;
>
> It is not clear to me how proxy_cache_bypass is different from proxy_no_cache.

The "proxy_no_cache" directive prevents caching of a response got
from an upstream (i.e., do not save to a cache).  In contrast, the
"proxy_cache_bypass" directive prevents returing a response from a
cache if it's there.

The warning in question is an artifact from 0.8.x time when
proxy_no_cache used to mean both "no cache" and "bypass".  It
probably should be removed.

--
Maxim Dounin
http://nginx.org/en/donation.html
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.