Forum: NGINX question about proxy_buffers

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Jérôme Loyet (Guest)
on 2009-06-09 20:56
(Received via mailing list)
Hi all,

I'm not sure how proxy_buffers are used.

1- requests to the backend can use only one buffer. If the reponse is
bigger than one buffer, some part of the response is buffered to disk.
2- requests to the backen can use multiple buffers. If the response is
bigger than on buffer, nginx will use more buffers (if available of
course).

So depending on the answer, I would use :
1- big buffers so that 80% of the requests can feet in on buffer. And
enough buffers so that 80% of the requests can be buffered
2- small buffers and a lots of buffers

don't know what to use :) any clue ?

thx
++ jerome
merlin corey (Guest)
on 2009-06-11 01:23
(Received via mailing list)
I'm not Igor so I cannot be totally sure, but my understanding of it
has always been 1- requests to the backend can use only one buffer and
the response gets buffered to disk.  As such, I generally start out
sites with log level info (might be too low) and try to tune down
requests getting buffered if at all possible.  I also try to stick
with magic numbers good for the architecture (so 8 and 16 based,
generally).

-- Merlin

2009/6/9 Jérôme Loyet <removed_email_address@domain.invalid>:
Jérôme Loyet (Guest)
on 2009-06-11 22:06
(Received via mailing list)
2009/6/10 merlin corey <removed_email_address@domain.invalid>:
> I'm not Igor so I cannot be totally sure, but my understanding of it
> has always been 1- requests to the backend can use only one buffer and
> the response gets buffered to disk.  As such, I generally start out
> sites with log level info (might be too low) and try to tune down
> requests getting buffered if at all possible.  I also try to stick
> with magic numbers good for the architecture (so 8 and 16 based,
> generally).
>
Thx for the answer merlin,

I did some test. The request I make to nginx is proxied to the
backend. There is only one request made at the same time.
I request a 5Mo file.

First: with minimum buffers:
proxy_buffers 2 4k;

I have this in the log: 2009/06/11 19:40:05 [warn] 24607#0: *104 an
upstream response is buffered to a temporary file
/LIBRE/nginx/proxy_temp/0000000000 while reading upstream, ....

Then I set 10 buffers of 1Mo:
proxy_buffers 10 1m;
and nothing appears in logs.

This tells me that a request can use more than one buffer.

I'll set up small buffers but a large number:
proxy_buffers 2048 8k;

Is it too much ?
Maxim D. (Guest)
on 2009-06-12 02:07
(Received via mailing list)
Hello!

On Thu, Jun 11, 2009 at 08:00:04PM +0200, Jérôme Loyet wrote:

>
>
> Then I set 10 buffers of 1Mo:
> proxy_buffers 10 1m;
> and nothing appears in logs.
>
> This tells me that a request can use more than one buffer.

Yes.

> I'll set up small buffers but a large number:
> proxy_buffers 2048 8k;
>
> Is it too much ?

Each connection that uses proxy can use up to 16M of memory with
this setting (even after actual connection to backend has been
already closed).  With 1024 client connections this means up to
16G of memory, probably a bit too many.

And keep in mind that:

- Doing reconfiguration will spawn new worker processes and memory
  usage by nginx nearly doubles for a while.

- You also need some memory for OS's disk cache.

Some hints about seveal big vs. many small buffers:

- Using bigger buffers may help a bit with CPU usage, but usually
  it's not a big deal.

- Using bigger buffers means less disk seeks if reply won't fit
  into memory and will be buffered to disk.

- On the other hand, small buffers allow better granularity (and
  less memory wasted for last partially filled buffer).

Since there are different load patterns (and different hardware)
it's hard to recommend some particular setting, but the about
hints should allow you to tune it for your system appropriately.

Maxim D.
merlin corey (Guest)
on 2009-06-12 02:16
(Received via mailing list)
2009/6/11 Jérôme Loyet <removed_email_address@domain.invalid>:
>
>
>
>>> course).
>>>
>>>
>>
>>
>
>

Nice work!  I guess for figuring out if it is too much you need to
look at your system and determine if it will be using too much memory,
but I would bet you can easily handle the 16MB + whatever structure
overhead for those 2048 buffers.  Possibly other factors like paging
and caching (on CPU level) might come into play, too, but probably not
at our (human) granuality.
Jérôme Loyet (Guest)
on 2009-06-12 02:38
(Received via mailing list)
2009/6/11 Maxim D. <removed_email_address@domain.invalid>:
>> > with magic numbers good for the architecture (so 8 and 16 based,
>>
> Yes.
>
>> I'll set up small buffers but a large number:
>> proxy_buffers 2048 8k;
>>
>> Is it too much ?
>
> Each connection that uses proxy can use up to 16M of memory with
> this setting (even after actual connection to backend has been
> already closed).  With 1024 client connections this means up to
> 16G of memory, probably a bit too many.

Do you mean that proxy_buffers are dedicated to each connection ?
I thought it was shared by all connections.
merlin corey (Guest)
on 2009-06-12 03:10
(Received via mailing list)
2009/6/11 Jérôme Loyet <removed_email_address@domain.invalid>:
> Do you mean that proxy_buffers are dedicated to each connection ?
> I thought it was shared by all connections.

Positive, I read it as each connection gets that many buffers, in
which case, I agree with Maxim you MIGHT have too much :).
Maxim D. (Guest)
on 2009-06-12 04:07
(Received via mailing list)
Hello!

On Fri, Jun 12, 2009 at 12:30:58AM +0200, Jérôme Loyet wrote:

[...]

> Do you mean that proxy_buffers are dedicated to each connection ?
> I thought it was shared by all connections.

Directive proxy_buffers specify maximum number and size of buffers
that can be used by one (each) connection.  Not every connection
uses so many buffers as they are allocated on demand, but every
connection can do so.

Original documentation clearly states "... number and size of
buffers for one connection ..." (in russian):

http://sysoev.ru/nginx/docs/http/ngx_http_proxy_mo...

Probably this was lost in translation (or was clarified later).
Feel free to fix wiki.

Maxim D.
Jérôme Loyet (Guest)
on 2009-06-12 09:54
(Received via mailing list)
2009/6/12 Maxim D. <removed_email_address@domain.invalid>:
>> >
> uses so many buffers as they are allocated on demand, but every
> connection can do so.
>
> Original documentation clearly states "... number and size of
> buffers for one connection ..." (in russian):
>
> http://sysoev.ru/nginx/docs/http/ngx_http_proxy_mo...
>
> Probably this was lost in translation (or was clarified later).
> Feel free to fix wiki.
>
OK Thanks you very much. In this case 16Mo (2048 * 8k is way way too
much)

I'll analyse my logs to know what is the distribution of my requests
size so that I can do the better compromise. Thx you guys
Jérôme Loyet (Guest)
on 2009-06-12 14:32
(Received via mailing list)
Le 12 juin 2009 07:45, Jérôme Loyet<removed_email_address@domain.invalid> a 
écrit :
>>> >> Is it too much ?
>> that can be used by one (each) connection.  Not every connection
>>
> OK Thanks you very much. In this case 16Mo (2048 * 8k is way way too much)
>
> I'll analyse my logs to know what is the distribution of my requests
> size so that I can do the better compromise. Thx you guys

After analysing my logs. I have found out:

1- on the dynamic part of the site : more than 80% of requests are
less or equal than 32ko
2- on the static part of the site : more than 80% of requests are less
or equal than 12ko

So I'll set this configuration.
proxy_buffers 4 8k; #32Ko to handle about 88% of static requests
proxy_buffers 4 4k; #16Ko to handle about 90% of dynamic requests

it seems more secure :) what do you think ?
This topic is locked and can not be replied to.