Hi all,
I’m not sure how proxy_buffers are used.
1- requests to the backend can use only one buffer. If the reponse is
bigger than one buffer, some part of the response is buffered to disk.
2- requests to the backen can use multiple buffers. If the response is
bigger than on buffer, nginx will use more buffers (if available of
course).
So depending on the answer, I would use :
1- big buffers so that 80% of the requests can feet in on buffer. And
enough buffers so that 80% of the requests can be buffered
2- small buffers and a lots of buffers
don’t know what to use 
any clue ?
thx
++ jerome
I’m not Igor so I cannot be totally sure, but my understanding of it
has always been 1- requests to the backend can use only one buffer and
the response gets buffered to disk. As such, I generally start out
sites with log level info (might be too low) and try to tune down
requests getting buffered if at all possible. I also try to stick
with magic numbers good for the architecture (so 8 and 16 based,
generally).
– Merlin
2009/6/9 Jérôme Loyet [email protected]:
2009/6/10 merlin corey [email protected]:
I’m not Igor so I cannot be totally sure, but my understanding of it
has always been 1- requests to the backend can use only one buffer and
the response gets buffered to disk. As such, I generally start out
sites with log level info (might be too low) and try to tune down
requests getting buffered if at all possible. I also try to stick
with magic numbers good for the architecture (so 8 and 16 based,
generally).
Thx for the answer merlin,
I did some test. The request I make to nginx is proxied to the
backend. There is only one request made at the same time.
I request a 5Mo file.
First: with minimum buffers:
proxy_buffers 2 4k;
I have this in the log: 2009/06/11 19:40:05 [warn] 24607#0: *104 an
upstream response is buffered to a temporary file
/LIBRE/nginx/proxy_temp/0000000000 while reading upstream, …
Then I set 10 buffers of 1Mo:
proxy_buffers 10 1m;
and nothing appears in logs.
This tells me that a request can use more than one buffer.
I’ll set up small buffers but a large number:
proxy_buffers 2048 8k;
Is it too much ?
2009/6/11 Jérôme Loyet [email protected]:
course).
Nice work! I guess for figuring out if it is too much you need to
look at your system and determine if it will be using too much memory,
but I would bet you can easily handle the 16MB + whatever structure
overhead for those 2048 buffers. Possibly other factors like paging
and caching (on CPU level) might come into play, too, but probably not
at our (human) granuality.
Hello!
On Thu, Jun 11, 2009 at 08:00:04PM +0200, Jérôme Loyet wrote:
Then I set 10 buffers of 1Mo:
proxy_buffers 10 1m;
and nothing appears in logs.
This tells me that a request can use more than one buffer.
Yes.
I’ll set up small buffers but a large number:
proxy_buffers 2048 8k;
Is it too much ?
Each connection that uses proxy can use up to 16M of memory with
this setting (even after actual connection to backend has been
already closed). With 1024 client connections this means up to
16G of memory, probably a bit too many.
And keep in mind that:
Some hints about seveal big vs. many small buffers:
-
Using bigger buffers may help a bit with CPU usage, but usually
it’s not a big deal.
-
Using bigger buffers means less disk seeks if reply won’t fit
into memory and will be buffered to disk.
-
On the other hand, small buffers allow better granularity (and
less memory wasted for last partially filled buffer).
Since there are different load patterns (and different hardware)
it’s hard to recommend some particular setting, but the about
hints should allow you to tune it for your system appropriately.
Maxim D.
2009/6/11 Jérôme Loyet [email protected]:
Do you mean that proxy_buffers are dedicated to each connection ?
I thought it was shared by all connections.
Positive, I read it as each connection gets that many buffers, in
which case, I agree with Maxim you MIGHT have too much :).
2009/6/11 Maxim D. [email protected]:
with magic numbers good for the architecture (so 8 and 16 based,
Yes.
I’ll set up small buffers but a large number:
proxy_buffers 2048 8k;
Is it too much ?
Each connection that uses proxy can use up to 16M of memory with
this setting (even after actual connection to backend has been
already closed). With 1024 client connections this means up to
16G of memory, probably a bit too many.
Do you mean that proxy_buffers are dedicated to each connection ?
I thought it was shared by all connections.
Hello!
On Fri, Jun 12, 2009 at 12:30:58AM +0200, Jérôme Loyet wrote:
[…]
Do you mean that proxy_buffers are dedicated to each connection ?
I thought it was shared by all connections.
Directive proxy_buffers specify maximum number and size of buffers
that can be used by one (each) connection. Not every connection
uses so many buffers as they are allocated on demand, but every
connection can do so.
Original documentation clearly states “… number and size of
buffers for one connection …” (in russian):
http://sysoev.ru/nginx/docs/http/ngx_http_proxy_module.html#proxy_buffers
Probably this was lost in translation (or was clarified later).
Feel free to fix wiki.
Maxim D.
2009/6/12 Maxim D. [email protected]:
uses so many buffers as they are allocated on demand, but every
connection can do so.
Original documentation clearly states “… number and size of
buffers for one connection …” (in russian):
Модуль ngx_http_proxy_module
Probably this was lost in translation (or was clarified later).
Feel free to fix wiki.
OK Thanks you very much. In this case 16Mo (2048 * 8k is way way too
much)
I’ll analyse my logs to know what is the distribution of my requests
size so that I can do the better compromise. Thx you guys
Le 12 juin 2009 07:45, Jérôme Loyet[email protected] a écrit :
Is it too much ?
that can be used by one (each) connection. Not every connection
OK Thanks you very much. In this case 16Mo (2048 * 8k is way way too much)
I’ll analyse my logs to know what is the distribution of my requests
size so that I can do the better compromise. Thx you guys
After analysing my logs. I have found out:
1- on the dynamic part of the site : more than 80% of requests are
less or equal than 32ko
2- on the static part of the site : more than 80% of requests are less
or equal than 12ko
So I’ll set this configuration.
proxy_buffers 4 8k; #32Ko to handle about 88% of static requests
proxy_buffers 4 4k; #16Ko to handle about 90% of dynamic requests
it seems more secure 
what do you think ?
any clue ?