I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can
see my nginx.conf here: http://pastebin.com/m63c18e1
My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This is
due to some disk activity every 5 seconds. Here are some samples of
“vmstat 1”:
procs -----------memory---------- —swap-- -----io---- -system–
----cpu----
0 2 0 3188220 41632 61408 0 0 0 2280 9119 6100 1 4
73 22
0 0 0 3187680 41632 61936 0 0 0 272 10516 8027 1
4 93 2
0 0 0 3188004 41632 61256 0 0 0 0 9446 6479 1 4
95 0
0 0 0 3188052 41632 61216 0 0 0 0 9533 6893 1 3
96 0
0 0 0 3187880 41632 62052 0 0 0 0 10639 8117 2
6 93 0
0 0 0 3186936 41632 64132 0 0 0 0 9797 6898 1 4
95 0
1 1 0 3189304 41636 61324 0 0 0 4692 9052 6414 1 4
65 29
0 3 0 3188508 41636 62656 0 0 0 1124 9107 6302 2 4
72 22
0 0 0 3187448 41636 63044 0 0 0 0 8601 5591 1 5
87 8
0 0 0 3184968 41636 65344 0 0 0 0 10273 7832 2
6 92 0
0 0 0 3184900 41636 65024 0 0 0 0 10104 7827 2
5 93 0
1 0 0 3181960 41636 68428 0 0 0 0 9675 7242 2 5
93 0
0 4 0 3183416 41636 68892 0 0 0 9528 7025 3921 1 2
55 42
0 0 0 3186076 41636 63620 0 0 0 440 8508 5839 1 5
60 35
As you can see in my nginx.conf, I have disabled logging (even error
log). I can not determine where my disk is actually writing, but what I
do know is that it is related to nginx. When our site gets busy (200mbit
traffic), iowait increases. There are no other important processes
running on the server.
My question now is: Is this normal behaviour and how can I reduce the
iowait?
Thanks, Edo.
On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
0 0 0 3186936 41632 64132 0 0 0 0 9797 6898 1 4 95 0
My question now is: Is this normal behaviour and how can I reduce the iowait?
If you serve large static content, then this is normal: nginx worker
processes wait on disk reads.
----- Original Message -----
From: “Igor S.” [email protected]
To: [email protected]
Sent: Tuesday, June 17, 2008 10:34 AM
Subject: Re: much iowait - how to reduce?
0 2 0 3188220 41632 61408 0 0 0 2280 9119 6100 1 4
95 0
1 0 0 3181960 41636 68428 0 0 0 0 9675 7242 2 5
running on the server.
Igor Sysoev
I am not running large static content, only some small images and some
php
generated html from my backends. Besides, the disk is only writing, not
reading:
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 8.00 0.00 1168.00 0 1168
Barry from wordpress.com told me that adding the line
“client_body_temp_path
/dev/shm;” would help, because of this quote: “If the request body is
more
than the buffer, then the entire request body or some part is written in
a
temporary file.”
(http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size).
Now this client body should get written to some shared memory, not to
the
disk.
Doesn’t seem to work though…
On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:
I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can
0 0 0 3187680 41632 61936 0 0 0 272 10516 8027 1 4
65 29
0 4 0 3183416 41636 68892 0 0 0 9528 7025 3921 1 2
My question now is: Is this normal behaviour and how can I reduce the
sda 8.00 0.00 1168.00 0 1168
Barry from wordpress.com told me that adding the line
“client_body_temp_path /dev/shm;” would help, because of this quote: “If
the request body is more than the buffer, then the entire request body or
some part is written in a temporary file.”
(http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size).
Now this client body should get written to some shared memory, not to the
disk.
Doesn’t seem to work though…
Do you have many uploads ?
Do you use 32-bit or 64-bit OS ?
Try to increase number of proxy_buffers:
proxy_buffer_size 4k;
proxy_buffers 64 4k;
Then backend responses up to 260K will be buffered in memory.
----- Original Message -----
From: “Igor S.” [email protected]
To: [email protected]
Sent: Tuesday, June 17, 2008 12:40 PM
Subject: Re: much iowait - how to reduce?
On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
0 2 0 3188220 41632 61408 0 0 0 2280 9119 6100 1
96 0
4
1 0 0 3181960 41636 68428 0 0 0 0 9675 7242 2
log). I can not determine where my disk is actually writing, but what I
the request body is more than the buffer, then the entire request body or
Try to increase number of proxy_buffers:
No uploads. All incomming data that gets stored is handled by the
backend
webservers, and probably all send to an independent MySQL server. I have
a
32bit OS.
I have been tuning a bit with proxy_buffer_size and proxy_buffers. Turns
out
that these settings reduce the iowait drastically for me:
proxy_buffer_size 32k;
proxy_buffers 512 32k;
I did a benchmark, and the server went to 20% CPU utilization, from
which
maybe 1% was iowait - that used to be 15% iowait. Tonight it will get
busy
on the server, so let’s see if we will have a nice proof of concept.
Anyhow, thanks for your time Igor, very much appreciated.
E.
On Tue, Jun 17, 2008 at 08:05:10PM +0200, Edo Frederix wrote:
see my nginx.conf here: http://pastebin.com/m63c18e1
0 0 0 3187680 41632 61936 0 0 0 272 10516 8027 1
93 0
5
0 4 0 3183416 41636 68892 0 0 0 9528 7025 3921 1
traffic), iowait increases. There are no other important processes
generated html from my backends. Besides, the disk is only writing, not
Now this client body should get written to some shared memory, not to the
proxy_buffers 64 4k;
proxy_buffer_size 32k;
proxy_buffers 512 32k;
I did a benchmark, and the server went to 20% CPU utilization, from which
maybe 1% was iowait - that used to be 15% iowait. Tonight it will get busy
on the server, so let’s see if we will have a nice proof of concept.
512*32k+32k is up to 16M per request. It’s too big. What is typical size
of your responses ? It’s better to set proxy_buffers to conform them
and allow to write large responses to a disk.
----- Original Message -----
From: “Igor S.” [email protected]
To: [email protected]
Sent: Tuesday, June 17, 2008 8:38 PM
Subject: Re: much iowait - how to reduce?
On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:
----cpu----
0 0 0 3188052 41632 61216 0 0 0 0 9533 6893 1
4
92 0
0 0 0 3186076 41636 63620 0 0 0 440 8508 5839 1
Now this client body should get written to some shared memory, not to
proxy_buffer_size 4k;
I have been tuning a bit with proxy_buffer_size and proxy_buffers. Turns
512*32k+32k is up to 16M per request. It’s too big. What is typical size
of your responses ? It’s better to set proxy_buffers to conform them
and allow to write large responses to a disk.
–
Igor S.
Igor Sysoev
The thing is that I have some files that I publish on the website. These
files are about 3mb large. A typical size of a response would probably
be
around 30kb. On my server I’m running with 4GB memory, so there is free
memory enough to run all buffers from the memory. I experienced that
iowait
would drop if I increased the number of buffers.
Given this info, what would you recommend?
E.
On Tue, Jun 17, 2008 at 10:11:55PM +0200, Edo Frederix wrote:
The thing is that I have some files that I publish on the website. These
files are about 3mb large. A typical size of a response would probably be
around 30kb. On my server I’m running with 4GB memory, so there is free
memory enough to run all buffers from the memory. I experienced that iowait
would drop if I increased the number of buffers.
Given this info, what would you recommend?
http://wiki.codemongers.com/NginxXSendfile
