Nginx high i/o

hi guys,

i have posted a question a while ago, and now i am coming back once
more.

i have now a dual quad core xeon system, running freebsd 7.2 latest
patch level, 64 bit, and 16 gb ram. using the latest nginx-devel.

the OS runs on a ssd, the db is set on a normal drive, and the site
files reside on another ssd.

i had a pretty similar config just instead i was using centos.

the i/o on the main drive (os drive) is constantly at 22 mb/s.

i have disabled nginx logging, and everything has been setup from the
ground. i have no idea why do i get such high i/o, because i have no
other apps that could cause this load.

could you please point me to some direction ? i don’t know what to do.

          ad4              ad5              ad6

KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s
54.56 428 22.79 0.00 0 0.00 32.89 4 0.14
55.05 492 26.46 0.00 0 0.00 26.38 8 0.21
55.24 452 24.40 0.00 0 0.00 36.36 5 0.20
53.20 457 23.76 22.31 19 0.42 19.88 8 0.16
53.33 464 24.15 0.00 0 0.00 48.47 23 1.11
54.14 403 21.32 0.00 0 0.00 16.18 217 3.43
52.75 406 20.90 0.00 0 0.00 17.49 178 3.04
55.75 402 21.90 0.00 0 0.00 13.42 12 0.16

Hello!

On Sat, Nov 21, 2009 at 12:21:52AM +0100, Stefanita Rares Dumitrescu
wrote:

i had a pretty similar config just instead i was using centos.

the i/o on the main drive (os drive) is constantly at 22 mb/s.

i have disabled nginx logging, and everything has been setup from
the ground. i have no idea why do i get such high i/o, because i
have no other apps that could cause this load.

Do you use proxy for big replies and/or have big client uploads?

If yes, it’s probably caused by disk buffering (you should have
warnings in logs, but you have no logs…) and you probably want to
tune your buffer sizes and/or proxy_temp_file_max_size.

Maxim D.

actually i got a bunch of errors, in error log, as i enabled it to get
some idea on what’s going on:

2009/11/21 08:47:44 [warn] 24393#0: *57036 an upstream response is
buffered to a temporary file /var/tmp/nginx/fastcgi_temp/1/56/0024741561
while reading upstream, client:

i am so sleepy right now, could you please tell me where do i get the
fastcgi buffers optimizations ? i searched on the wiki but i can’t think
straight.

Hello!

On Sat, Nov 21, 2009 at 01:39:38AM +0100, Stefanita Rares Dumitrescu
wrote:

can’t think straight.
Start from tuning fastcgi_buffers:

http://wiki.nginx.org/NginxHttpFcgiModule#fastcgi_buffers

If you haven’t had enough memory to deligate it for buffers you
may consider using fastcgi_max_temp_file_size 0; - this will
disable disk buffering at cost of not reading data from backend
connection (i.e. keeping backend busy).

You may also consider investigating why your backend replies are
so big (fastcgi_buffers should be 64k in total on your platform,
this is much more than typical generated html…) and moving them
to nginx itself if possible (e.g. by using X-Accel-Redirect).

Maxim D.

Hello!

On Sat, Nov 21, 2009 at 02:19:16PM +0100, Stefanita Rares Dumitrescu
wrote:

    fastcgi_max_temp_file_size 0;

This directive turns off fastcgi buffering according to the source code.

i did this, and the buffer errors are gone. however, the pages load
very very slow. take about 5-10 seconds to load the page.

As I already said, switching off disk buffering has it’s cost.
Without buffering backend processes will be busy for much longer
time. There is no surprise that the same number of backend
processes is no longer enough.

Maxim D.

fastcgi_buffers 64 64k;
fastcgi_buffer_size 64k;

i made the following mods in the fastcgi section, and waiting for
results. from what i understand:

     fastcgi_buffers 64 64k;

the first 64 is the number of buffers and the second is the buffer size,
so on a server that takes around 500 hits per second should i increase
it, or the value should be just fine ?

fastcgi_max_temp_file_size 0;
This directive turns off fastcgi buffering according to the source code.

i did this, and the buffer errors are gone. however, the pages load very
very slow. take about 5-10 seconds to load the page.

could it be my php-fpm.conf ?

the thing is why i am going nuts about this is because the same exact
config (same nginx/php-fpm), i am running on a centos server, and i am
able to push over 200 mb/s and everything works perfectly fine.

i switched over to freebsd to squeeze some more performance out of the
server though.

also i have noticed that when more and more connections are established
to the server, my php-cgi processes are frozen in “sbwait” state:

32514 nobody 1 4 0 60816K 14252K sbwait 4 0:12 0.00%
php-cgi
32562 nobody 1 4 0 60816K 14212K sbwait 2 0:11 0.00%
php-cgi
32577 nobody 1 4 0 60816K 14240K sbwait 6 0:11 0.00%
php-cgi
32479 nobody 1 4 0 60816K 14244K sbwait 0 0:11 0.00%
php-cgi
32475 nobody 1 4 0 60816K 14256K sbwait 0 0:11 0.00%
php-cgi
32497 nobody 1 4 0 60816K 14228K sbwait 0 0:11 0.00%
php-cgi
32527 nobody 1 4 0 60816K 14208K sbwait 1 0:11 0.00%
php-cgi
32564 nobody 1 4 0 60816K 14192K sbwait 4 0:10 0.00%
php-cgi

posted php-fpm.conf below.

<?xml version="1.0" ?>
     All relative paths in this config are relative to php's install

prefix

     <section name="global_options">

             Pid file
             <value

name=“pid_file”>/usr/local/kphp/logs/php-fpm.pid

             Error log file
             <value

name=“error_log”>/usr/local/kphp/logs/php-fpm.log

             Log level
             <value name="log_level">notice</value>

             When this amount of php processes exited with SIGSEGV

or SIGBUS …
10

             ... in a less than this interval of time, a graceful

restart will be initiated.
Useful to work around accidental curruptions in
accelerator’s shared memory.
1m

             Time limit on waiting child's reaction on signals from

master
5s

             Set to 'no' to debug fpm
             <value name="daemonize">yes</value>

     </section>

     <workers>

             <section name="pool">

                     Name of pool. Used in logs and stats.
                     <value name="name">nginx</value>

                     Address to accept fastcgi requests on.
                     Valid syntax is 'ip.ad.re.ss:port' or just

‘port’ or ‘/path/to/unix/socket’
127.0.0.1:8000

                     <value name="listen_options">

                             Set listen(2) backlog
                             <value name="backlog">-1</value>

                             Set permissions for unix socket, if one

used.
In Linux read/write permissions must be
set in order to allow connections from web server.
Many BSD-derrived systems allow
connections regardless of permissions.


0666

                     Additional php.ini defines, specific to this

pool of workers.



                     Unix user of processes
                     <value name="user">nobody</value>

                     Unix group of processes
                     <value name="group">nobody</value>

                     Process manager settings
                     <value name="pm">

                             Sets style of controling worker process

count.
Valid values are ‘static’ and
‘apache-like’
static

                             Sets the limit on the number of

simultaneous requests that will be served.
Equivalent to Apache MaxClients
directive.
Equivalent to PHP_FCGI_CHILDREN
environment in original php.fcgi
Used with any pm_style.
128

                             Settings group for 'apache-like' pm 

style

                                     Sets the number of server

processes created on startup.
Used only when ‘apache-like’
pm_style is selected
32

                                     Sets the desired minimum number

of idle server processes.
Used only when ‘apache-like’
pm_style is selected
32

                                     Sets the desired maximum number

of idle server processes.
Used only when ‘apache-like’
pm_style is selected
64

                             </value>

                     </value>

                     The timeout (in seconds) for serving a single

request after which the worker process will be terminated
Should be used when ‘max_execution_time’ ini
option does not stop script execution for some reason
‘0s’ means ‘off’
0s

                     The timeout (in seconds) for serving of single

request after which a php backtrace will be dumped to slow.log file
‘0s’ means ‘off’
0s

                     The log file for slow requests
                     <value name="slowlog">logs/slow.log</value>

                     Set open file desc rlimit
                     <value name="rlimit_files">2048</value>

                     Set max core size rlimit
                     <value name="rlimit_core">0</value>

                     Chroot to this directory at the start, absolute

path

                     Chdir to this directory at the start, absolute 

path

                     Redirect workers' stdout and stderr into main

error log.
If not set, they will be redirected to
/dev/null, according to FastCGI specs
yes

                     How much requests each process should execute

before respawn.
Useful to work around memory leaks in 3rd party
libraries.
For endless request processing please specify 0
Equivalent to PHP_FCGI_MAX_REQUESTS
500

                     Comma separated list of ipv4 addresses of

FastCGI clients that allowed to connect.
Equivalent to FCGI_WEB_SERVER_ADDRS environment
in original php.fcgi (5.2.2+)
Makes sense only with AF_INET listening socket.
127.0.0.1

                     Pass environment variables like LD_LIBRARY_PATH
                     All $VARIABLEs are taken from current 

environment

$HOSTNAME
/usr/local/bin:/usr/bin:/bin
/tmp
/tmp
/tmp
$OSTYPE
$MACHTYPE
2

             </section>

     </workers>

If you’re seeing buffering to disk instead of to ram (and you don’t
want the i/o penalty of buffering to disk), then the ram buffer is not
sufficient. I would increase it so long as you have sufficient ram to
do so.

On Sat, Nov 21, 2009 at 10:48 AM, Stefanita Rares Dumitrescu

flawless. it works now perfectly. no more disk buffering.

what i wanted to ask. regarding that 64 number:

fastcgi_buffers 64

buffers per second. what is the logic of it ? i mean … how does it
improve performance or decrease it?

Hello!

On Sat, Nov 21, 2009 at 07:48:07PM +0100, Stefanita Rares Dumitrescu
wrote:

size,
Yes. Note well - this numbers specify maximum number of buffers
allocated per request.

so on a server that takes around 500 hits per second should i
increase it, or the value should be just fine ?

Request rate isn’t relevant here. It’s the number of concurent
requests that is relevant, including maximum allowed one.

With 64 64k buffers it is possible for request to eat up to 4M of
memory. With default settings (worker_connections 1024;
worker_processes 1) this gives up to 4G (4M * 1 * 1024).
Shouldn’t be the problem for server with 16G of memory.

If you have more worker processes and/or more worker connections
allowed you may consider lowering this value a bit to make sure
your server can’t be DoS’ed by eating all memory for buffers and
forcing swap activity.

Maxim D.

thanks a bunch. now i understand everything. good nginx lesson today :slight_smile:

right now i get around 12G memory free, so i might see how it performs
in the future and adjust.

i have worker_connections 8192;

because there are large traffic spikes, and i need to prevent them
server from working slow.