I assume this has been covered before but despite lots of searching,
cannot find it.
We run many nginx servers with fast-cgi for PHP, but here in China the
connections can be slow or dynamic, so suddenly we have 1000 connections
in ‘write’ status in Nginx. This should be no problem, but these seem
to be backing up the fast-cgi processes, eventually running out and the
whole system locks up from a users’ perspective, with 502 errors since
nginx can’t find any more fast-cgi processes to talk with (they are all
busy).
Errors we get are usually "upstream timed out (110: Connection timed
out) while reading response header from upstream, client: 61.149.175.16,
server: 121.13.44.145, request: "GET /…
In this scenario, we’d need 1000+ fast-cgi processes to handle all the
open connections. We’d prefer to run 10-20 php connections, which can
easily handle all the performance needs.
We thought, this is simple, just add more buffering so all the PHP
output is in memory in Nginx, and it will close the connection to PHP
and another user can use it. These are big servers, with 8-16 cores and
24GB+ RAM so we have plenty of power and memory. So we added bigger
buffers to 64KB and added thousands of buffers, etc. but Nginx’s memory
size didn’t really increase (very small at 30-50MB) and the problem
didn’t go away. And we have no buffering to disk messages in the logs.
So if the buffers are big enough and/or we have disks space, I am
thinking nginx will ALWAYS buffer ALL the fastcgi data and the
connection will close, so we should NEVER see fastcgi waiting for nginx
to write data to a client - is this correct?
First, are these buffer settings per connection or for all connections ?
I assume fastcgi_buffer_size is per connection. But if fastcgi_buffers
is per connection, why have a buffer count, why not just say 32K, 64K,
etc.? So I’m guessing this is the total buffers available to the
server, in blocks of the buffer size, for example fastcgi_buffers 1024
64k gives me 64MB of total buffer space. If I have 100 connections, I
can buffer about 600KB each, etc. before nginx starts buffering to
files.
Without a fix we are running 1000 cgi processes and 1-2K nginx
connections. This works but if we ever get real load, we’ll have a 250
load average, like we used to see on loaded Apache systems. We need a
way to use 10-20 PHP processes on a few thousand slow connections.
I assume we have buffering problems, but maybe there is a close or other
issue that prevents the php from being re-used, but this works great
when the connections are fast and the % of writers is small.
The fastcgi engine is php5-cgi; maybe we should use spawn-cgi from
lighttpd.
Key configis are:
events {
worker_connections 4096;
use epoll;
multi_accept off;
}
http {
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 15;
tcp_nodelay on;
server {
listen 80;
server_name 120.136.43.145 abc.com.cn 127.0.0.1;
root /var/www/abc;
access_log /var/log/nginx/abc.com_access.log;
error_log /var/log/nginx/abc.com_error.log;
index index.html index.php index.htm;
location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_buffer_size 64k;
fastcgi_buffers 4096 64k;
fastcgi_param SCRIPT_FILENAME
/var/www/abc$fastcgi_script_name;
include fastcgi_params;
}
}
Posted at Nginx Forum: