Forum: NGINX Php-fpm requests eventually goes to queue!

13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 09:37
(Received via mailing list)
Hello,

     We're using nginx + php-fpm. Please check the following
configurations
in php-fpm and sysctl in order to handle large amount of php-fpm
request,
but still 1000+ requests are getting into queue every 15min.

php-fpm.d/stats.conf

[stats]
listen = 127.0.0.1:9000
user = apache
group = apache
request_slowlog_timeout = 5s
slowlog = /var/log/php-fpm/stats-slow.log
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 250
pm.start_servers = 40
pm.min_spare_servers = 20
pm.max_spare_servers = 40
pm.max_requests = 40000
listen.backlog = -1
request_terminate_timeout = 300s
rlimit_files = 13107200
rlimit_core = unlimited
env[HOSTNAME] = $HOSTNAME
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
pm.status_path = /status

The main config parameters of sysctl.conf :

vm.overcommit_memory = 1
fs.file-max = 7000000
net.ipv4.tcp_max_syn_backlog = 70000
net.core.netdev_max_backlog = 4096
net.core.somaxconn=65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_tw_reuse = 1

/etc/security/limits.conf
root soft nofile 700000
root hard nofile 700000


We've 72G of Ram and also writing on disk is 100+MB/s on Sas drives
which
makes high io util% most of the time.

Any clue why requests are still getting into php-fpm  queue and max
children also reached errors occuring, even max_children are 250 *
40000. ??

Php-fpm status :

pool:                 stats
process manager:      dynamic
start time:           22/May/2014:12:17:39 +0500
start since:          1140
accepted conn:        228244
listen queue:         579
max listen queue:     1970
listen queue len:     65535
idle processes:       167
active processes:     9
total processes:      176
max active processes: 250
max children reached: 1



Regards.
Shahzaib
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 10:25
(Received via mailing list)
These are the new status for php-fpm now :

pool:                 stats
process manager:      dynamic
start time:           22/May/2014:12:17:39 +0500
start since:          3975
accepted conn:        866645
listen queue:         0
max listen queue:     2163
listen queue len:     65535
idle processes:       153
active processes:     2
total processes:      155
max active processes: 250
max children reached: 4





On Thu, May 22, 2014 at 12:37 PM, shahzaib shahzaib
Ba5061f1803f9ab0cfc3d53366d49546?d=identicon&s=25 Lord Nynex (Guest)
on 2014-05-22 10:28
(Received via mailing list)
This does not seem like an nginx issue?
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 10:32
(Received via mailing list)
Following is the nginx config :

user  nginx;
# no need for more workers in the proxy mode
worker_processes  16;
error_log  /var/log/nginx/error.log crit;
worker_rlimit_nofile 409600;
events {
 worker_connections 10240; # increase for busier servers
 use epoll; # you should use epoll here for Linux kernels 2.6.x
}
http {
 server_name_in_redirect off;
 server_names_hash_max_size 10240;
 server_names_hash_bucket_size 1024;
 include    mime.types;
 default_type  application/octet-stream;
 server_tokens off;
 sendfile on;
 tcp_nopush on;
 tcp_nodelay on;
 keepalive_timeout  5;
 gzip on;
 gzip_vary on;
 gzip_disable "MSIE [1-6]\.";
 gzip_proxied any;
 gzip_http_version 1.1;
 gzip_min_length  1000;
 gzip_comp_level  6;
 gzip_buffers  16 8k;
# You can remove image/png image/x-icon image/gif image/jpeg if you have
slow CPU
 gzip_types    text/plain text/xml text/css application/x-javascript
application/xml application/xml+rss text/javascript
application/atom+xml;
 ignore_invalid_headers on;
 client_header_timeout  3m;
 client_body_timeout 3m;
 send_timeout     3m;
 reset_timedout_connection on;
 connection_pool_size  256;
 client_header_buffer_size 256k;
 large_client_header_buffers 4 256k;
 client_max_body_size 200M;
 client_body_buffer_size 128k;
 request_pool_size  32k;
 output_buffers   4 32k;
 postpone_output  1460;
 client_body_in_file_only on;
 log_format bytes_log "$msec $bytes_sent .";
 include "/etc/nginx/conf.d/virtual.conf";
}
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 13:03
(Received via mailing list)
Anyone ?
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-05-22 14:10
(Received via mailing list)
Hello!

On Thu, May 22, 2014 at 04:03:37PM +0500, shahzaib shahzaib wrote:

> Anyone ?

[...]

> > On Thu, May 22, 2014 at 1:27 PM, Lord Nynex <lordnynex@gmail.com> wrote:
> >
> >> This does not seem like an nginx issue?

--
Maxim Dounin
http://nginx.org/
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 16:35
(Received via mailing list)
Met you after a long time maxim :) how's everything ?
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 16:52
(Received via mailing list)
You think, issue is not related to nginx but nginx and php-fpm work
together,  that is the reason i posted the question on nginx forums.
56080a668b706e94fb2bbe9ee727d006?d=identicon&s=25 Reinis Rozitis (Guest)
on 2014-05-22 16:53
(Received via mailing list)
> pm.max_children = 250
> pm.start_servers = 40
> pm.max_requests = 40000

> Any clue why requests are still getting into php-fpm  queue and max
> children also reached errors occuring, even max_children are 250 * 40000.
> ??


Your math is wrong - there is no such 250 * 40000, because
pm.max_requests
is just a number of requests after which the php child restarts (to
avoid
possible memory leak etc (it may also be 0)).

Your (maximum simultaneously running) concurrent requests are still 250
(~pm.max_children), so bassically if your request takes longer than 1
second
and there are more than 250 requests per second you will end up with a
queue.


You either speed up the php code (and everything besides it) / increase
the
FPM pool (more children or more pools/fpm backends) or use caching like
some
frontend cache (varnish/squid etc) or nginx with fastcgi_cache.

rr
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-05-22 17:09
(Received via mailing list)
Thanks, my math is not wrong my concept about php-fpm was wrong. I was
thinking each child can handle 40000 requests/sec. :(
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.