Re: Quick performance deterioration when No. of clients increases

Hi Nick,

There was a discussion recently that yielded a marked performance
difference between smaller and higher levels of gzip compression. I’ll
echo the concerns over CPU power and would also suggest trying
gzip_comp_level 1.



On Fri, Oct 11, 2013 at 3:07 AM, [email protected] wrote:

Message: 4
Date: Fri, 11 Oct 2013 11:07:22 +0300
From: Nikolaos M. [email protected]
To: [email protected]
Subject: Quick performance deterioration when No. of clients increases
Message-ID: [email protected]
Content-Type: text/plain; charset=“utf-8”; Format=“flowed”


I am trying to migrate a Joomla 2.5.8 website from Apache to NGINX 1.4.2
with php-fpm 5.3.3 (and MySQL 5.5.34) on a CentOS 6.4 x86_64 Virtual
Machine (running under KMS).

The goal is to achieve better peak performance: This site has occasional
high peaks; while the normal traffic is ~10 req/sec, it may reach > 3000
req/sec for periods of a few hours (due to the type of services the site
provides - it is a non-profit, real-time seismicity-related site - so
php caching should not be more than 10 seconds).

The new VM (using Nginx) currently is in testing mode and it only has
1-core CPU / 3 GB of RAM. We tested performance with loadimpact and the
results are attached.

You can see at the load graph that as the load approaches 250 clients,
the response time increases very much and is already unacceptable (this
happens consistently). I expected better performance, esp. since caching
is enabled. Despite many efforts, I cannot find the cause of the
bottleneck, and how to deal with it. We would like to achieve better
scaling, esp. since NGINX is famous for its scaling capabilities. Having
very little experience with Nginx, I would like to ask for your
assistance for a better configuration.

When this performance deterioration occurs, we don’t see very high CPU
load (Unix load peaks 2.5), neither RAM exhaustion (System RAM usage
appears to be below 30%). [Monitoring is through Nagios.]

Can you please guide me on how to correct this issue? Any and all
suggestions will be appreciated.

Current configuration, based on info available on the Internet, is as
follows (replaced true domain/website name and public IP address(es)):

=================== Nginx.conf ===================

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/;

worker_rlimit_nofile 200000;

events {
worker_connections 8192;
multi_accept on;
use epoll;

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 64;

 log_format  main  '$remote_addr - $remote_user [$time_local]

“$request” ’
'$status $body_bytes_sent “$http_referer” ’
‘“$http_user_agent” “$http_x_forwarded_for”’;

 log_format cache  '$remote_addr - $remote_user [$time_local]

“$request” ’
'$status $upstream_cache_status $body_bytes_sent
“$http_referer” ’
‘“$http_user_agent” “$http_x_forwarded_for”’;

 fastcgi_cache_path /var/cache/nginx levels=1:2

keys_zone=microcache:5m max_size=1000m;

 access_log  /var/log/nginx/access.log  main;

 sendfile           on;

 tcp_nopush         on;
 tcp_nodelay        on;
 keepalive_timeout  2;

 types_hash_max_size 2048;
 server_tokens off;

 keepalive_requests 30;

 open_file_cache max=5000 inactive=20s;
 open_file_cache_valid 30s;
 open_file_cache_min_uses 2;
 open_file_cache_errors on;

 gzip on;
 gzip_static on;
 gzip_disable "msie6";
 gzip_http_version 1.1;
 gzip_vary on;
 gzip_comp_level 6;
 gzip_proxied any;
 gzip_types text/plain text/css application/json

application/x-javascript text/xml application/xml application/xml+rss
text/javascript application/javascript text/x-js;
gzip_buffers 16 8k;

 include /etc/nginx/conf.d/*.conf;



================ website config ==================

server {
listen 80;
access_log /var/webs/wwwexample/log/access_log main;
error_log /var/webs/wwwexample/log/error_log warn;
root /var/webs/wwwexample/www/;

 index  index.php index.html index.htm index.cgi default.html

default.htm default.php;
location / {
try_files $uri $uri/ /index.php?$args;

 location /nginx_status {
    stub_status on;
    access_log   off;
    deny all;

 location ~*

/(images|cache|media|logs|tmp)/.*.(php|pl|py|jsp|asp|sh|cgi)$ {
return 403;
error_page 403 /403_error.html;

 location ~ /\.ht {
     deny  all;

 location /administrator {
     deny all;

 location ~ \.php$ {

     # Setup var defaults
     set $no_cache "";
     # If non GET/HEAD, don't cache & mark user as uncacheable for 1

second via cookie
if ($request_method !~ ^(GET|HEAD)$) {
set $no_cache “1”;
# Drop no cache cookie if need be
# (for some reason, add_header fails if included in prior
if ($no_cache = “1”) {
add_header Set-Cookie “_mcnc=1; Max-Age=2; Path=/”;
add_header X-Microcachable “0”;
# Bypass cache if no-cache cookie is set
if ($http_cookie ~* “_mcnc”) {
set $no_cache “1”;
# Bypass cache if flag is set
fastcgi_no_cache $no_cache;
fastcgi_cache_bypass $no_cache;
fastcgi_cache microcache;
fastcgi_cache_key $scheme$host$request_uri$request_method;
fastcgi_cache_valid 200 301 302 10s;
fastcgi_cache_use_stale updating error timeout invalid_header
fastcgi_pass_header Set-Cookie;
fastcgi_pass_header Cookie;
fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

     try_files $uri =404;
     include /etc/nginx/fastcgi_params;
     fastcgi_param PATH_INFO $fastcgi_script_name;
     fastcgi_intercept_errors on;

     fastcgi_buffer_size 128k;
     fastcgi_buffers 256 16k;
     fastcgi_busy_buffers_size 256k;
     fastcgi_temp_file_write_size 256k;
     fastcgi_read_timeout 240;

     fastcgi_pass unix:/tmp/php-fpm.sock;

     fastcgi_index index.php;
     include /etc/nginx/fastcgi_params;
     fastcgi_param SCRIPT_FILENAME 



 location ~* \.(ico|pdf|flv)$ {
     expires 1d;

 location ~* \.(js|css|png|jpg|jpeg|gif|swf|xml|txt)$ {
     expires 1d;


================= php-fpm.conf ===================
pid = /var/run/php-fpm/
error_log = /var/log/php-fpm/error.log

daemonize = no

============== php-fpm.d/www.conf ================

listen = /tmp/php-fpm.sock
listen.allowed_clients =
user = nginx
group = nginx

pm = dynamic
pm.max_children = 1024
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

slowlog = /var/log/php-fpm/www-slow.log

php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
php_admin_value[memory_limit] = 128M

php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session


================ mysql my.cnf ====================


query_cache_limit = 2M
query_cache_size = 200M
key_buffer = 100M
join_buffer = 2M
table_cache= 150M
sort_buffer= 2M



=============== mysqltuner report ================

MySQLTuner 1.2.0 - Major H. [email protected]

-------- General Statistics

[–] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.34
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics

[–] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[–] Data in MyISAM tables: 9M (Tables: 80)
[–] Data in InnoDB tables: 1M (Tables: 65)
[–] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[–] Data in MEMORY tables: 0B (Tables: 4)
[!!] Total fragmented tables: 66

-------- Security Recommendations

[OK] All database users have passwords assigned

-------- Performance Metrics

[–] Up for: 12h 51m 16s (21K q [0.471 qps], 1K conn, TX: 10M, RX: 1M)
[–] Reads / Writes: 55% / 45%
[–] Total buffers: 694.0M global + 21.4M per thread (151 max threads)
[!!] Maximum possible memory usage: 3.8G (135% of installed RAM)
[OK] Slow queries: 0% (0/21K)
[OK] Highest usage of available connections: 23% (36/151)
[OK] Key buffer size / total MyISAM indexes: 150.0M/5.1M
[OK] Key buffer hit rate: 99.3% (51K cached / 358 reads)
[OK] Query cache efficiency: 80.9% (10K cached / 13K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 55 sorts)
[OK] Temporary tables created on disk: 8% (5 on disk / 60 total)
[OK] Thread cache hit rate: 98% (36 created / 1K connections)
[OK] Table cache hit rate: 20% (192 open / 937 opened)
[OK] Open file limit used: 0% (210/200K)
[OK] Table locks acquired immediately: 99% (4K immediate / 4K locks)
[!!] Connections aborted: 8%
[OK] InnoDB data size / buffer pool: 1.1M/128.0M


Please advise.

Thanks and Regards,