Forum: NGINX Curious problem with PHP

F7ec36678a4102de7a6895d299a60819?d=identicon&s=25 Ian (Guest)
on 2014-06-28 21:39
(Received via mailing list)
Hi all,

I have been investigating a performance problem with my
nginx/init-fastcgi/PHP set up and I'm not having much success. All ideas
gratefully received.

It appears as if the PHP worker processes return their reply (in 100ms
or so), and then hang about for 500-700ms before being able to handle
the next message. This results in worker starvation under remarkably low
loads.

The application is a chat room. I am using the Nginx HTTP push module.
POSTed messages are passed to PHP, which uses its CURL module to write
to the /publish page. PHP also talks to MySQL for data storage.
Javascript fetches from the /subscribe address, updates the screen and
requests again. GETs from /subscribe do not involve PHP.

I know the delay is after the reply, because Firebug shows the response
time is 100 to 200ms - as expected. Ping times are 100ms plus).

With 5 PHP worker processes and three people paste/sending as fast as
they can, it is very easy to overload the system.  (I can nearly do it
on my own).  Each browser stores the POSTs waiting for the previous
reply thus each browser can only have one message in PHP at a time. Thus
there cannot be more than three messages in the PHP system at once.  So
how can 5 workers be overloaded?

I have stopped and counted 5 of my own messages coming through over the
next 7-8 seconds.  It appears from comments made on the phone, that all
messages arrive together - just late.  No messages are dropped.

So it appears that worker starvation is happening - but not long
response times. Server load is also slight (machine load is pushed up
from 0.3% to 2% out of 400% - 4CPUs).

Conclusion: the worker processes are hanging about after sending their
reply for some considerable time (guess 500-700ms) before they can
handle a new message.

In an effort to identify what might cause the PHP worker process to hang
about I have:
      released the session as early as possible - no change.
      switched the MySQL processing from a transaction to auto-commit -
this made things slightly worse.
      doubled the VPS memory - this had minimal effect but removes the
possibility of swapping
      attempted to use keep-alive between nginx and the fastcgi process
(and failed. Gave up as this will probably going to make things worse -
no keep alive is the default).

Using nginx 1.4.4 , PHP 5.3.10

Has anyone got an idea as what might be happening? I'm stumped.

Thanks

Ian
671d9faabfe3d3382be736b93fbfa1d5?d=identicon&s=25 Steve Holdoway (Guest)
on 2014-06-29 08:31
(Received via mailing list)
Hi,

On Sat, 2014-06-28 at 20:39 +0100, Ian wrote:
>
> they can, it is very easy to overload the system.  (I can nearly do it
> response times. Server load is also slight (machine load is pushed up
> this made things slightly worse.
> Thanks
>
> Ian

First check: nginx/init-fastcgi/PHP setup... you *do* mean nginx /
php-fpm don't you??

There'll be at least some suggestions to get a bit more up-to-date with
nginx 1.6+, php 5.4+, but, ignoring those...

My first step ( I'm a sysadmin, so look at every application as a black
box and measure what's happening to the infrastructure ) would be to
install some kind of monitoring, munin being my preference, which has
loads of useful extras to monitor exactly what the server, nginx, mysql,
php-fpm, APC ( you have installed it I hope! ), and everything else is
doing. It's all well and good adding extra memory for example, but if
you don't configure the infrastructure to use it, then it may just get
wasted.

It could be a database problem. How much resource is allocated to this,
and are you saving session data in it? There's a simple script out there
called mysql_tuner.sh which will give you a quick overview of how the
database resources are being used.

Hopefully these will aid you in identifying the bottleneck.

Steve
--
Steve Holdoway BSc(Hons) MIITP
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.