PHP/FCGI balancing

igorclark · June 17, 2008, 5:46pm

Hi there, two related questions, one quickie, one more involved

Is it possible to specify weighting between upstream FastCGI
servers? If not in nginx config, then using some clever scheme?
Your thoughts on the following situation would be welcome!

Our application is 80% Flash with many AMF callbacks to dynamic PHP/
MySQL services. A majority of the actual bandwidth going out will be
on SWFs and images.

Today we’re pushing 20mbps in the pre-launch sign-up phase; the main
campaign doesn’t launch till next week.

We have 4 servers, quad-core Xeon 2.33ghz. 2 have 8GB RAM, the other 2
have 4GB.

The current set up is thus:

one 8GB is nginx, front-end reverse proxy and static web server;
one of the 4GB boxes is php/fcgi;
the other 4GB box is mysql.

The second 8GB has only just become available as it was being used on
another, now closed, project.

The 3 deployed boxes are all handling the current traffic without
breaking a sweat, low loads, high idle CPU percentages, plenty of RAM
free, no waiting on IO. However of course as the new gear has become
available and will be sitting around doing nothing otherwise, I want
to use it to best effect in next week’s campaign.

We have a hardware load balancer available, weighted round robin, in
front of the 2 8GB boxes.

I’m considering whether to:

leave all web serving to 8GB box 1, and dedicate 8GB box 2 to PHP
application serving, or
load-balance web traffic between both 8GB boxes, say 70% to existing
box and 30% to new box, and run PHP app server on new 8GB box.

As per Q1, I’d like to be able to specify some weighting on the
fastcgi–>PHP traffic so that I could tune which box processes what a
bit more finely, but from what I can see in the nginx documentation,
this isn’t possible.

The latter way seems to spread the load, but the former way dedicates
machines to doing one thing at a time. Given that the nginx box is
currently running at loads of less than 0.1 and has over 2GB of RAM
free, perhaps the extra nginx muscle isn’t necessary.

There is a bit of processing overhead on the PHP side, as the AMF
codec processing has to be done in PHP (the PHP AMF C extension has a
bug with AMF3 which breaks our app) - although we’re using APC, which
seems to speed things up nicely.

What do you think would be most sensible?

Many thanks for any thoughts.
Igor

igorclark · June 17, 2008, 7:06pm

Hello!

On Tue, Jun 17, 2008 at 04:36:04PM +0100, Igor C. wrote:

Hi there, two related questions, one quickie, one more involved

Is it possible to specify weighting between upstream FastCGI servers?
If not in nginx config, then using some clever scheme?

http://wiki.codemongers.com/NginxHttpUpstreamModule#server

have 4GB.
The 3 deployed boxes are all handling the current traffic without

leave all web serving to 8GB box 1, and dedicate 8GB box 2 to PHP
The latter way seems to spread the load, but the former way dedicates
machines to doing one thing at a time. Given that the nginx box is
currently running at loads of less than 0.1 and has over 2GB of RAM
free, perhaps the extra nginx muscle isn’t necessary.

There is a bit of processing overhead on the PHP side, as the AMF codec
processing has to be done in PHP (the PHP AMF C extension has a bug with
AMF3 which breaks our app) - although we’re using APC, which seems to
speed things up nicely.

What do you think would be most sensible?

It’s unlikely that nginx box will be bottleneck unless you have IO-bound
workload (i.e. many static files). Looks like another PHP server
whould be better choice.

Maxim D.

igorclark · June 17, 2008, 7:43pm

On 17 Jun 2008, at 17:57, Maxim D. wrote:

Hello!

On Tue, Jun 17, 2008 at 04:36:04PM +0100, Igor C. wrote:

Hi there, two related questions, one quickie, one more involved

Is it possible to specify weighting between upstream FastCGI
servers? If not in nginx config, then using some clever scheme?

http://wiki.codemongers.com/NginxHttpUpstreamModule#server

Superb.

Your thoughts on the following situation would be welcome!
[…]
What do you think would be most sensible?

It’s unlikely that nginx box will be bottleneck unless you have IO-
bound workload (i.e. many static files). Looks like another PHP
server whould be better choice.

And with the weighting on the PHP that makes even more sense.

Thanks very much, Maxim.

Best wishes
Igor

igorclark · June 18, 2008, 11:03am

Igor C. wrote:

bit more finely, but from what I can see in the nginx documentation,
seems to speed things up nicely.

What do you think would be most sensible?
Would it not be better to use the two 8’s for MySQL and PHP (same set-up
on both, with the mysql db replicating from one to the other), and have
the two 4’s as nginx servers in a fail-over scenario? I’d’ve thought
that nginx wouldn’t benefit as much from the ram as PHP & MySQL would,
and ensuring that if one of the “app” servers goes down its not the end
of the world as you can still serve content.

–

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]

Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

igorclark · June 18, 2008, 11:26am

Thanks Philip,

That’s a good suggestion. I had considered swapping them around to
make the most of what we now have, but there’s so much stuff built and
running live on the existing nginx <–> php <–> mysql setup already
that I’m really wary of making wholesale changes like that before the
main thing goes live next week, and unfortunately I definitely don’t
have time to start getting into DB replication issues. The main
campaign’s only live for a week! Hence my wish to make the best use of
the unexpected extra box in line with the existing setup rather than
making big changes.

Appreciate your reponse. I’ll bear it mind on the next project!

Best wishes
Igor

igorclark · June 18, 2008, 12:46pm

On Wed, Jun 18, 2008 at 09:51:02AM +0100, Phillip B Oldham wrote:

fastcgi–>PHP traffic so that I could tune which box processes what a
bug with AMF3 which breaks our app) - although we’re using APC, which
seems to speed things up nicely.

What do you think would be most sensible?
Would it not be better to use the two 8’s for MySQL and PHP (same set-up
on both, with the mysql db replicating from one to the other), and have
the two 4’s as nginx servers in a fail-over scenario? I’d’ve thought
that nginx wouldn’t benefit as much from the ram as PHP & MySQL would,
and ensuring that if one of the “app” servers goes down its not the end
of the world as you can still serve content.

nginx itself does not require much memory as MySQL or PHP.
However, the memory may be used by OS VM cache.

igorclark · June 18, 2008, 12:55pm

Igor, with a large amount of RAM to play with, would nginx find it
beneficial to have its
client-body-temp-path/fastcgi-temp-path/proxy-temp-path pointing to a
tmpfs (ramdisk) directory?

Igor S. wrote: