Again: Workaround found for request queuing vs. num_processors, accept/close

Hello all.
I too found out that I sometimes have some action that can take up to 10
seconds in my rails application. I’ve read all arguments Zed made about
polling/and inbox strategies, and I think I just can’t work around my
feeling
that a “wrong” request that takes up too much time should be able to
lock
subsequent requests in mongrels queue. That’s what I have a balancer
for,
right? :slight_smile:
So I tried Robert M.s hint[1] and used
BalancerMember http://foo max=1 acquire=100
to see if that helps. But I may have hit a bug[2], and this does not
work for
me.
Does this work for someone? With what version of Apache? I am running
2.2.4.
Or do all of you have only very fast requests that do not take more than
a
few seconds max to complete? Or do you run a different load balancer?
HAProxy?
I also think that the overall speed of the application would incerase
even
with relatively fast requests, because the balancer would forward the
requests always to a free mongrel that has nothing to do, and can handle
the
request immediately.

BTW, it looks like the acquire parameter is not in seconds, but in
milliseconds, so I tried acquire=100 as well, but no difference.
From[2]
/* Acquire timeout in milliseconds.
* If set this will be the maximum time to
* wait for a free connection.
*/

Maybe this belongs to the Apache ml and not to mongrel, but I think
maybe some
of you hit this problem and solved it somehow.

Another issue: on my system,
mongrel_rails --version
Mongrel Web Server 1.1.3 (installed as a gem)
the --num-procs parameter does not seem to work. To debug the problem I
added
STDERR.puts “TEST: #{worker_list.length} #{@num_processors}”
on line 279 of /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb,
and
to my surprise it looks like the parameter is ignored. If I run it

$ mongrel_rails start -a 127.0.0.1 -p 8999 --num-procs 1 &
[1] 18617
** Starting Mongrel listening at 127.0.0.1:8999
** Starting Rails with development environment…
config/boot.rb:28:Warning: require_gem is obsolete. Use gem instead.
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready. TERM => stop. USR2 => restart. INT => stop (no
restart).
** Rails signals registered. HUP => reload (without restart). It might
not
work well.
** Mongrel 1.1.3 available at 127.0.0.1:8999
** Use CTRL-C to stop.

and then:

$ w3m -dump http://localhost:8999/test >/dev/null
TEST: 0 950
$

So it looks like @num_processors is still at the default value of 950,
and not
1 as I told it on the command line. Does the deprecation warning maybe
have
something to do with this (I don’t think so, but you never know…)?

Thank you!
Simon

[1]http://rubyforge.org/pipermail/mongrel-users/2007-October/004145.html
[2][Apache-SVN] Contents of /httpd/httpd/trunk/modules/proxy/mod_proxy.c

On Jan 23, 2008 12:18 PM, Simon S. [email protected] wrote:

Or do you run a different load balancer [for limiting the number of outstanding requests].

Many people do it with HAProxy. EngineYard built an Nginx plugin that
does the same (don’t think they released it yet).


Alexey V.
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

On Wednesday 23 January 2008 21:36:39 Alexey V. wrote:

Many people do it with HAProxy. EngineYard built an Nginx plugin that
does the same (don’t think they released it yet).

Thanks for your replay. At least now I know that someone understood my
poor
English :slight_smile:

On Wed, 23 Jan 2008, Simon S. wrote:


So it looks like @num_processors is still at the default value of 950, and not
1 as I told it on the command line. Does the deprecation warning maybe have
something to do with this (I don’t think so, but you never know…)?

hum… this seems like a bug. Could you please open a ticket for it?

Cheers,

filipe {
@ icewall.org
GPG 1024D/A6BA423E
http://filipe.icewall.org/
}

On Wednesday 23 January 2008 22:07:45 Filipe wrote:

hum… this seems like a bug. Could you please open a ticket for it?

sure.
http://rubyforge.org/tracker/index.php?func=detail&aid=17428&group_id=1306&atid=5145

On Wednesday 23 January 2008 23:03:21 Ezra Z. wrote:

Actually the nginx fair balancer has been released for a while. You
can read up on it and get it here:

http://wiki.codemongers.com/NginxHttpUpstreamFairModule

Hmmm. I can’t use a no-name web server with a load balancer that is a
set of
patches for that server on our intranet production server. At the first
problem I would lose my job.

On Jan 23, 2008, at 2:36 PM, Simon S. wrote:

problem I would lose my job.
That’s too bad then, you will have to suffer on with apache i guess.
Nginx is a lot more solid then apache is in stability and performance.
I’m personally running over 1000 nginx daemons with absolutely no
problems to speak of with it.

Cheers-

On Jan 23, 2008, at 12:36 PM, Alexey V. wrote:

On Jan 23, 2008 12:18 PM, Simon S. [email protected]
wrote:

Or do you run a different load balancer [for limiting the number of
outstanding requests].

Many people do it with HAProxy. EngineYard built an Nginx plugin that
does the same (don’t think they released it yet).

Actually the nginx fair balancer has been released for a while. You
can read up on it and get it here:

http://wiki.codemongers.com/NginxHttpUpstreamFairModule

Cheers-

On Wednesday 23 January 2008 23:45:43 Ezra Z. wrote:

That’s too bad then, you will have to suffer on with apache i guess.

Probably.

Nginx is a lot more solid then apache is in stability and performance.
I’m personally running over 1000 nginx daemons with absolutely no
problems to speak of with it.

Do you have your architecture documented somewhere online? Can I read
it?
I need Apache for kerberos for SSO, and I don’t think nginx supports
that
(could not find it). I could put Apache in front of nginx balancing the
mongrels, but at that point it’s maybe better to use haproxy like the
the
rubyworks guys suggest[1] (no need for patches).
I’m open for suggestions.

[1]http://rubyworks.rubyforge.org/manual/haproxy.html

On Jan 23, 2008, at 2:57 PM, Simon S. wrote:

Do you have your architecture documented somewhere online? Can I
[1]http://rubyworks.rubyforge.org/manual/haproxy.html
Hey Simon-

I don’t have any whitepapers about our architecture currently
available. But the 50,000 ft view is like this:

Http request from client → LVS redundant load balancers → Nginx →
mongrel_cluster.

The nginx and mongrel clusters run on the same Xen VM. with multiple
VM’s behind the LVS load balancers.

You are correct that nginx does not support kerberos so you will need
to keep apache. Haproxy is probably a good solution for you in this
case.

Cheers-

On Jan 23, 2008 4:42 PM, Ezra Z. [email protected] wrote:

You are correct that nginx does not support kerberos so you will need
to keep apache. Haproxy is probably a good solution for you in this
case.

+1

About Nginx: besides EngineYard and a lot of other Rails deployments,
Nginx serves something like one-fourth of Russian internet traffic,
IIRC. It’s not exactly no-name, just hasn’t been discovered by the
rest of the world until recently.


Alexey V.
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]

On Wed, 23 Jan 2008 23:36:34 +0100
Simon S. [email protected] wrote:

On Wednesday 23 January 2008 23:03:21 Ezra Z. wrote:

Actually the nginx fair balancer has been released for a while. You
can read up on it and get it here:

http://wiki.codemongers.com/NginxHttpUpstreamFairModule

Hmmm. I can’t use a no-name web server with a load balancer that is a set of
patches for that server on our intranet production server. At the first
problem I would lose my job.

Uh, but, Apache is quite literally “a patchy” version of NCSA httpd. :slight_smile:


Zed A. Shaw

On Wed, 23 Jan 2008 14:45:43 -0800
Ezra Z. [email protected] wrote:

That’s too bad then, you will have to suffer on with apache i guess.
Nginx is a lot more solid then apache is in stability and performance.
I’m personally running over 1000 nginx daemons with absolutely no
problems to speak of with it.

Yep, a single nginx server on a 256M virtual machine with just 2
workers handled a simultaneous Redditing, TechCrunch, TechMeme, and
Slashdotting without so much as a hitch in performance. That’s 170k
uniq visitors in 3 days.

To put this into perspective, I had people ask me to take links off my
pages because the trail-off onto their pages was killing their server.

Of course, I did do some special tuning, but that’s another story.


Zed A. Shaw

Ezra Z. wrote:

Hey Simon-

I don’t have any whitepapers about our architecture currently
available. But the 50,000 ft view is like this:

Http request from client → LVS redundant load balancers → Nginx →
mongrel_cluster.

The nginx and mongrel clusters run on the same Xen VM. with multiple
VM’s behind the LVS load balancers.

Cool, good to know. This setup would be a bit overkill to me, but good
to know how you set things up and how they work.

You are correct that nginx does not support kerberos so you will need
to keep apache. Haproxy is probably a good solution for you in this
case.

All right. I’ve set up apache as reverse-/proxy serving haproxy
balancing the mongrels.
Just to help someone who would like to go down the same road, this is my
apache site config:

<Virtualhost *:80>
ServerName whatever
DocumentRoot /home/myhome/MyHome/public/

ProxyPass /images !
ProxyPass /stylesheets !
ProxyPass /javascripts !

Alias /images /home/myhome/MyHome/public/images
Alias /stylesheets /home/myhome/MyHome/public/stylesheets
Alias /javascripts /home/myhome/MyHome/public/javascripts

ProxyPass / http://127.0.0.1:8000/
ProxyPassReverse / http://127.0.0.1:8000
ProxyPreserveHost on

<Directory /home/myhome/MyHome/public>
Options ExecCGI FollowSymLinks
AllowOverride all
Order allow,deny
Allow from all

and my haproxy.cfg file (took it from rubyworks):

global
web server
maxconn 500
socket, therefore
log 127.0.0.1 local0
user nobody
group nogroup
defaults
log global
mode http
balance roundrobin
maxconn 500
option httplog
option abortonclose
retries 3
redispatch
clitimeout 120000
contimeout 120000
srvtimeout 120000
listen rails :8000
server rails-1 localhost:8001 maxconn 1 check
server rails-2 localhost:8002 maxconn 1 check
server rails-3 localhost:8003 maxconn 1 check

to test I set the clitimeout, contimeout and srvtimeout to high numbers,
so I can see how long actions/requests get handled.
The “maxconn 1” parameter does exactly what it says (one request per
mongrel).
I have to test this setup a bit more, and see what haproxy does if a
mongrel goes down or is disabled.

Thank you all for your feedback.

Simon

On 24 Jan 2008, at 01:46, Zed A. Shaw wrote:

workers handled a simultaneous Redditing, TechCrunch, TechMeme, and
Slashdotting without so much as a hitch in performance. That’s 170k
uniq visitors in 3 days.

To put this into perspective, I had people ask me to take links off my
pages because the trail-off onto their pages was killing their server.

Just a “me too” post - nginx is solid. We ran a site with a ~1Mb
homepage weight (once all the graphics and video clips all over it had
loaded - yeah, I know …) on nginx with 2 workers, and it took 570k
PIs across 65k unique visitors in one day, and that mostly in a 5-hour
period. OK, it was a dedicated dual Xeon with 4GB RAM and not much on
it apart from CentOS base + nginx, but I kept a close eye and I didn’t
see it go above 500Mb, and the process load was low enough to make a
brave man weep.

Of course, I did do some special tuning, but that’s another story.

A story you might be happy to share, perhaps?

i


Igor C. // POKE // 10 Redchurch Street // E2 7DD // +44 (0)20 7749
5355 // www.pokelondon.com