Ruby Forum Rails deployment > large scale deployment

Posted by Damien Hunter (Guest)
on 04.01.2008 11:30
(Received via mailing list)
I'm looking to deploy a digital sign-up/appointment sheet application
at a university. Can anyone give suggestions on the best setups to
handle sporadic use by 8,000 individuals?

Two major factors here are reliability and cost: meaning reliability
needs to be high, and cost needs to be low. However, the data is being
served on the same LAN as the clients.
Posted by Damien Hunter (Guest)
on 04.01.2008 11:39
(Received via mailing list)
To be more specific, let's ballpark that at any given second there are
40 users sending server requests. At rare peak times there may be
100+.
Posted by Jason Roelofs (Guest)
on 04.01.2008 13:38
(Received via mailing list)
Note: There's no such thing as a "best" setup. For your simple case, the
following will do.

From my experience, you'll be fine with a single beefy server, a good 
HTTP
server front-end (I highly recommend nginx) proxying off to mongrels 
(start
with 10 or so, play around with the number to see what kind of 
performance
you can sustain).

For my server setup this way, I easily pull 300 req / sec.

Jason
Posted by Ericson Smith (Guest)
on 04.01.2008 15:52
(Received via mailing list)
I'd suggest you do the following:

1. Get two servers, one for web and another for DB
2. Web server should have 4 gigs of ram at least and at least 4 cores (8
would be better)
3. DB well scaled and connected directly to web server
4. Go with Litespeed web server. Why? Because here you have only one 
tier to
content with instead of ngix + mongrel + your app, or apache + mongrel +
your app. Besides its super easy to setup.

We have that setup at http://www.funadvice.com and we get a few hundred
thousand requests per day with typical server load of 0.3 even at peak 
hours
of 3pm EST. One of the biggest things with Litespeed is that you no 
longer
have to worry about keeping your mongrels up, scaling them, or having 
them
not consume too much memory along with your app. Litespeed controls all 
that
crap for you my gracefully restarting and scaling depending on traffic.

There's a free version, or an enterprise version to pick your poison.

- Ericson Smith
CTO
Funadvice.com
Posted by Alexey Verkhovsky (Guest)
on 04.01.2008 19:22
(Received via mailing list)
On Jan 4, 2008 7:51 AM, Ericson Smith <esconsult1@gmail.com> wrote:
> 4. Go with Litespeed web server.

Litespeed, as far as I have seen, works pretty well, but... Their free
version has this little downside:
  "Multi-Processor Scalability: No"

If that means limiting the server to a single request handler thread
(what else could it be?), then you'd have to buy the non-free version
for production use. Correct me if I'm wrong.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]
Posted by Philip Hallstrom (Guest)
on 04.01.2008 19:25
(Received via mailing list)
>> 4. Go with Litespeed web server.
>
> Litespeed, as far as I have seen, works pretty well, but... Their free
> version has this little downside:
>  "Multi-Processor Scalability: No"
>
> If that means limiting the server to a single request handler thread
> (what else could it be?), then you'd have to buy the non-free version
> for production use. Correct me if I'm wrong.

Why?  All depends on the amount of traffic.  My memory is the original
poster wasn't going to have that many users (despite the subject) but I
could be wrong.

Sure, you'd get more from the production version, but the free should do
pretty well for quite a bit of traffic.
Posted by Ericson Smith (Guest)
on 04.01.2008 19:29
(Received via mailing list)
We use the free on another pretty large site with over a million uniques 
per
month. Works pretty well for us.

- Ericson Smith
CTO
http://www.funadvice.com
Posted by Alexey Verkhovsky (Guest)
on 04.01.2008 20:06
(Received via mailing list)
On Jan 4, 2008 1:44 PM, Philip Hallstrom <rails@philip.pjkh.com> wrote:
> > If that means limiting the server to a single request handler thread
> > (what else could it be?), then you'd have to buy the non-free version
> > for production use.
> Why?  All depends on the amount of traffic.

Of course. However, on a quad core commodity hardware of today, this,
very roughly speaking, limits your throughput to one fourth of what
your server is otherwise capable of. Even less, if you have a separate
DB box. Ruby is a slow language, therefore Rails apps are almost
invariably CPU-bound. Assuming I understand the meaning of
"Multi-Processor Scalability: No" correctly.

> We use the free on another pretty large site with over a million uniques per month
Question is: how many dynamic page hits per second at peak load does
that translate to? Single core, with Rails == 10, maximum.

By the way, if you are going to handle one request at a time, anyway,
single Mongrel on port 80 works just fine, too.
Litespeed (paid edition) is good for handling multiple low traffic
sites (where you are bottlenecked by RAM, not CPU). It's not bad for
hosting a single application, either, as long as you are OK with
paying money for commercial close-sourced middleware, but a Mongrel
cluster is free and not all that hard to set up and run, either.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]
Posted by Ericson Smith (Guest)
on 04.01.2008 20:35
(Received via mailing list)
Really, it comes down to this... Does your site have a revenue model?

If it does, the $1299 (or $699 per year, or $65 per month) that 
Litespeed
charges for the 4 CPU version, is chump change. If you run a database on 
the
same server, then you will only need the 2 CPU version for $799 (since 
you
need the other 2 CPU's for OS and database activity anyway). If you 
don't
have a revenue model, then go with one of the free stacks instead.

I don't work for litespeed or anything, we just use their software (both
free and paid versions). And really, we do have a revenue model (and are 
not
funded), and after 25 years in the programming deployment webmaster biz, 
i
really dont have time to waste with a stack that bombs out, so 
discovering
litespeed a few months ago was a big boon for us.

I just feel sad when I hear people crying over deployment issues with 
rails
when there is at least one decent solution out there available, and they
just stick their heads in their sand with Apache + Mongrel or nginx +
Mongrel, or Apache + fcgi or lighty + mongrel (all sheperded by monit). 
All
those are pretty good solutions, and we've tried them all, none have 
come
close to making it possible to pretty much relax about the web serving 
part
of the deployment as litespeed has.

Zed brought us out of the fcgi ghetto with Mongrel. Amen to that. Its 
time
that someone wrote a good mod_ruby for apache or something. That to me 
is
where we need to head.

- Ericson

On Jan 4, 2008 2:05 PM, Alexey Verkhovsky <alexey.verkhovsky@gmail.com>
Posted by Jason Roelofs (Guest)
on 04.01.2008 23:34
(Received via mailing list)
How does litespeed compare to Swiftiply, if there's anyone who has used
either / both? I am quite enjoying the nginx + Mongrel setup, but we are 
not
yet to the point of heavy traffic and needing to scale to multiple 
servers
and the like.

Jason
Posted by Ericson Smith (Guest)
on 05.01.2008 00:14
(Received via mailing list)
It seems that you still have to manually start Swiftiply processes as 
your
traffic scales. You set the mins and maxes and litespeed does it for 
you.
This is what I meant by not having to worry about the web serving 
deployment
aspect of your application.

Swiftiply does still seem like the answer to some of the large issues 
with
mongrel though. Based on my cursory reading of the site, it still seems 
that
you will have the problem of stopping/starting/restarting processes 
though.
Lets face it, our Rails apps will from time to time have memory leaks 
and
crash their containers. You may be lucky to write the perfect app that 
does
neither, but in my experience, this is a pretty common thing.

Having something that manages that aspect is a no brainer.

- Ericson Smith
CTO
http://www.funadvice.com
Posted by Thibaut Barrère (thbar)
on 05.01.2008 03:18
(Received via mailing list)
> 4. Go with Litespeed web server. Why? Because here you have only one tier to
> content with instead of ngix + mongrel + your app, or apache + mongrel +
> your app. Besides its super easy to setup.

+1 on this one. I don't work for Litespeed neither, but I really
recommend having a look at LiteSpeed (either standard or enterprise
edition), pretty much for the same reasons Ericson Smith describes.

I've seen LS overlooked by some because it was not open-source, but
really their software is great, simple to use and reliable.

hope this helps!

Thibaut Barrère / LoGeek
--
http://blog.logeek.fr - about writing software
http://evolvingworker.com - tools for a better day
Posted by tmornini@engineyard.com (Guest)
on 05.01.2008 08:01
(Received via mailing list)
On Jan 4, 2:59 pm, "Ericson Smith" <esconsu...@gmail.com> wrote:

> It seems that you still have to manually start Swiftiply processes as your
> traffic scales. You set the mins and maxes and litespeed does it for you.
> This is what I meant by not having to worry about the web serving deployment
> aspect of your application.

The problem with auto start scaling is that it destabilizes the
environment. I don't mean that you'll crash more, but that your
performance will change over time, and sometimes in ways that you
might not predict.

For high volume sites, you want to bring up the system initially into
an optimal state, and then keep it there, IMHO.

Auto starting processes is good for low end stuff, but not really
applicable to high volume sites

> Swiftiply does still seem like the answer to some of the large issues with
> mongrel though. Based on my cursory reading of the site, it still seems that
> you will have the problem of stopping/starting/restarting processes though.
> Lets face it, our Rails apps will from time to time have memory leaks and
> crash their containers. You may be lucky to write the perfect app that does
> neither, but in my experience, this is a pretty common thing.
>
> Having something that manages that aspect is a no brainer.

You can use monit or god to monitor those processes. One advantage of
this method is that you only need to make sure your process monitor
starts automatically at startup, and then it will handle starting the
application servers too. This is, of course, exactly the same behavior
as LightSpeed, so I'm not claiming advantages here.

We'll soon be releasing an open source plug-in for nginx called "nginx
fair proxy" that using a much improved proxy algorithm that limits the
number of connections nginx will open against mongrel. We've seen far
better performance, and much less mongrel bloat on high volume sites
during our testing.

nginx + mongrel ++ (with or without evented mongrel)

Also, take a look at merb with datamapper for applications that need
really high performance.
Posted by Raymond O'Connor (nappin)
on 05.01.2008 12:41
Regarding litespeed and the multiprocessor scalability:  I do notice it 
spawns several ruby processes when you start the server. As far as I can 
tell, there's nothing in the free version that keeps these ruby 
processes from being run on multiple cores...    So I think you do get 
some advantage with multiple cores even with the free version.  Correct 
me if I'm wrong though...

But just to echo what everyone else has been saying, our company has had 
great success with the litespeed server.  We've been using it for about 
6 months now with several rails apps and have had no problems.  And 
whenever I have a question, their support people on their forum usually 
respond within a few hours with an answer.

-Ray

Posted by Ericson Smith (Guest)
on 05.01.2008 15:02
(Received via mailing list)
That is correct. The ruby processes actually will run on multiple cores. 
Its
only the dispatcher that looks and traffic and dishes out to the ruby 
lsapi
processes that actually runs on one core.

For tmornini@engineyard.com:
Of course unlimited scaling will destabilize your environment. That's 
why
you do some basic investigations and set your min and max values for 
spawned
lsapi children. Its the same issue if the startup 20 mongrels on a dual 
core
machine. The point is if I know that my theorerical maximums are 10 
mongrels
(lsapi clients). Then once I set that, I no longer have to worry about 
it.
Litespeed will put them up or tear them down -- up to the maximums I 
set, or
down to the minimums -- based on traffic, memory and load. So 
destabilizing
your environment is not an issue. After all Apache does the same thing
governed my max child clients.

And the gravy on the steak? You don't need monit for all of that.

- Ericson Smith
CTO
http://www.funadvice.com

On Jan 5, 2008 6:41 AM, Raymond O'Connor 
<ruby-forum-incoming@andreas-s.net>
Posted by Chris Richards (chris)
on 05.01.2008 15:47
(Received via mailing list)
Does multiprocessor mean it will only use one physical processor or
just one core?  If it's just one processor then a quad core would be
fine.

Chris

On Jan 5, 11:41 am, Raymond O'Connor <ruby-forum-incom...@andreas-
Posted by Ericson Smith (Guest)
on 05.01.2008 16:28
(Received via mailing list)
There is one parent process that starts up child processes. The parent
process is somewhat akin to a proxy (apache etc) dispatching requests to
multiple mongrels. The parent process for the free version runs on 1 
CPU.
However each child process will run on its own processor.

I believe litespeed says the free version can handle up to 150 requests 
per
second, which means you can handle nearly 13 million hits per day with
decent hardware. Pretty good for the free version I think.

- Ericson Smith
CTO
http://www.funadvice.com
Posted by Philip Hallstrom (Guest)
on 05.01.2008 16:43
(Received via mailing list)
> I believe litespeed says the free version can handle up to 150 requests per

150 *simultaneous* *connections*. Very different from 150 r/s.

But yeah, it's still plenty for a lot of sites out there.
Posted by tmornini@engineyard.com (Guest)
on 05.01.2008 21:39
(Received via mailing list)
On Jan 5, 6:02 am, "Ericson Smith" <esconsu...@gmail.com> wrote:

> Of course unlimited scaling will destabilize your environment. That's why
> you do some basic investigations and set your min and max values for spawned
> lsapi children. Its the same issue if the startup 20 mongrels on a dual core
> machine.

No, it's decidedly not the same. If you startup 20 mongrels on a dual
core machine, then the performance will be the same whether those 20
are in use or not. It's not an issue of CPU, to a large extent, but of
memory...

> The point is if I know that my theorerical maximums are 10 mongrels
> (lsapi clients). Then once I set that, I no longer have to worry about it.

But why wouldn't you want all 10 running if 10 is the maximum?

> Litespeed will put them up or tear them down -- up to the maximums I set, or
> down to the minimums -- based on traffic, memory and load.

Why would you pay the penalty of processes startup and teardown?

> So destabilizing your environment is not an issue.

Sure it is. What happens if you have high traffic during a cron job?
You'll have max mongrels running, using all available RAM with enough
left over to allow the machine to run efficiently. The cron job starts
and *bam* you're in swap.

Now, I know what you're going to say: You've set the maximum too high.
Yes, I agree with that technically, but in practice, a LOT of people
forget about daily, weekly, and monthly cron jobs, etc.

Also, people tend to log into production machine and run code for
various purposes, sometimes as part of routine. With fluctuating
number of mongrel processes, this can work for a long time, then
suddenly take the machine into swap for no apparent reason.

Technically, you're 100% right, all of this can be avoided with your
method. Operationally, I'm quite convinced it's not a good idea. :-)

> After all Apache does the same thing governed my max child clients.

Yes, and in my experience large websites don't use this feature for
the reasons I've mentioned above. :-)

> And the gravy on the steak? You don't need monit for all of that.


But you do need monit to monitor lightspeed itself, so you've gained
nothing. :-)
Posted by Ericson Smith (Guest)
on 05.01.2008 22:41
(Received via mailing list)
Lets start with the last thing u said. Litespeed has basically the same 
kind
of stability of apache. Do you really need monit to monitor apache? 
Maybe if
you're seriously having some kind of major app that will break the
underlying web server, then yes, but for 99% of us, no.

I think what you're forgetting I can also set litespeed to adjust based 
on
memory AND process limits per user.

I can set 2 min and 10 max and forget about it. Litespeed adds more if
necessary, then based on capacity, gracefully ramps down if I dont need
them, thus leaving the system for other things. A great example is if 
I'm
running CRON's in the middle of the night, this is when my traffic is 
low.
Why would I want 10 instances running? I can use the resources used from 
not
having 8 useless instances running doing nothing.

My point in all of this is not to knock the great open source software 
out
there which I have used and will continue to do so, but just to point 
out
that Litespeed is an extremely credible contender for Rails. It really 
makes
it super simple to get going with top-notch performance as a great side
bonus. For guys like me who want to spend time coding and dealing with 
the
strategic issues on my site (our core competence), Litespeed is a 
godsend.

- Ericson Smith
CTO
http://www.funadvice.com
Posted by Alexey Verkhovsky (Guest)
on 05.01.2008 23:49
(Received via mailing list)
On Jan 5, 2008 1:38 PM, tmornini@engineyard.com <tmornini@gmail.com> 
wrote:
> We'll soon be releasing an open source plug-in for nginx called "nginx
> fair proxy" that using a much improved proxy algorithm that limits the
> number of connections nginx will open against mongrel.

That would be awesome.

--
Alexey Verkhovsky
CruiseControl.rb [http://cruisecontrolrb.thoughtworks.com]
RubyWorks [http://rubyworks.thoughtworks.com]
Posted by Philip Hallstrom (Guest)
on 06.01.2008 04:53
(Received via mailing list)
>> The point is if I know that my theorerical maximums are 10 mongrels
>> (lsapi clients). Then once I set that, I no longer have to worry about it.
>
> But why wouldn't you want all 10 running if 10 is the maximum?

Not as useful for consistently large traffice sites, but to take it
another direction, I love litespeed for the fact that I can run several
Rails sites on my little 1gb box that also does postfix/postgresql/mysql
and all sorts of other things and not have to have 6-7 mongrel processes
running constantly.  The sites are so small that most of the time there
aren't any lsapi processes running.

True, this isn't large scale, but it's a nice feature :)

I also suspect that we all may have different ideas of large scale :)

>> And the gravy on the steak? You don't need monit for all of that.
>
> But you do need monit to monitor lightspeed itself, so you've gained
> nothing. :-)

What monitors monit?  :)