Ruby Forum Mongrel > Mongrel performance study

Hi mongrelians,

I found myself with an unloaded full production server recently which
presented an opportunity to run some performance tests against Mongrel
and nginx. I thought this list might have some interest in the results
of my tinkering!

I actually ran two sets of studies, and Wayne Seguin kindly provided a
lot of guidance and support in getting them running. Thanks Wayne!
Also, EngineYard provided the hardware for the test.

The first test looked at Mongrels running with nginx fair proxy vs.
without it:

http://www.misuse.org/science/2008/04/07/thin-ruby-on-rails-nginx-fair-proxy-performance-testing/

The second test looked at Mongrels vs Thin as the application handler
for Rails:

http://www.misuse.org/science/2008/04/07/thin-vs-mongrel-a-ruby-on-rails-performance-shootout/

Overall the results were not so surprising but at least there's a
little more hard data for people to look at while making decisions
about this stuff.

Summary 1: Fair proxy can boost performance significantly on sites that
have a mix of 95% fast (<1s) and 5% slow-ish (4s) pages.

Summary 2: Unix sockets are probably 4% faster than IP sockets (and
therefore Thin is about 4% faster than Mongrel).

One thing I did notice was that under heavy load nginx fair proxy
caused some weird "clustering" of requests - the graphs (unpublished)
show it pretty clearly while the statistics (published above) do not.
So while fair proxy looks pretty good overall, if you are heavily
loaded, I might recommend from the study that you do not use fair
proxy, at least the version I tested (current stable release).

Your comments and feedback are welcome. I hope the studies' information
and quality are usable to you. It's a very small contribution back to
this great community and the people who have contributed so much more.

Steve

On Mon, 07 Apr 2008 23:14:47 -0700
Steve Midgley <public@misuse.org> wrote:

> Hi mongrelians,
> 
> http://www.misuse.org/science/2008/04/07/thin-ruby-on-rails-nginx-fair-proxy-performance-testing/

Yay! You mentioned standard deviations.  Bad! You didn't give any.
That'd help people figure out if the 4% is even worth it.

> The second test looked at Mongrels vs Thin as the application handler 
> for Rails:
> 
> http://www.misuse.org/science/2008/04/07/thin-vs-mongrel-a-ruby-on-rails-performance-shootout/

"Thin is the newest incarnation of Zed Shaw’s (and now community
managed) masterwork Mongrel."

That statement isn't correct.  Please don't make it since I don't think
the author of Thin or the authors of Mongrel would say that.  Oh, well
they'd say my stuff is a masterwork, just not that Thin is an
incarnation of Mongrel. :-)

> Overall the results were not so surprising but at least there's a 
> little more hard data for people to look at while making decisions 
> about this stuff.
> 
> Summary 1: Fair proxy can boost performance significantly on sites that 
> have a mix of 95% fast (<1s) and 5% slow-ish (4s) pages.
> 
> Summary 2: Unix sockets are probably 4% faster than IP sockets (and 
> therefore Thin is about 4% faster than Mongrel).

I don't think it's unix sockets vs. ip sockets.  These days those
aren't really that much faster than a localhost connection thanks to
advancements in performance for internal IPC.

Still, 4% increase for all that work done in Thin is kind of lame.
These kinds of results keep bringing me back to the real problem:
Ruby.  It's not what IO loop you use, or threading model, or socket
type, but more just that Ruby is slow and it's processing sucks.

Another comment I'd make is, why not also release the methodology you
gave.  I'm actually working on a presentation for RuPy so if you'd like
to hack on a repeatable study with an automated report let me know.
Could be a good test for what I'm writing for the conference.

--
Zed A. Shaw
- Hate: http://savingtheinternetwithhate.com/
- Good: http://www.zedshaw.com/
- Evil: http://yearofevil.com/

On 8 Apr 2008, at 08:20, Zed A. Shaw wrote:
>
> I don't think it's unix sockets vs. ip sockets.  These days those
> aren't really that much faster than a localhost connection thanks to
> advancements in performance for internal IPC.
>
> Still, 4% increase for all that work done in Thin is kind of lame.
> These kinds of results keep bringing me back to the real problem:
> Ruby.  It's not what IO loop you use, or threading model, or socket
> type, but more just that Ruby is slow and it's processing sucks.

I believe this is all correct, however there are other advantages to
unix sockets for certain setups. They're easier to secure (this is not
always relevant, but it can be). In some scenarios they can be easier
to manage too (you can glob them, it's harder to 'glob' using
netstat). I know that was a discussion about performance, this is just
my $0.02 :-)

>  I'm actually working on a presentation for RuPy so if you'd like
>  to hack on a repeatable study with an automated report let me know.

ebb includes four different benchmarks that i often use to measure
against thin, mongrel, fastcgi, and evented mongrel.
http://github.com/ry/ebb/tree/master/benchmark
but i will present these at RuPy too!

ry