Article: High-Performance Ruby On Rails Setups Test

Rails
1

This week we have started one new project with Ruby on Rails as primary
framework. My first task was to prepare runtime environment for it on
one of our development servers. When I have tried to research how people
doing it, I noted that there is no information about how to deploy rails
application with nginx as frontend and what is performance of such
solution. Before blindly make any decisions about future platform Iâ??ve
decided to make some performance tests of the some most popular rails
web servers/frontends. Results of these tests you can find here with
configuration samples for all software that I have used.

Details:
http://blog.kovyrin.net/2006/08/22/high-performance-rails-nginx-lighttpd-mongrel

2

Alexey Kovyrin wrote:

Details:
High-Performance Ruby On Rails Setups Test: mongrel vs lighttpd vs nginx :: Oleksiy Kovyrin

First of all - great initiative. But - as is posted in the comments on
your blog - testing the default rails index.html page really isn’t
performance testing rails at all - since that’s a static html page. You
need to create an app that probably hits a database on every page view
or at least hits and then caches.

I’m also questioning how you came up with testing 100 simultaneous
connections from one single computer. I’m sure the tcp/ip stack of the
OS in question will be a factor here - and 100 connections
simultaneously is a LOT.

3

As they’ve already told you, there’s a need to make the same tests with
some kind of DB query… Here are my results:

Test machine: Sun Fire X4100 2xOpteron 248, Debian 3.1, kernel
2.6.8-smp, 2GB RAM

Servers tested:

  • Webrick 1.3.1
  • Lighttpd 1.4.11 + 5 fcgi spawns
  • Mongrel 0.3.13.3
  • Pound & 5xMongrel cluster
    (apache will be included in the next round)

The test:

  • New rails application, the database is a list of the results of a
    lottery game in my country (Kino). Modified the ‘list’ action of the
    scaffold-created controller to display all the 350 rows. The resulting
    page is about 500Kb. There’s no cache, the environment is production.
    The log was erased after every test.

The benchmark:
Three different tests:
T1: 20 concurrent connections, 500 total requests
T2: 30 concurrent connections, 500 total requests
T3: 40 concurrent connections, 500 total requests

The command:
/usr/sbin/ab -c $CONN -n $TOTAL http://cerebro:3000/sorteos
Run from another machine in the LAN.

The results:
The test was run 10 times. Results here are the average.

Webrick (only T1)
T1:

  • Total: 460.43 [s]
  • Req/s: 1.09
  • Rate: 570.63 [s]
  • Time/req: 920.00 [ms]

Lighttpd + 5 fcgi spawns:
T1:

  • Total: 207.15 [s]
  • Req/s: 2.41
  • Rate: 1268.18 [s]
  • Time/req: 414.31 [ms]
    T2:
  • Total: 216.23 [s]
  • Req/s: 2.31
  • Rate: 1214.93 [s]
  • Time/req: 432.47 [ms]
    T3:
  • Total: 215.67 [s]
  • Req/s: 2.32
  • Rate: 1218.07 [s]
  • Time/req: 431.35 [ms]

Mongrel (only T1)
T1:

  • Total: 540.90 [s]
  • Req/s: 0.92
  • Rate: 486.05 [s]
  • Time/req: 1081.82 [ms]

Pound + 5xMongrel Cluster
T1:

  • Total: 290.40 [s]
  • Req/s: 1.72
  • Rate: 908.77 [s]
  • Time/req: 580.81 [ms]
    T2:
  • Total: 324.54 [s]
  • Req/s: 1.54
  • Rate: 813.96 [s]
  • Time/req: 649.07 [ms]

In my scenario, the best option appears to be lighttpd + fcgi.

Regards,
Rolando.-

4

Zed S. wrote:

You also have to run each test multiple times and take the mean of them
in order to rule out accidentally getting a bad read. Not to mention a
bunch of other stuff.

quoting myself:

The results:
The test was run 10 times. Results here are the average.

All of these are so pathetically low that I seriously doubt your test is
configured correctly. These are so low that you’re either running this
against a worst case setup or in development mode or something. In
fact, these numbers are so low it doesn’t matter how fast your setup is,
it can’t service any kind of reasonable requests rate.

The rendered page is a 500Kb page. This is the controller code:

def list
#@sorteo_pages, @sorteos = paginate :sorteos, :per_page => 10
@sorteos = Sorteo.find(:all)
end

The view code is the one generated by the scaffold, minus the pagination
stuff.

In your situation there’s probably so many confounding factors that
you’d need to work on the setup to find the simplest best case as a
comparison (like the control in an experiment). Until you do that you
really have no idea how fast each config could be.

I could do the same test our friend Alexey did: hit the index page.

For example, how is it possible that you have such a low performance and
yet with 5 mongrels or fcgi processes you don’t see a 5x increase?
That’s the expected response. Also, why 5? How come there’s no test
for lighttp+1 fcgi?

Yes… I should’ve tested lighttpd+1 fcgi.

Is this with everything running on the same machine
(even your ab run)?

again, quoting myself:

The command:
/usr/sbin/ab -c $CONN -n $TOTAL http://cerebro:3000/sorteos
Run from another machine in the LAN.

How many of these runs did you do? Just one? How
do you know that one wasn’t just bad/good luck? Why are you increasing
the concurrency but not the number of connections?

Again, each test was run 10 times. The results in the post are the
average. Also, you have to consider that the rendered page was about
500Kb. There were no images, no external stylesheets, etc.

Just some minor criticisms as I don’t think the test is right for any of
the tested servers (I know fcgi does better than 2.3 req/s, as well as
mongrel).

Did you read the whole post? I was trying to hit a worst case
scenario, not an average one. That’s why I removed the pagination, there
was no cache (except the default in the production environment), that’s
why I rendered a whole ~350 rows html-table to the browser.
I might run the same test, but this time with pagination (to reduce the
size of the result for the query and the rendered page)…

–
Zed A. Shaw

Regards,
rolando.-

5

On Wed, 2006-08-23 at 14:50 +0200, Rolando A. wrote:

As they’ve already told you, there’s a need to make the same tests with
some kind of DB query… Here are my results:

You also have to run each test multiple times and take the mean of them
in order to rule out accidentally getting a bad read. Not to mention a
bunch of other stuff.

> Webrick (only T1) > T1: - Req/s: 1.09

Lighttpd + 5 fcgi spawns:
T1: - Req/s: 2.41
T2: - Req/s: 2.31
T3: - Req/s: 2.32

Mongrel (only T1)
T1: - Req/s: 0.92

Pound + 5xMongrel Cluster
T1: - Req/s: 1.72
T2: - Req/s: 1.54

All of these are so pathetically low that I seriously doubt your test is
configured correctly. These are so low that you’re either running this
against a worst case setup or in development mode or something. In
fact, these numbers are so low it doesn’t matter how fast your setup is,
it can’t service any kind of reasonable requests rate.

In your situation there’s probably so many confounding factors that
you’d need to work on the setup to find the simplest best case as a
comparison (like the control in an experiment). Until you do that you
really have no idea how fast each config could be.

For example, how is it possible that you have such a low performance and
yet with 5 mongrels or fcgi processes you don’t see a 5x increase?
That’s the expected response. Also, why 5? How come there’s no test
for lighttp+1 fcgi? Is this with everything running on the same machine
(even your ab run)? How many of these runs did you do? Just one? How
do you know that one wasn’t just bad/good luck? Why are you increasing
the concurrency but not the number of connections?

Just some minor criticisms as I don’t think the test is right for any of
the tested servers (I know fcgi does better than 2.3 req/s, as well as
mongrel).

–
Zed A. Shaw

http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 – Come get help.

6

Zed S. wrote:

Just some minor criticisms as I don’t think the test is right for any of
the tested servers (I know fcgi does better than 2.3 req/s, as well as
mongrel).

oh!.. one more thing. Don’t check the req/s, check to see if the rate
is according your results.
The request I made is not a standard one, so that might be the reason of
the low req/s, but check for the rate.
(btw, the unit for the rate I put in my post is [KB/s], not [s]…)

–
Zed A. Shaw

regards,
Rolando.-

7

On Thu, 2006-08-24 at 15:21 +0200, Rolando A. wrote:

Zed S. wrote:
Did you read the whole post? I was trying to hit a worst case
scenario, not an average one. That’s why I removed the pagination, there
was no cache (except the default in the production environment), that’s
why I rendered a whole ~350 rows html-table to the browser.
I might run the same test, but this time with pagination (to reduce the
size of the result for the query and the rendered page)…

Yes I read it, I was just giving you some friendly advice on how to do
it again so that it actually showed something. Your worst case scenario
is basically making it so that your performance measurement has no
information. You’ve confounded the analysis with all the extra
conditions you put on it, when it’s not really necessary.

For example, how do I know your “worst case scenario” isn’t really:

def sleep_for_days
sleep 60
end

I don’t, so without a baseline to compare with on the same machine I
have nothing to go on (unless you post your code). What if you run your
baseline and come up with say 20 req/sec? Well, I’d assume your system
is setup poorly since I can get much smaller machines to do better than
that. Without a baseline, I have nothing to compare against.

Now I understand it was done quickly, but if you post measurements like
that without taking the time to do them right then it just gives
everyone a bunch of FUD to deal with. A better approach would be to
have done all your analysis for the baseline fastest case (as a control)
then your analysis.

Also, don’t just say X is faster than Y. Really clarify it, “With a
blah controller that has a blah condition with blah database and no
caching on foo machine, then X is faster than Y.”

Because, just like the nightly news, all people will post on their blogs
is:

“X is faster than Y”.

Which does us all a disservice in the end.

–
Zed A. Shaw

http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 – Come get help.

8

Zed S. wrote:

Now I understand it was done quickly, but if you post measurements like
that without taking the time to do them right then it just gives
everyone a bunch of FUD to deal with. A better approach would be to
have done all your analysis for the baseline fastest case (as a control)
then your analysis.

Ok, I give you that. I’m preparing a baseline test.

Also, don’t just say X is faster than Y. Really clarify it, “With a
blah controller that has a blah condition with blah database and no
caching on foo machine, then X is faster than Y.”

I never explicitly said “X is faster than Y”, I actually said “in my
scenario, the best option appears to be lighttpd + fcgi”. There’s an
“appears” and “in my scenario”.

Because, just like the nightly news, all people will post on their blogs
is:

“X is faster than Y”.

Which does us all a disservice in the end.

I totally agree with you.

–
Zed A. Shaw
http://www.zedshaw.com/
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 – Come get help.

regards,
rolando.-

9

Alexey Kovyrin wrote:

I noted that there is no information about how to deploy rails
application with nginx as frontend and what is performance of such
solution.

The day after you wrote this, Ezra Z. posted this on his blog:

“Nginx, my new favorite front end for mongrel cluster”
http://brainspl.at/articles/2006/08/23/nginx-my-new-favorite-front-end-for-mongrel-cluster

regards

Justin

10

“Nginx, my new favorite front end for mongrel cluster”

Ruby on Rails Blog / What is Ruby on Rails for?

Thanks for the reference Justin,
-andy