Help! Why is my ActionCaching that slow?

Rails
1

I’ve just did some performance-testing with httperf on a virtual server
(with a max. of 10 customers per machine) like:

httperf --server www.myHOST.com --port 80 --uri /mycontroller/68
–num-conns 400 --rate 45

When any caching is switched-off I get up to 26 req/s. With action
caching enabled mongrel tells my something about 240 up to 540 req/s. In
fact httperf attests a performance of just around 30 req/s!

Since it’s always the same req. I could imagine that the database does
quite a good job with it’s caching, so that the actual number might be
lower (around 10 or so). But why is filebased Action-Caching so slow?
Lighttd can handle > 300 req./s !

Thanks
David

2

Any ideas at all?

3

What does a baseline perf test against an action that doesn’t hit the
database give you? I’m guessing it’ll be around 30 req/s.

Mongrel is fast, but definitely not as as fast as lighttpd when it
comes to serving static content. If you’re running mongrels behind
lighttpd, set it up to serve up the cached pages on behalf of
mongrel. This’ll speed up your perf and let mongrel handle the
generation of other pages on your site.

This can be done multiple ways depending on what version of lighttpd
you’re using…

On Mar 8, 10:12 pm, David S. [email protected]

4

Frist, thanks for your reply!

Eden Li wrote:

What does a baseline perf test against an action that doesn’t hit the
database give you? I’m guessing it’ll be around 30 req/s.

Hardly. It’s about 18 req/s.

I’ve done a simple test with IRB on my local AMD Athlon 2600+ and on the
virtual server:

10000000.times do
i = 1
end

…took about 4.6s on my local machine and about 11s on my virtual
server, running on a Intel P4 2.80 GHz. Which means that the v-server is
offering me just about 40% of the CPU power of a four year old desktop
PC!

Would you say that Mongrel / Rails is fast enough for a havy loaded
business web application? Let’s say up to 1.500 req/s. Of course I can
cluster a bit and use maybe four servers each with a dual core Opteron.
Would you say it could do that?

Mongrel is fast, but definitely not as as fast as lighttpd when it
comes to serving static content. If you’re running mongrels behind
lighttpd, set it up to serve up the cached pages on behalf of
mongrel. This’ll speed up your perf and let mongrel handle the
generation of other pages on your site.

Well, I need to authenticate users. That’s why I thought ActionCaching
would be the ideal solution.

This can be done multiple ways depending on what version of lighttpd
you’re using…

Is there another trick with lighttd?

5

Well, I need to authenticate users. That’s why I thought ActionCaching
would be the ideal solution.

Yeah, well now you’re stuck. Action-level caching will allow you to
authenticate, but mongrel will definitely have to touch the request.

I guess there is no way around? Mongrel has to serve the hole request?!
Or is there an option that lighttd could takeover after the
authentication???!

From my own testing on my servers, I’ve noticed that my setup was able
to serve up a whole lot more than httperf was telling me I could serve
up.

I did so. However adding more mongrels does not improve my performance
that much. Maybe it’s the V-Server. What type of server you’re running
on? Virtual or dedicated server?! And what are your performance figures?

6

Would you say that Mongrel / Rails is fast enough for a havy loaded
business web application? Let’s say up to 1.500 req/s. Of course I can
cluster a bit and use maybe four servers each with a dual core Opteron.
Would you say it could do that?

Is that 1500 pages per second? Or requests? I just checked three
sites
(pet project, work site, and rubyonrails) and just loading the homepage
resulted in 13, 57, and 21 requests (images, javascript, stylesheets).
I
know our work is high, so let’s say 20 is average… that means you want
to handle 75 pages/sec which should be very doable on four servers…

-philip

7

David S. wrote:

Would you say that Mongrel / Rails is fast enough for a havy loaded
business web application? Let’s say up to 1.500 req/s. Of course I can
cluster a bit and use maybe four servers each with a dual core Opteron.
Would you say it could do that?

If you clustered, you would be able to handle lots and lots of
requests. There are a number of high-traffic sites out there running
mongrel+Rails, so this definitely is scalable. With a caveat – your
Rails code is optimized and not relying on stuff like RMagick, etc…

Well, I need to authenticate users. That’s why I thought ActionCaching
would be the ideal solution.

Yeah, well now you’re stuck. Action-level caching will allow you to
authenticate, but mongrel will definitely have to touch the request.

From my own testing on my servers, I’ve noticed that my setup was able
to serve up a whole lot more than httperf was telling me I could serve
up. It’s probable my testing methods were out of whack, so you might
want to check your httperf args are up to snuff. Here’s a really good
tutorial on using httperf with mongrel. Unfortunately, I haven’t gone
through it yet.

Oh, and here’s a for-pay screencast about httperf:
http://peepcode.com/products/benchmarking-with-httperf. It looks like
it’ll be useful, but YMMV.

8

Is that 1500 pages per second? Or requests?

Yes, that’s 1500 requests / second. That what a havy load application
should be able to serve.

I just checked three

sites
(pet project, work site, and rubyonrails) and just loading the homepage
resulted in 13, 57, and 21 requests (images, javascript, stylesheets).

A serious public site should have a limit per C Class segment or
IP-Address via iptables. So the above sites should be able to serve far
more than that.

BTW are there any “out-of-box” tools that could do the authenication
instead of rails? I mean so that the pages could be served at the speed
of the webserver instead of bugging rails. Like lighttpd’s Lua for
instance? Or a native authenication service??!

9

I just checked three

sites
(pet project, work site, and rubyonrails) and just loading the homepage
resulted in 13, 57, and 21 requests (images, javascript, stylesheets).

A serious public site should have a limit per C Class segment or
IP-Address via iptables. So the above sites should be able to serve far
more than that.

Sorry, what I meant to say was that a single loading of their homepage
resulting in 13, 57, and 21 total requests, not that the site could only
support that many requests/sec.

-philip

10

Philip H. wrote:

Sorry, what I meant to say was that a single loading of their homepage
resulting in 13, 57, and 21 total requests, not that the site could only
support that many requests/sec.

-philip

Good point! Maybe I missunderstood the “experts” I’ve talked with. They
were talking about 1,500 req/s for a forum / chatting plattform. I was
told that they are also using a lot of AJAX and that the bottleneck is a
mySQL database. That’s why they are now using a cluster. Probably that
figure included requests to static content like stylesheets, images, and
stuff as well.

But I thought most of that is cached by the browser anyway? If the
clients are frequent users of that portal, wouldn’t the browser avoid
loading that stuff and request only the dynamic content?!

However I’m trying to avoid having havy database load with a intelligent
caching strategy. I hope that mongrel will support me on this way.

At work I’m maintaining the source-code of a JSP-based web-application
used by about 80.000 active customers (meaning customers that log-in at
least once a month or so). I’ve had a look into Tomcat’s log-files and
found 40 dynamic requests logged for one second somewhere at the
afternoon on one of the two machines.

So maybe 100…200 requests/sec. would already be ok for a, let’s say
medium sized, web-application. If I’m lucky and the site get’s even more
traffic, I guess I should still be able to serve 1,000 req. with more
server-hardware, more mongrels, and a lot of caching so that the
database won’t get overloaded?!

Does ANYONE knows some more performance figures of rails based
web-sites? I would be interested in how many requests they can serve and
what type of hardware and software-setup they are using. Too bad the
author of “Agile web development with rails” stopped talking about these
figures after the first book.

11

On Mar 9, 2007, at 4:47 PM, Philip H. wrote:

We did this with 20 servers running apache and 4 mongrels each, and
three
separate media servers (for video). None of the servers were
overworked
(load < 1) so I imagine we could have gotten by with a lot fewer,
but the
year before we were PHP and were slammed and our traffic triples about
this time every year so we didn’t want to take any chances :slight_smile:

Hey, I just wanted to dispel a potential myth here.

Many people feel that a server with load over 1.0 is overworked, but
that’s
not the way the master Unix folks saw it.

A process that’s blocking on anything, including I/O, network, etc.
(think
MySQL query) is “waiting to run” and therefore counts towards the run
queue.

The best tuners I’ve known feel that a load of 4.0 is saturated, and
target
2-3 for day-to-day operations, sometimes much higher if there are
external
dependencies of any sort.

Load numbers in and of themselves don’t mean much. Why is the load high?
What is causing the load to increase? If it’s not a local resource
constraint, i.e. CPU or Disk I/O, then it’s very easy to entirely
misunderstand load as it related to performance.

My point is not to suggest you don’t know this, but to make sure nobody
takes you statement above and misconstrues its meaning.

–
– Tom M., CTO
– Engine Y., Ruby on Rails Hosting
– Reliability, Ease of Use, Scalability
– (866) 518-YARD (9273)

12

mySQL database. That’s why they are now using a cluster. Probably that
figure included requests to static content like stylesheets, images, and
stuff as well.

That’s a busy forum, but depending on how they use AJAX I could see it
reaching that.

But I thought most of that is cached by the browser anyway? If the
clients are frequent users of that portal, wouldn’t the browser avoid
loading that stuff and request only the dynamic content?!

Guess it depends… on our site most of the pages include their own CSS
and specific images as the content is pretty different from section to
section, but yeah, after awhile browsers should cash that.

But my point was that lighttpd/nginx/apache/etc should be able to serve
thousands of requests per second for static content. So that really
shouldn’t factor into things unless you’re youtube or flickr :slight_smile:

However I’m trying to avoid having havy database load with a intelligent
caching strategy. I hope that mongrel will support me on this way.

I’m not sure how mongrel would help specifically with this. Rails page
caching and fragment caching would though. Look into memcache as well
and
the various plugins that tie memcache into AR’s find methods and Rails
caching in general. Memcache is a life saver for us.

database won’t get overloaded?!

Does ANYONE knows some more performance figures of rails based
web-sites? I would be interested in how many requests they can serve and
what type of hardware and software-setup they are using. Too bad the
author of “Agile web development with rails” stopped talking about these
figures after the first book.

Sometime last year our corp site did 8,996,175 pages and 63,571,374
requests in one day. That works out to um… about 100 pages/sec and
735
requests/sec.

And while I don’t trust alexia exactly, for comparision with some other
rails sites…

http://img312.imageshack.us/img312/3569/alexavd9.png

We did this with 20 servers running apache and 4 mongrels each, and
three
separate media servers (for video). None of the servers were overworked
(load < 1) so I imagine we could have gotten by with a lot fewer, but
the
year before we were PHP and were slammed and our traffic triples about
this time every year so we didn’t want to take any chances :slight_smile:

-philip

13

Philip H. wrote:

But my point was that lighttpd/nginx/apache/etc should be able to serve
thousands of requests per second for static content. So that really
shouldn’t factor into things unless you’re youtube or flickr :slight_smile:

True.

I’m not sure how mongrel would help specifically with this. Rails page
caching and fragment caching would though.

I’ve forgot that it is rails who does the caching. What would you say?
Is mongrel still the option #1 for havy loaded apps? Or would you say
fcgi is doing a better job here?!

Look into memcache as well

I will. I’ve already tested with :memory_cache, which did not helped to
increase the req/s figure that much. It was a raise of about 10%.

In speaking of memcache. (How) could content be invalided? On the
file-system it’s easy to write a .sh-script which starts deleting after
the 100,000st file, ordered by last access. It that possible with
memcache as well?

What raise of performance did you noticed with memcache in your
environment?

the various plugins that tie memcache into AR’s find methods and Rails
caching in general. Memcache is a life saver for us.

Various plugins? Is there more to recommend than the well known
MemCacheStore Ruby interface?

Sometime last year our corp site did 8,996,175 pages and 63,571,374
requests in one day. That works out to um… about 100 pages/sec and
735
requests/sec.

All pages served by rails? Or most served by apache?!

We did this with 20 servers running apache and 4 mongrels each, and

Just 4 mongrels per server? Was that the optimum configuration for you?

14

I’m not sure how mongrel would help specifically with this. Rails page
caching and fragment caching would though.

I’ve forgot that it is rails who does the caching. What would you say?
Is mongrel still the option #1 for havy loaded apps? Or would you say
fcgi is doing a better job here?!

My personal journey has been fcgi -> mongrel -> litespeed. My
understanding is that mongrel is preferred over fcgi and that
nginx/mongrel seem to be the current favorite, but search the archives
as
there are others who have done some research on it :slight_smile:

In speaking of memcache. (How) could content be invalided? On the
file-system it’s easy to write a .sh-script which starts deleting after
the 100,000st file, ordered by last access. It that possible with
memcache as well?

Memcache has built in expiration. When you store something into
memcache
you can tell it when to expire (in seconds).

What raise of performance did you noticed with memcache in your
environment?

We’d be dead without it :slight_smile: We use it to generate a lot of aggregate
queries as well as fragment cache a lot of content.

the various plugins that tie memcache into AR’s find methods and Rails
caching in general. Memcache is a life saver for us.

Various plugins? Is there more to recommend than the well known
MemCacheStore Ruby interface?

Hrm. I don’t know their names off hand, but there are plugins to use
memcache for sessions, fragment caching, and hook them into
Model.find(…)

Sometime last year our corp site did 8,996,175 pages and 63,571,374
requests in one day. That works out to um… about 100 pages/sec and
735
requests/sec.

All pages served by rails? Or most served by apache?!

With the exception of the homepage which was cached for one minute then
regenerated all the pages were served by rails.

We did this with 20 servers running apache and 4 mongrels each, and

Just 4 mongrels per server? Was that the optimum configuration for you?

Yep. Moving to 5 hurt performance. See the mongrel site for details.

-philip

15

Philip H. wrote:

My personal journey has been fcgi -> mongrel -> litespeed.

How much faster is Ruby LSAPI than mongrel incl. the required load
balancer?

Memcache has built in expiration. When you store something into
memcache
you can tell it when to expire (in seconds).

That won’t do it for me. I intend to keep as many pages as possible in
the cache. Therefore I’d like to expire only those cache-items which
have been rarely requested (ordered by last access-time => ls -h).

16

Tom M. wrote:

The best tuners I’ve known feel that a load of 4.0 is saturated, and
target
2-3 for day-to-day operations, sometimes much higher if there are
external
dependencies of any sort.

Well, I would say havy load is when the web-server or particularly
mongrel starts returning 5xx. However that’s before the load reaches a
value of 4. And it is noticable that most of the load is “system”. Since
the request gets cached, what could be the bottleneck? The sockets on
the virtual server?!

Hope the performance gets much better when moving to a dedicated server.

17

On Mar 9, 2007, at 4:01 PM, Philip H. wrote:

relevant.
wipe
the cache only if the data that uses it has been updated. Cause if it
hasn’t, you might as well let it sit there forever :slight_smile:

-philip

Memcached afaik will push older items out of the cache as you put
new ones into the cache. So you can just keep stuffing things into
memcached and the older unused cached items will get pushed out of
memcached when new items come in and it has used all its configured
memory.

Cheers-
– Ezra Z.
– Lead Rails Evangelist
– [email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)

18

Philip H. wrote:

My personal journey has been fcgi -> mongrel -> litespeed.

How much faster is Ruby LSAPI than mongrel incl. the required load
balancer?

That I can’t tell you as I haven’t done the homework myself. All of our
stuff is behind a hardware load balancer though so that part isn’t
relevant.

Memcache has built in expiration. When you store something into
memcache you can tell it when to expire (in seconds).

That won’t do it for me. I intend to keep as many pages as possible in
the cache. Therefore I’d like to expire only those cache-items which
have been rarely requested (ordered by last access-time => ls -h).

Then you want cache sweeping and observers and set something up to wipe
the cache only if the data that uses it has been updated. Cause if it
hasn’t, you might as well let it sit there forever :slight_smile:

-philip

19

What are you thinking about the approach (I don’t know if it will work)
to do the authenication by lighttd’s Caching Meta Language (cml) build
on Lua and to return the cache as page cache directly from lighttpd
(instead of mongrel)?

20

So you can just keep stuffing things into

memcached and the older UNUSED cached items will get pushed out of
memcached when new items come in and it has used all its configured
memory.

=> That’s exactly what I need. And it is actually the thing you’d expect
from a well-designed caching system:)