Feedback wanted on current setup, and a question about cachi

Rails Deployment
1

Here’s where I explain my current setup, and ask for feedback. I’m
looking for a solution that’s easy to set up and Just Works. This is
for http://www.tanga.com. Due to the nature of the site, traffic is
awfully spikey, meaning that we can get 100x normal traffic in a matter
of seconds. I’m being hosted at Textdrive.

An accelerator is sort of like a VPS, I suppose. Running on
OpenSolaris.

In a few days, I should have:

* Four 1/16 accelerators. Each of these will be running 10

Mongrels. A Big IP will be doing load balancing feeding the requests
directly to the Mongrels. SSL processing is done by the Big IP (I
hope).

* One 1/8 accelerator doing database (postgresql). It also sends

out a ton of email and sms notifications.

* All the accelerators share the same /home mount point that's on a

x4500. All the Mongrels on the different Accelerators run off the same
Rails code (i.e. the code is in /home/deploy/apps/tanga). Each Mongrel
writes to the same production.log file that’s in /home.

QUESTION: How should page/fragment caching work? I guess we need a
webserver in there somewhere that determines whether the static cache
file should be sent or if the request needs to be sent to the Mongrels.
I’m not opposed to using something like BingoDisk, but that doesn’t
solve the issue of how the cache issue should work. Right now, Apache
first checks for the existence of the requested file, and if it doesn’t
find it, it sends the request to Mongrel.

2

Memcached fragments?

http://blog.zvents.com/2006/11/3/rails-plugin-extended-fragment-cache

Eliot

3

The issue is how the cached files get to the user. I don’t think
memcache helps with that.

4

On Jan 24, 2007, at 2:28 PM, Joe Van D. wrote:

* All the accelerators share the same /home mount point that's

solve the issue of how the cache issue should work. Right now, Apache
first checks for the existence of the requested file, and if it
doesn’t
find it, it sends the request to Mongrel.

You will have to step up to something like memcached for your
fragment store, page caching is harder to do when you go straight to
the mongrels from the hardware load balancers, I never came up with a
solution to page caching that I liked with a setup like this.

What I do when I have nodes like that behind a hardware l/b is run
one nginx server on each node. Nginx serves static content and page
caches and proxies to mongrel for the rest. This neatly sidesteps the
hassle of an asset_host. Especially since you have the shared
filesystem to put cached pages and fragments on you may as well take
advantage of it.

Also one flaw I see in your setup is that you have all the nodes
logging their production log to the same file on a shared filesystem.
This is going to degrade performance a lot when you have 4 nodes
trying to get a lock so they can write to the logs. And rails does
this all day long. It’s one thing to have a cluster of mongrels on
the same host write to the same logs, thats fine,. It’s another thing
to have 4 server nodes that share the same filesystem that are each
running clusters of mongrels.

We use GFS as our shared filesystem in a totally similar way to what
you are doign but GFS has something called context dependent
symlinks. These allow rails to blissfully write to the production log
at the same path on the shared filesystem, but really the log
directory is just a symlink to the node’s hostname.

Like this:

ey00-s00076 shared # ls -lsa
total 36
4 drwxrwxr-x 6 ez ez 3864 Dec 14 12:08 .
4 drwxr-xr-x 4 ez ez 3864 Jan 23 17:08 …
4 lrwxrwxrwx 1 ez ez 9 Dec 2 11:30 log -> @hostname
4 drwxr-xr-x 2 ez ez 3864 Jan 24 01:26 ey00-s00048
4 drwxr-xr-x 2 ez ez 3864 Jan 24 01:27 ey00-s00069

see the log -> @hostname ? that makes it so each node in the cluster
can still write logs to app/shared/log, but the file will be actually
in a dir named after the current hostname.

Separating the logs out increased the performance of rails apps in a
cluster like this at least 20%

Cheers-
– Ezra Z.
– Lead Rails Evangelist
– [email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)

5

On Jan 24, 4:03 pm, Ezra Z. [email protected] wrote:

* All the accelerators share the same /home mount point that's

solve the issue of how the cache issue should work. Right now, Apache
first checks for the existence of the requested file, and if it
doesn’t
find it, it sends the request to Mongrel.You will have to step up to something like memcached for your
fragment store, page caching is harder to do when you go straight to
the mongrels from the hardware load balancers, I never came up with a
solution to page caching that I liked with a setup like this.

Oh, sorry. The cached files will be stored on a NFS drive (for now).
I’ll look at using memcache later. My question was more about how to
get the static/cached files to the user, which it looks like nginx
might do.

trying to get a lock so they can write to the logs. And rails does
this all day long. It’s one thing to have a cluster of mongrels on
the same host write to the same logs, thats fine,. It’s another thing
to have 4 server nodes that share the same filesystem that are each
running clusters of mongrels.

Hm, good point.

4 drwxrwxr-x 6 ez ez 3864 Dec 14 12:08 .
cluster like this at least 20%
So, if each machine’s log is written to a different file, doesn’t it
make it difficult to look through each production.log file when you’re
trying to track down problems? I wonder if there’s a good way to
combine all the log files when you’re debugging.

6

Hi~

On Jan 24, 2007, at 5:16 PM, Joe Van D. wrote:

An accelerator is sort of like a VPS, I suppose. Running on
out a ton of email and sms notifications.
webserver in there somewhere that determines whether the static
fragment store, page caching is harder to do when you go straight to
one nginx server on each node. Nginx serves static content and page
this all day long. It’s one thing to have a cluster of mongrels on
at the same path on the shared filesystem, but really the log
4 drwxr-xr-x 2 ez ez 3864 Jan 24 01:27 ey00-s00069
So, if each machine’s log is written to a different file, doesn’t it
make it difficult to look through each production.log file when you’re
trying to track down problems? I wonder if there’s a good way to
combine all the log files when you’re debugging.

Since all the real log directories live on the shared filesystem you
can tail all the production logs at once like this:

$ tail -f /data/appname/shared/ey00-s000*/production.log

That will tail all the log files at once with labels for which file
entries come from.

Cheers-
– Ezra Z.
– Lead Rails Evangelist
– [email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)