Sporadic clustering + caching problem

So, I’ve seeing some quirking behavior with my app after launching it
into my production environment (railsmachine). The weirdness only
(sporadically) occurs when I’m running a mongrel cluster. The app runs
fine (locally and in production) when I run on a single mongrel. I’m
hoping somebody on the forum can help me out, as I’ve run out of things
to try given my limited rails experience :slight_smile:


My railsmachine environment consists of Apache load-balancing 2 mongrel
instances connecting to a single (non-clustered) MySQL instance. All
these processes run in a single VPS. I set up the VPS per the
RailsMachine “operating manual”:
https://support.railsmachine.com/index.php?pg=kb.book&id=4 . Rails
version is 1.2.3 (I can list my gems if requested, but I don’t think
they’re applicable to the problem)

The Problem

The problem is sporadic and difficult to reproduce, but when it does
occur, it usually looks like this:

  • Go to the “edit” page and update an existing record (e.g. set one of
    the boolean flags on the record from true to false or vice-versa). The
    changes are correctly persisted to the DB via ActiveRecord (I verified
    this with mysql client).
  • Load the “view” page that displays all records: the OLD version of the
    record is displayed
  • Refresh (via the browser’s refresh button) the “view” page: The
    changes are now reflected in the view (i.e. the view is in sync with
    the DB state)
  • Refresh the “view” page again: The view toggles back to the OLD
    version of the record.
  • and so on…

Given this behavior, I’m guessing the problem is due to one of the
mongrels caching a stale version of the page (or model?). Perhaps the
mongrel that performed DB update flushed it’s cache while the other
mongrel kept the stale version of the page/model? In this case, the
fact that the view page toggles between stale and fresh on browser would
suggest that the Apache load balancing is performed round-robin.
However, this is just a hypothesis…


My application is fairly simple

  • 4 database tables
  • no session state
  • no page or action caching
  • simple (hand-coded) caching of static database tables

But, I wanted to boil the problem to something really simple for
debugging purposes. SO, I wrote the following controller:

START games_controller.rb

class GamesController < ApplicationController
session :off

View Games Status Page

def view
user_id = params[:user_id]

# This is an ActiveRecord "dynamic finder"
@participant = Participant.find_by_user_id(user_id)

# a Participant has_many Games. I'm using (true) here to force
# a collection refresh (just to be safe)
@games = @participant.games(true)


END games_controller.rb

I can reproduce the bug even with this simple controller, leading me to
believe the problem must lie in my environment and not my code. I
verified (via the logs) the controller code is executing on each request
(i.e. rails isn’t caching my action). After some futile
logging/debugging efforts, I then tried turning the following knobs
(testing between each step, of course):

  • Turn off rails action caching
  • Turn off the MySQL query_cache
  • Restart MySQL, Apache. Redeploy mongrels (via capistrano)

However, none of this seemed to help. Here is the production environment
configuration file I ended up with (excluding app-specific ENV entries)

START production.rb

config.cache_classes = true
config.action_controller.consider_all_requests_local = false
config.action_controller.perform_caching = false

END production.rb

So, I’m at a loss at this point. I’m a rails noob (Java’s my main
thing), so I can only assume that I’m doing something stupid on my VPS.
I contacted railsmachine support, and, unfortunately, they were of
little help (“Did you check the logs?”). Any help you could offer would
be greatly appreciated. Solutions, suggestions, anything :slight_smile:

Thanks in advance!