Scaling/optimizing a slow ruby on rails application

Hi everyone,

My name is Michael S. and I’m the co-founder & CTO of a
rails-based start-up. We have built a product for other businesses to
help their users adopt their products.

We’ve built the application, delivered to a few customers and now we’ve
hit scaling and performance issues.

I’d love your feedback on how we proceed as we try to resolve them:

  1. These are the problems we’ve encountered:
    The main user dashboard takes between 7-15 seconds to load. The maximum
    requests per second that we can support are 100-150.

  2. Why we think they’re happening:
    We’re using couchdb with an ORM that has our data modeled relationally.
    This causes the ORM to launch an extraordinary amount of http requests
    to couchdb to load all our data. The more data and users in the project,
    the longer the delay. There’s also potentially inefficient code, extra
    loops etc. We’re not using caching to it’s fullest extent.

  3. What we’re using/what we’ve done:
    Our architecture: rails ‘3.2.12’, ruby 1.9.3p374, nginx + unicorn (main
    server), elasticsearch (separate server), couchdb (separate server). All
    on AWS.
    We’ve added some view level caching where we can get away with stale
    data.
    We’ve tried using higher tiered/ssd aws servers but it doesn’t look to
    be our main bottleneck.

  4. How we plan on completing it:
    We’re mostly done a rewrite to migrate our data and data model to
    activerecord. We’ve hit a snag in migrating s3 attachment code built
    into the couchdb ORM that we’re using. Following the completion of the
    migration to sql/activerecord we are going to do the following in an
    attempt to increase performance:

A. Use percono mysql server.

B. Run tools such as bullet to analyze performance and where we might
have bad queries/lookups.

C. Update to latest Rails and Ruby versions.

D. Look into Puma and paralelization of DB access calls.

E. Break apart the application into separate services.

  1. What we don’t know right now:
    Whether switching to Activerecord or any of the above ideas will
    inherently give us any performance benefits.
    Whether we will have any other unexpected surprises or loss of
    functionality due to switching to ActiveRecord.
    How long this all will take and whether there’s other areas that we need
    to focus to get sub 1 second performance that we’re missing.

Do you all have any feedback/ideas/insights completing projects like
these?

In particular, I would be curious about:

a) How would you approach it?

b) Anything that’s worked for you in the past on suitable projects?

c) Anything to avoid doing?

d) Where to find good additional team members & advisors to help us
complete the project?
e) How long to expect it to take?

Here are our application rake stats:
http://pastie.org/private/qrxkytk4uur9odndydv5dq

On Tuesday, 22 July 2014 23:44:12 UTC-4, Ruby-Forum.com User wrote:

I’d love your feedback on how we proceed as we try to resolve them:

http://stackoverflow.com/questions/24901078/scaling-optimizing-a-slow-ruby-on-rails-application

http://www.google.com/url?q=http%3A%2F%2Fstackoverflow.com%2Fquestions%2F24901078%2Fscaling-optimizing-a-slow-ruby-on-rails-application&sa=D&sntz=1&usg=AFQjCNELHm0mSM80NfWTSFPZ6aVzPYMfcA

Some thoughts:

  • as mentioned on SO, New Relic is a good place to start. Figure out
    where
    the time is going in the slow requests, and work from there.

  • the rewrite to an RDBMS may help, may not. It’s highly dependent on
    why
    CouchDB was chosen in the first place - if your application depends
    heavily
    on some of the features of CouchDB (MapReduce, or async replication) it
    may
    make things worse.

  • if minor parts are hard to port from Couch, skip them. Get something
    working with the new design, and benchmark it.

  • Above everything else, MEASURE. Most of the things you listed on
    StackOverflow could improve performance, but could decrease it.

–Matt J.