Scaling/optimizing a slow ruby on rails application

Rails
1

Hi everyone,

My name is Michael S. and I’m the co-founder & CTO of a
rails-based
start-up. We have built a product for other businesses to help their
users
adopt their products.

We’ve built the application, delivered to a few customers and now we’ve
hit scaling and performance issues.

I’d love your feedback on how we proceed as we try to resolve them:

  1. These are the problems we’ve encountered:

The main user dashboard takes between 7-15 seconds to load. The maximum
requests per second that we can support are 100-150.

  1. Why we think they’re happening:

We’re using couchdb with an ORM that has our data modeled relationally.
This causes the ORM to launch an extraordinary amount of http requests
to
couchdb to load all our data. The more data and users in the project,
the
longer the delay. There’s also potentially inefficient code, extra loops
etc. We’re not using caching to it’s fullest extent.

  1. What we’re using/what we’ve done:

Our architecture: rails ‘3.2.12’, ruby 1.9.3p374, nginx + unicorn (main
server), elasticsearch (separate server), couchdb (separate server). All
on
AWS.

We’ve added some view level caching where we can get away with stale
data.

We’ve tried using higher tiered/ssd aws servers but it doesn’t look to
be
our main bottleneck.

  1. How we plan on completing it:

We’re mostly done a rewrite to migrate our data and data model to
activerecord. We’ve hit a snag in migrating s3 attachment code built
into
the couchdb ORM that we’re using. Following the completion of the
migration
to sql/activerecord we are going to do the following in an attempt to
increase performance:

A. Use percono mysql server.

B. Run tools such as bullet to analyze performance and where we might
have
bad queries/lookups.

C. Update to latest Rails and Ruby versions.

D. Look into Puma and paralelization of DB access calls.

E. Break apart the application into separate services.

  1. What we don’t know right now:

Whether switching to Activerecord or any of the above ideas will
inherently
give us any performance benefits.

Whether we will have any other unexpected surprises or loss of
functionality due to switching to ActiveRecord.

How long this all will take and whether there’s other areas that we need
to
focus to get sub 1 second performance that we’re missing.

Do you all have any feedback/ideas/insights completing projects like
these?

In particular, I would be curious about:

a) How would you approach it?

b) Anything that’s worked for you in the past on suitable projects?

c) Anything to avoid doing?

d) Where to find good additional team members & advisors to help us
complete the project?

e) How long to expect it to take?

Here are our application rake stats:
http://pastie.org/private/qrxkytk4uur9odndydv5dq

2

On Wed, Jul 23, 2014 at 10:59 AM, Michael S. [email protected]
wrote:

We’ve built the application, delivered to a few customers and now we’ve
The main user dashboard takes between 7-15 seconds to load. The maximum
etc. We’re not using caching to it’s fullest extent.

  1. What we’re using/what we’ve done:

Our architecture: rails ‘3.2.12’, ruby 1.9.3p374, nginx + unicorn (main
server), elasticsearch (separate server), couchdb (separate server). All on
AWS.

one easy way you can improve is to move to a more recent version of
ruby.
there’s a big improvement between
2.0 and 1.9.3 (and it’s already 2.1.2) so you might want to look into
that
first since it’s easier than improving the
code performance.

activerecord. We’ve hit a snag in migrating s3 attachment code built into
B. Run tools such as bullet to analyze performance and where we might

Whether we will have any other unexpected surprises or loss of
In particular, I would be curious about:

https://groups.google.com/d/msgid/rubyonrails-talk/dab7f05f-a3bd-4716-8f9d-7b8007f5582d%40googlegroups.com

https://groups.google.com/d/msgid/rubyonrails-talk/dab7f05f-a3bd-4716-8f9d-7b8007f5582d%40googlegroups.com?utm_medium=email&utm_source=footer

.
For more options, visit https://groups.google.com/d/optout.

–

3

On Wednesday, July 23, 2014 3:59:56 AM UTC+1, Michael S. wrote:

We’ve
built the application, delivered to a few customers and now we’ve hit
scaling and performance issues.

First off storing relational data in couchdb doesn’t sound right - it’s
not really what it is designed for. I used couchdb on a project once
many moons ago and things got a lot easier once we stopped trying to use
it as a relational datastore.

It seems to me that your first step is to establish a performance
baseline: write some automata le performance tests (I don’t mean tests
in the sense of rspec - just something that exercises your website in a
reasonably realistic way). Use tools such as stack_profiler to find out
where the time is really going. Tools such as new relic or
rack-mini_profiler can also provide insight into what real users are
seeing.

What we’re using/what we’ve done:

Our architecture:
rails ‘3.2.12’, ruby 1.9.3p374, nginx + unicorn (main server),
elasticsearch (separate server), couchdb (separate server). All on
AWS.

In my experience elasticsearch is blazing fast, scales well horizontally
and doesn’t constrict you in quite the way couchdb does (or at least did

  • couchdb has almost certainly changed since I last used it). You can
    also use it as a straight up document oriented datastore - I remember
    reading about some of the early moving elasticsearch folk replacing
    couchdb with elasticsearch

We’ve added some
view level caching where we can get away with stale data.

We’ve tried using
higher tiered/ssd aws servers but it doesn’t look to be our main
bottleneck.

increase performance:

A. Use
percono mysql server.

It may be a bit late but I would seriously consider postgres. Its richer
set of data types (arrays, json etc) might ease the transition and it
has a pretty awesome feature set.

B.
Run tools such as bullet to analyze performance and where we might
have bad queries/lookups.

C.
Update to latest Rails and Ruby versions.

Never hurts - also consider GC tuning parameters.

e)
How long to expect it to take?

I think this is largely unanswerable by someone unfamiliar with your
app. If you’re forcing relational style data into a document oriented
datastore like couchdb then storing it in a relational db should make it
easier/faster. On the other hand if you do play to couchdb’s strengths
in other ways then that might make the process more difficult (eg if
some of your documents store arbitrary key/pairs of data then that can
be a pain in the relational world)

Where possible, break the problem down so that you can prove some of
these points early (breaking up into services could be a strategy here,
if it makes it easier for you to move part of your data across without
having to rewrite everything). Writing a prototype that handles some of
your hot paths could also provide some key early validation.

I wouldn’t obsess too much about tools like bullet - thinking about your
data access patterns before this conversion will probably make the
biggest difference (although nothing wrong with using it to mop things
up afterwards if you find yourself having to include 27 tables then you
should probably rethink your design)

Fred

4

On Tuesday, July 22, 2014 9:59:56 PM UTC-5, Michael S. wrote:

longer the delay. There’s also potentially inefficient code, extra loops
We’ve added some view level caching where we can get away with stale data.
the couchdb ORM that we’re using. Following the completion of the migration
have bad queries/lookups.

functionality due to switching to ActiveRecord.

Here are our application rake stats:
http://pastie.org/private/qrxkytk4uur9odndydv5dq

Without know the internal workings of your application, it is impossible
to
give specific answers…but if you are experiencing 15 second delays
with
only a few users, I would take a hard look at the data storage layer.
As
F. Cheung mentioned, I don’t think you have your data correctly
architected
with CouchDB. Consider Postgres as your DB, it is robust and has some
great features. With only a few users, I doubt you will see much speed
improvement upgrading your Ruby or Rails versions. Your immediate
concern
should be the data architecture.