Spec/Test Speed

Testing RSpec
1

Just wanted to pick some smart people about this topic: What are you
guys doing to increase the speed of your specs?

I’m a big fan of autotest, but right now my current project has 438
specs (for rails). Most of them are in the model, and for all of
them we are hitting the database (they are more functional specs than
unit-tests). The whole suite takes 112 seconds (meaning the red to
green cycle leaves me learning how to juggle, just like Jim W.
with make). Any recommendations, short of mocking/stubbing to get
around this? Some of the suggestions I’ve heard are:

  • In Memory Database w/ Sqlite. As far as I know, no one has gotten
    this to work (with rspec, at least)
  • Using faster hardware. I don’t have the budget right now for a
    better machine. One interesting idea is this one: http://
    DeepTest Preview - Dan Manges's Blog, although as far as I know,
    it doesn’t work for rspec.
  • Only running specs per file, and using continous integration to do
    the rest. Right now this isn’t an option for testing my User model,
    which has 92 examples, and takes 20 seconds. Also, I’d really rather
    not commit before I know the whole test suite passes
  • Other ideas?

Scott

2

Scott T. wrote:

  • Other ideas?

Scott

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

  1. Use the Ruby profiler to see where it’s spending its time.

  2. Recompile the Ruby interpreter with full optimization if you haven’t
    already. Assuming you’re on a platform supported by gcc, download the
    Ruby source and type

    export CFLAGS=‘-O3 -march=’

before the “./configure”. is your chip architecture, for
example “pentium3”, “pentium4”, “athlon-xp”, …

If your test suite and application are fully open source and you don’t
mind someone playing with it, I’m collecting benchmark suites for the
Ruby interpreter and would love to have a nice heavy “rspec” run as an
example of real-world Ruby usage. Contact me off-list if you’re
interested, or if you want me to show you how to profile the Ruby
interpreter itself on your test case.

3

http://rubyforge.org/mailman/listinfo/rspec-users
Unfortunately, the app isn’t open source, although I do have one
other app, and a rails plugins which both have specs:

http://rubyforge.org/projects/replacefixtures/
http://rubyforge.org/projects/datedbackup/

Those don’t hit a database, so they are incredibly fast:

Finished in 0.19639 seconds

257 examples, 2 failures, 4 pending

I’m going to give the profiler a shot, plus compiling ruby from
source (I’m using the fink package manager, after having ditched
darwin ports). I’ll see where that leaves me.

As for running the profiler - is there an easy way to wrap the blocks/
closures given to an ‘it’ example with the profiler? (I haven’t
contacted you off list since I’m sure others would benefit from this
information).

Other, more general ideas are still welcome…

Scott

4

Okay - Now I think we are getting off topic, but…

Ah … you’re on a Mac … Intel or PPC? If Intel, you might want to
look into their VTune tool (I think it runs on MacOS). Even if you
don’t
have the latest whiz-bang multi-core Intel chip, VTune will tell you
stuff you probably never knew you should care about.

I’ll check that out.

(I’m an Intel-free zone, and so I use AMD’s CodeAnalyst.) :slight_smile:

Database, eh? Maybe you should be recompiling the database from source
too. PostgreSQL? MySQL? There are lots of tweaking gizmos for them,
plus a benchmark called “MySQL Super Smack”.

I’ve heard of it. Any to recommend?

As an aside: how correlated do you think performance of specs is to
real performance of a production rails app? I’m asking this as
knowing almost nothing about mysql (I learned about the query_cache
just the other day) - as a developer who just wants my app to run (it
crashes at least once a day).

Scott

5

In-memory with sqlite worked fine with rspec-0.8 (it’s been a while
since I did this!). Google for instructions or drop me a line. Don’t
forget autotest/zentest to run only those tests which have ben affected
by changes.

Rgds,
Jerry

6

Scott T. wrote:

257 examples, 2 failures, 4 pending
Other, more general ideas are still welcome…

Ah … you’re on a Mac … Intel or PPC? If Intel, you might want to
look into their VTune tool (I think it runs on MacOS). Even if you don’t
have the latest whiz-bang multi-core Intel chip, VTune will tell you
stuff you probably never knew you should care about.

(I’m an Intel-free zone, and so I use AMD’s CodeAnalyst.) :slight_smile:

Database, eh? Maybe you should be recompiling the database from source
too. PostgreSQL? MySQL? There are lots of tweaking gizmos for them,
plus a benchmark called “MySQL Super Smack”.

7

On Oct 4, 2007, at 6:11 AM, Jerry West wrote:

In-memory with sqlite worked fine with rspec-0.8 (it’s been a while
since I did this!). Google for instructions or drop me a line. Don’t
forget autotest/zentest to run only those tests which have ben
affected
by changes.

Actually, on my current rails project I can’t use sqlite, because of
a bug in rails:

http://dev.rubyonrails.org/ticket/9385

So out goes that idea (at least for the forseeable future). What
speed difference do you see with in-memory database?

I’m already using ZenTest. When going from red → (subset) green =>
(all) green, the last step is killing me (because I have to wait
around for 2 minutes for all the specs to fail. This usually results
in me spending a lot of time responding to emails while I should be
developing :wink:

Scott

8

Scott,

I don’t really have a lot to contribute on how to make it faster,
other than to outline what we’ve been doing on our projects.

On one of our current projects we have the following 2570 examples
that run in ~70 seconds on our pairing stations (mac minis, 1.83
c2d). In general across our various machines is at or a little more
than a minute for specs for controllers, models, helpers, lib, views,
and plugins. Our Story suite takes longer, but it’s still under
development so I don’t really count it at this point. We have Ruby
1.8.6 installed from MacPorts on all machines, as well as MySQL 5.
(current as of a month ago) from macports.

We make good use of mocking and stubbing through our controller
tests, and little use of fixtures. We primarily use the
spec_attribute_helper (or factory method) as Luke R. and Dan
Manges have outlined in their respective blog articles. I’ve been
looking at deep test, or possible spec_distributed as a way to speed
things up more. Our main issue is our precommit task (rake cruise
which our ci server also runs) executes rcov to test for full
coverage and adds 15-25 seconds to the whole thing bringing it up to
a minute and a half.

We also make heavy use of some simple custom rcov tasks ala rake
app:coverage:models, rake app:coverage:lib, etc to help when we are
working on slices of the apps. So far we have found that regardless
of the wait (and the increased chances of wandering about to other
tasks) we always want to run the full stack locally and not just a
slice.

Then again all this strikes me as rather funny as I remember waiting
5 or 6 minutes back in my .NET days for a quarter our tests to run,
with no coverage report. I guess ruby and rspec spoiled me :slight_smile:

  • Chad

NOTE: Our project is on EdgeRails (few revisions back) and Trunk RSpec

9

On Oct 4, 2007, at 6:11 AM, Jerry West wrote:

In-memory with sqlite worked fine with rspec-0.8 (it’s been a while
since I did this!). Google for instructions or drop me a line. Don’t
forget autotest/zentest to run only those tests which have ben
affected
by changes.

Actually, on my current rails project I can’t use sqlite, because of
a bug in rails:

http://dev.rubyonrails.org/ticket/9385

So out goes that idea (at least for the forseeable future). What
speed difference do you see with in-memory database?

I’m already using ZenTest. When going from red → (subset) green =>
(all) green, the last step is killing me (because I have to wait
around for 2 minutes for all the specs to fail. This usually results
in me spending a lot of time responding to emails while I should be
developing :wink:

Scott

10

On Oct 7, 2007, at 12:31 AM, Chad H. wrote:

Scott,

I don’t really have a lot to contribute on how to make it faster,
other than to outline what we’ve been doing on our projects.

On one of our current projects we have the following 2570 examples
that run in ~70 seconds on our pairing stations (mac minis, 1.83

I assume your “pairing stations” are two separate mac-minis, in which
you practice pair programming? Or is this a cluster of two mac-minis?

But this sounds great - 70 seconds for 2500 specs. How many of those
are model specs (that hit the database)?

Manges have outlined in their respective blog articles. I’ve been
looking at deep test, or possible spec_distributed as a way to speed
things up more. Our main issue is our precommit task (rake cruise

The factory method (or attribute_helper) still hits the database. I
don’t see it as any sort of performance gain. In fact, I’ve even
developed a plugin around the Factory idea, and it was only when I
started using it in all of my tests that the speed really started to
affect me (I was using mocking/stubbing, with much frustration prior
to that point). But to me it’s pretty clear the plugin (or the
factory) is not the problem - the hit is the database. DHH saw this
hit, and since they were using fixtures,he found that creating the
fixtures, and then wrapping each test in a transaction was a huge
performance gain. I wonder if the same would be true with setups/
before(:each)…

The obvious thing to do to solve the performance problem is to remove
the hit to the database. The question is: At what level of
abstraction should this be done? The one camp (which would include
fellows like Jay Fields), would mock/stub everything they don’t
write. For me, I see testing as more than testing - it’s the
documentation to my code which never lies to me (this documentation
is so good, that I can give it to my boss, who is not a programmer).
For that reason (testing is not about testing), I could never stub/
mock AR inline - that is, in the tests themselves (although if I had
some sort of external plugin to do this for me, I may feel
differently about it).

The other alternatives seem to be: b) writing a sql parser, and c)
speeding up the database (deeptest, in-memory databases, more
hardware, etc). The sql parser is a big job, and speeding up the
database doesn’t seem to give the performance improvement that I’m
looking for.

As for deeptest, I wouldn’t mind helping out with getting it to work
for rspec (email me off-list at [email protected], if you are
interested in my help). I haven’t explored spec_distributed at all
(in fact, I didn’t even know it existed, although I had a similar
thought a few days ago).

which our ci server also runs) executes rcov to test for full
coverage and adds 15-25 seconds to the whole thing bringing it up to
a minute and a half.

Coverage is not a big point for me. I’m happy with running rcov only
once a week, assuming that I have 100% right now, and I develop
everything test-first. Right now I’m no where near 100%, so I’m not
too worried about it.

Regards,

Scott

11

On 10/6/07, Scott T. [email protected] wrote:

1.8.6 installed from MacPorts on all machines, as well as MySQL 5.
don’t see it as any sort of performance gain. In fact, I’ve even
The obvious thing to do to solve the performance problem is to remove
the hit to the database. The question is: At what level of
abstraction should this be done? The one camp (which would include
fellows like Jay Fields), would mock/stub everything they don’t
write. For me, I see testing as more than testing - it’s the
documentation to my code which never lies to me (this documentation
is so good, that I can give it to my boss, who is not a programmer).

I think you’d be far better served by user stories in this case. The
user story is the tool we use to communicate with non-geeks, be it
customers or our boss. Low-level specs fall squarely in the developer
realm, as far as I’m concerned. They’re easy to read because we don’t
like to torture ourselves. I think that they’re fantastic for
communicating intent with other developers, but that’s where it ends.

So, when you move documentation as a goal from specs over to stories,
that frees specs to take advantage of techniques like mocks to help
with your design. Documentation is a pleasant side effect of BDD, not
a goal.

Pat

12

On 10/6/07, Scott T. [email protected] wrote:

other than to outline what we’ve been doing on our projects.

spec_attribute_helper (or factory method) as Luke R. and Dan
to that point). But to me it’s pretty clear the plugin (or the
write. For me, I see testing as more than testing - it’s the
So, when you move documentation as a goal from specs over to stories,
adds a lot of extra noise to a rails test, where it doesn’t in the
other projects that I’ve worked on.

Well, one of the main parts of programming is making tradeoffs.
You’re probably not going to run 2500 tests that hit the db in under
30 seconds. So you either don’t hit the db or you don’t run all the
tests all the time. You can certainly change autotest to not
automatically run the full suite after you go red->green. Only have
it run the specs for the files you changed.

Break your specs into fast and slow groups. Run the fast ones
continuously and the slow ones left often. Work on moving one or two
slow specs into the fast group every day. You’ll probably discover
some techniques that you can share with all of us :slight_smile:

As far as mocks adding a lot of extra noise, I find in general that
when mocks are painful for me it’s because my code is too tightly
coupled. Just like my body tells me something is wrong through pain,
so does my code.

But whether or not you call the specs “stories” or “integration
specs” (it is, after all, all a continuum of what is considered an
“integration” test), I still have no idea how to run them any faster.

Well, I don’t want to beat this to death, but I think it’s dangerous
to look at specs and stories as simply different degrees of the same
concept. The fundamental difference is that you should not change a
story without consulting a customer, and you should not have to
consult a customer to change a spec. Stories represent assumptions
and business value. Assumptions are learned through conversations
with the customer, and only the customer can define business value.
Specs are merely a tool to help us generate that business value, and
velocity goes way down if we have to worry that changing some specs
will impact the customer in some way. Worse, if we don’t worry about
it, we may end up changing some assumptions around and thus deliver
the wrong product.

In the ever popular analogy of car mechanics, I would want my mechanic
to ask me before replacing an axle on my car, but I would not examine
the wrench he chooses to perform the maintenance.

Pat

13

On Oct 7, 2007, at 1:47 AM, Pat M. wrote:

that run in ~70 seconds on our pairing stations (mac minis, 1.83
views,
looking at deep test, or possible spec_distributed as a way to speed
fixtures, and then wrapping each test in a transaction was a huge

a goal.
Documentation is certainly a goal of BDD (Just not necessarily for
the business user - as you pointed out). I don’t write my specs to
be understandable by the business user, although it just often
happens that they are (at least in this rails context, since even
most of the fields of the database are more-or-less user facing).
The point that I was trying to hammer home is that mocking/stubbing
adds a lot of extra noise to a rails test, where it doesn’t in the
other projects that I’ve worked on.

But whether or not you call the specs “stories” or “integration
specs” (it is, after all, all a continuum of what is considered an
“integration” test), I still have no idea how to run them any faster.

Scott

14

performance gain. I wonder if the same would be true with setups/
before(:each)…

Scott,

Where is this factory plugin you speak of?

Thanks,
Ben

15

On Oct 7, 2007, at 11:42 AM, Pat M. wrote:

I don’t really have a lot to contribute on how to make it faster,
But this sounds great - 70 seconds for 2500 specs. How many of
1.8.6 installed from MacPorts on all machines, as well as MySQL 5.
cruise
factory) is not the problem - the hit is the database. DHH saw
fellows like Jay Fields), would mock/stub everything they don’t
realm, as far as I’m concerned. They’re easy to read because we
a goal.
Well, one of the main parts of programming is making tradeoffs.
You’re probably not going to run 2500 tests that hit the db in under
30 seconds. So you either don’t hit the db or you don’t run all the
tests all the time. You can certainly change autotest to not
automatically run the full suite after you go red->green. Only have
it run the specs for the files you changed.

Break your specs into fast and slow groups. Run the fast ones
continuously and the slow ones left often. Work on moving one or two
slow specs into the fast group every day. You’ll probably discover
some techniques that you can share with all of us :slight_smile:

My current thought is this:

  1. Use sqlite in memory database when developing (but use the CI
    server to run the specs against mysql)
  2. Start breaking up the spec files into more managable groups.
    Right now 100 of our 500 specs are in one file (the user_spec.rb).
    It’s a social networking app, so it makes sense that most of the
    specs would be centered around the user. But the description blocks
    should show some sort of logical grouping, which can then be split up
    into different spec files (user_with_items_spec.rb,
    user_with_comments_spec.rb, and so on).
  3. Use autotest (or a custom tool) to rerun this one file that I’m
    changing. It’s pretty obvious that the mapping to autotest is flawed
    (Dave A. has talked about the flaws of the one-to-one mapping of
    spec file to lib file (or app) several times). Maybe some sort of
    tool which could inflect that any change to the
    user_with_items_spec.rb should rerun with a change either to the User
    or to the Item model, as well as the spec file.

As far as mocks adding a lot of extra noise, I find in general that
when mocks are painful for me it’s because my code is too tightly
coupled. Just like my body tells me something is wrong through pain,
so does my code.

Uhum…I don’t think It’s my code that’s too tightly coupled (clears
throat rails). Seriously, though, I tend to agree with you - but
here I don’t have an option of rewriting rails test-first to be more
decoupled.

and business value. Assumptions are learned through conversations
with the customer, and only the customer can define business value.
Specs are merely a tool to help us generate that business value, and
velocity goes way down if we have to worry that changing some specs
will impact the customer in some way. Worse, if we don’t worry about
it, we may end up changing some assumptions around and thus deliver
the wrong product.

I agree with what you’ve said. But with rails (and a fat model), the
specs for the model almost sound like a user story (albeit, not a
full stack one). Here is a tiny part of the specdoc which I sent to
my boss:

The User

  • should only find the one item which is not ignored, if there are
    two items in the queue, and one is ignored

  • should only find one item in the review queue, if there are two
    items in the queue, but only one is ignored

  • should see all items in the review queue, if none have yet been
    ignored (and two are in the queue)

  • should not be allowed to ignore items from review queue if he has
    already reviewed 20 items

  • should raise an error if a nil item is given to be skipped

  • should raise an error if no item is given to be skipped

  • should not skip the item if the user is not allowed to skip the item

Obviously you can see I didn’t write these things for him - (the ones
at the end prove it. A “nil item” doesn’t mean anything to my boss.
But the other ones clearly show business value. Once again, (some of
these) specs just happen to be things which my boss can read. I
wrote those specs without mocking and stubbing, but they could as
just as well been mocked/stubbed to death.

Scott

16

On Oct 7, 2007, at 3:01 PM, Ben M. wrote:

fixtures, and then wrapping each test in a transaction was a huge
performance gain. I wonder if the same would be true with setups/
before(:each)…

Scott,

Where is this factory plugin you speak of?

http://replacefixtures.rubyforge.org/

There is also a screencast at the top of that page. I would highly
advise you watch it.

Scott

17

The Stories (ie Story Runner full integration tests that hit the db)
are fairly slow by comparison, I agree. We have those run on our CI
server, and only locally when we modify them. That’s our approach
for handling integration testing.

Re: The mac mini’s, yeah they were pairing stations (one per pair of
users, no distributed spec running yet). Our main specs are built
around heavier use of mocking/stubbing.

  • Chad