Community poll about testing and specs

Hi everyone,

I’m currently reviewing and revising the Testing chapter of my
upcoming book The Rails Way and I would like to include perspectives
from the wider Rails community about the subject. If you have a few
moments to respond, please help me with your feedback and opinions.
This request is pretty much open-ended, but here are some questions
that may serve as inspiration.

  1. Are test-driven development (tdd) or behavior-driven development
    (bdd) important aspects of your day-to-day practice? Did you pick up
    tdd from your involvement with Rails or bring it with you from another
    technology?

  2. Is there palpable peer pressure in the Rails community to adopt
    progressive testing practices such as tdd and bdd? (Or does that
    strike you as Agilist hogwash?)

  3. On your production projects, do you monitor and strive towards
    achieving 100% test-coverage? Why or why not? If you do keep track, do
    you use the built-in rake task, RCov, or some other method of gauging
    your code coverage?

  4. Unit, Functional, Integration tests…? Selenium and Watir…? All
    equally important in Rails? Or are some kinds of tests more important
    and valuable to you than others?

  5. Do you hate fixtures? Can I accurately say it is the Rails way to
    hate (or at least complain about) fixtures? Do you make use of mocking
    and stubbing instead? Mocha? Some combination?

  6. Have you switched, or do you plan to switch over to RSpec for your
    testing and specs? Is it important to you that I include some coverage
    of RSpec in a book entitled “The Rails Way” or do you feel it would be
    inappropriate?

Your comments are greatly appreciated.

Obie F.

Pre-order my book The Rails Way today!
http://www.amazon.com/dp/0321445619

On 7/23/07, Obie F. [email protected] wrote:

  1. Are test-driven development (tdd) or behavior-driven development
    (bdd) important aspects of your day-to-day practice? Did you pick up
    tdd from your involvement with Rails or bring it with you from another
    technology?

I’d like to say that we’re totally dedicated to TDD, but in practice
that hasn’t happened. However some of our most painful, annoying bugs
have resulted from “fixes” that weren’t test-driven. Once you slip up
here and there by just writing code, it becomes easier to justify
subsequent small changes without writing tests. But we all know about
code debt, broken windows, etc.

Personally, I’m in love with RSpec. When I started working for my new
company I was the only Ruby guy. Gradually two of the other fellows
have started writing Ruby code. They haven’t really been bitten by
the RSpec bug yet, but I think it’s just because they haven’t had time
to really get into it. One issue is that I work remotely so I don’t
get to show them the little tricks as much. I’m moving in a week
though so hopefully that’ll take care of that, and we’ll be a bunch of
RSpec-happy hackers.

  1. Is there palpable peer pressure in the Rails community to adopt
    progressive testing practices such as tdd and bdd? (Or does that
    strike you as Agilist hogwash?)

I don’t think there’s peer pressure, but TDD is certainly encouraged.
As you know, Rails puts a bunch of tests in place for you. It’s
really easy to write tests. You certainly don’t have to though.

That brings up a more interesting point (to me, anyway). There’s no
peer pressure to do anything in the Rails community. DHH picked a
language that made him happy. Rails was born from code that he felt
was beautiful. That kind of set the tone for the Rails community. So
much about Rails is subjective. People either “get it” or they don’t.
But there’s no malice from those who do towards those who don’t -
just gentle encouragement.

  1. On your production projects, do you monitor and strive towards
    achieving 100% test-coverage? Why or why not? If you do keep track, do
    you use the built-in rake task, RCov, or some other method of gauging
    your code coverage?

We use RCov, but we don’t strive for 100% coverage. It’s just not
practical really. Also, we’ve come to rely on RSpec less as a
regression testing tool and more of a design tool. We can only be
confident in our code if all of our acceptance tests are running. We
use RSpec to prevent us from writing crappy code that we have to keep
running :slight_smile:

  1. Unit, Functional, Integration tests…? Selenium and Watir…? All
    equally important in Rails? Or are some kinds of tests more important
    and valuable to you than others?

I like to start with functional tests because it’s the fastest,
easiest way to discover the objects and APIs in the system. We end up
with a very tight, clean API that’s useful.

Unit tests are super important. They allow us to refactor the code
base and get instant feedback when something breaks (or needs to
change).

We use Selenium for the integration/acceptance tests. They’re kind of
a pain to run, but they exercise all the important use cases.

The functional and unit tests are all written in RSpec, and I use
autotest to keep an eye on things for me. The selenium test suite
runs nightly, but we can also kick it off on the testing server
whenever we have to. When using RSpec as a design tool, it can’t
catch all the regressions, so we need Selenium tests.

  1. Do you hate fixtures? Can I accurately say it is the Rails way to
    hate (or at least complain about) fixtures? Do you make use of mocking
    and stubbing instead? Mocha? Some combination?

I don’t hate them. For the most part I don’t use them though…as I
mentioned earlier, in functional tests I mock out all the AR stuff,
because I’m just trying to figure out the design. Fixtures can come
in handier for unit tests, actually. For example we needed to
generate several reports that used a bunch of complicated SQL. Any
time you’re writing custom SQL you definitely need to hit the database
to verify that your code actually works. Fixtures were perfect there
since we didn’t want a lot of setup code.

To me, fixtures should be used when the code base is small, and you
can use the functional tests as mini integration tests. They can get
messy though as the code base gets larger, and of course they can slow
things down.

I also think that larger code bases will benefit more from RSpec’s
design capabilities than its testing capabilities. This might be
because fixtures can get out of hand, but it may simply be a matter of
mindset - if you’re focus your energy on design, you’re more likely to
end up with a good one.

If you use mocks, you HAVE to have some external acceptance tests.
It’s nice to use RSpec to discover the interfaces in your system, but
at the end of the day you rely on an implementation. You have to have
something in place to verify that your implementation is still working
as expected.

Note that you should always have acceptance tests of some sort. I
just feel that functional tests that use fixtures (and thus the AR
implementation) count as an acceptance test. They’re just not good
for when your app gets big.

  1. Have you switched, or do you plan to switch over to RSpec for your
    testing and specs? Is it important to you that I include some coverage
    of RSpec in a book entitled “The Rails Way” or do you feel it would be
    inappropriate?

Apparently I’ve been using RSpec for over a year now! [1]

I think you should definitely include it in there. I know a lot of
people look at RSpec and first think to themselves that it’s nothing
special, there’s no real point in learning it. However some really,
really sharp dudes (courtenay, topfunky) said that at the beginning
and have since admitted they just missed the beauty that is RSpec.
Again I think it’s probably a subjective thing…but if really smart
people are quick to dismiss it at the beginning, think of what the
rest of us would do. Maybe some of your readers won’t care, but I
think a lot more are going to get it and will love you for it.

I can honestly say that RSpec is the best thing about Rails coding.
Writing Rails apps is easy, it’s designing them well that’s hard.
RSpec is a great tool in that regard.

I wish someone would write a Ruby-based BDD book now :slight_smile:

Your comments are greatly appreciated.

Wow that got long. Hope some of it is useful.

Pat

On 7/23/07, Pat M. [email protected] wrote:

Apparently I’ve been using RSpec for over a year now! [1]

I doubt anyone cares, but the link that I meant to include is
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/192905

I may be biased, but I would really like and have been looking for an
alternative BDD choice besides RSpec.

What others are there? I have seen a couple by the likes of Ryan, even
Rick. Are any better? What about a conrast of testing libraries
highlighting the benefits and opinions of each?

I really want to get into testing beyond Test/Unit, but I detest the
RSpec library out of spite :slight_smile:

Nathaniel.

On 7/23/07, Obie F. [email protected] wrote:

  1. Are test-driven development (tdd) or behavior-driven development
    you use the built-in rake task, RCov, or some other method of gauging
  2. Have you switched, or do you plan to switch over to RSpec for your
    Pre-order my book The Rails Way today!
    http://www.amazon.com/dp/0321445619


Nathaniel Steven Henry Brown

Toll Free: 1-877-446-4647
Vancouver: 604-724-6624

On 24/07/07, Obie F. [email protected] wrote:

  1. Are test-driven development (tdd) or behavior-driven development
    (bdd) important aspects of your day-to-day practice? Did you pick up
    tdd from your involvement with Rails or bring it with you from another
    technology?

Yes, TDD is a core part of my day-to-day practice. I brought it with me
from
the world of Java.

  1. Is there palpable peer pressure in the Rails community to adopt

progressive testing practices such as tdd and bdd? (Or does that
strike you as Agilist hogwash?)

I don’t think there is much, but it’s a very subjective question.

  1. On your production projects, do you monitor and strive towards

achieving 100% test-coverage? Why or why not? If you do keep track, do
you use the built-in rake task, RCov, or some other method of gauging
your code coverage?

No. 100% test coverage doesn’t guarantee anything works. You can get
100%
test coverage with no assertions. Test coverage can be useful in
pointing
out what code is not being executed by your tests, but I don’t think
aiming
for 100% test coverage is sensible. We don’t use it, but Heckle could
help
find ineffective tests.

  1. Unit, Functional, Integration tests…? Selenium and Watir…? All

equally important in Rails? Or are some kinds of tests more important
and valuable to you than others?

A combination of different types of test are important, but I think it’s
most efficient to aim for more coverage from unit tests. And I mean unit
tests that test a single class in isolation as much as possible - not
unit
tests in the Rails sense.

  1. Do you hate fixtures? Can I accurately say it is the Rails way to

hate (or at least complain about) fixtures? Do you make use of mocking
and stubbing instead? Mocha? Some combination?

Yes I hate fixtures (
Rails fixtures - help or hindrance - James Mead).
I use Mocha.

  1. Have you switched, or do you plan to switch over to RSpec for your

testing and specs? Is it important to you that I include some coverage
of RSpec in a book entitled “The Rails Way” or do you feel it would be
inappropriate?

As a keen user of xUnit variations over the years, I’ve struggled to see
the
real benefits of switching to RSpec especially as the API kept changing
so
often. But the 1.0 release does look to have some nice terminology and
syntactic sugar. My one hesitation with it is what I feel is an
over-emphasis on the removal of duplication into setup-type methods. I
prefer to make my objects and therefore my tests simpler - in fact I
like to
end up with entirely self-contained test methods (even at the expense of
some duplication). But I realise this is probably just a matter of
taste.


James.

By the way… just checked out the Index of the book. I believe this is
going to be the #3 book in my collection. AWDWR and Programming Ruby
are the foundation, let’s see where this one takes us when its done :slight_smile:

Can’t wait to see it finished.

On 7/24/07, Nathaniel B. [email protected] wrote:

Nathaniel.

that may serve as inspiration.
3) On your production projects, do you monitor and strive towards
and stubbing instead? Mocha? Some combination?
Best Online Casino in Australia | Top Licensed Casinos for Gamblers

Toll Free: 1-877-446-4647
Vancouver: 604-724-6624


Nathaniel Steven Henry Brown

Toll Free: 1-877-446-4647
Vancouver: 604-724-6624

On 7/24/07, Nathaniel B. [email protected] wrote:

I may be biased, but I would really like and have been looking for an
alternative BDD choice besides RSpec.

a lot of people use
test/spec(http://test-spec.rubyforge.org/test-spec/) and
test/spec/rails (usually at
http://plugins.require.errtheblog.com/wiki/TestSpecRails)

-chris

On 7/23/07, Obie F. [email protected] wrote:

  1. Are test-driven development (tdd) or behavior-driven development
    (bdd) important aspects of your day-to-day practice? Did you pick up
    tdd from your involvement with Rails or bring it with you from another
    technology?

Yes, TDD/BDD/E(example)DD/etc are part of my daily practice. I’ve been
practicing TDD since 2003, at first in java.

  1. Is there palpable peer pressure in the Rails community to adopt
    progressive testing practices such as tdd and bdd? (Or does that
    strike you as Agilist hogwash?)

Whether or not you’re doing tests first, testing does seem very
important to the community. My sense is that writing them first is
less important to most people than making them expressive and DRY,
which aligns well with the sensibilities that bring many people to
Ruby and Rails.

  1. On your production projects, do you monitor and strive towards
    achieving 100% test-coverage? Why or why not? If you do keep track, do
    you use the built-in rake task, RCov, or some other method of gauging
    your code coverage?

Ditto what James M. says elsewhere in this thread.

  1. Unit, Functional, Integration tests…? Selenium and Watir…? All
    equally important in Rails? Or are some kinds of tests more important
    and valuable to you than others?

As a matter of practice, I tend to have integration tests (rails-style
or in-browser) coupled with highly isolated granular examples of
individual objects. Ironically, I tend to use mocks for controller,
view and helper examples while I let the model examples (“unit tests”
in rails) talk to the db.

  1. Do you hate fixtures? Can I accurately say it is the Rails way to
    hate (or at least complain about) fixtures? Do you make use of mocking
    and stubbing instead? Mocha? Some combination?

I like to see everything I need to understand an example in one place.
My main beef with fixtures is that you have to look in another file to
understand the state of the objects in your tests. That said, I think
that ideas like fixture scenarios are promising, as long as you name
things well.

  1. Have you switched, or do you plan to switch over to RSpec for your
    testing and specs? Is it important to you that I include some coverage
    of RSpec in a book entitled “The Rails Way” or do you feel it would be
    inappropriate?

Well, I’ll try to respond in an un-biased way (wish me luck). To me,
The Rails Way is an ongoing experiment in search of new ways to
approach software that are both enjoyable from an aesthetic
perspective, and pragmatic. Whether RSpec is alive and well or just a
distant memory 5 years down the line doesn’t really matter. What is
important about RSpec is the discussion and exploration that it is
part of and, in part, inspires.

I think that a section on BDD, touching at least the surfaces of
RSpec, test/spec and simply-bdd would definitely be appropriate. It
would also be great to see this in the context of a discussion of
testing practices - dispelling what I view as a myth that if you’re
using x tool then you’re doing y practice.

Your comments are greatly appreciated.

Looking forward to this book, Obie. Good luck!

Cheers,
David

  1. Are test-driven development (tdd) or behavior-driven development
    (bdd) important aspects of your day-to-day practice? Did you pick up
    tdd from your involvement with Rails or bring it with you from another
    technology?

TDD is very important. I learned it from Rails, and now long for it when
I
have to work in other languages, especially ones where there are no good
testing frameworks.

  1. Is there palpable peer pressure in the Rails community to adopt

progressive testing practices such as tdd and bdd? (Or does that
strike you as Agilist hogwash?)

To an extent. People follow what the community “experts” do. I think
that
there’s more peer pressure in terms of what to use. RSpec is a good
example of that… lots of people are saying it’s the thing to use.

  1. On your production projects, do you monitor and strive towards

achieving 100% test-coverage? Why or why not? If you do keep track, do
you use the built-in rake task, RCov, or some other method of gauging
your code coverage?

We use RCov to track coverage and shoot for close to 99%. We use RCov
mostly
to tell us what things a developer didn’t test.

  1. Unit, Functional, Integration tests…? Selenium and Watir…? All

equally important in Rails? Or are some kinds of tests more important
and valuable to you than others?

Unit tests, functional tests. Integration tests for situations that have
lots of redirection. Selenium too.

  1. Do you hate fixtures? Can I accurately say it is the Rails way to

hate (or at least complain about) fixtures? Do you make use of mocking
and stubbing instead? Mocha? Some combination?

Love / hate. I use them for simple things like populating data, but I
tend
to create instances in test helpers instead.

  1. Have you switched, or do you plan to switch over to RSpec for your

testing and specs? Is it important to you that I include some coverage
of RSpec in a book entitled “The Rails Way” or do you feel it would be
inappropriate?

I really would like to know what the benefits are. Lots of people are
switching and suggesting that it’s the better way, yet the core seems to
like test:unit. To me it looks like the same thing with some nice new
features and a DSL, so I’m not fully convinced I want to change my
practices
(and support two testing frameworks). If there are compelling reasons
though, then it’s worth it.

I’m writing a book for O’Reilly aimed at Windows developers, and I am
not
planning to cover RSpec at this time. I don’t think it’s appropriate to
include it in your book though if the community deems it important. (It
would be a slam-dunk if core decided to use rspec).

I think that a section on BDD, touching at least the surfaces of
RSpec, test/spec and simply-bdd would definitely be appropriate. It
would also be great to see this in the context of a discussion of
testing practices - dispelling what I view as a myth that if you’re
using x tool then you’re doing y practice.

Actually, leave my simply-bdd plugin out. It started as a 6-line
joke, and I’m glad Chris was able make test/spec :slight_smile:


Rick O.
http://lighthouseapp.com
http://weblog.techno-weenie.net
http://mephistoblog.com

Awesome responses so far, everyone. Thanks and keep them coming! I’ll
post my own feelings and some sort of wrap-up in a couple of days.

Obie F.

Pre-order my book The Rails Way today!
http://www.amazon.com/dp/0321445619