The ultimate test would be one that is focused on one thing such that
the test would
- break every time that thing broke
- break only when that thing broke
- give detailed feedback enabling you to focus on the thing’s
subpart necessary to identify and fix the problem
The ultimate test suite would be the set of such tests that covered
every single concern that exists in a project.
Some of these are concerns are easier to test than others. Some are
important and lend themselves to automation, thus enjoy great tool
support. Others take something far more abstract like a person’s
When considering a new test, I should ask myself what problem the
test solves, and what problem I really want the test to solve. I try
to write the test in terms of the second. For example, if I were to
write the following test:
it “should allow deposits and withdrawals” do
@account.balance.should == 55
The test would be valuable when the problem is that you need to track
how much money people have in their accounts.
If you faced a problem such as “Make sure the user sees an error when
they withdraw more than their balance,” you would not want a test
it “should have an overdraft error” do
post :withdrawals, :amount => 1_000_000_000_000 # even bill can’t do
If this test breaks at some point, we would make a change to the test
or production code in order to make it pass. We wouldn’t think of the
problem it was intended to solve though, because we are absorbed in
the problem that initiated the break-inducing change. The danger in
this is that the test appears to be robustly covering something
useful, but in effect can let problems leak through. Imagine that we
remove an important partial from a view. This test, whose ultimate
goal was to ensure that users see an error message, failed to catch
the problem where the message wasn’t displayed at all.
So you should write tests expressing the same level abstraction as the
problem you want them to solve. Somewhere you would need a test like:
it “should have an overdraft error” do
@page.should include(“Can’t withdraw more than your balance”)
Now you would need some way to get @page. Perhaps you make a request
to a controller, it hits a database, renders a response, and assigns
the response body to @page.
Or maybe you rendered a view using a fake @withdrawal, and you’ve got
another test that verifies that you assign @withdrawal in the
controller, and another test verifies that Withdrawal objects get an
error when you try to create them for an amount greater than the
target account balance. And you’ve got a test that you label an
Integration test to signal the fact that it integrates all of these
You should only write integration tests that check across valuable
boundaries. This does not restrict it to stuff like company-specific
code using an ORM framework, though. Because you should only write
tests that are valuable, sets of layered tests forms a small subsystem
requiring integration testing.
It is sometimes useful to have layers of tests that enable you to
localize problems. Other times the types of problems you solve will
be trivial or obvious and won’t require localization.
As a simple rule, more tests == more overhead. But if you’re missing
certain tests then you will not notice certain problems when they
appear. The art of all of this is identifying the set of tests that
maximizes your confidence and ability to produce valuable software.
With all that theory out of the way, what can we say about the tests
it “should return 405 (Method Not Allowed) if HTTP_ACCEPT is text/
@request.env[“HTTP_ACCEPT”] = “text/html”
response.response_code.should == 405
This test would be good in a situation where we had published an API
stating only XML requests were allowed.
assigns[:model].should == model
it “should render feed template” do
This test would be valuable in a context where the XML feed output is
complex. In that case, testing the output directly might not
sufficiently enable us to localize issues.
If you could write tests that examine the response body, without
reducing the clarity of the example group, you should do so. Fewer
tests == less overhead.
Obviously, both are very basic in their implementation, but still, I
ask… If you were writing the specs, which way would you write them?
Thanks for any guidance.
I hope that, despite the typical “it-all-depends-on-context”, I was
able to give you some insight into identifying and analyzing possible