Spec'ing via features

Testing RSpec
1

As I work with Rails TestUnit tests I am reconsidering how to use
cucumber features. It seems to me that it might be best to have a
coherent view of how to arrange my test suites before I get much further
into this. Now, so far I have considered three possibilities:

  1. Use features exclusively. Create a feature file for each model, one
    for each controller and possibly an additional one for those views that
    need separate tests. Name then with the nomenclature x_model.feature,
    x_model_controller.feature and where desired, x_view.feature

  2. Use features exclusively. Create one feature file for each
    non-decomposable piece of system functionality. Test models,
    controllers and views within each feature file.

  3. Use features and RSpec. Well, is that not what I am doing with
    features alone? Or, are their cases when testing with RSpec spec files
    are a better choice than a features scenario?

  4. Use features and TestUnit. Naaahhh…

  5. Some other way of which I have not considered.

In the interest of enlightenment and from a desire to avoid unnecessary
back tracking as my project develops, what do experienced practitioners
suggest as the favoured way of arranging test suites.

Yes, I have done the goggle thing but I really have not found anything
terribly useful about organizing tests specifically for Rails.

2

James B. wrote:

into this. Now, so far I have considered three possibilities:

Ok, five…

3

Its tempting to see features as a kind of extended test tool, I
certainly looked at them in that way initially. However I think this
isn’t really the most profitable way to look at them, nor how they
were designed. It would probably be worthwhile if you looked up some
articles on BDD, and perhaps pondered a little further on the ideas of
customer developer communication, and of features driving development.
Also have a look at the declarative vs imperative feature stuff that
has appreared in this list and in Ben M.'s articles.

Once you do this you might agree that features are not an appropriate
tool for doing low-level testing, like testing individual models and
controllers. They certainly can reduce the need to do some tests in
the normal rails/rspec testing stack e.g. view testing but there much
more a design/communication tool than a testing tool

In particular I don’t see that driving unit tests with features as
being particularly productive because

  1. Customers don’t need to know the details of unit tests (they’re is
    to much detail for them to comprehend). In fact writing unit tests as
    features actually pollutes the feature space meaning that you end up
    having to try and seperate one kind of feature from another
  2. Programmers should be able to understand unit tests (they don’t
    need the story format)
  3. Unit tests are by nature very detailed and the step matching step
    writing mechanism is pretty inefficient at expressing this, even with
    FIT tables
  4. Features run slowly in comparison to well written test suites, so
    running autospec with them gets unproductive fairly quickly. This will
    be made worse if you create lots of extra features.

To summarize see features as things that drive development and bridge
the customer developer divide, then once you start developing
something use unit tests to ensure the intimate details of your models
and to help with regression etc… Then have a ponder about what you
need in the controller, view testing area. This should work fairly
well in Rails if you use ‘fat model skinny controller’.

There’s definitely the need for a really insightful blog article(s)
into the detail of moving through this process of feature to
implementation via unit test etc… I get the feeling there’s quite
alot of personal style involved and certainly haven’t worked out how
to do it myself yet

Anyhow HTH

Andrew

2008/11/21 James B. [email protected]:

4

On 21 Nov 2008, at 18:15, James B. wrote:

x_model_controller.feature and where desired, x_view.feature
4. Use features and TestUnit. Naaahhh…
terribly useful about organizing tests specifically for Rails.
No doubt it’s (3) for me. It’s all about working from the outside in.

Cucumber is for Acceptance Testing, and those acceptance tests should
not be cluttered up with edge cases. They should ideally be readable
by your stakeholders / customers.

Equally, when you hit something complicated in a class that’s likely
to be pulled in different directions by different features, it’s
responsible to start writing unit tests for it. Whether you choose to
use RSpec or Test::Unit for those unit tests is really up to you.
Those unit tests can then be used by the next programmer who comes
along to change that class to satisfy another feature as well as yours.

I am absolutely loving doing BDD from the outside in. Driving early
changes from Cucumber acceptance tests means you have so much more
space within which to refactor (not being hemmed in my unit tests /
mocks) as the design emerges during those first few iterations. It’s
been nothing short of a revelation for me.

cheers,
Matt

5

On 24 Nov 2008, at 18:29, Pat M. wrote:

  1. Use features exclusively. Create a feature file for each model,

unnecessary
Agreed.
$1 more than he has available in his account?

Lately I’ve been putting more and more stuff into ATs. I’m finding it
valuable to keep tests for domain logic separate from plain ol unit
tests…meaning that my Account object may be tested mostly with
Cucumber, but helper objects such as a stats aggregator will be spec’d
with code-level examples.

Yep. Agreed, you put it better than I could.

I am freaking people out at work because I seem to have practically
stopped writing unit tests, but I’m still certain that I don’t lift a
finger without some automated test to force me into it. I’m just
finding that ATs give me a much bigger lever in terms of how much code
they drive out with a single test.

I have a feeling that people who’ve developed with FIT / JUnit must
have been through this loop before. Any war stories out there?

cheers,
Matt

6

Matt W. [email protected] writes:

that
are a better choice than a features scenario?

Yes, I have done the goggle thing but I really have not found anything
terribly useful about organizing tests specifically for Rails.

No doubt it’s (3) for me. It’s all about working from the outside in.

Agreed.

Cucumber is for Acceptance Testing, and those acceptance tests should
not be cluttered up with edge cases. They should ideally be readable
by your stakeholders / customers.

I disagree with the part about edge cases. Acceptance Tests are about
defining and verifying business value, and edge cases are supremely
valuable to businesses. What happens when an ATM user tries to withdraw
$1 more than he has available in his account?

Lately I’ve been putting more and more stuff into ATs. I’m finding it
valuable to keep tests for domain logic separate from plain ol unit
tests…meaning that my Account object may be tested mostly with
Cucumber, but helper objects such as a stats aggregator will be spec’d
with code-level examples.

Pat

7

I’m working on a Rails project with someone else, and getting
different results on specs. After a lot of poking around for
differences, the only thing I can find is autospec in /opt/local/bin/
autospec vs. /usr/bin/autospec Seems like a big clue, but who
actually installs this - rspec, ZenTest? I can’t seem to google up
any clues as to what actually creates this, or how it might be
different on different machines. We both have the same version of
rspec, rspec-rails, zen-test, etc. and both on the same version of OS
X, both on the same project cloned out of git, same ruby out of port,
etc… So it’s not obvious why our specs aren’t running the same. This
is the only difference I can find - any theories of how it could be
different or what that might imply are welcome.

Thanks,
Steve

8

On Mon, Nov 24, 2008 at 10:29 AM, Pat M. [email protected] wrote:

I disagree with the part about edge cases. Acceptance Tests are about
defining and verifying business value, and edge cases are supremely
valuable to businesses. What happens when an ATM user tries to withdraw
$1 more than he has available in his account?

Withdrawing $1 more than available is an edge case, vulnerable to an
off-by-one error, and doesn’t need to be shown to business, in my
opinion.
Testing that is a means of detecting errors - it’s not a specification.

What happens when an ATM user tries to withdraw $100 more than available
is
not an edge case, and should be shown to business.

I realize it’s a fine point - I’m just responding to whether the
business
needs to see what we call “edge cases.”

///ark

9

Hi guys,

I’m having trouble figuring out where the line between writing a spec or
a feature is.
Since I started with rspec stories, I have the idea that stories where
just the evolution of specs.
My main reason for this was the re-usability of steps throughout
stories, which I think is great.
On contraire, at our company, we have lots of very similar specs,
that check very similar things over and over,
which doesn’t’ seem to bother anyone too much, apart from myself.
They are simply not DRY.

I have the idea that using steps and resources we can come up with a
very coherent test framework, that will
simplify the most common tests, and allow us to concentrate on the
really specific things of our project.

I can vouch this idea with a very helpful number of steps
(which I’m still polishing with the help of my colleagues and would love
to share and extend)
Which either check things in the database or in the views, in a highly
reusable manner,
thus with 2 simple steps and a non-complex regexp,
you can basically substitute hundreds of lines of specs that do the same
job.

Now that I’m at it, let me point out controller and view specs, which if
I’m still up to date with the list,
have become less important to test with specs, either because we are
writing more restful applications,
or because Then steps and view specs cover the same areas of testing.
Now the ones that are left to discuss further are the model specs, or
unit tests, which is where I see the most repeated test code,
and believe it could be dryed up by using the concept behinds steps.

Last but not least, I don’t see why we should follow the MVC pattern of
rails to structure our test code.
It feels wrong to have to move all those things around when you refactor
(I know this is not a problem in static languages,
but still, there is a clear coupling between implementation code and
test code which I’m concerned with)

Looking forward to your comments and suggestions.

Kind Regards,

Raimond G.

10

On Nov 24, 2008, at 2:03 PM, Steven R. wrote:

port, etc… So it’s not obvious why our specs aren’t running the
same. This is the only difference I can find - any theories of how
it could be different or what that might imply are welcome.

What versions of rspec & zentest are the two of you running?

Does one of you have --reverse in spec.opts?

Scott

11

On Nov 24, 2008, at 2:05 PM, Scott T. wrote:

Does one of you have --reverse in spec.opts?

rspec/rspec-rails 1.1.11, ZenTest 3.11.0

we both have --reverse in spec.opts

SR

12

On Mon, Nov 24, 2008 at 6:18 PM, Steven R. [email protected]
wrote:

What versions of rspec & zentest are the two of you running?

Does one of you have --reverse in spec.opts?

rspec/rspec-rails 1.1.11, ZenTest 3.11.0

we both have --reverse in spec.opts

both have mtime as loadby order?

Also, will be nice to know what kind of differences you’re getting.

Just output or errors?

–
Luis L.
AREA 17

Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams

13

On Mon, Nov 24, 2008 at 2:18 PM, Mark W. [email protected] wrote:

What happens when an ATM user tries to withdraw $100 more than available is
not an edge case, and should be shown to business.

I realize it’s a fine point - I’m just responding to whether the business
needs to see what we call “edge cases.”

If the customer feels the scenario is important then it will be a
scenario. I would not rely on that scenario for exhaustive coverage
that withdrawing money from the account worked in all cases. Nor would
I go write more scenarios to cover withdrawing $1, $2, $100, etc. T

This is where the value of object-level code examples come into play.
Features/scenarios are intended to provide clear communication for the
behaviour of the system and the value it provides. They are not
intended to be exhaustive. Object level code examples (the specs) are
intended to be exhaustive. This is where we affect the design and
maintainability of the objects in our code. This is where we as
developers are disciplined to make sure that we have good examples and
regression against withdrawing too much money from an account, and not
worrying about other objects in the system unrelated to the behaviour
of an Account.

Object level code examples and scenarios combined give us a huge
amount of confidence in refactoring the system. One without the other
drastically reduces this confidence (at least for me). They work best
together. I don’t see it as a either-or proposition.

Well, the only time I see it as either-or is in the case of things
that end up like controllers. On a thread not to long ago Pat M.
mentioned that he stopped writing a lot of controller specs, and he
let the scenarios cover them. I agree with Pat, and I have since
started doing this where my controller doesn’t have logic that is not
tested outside of a scenario. This requires discipline to not let
things creep into your controller actions or filters, and to ensure
your controller is only used to wire together incoming requests with
outgoing responses. In this case, writing object level code examples
is largely unneeded IMO. However, as soon as some kind of logic is
introduced that isn’t adequately covered by a scenario, I’m still
writing controller specs.

my 2 cents,

–
Zach D.
http://www.continuousthinking.com

14

On Nov 24, 2008, at 2:21 PM, Luis L. wrote:

the only
rspec/rspec-rails 1.1.11, ZenTest 3.11.0

we both have --reverse in spec.opts

both have mtime as loadby order?

Yes - they both have the loadby order because they’re running in
unmodified git clones of the same project - so whatever the
differences are, they’re in the machine environment, not in flags and
files within the Rails project.

Also, will be nice to know what kind of differences you’re getting.
Just output or errors?

The differences are errors. Still would like to know where /opt/local/
bin/autospec comes from if anyone knows . . .

SR

15

On Mon, Nov 24, 2008 at 7:23 PM, Steven R. [email protected]
wrote:

/opt/local/bin/autospec comes from if anyone knows . . .

Is weird that you have autospec installed in two different places,
unless there was some forced GEM_PATH and GEM_HOME for it.

Both installations are using the bundled ruby version that came with
OSX?

If you post the errors will be much helpful, is hard to figure out
things and assist you without the proper details.

Luis L.
AREA 17

Human beings, who are almost unique in having the ability to learn from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams

16

On Mon, Nov 24, 2008 at 1:07 PM, Pat M. [email protected] wrote:

In my hypothetical example, the
specification is what to do when someone withdraws more than they have
in their account. Then when you write scenarios, the simplest thing you
can do to show that is the edge case itself, which is in fact valuable.
$1 over is (probably) the same as $100 over, and I think that $100
raises more questions than it answers. Why is it $100? Is there some
business rule that charges a penalty if they go $100 over the limit, but
not less?

I was addressing the general idea of presenting what we call “edge
cases” to
business users. Put it this way: if edge cases were necessary in
general,
then in addition to the scenario with $1, we’d need one with $0.01 and
one
with $0.

But to respectfully agree with you, if the edge case is actually a
business
edge case and not just a test for, e.g., an off-by-one error, then I’ll
go
along with Zach and you and say that $1 is a good scenario.

///ark

17

“Mark W.” [email protected] writes:

On Mon, Nov 24, 2008 at 10:29 AM, Pat M. [email protected] wrote:

I disagree with the part about edge cases.  Acceptance Tests are about
defining and verifying business value, and edge cases are supremely
valuable to businesses.  What happens when an ATM user tries to withdraw
$1 more than he has available in his account?

Withdrawing $1 more than available is an edge case, vulnerable to an off-by-one error, and doesn’t need to be shown to business, in my
opinion. Testing that is a means of detecting errors - it’s not a specification.

I respectfully disagree :slight_smile: In my hypothetical example, the
specification is what to do when someone withdraws more than they have
in their account. Then when you write scenarios, the simplest thing you
can do to show that is the edge case itself, which is in fact valuable.
$1 over is (probably) the same as $100 over, and I think that $100
raises more questions than it answers. Why is it $100? Is there some
business rule that charges a penalty if they go $100 over the limit, but
not less?

Pat

18

On 25/11/2008, at 7:29 AM, Pat M. wrote:

Lately I’ve been putting more and more stuff into ATs. I’m finding it
valuable to keep tests for domain logic separate from plain ol unit
tests…meaning that my Account object may be tested mostly with
Cucumber, but helper objects such as a stats aggregator will be spec’d
with code-level examples.

Pat

Just wondering a bit on this … the bit about your Account object
being tested mostly through Cucumber vs unit tests.

In my mind (as I think about this) cucumber is more about testing than
rspec is, meaning Acceptance Tests are more about testing where unit
tests when using TDD is more about design than testing. Do I make
sense? Email is not the best medium for conversations :- )

Now from what I have read of your writings I am assuming that you are
using TDD … and so just from what I read above I am wondering if
that statement is true because TDD is driving the implementation of
Account, and so you will have a 100% coverage of the Account class
behaviour, but when it comes to testing, the AT is doing more of that
(meaning testing). Is that it? IIRC, in Kent Beck’s TDD book there
is the example of TDD’ing a triangle (or something), but it was
something that showed that to TDD the triangle you ended up with less
tests than if you were testing the triangle.

How’s that sound?

Cheers
Shane

19

Shane M. [email protected] writes:

that statement is true because TDD is driving the implementation of
Shane

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

Here’s my latest Theory of Testing, in a nutshell:

Logic in a system falls into one of two categories, business logic and
supporting (infrastructural?) logic. Business logic is often then
further subdivided into domain logic and application logic. Domain
logic is the logic essential to the domain being used and would remain
consistent between applications, whereas app logic is unique to each
application. Most domain models I’ve seen used in Rails apps are used
only in that Rails app, and tend to be simple enough, that this
distinction doesn’t matter a whole deal. Well, actually I think Rails
devs just don’t think in those terms, but whatever.

It’s this business logic that is important to capture in acceptance
tests, in my opinion. We can write ATs for domain logic, as with the
example I gave for the ATM withdrawal. And we can also write ATs for
application logic, which is stuff like showing a success message on a
page, or sending off an email, or whatever.

I think some of the confusion comes from “edge case” having a pretty
particular meaning to developers, but in the example I gave there’s a
lot of overlap between an important business scenario and a programmer’s
edge case. I’m finding that when that occurs, and I can be confident in
my code after writing acceptance tests to cover the business scenarios,
I don’t need to write object examples.

Then there are a bunch of objects that provide neither domain nor
application logic - they’re basically just the result of refactoring to
keep your design clean. These are the objects I write examples for.
They don’t matter to the customer, but it is important to me that they
work and are well-designed. And actually, I’m starting to relax on
those a bit as well sometimes…Yehuda advocates a style that is
basically JUST acceptance testing (though at various levels, because
sometimes the “customer” is a business user and sometimes it’s another
programmer using your API). While I don’t go as far as he does, I am
less strict about directly testing every single method of every single
object than I was just a few months ago.

Wow, if that’s it in a nutshell… :slight_smile:

Pat

20

Steven R. [email protected] writes:

I don’t really expect anyone else to slog through that - I would
really just like the one little tidbit “where does autospec come
from?”

autospec is installed as part of the rspec gem.

Pat