Role of stories vs specs

Testing RSpec
1

I’ve been thinking about the role that stories played compared to
specs, and that “should we use should” thread brought those thoughts
up again.

First I want to discuss whether or not specs are authoritative
regarding the desired behavior of a system. I would say they’re a
good approximation of how the system currently runs according to the
developer’s understanding. I say they’re an approximation because if
you use mocks everywhere (as I believe you ought to) then you’re not
performing integrations and the specs are necessarily incomplete.

Why does that idea matter? Traditionally, we’ve leaned heavily on a
suite of unit tests to perform refactorings. In fact, Fowler says the
first thing you have to do in order to refactor is grow a set of
comprehensive unit tests.

Refactoring, by definition, is changing the structure without changing
the observable behavior. Implicit in using tests to enable
refactoring is the idea that those tests precisely define what the
behavior of a system should be. If we say that our specs are not
actual specifications of desired behavior, then specs aren’t
sufficient safety nets for refactoring.

This mostly comes from my experience that refactoring is often a PITA
when using mocks. Mocks isolate dependencies, which is great, and
makes it easy to change code without affecting parts of the system
that really shouldn’t be affected. However they make simple, standard
refactorings like “move method” more difficult.

If I say that stories are authoritative and specs aren’t, then that
means I ought to rely on stories to provide the safety net for my
refactoring. When taking those baby steps, it’s important to keep the
stories green rather than the specs.

Another result of this thinking is that more specification should be
done in stories than I originally thought. Edge cases are my prime
example. I used to think I should write a story that expresses the
general desired behavior, and then cover any edge cases in specs.
However edge cases should be covered by stories because stories define
the desired behavior of the system. Also, edge cases presumably
represent some business value to the customer. Finally, it’s better
to keep all of the desired behavior specs in one place. Stories are
naturally more verbose than specs though.

That paragraph feels awkward to me, because I think I’m missing some
vocabulary. It suggests to me that the collection of all user stories
is the app specification - “spec” for short. What is the collection
of all descriptions, the object-level specs? I’ve always just said
“specs,” but I don’t think that really fits anymore with my current
thinking. Not to mention it’s just confusing if I also use “spec” to
mean the collection of all user stories. I know that an individual
describe block is a “description,” but maybe the whole suite should be
called “descriptions” instead of “specs?”

What do you guys think? Is it useful to draw a line and say that
stories are authoritative and specs aren’t? Do you think it becomes
less crucial to use object-level specs to enable refactoring, and
instead lean on stories more?

Pat

2

On Nov 13, 2007 1:10 PM, Pat M. [email protected] wrote:

Why does that idea matter? Traditionally, we’ve leaned heavily on a
suite of unit tests to perform refactorings. In fact, Fowler says the
first thing you have to do in order to refactor is grow a set of
comprehensive unit tests.

Refactoring, by definition, is changing the structure without changing
the observable behavior.

IMO, that definition refers to a system, not individual objects. When
you change objects, you usually have to change the examples.

Keep in mind that that goes nearly unnoticed in java or .NET because
of the refactoring tools available on those platforms. Change a method
name once and it changes it everywhere, including the examples. Move
a method from one object to another and you get a dialog that let’s
you opt to leave the existing method in place delegating to the new
location, or change all the consumers of that method in the system,
including the examples, to use the new location.

Implicit in using tests to enable
refactoring is the idea that those tests precisely define what the
behavior of a system should be.

Not so much about precision as it is about comprehensiveness.

If we say that our specs are not
actual specifications of desired behavior, then specs aren’t
sufficient safety nets for refactoring.

Again - this word “specification” is causing trouble.

Examples have three different roles. When you first write them, they
are examples of how you want an object to behave. After you get them
to pass, they become both executable documents of the system as it is
and regression tests.

In their role as regression tests, they serve the same role as unit
tests do in refactoring.

This mostly comes from my experience that refactoring is often a PITA
when using mocks. Mocks isolate dependencies, which is great, and
makes it easy to change code without affecting parts of the system
that really shouldn’t be affected. However they make simple, standard
refactorings like “move method” more difficult.

Again - this is a solved problem in other platforms and will
eventually be solved in ours.

If I say that stories are authoritative and specs aren’t, then that
means I ought to rely on stories to provide the safety net for my
refactoring. When taking those baby steps, it’s important to keep the
stories green rather than the specs.

On some level that makes sense - but it’s going to slow you down quite
a bit when you start building up a significant suite of stories.

The way I’ve always dealt with this was to rely on the examples to get
me through a refactoring, and then run the stories before committing.
That allows me to stay focused on small problems and progress
steadily.

By the way, when I say “always dealt with this”, I’m talking about
some years of experience using FitNesse for stories and xUnit for
examples. So although the Story Runner is new, the idea of having two
separate environments - one for system-level examples and one for
object-level examples - has been around for a while.

Another result of this thinking is that more specification should be
done in stories than I originally thought. Edge cases are my prime
example. I used to think I should write a story that expresses the
general desired behavior, and then cover any edge cases in specs.

I think they should be covered in both stories and examples. The thing
is that they get covered in different ways.

For one, the language in which they are expressed is different -
because the stories are about the system and the examples are about
the objects in the system. And there will be some duplicated effort
between the stories and the examples - and that is a GOOD thing.

Additionally, when dealing with stories, you may specify that a
particular error message shows up on the screen. However, when you get
down to the individual objects, you would likely have a single example
showing that the screen will show whatever error message you throw at
it, and then move down to the model to generate this particular edge
case message.

However edge cases should be covered by stories because stories define
the desired behavior of the system.

Definitely!

Also, edge cases presumably
represent some business value to the customer.

Definitely!

Finally, it’s better
to keep all of the desired behavior specs in one place.

Here’s where we disagree. I think that it’s a matter of who the
audience is for the different tools.

describe block is a “description,” but maybe the whole suite should be
called “descriptions” instead of “specs?”

The vocabulary that we seem to be moving to in the trunk is this:

Story Framework

  • Stories
  • Scenarios
  • Steps

Example Framework

  • ExampleGroups (what you referred to as descriptions)
  • Examples

I’m not completely comfortable with this yet in terms of being able to
talk about it, so suggestions are welcome. The problem with using Spec
or Example is that either can easily refer to the coarse grained
stories or the fine grained object level examples. Maybe we should
call them Coarse Grained Examples and Fine Grained Examples :slight_smile:
Seriously - I welcome other thoughts on this.

What do you guys think? Is it useful to draw a line and say that
stories are authoritative and specs aren’t?

I don’t find this useful. To me they are two completely separate
animals that work together to describe the behaviour of a system. They
are equally authoritative (or not). They both have a life cycle in
which they move from examples of what we’d like to examples of what we
have.

Do you think it becomes
less crucial to use object-level specs to enable refactoring, and
instead lean on stories more?

I think it’s useful to use the stories as a periodic sanity check (as
I mentioned above), but I think that if you start using stories as
your safety net while you’re in the rapid flow of refactoring, you’re
begging for a significant drop in productivity. They’re just going to
take to long to run and won’t help you to isolate problems as they
arise.

Thanks for the thought provoking post!

Cheers,
David

3

On Nov 13, 2007 11:51 AM, David C. [email protected] wrote:

performing integrations and the specs are necessarily incomplete.
you change objects, you usually have to change the examples.
I agree completely. That opinion is what spurred this line of thought.

Keep in mind that that goes nearly unnoticed in java or .NET because
of the refactoring tools available on those platforms. Change a method
name once and it changes it everywhere, including the examples. Move
a method from one object to another and you get a dialog that let’s
you opt to leave the existing method in place delegating to the new
location, or change all the consumers of that method in the system,
including the examples, to use the new location.

Which is fantastic, but what strategies should Ruby dudes use in the
mean time?

If I say that stories are authoritative and specs aren’t, then that
means I ought to rely on stories to provide the safety net for my
refactoring. When taking those baby steps, it’s important to keep the
stories green rather than the specs.

On some level that makes sense - but it’s going to slow you down quite
a bit when you start building up a significant suite of stories.

That was my main concern. You can get 1200 examples running in under
5 seconds. Probably can’t do that with stories.

The way I’ve always dealt with this was to rely on the examples to get
me through a refactoring, and then run the stories before committing.
That allows me to stay focused on small problems and progress
steadily.

Herein lies the (my) problem. I find that when I refactor the
examples break. You’ve already said that it’s okay, because tools
ought to change the examples appropriately. I don’t personally have
an issue with breaking specs as I refactor…but that’s because I’ve
learned to deal with it, and know what kind of breaking changes I’m
comfortable with. My main point is that if I have to go in and
manually change specs, then in general I don’t consider the specs to
be my comfortable safety net. They’ll get there when we have better
tool support, but I have to do stuff today.

I suppose the real problem is that I can’t quantify what impact that
all has on my velocity. So I just end up focusing on the pain that it
causes me. But, I think getting rid of the pain is a very worthwhile
endeavor on its own.

the objects in the system. And there will be some duplicated effort
between the stories and the examples - and that is a GOOD thing.

Agreed. The more-than-occasional-less-than-frequent duplication
between stories and examples doesn’t bother me. In my mind they’re
completely separate systems, it’s just that sometimes I happen to be
working on such a simple level that they appear to be the same.

Here’s where we disagree. I think that it’s a matter of who the
audience is for the different tools.

Let’s say in some banking software we have a transfer screen, and
there are three possible errors: insufficient funds, source account
frozen, target account frozen. Would you write a scenario for each of
the possible errors, and an example for each? This probably isn’t a
good example because I think you should :slight_smile: but it’s easy to see how
it would lead to more duplication than I’m comfortable with. Every
time you need to specify business value you have to stick it in a
story and an example.

I’m not completely comfortable with this yet in terms of being able to
talk about it, so suggestions are welcome. The problem with using Spec
or Example is that either can easily refer to the coarse grained
stories or the fine grained object level examples. Maybe we should
call them Coarse Grained Examples and Fine Grained Examples :slight_smile:
Seriously - I welcome other thoughts on this.

There’s another part missing, which is the set of all stories, and the
set of all example groups.

What do you guys think? Is it useful to draw a line and say that
stories are authoritative and specs aren’t?

I don’t find this useful. To me they are two completely separate
animals that work together to describe the behaviour of a system. They
are equally authoritative (or not). They both have a life cycle in
which they move from examples of what we’d like to examples of what we
have.

I agree that they’re completely separate beasts. I have a feeling
we’ll disagree on the point of what is authoritative. I think that
stories clearly define how much of the system is done, and examples
don’t do that at all. However, I’m not sure yet if it’s an important
distinction to make, and hopefully over the next few months I’ll gain
deeper insight.

Thanks for the thought provoking post!

Thanks so much for your reply.

Cheers,
Pat

4

On Nov 13, 2007 4:00 PM, David C. [email protected] wrote:

is responsible for producing the correct error message.
When I gave that example I thought it sucked, because I would write a
story and example for each too. But I guess I can say it served as a
bit of a sanity check.

Anyway, a more gray-area example (imo) is Rails validations. We’ve
got a user registration page, and it requires the user to fill in
their name, email address, and a password.

Would you write a scenario where nothing is filled in? Where
everything but the name is filled in? And so on for each attribute.
How about combinations of missing stuff?

It seems to me like that story is on a VERY high level and all we
really care about is verifying that the plumbing works (so maybe I
need to think of this as a story for the UI domain instead of the
business domain?). If the user fills in everything, make sure there’s
a new record. If they don’t fill it in, make sure they’re shown the
registration page again. It’s cumbersome to try to cover every
possible scenario even though you “should.”

Would you write a story at a lower level, which actually tries to save
the objects to the db and verifies that nothing changes? That
definitely duplicates what’s going on in the model-level specs.

Hopefully this thread hasn’t died…

Cheers,
Pat

5

On Nov 13, 2007 5:17 PM, Pat M. [email protected] wrote:

On Nov 13, 2007 11:51 AM, David C. [email protected] wrote:

Keep in mind that that goes nearly unnoticed in java or .NET because
of the refactoring tools available on those platforms. Change a method
name once and it changes it everywhere, including the examples. Move
a method from one object to another and you get a dialog that let’s
you opt to leave the existing method in place delegating to the new
location, or change all the consumers of that method in the system,
including the examples, to use the new location.

Which is fantastic, but what strategies should Ruby dudes use in the mean time?

I won’t speak for all, or say what one should use. But what I do is I
change things one step at a time with autotest running in the
background. When autotest tells me something failed, I look at it.
When it tells me nothing failed and I expected something to, I look at
that.

Whether or not you have tool support, refactoring is its own world of
study - taking steps that move you in the right direction and keep the
bar green as much as possible.

If I say that stories are authoritative and specs aren’t, then that
means I ought to rely on stories to provide the safety net for my
refactoring. When taking those baby steps, it’s important to keep the
stories green rather than the specs.

On some level that makes sense - but it’s going to slow you down quite
a bit when you start building up a significant suite of stories.

That was my main concern. You can get 1200 examples running in under
5 seconds. Probably can’t do that with stories.

Exactly

comfortable with. My main point is that if I have to go in and
manually change specs, then in general I don’t consider the specs to
be my comfortable safety net. They’ll get there when we have better
tool support, but I have to do stuff today.

Actually, if I change something and it causes an example to fail, that
suggests to me that the examples ARE a good safety net! Just as in the
TDD cycle, the failures actually give you confidence that you’re
touching the code you think you’re touching.

I suppose the real problem is that I can’t quantify what impact that
all has on my velocity. So I just end up focusing on the pain that it
causes me. But, I think getting rid of the pain is a very worthwhile
endeavor on its own.

You may just have to will it away. I find that this does not slow me
down. I’m slower on a step by step basis than I am with powerful
tools, but since I can solve the same problems with so much less code
in Ruby, it sort of balances itself out.

Let’s say in some banking software we have a transfer screen, and
there are three possible errors: insufficient funds, source account
frozen, target account frozen. Would you write a scenario for each of
the possible errors, and an example for each?

Without even batting an eye!

The scenario is going to reflect how things look from the outside. The
example is going to be targeted right at the part of the system that
is responsible for producing the correct error message.

This probably isn’t a
good example because I think you should :slight_smile: but it’s easy to see how
it would lead to more duplication than I’m comfortable with. Every
time you need to specify business value you have to stick it in a
story and an example.

This used to bother me too, but that was a long time ago. The thing is
that you want the examples to be comprehensive so you can refactor.
You also want the stories to be comprehensive, so you know what the
system does. And so you can refactor :wink:

I’m not completely comfortable with this yet in terms of being able to
talk about it, so suggestions are welcome. The problem with using Spec
or Example is that either can easily refer to the coarse grained
stories or the fine grained object level examples. Maybe we should
call them Coarse Grained Examples and Fine Grained Examples :slight_smile:
Seriously - I welcome other thoughts on this.

There’s another part missing, which is the set of all stories, and the
set of all example groups.

Anybody got any ideas?

Cheers,
David

6

On Nov 14, 2007 5:50 PM, Pat M. [email protected] wrote:

example is going to be targeted right at the part of the system that
Would you write a scenario where nothing is filled in? Where

Would you write a story at a lower level, which actually tries to save
the objects to the db and verifies that nothing changes? That
definitely duplicates what’s going on in the model-level specs.

Keep in mind that stories/scenarios and specs each play multiple roles.

Scenarios start off as a collaboration tool for the whole team to
understand what’s about to be developed. Then code is developed, the
scenarios pass, and now they serve two new purposes. Assuming they are
well maintained (they are kept well organized and passing), scenarios
document the system as it actually is and they are system-level
regression tests.

Similarly, examples start as a means of designing subsets of the
system at a very granular level. Then the become documentation of your
objects as they really are, and object-level regression tests.

Also consider that both levels have a red/green/refactor life cycle,
though less apparent at the higher levels. So they are both constantly
subject to change in response to new and changing requirements. But
they have subtly different forces of change. At a high level, they are
both changing because the business rules grow or change. But specs
change much more frequently as the design changes.

In the end, it’s about finding a process that works for you and your
team throughout the lifecycle of a project. For me, the duplication
between stories and specs helps me to stay productive all the time. It
is faith in combination of the two alerting me of behaviour-breaking
changes that keeps me moving.

If on your team you find it more pragmatic to express things in one
place or the other, and you find that doing so keeps you agile
(remember - agility is not only about speed - it’s about responding to
change) throughout the project, then that’s OK.

My experience tells me that I’m a happier person when I’ve got robust
sets of stories and specs that live independently and both cover the
entire system from their unique perspective.

FWIW.

Hopefully this thread hasn’t died…

No - it was just on vacation. Though I would welcome more voices in it
if anybody else has an opinion to express.

Cheers,
David