Best practices for sharing state between story steps?

Testing RSpec
1

Hi all,

I’m just getting into RSpec stories and liking them (especially with
webrat), but I’m finding it tricky to write steps that are
self-contained and reusable, particularly where features intersect.
What approaches do people use to achieve this?

(Maybe a better question is, do people bother? One of the things I like
about story-runner is the way I’m building a DSL for integration testing
my application, but should I just write the scenario I need, write the
steps to make it run and forget about reusing steps?)

As an example of where I’m having trouble, say I’m writing a blog (since
it’s the Web 2.0 version of Hello World), so I have posts and comments.
I want to write a scenario something like

Given a post
And some comments for the post
When I view the post
Then I should see the comments

How do I tell the “Given some comments” step which post to attach the
comments to?

I can do it by having “Given a post” set a @post instance variable and
having “Given some comments” use that, but it feels like global
variables all over again. My steps (probably in different files) are
coupled via the @post instance variable, and other steps can clobber it,
and if I forget to clear it I might pollute later steps, and if I want
to refer to more than one post (@post1, @post2) I have to rewrite all my
steps…

I noticed a little note on the Cucumber wiki
(GitHub: Let’s build from here · GitHub) explicitly
advising against using @variables for this, I’m guessing for these
reasons.

I could avoid the first issue by combining the top two steps into “Given
a post with some comments” but then I still have the problem of which
post “When I view the post” should GET. Maybe “When I view a post with
comments”…?

Any advice, criticism or sympathy appreciated :slight_smile:

Sam

2

This thread may help give you some insight…

http://www.mail-archive.com/[email protected]/msg05382.html

Zach

On Sun, Sep 7, 2008 at 7:58 PM, Sam S. [email protected] wrote:

steps to make it run and forget about reusing steps?)

steps…

Any advice, criticism or sympathy appreciated :slight_smile:

Sam

Posted via http://www.ruby-forum.com/.

rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

–
Zach D.
http://www.continuousthinking.com

3

On Sep 7, 2008, at 7:58 PM, Sam S. wrote:

What approaches do people use to achieve this?

Perhaps I’m bucking what others have advised against, so take it for
what it’s worth.

I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to “I”), @current_project (corresponds to
“the project”), etc. I am careful to be consistent. There’s only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to ‘states’ in my app, which might be in the
session or part of a nested URL.

With regard to reusable steps, I have some steps that are generic,
some are app specific, and some are (group of) story/feature specific:

  1. The generic ones are very reusable, for example

    When “I click the $link link” do |link|
    clicks_link link.strip_quotes
    end

    Then “the browser should show $a_or_an $tag tag with $contents” do
    |_, tag, contents|
    response.should have_tag(tag, contents.strip_quotes)
    end

  2. The app specific ones are intended to be reusable in any story but
    only my app, may do some database initialization (like fixtures),
    logging in, for example:

    Given “a $pname project with $settings”
    Given “I am logged in as a $role member”

Note, $settings is in the format
Given a Test project with foo: 1, bar: “bar value”, and baz: yadda
yadda

creates a default project overridden with the specified settings
(using to_hash_from_story, http://www.mail-archive.com/rspec-
[email protected]/msg05771.html )

  1. And the feature-set specific steps are not very reusable at all.
    As it turns out, there’s relatively few of these, and they tend to
    reflect aggregates of more granular steps already tested in a
    different story or scenario.

hope this helps,

linoj

4

Jonathan L. wrote:

I make some limited use of global (instance) variables that
correspond to english language pronouns. I have things like
@current_user (corresponds to “I”), @current_project (corresponds to
“the project”), etc. I am careful to be consistent. There’s only a
handful of these, but I find it extremely convenient. Note, these
also tend correspond to ‘states’ in my app, which might be in the
session or part of a nested URL.

So could I summarise this approach as, “use @variables, with good names
and consistent principles for use”?

One problem I’ve had is that some steps implicitly have to be preceded
by other steps, so that the @variables get set up right (e.g. “Given
some comments” refers to @post, which has to be set beforehand by “Given
a post”). I can make the wording of the steps more explicit, but then I
end up with clunky steps like “Given some existing comments by the user
for the post”.

Generally I’ve found with this sort of approach I end up writing fairly
imperative-style scenarios (as in
http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/),
and they’re quite brittle with respect to small changes to steps.

Do you encounter these problems at all?

5

Sam S. wrote:

and consistent principles for use"?

I’m of the opinion “do what works”. I know there are purists that say
don’t do this and don’t do
that, but when you come down to it you have to use variables between
steps, look at the rails
examples and it sets a response variable between steps.

What I do is horrible :slight_smile: but “it works”, I set GLOBAL variables (yup
$current_state), because I
found that sometimes @variable didn’t get setup properly in some cases.

Then I use the listeners to clear those variables between scenarios, so
every scenario I have a
listener (effectively before_scenario), that clears all the global
variables I use between steps,
this avoids errors where I don’t set up something in a step but the
scenario passes because it just
happens to have the right value in a global set by a previous test.

This does require some maintenance, however I could use a global hash
for my
inter-step-dependencies, and just $hash.clear in the before_scenario
listener.

Ahh I remember why I had to use $variable and not @variable, for some
reason the before_scenario
listener does not have access to the same scope as the scenario so I
have to use $variables.

So most of my scenarios look like this…

before_scenario do
$v1= nil
$v2= nil
end

Scenario “xxx” do
Given “something…”
When “I do something” # this will set $v1 and $v2
Then “check it was done” # this will check $v1 and $v2
end

I know purists will hate this :wink: But “it works for me”

BTW I use stories entirely for integration testing, testing the entire
stack by using the same
inputs (in my case a WEB based REST API) to the stack as a client would.

I also do nasty things like delve into the database directly in a Then
to check the database
actually got setup the way I expected by a previous When, as well as
checking the returned XML from
the Rest-API call.

As always YMMV

6

Zach D. wrote:

This thread may help give you some insight…

http://www.mail-archive.com/[email protected]/msg05382.html

Thanks for pointing me there. The first insight it’s given me is that
I’m actually asking two different (though related) questions:

  1. What’s the best way to build reusable steps, and is it a good idea?
  2. What’s the best way to share state between the steps in a single
    scenario?

That thread addresses 1) directly, and sounds like there’s some evidence
that reusable steps aren’t such a good idea. In which case 2) pretty
much drops out - if I don’t try so hard to reuse my steps then there are
fewer disadvantages to the obvious global @variable tactic.

If you take the view that reusable steps may be unhelpful, what do you
find is a good way to organise them? I can see an extreme case - each
plain-text foo.story gets its own steps defined in foo.rb, so there are
no cross-file @variable conventions to worry about. In this way foo.rb
is basically a clarification in code of foo.story, rather than part of
an app-specific language (ASL?) for integration testing. Does anyone do
this, and do they find it a useful way of looking at the problem?

7

On Tue, Sep 9, 2008 at 2:21 PM, Dan N. [email protected] wrote:

sensible way to share state across steps. You can think of the object the
steps run in as a “world” (it might even still be called that - it was at
one point) where you can set and verify state.

The debate seems to be whether step definitions should be stateful or
not.
In practice this is achieved by setting one or more @variables in a
step and reusing them in a different step - all within a scenario.

Both are fine, but beware that as your codebase grows and you have
hundred or so step definitions, things can become really hard to
maintain.
It won’t bite you when you start on a new codebase, but in my
experience stateful steps will bite you later.

Having experienced this pain in one project, I decided to try out
stateless steps exclusively on the next project. I’ve found stateless
steps to be a tad more verbose than stateful ones, both in the text
and in the implementation (because you have to pass “identifiers”
around). Still, the improved maintainability of my features and
scenarios using this approach outweighs this slight increase in
verbosity.

Here is an example of a stateless scenario: (As a convention I’m
“quoting” variables)

You can see the extra verbosity I’m talking about in the repetition of
“Aslak”, “Beerfest” etc.

I’m still pragmatic about this of course, but now you know some of my
experience.

Aslak

8

Hi Jim.

I guess I’m not a purist then - that looks fine to me, and it’s probably
something I would consider doing too.

The thing to bear in mind is that there is magic going on when you run
steps. Each step in a scenario is run in the context of the same object
instance (which you don’t get to see explicitly) by pixies, which means
any
@variables you set in one step should be visible to any other steps in
the
same scenario. I don’t think of this as global state - it’s simply a
sensible way to share state across steps. You can think of the object
the
steps run in as a “world” (it might even still be called that - it was
at
one point) where you can set and verify state.

There are bound to be pathological cases though, such as the
before_scenario
and after_scenario listeners. Perhaps there’s a bug there, in that they
should be running in the same object instance as the scenario steps
themselves. Perhaps not. In the latter case I would definitely go with
$globals to communicate state into your scenario, as long as you promise
never to use them in your application code. Never, you hear me?

Cheers,
Dan

2008/9/8 Jim M. [email protected]

9

On Tue, Sep 9, 2008 at 8:28 PM, Jim M. [email protected] wrote:

I wonder what exactly is meant by stateless steps? It seems to me that you
What I mean by a stateless step is a step that doesn’t assign or use
any @variables in its own context.

10

Dan N. wrote:

Hi Jim.

I guess I’m not a purist then - that looks fine to me, and it’s probably
something I would consider doing too.

I’d never call you a purist Dan :wink: But I do feel less dirty now,
although after reading Aslaks post
I wonder what exactly is meant by stateless steps? It seems to me that
you have to set some kind of
variable between steps to communicate what was done in one and what
needs to be checked in another.

Having been guilty of writing unit tests that were not stateless (ie one
unit test depended on the
result of a previous one, which BTW I really avoid doing), I did not
consider setting variables
between steps to be stateful.

scenario steps themselves. Perhaps not. In the latter case I would
definitely go with $globals to communicate state into your scenario, as
long as you promise never to use them in your application code. Never,
you hear me?

Hmmm well OK I’ll try to never use them :wink:

–
Jim M., http://blog.wolfman.com

11

aslak hellesoy wrote:

The debate seems to be whether step definitions should be stateful or not.
In practice this is achieved by setting one or more @variables in a
step and reusing them in a different step - all within a scenario.

I think that is the debate, but I’d like to point out that there is
always state between steps, it
is just a matter of where it is being kept. In your example for instance
most of the state is being
kept in the database between steps (ie between Given an “active”
site_user names “aslak” and he
following steps.

(As a side question how do you clean up the database between tests, so
the “state” from the previous
Scenario doesn’t affect the next Scenario?) In some cases I use a
randomly generated record each time.

Here is an example of a stateless scenario: (As a convention I’m
“quoting” variables)

That is quite interesting, however as I pointed out above there is state
between steps, in the
database and in the response.

I actually do something very similar, with named resources in each step,
however I find I still need
to use variables between steps, here is an example… (Not rails)

Scenario: testing a pet
Given I have a pet named “my dog”
When I stroke “my dog”
Then “my dog”'s health goes up

In the Given I create a pet named “my dog” in the database either
directly or through the REST-ful
API to my Web app, however in both cases I need the returned id of the
created pet to make any
further calls on it.

In the When clause I need to make an API call that pets “my dog”, the
API is POST stroke/345
So what I do is in the Given I assign the returned id of the new record
to $petid, then in the When
clause I do something like, $http_req.post(“stroke”, id => $petid).

Now I needed to carry that variable $petid around otherwise how would I
be able to make the call in
When.

Similarly when I want to check my pets health in the Then clause I still
need that $petid, whether I
check the value directly in the database or make an API call that
returns my pets health.

I don’t see anyway around keeping state in these cases.

Now I do clear that state ($petid= nil) in the before_scenario hook, so
another scenario which may
be written badly and not assign the $petid, won’t succeeed due to a
previously successful scenario.

I’d definitely be interested in better ways to do this though, as I hate
passing global variables
around (as I said in an earlier post I can’t use @variable because the
before_scenario does not seem
to have access to the same scope as the scenario that is about to run).
Although I think someone
said each scenario has its own variables? I didn’t notice that behavior.

12

On Tue, Sep 9, 2008 at 8:52 PM, Jim M. [email protected] wrote:

steps (ie between Given an “active” site_user names “aslak” and he following
steps.

There is persistent state (database) and object state (the object that
serves as a context for a scenario).

I’m not saying that state in and of itself is bad. However, coupling
is bad - for maintenance reasons. If the steps share object state (in
@variables) they become coupled.

So why is coupling bad for maintenance? When your step library grows
to several dozens or like in my case, over a hundred, you want to be
able to cherry-pick steps to build new scenarios. If you have coupled
them with object state, you can’t do that. These steps will simply not
work unless they’re used alongside the steps they are @coupled to.

That’s when you start to add new step definitions with a similar
semantic and wording, but with a different implementation - to work
around the other steps that you realised you can’t reuse. Now things
get confusing. You have Given /there is a contract named (.)/ and a
Given /there is an existing contract named (.)/ etc. etc.

(As a side question how do you clean up the database between tests, so the
“state” from the previous Scenario doesn’t affect the next Scenario?) In
some cases I use a randomly generated record each time.

Cucumber starts a transaction when a scenario starts and rolls it back
when it ends (when on Rails).

Here is an example of a stateless scenario: (As a convention I’m
“quoting” variables)

That is quite interesting, however as I pointed out above there is state
between steps, in the database and in the response.

That’s not the coupling kind of state I’m talking about. That’s ok.

or through the REST-ful API to my Web app, however in both cases I need the

I’d definitely be interested in better ways to do this though, as I hate
passing global variables around (as I said in an earlier post I can’t use
@variable because the before_scenario does not seem to have access to the
same scope as the scenario that is about to run). Although I think someone
said each scenario has its own variables? I didn’t notice that behavior.

In Cucumber a Before block has access to the same scope as the steps.

13

On 9 Sep 2008, at 19:52, Jim M. wrote:

kept in the database between steps (ie between Given an “active”
site_user names “aslak” and he following steps.

(As a side question how do you clean up the database between tests,
so the “state” from the previous Scenario doesn’t affect the next
Scenario?) In some cases I use a randomly generated record each time.

I think the scenarios are each wrapped in a transaction, so (as long
as you’re using the right type of database / tables) the slate should
be wiped clean between each one.

In the When clause I need to make an API call that pets “my dog”,
database or make an API call that returns my pets health.
have access to the same scope as the scenario that is about to
run). Although I think someone said each scenario has its own
variables? I didn’t notice that behavior.

I’ve been thinking about this a bit, after reading Aslak’s advice
which went contrary to my instincts. It seems like you could probably
do something like this instead:

Scenario: testing a pet
Given I have a pet
When I stroke the pet
Then the pet’s health goes up

Because your Given step creates just one Pet instance, you can safely
refer to it as “the pet” in the other steps, and simply call Pet.find
(:first) in the step matcher.

See?

In a more complicated scenario:

Scenario: testing a pet
Given I have two pets named “my dog” and “my cat”
When I stroke “my dog”
Then “my cat” should be pissed off

Now your step matcher needs to refer to a particular pet, but can’t
it just use the name as a key, since that will be unique within the
set of two Pet records you created in your Given step?

When /I stroke “(.*)”/ |pet_name| do
Pet.find_by_name(pet_name).stroke
end

Does that make sense to you?

cheers,
Matt

http://blog.mattwynne.net

In case you wondered: The opinions expressed in this email are my own
and do not necessarily reflect the views of any former, current or
future employers of mine.

14

Thanks for the hints, but I think I did point out I am not using Rails,
I write Integration tests,
that talk directly to the web application via HTTP running the full
stack.

So Active record is not available neither are transactions for the
database.

Sorry for the confusion.

If I were using Rails I would undoubtedly do it the way you suggest :slight_smile:

15

On Tue, Sep 9, 2008 at 3:52 PM, Matt W. [email protected] wrote:

I think that is the debate, but I’d like to point out that there is always
you’re using the right type of database / tables) the slate should be wiped
clean between each one.

I don’t think this works as you may expect since My
SQL and PostgreSQL don’t support nested transactions, which is what
would happen if your test environment wrapped stories/scenarios in
transactions and your application utilized transactions.

Granted there has been hopes of getting savepoints fix the problem,
but the ticket has not been resolved:

http://rails.lighthouseapp.com/projects/8994/tickets/383-activerecord-should-use-savepoints-for-nested-transactions

–
Zach D.
http://www.continuousthinking.com

16

aslak hellesoy wrote:

them with object state, you can’t do that. These steps will simply not
work unless they’re used alongside the steps they are @coupled to.

I agree and have run into this problem, but I have not found a better
way in the environment I am using.

Cucumber starts a transaction when a scenario starts and rolls it back
when it ends (when on Rails).

Nice if you are using Rails, which I am not :slight_smile: I was curious if anyone
had run into the same
problems I do, because I don’t use rails, I test directly against the
HTTP API, so the database gets
all this test data, you can’t use transactions, so the database
potentially has this state from
previous Scenarios.

Right now in my Before_Scenario I generally try to clear the database,
but that gets hard when it is
relational with a ton of foreign key constraints, and you can’t simply
delete or truncate a table.
As I said one work around is I create randomly named records every run,
so as to avoid collision
with previous data. I wonder if anyone has a better idea?

BTW my Web service is written in Java/Jetty/Spring so I have no
Rails/Ruby niceties on that side, so
it is nice to be able to use Ruby in the integration tests.

That is quite interesting, however as I pointed out above there is state
between steps, in the database and in the response.

That’s not the coupling kind of state I’m talking about. That’s ok.

Ok I think I see the difference, by the state being in the Database or
the response variable it does
not implicitly couple steps like an @variable would. But it still
requires the database to have been
setup in a certain state by certain preceding Givens or Whens, which is
a kind of coupling. But
maybe I am stretching too far on that argument? I still feel coupling is
coupling and state is state :slight_smile:

In Cucumber a Before block has access to the same scope as the steps.

Good that’ll make the transition easier I hope.

17

Thanks everyone for your responses - this is a really helpful discussion
and every post has added to my understanding. Thanks also for
disagreeing with each other, as it makes me feel less stupid for asking
the question :slight_smile:

Jim M. wrote:

Ok I think I see the difference, by the state being in the Database or
the response variable it does
not implicitly couple steps like an @variable would. But it still
requires the database to have been
setup in a certain state by certain preceding Givens or Whens, which is
a kind of coupling. But
maybe I am stretching too far on that argument? I still feel coupling is
coupling and state is state :slight_smile:

It seems to me that with

“Given an existing post ‘lol internet’” # creates a post with that title
“Then the page should include ‘lol internet’” # assumes a post with that
title

the assumptions are in line with what you’d naturally understand from
the English, whereas

“Given an existing post ‘lol internet’” # creates and stores in
@the_post
“Then I should see the post title” # verifies @the_post matches what’s
in the db

seems to make more assumptions than are obvious from the English.

In fact, couldn’t one argue that the second example is redundantly
storing the post in two places (the database and @the_post)? I think
this is what Aslak means by distinguishing “stateful” and “stateless” -
@variables are extra state, on top of the application state which must
change either way or you’re not testing anything.

18

Sam S. wrote:

@the_post
“Then I should see the post title” # verifies @the_post matches what’s
in the db

seems to make more assumptions than are obvious from the English.

In fact, couldn’t one argue that the second example is redundantly
storing the post in two places (the database and @the_post)? I think
this is what Aslak means by distinguishing “stateful” and “stateless” -
@variables are extra state, on top of the application state which must
change either way or you’re not testing anything.

Thats true in that case. I also wouldn’t do what you show in the second
case. If I can get the
result from the API or the database I would.

In my case I store data between states that I can’t get anywhere else,
or that tells me where to get
the results from.

19

On Tue, Sep 9, 2008 at 11:31 PM, Jim M. [email protected] wrote:

to several dozens or like in my case, over a hundred, you want to be

Nice if you are using Rails, which I am not :slight_smile: I was curious if anyone had
run into the same problems I do, because I don’t use rails, I test directly
against the HTTP API, so the database gets all this test data, you can’t use
transactions, so the database potentially has this state from previous
Scenarios.

Sorry for not paying attention to this earlier.

I’ve done that on several projects (using selenium/watir mostly). Some
things you could try:

  1. HTTP database endpoint
    A couple of years ago I was using Selenium against a Java web app. We
    implemented a “special” HTTP endpoint for testing only that we would
    access before a test. It would put the database in a pristine state.
    It’s clunky, but it works.

  2. Ruby database endpoint
    Another way to do it is to access the database directly from Ruby
    before the test runs and clear it there.

  3. Turn off constraints
    Approaches 1 and 2 may be hard to do if you have complex constraints.
    One strategy is to turn off constraints in your test database and just
    do brute force deletes along with 1 or 2.

  4. Incremental, semi-random data
    This is the approach you are describing. The problem with this is risk
    of false positives and intermittent failures. How risky it is depends
    on your app of course.

  5. Nested transactions
    Using 1 or 2, start a transaction in the beginning of a scenario and
    roll it back at the end. Only works if your database supports nested
    transactions.

I don’t know if there is any good literature or discussion groups
about these topics, but there ought to be.

Aslak

20

On 10.9.2008, at 0.29, Zach D. wrote:

I don’t think this works as you may expect since My
SQL and PostgreSQL don’t support nested transactions, which is what
would happen if your test environment wrapped stories/scenarios in
transactions and your application utilized transactions.

I’ve been bitten by this a couple of times. A common idiom in the
Rails world is (was at least) to use save! to raise an exception when
an object couldn’t be saved. In a rescue block you could then do the
things you wanted to do when saving failed. Raising the exception also
rolled back a transaction, so it was a nice way to ensure some
transactional logic:

MyModel.transaction do
MyModel.save!
OtherModel.save!
end

If either line in the transaction failed, it was ensured that neither
of them was persisted in the db. This helped in many cases where you
wanted either all or none of your updates to get into the db. However,
with stories these cases always fail, because of the reason Zach
mentions.

The question whether a case where someone posts an empty email address
is an exceptional state in strict PragProg sense might be a bit
questionable, of course, but it’s beyond the point here…

//jarkko

–
Jarkko L.

http://www.railsecommerce.com
http://odesign.fi