[Cucumber, BDD] When not to use Cucumber?

Testing RSpec
1

Hi all,
Just wanted to get some feedback and opinions on something… I have
been using Cucumber/Story Runner since plaintext support was introduced.
(So, since October/November of 2007.) Throughout this time I have used
it on various projects. Some projects had customer involvement in help
writing the scenarios, some I mostly wrote the scenarios after gathering
requirements and then got verification from the customer, and others
involved no customer at all (i.e. I was the customer). In all three
cases I have gotten a lot of benefit from the upfront analysis that
writing the feature forces you to do and the outside-in development that
follows. That said, the business analysis and outside-in development
can be accomplished with any tool (i.e. rspec with webrat for webapps).
In the past I have always defaulted on using the GWT (Given, When, Then)
syntax for most of my acceptance tests, but that may be because I am so
accustomed to that workflow by now.

So… I’m curious what people’s thoughts are. When writing acceptance
tests how do you choose which tool is best for the job? I often hear
people complaining about the GWT syntax and they see no benefit of it
over the frameworks that provide contexts (i.e rspec, shoulda, etc).[1]
In what cases, if any, do you think the GWT gets in the way and how? In
particular, if you are writing an app or lib whose customers/users are
all developers do you still think there is value in the GWT syntax that
Cucumber provides?

I have my own thoughts on the matter, but in playing devil’s advocate I
want to get other people’s thoughts before I skew you with mine. :slight_smile:

Thanks,
Ben

[1] The Practicality of the RSpec Story Runner - Dan Manges's Blog

2

Ben M. wrote:

So… I’m curious what people’s thoughts are. When writing acceptance
tests how do you choose which tool is best for the job? I often hear
people complaining about the GWT syntax and they see no benefit of it
over the frameworks that provide contexts (i.e rspec, shoulda, etc).[1]
In what cases, if any, do you think the GWT gets in the way and how? In
particular, if you are writing an app or lib whose customers/users are
all developers do you still think there is value in the GWT syntax that
Cucumber provides?

My own experience is quite limited, but I did start out trying to get
what I desired done through TestUnit, then RSpec, then RSpec Story
Runner and finally Cucumber. In the end it was Cucumber that clarified
my approach for me, mainly by clearing away all (or at least most) of
the nitty gritty details of configuring and using the tool itself. I
found that, coming from a non OO programming background, that the degree
of mastery required of the other tools mentioned, together with the
effort I had to make to get the tool to behave as I wished (desires that
were often malformed by lack of understanding in any case), distracted
considerably from the task at hand, programming the application.

I have progressed on my current project far, far more since discovering
Cucumber this past November than in all the time before. The
bifurcation between what and how seems far more natural to me with the
Cucumber features/step definitions file split than with the unified
expectation method/test steps of RSpec and the test method/assert
statements of TestUnit. Cucumber feature files just seem to permit one
to concentrate solely on “what” in the one case and then, with the
separate step definitions file, deal with the “how” in the other,
without crossing the line between the two.

Given, When and Then seemed contrived and far too stiff to me to begin
with. However, I discovered that much of my initial discomfort arose
from too vague a conception of what is was that was desired. The
benefit of GWT stylistic conventions is that it really does make one
consider deeply where one is starting from, what one is actually trying
to accomplish, and how, exactly, does one tell if it has been
accomplished.

If this sort of self-analysis is so ingrained that one need not think of
it then GWT probably does not provide a measurable benefit. So, perhaps
deeply experienced developers and programmers might not get as much from
it as I have.

On the other hand, I have frequently discovered in myself a lamentable
tendency to believe that I understood what a problem was simply because
I wanted to believe and not because of any demonstrable evidence to that
effect. Sitting down and writing out what you believe to be the truth
often reveals the flaws in ones assumptions. The GWT litany seems to
provide a very useful, if formulaic, examination of exactly what one is
setting out to accomplish thereby sometimes raising the question of why
it needs to be done at all. I find that it thereby clears away much
that is irrelevant to the question at hand.

I also hold to the opinion that when writing an end user application the
only testing that can be consider definitive is what is described as
integration or acceptance testing. I accept that there are pragmatic
reasons why, because of limited time and equipment resources, unit and
functional tests are viewed as attractive alternatives. But, that
should not blind one the the fact that they really are alternatives,
chosen because of environmental considerations and not because of
inherent superiority. The only test that means anything to an end user
is one that shows the application works as intended in the environment
it is deployed to.

Libraries and utilities have a different audience and their testing is
perhaps better handled solely in unit and functional tests, but I cannot
say so with any conviction.

3

My own experience is quite limited, […]

Hi James, are you saying that you only use Cucumber and not RSpec at
all?

4

On 23/01/2009, at 12:04 PM, Ben M. wrote:

So… I’m curious what people’s thoughts are. When writing
acceptance tests how do you choose which tool is best for the job?
I often hear people complaining about the GWT syntax and they see no
benefit of it over the frameworks that provide contexts (i.e rspec,
shoulda, etc).[1] In what cases, if any, do you think the GWT gets
in the way and how? In particular, if you are writing an app or lib
whose customers/users are all developers do you still think there is
value in the GWT syntax that Cucumber provides?

I’ve been using RSpec since July, and love it. For the last few
months, I’ve been reading most of the Cucumber posts on this list.
Last week, I thought I’d give Cucumber a try. It feels like a really
fantastic tool for determining business requirements, functionality,
etc. From what I can tell though, there seems to be a lot of overlap
between RSpec specs, and Cucumber features and stories. For that
reason, I haven’t delved much further into Cucumber, because I don’t
want to be repeating myself, and having to update two sets of files
whenever I make a change.

However, if that’s not the case, please correct me!
Nick

5

I’ve been using story runner for the first time over the past 4 months
or so
and I’ve found the invaluable part to be the plain text support. Having
a
shared artifact that can be used by the customer, developers and testers
is
a huge boost towards achieving ubiquitous language. We use the story
kickoff
discussion to refine the language used in each scenario together,
ensuring
that the language makes sense to everyone and is consistent with other
stories.

In the sense that we want a way to store the feature/story in an
executable
format that we can all understand, a tool that supports plain text
stories
is the best choice for us. You can implement a scenario just as easily
using
plain rspec but then it becomes less accessible to other people in the
team.

If I was choosing a testing approach for a project that had technical
people
as customers I would still use the GWT format. I think it helps you
continually focus on what you are really trying to achieve, otherwise it
can
be too easy to burrow down into the technical details and lose sight of
that. No matter what the nature of project, every thing that you do is
driven by the goal of adding some sort of value. Continually keeping
that
goal in mind helps you make better choices along the way.

I agree with James that having that separation between the “what” and
“how”
is very useful. This approach is amenable to a more black box style of
testing appropriate for acceptance tests. You should theoretically be
able
to completely change how your system works without touching the
story/feature text. Your mileage would of course vary with this
depending on
whether or not you use a more declarative or imperative approach (as Ben
describes in
http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/
).

My current recipe for testing spread is to automate the acceptance
scenarios
using a tool like story runner/cucumber and then use specs for targeted
integration and unit level testing. The automated acceptance scenarios
cover
the main scenarios that the customer cares about while more specialised
cases (or more technical aspects) are covered at the spec level. The
specs
will typically gravitate towards the white box end of the scale
(describing
the internals in more detail).

  • Brent
6

Fernando P. wrote:

My own experience is quite limited, […]

Hi James, are you saying that you only use Cucumber and not RSpec at
all?

No, I do use RSpec and TestUnit for that matter, but not to any extent
at the present time. I find that cucumber features are much more
conducive to driving out the design than RSpec expectations. Once the
essence of the application is laid out and working then, if necessary, I
will consider adding more technical type expectations via RSpec.

Frankly, at the moment, I do not see a compelling case to move from step
definitions into formal RSpec expectations. Again, my experience in
these matters is limited and perhaps as it grows my appreciations will
change as well.

I do note however, that once one finds a tool that suits ones needs then
it is somewhat inefficient to use another without some compelling
advantage to be gained. What advantage does formal RSpec constructions
provide over cucumber step definitions using RSpec and RSpec-Rails
methods?

7

See comments inline…

On 1/23/09 1:46 PM, James B. wrote:

Cucumber provides?
effort I had to make to get the tool to behave as I wished (desires that
were often malformed by lack of understanding in any case), distracted
considerably from the task at hand, programming the application.

Interesting, thanks for sharing what your background is. I have seen
other developers who don’t have a strong OO background struggle with the
concepts of isolation have a really hard time understanding how and why
to use rspec. However, they seem to love Cucumber and it makes total
sense to them.

I have progressed on my current project far, far more since discovering
Cucumber this past November than in all the time before. The
bifurcation between what and how seems far more natural to me with the
Cucumber features/step definitions file split than with the unified
expectation method/test steps of RSpec and the test method/assert
statements of TestUnit. Cucumber feature files just seem to permit one
to concentrate solely on “what” in the one case and then, with the
separate step definitions file, deal with the “how” in the other,
without crossing the line between the two.

Ahh… that is a very good observation. When I write examples in rspec I
start out with the example description first (obviously) but then I like
to jump straight to my expectation. Meaning, I will say something like
“dog.should be_happy” before I have even created the dog object. I like
to do this for the same reason- I first write what I want and then I go
a line above and create the how/given. A lot of people complain about
the separation of code form story in plaintext features, but as you have
pointed out that separation can be a very good thing.

deeply experienced developers and programmers might not get as much from
that is irrelevant to the question at hand.

+1. I totally agree with this. When it is difficult for me to write
scenarios I find that the reason is that I fail to really understand
what I am trying to accomplish. By flushing the acceptance criteria out
in the GWT language I come out with a much better understanding of the
problem and know exactly where to begin.

So here is where I disagree with you. :slight_smile: I do agree that all the user
cares about in the end is that the desired outcome is reached and the
business value is provided. So with that mindset features are the only
kind of tests a customer really cares about. Unit tests however are
more for the developer. They aid in design and provide much more
detailed documentation than the customer testes (acceptance tests). I
don’t think that cucumber features help me design my objects that much
better unless I have focused object level examples telling me where the
pain points are in my design. Eventually the customer will be greatful
that you have unit tests (even if they don’t know about them) because
they will help lower the maintenace of the project by encouraging better
design and providing tighter and more focused feedback on the smaller
parts of the system. IMHO, of course.

Thanks for the feedback,
Ben

8

On 1/23/09 10:04 PM, Brent Snook wrote:

snip

If I was choosing a testing approach for a project that had technical
people as customers I would still use the GWT format. I think it helps
you continually focus on what you are really trying to achieve,
otherwise it can be too easy to burrow down into the technical details
and lose sight of that. No matter what the nature of project, every
thing that you do is driven by the goal of adding some sort of value.
Continually keeping that goal in mind helps you make better choices
along the way.

This is at the herat of what I was asking. Even if every “customer” of
the project is fluent in ruby would you still bother with the plaintext
GWT features of Cucumber? I think you have nailed it… Or at least,
have confirmed my thoughts on the matter. :slight_smile: Business value is business
value no matter who is the customer. While other tools can be used to
automate acceptance criteria Cucumber provides the huge advantage of
helping you focus and analyze on what the desired business value is. I
have talked to other people who have claimed that the GWT gets in the
way though, so YMMV. So, I wouldn’t be surprised that in some technical
domains the overhead of adding features just doesn’t make sense.

Thanks for your thoughts,
Ben

9

Brent Snook wrote:

the internals in more detail).

Hi, I think this is the most used approach nowadays, high level testing
with
features and more specific testing, that end users don’t care about at
the
spec level. However, I prefer the distinction between, declarative
stories
and imperative stories, which I think can be applied to these two levels
also.

For example, the end user or the business person, most probably gets
bored
of reading a very long imperative story of clicking links and filling
out
forms, plus other setup steps that might seem unnecessary to him, but
are
needed sometimes due to a not fully correct design. Whilst a developer
will
appreciate an imperative story as it tells him/her exactly what to do.

In my opinion, even at the spec level we can use the philosophy behind
steps
(re-usability) to reduce the redundancies inherent to spec definitions.
Trivial cases of you should see this in the page (view specs), or this
object should have this attribute set to true (model specs), happen all
the
time, and seem like a great place to use variables inside step
definitions.

Other cases are more complex and the use of regular expressions to
encapsulate them all, can certainly be very difficult, maybe the type of
an
attribute is a datetime and needs to be parsed, thus not being
completely
re-usable to say a boolean field.

Still we could account for this variations by checking the type of the
attribute, at the expense of increasing the complexity of the step.
This is
where the argument continues, people support the view of writing
repetitive
specs considering that they are clearer. I still have doubts about
this.
In my opinion this will certainly be a complex problem to solve, but
once we
tackle it a number of times, it will become trivial, we’ll figure out an
elegant step_helper to deal with this situations, and move on to the
next
hard problem.

The hardest problem seems to be agreeing on a language to write the
stories
including minimal variations in the syntax and grammar of our steps, to
be
able to come up with a re-usable set of step definitions out-of-the-box,
a
set of steps that will allow us to write stories and jump straight to
the
failing nature of a step, due to it not being implemented, once its
implemented the step will be passing, without further need to write
trivial
step definitions. This would be a great time saver and allow us to
concentrate on harder problems very specific to our application that are
not
accounted for in the general step definitions.

In conclusion, when not to use cucumber? never. Always use cucumber,
everyone always use cucumber, lets find this hard problems to re-use
steps
and tackle them together, find a solution and move on to the next
road-blocker. Use Rspec but as the underlying language to make cucumber
work, not to divide your application into model view controller specs,
separate your tests into user and developer features/steps.
Distributed testing is a must to use cucumber effectively, and the use
of
mocks is also unrefutable[1], but only as an alternative to an inability
to
test something in a classical way[2]

Rai

[1]http://rubyconf2008.confreaks.com/testing-heresies.html
[2]Mocks Aren't Stubs

–
View this message in context:
http://www.nabble.com/-Cucumber%2C-BDD--When-not-to-use-Cucumber--tp21628693p21639914.html
Sent from the rspec-users mailing list archive at Nabble.com.

10

On 1/24/09 5:02 AM, voodoorai2000 wrote:

Rai

[1]http://rubyconf2008.confreaks.com/testing-heresies.html
[2]Mocks Aren't Stubs

Well, I have learned never to say never. :wink: As far as the mocking
goes… I always use real objects in features unless I need to make an
external network call to another system (i.e. a web services call, in
which case I like to use fakeweb[1]). That said, one of the things I
really like about having features is that it allows me to use mock
objects in my object level examples without any fear of integration
point failures. I find it very hard to test objects under total
isolation without using mocks to remove dependencies and I gain a lot of
design benefit from using mocks. I really don’t want to get into the
whole classic vs mockist debate though as it was just beat to death on
the TDD mailing list.[2] :slight_smile:

-Ben

[1] GitHub - chrisk/fakeweb: Ruby test helper for injecting fake responses to web requests
[2]
http://tech.groups.yahoo.com/group/testdrivendevelopment/message/29674

11

Hi,

I’ve been using cucumber, and story runner before that since plain
text stories were introduced 1.5 years ago. I also use rspec for
model and controller specing/testing. I’ll answer the opposite
question: when do I use cucumber?

I am a team of one, and I am my own client, so the way I use cucumber
may be different than other people’s experience. As outlined in my
recent presentation (http://www.slideshare.net/linoj/cucumber-how-i-
slice-it-presentation-924254 ) I’ve found myself using cucumber for:

  • Talking to myself (planning)
  • Keeping track of what I’m working on (outlining)
  • Automating my manual testing (scripting)
  • View specs
  • Regression testing (confidence)

Plain text stories let me plan out my app from the end-user point of
view. When I’m my own client, Given/When/Then helps me structure
conversations with myself. Scenarios let me explore variations.
However, I do not shoot for full coverage or many edge cases, I leave
that for the controller and model specs. These tend to be short
atomic G/W/T scenarios.

When taking the BDD approach (which TBH is sometimes), it can take
days to implement a single scenario, drilling down from feature to
controller spec to model spec, etc, at which point the story provides
a valuable context for what to work on next as i “pop the to-do
stack” and work my way through the steps.

However, in reality, sometimes I jump in and implement stuff first,
and test it as I go by clicking in the browser (oh my!). At a certain
point I’m repeating the same set of clicks so I write a cucumber
story to more easily script the scenario. In this case there may be a
high level Given step(s), and a sequence of multiple When’s, with a
few Then’s inbetween (to validate where I landed, flash messages, or
whatever).

I’ve also started using cucumber in place of view specs. These
scenarios will often be a simple Give/When with a whole bunch of
Then’s that validate elements on the page.

By the way, I’ve written utility steps for viewing the response in a
browser (http://www.vaporbase.com/postings/Show_me_the_response ),
and I often use rdebug to work through my code.

Finally, I love to run my suite of scenarios as regression tests,
especially at stabilization points in my development, to make sure
everything is still operational. I even have a “cap deploy:features”
to run them on my staging server.

linoj