Variance in behaviour: rake features vs. autotest

$ rake features

56 steps passed

$ set
AUTOFEATURE=true
BASH=/bin/bash

$ autotest

script/cucumber features --format progress --format autotest --out
/tmp/autotest-cucumber.1527.0
FFF____F__F__F__F__F_P_F_P_FP_P_F_P_P_F_P_P_F_P_P_F_P_P_

Pending Scenarios:

  1. Identify and record legal entities associated with a transaction
    (Attempt to ADD a VALID entity via a web form)

Failed:

uninitialized constant Location
./features/locations/step_definitions/location_steps.rb:7:in Given /should not already have a location/' features/locations/location.feature:12:inGiven I should not already
have a location’

Can anyone tell me what I am doing wrong with my setup so that rake
features pass all tests but autotest does not?

James B. wrote:

$ autotest

script/cucumber features --format progress --format autotest --out
/tmp/autotest-cucumber.1527.0
FFF____F__F__F__F__F_P_F_P_FP_P_F_P_P_F_P_P_F_P_P_F_P_P_

On closer inspection, this looks very much like the problem that I have
with running cucumber from the cli.

James B. wrote:

James B. wrote:

$ autotest

script/cucumber features --format progress --format autotest --out
/tmp/autotest-cucumber.1527.0
FFF____F__F__F__F__F_P_F_P_FP_P_F_P_P_F_P_P_F_P_P_F_P_P_

On closer inspection, this looks very much like the problem that I have
with running cucumber from the cli.

But while $ cucumber features -r features corrects the cli problem I do
not know how to incorporate this knowledge into the autotest setup. Any
ideas?

James B. wrote:

On closer inspection, this looks very much like the problem that I have
with running cucumber from the cli.

But while $ cucumber features -r features corrects the cli problem I do
not know how to incorporate this knowledge into the autotest setup. Any
ideas?

James,
As the cucumber wiki page says about autotest
(GitHub: Let’s build from here · GitHub) if
you want to override the arguments used by autotest you need to define
an ‘autotest’ profile in your cucumber.yml file. I added information
about profiles here:
GitHub: Let’s build from here · GitHub.

So, in your case you would simply need to create a cucumber.yml file at
the root of your project with this as the contents:

default: -r features
autotest: -r features

(I added the default so that when you use the cli it will automatically
use those args too.)

To answer your original question, you must specify the requiring of the
features dir because cucumber, by default, only loads the ruby files in
the feature’s dir and children dirs. So when you have a feature that is
in a subdirectory of your main features dir you have to explicitly
require the ruby files under features. Make sense?

-Ben

Ben M. wrote:

As the cucumber wiki page says about autotest
(GitHub: Let’s build from here · GitHub) if
you want to override the arguments used by autotest you need to define
an ‘autotest’ profile in your cucumber.yml file. I added information
about profiles here:
GitHub: Let’s build from here · GitHub.

So, in your case you would simply need to create a cucumber.yml file at
the root of your project with this as the contents:

default: -r features
autotest: -r features

(I added the default so that when you use the cli it will automatically
use those args too.)

To answer your original question, you must specify the requiring of the
features dir because cucumber, by default, only loads the ruby files in
the feature’s dir and children dirs. So when you have a feature that is
in a subdirectory of your main features dir you have to explicitly
require the ruby files under features. Make sense?

-Ben

I follow part of this logic, but not all. Given a structure like this:

features
|-- entities
| |-- entity.feature
| -- step_definitions | – entity_steps.rb
|-- locations
| |-- location.feature
| -- step_definitions | – location_steps.rb
|-- sites
| -- step_definitions |-- step_definitions | – webrat_steps.rb
-- support – env.rb

I would expect that “$ cucumber features” would load all rb files in and
under features abd pre-load the required files before processing the
feature files. If what you are saying is that this command simply sets
up a queue, explicit or implicit, to process .feature files in order of
occurrence whereas the “-r features” argument tells cucumber to
pre-process the .rb files and load all required libraries first then,
yes, your explanation makes sense. Given that it is possible to run
cucumber against a single feature file this is perhaps the only
reasonable way of approaching the problem.

I appreciate your advice with respect to autotest settings. Unhappily,
I had already tried several variants of what you proposed and they all
failed. For example:

$ rake features

56 steps passed
$
$ # empty cucumber.yml file
$
$ cat cucumber.yml
$
$ cucumber features -r features

56 steps passed
$
$ # cucumber default set
$
$ cat cucumber.yml
default: -r features
$
$ cucumber features

14 steps failed
30 steps skipped
12 steps pending
$
$ let us try autotest
$
$ cat cucumber.yml
autotest: -r features
$
$ cat .autotest
require ‘autotest/redgreen’

module Autotest::GnomeNotify
… code to generate popup notice to gnome desktop
Autotest.add_hook :green do |at|
notify SUCCESS_STOCK_ICON, “All Tests Passed.”, “”
end

end
$
$ autotest
loading autotest/cucumber_rails
/usr/bin/ruby -I.:lib:test -rtest/unit -e “%w[test/unit/vendor_test.rb
test/unit/notification_test.rb test/unit/client_test.rb
test/unit/location_test.rb
test/functional/entity_client_controller_test.rb test/unit/site_test.rb
test/functional/clients_controller_test.rb
test/unit/entity_test.rb].each { |f| require f }” | unit_diff -u
Loaded suite -e
Started

Finished in 1.85908 seconds.

16 tests, 22 assertions, 0 failures, 0 errors

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.6981.0

… some time later…
Interrupt a second time to quit
$
$ # Hmmm. No feature test ran. Let us try this instead.
$ autotest features
loading autotest/cucumber_rails
/usr/bin/ruby -I.:lib:test -rtest/unit -e “%w[test/unit/site_test.rb
test/unit/client_test.rb test/functional/clients_controller_test.rb
test/unit/location_test.rb
test/functional/entity_client_controller_test.rb
test/unit/notification_test.rb test/unit/vendor_test.rb
test/unit/entity_test.rb].each { |f| require f }” | unit_diff -u
Loaded suite -e
Started

Finished in 1.666869 seconds.

16 tests, 22 assertions, 0 failures, 0 errors

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.6990.0

Nothing happening. No cucumber tests. Lets touch a feature file

from another terminal session

touch features/entity/entity.feature

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.6990.1

Ahhh. Something happened. Let us see what is in the output file

cat tmp/autotest-cucumber.6990.1

Nothing.

Interrupt a second time to quit

$ # Let us try something else.
$ cat cucumber.yml
autotest: features -r features
$
$ autotest
loading autotest/cucumber_rails
/usr/bin/ruby -I.:lib:test -rtest/unit -e “%w[test/unit/client_test.rb
test/functional/entity_client_controller_test.rb
test/unit/vendor_test.rb test/functional/clients_controller_test.rb
test/unit/site_test.rb test/unit/entity_test.rb
test/unit/notification_test.rb test/unit/location_test.rb].each { |f|
require f }” | unit_diff -u
Loaded suite -e
Started

Finished in 1.674592 seconds.

16 tests, 22 assertions, 0 failures, 0 errors

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.7043.0
Feature: Identify and record the location of parties to transactions #
features/locations/location.feature

35 steps passed
5 steps failed
16 steps skipped

what happens when we touch a feature file from another session?

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.7043.1 -s ‘Test if Fixtures are loaded first’ -s
‘The common name should display with initial capitals’
Feature: Identify and record legal entities associated with a
transaction # features/entities/entity.feature

4 steps passed
5 steps failed
16 steps skipped

So, depending upon the way cucumber is invoked, either all the tests
pass, all the tests are skipped, or some of the tests pass and some fail
one way and others pass and fail when invoked another way. This seems
problematic for testing and it is far beyond my modest abilities to
explain or rectify.

Regards,

James B. wrote:

So, depending upon the way cucumber is invoked, either all the tests
pass, all the tests are skipped, or some of the tests pass and some fail
one way and others pass and fail when invoked another way. This seems
problematic for testing and it is far beyond my modest abilities to
explain or rectify.

Well, I have localized the source of the error and I have resolved one
of the causes.

One of the feature tests was to assure that the test table was empty to
begin with. This was passing under cucumber and rake features while
failing under autotest. It failed because the testunit tests were
loading a fixture into that table. This I resolved by changing the
feature step to destroy_all that table.

The four remaining failures are all located in a test that uses a “More
Examples” construct. So, in fact, there is just one failing test that
is hit four times. This test follows:

Feature

Scenario: The common name should display with initial capitals
Given one valid entity
And I am on the edit entity page
When I fill in “Common Name” with " ANYThing WronG wITh tHiS? "
And I commit the update
Then I should see “Anything Wrong With This?”
And I should see an update success confirmation

More Examples:

Steps


def build_valid_entities(n=1)
Entity.transaction do
Entity.destroy_all
n.to_i.times do |n|
Entity.create! :entity_name => “Entity #{n}”,
:entity_legal_name => “Entity #{n} Legal Name”,
:entity_legal_form => “CORP”
end
end
end

Given /(\d+) valid entit/ do |n|
build_valid_entities(n)
end

Given /one valid entity/ do
build_valid_entities(1)
end

Given /I am on the edit entity page/ do
visits “/entities/1/edit”
end

The error is that there exists no entity with id=1. Ideas?

Regards,

James B. wrote:

I suggest that, as helpful as this is, it might be much improved if you
could discuss the default behaviour of cucumber with an empty
cucumber.yml and what this equates to with the options explicitly set.
Following illumination of this point then describe what one might like
to alter together with the reasons why and show the setting for that.

For the default behavior type “cucumber --help”. The first flag reads:
" -r, --require LIBRARY|DIR Require files before executing the
features.
If this option is not specified,
all *.rb files that
are siblings or below the features
will be autorequired
This option can be specified
multiple times.
"

So… the autorequiring of the ruby files next to the passed in
feature(s) to be ran is the default behaviour.

I have found that when approaching a new technology, instructions and
examples are ofttimes needlessly opaque simply because there is no
mapping provided between out-of-the-box behaviour and the available
alternatives. Once one is familiar with the tool such things are
obvious, which no doubt is why they are so often omitted form
discussion. But to people just beginning to explore a new environment
such examples are extremely helpful.

Great idea! Someone should probably add that to the wiki…

-Ben

Ben M. wrote:

As the cucumber wiki page says about autotest
(GitHub: Let’s build from here · GitHub) if
you want to override the arguments used by autotest you need to define
an ‘autotest’ profile in your cucumber.yml file. I added information
about profiles here:
GitHub: Let’s build from here · GitHub.

I suggest that, as helpful as this is, it might be much improved if you
could discuss the default behaviour of cucumber with an empty
cucumber.yml and what this equates to with the options explicitly set.
Following illumination of this point then describe what one might like
to alter together with the reasons why and show the setting for that.

I have found that when approaching a new technology, instructions and
examples are ofttimes needlessly opaque simply because there is no
mapping provided between out-of-the-box behaviour and the available
alternatives. Once one is familiar with the tool such things are
obvious, which no doubt is why they are so often omitted form
discussion. But to people just beginning to explore a new environment
such examples are extremely helpful.

Ben M. wrote:

GitHub: Let’s build from here · GitHub.

Great idea! Someone should probably add that to the wiki…

-Ben

Sorry if that last remark came across a little snide. What I meant to
say was that the wiki is a community effort and it would be great if you
could improve it with your insight and point of view.

The wiki instructions probably do suffer from what you described because
most of the wiki has been written by people who have been involved in
cucumber/rspec story runner from the beginning (or who are familiar with
the code base.) So our instructions may not be as helpful to newcomers
as we had hoped. Since you are coming from a different perspective it
would be extremely helpful for your insights to be added to the wiki to
help other people who are beginning and will probably have the same
questions you have had.

Thanks,
Ben

Ben M. wrote:

would be extremely helpful for your insights to be added to the wiki to
help other people who are beginning and will probably have the same
questions you have had.

As soon as I have an insight, my wife will faint…

I would love to assist in this manner, the difficulty being that at
present I posses no idea what settings the default cucumber behaviour
maps to. Further, it is evident that changing one setting in
cucumber.yml, or perhaps setting any setting in it, changes the default
behaviour in non-intuitive ways. For example, an empty cucumber.yml
file causes autotest to display results one way while simply setting the
libraries to load (-r lib) alters the autotest result display. The
connection between these two events, although undeniable, is not
evident.

James B. wrote:

The error is that there exists no entity with id=1. Ideas?

I have resolved all of the test result differences between running
cucumber features -r features, rake features, and autotest. These were
dependent on whether testunit tests and their associated fixtures were
invoked immediately prior to cucumber or not.

The fix to the last failing test was to eliminate the literal id=1 in
the view test and use this construct instead:

Given /I am on the edit entity page/ do
@party = Entity.first
visits “/entities/#{@party.id}/edit”
end

Now, I still have a number of issues with respect to configuring
autotest to work with cucumber. If I have nothing in my cucumber.yml
file then I see this:

script/cucumber features --format progress --format autotest --out
/tmp/autotest-cucumber.7817.0
FFF____F__F__F__F__F_P_F_P_FP_P_F_P_P_F_P_P_F_P_P_F_P_P_

If I change cucumber.yml, while autotest is running, to this:

autotest: features -r features

Then I see this:

script/cucumber --profile autotest --format autotest --out
/tmp/autotest-cucumber.7817.1 -s ‘No locations should exist’ -s ‘Test if
Fixtures are loaded first’ -s ‘Record Entity basic identification
information’ -s ‘The “common name” attribute should be normalized before
saving’ -s ‘Delete entity’ -s ‘Attempt to ADD a VALID entity via a web
form’ -s ‘Attempt to ADD an EMPTY entity via a web form’ -s ‘Attempt to
ADD an UNNAMED entity via a web form’ -s ‘The common name should display
with initial capitals’
Feature: Identify and record the location of parties to transactions

followed by a complete rerun of the entire test suite including
testunit. However, I do not see this:

PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP

at the top of the test run before displaying the steps and tracebacks on
the console. Which, I admit, does not say much but I would like to know
how to get it nonetheless. Adding --format progress by itself seems to
just provide a string of dots … etc.

I would also like to know how to see both the progress bar and the
pretty format console display of the tests. If I add --format pretty
after --format progress then the dots are interspersed with the step
results. Like so:

. And I am on the edit entity page

features/entities/step_definitions/entity_steps.rb:125

. When I fill in “Common Name” with " ANYThing WronG wITh tHiS? "

features/step_definitions/webrat_steps.rb:12

. And I commit the update

Note the leading ‘.’ which prefaces each passing line.

Finally, my custom notice command for autotest seems not to get called
when running cucumber although it works perfectly well with the TestUnit
tests.

Autotest.add_hook :red do |at|
notify ERROR_STOCK_ICON, “Failing Tests”, “#{at.files_to_test.size}
tests failed.”
end

Suggestions welcome.

Andrew P. wrote:

Bit of a guess but

def build_valid_entities(n=1)
Entity.transaction do
Entity.destroy_all
n.to_i.times do |x|
Entity.create! :entity_name => “Entity #{x}”,
:entity_legal_name => “Entity #{n} Legal Name”,
:entity_legal_form => “CORP”
end
end
end

Note the x inside the blcok. Probably not a good idea to use same
variable inside and outside block

2008/11/27 James B. [email protected]:

True, true. Although in this case the value assigned to
:entity_legal_name was meaningless in other circumstances this could
have been a subtle error to find if the test data was expected to be
unique. As I discovered and have reported, my residual discrepancies in
autotest results was due to the loading of TestUnit fixtures leaving the
database in a populated state. Altering the tests to allow for
pre-existing entries resolved the difference in observed behaviour.

As for my custom notifier for autotest, the reason that this does not
work is that cucumber does not link its results to the :red, :green,
:all_good hooks of autotest. I have opened a ticket for a feature
request to have this functionality added to cucumber.

http://rspec.lighthouseapp.com/projects/16211/tickets/108-add-support-for-autotest-red-green-hooks

Ben M. wrote:

Sorry if that last remark came across a little snide. What I meant to
say was that the wiki is a community effort and it would be great if you
could improve it with your insight and point of view.

The wiki instructions probably do suffer from what you described because
most of the wiki has been written by people who have been involved in
cucumber/rspec story runner from the beginning (or who are familiar with
the code base.) So our instructions may not be as helpful to newcomers
as we had hoped. Since you are coming from a different perspective it
would be extremely helpful for your insights to be added to the wiki to
help other people who are beginning and will probably have the same
questions you have had.

Thanks,
Ben

How is this?

James B. wrote:

as we had hoped. Since you are coming from a different perspective it

That’s a really good write up. Thanks!


Joseph W.

Bit of a guess but

def build_valid_entities(n=1)
Entity.transaction do
Entity.destroy_all
n.to_i.times do |x|
Entity.create! :entity_name => “Entity #{x}”,
:entity_legal_name => “Entity #{n} Legal Name”,
:entity_legal_form => “CORP”
end
end
end

Note the x inside the blcok. Probably not a good idea to use same
variable inside and outside block

2008/11/27 James B. [email protected]:


rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users

On Fri, Nov 28, 2008 at 10:48 PM, James B. [email protected]
wrote:

as we had hoped. Since you are coming from a different perspective it
would be extremely helpful for your insights to be added to the wiki to
help other people who are beginning and will probably have the same
questions you have had.

Thanks,
Ben

How is this?
GitHub: Let’s build from here · GitHub

I think that’s a great introduction to Cucumber. I also think it’s a bit
long. There are a few paragraphs that I don’t think provide useful or
relevant information, or that repeat information found elsewhere on the
wiki. My comments in order:

  1. Where to start
    The first

     box and the following paragraph can be skipped.

  2. Where do I put Tests?
    First - I don’t like calling them tests. Let’s call them features and
    rather
    explain on the top that in Cucumber we caqll tests features.

  3. How do I Write Tests?
    Is the explanation of the Given/When etc methods really necessary? The
    alias
    comment is wrong - the implementations are different.
    We use the term “step definition” for regexp+block on the other pages
    (not
    step matcher).

“Worthy of note is that, inside the step files, one must enclose the
arguments to the Given/When/Then/And methods with a string delimiter”:
This is only a convention to make it easier to know what’s a variable
when
reading/editing a feature. It is absolutely not a must.

  1. What way do I run the tests?
    This has a separate page already. It’s better if you edit this with
    additional info and link to it from your page.

  2. Anything else?
    The last code snippet is not needed. It’s already at the top of
    webrat_steps.rb (added recently).

It would be great if you could remove redundant info and link instead -
expand other existing pages where there is missing info. And all in all

very good stuff. Thanks!

BTW - has anyone seen all the cool tools that are cropping up around
Cucumber? GitHub: Let’s build from here · GitHub

Cheers,
Aslak

No it isn’t you can have

When /determine the party (.*)/ do

And call that in in other nested step using a string literal

When
When “deterimine the party foo”
end

In fact I never enclose the arguments with a string delimiter as I
think it makes the feature less clear - don’t think feature writers
should have to worry about having correct quotes. just my 2c

2008/11/28 James B. [email protected]:

aslak hellesoy wrote:

“Worthy of note is that, inside the step files, one must enclose the
arguments to the Given/When/Then/And methods with a string delimiter”:

This is only a convention to make it easier to know what’s a variable
when reading/editing a feature. It is absolutely not a must.

It is a must inside step files:

When /determine the party “(.*)”/ do |n|
When obtain the party n
end

results in

And I determine the party "corporate form"              # 

features/entities/
step_definitions/entity_steps.rb:56
undefined method `party’ for
#<ActionController::Integration::Session:0x7e
e6c3b8> (NoMethodError)

While

When /determine the party “(.*)”/ do |n|
When “obtain the party “#{n}””
end

passes.

Scenario: Record Entity basic identification information #
features/entities/
entity.feature:17

And I determine the party "corporate form"              # 

features/entities/
step_definitions/entity_steps.rb:56
Then I should save the party information #
features/entities/
step_definitions/entity_steps.rb:60

Andrew P. wrote:

No it isn’t you can have

When /determine the party (.*)/ do

And call that in in other nested step using a string literal

When
When “deterimine the party foo”
end

In fact I never enclose the arguments with a string delimiter as I
think it makes the feature less clear - don’t think feature writers
should have to worry about having correct quotes. just my 2c

Non, you misunderstood what I wrote, although in retrospect I should
have been more explicit and precise with my terminology. It is NOT the
feature file that I was speaking of. It was within the step_definitions
file, as your example employs itself. When I referred to step files it
was to the step_definitions.rb files, a clarification that I have since
made to the article.