[rspec] loading specs as an object model without running them

martin_j_andrews · April 29, 2009, 8:24am

I’d like to load all of my examples into memory without actually
running them, so I can traverse the object tree and get descriptions
etc. I’m finding it tricky to figure out how to do that though.
There seems to be an inherent assumption in the framework that that
examples will actually run.

So basically, I want to do something like this:

examples = load_examples_without_running
examples.each {|example| puts example.description}

def load_examples_without_running

what goes here?

end

Does anyone have an idea about how I might do that?

– Marty A.

martin_j_andrews · April 29, 2009, 12:55pm

On Wed, Apr 29, 2009 at 1:23 AM, Marty A. [email protected]
wrote:

I’d like to load all of my examples into memory without actually
running them, so I can traverse the object tree and get descriptions
etc. I’m finding it tricky to figure out how to do that though.
There seems to be an inherent assumption in the framework that that
examples will actually run.

So basically, I want to do something like this:

examples = load_examples_without_running
examples.each {|example| puts example.description}

There is a --dry-run command line switch that will do this for you,
but I’ve been considering deprecating it because it can’t work
consistently with matcher-generated descriptions:

describe RSpec do
it { should be_concise }
end

$ spec rspec_spec.rb
RSpec
should be concise

In this case, the be_concise matcher generates the text “should be
concise” when it is run. If you don’t run it, there is no description.
As this approach becomes more and more popular, the --dry-run not only
becomes less useful, it actually becomes somewhat of a frustration.

As for getting a handle on the objects, this is something that is not
really baked into RSpec, but will definitely be baked into rspec-2.0.
Work on this will start later this year (probably not until after the
rspec book is off to print). Goals for it include a very clean
separation of the DSL and the underlying object model and a very
lightweight runner.

So in the short run, your best option is to use --dry-run and
understand that you only get value out of it if you write docstrings
for all of your examples.

Cheers,
David

martin_j_andrews · April 29, 2009, 1:10pm

On 29/04/2009, at 8:54 PM, David C. wrote:

examples = load_examples_without_running
examples.each {|example| puts example.description}

[snip]

As for getting a handle on the objects, this is something that is not
really baked into RSpec, but will definitely be baked into rspec-2.0.
Work on this will start later this year (probably not until after the
rspec book is off to print). Goals for it include a very clean
separation of the DSL and the underlying object model and a very
lightweight runner.

So in the short run, your best option is to use --dry-run and
understand that you only get value out of it if you write docstrings
for all of your examples.

Ok. It’s actually a bit more complicate than that. Let me give you
an insight into where I’m headed.

I’m working into a big corporate in Australia, and they are demanding
traceability of tests back to requirements. I’m already using Mingle
to manage story cards, and I’m hoping to let the testers build in the
traceability via code. So I’ve got a Mixin to the examples so that I
can declare the id of the story that the test maps to. It looks
something like this:

describe FunctionalArea do
it “automates some acceptance test declared on a story” do
traces_to_story 35
# test implementation
end
end

So now, I want to be able to walk the object tree, getting the
descriptions and the story id’s. Mingle has a nice RESTful API that
I’ve already used to slurp up detail about stories. Now I just want
to wire in this last bit of information to tick the traceability box
without any human needing to do the work.

I’ll have a look at the --dry-run option and see where it gets me. If
you’ve got any more tips, let me know. I’ll send in a code snippet if
I manage to get it going

– Marty

martin_j_andrews · April 29, 2009, 8:28am

On 29 Apr 2009, at 07:23, Marty A. wrote:

def load_examples_without_running

what goes here?

end

Does anyone have an idea about how I might do that?

Use the dry-run option?

Matt W.
http://blog.mattwynne.net

martin_j_andrews · April 29, 2009, 1:43pm

On Wed, Apr 29, 2009 at 6:09 AM, Marty A. [email protected]
wrote:

examples will actually run.
Work on this will start later this year (probably not until after the

test implementation

you’ve got any more tips, let me know. I’ll send in a code snippet if I
manage to get it going

There is something you can almost use for this, but it’s
incomplete at the moment. RSpec let’s you write your own output
formatters, which receive messages for every example group and example
as they are run. Currently, the object passed to example_started (and
a few other methods) is an ExampleGroupProxy, which includes an
options hash passed to the example. So if you do this:

it “automates some acceptance test declared on a story”,
:traces_to_story => 25 do

…

end

Then you could do stuff like:

def example_started(example_proxy)
traces_to_story(example_proxy.options[:traces_to_story]
end

This should solve your problem assuming that you have docstrings for
all of your examples. The big missing piece at this moment is that the
ExampleGroupProxy (passed to example_group_started) doesn’t include
the options hash, but I’ll happily add that tonight.

To see this in action, try this:

in options_formatter.rb

require ‘spec/runner/formatter/base_text_formatter’

class OptionsFormatter < Spec::Formatter::BaseTextFormatter
def example_started(proxy)
puts proxy.options.inspect unless proxy.options.empty?
end
end

in example_spec.rb

require ‘spec’

describe “example” do
it “supports options”, :like => ‘these’ do
# no need for anything here
end
end

Now, on the command line:

$ spec --require options_formatter.rb example_spec.rb --format
OptionsFormatter
{:like => ‘these’}

Formatter API is here:
http://rspec.rubyforge.org/rspec/1.2.4/classes/Spec/Runner/Formatter/BaseFormatter.html

Let me know if this is a good direction for you and I’ll add the
options hash to the example group proxy.

Cheers,
David

martin_j_andrews · April 29, 2009, 2:16pm

On Wed, Apr 29, 2009 at 6:50 AM, Marty A. [email protected]
wrote:

understand that you only get value out of it if you write docstrings
like
descriptions
manage to get it going

…

ExampleGroupProxy (passed to example_group_started) doesn’t include

require ‘spec’
OptionsFormatter
option. Most of the examples are actually integration tests that call out
to other tools as they run, so the build is S…L…O…W (over an hour). I
want to be able to generate the report without actually running the tests.

I’m really sorry about this, but I just couldn’t wait for tonight to add
this:

Added options hash to example group proxy. · dchelimsky/rspec@a4b3aac · GitHub

The release will have to wait for tonight, but you can easily build
the gem locally: http://wiki.github.com/dchelimsky/rspec/edge

Cheers,
David

martin_j_andrews · April 29, 2009, 2:23pm

[snip]

I’m really sorry about this, but I just couldn’t wait for tonight to
add this:

Added options hash to example group proxy. · dchelimsky/rspec@a4b3aac · GitHub

The release will have to wait for tonight, but you can easily build
the gem locally: http://wiki.github.com/dchelimsky/rspec/edge

Thanks David

I’m in Australia, and it’s already getting late here. Your tonight
will be during the day tomorrow for me. I’ll give it a go then.

– Marty

martin_j_andrews · April 30, 2009, 12:00am

So basically, I want to do something like this:

examples = load_examples_without_running
examples.each {|example| puts example.description}

[snip]

# test implementation

end
end

I would have expected Cucumber to map stories to scenarios/examples
by having them in the same file (and a @tag if someone insists that
a number referrring an external doc is better for tracability).

I would have expected RSpec to cover a lower level that is of no
interest to the person wanting traceability, as that person can not
judge whether the code works anyway (YMMV). Besides, a spec can
apply to more than one requirement; or to no particular requirement at
all.

I’m expecting this from regulations that deal with medical devices.
Validation has to happen on story level; risk mitigation may apply
to lower levels, though. Our department is working on several projects
that’ll result in medical devices and we are learning how regulations,
audits and things work for real.

So I’d like to hear why you are trying to do what you do.

There can be other requirements for other reasons. What traceability
are you looking for in these tests? Why do you look at the RSpec
level?

Bye,
Kero.

How can I change the world if I can’t even change myself?
– Faithless, Salva Mea

martin_j_andrews · April 30, 2009, 12:30am

On 30/04/2009, at 7:59 AM, Kero van Gelder wrote:

manage story cards, and I’m hoping to let the testers build in the

I’m expecting this from regulations that deal with medical devices.
Validation has to happen on story level; risk mitigation may apply
to lower levels, though. Our department is working on several projects
that’ll result in medical devices and we are learning how regulations,
audits and things work for real.

So I’d like to hear why you are trying to do what you do.

Sure.

There can be other requirements for other reasons. What traceability
are you looking for in these tests? Why do you look at the RSpec
level?

I don’t necessarily disagree with what you are saying, but the project
was going long before cucumber existed, and we already had a heavy
investment in tests using rspec. It’s a data warehousing project, so
there’s little or no UI to speak of. We built a DSL on top of RSpec
to allow us to manipulate data and verify results after ETL jobs run.
Most of tests set up data, run the ETL job (calling out to an external
tool) and then verify that the results look the way they’re supposed to.

So to us, RSpec isn’t a ‘lower level’. Perhaps if the project was
starting now, I’d consider cucumber. Our investment in a DSL on RSpec
is reasonably heavy though, and is working really well for us. I’d
rather enhance that a bit further than switch frameworks for the sake
of one feature.

– Marty

martin_j_andrews · April 29, 2009, 1:51pm

On 29/04/2009, at 9:41 PM, David C. wrote:

running them, so I can traverse the object tree and get
[snip]

Mingle to

test implementation

needing to do
formatters, which receive messages for every example group and example

def example_started(example_proxy)
traces_to_story(example_proxy.options[:traces_to_story]
end

This should solve your problem assuming that you have docstrings for
all of your examples. The big missing piece at this moment is that the
ExampleGroupProxy (passed to example_group_started) doesn’t include
the options hash, but I’ll happily add that tonight.

Yep - no problems there. I reckon about 98% have docstrings. I can
easily add the remainders.

end

options hash to the example group proxy.
I reckon this would be great if it will still work with the --dry-
run option. Most of the examples are actually integration tests that
call out to other tools as they run, so the build is S…L…O…W (over
an hour). I want to be able to generate the report without actually
running the tests.

– Marty