Forum: RSpec [rspec] loading specs as an object model without running them

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Marty A. (Guest)
on 2009-04-29 10:24
(Received via mailing list)
I'd like to load all of my examples into memory without actually
running them, so I can traverse the object tree and get descriptions
etc.  I'm finding it tricky to figure out how to do that though.
There seems to be an inherent assumption in the framework that that
examples will actually run.

So basically, I want to do something like this:

examples = load_examples_without_running
examples.each {|example| puts example.description}

def load_examples_without_running
 # what goes here?
end


Does anyone have an idea about how I might do that?

-- Marty A.
Matt W. (Guest)
on 2009-04-29 10:28
(Received via mailing list)
On 29 Apr 2009, at 07:23, Marty A. wrote:

>
> def load_examples_without_running
>  # what goes here?
> end
>
>
> Does anyone have an idea about how I might do that?

Use the dry-run option?

Matt W.
http://blog.mattwynne.net
http://www.songkick.com
David C. (Guest)
on 2009-04-29 14:55
(Received via mailing list)
On Wed, Apr 29, 2009 at 1:23 AM, Marty A. <removed_email_address@domain.invalid>
wrote:
> I'd like to load all of my examples into memory without actually
> running them, so I can traverse the object tree and get descriptions
> etc.  I'm finding it tricky to figure out how to do that though.
> There seems to be an inherent assumption in the framework that that
> examples will actually run.
>
> So basically, I want to do something like this:
>
> examples = load_examples_without_running
> examples.each {|example| puts example.description}

There is a --dry-run command line switch that will do this for you,
but I've been considering deprecating it because it can't work
consistently with matcher-generated descriptions:

describe RSpec do
  it { should be_concise }
end

$ spec rspec_spec.rb
RSpec
  should be concise

In this case, the be_concise matcher generates the text "should be
concise" when it is run. If you don't run it, there is no description.
As this approach becomes more and more popular, the --dry-run not only
becomes less useful, it actually becomes somewhat of a frustration.

As for getting a handle on the objects, this is something that is not
really baked into RSpec, but will definitely be baked into rspec-2.0.
Work on this will start later this year (probably not until after the
rspec book is off to print). Goals for it include a very clean
separation of the DSL and the underlying object model and a very
lightweight runner.

So in the short run, your best option is to use --dry-run and
understand that you only get value out of it if you write docstrings
for all of your examples.

Cheers,
David
Marty A. (Guest)
on 2009-04-29 15:10
(Received via mailing list)
On 29/04/2009, at 8:54 PM, David C. wrote:

>> examples = load_examples_without_running
>> examples.each {|example| puts example.description}

[snip]

> As for getting a handle on the objects, this is something that is not
> really baked into RSpec, but will definitely be baked into rspec-2.0.
> Work on this will start later this year (probably not until after the
> rspec book is off to print). Goals for it include a very clean
> separation of the DSL and the underlying object model and a very
> lightweight runner.
>
> So in the short run, your best option is to use --dry-run and
> understand that you only get value out of it if you write docstrings
> for all of your examples.

Ok.  It's actually a bit more complicate than that.  Let me give you
an insight into where I'm headed.

I'm working into a big corporate in Australia, and they are demanding
traceability of tests back to requirements.  I'm already using Mingle
to manage story cards, and I'm hoping to let the testers build in the
traceability via code.  So I've got a Mixin to the examples so that I
can declare the id of the story that the test maps to.  It looks
something like this:

describe FunctionalArea do
   it "automates some acceptance test declared on a story" do
     traces_to_story 35
     # test implementation
   end
end

So now, I want to be able to walk the object tree, getting the
descriptions *and* the story id's.  Mingle has a nice RESTful API that
I've already used to slurp up detail about stories.  Now I just want
to wire in this last bit of information to tick the traceability box
without any human needing to do the work.

I'll have a look at the --dry-run option and see where it gets me.  If
you've got any more tips, let me know.  I'll send in a code snippet if
I manage to get it going :)

-- Marty
David C. (Guest)
on 2009-04-29 15:43
(Received via mailing list)
On Wed, Apr 29, 2009 at 6:09 AM, Marty A. <removed_email_address@domain.invalid>
wrote:
>>> examples will actually run.
>> Work on this will start later this year (probably not until after the
>
>    # test implementation
> you've got any more tips, let me know.  I'll send in a code snippet if I
> manage to get it going :)

There *is* something you can *almost* use for this, but it's
incomplete at the moment. RSpec let's you write your own output
formatters, which receive messages for every example group and example
as they are run. Currently, the object passed to example_started (and
a few other methods) is an ExampleGroupProxy, which includes an
options hash passed to the example. So if you do this:

it "automates some acceptance test declared on a story",
    :traces_to_story => 25 do
# ...
end

Then you could do stuff like:

def example_started(example_proxy)
  traces_to_story(example_proxy.options[:traces_to_story]
end

This should solve your problem assuming that you have docstrings for
all of your examples. The big missing piece at this moment is that the
ExampleGroupProxy (passed to example_group_started) doesn't include
the options hash, but I'll happily add that tonight.

To see this in action, try this:

* in options_formatter.rb

require 'spec/runner/formatter/base_text_formatter'

class OptionsFormatter < Spec::Runner::Formatter::BaseTextFormatter
  def example_started(proxy)
    puts proxy.options.inspect unless proxy.options.empty?
  end
end

* in example_spec.rb

require 'spec'

describe "example" do
  it "supports options", :like => 'these' do
    # no need for anything here
  end
end

Now, on the command line:

$ spec --require options_formatter.rb example_spec.rb --format
OptionsFormatter
{:like => 'these'}

Formatter API is here:
http://rspec.rubyforge.org/rspec/1.2.4/classes/Spe...

Let me know if this is a good direction for you and I'll add the
options hash to the example group proxy.

Cheers,
David
Marty A. (Guest)
on 2009-04-29 15:51
(Received via mailing list)
On 29/04/2009, at 9:41 PM, David C. wrote:

>>>> running them, so I can traverse the object tree and get
>> [snip]
>>>
>> Mingle to
>>    # test implementation
>> needing to do
> formatters, which receive messages for every example group and example
>
> def example_started(example_proxy)
>  traces_to_story(example_proxy.options[:traces_to_story]
> end
>
> This should solve your problem assuming that you have docstrings for
> all of your examples. The big missing piece at this moment is that the
> ExampleGroupProxy (passed to example_group_started) doesn't include
> the options hash, but I'll happily add that tonight.

Yep - no problems there.  I reckon about 98% have docstrings.  I can
easily add the remainders.



> end
>
> options hash to the example group proxy.
I reckon this would be great *if* it will still work with the --dry-
run option.  Most of the examples are actually integration tests that
call out to other tools as they run, so the build is S..L..O..W (over
an hour).  I want to be able to generate the report without actually
running the tests.

-- Marty
David C. (Guest)
on 2009-04-29 16:16
(Received via mailing list)
On Wed, Apr 29, 2009 at 6:50 AM, Marty A. <removed_email_address@domain.invalid>
wrote:
>>>>>
>>>
>>>> understand that you only get value out of it if you write docstrings
>>> like
>>> descriptions
>>> manage to get it going :)
>> # ...
>> ExampleGroupProxy (passed to example_group_started) doesn't include
>>
>> require 'spec'
>> OptionsFormatter
> option.  Most of the examples are actually integration tests that call out
> to other tools as they run, so the build is S..L..O..W (over an hour).  I
> want to be able to generate the report without actually running the tests.

I'm really sorry about this, but I just couldn't wait for tonight to add
this:

  http://github.com/dchelimsky/rspec/commit/a4b3aac6...

The release *will* have to wait for tonight, but you can easily build
the gem locally: http://wiki.github.com/dchelimsky/rspec/edge

Cheers,
David
Marty A. (Guest)
on 2009-04-29 16:23
(Received via mailing list)
[snip]

> I'm really sorry about this, but I just couldn't wait for tonight to
> add this:
>
>  http://github.com/dchelimsky/rspec/commit/a4b3aac6...
>
> The release *will* have to wait for tonight, but you can easily build
> the gem locally: http://wiki.github.com/dchelimsky/rspec/edge

Thanks David :)

I'm in Australia, and it's already getting late here.  Your tonight
will be during the day tomorrow for me.  I'll give it a go then.

-- Marty
Kero van Gelder (Guest)
on 2009-04-30 02:00
(Received via mailing list)
>>> So basically, I want to do something like this:
>>>
>>> examples = load_examples_without_running
>>> examples.each {|example| puts example.description}

[snip]

>     # test implementation
>   end
> end

I would have expected Cucumber to map stories to scenarios/examples
by having them in the same file (and a @tag if someone insists that
a number referrring an external doc is better for tracability).

I would have expected RSpec to cover a lower level that is of no
interest to the person wanting traceability, as that person can not
judge whether the code works anyway (YMMV). Besides, a spec can
apply to more than one requirement; or to no particular requirement at
all.

I'm expecting this from regulations that deal with medical devices.
Validation has to happen on story level; risk mitigation may apply
to lower levels, though. Our department is working on several projects
that'll result in medical devices and we are learning how regulations,
audits and things work for real.

So I'd like to hear why you are trying to do what you do.

There can be other requirements for other reasons. What traceability
are you looking for in these tests? Why do you look at the RSpec
level?

Bye,
Kero.
___
How can I change the world if I can't even change myself?
  -- Faithless, Salva Mea
Marty A. (Guest)
on 2009-04-30 02:30
(Received via mailing list)
On 30/04/2009, at 7:59 AM, Kero van Gelder wrote:

>> manage story cards, and I'm hoping to let the testers build in the
>
> I'm expecting this from regulations that deal with medical devices.
> Validation has to happen on story level; risk mitigation may apply
> to lower levels, though. Our department is working on several projects
> that'll result in medical devices and we are learning how regulations,
> audits and things work for real.
>
> So I'd like to hear why you are trying to do what you do.

Sure.


> There can be other requirements for other reasons. What traceability
> are you looking for in these tests? Why do you look at the RSpec
> level?

I don't necessarily disagree with what you are saying, but the project
was going long before cucumber existed, and we already had a heavy
investment in tests using rspec.  It's a data warehousing project, so
there's little or no UI to speak of.  We built a DSL on top of RSpec
to allow us to manipulate data and verify results after ETL jobs run.
Most of tests set up data, run the ETL job (calling out to an external
tool) and then verify that the results look the way they're supposed to.

So to us, RSpec isn't a 'lower level'.  Perhaps if the project was
starting now, I'd consider cucumber.  Our investment in a DSL on RSpec
is reasonably heavy though, and is working really well for us.  I'd
rather enhance that a bit further than switch frameworks for the sake
of one feature.

-- Marty
This topic is locked and can not be replied to.