Simple == with prettier error messages + good documentation

Testing RSpec
1

Hi all,

I’ve found myself writing a thing I think is less than optimal, looking
for suggestions. The context is, I’m testing a result, and as a part of
that test, I might verify two or three things, which are individually
relevant but not really discrete results (?).

Here’s my thinking process, using a toy example:

foo.should == bar (or foo.should_not be_nil)

expected not to be nil, but was

(hm, not very informative)

if( foo == nil )
“failure to setup foo”.should == “foo should be set to the thing
that will be rendered”
end

expected “foo should be set to the thing that will be rendered”,
got “failure to setup foo” (using ==)

I’ve used this, by example, for a test on a dependency (imagemagick),
where if the dependency isn’t found, I show a decent message with info
the tester can use to resolve it. And, as I mentioned, I’ve used it for
revealing more details in cases where the it “” + the generic error
aren’t informative.

I’m satisfied using this method for things like detecting a failure to
use a test-helper correctly - works fine, doesn’t get in my way as part
of the documentation. Which brings me to the problem I’m concerned
about:

With this method, nothing come out in the generated spec-docs to
represent the thing I’m conditionally requiring.

I guess I could get more fine-grained with my it()'s, but I’ve been
preferring a more general statement for it(), that gives the sense
without the detail.

Any suggestions?

Thanks,

Randy

2

On Thu, Jan 29, 2009 at 12:00 PM, [email protected] wrote:

got “failure to setup foo” (using ==)
With this method, nothing come out in the generated spec-docs to represent
the thing I’m conditionally requiring.

I guess I could get more fine-grained with my it()'s, but I’ve been
preferring a more general statement for it(), that gives the sense without
the detail.

Any suggestions?

I can’t think of anything that wouldn’t result in something that
requires
more writing as of now. Maybe we need a new construct like:

it “does something” do
with_message “this is a more specific message” do
foo.should == bar
end
end

WDYT?

3

On 29/01/2009, at 1:00 PM, [email protected] wrote:

got “failure to setup foo” (using ==)
concerned about:
Thanks,

Randy

Hi Randy. I’m not 100% sure what you’re asking. In short, are you
wondering how to generate expectation failure messages that are more
verbose and contextual?

Cheers,
Nick

4

On Thu, Jan 29, 2009 at 7:25 PM, David C. [email protected]
wrote:

will be rendered"

I think that would be useful. Maybe make it more explicit that it’s an
error message:

on_error “bla” do
…
end

5

On Thu, Jan 29, 2009 at 8:18 PM, David C. [email protected]
wrote:

foo.should == bar (or foo.should_not be_nil)

I guess I could get more fine-grained with my it()'s, but I’ve been
with_message “this is a more specific message” do
…
end

on_failure “…” do ???

uh crap. strike last remark

6

On 29/01/2009, at 2:18 PM, David C. wrote:

I’ve found myself writing a thing I think is less than optimal,

expected “foo should be set to the thing that will be rendered”,
informative.

it “does something” do
on_error “bla” do
…
end

on_failure “…” do ???

I like “on_failure”, as it’s consistent with RSpec’s output. Eg:
31 examples, 0 failures

What could be done to make the construct more sentence-like? If used
in this manner:

it ‘should do something’ do
on_failure “@foo is nil” do
@foo.should_not be_nil
end
end

It reads like this to me:
If “@foo is nil” fails, execute the block.

These are a bit verbose, but what do you think these approaches?:
http://gist.github.com/54726

7

On Thu, Jan 29, 2009 at 2:38 PM, Nick H. [email protected]
wrote:

Hi all,

expected “foo should be set to the thing that will be rendered”,
I’m satisfied using this method for things like detecting a failure to
preferring a more general statement for it(), that gives the sense
foo.should == bar
end
on_failure “@foo is nil” do
@foo.should_not be_nil
end
end

It reads like this to me:
If “@foo is nil” fails, execute the block.

These are a bit verbose, but what do you think these approaches?:
http://gist.github.com/54726

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

8

On Thu, Jan 29, 2009 at 1:02 PM, aslak hellesoy
[email protected]wrote:

that test, I might verify two or three things, which are individually
if( foo == nil )
the

I think that would be useful. Maybe make it more explicit that it’s an
error message:

on_error “bla” do
…
end

on_failure “…” do ???

9

On 29/01/2009, at 4:27 PM, aslak hellesoy wrote:

http://gist.github.com/54726

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

Upped:

54758’s gists · GitHub

Putting the message at the end is a great idea! I still think that
naming the method (and thus starting the block) with “on_failure” is a
bit misleading when read aloud…but maybe that’s not a big concern
for you guys?

What do you think about this modification to #3?:

10

On Thu, Jan 29, 2009 at 3:27 PM, aslak hellesoy
[email protected] wrote:

On Thu, Jan 29, 2009 at 12:00 PM, [email protected] wrote:

end
informative.

it “does something” do
on_error “bla” do

http://gist.github.com/54726

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

Upped:

54758’s gists · GitHub

Somebody move this to a ticket please :slight_smile:

11

On Thu, Jan 29, 2009 at 10:14 PM, David C. [email protected]
wrote:

foo.should == bar (or foo.should_not be_nil)

I guess I could get more fine-grained with my it()'s, but I’ve been
with_message “this is a more specific message” do
…
it ‘should do something’ do

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

Upped:

12

I’m starting to wonder if this is a good idea to begin with. I had
started to suggest nested it()s:

it “…” do

it “something more” do
end
end

… but it’s already handled by the existing nested describe(),
before(), it() system.

I guess if we were shooting for flexibility, we might ask what the
simplest way is to the next level of flex? One route might be a detail
level on describe() blocks. So, suggestion #5:

Then when generating spec docs, you could vary the detail level on
what’s generated, to generate only up to level-2 documentation, or to
generate full-detail documentation if you prefer.

Nested it()s could do the same thing, I suppose. Not sure what side
effects would come into play. Pretty clearly, a nested it() wouldn’t
have an embedded transaction or respect the outer before() block, since
the outer if() would handle both those things.

Randy

From: aslak hellesoy [email protected]
To: rspec-users [email protected]
Sent: Thursday, January 29, 2009 1:27:02 PM
Subject: Re: [rspec-users] simple == with prettier error messages + good
documentation

On Thu, Jan 29, 2009 at 10:14 PM, David C. [email protected]
wrote:

foo.should == bar (or foo.should_not be_nil)

I guess I could get more fine-grained with my it()'s, but I’ve been
with_message “this is a more specific message” do
…
it ‘should do something’ do

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

Upped:

13

On Thu, Jan 29, 2009 at 3:50 PM, [email protected] wrote:

Nested it()s could do the same thing, I suppose. Not sure what side effects
would come into play. Pretty clearly, a nested it() wouldn’t have an
embedded transaction or respect the outer before() block, since the outer
if() would handle both those things.

For a number of technical reasons, the nested code example (it()) is a
non-starter.

The idea of having some means of flagging examples so they report
differently is possible. Right now, example groups and examples each
take a hash in their declarations:

describe “an example group”, :with => "some, :args => “like this” do
it “an example”, :with => “some”, :other => “args”
…
end
end

Those are accessible to the runner, and are used internally for
assorted filtering. As of this moment, they don’t make it past the
reporters, because the object that goes to formatters is a wrapper w/
limited information instead of the real example. We could add the
options to that wrapper though, and then you’d be able to write a
custom formatter that would do anything you like based on the
configuration of those options.

WDYT about that?

14

On 29 Jan 2009, at 22:20, David C. wrote:

of
(hm, not very informative)

got “failure to setup foo” (using ==)
error
the
sense
without
the detail.

I really do suggest setting up the context in the before() block of a
describe (example group), then using each ‘it’ (example) to hold an
assertion. It’s a tried and tested way of doing it for me, but it does
take some getting used to when you move over from xUnit style tests.

end
end

http://gist.github.com/54726

I’ll take that gist and raise you one:

Suggestion #1.rb · GitHub (Suggestion #3)

Upped:

54758’s gists · GitHub

I do find myself hitting this sometimes, especially when I use rspec
assertions in Cucumber steps - I usually stick to one assertion per
example in RSpec so it isn’t a problem when I’m writing unit tests.

However we do have a couple of alternatives already, without extending
RSpec any more:
(1) just use traditional test/unit assertions instead (works OK in
Cucumber steps, though obviously they don’t read quite so well)
(2) write a custom matcher

We could also pretty easily extend the existing matchers, just taking
a failure message in the factory method, e.g:

I agree this discussion should move to a ticket. But it seems I am to
lazy to do it myself :wink:

Matt W.
http://blog.mattwynne.net

15

I do like try_to do…end.or_report - much better for the case where I’m
not trying to generate documentation.

I’m not sure it works so well for the case where I want to generate an
extra element of documentation, though this might be just an
implementation detail.

.or_report
.or_document
.or_gripe
.or_complain

One of the first two could generate documentation, and one of the others
might generate none.

Randy

From: Nick H. [email protected]

On 29/01/2009, at 4:27 PM, aslak hellesoy wrote:

54758’s gists · GitHub

Putting the message at the end is a great idea! I still think that
naming the method (and thus starting the block) with “on_failure” is a
bit misleading when read aloud…but maybe that’s not a big concern for
you guys?

What do you think about this modification to #3?:

16

On Fri, Jan 30, 2009 at 12:00 PM, Rick DeNatale
[email protected] wrote:

Nice to see you step back from the list of suggestions and nicely summarize
the problem.

I think that using the example docstring works well enough if you are
religulous about the 1 expectation per example approach.

In general I think what is wanted is to have a string to override what is
reported on a failed expectation, which prompted me to make yet another
suggestion

Suggestion #1.rb · GitHub

Now that I’ve started a ticket, would you kindly put your response in
the ticket instead?

Thanks

17

On Thu, Jan 29, 2009 at 7:33 PM, David C.
[email protected]wrote:

I agree this discussion should move to a ticket. But it seems I am to
lazy
to do it myself :wink:

Lighthouse - Beautifully Simple Issue Tracking

Nice to see you step back from the list of suggestions and nicely
summarize
the problem.

I think that using the example docstring works well enough if you are
religulous about the 1 expectation per example approach.

In general I think what is wanted is to have a string to override what
is
reported on a failed expectation, which prompted me to make yet another
suggestion

–
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale

18

On Fri, Jan 30, 2009 at 1:36 PM, David C.
[email protected]wrote:

suggestion

Suggestion #1.rb · GitHub

Now that I’ve started a ticket, would you kindly put your response in
the ticket instead?

Thanks

Done

–
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale

19

Passing the it() and describe() args through to the reporting layer
sounds like a great idea. I’ve thought more about the reporting-levels
approach. The numeric as I postulated earlier is probably not the most
useful thing.

As a product manager, I want to be able to flexibly present the
appropriate documentation to different audiences, to avoid the need for
maintaining multiple docs.

describe “an example group”, :audience => :execs
describe “an example group”, :audience => :owner
describe “an example group”, :audience => :users
describe “an example group”, :audience => :developers

These could be entirely ad-hoc, allowing the developer to select the
desired labels for the situation at hand.

When generating reports, you’d state which audiences to generate docs
for (maybe with a default set coded into a helper?).

When generating reports to text, any examples or groups that were
skipped would be simply tabulated, and the summary would be displayed at
the bottom:

Skipped 7 example groups and 27 examples for Users.
Skipped 3 example groups and 12 examples for Developers.

And when generating reports to HTML, any docs for not-specified roles
could simply be folded/hidden by default, allowing the viewer to drill
down. Of course, this could be skipped by setting an option for static
HTML purposes, and the skipped items could be reported in the same way
as the text version.

Does this seem like a useful way to treat the various detail levels of
documentation we want to present to different audiences?

Randy

From: David C. [email protected]

Randy:

54764’s gists · GitHub

Then when generating spec docs, you could vary the detail level on what’s
generated, to generate only up to level-2 documentation, or to generate
full-detail documentation if you prefer.

Nested it()s could do the same thing, I suppose. Not sure what side effects
would come into play. Pretty clearly, a nested it() wouldn’t have an
embedded transaction or respect the outer before() block, since the outer
if() would handle both those things.

For a number of technical reasons, the nested code example (it()) is a
non-starter.

The idea of having some means of flagging examples so they report
differently is possible. Right now, example groups and examples each
take a hash in their declarations:

describe “an example group”, :with => "some, :args => “like this” do
it “an example”, :with => “some”, :other => “args”
…
end
end

Those are accessible to the runner, and are used internally for
assorted filtering. As of this moment, they don’t make it past the
reporters, because the object that goes to formatters is a wrapper w/
limited information instead of the real example. We could add the
options to that wrapper though, and then you’d be able to write a
custom formatter that would do anything you like based on the
configuration of those options.

WDYT about that?

20

On Sat, Jan 31, 2009 at 12:02 PM, [email protected] wrote:

describe “an example group”, :audience => :users

Skipped 7 example groups and 27 examples for Users.
Skipped 3 example groups and 12 examples for Developers.

And when generating reports to HTML, any docs for not-specified roles could
simply be folded/hidden by default, allowing the viewer to drill down. Of
course, this could be skipped by setting an option for static HTML purposes,
and the skipped items could be reported in the same way as the text version.

Does this seem like a useful way to treat the various detail levels of
documentation we want to present to different audiences?

Hey Randy,

I just committed code that makes the options available to the
formatters for both example groups and individual examples. If you
grab the latest from git, take a look at
examples/passing/filtered_formatter_example.rb and you’ll see how it
works.

This should let you experiment with different ways of filtering
things. The approach you suggest does seem sound, but I’m not prepared
to add any first class support for anything like that to rspec proper
until its been proven effective.

I’m also interested in figuring out how to apply this to the actual
run, not just the reporting. i.e. use options as a tagging system to
determine which groups and/or individual examples to run. This would
require some entry point earlier in the process, and I’m not quite
sure the best approach yet. Ideas are welcome.

Cheers,
David