Forum: RSpec simple == with prettier error messages + good documentation

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
942a74b1b71ca62234ae690b44699ebc?d=identicon&s=25 unknown (Guest)
on 2009-01-29 19:14
(Received via mailing list)
Hi all,

I've found myself writing a thing I think is less than optimal, looking
for suggestions.  The context is, I'm testing a result, and as a part of
that test, I might verify two or three things, which are individually
relevant but not really discrete results (?).

Here's my thinking process, using a toy example:

  foo.should == bar (or foo.should_not be_nil)

> expected not to be nil, but was

(hm, not very informative)

  if( foo == nil )
    "failure to setup foo".should == "foo should be set to the thing
that will be rendered"
  end

> expected "foo should be set to the thing that will be rendered",
> got "failure to setup foo" (using ==)

I've used this, by example, for a test on a dependency (imagemagick),
where if the dependency isn't found, I show a decent message with info
the tester can use to resolve it.  And, as I mentioned, I've used it for
revealing more details in cases where the it "" + the generic error
aren't informative.

I'm satisfied using this method for things like detecting a failure to
use a test-helper correctly - works fine, doesn't get in my way as part
of the documentation.  Which brings me to the problem I'm concerned
about:

With this method, nothing come out in the generated spec-docs to
represent the thing I'm conditionally requiring.

I guess I could get more fine-grained with my it()'s, but I've been
preferring a more general statement for it(), that gives the sense
without the detail.

Any suggestions?

Thanks,

Randy
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-29 19:54
(Received via mailing list)
On Thu, Jan 29, 2009 at 12:00 PM, <r_j_h_box-sf@yahoo.com> wrote:

>
> > got "failure to setup foo" (using ==)
> With this method, nothing come out in the generated spec-docs to represent
> the thing I'm conditionally requiring.
>
> I guess I could get more fine-grained with my it()'s, but I've been
> preferring a more general statement for it(), that gives the sense without
> the detail.
>
> Any suggestions?
>

I can't think of anything that wouldn't result in something that
requires
more writing as of now. Maybe we need a new construct like:

it "does something" do
  with_message "this is a more specific message" do
    foo.should == bar
  end
end

WDYT?
49de4cd2f26705785cbef2b15a9df7aa?d=identicon&s=25 Nick Hoffman (nickh)
on 2009-01-29 20:04
(Received via mailing list)
On 29/01/2009, at 1:00 PM, r_j_h_box-sf@yahoo.com wrote:
>
> > got "failure to setup foo" (using ==)
> concerned about:
> Thanks,
>
> Randy

Hi Randy. I'm not 100% sure what you're asking. In short, are you
wondering how to generate expectation failure messages that are more
verbose and contextual?

Cheers,
Nick
0be0e4aa42aacd9a8a95c792de273ca7?d=identicon&s=25 Aslak Hellesøy (aslakhellesoy)
on 2009-01-29 20:16
(Received via mailing list)
On Thu, Jan 29, 2009 at 7:25 PM, David Chelimsky <dchelimsky@gmail.com>
wrote:
>>
>> will be rendered"
>>
>>
>
I think that would be useful. Maybe make it more explicit that it's an
error message:

on_error "bla" do
  ...
end
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-29 20:35
(Received via mailing list)
On Thu, Jan 29, 2009 at 1:02 PM, aslak hellesoy
<aslak.hellesoy@gmail.com>wrote:

> >> that test, I might verify two or three things, which are individually
> >>   if( foo == nil )
> the
> >>
> >
> I think that would be useful. Maybe make it more explicit that it's an
> error message:
>
> on_error "bla" do
>  ...
> end


on_failure "..." do ????
0be0e4aa42aacd9a8a95c792de273ca7?d=identicon&s=25 Aslak Hellesøy (aslakhellesoy)
on 2009-01-29 21:40
(Received via mailing list)
On Thu, Jan 29, 2009 at 8:18 PM, David Chelimsky <dchelimsky@gmail.com>
wrote:
>> >>
>> >>   foo.should == bar (or foo.should_not be_nil)
>> >>
>> >>
>> >> I guess I could get more fine-grained with my it()'s, but I've been
>> >   with_message "this is a more specific message" do
>>  ...
>> end
>
> on_failure "..." do ????
>

uh crap. strike last remark
49de4cd2f26705785cbef2b15a9df7aa?d=identicon&s=25 Nick Hoffman (nickh)
on 2009-01-29 22:13
(Received via mailing list)
On 29/01/2009, at 2:18 PM, David Chelimsky wrote:
> >> I've found myself writing a thing I think is less than optimal,
> >>
> >> > expected "foo should be set to the thing that will be rendered",
> >> informative.
> >>
> > it "does something" do
> on_error "bla" do
>  ...
> end
>
> on_failure "..." do ????

I like "on_failure", as it's consistent with RSpec's output. Eg:
   31 examples, 0 failures

What could be done to make the construct more sentence-like? If used
in this manner:

it 'should do something' do
   on_failure "@foo is nil" do
     @foo.should_not be_nil
   end
end

It reads like this to me:
   If "@foo is nil" fails, execute the block.

These are a bit verbose, but what do you think these approaches?:
http://gist.github.com/54726
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-29 22:19
(Received via mailing list)
On Thu, Jan 29, 2009 at 2:38 PM, Nick Hoffman <nick@deadorange.com>
wrote:
>> >> Hi all,
>> >>
>> >> > expected "foo should be set to the thing that will be rendered",
>> >> I'm satisfied using this method for things like detecting a failure to
>> >> preferring a more general statement for it(), that gives the sense
>> >     foo.should == bar
>> end
>  on_failure "@foo is nil" do
>    @foo.should_not be_nil
>  end
> end
>
> It reads like this to me:
>  If "@foo is nil" fails, execute the block.
>
> These are a bit verbose, but what do you think these approaches?:
> http://gist.github.com/54726

I'll take that gist and raise you one:

http://gist.github.com/54750 (Suggestion #3)
0be0e4aa42aacd9a8a95c792de273ca7?d=identicon&s=25 Aslak Hellesøy (aslakhellesoy)
on 2009-01-29 22:29
(Received via mailing list)
On Thu, Jan 29, 2009 at 10:14 PM, David Chelimsky <dchelimsky@gmail.com>
wrote:
>>> >>
>>> >>   foo.should == bar (or foo.should_not be_nil)
>>> >>
>>> >>
>>> >> I guess I could get more fine-grained with my it()'s, but I've been
>>> >   with_message "this is a more specific message" do
>>>  ...
>> it 'should do something' do
>
> I'll take that gist and raise you one:
>
> http://gist.github.com/54750 (Suggestion #3)
>

Upped:

http://gist.github.com/54758
49de4cd2f26705785cbef2b15a9df7aa?d=identicon&s=25 Nick Hoffman (nickh)
on 2009-01-29 22:48
(Received via mailing list)
On 29/01/2009, at 4:27 PM, aslak hellesoy wrote:
>>>
>>> http://gist.github.com/54726
>>
>> I'll take that gist and raise you one:
>>
>> http://gist.github.com/54750 (Suggestion #3)
>
> Upped:
>
> http://gist.github.com/54758

Putting the message at the end is a great idea! I still think that
naming the method (and thus starting the block) with "on_failure" is a
bit misleading when read aloud...but maybe that's not a big concern
for you guys?

What do you think about this modification to #3?:
http://gist.github.com/54770
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-29 22:57
(Received via mailing list)
On Thu, Jan 29, 2009 at 3:27 PM, aslak hellesoy
<aslak.hellesoy@gmail.com> wrote:
>>>> > On Thu, Jan 29, 2009 at 12:00 PM, <r_j_h_box-sf@yahoo.com> wrote:
>>>> >>
>>>> >>   end
>>>> >> informative.
>>>> >>
>>>> > it "does something" do
>>>> on_error "bla" do
>>>
>>> http://gist.github.com/54726
>>
>> I'll take that gist and raise you one:
>>
>> http://gist.github.com/54750 (Suggestion #3)
>>
>
> Upped:
>
> http://gist.github.com/54758

Somebody move this to a ticket please :)
942a74b1b71ca62234ae690b44699ebc?d=identicon&s=25 unknown (Guest)
on 2009-01-29 23:09
(Received via mailing list)
I'm starting to wonder if this is a good idea to begin with.  I had
started to suggest nested it()s:

it "..." do

  it "something more" do
  end
end

... but it's already handled by the existing nested describe(),
before(), it() system.

I guess if we were shooting for flexibility, we might ask what the
simplest way is to the next level of flex?  One route might be a detail
level on describe() blocks.  So, suggestion #5:

http://gist.github.com/54764

Then when generating spec docs, you could vary the detail level on
what's generated, to generate only up to level-2 documentation, or to
generate full-detail documentation if you prefer.

Nested it()s could do the same thing, I suppose.  Not sure what side
effects would come into play.  Pretty clearly, a nested it() wouldn't
have an embedded transaction or respect the outer before() block, since
the outer if() would handle both those things.

Randy




________________________________
From: aslak hellesoy <aslak.hellesoy@gmail.com>
To: rspec-users <rspec-users@rubyforge.org>
Sent: Thursday, January 29, 2009 1:27:02 PM
Subject: Re: [rspec-users] simple == with prettier error messages + good
documentation

On Thu, Jan 29, 2009 at 10:14 PM, David Chelimsky <dchelimsky@gmail.com>
wrote:
>>> >>
>>> >>   foo.should == bar (or foo.should_not be_nil)
>>> >>
>>> >>
>>> >> I guess I could get more fine-grained with my it()'s, but I've been
>>> >   with_message "this is a more specific message" do
>>>  ...
>> it 'should do something' do
>
> I'll take that gist and raise you one:
>
> http://gist.github.com/54750 (Suggestion #3)
>

Upped:

http://gist.github.com/54758
942a74b1b71ca62234ae690b44699ebc?d=identicon&s=25 unknown (Guest)
on 2009-01-29 23:11
(Received via mailing list)
I do like try_to do...end.or_report - much better for the case where I'm
not trying to generate documentation.


I'm not sure it works so well for the case where I want to generate an
extra element of documentation, though this might be just an
implementation detail.

.or_report
.or_document
.or_gripe
.or_complain

One of the first two could generate documentation, and one of the others
might generate none.

Randy




________________________________
From: Nick Hoffman <nick@deadorange.com>

On 29/01/2009, at 4:27 PM, aslak hellesoy wrote:
> http://gist.github.com/54758

Putting the message at the end is a great idea! I still think that
naming the method (and thus starting the block) with "on_failure" is a
bit misleading when read aloud...but maybe that's not a big concern for
you guys?

What do you think about this modification to #3?:
http://gist.github.com/54770
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-29 23:55
(Received via mailing list)
On Thu, Jan 29, 2009 at 3:50 PM,  <r_j_h_box-sf@yahoo.com> wrote:
>
> Nested it()s could do the same thing, I suppose.  Not sure what side effects
> would come into play.  Pretty clearly, a nested it() wouldn't have an
> embedded transaction or respect the outer before() block, since the outer
> if() would handle both those things.

For a number of technical reasons, the nested code example (it()) is a
non-starter.

The idea of having some means of flagging examples so they report
differently is possible. Right now, example groups and examples each
take a hash in their declarations:

describe "an example group", :with => "some, :args => "like this" do
  it "an example", :with => "some", :other => "args"
  ...
  end
end

Those are accessible to the runner, and are used internally for
assorted filtering. As of this moment, they don't make it past the
reporters, because the object that goes to formatters is a wrapper w/
limited information instead of the real example. We could add the
options to that wrapper though, and then you'd be able to write a
custom formatter that would do anything you like based on the
configuration of those options.

WDYT about that?
Cdf378de2284d8acf137122e541caa28?d=identicon&s=25 Matt Wynne (mattwynne)
on 2009-01-30 01:22
(Received via mailing list)
On 29 Jan 2009, at 22:20, David Chelimsky wrote:

>>>>>>> of
>>>>>>> (hm, not very informative)
>>>>>>>> got "failure to setup foo" (using ==)
>>>>>>> error
>>>>>>> the
>>>>>>> sense
>>>>>>> without
>>>>>>> the detail.

I really do suggest setting up the context in the before() block of a
describe (example group), then using each 'it' (example) to hold an
assertion. It's a tried and tested way of doing it for me, but it does
take some getting used to when you move over from xUnit style tests.

>>>>>> end
>>>>> end
>>>>
>>>> http://gist.github.com/54726
>>>
>>> I'll take that gist and raise you one:
>>>
>>> http://gist.github.com/54750 (Suggestion #3)
>>>
>>
>> Upped:
>>
>> http://gist.github.com/54758

I do find myself hitting this sometimes, especially when I use rspec
assertions in Cucumber steps - I usually stick to one assertion per
example in RSpec so it isn't a problem when I'm writing unit tests.

However we do have a couple of alternatives already, without extending
RSpec any more:
(1) just use traditional test/unit assertions instead (works OK in
Cucumber steps, though obviously they don't read quite so well)
(2) write a custom matcher

We could also pretty easily extend the existing matchers, just taking
a failure message in the factory method, e.g:

http://gist.github.com/54849

I agree this discussion should move to a ticket. But it seems I am to
lazy to do it myself ;)


Matt Wynne
http://blog.mattwynne.net
http://www.songkick.com
8f6f95c4bd64d5f10dfddfdcd03c19d6?d=identicon&s=25 Rick Denatale (rdenatale)
on 2009-01-31 12:35
(Received via mailing list)
On Thu, Jan 29, 2009 at 7:33 PM, David Chelimsky
<dchelimsky@gmail.com>wrote:

>
> > I agree this discussion should move to a ticket. But it seems I am to
> lazy
> > to do it myself ;)
>
> http://rspec.lighthouseapp.com/projects/5645-rspec...
>
>
Nice to see you step back from the list of suggestions and nicely
summarize
the problem.

I think that using the example docstring works well enough if you are
religulous about the 1 expectation per example approach.

In general I think what is wanted is to have a string to override what
is
reported on a failed expectation, which prompted me to make yet another
suggestion

http://gist.github.com/55165

--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-01-31 12:51
(Received via mailing list)
On Fri, Jan 30, 2009 at 12:00 PM, Rick DeNatale
<rick.denatale@gmail.com> wrote:
> Nice to see you step back from the list of suggestions and nicely summarize
> the problem.
>
> I think that using the example docstring works well enough if you are
> religulous about the 1 expectation per example approach.
>
> In general I think what is wanted is to have a string to override what is
> reported on a failed expectation, which prompted me to make yet another
> suggestion
>
> http://gist.github.com/55165

Now that I've started a ticket, would you kindly put your response in
the ticket instead?

Thanks
8f6f95c4bd64d5f10dfddfdcd03c19d6?d=identicon&s=25 Rick Denatale (rdenatale)
on 2009-01-31 16:06
(Received via mailing list)
On Fri, Jan 30, 2009 at 1:36 PM, David Chelimsky
<dchelimsky@gmail.com>wrote:

> >>
> > suggestion
> >
> > http://gist.github.com/55165
>
> Now that I've started a ticket, would you kindly put your response in
> the ticket instead?
>
> Thanks


Done


--
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
942a74b1b71ca62234ae690b44699ebc?d=identicon&s=25 unknown (Guest)
on 2009-01-31 19:14
(Received via mailing list)
Passing the it() and describe() args through to the reporting layer
sounds like a great idea.  I've thought more about the reporting-levels
approach.  The numeric as I postulated earlier is probably not the most
useful thing.

As a product manager, I want to be able to flexibly present the
appropriate documentation to different audiences, to avoid the need for
maintaining multiple docs.

describe "an example group", :audience => :execs
describe "an example group", :audience => :owner
describe "an example group", :audience => :users
describe "an example group", :audience => :developers

These could be entirely ad-hoc, allowing the developer to select the
desired labels for the situation at hand.

When generating reports, you'd state which audiences to generate docs
for (maybe with a default set coded into a helper?).

When generating reports to text, any examples or groups that were
skipped would be simply tabulated, and the summary would be displayed at
the bottom:

  Skipped 7 example groups and 27 examples for Users.
  Skipped 3 example groups and 12 examples for Developers.


And when generating reports to HTML, any docs for not-specified roles
could simply be folded/hidden by default, allowing the viewer to drill
down.  Of course, this could be skipped by setting an option for static
HTML purposes, and the skipped items could be reported in the same way
as the text version.

Does this seem like a useful way to treat the various detail levels of
documentation we want to present to different audiences?

Randy




________________________________
From: David Chelimsky <dchelimsky@gmail.com>

Randy:
> http://gist.github.com/54764
>
> Then when generating spec docs, you could vary the detail level on what's
> generated, to generate only up to level-2 documentation, or to generate
> full-detail documentation if you prefer.
>
> Nested it()s could do the same thing, I suppose.  Not sure what side effects
> would come into play.  Pretty clearly, a nested it() wouldn't have an
> embedded transaction or respect the outer before() block, since the outer
> if() would handle both those things.

For a number of technical reasons, the nested code example (it()) is a
non-starter.

The idea of having some means of flagging examples so they report
differently is possible. Right now, example groups and examples each
take a hash in their declarations:

describe "an example group", :with => "some, :args => "like this" do
  it "an example", :with => "some", :other => "args"
  ...
  end
end

Those are accessible to the runner, and are used internally for
assorted filtering. As of this moment, they don't make it past the
reporters, because the object that goes to formatters is a wrapper w/
limited information instead of the real example. We could add the
options to that wrapper though, and then you'd be able to write a
custom formatter that would do anything you like based on the
configuration of those options.

WDYT about that?
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2009-02-02 09:47
(Received via mailing list)
On Sat, Jan 31, 2009 at 12:02 PM,  <r_j_h_box-sf@yahoo.com> wrote:
> describe "an example group", :audience => :users
>
>   Skipped 7 example groups and 27 examples for Users.
>   Skipped 3 example groups and 12 examples for Developers.
>
> And when generating reports to HTML, any docs for not-specified roles could
> simply be folded/hidden by default, allowing the viewer to drill down.  Of
> course, this could be skipped by setting an option for static HTML purposes,
> and the skipped items could be reported in the same way as the text version.
>
> Does this seem like a useful way to treat the various detail levels of
> documentation we want to present to different audiences?

Hey Randy,

I just committed code that makes the options available to the
formatters for both example groups and individual examples. If you
grab the latest from git, take a look at
examples/passing/filtered_formatter_example.rb and you'll see how it
works.

This should let you experiment with different ways of filtering
things. The approach you suggest does seem sound, but I'm not prepared
to add any first class support for anything like that to rspec proper
until its been proven effective.

I'm also interested in figuring out how to apply this to the actual
run, not just the reporting. i.e. use options as a tagging system to
determine which groups and/or individual examples to run. This would
require some entry point earlier in the process, and I'm not quite
sure the best approach yet. Ideas are welcome.

Cheers,
David
This topic is locked and can not be replied to.