Forum: RSpec How to spec a (Trollop-based) binary ’s internal state?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Cec345a59245af9d06e4438a413f4eb5?d=identicon&s=25 Shot (Piotr Szotkowski) (Guest)
on 2008-11-07 14:53
(Received via mailing list)
I’m trying to spec a ‘binary’, and as previously discussed on this list,
I’m trying to do it ‘from outside’ – i.e., by calling it with Kernel#`
and observing the (side-)effects.

Unfortunately, this doesn’t really let me spec expectations about its
internals. Let’s assume I have a -d flag and I parse it with Trollop to
set Conf.debug:

opts = Trollop::options do
  opt :debug, 'Print debug info', :default => false
end
Conf.debug = opts[:debug]

To have a spec that actually expects
this, I’d write something like this:

it 'should set the debug option properly' do
  `binary -d`
  Conf.should_receive(:debug=).with true
end

This obviously doesn’t work, as the Kernel#` call is executed in
a context that is not visible from the current Ruby interpreter.
I assume that eval()-ing the binary (much like Rick Bradley does¹
for specing flog) would be the way to go, but then I don’t know how
to pass the -d flag to Trollop.

¹
http://rubyhoedown2008.confreaks.com/11-rick-bradl...

What would be the best practice in this case? Prepare ARGV for Trollop
so that it believes we are calling the binary with the -d flag and then
eval the binary, or run the binary with Kernel#`, pass it something
that should generate debug output and then check whether the output
was generated (rather than check whether the binary set Conf.debug)?

-- Shot
42172acdf3c6046f84d644cb0b94642c?d=identicon&s=25 Pat Maddox (pergesu)
on 2008-11-07 17:11
(Received via mailing list)
"Shot (Piotr Szotkowski)" <shot@hot.pl> writes:

> end
> This obviously doesn’t work, as the Kernel#` call is executed in
> that should generate debug output and then check whether the output
> was generated (rather than check whether the binary set Conf.debug)?
>
> -- Shot

When you use backticks to execute the binary, it runs in a separate
process, so you can't use mock expectations like that.

I think you're kind of missing the intention though...what I think you
want to do is write features with cucumber that use the built binary
from the outside.  But when you write code-level examples, you are not
going to run the binary.  You're just writing examples at a lower level,
directly for the objects.  Make sense?

Pat
Cec345a59245af9d06e4438a413f4eb5?d=identicon&s=25 Shot (Piotr Szotkowski) (Guest)
on 2008-11-07 18:36
(Received via mailing list)
Pat Maddox:

> "Shot (Piotr Szotkowski)" <shot@hot.pl> writes:

>> it 'should set the debug option properly' do
>>   `binary -d`
>>   Conf.should_receive(:debug=).with true
>> end

> When you use backticks to execute the binary, it runs in
> a separate process, so you can't use mock expectations like that.

Right, that’s why I suggested I could Kernel#eval the binary’s contents
in the current process instead. This would require tricking Trollop, but
I assume I could trick it by hand-crafting ARGV.

> I think you're kind of missing the intention though...

Yeah, I thought so – that’s why I’m asking. :) The previous question
ended up with the conclusion that RSpecing a ‘binary’ through Kernel#`
is a sane way to go:

¹
http://groups.google.com/group/rspec/browse_thread...

> what I think you want to do is write features with
> cucumber that use the built binary from the outside.

Ah, Cucumber. :) I’ve yet to write my first story; I somehow assumed
these are more oriented toward non-programmers and/or for driving
implementation on a rather higher level, and have to be ‘implemented’
on the RSpec level somehow anyway. I take that these assumptions are
flawed. :)

> But when you write code-level examples, you are not going to run the
> binary. You're just writing examples at a lower level, directly for
> the objects. Make sense?

Hm, I guess I simply need to read up on Cucumber and stories;
I still can’t see how this kind of specification would differ
(in the end) from Kernel#`-based RSpec – i.e., why implementing
the code behind these stories would side-step the underlying issue.
(But the last time I read on stories was pre-Cucumber.)

Is there an example of story/Cucumber-based specification
of a binary’s behaviour available somewhere out there?

-- Shot
2ce9c0106b5851b2294ba5eb9f5c04bd?d=identicon&s=25 Ashley Moran (Guest)
on 2008-11-10 12:24
(Received via mailing list)
On Nov 07, 2008, at 5:24 pm, Shot (Piotr Szotkowski) wrote:

> Right, that’s why I suggested I could Kernel#eval the binary’s
> contents
> in the current process instead. This would require tricking Trollop,
> but
> I assume I could trick it by hand-crafting ARGV.

You might find Rick Bradley's talk[1] on regression testing flog
useful here.  He starts by speccing the binary at a Ruby level.  Note,
however, that later in the talk he admits he made a mistake in not
writing end-to-end tests soon enough.  I think he should have run the
binary as a black box and just pushed binary code into lib sooner.

> Ah, Cucumber. :) I’ve yet to write my first story; I somehow assumed
> these are more oriented toward non-programmers and/or for driving
> implementation on a rather higher level, and have to be ‘implemented’
> on the RSpec level somehow anyway. I take that these assumptions are
> flawed. :)

Yes and no.  I wouldn't say they are oriented towards the end user -
Cucumber files are code and must be formatted strictly.  But they are
readable enough that they can be understood by the end user.  You
don't actually need to use RSpec in Cucumber steps - just raising an
exception when you see something you don't like is enough - but the
matchers make the steps readable.

I recommending stopping the line until you have your current
functionality described in Cucumber feature definitions, it will save
you a lot of pain in the long run.


> Hm, I guess I simply need to read up on Cucumber and stories;
> I still can’t see how this kind of specification would differ
> (in the end) from Kernel#`-based RSpec – i.e., why implementing
> the code behind these stories would side-step the underlying issue.
> (But the last time I read on stories was pre-Cucumber.)

Cucumber with Kernel#` and RSpec with Kernel#` would achieve the same
thing, the difference being the structure of the files.  Cucumber is
optimised to deal with higher-lever concepts, where natural language
is a more important part of the spec.  RSpec is more suited to lower-
level specs where the aim is to cover every edge case or object
interaction.  This would be too cumbersome with Cucumber.


> Is there an example of story/Cucumber-based specification
> of a binary’s behaviour available somewhere out there?

There is now :)

Well, it's a really crappy, trivial example I just made it about 10
mins.  No use of STDERR, no file input/output, no network
connections.  Flagrant abuse of FIT tables.  Oh, and wanton sharing of
data between steps.  But apart from that it's model code.

Don't take any of this as best practice!  It's just to show the
concept (seeing how "sort" is presumably written in C):

# sort.feature

   Feature: sort
     So that I can search faster
     As a person that likes searching
     I want to sort data

     Scenario: data ordered already
       Given the following data:
         | STDIN  |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |
       When I sort the data
       Then the output should be:
         | STDOUT |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |

     Scenario: data out of order
       Given the following data:
         | STDIN  |
         | pear   |
         | mango  |
         | apple  |
         | cherry |
         | banana |
       When I sort the data
       Then the output should be:
         | STDOUT |
         | apple  |
         | banana |
         | cherry |
         | mango  |
         | pear   |


# steps/sort.rb

   require 'spec'

   Given %r/^the following data:$/ do |table|
     @stdin = table.hashes.map { |r| r["STDIN"] }.join("\n")
   end

   When %r/^I sort the data$/ do
     @result = `sort <<END\n#@stdin\nEND\n`.split("\n")
   end

   Then %r/^the output should be:$/ do |table|
     @expected = table.hashes.map { |r| r["STDOUT"] }
     @result.should == @expected
   end

Hope this helps.

Ashley

[1]
http://rubyhoedown2008.confreaks.com/11-rick-bradl...


--
http://www.patchspace.co.uk/
http://aviewfromafar.net/
Cec345a59245af9d06e4438a413f4eb5?d=identicon&s=25 Shot (Piotr Szotkowski) (Guest)
on 2008-11-14 21:55
(Received via mailing list)
Ashley Moran:

> On Nov 07, 2008, at 5:24 pm, Shot (Piotr Szotkowski) wrote:

>> Right, that’s why I suggested I could Kernel#eval the binary’s contents
>> in the current process instead. This would require tricking Trollop, but
>> I assume I could trick it by hand-crafting ARGV.

> You might find Rick Bradley's talk[1] on regression testing flog
> useful here. He starts by speccing the binary at a Ruby level.

Heh – I brought exactly this talk up twice when talking about specing
binaries here (it was what pointed me in the direction that it would
be actually good to spec/test them, actually). :)

> Note, however, that later in the talk he admits he made a mistake in
> not writing end-to-end tests soon enough. I think he should have run
> the binary as a black box and just pushed binary code into lib sooner.

Right. I think I agree that’s the best
(and most elegant) way to test binaries.

In my case, asfter some specing and some coding, it turns out I can’t
really do this the way I wanted; most of the cases I need to test for
are triggered with input that makes the binary run a long time (it’s
the code behind my PhD thesis, so tons of computations). As I’m testing
a black box with Kernel#`, I can’t mock/stub anything out.

Seeing as my binary grew to over 80 lines, I guess my best bet is to
pull out the gist of it into a separate Runner class, spec it (with
mocks/stubs/fixtures) so it runs in a sane time and then only create
a think binary wrapper around it (which will basically parse the
options, set the config and then run the runner).

Thanks a lot for the nice Cucumber example! It was most informative.

-- Shot
Cdf378de2284d8acf137122e541caa28?d=identicon&s=25 Matt Wynne (mattwynne)
on 2008-11-15 00:01
(Received via mailing list)
On 14 Nov 2008, at 20:56, Shot (Piotr Szotkowski) wrote:
> Seeing as my binary grew to over 80 lines, I guess my best bet is to
> pull out the gist of it into a separate Runner class, spec it (with
> mocks/stubs/fixtures) so it runs in a sane time and then only create
> a think binary wrapper around it (which will basically parse the
> options, set the config and then run the runner).

Have a look at the source code for cucumber's binary, or rspec's
'spec' command. They're both about four lines long creating a
(eminently spec-able) class that does the rest.

cheers,
Matt
2ce9c0106b5851b2294ba5eb9f5c04bd?d=identicon&s=25 Ashley Moran (Guest)
on 2008-11-15 12:00
(Received via mailing list)
On Nov 14, 2008, at 8:56 pm, Shot (Piotr Szotkowski) wrote:

> Heh – I brought exactly this talk up twice when talking about specing
> binaries here (it was what pointed me in the direction that it would
> be actually good to spec/test them, actually). :)


Ha, so you did :)

There was a delay after I read your email to when I finally replied,
in that time it clearly left my head...


> In my case, asfter some specing and some coding, it turns out I can’t
> really do this the way I wanted; most of the cases I need to test for
> are triggered with input that makes the binary run a long time (it’s
> the code behind my PhD thesis, so tons of computations). As I’m
> testing
> a black box with Kernel#`, I can’t mock/stub anything out.

Well, like Matt says, all the binary needs to do is get your code up
and running.  For that, I imagine you can use selectively-chosen
inputs that don't produce long run times.  The cucumber files just
need to prove that all the command line options work, and that the app
is directing IO to and from the right places.  The details of the
algorithms can be dealt with in your spec suite.

Ashley


--
http://www.patchspace.co.uk/
http://aviewfromafar.net/
This topic is locked and can not be replied to.