How to spec a (Trollop-based) binary ’s internal state?


#1

I’m trying to spec a ‘binary’, and as previously discussed on this list,
I’m trying to do it ‘from outside’ – i.e., by calling it with Kernel#`
and observing the (side-)effects.

Unfortunately, this doesn’t really let me spec expectations about its
internals. Let’s assume I have a -d flag and I parse it with Trollop to
set Conf.debug:

opts = Trollop::options do
opt :debug, ‘Print debug info’, :default => false
end
Conf.debug = opts[:debug]

To have a spec that actually expects
this, I’d write something like this:

it ‘should set the debug option properly’ do
binary -d
Conf.should_receive(:debug=).with true
end

This obviously doesn’t work, as the Kernel#` call is executed in
a context that is not visible from the current Ruby interpreter.
I assume that eval()-ing the binary (much like Rick B. does¹
for specing flog) would be the way to go, but then I don’t know how
to pass the -d flag to Trollop.

¹
http://rubyhoedown2008.confreaks.com/11-rick-bradley-flog-test-new.html

What would be the best practice in this case? Prepare ARGV for Trollop
so that it believes we are calling the binary with the -d flag and then
eval the binary, or run the binary with Kernel#`, pass it something
that should generate debug output and then check whether the output
was generated (rather than check whether the binary set Conf.debug)?

– Shot


#2

“Shot (Piotr S.)” removed_email_address@domain.invalid writes:

end
This obviously doesn’t work, as the Kernel#` call is executed in
that should generate debug output and then check whether the output
was generated (rather than check whether the binary set Conf.debug)?

– Shot

When you use backticks to execute the binary, it runs in a separate
process, so you can’t use mock expectations like that.

I think you’re kind of missing the intention though…what I think you
want to do is write features with cucumber that use the built binary
from the outside. But when you write code-level examples, you are not
going to run the binary. You’re just writing examples at a lower level,
directly for the objects. Make sense?

Pat


#3

Pat M.:

“Shot (Piotr S.)” removed_email_address@domain.invalid writes:

it ‘should set the debug option properly’ do
binary -d
Conf.should_receive(:debug=).with true
end

When you use backticks to execute the binary, it runs in
a separate process, so you can’t use mock expectations like that.

Right, that’s why I suggested I could Kernel#eval the binary’s contents
in the current process instead. This would require tricking Trollop, but
I assume I could trick it by hand-crafting ARGV.

I think you’re kind of missing the intention though…

Yeah, I thought so – that’s why I’m asking. :slight_smile: The previous question
ended up with the conclusion that RSpecing a ‘binary’ through Kernel#`
is a sane way to go:

¹
http://groups.google.com/group/rspec/browse_thread/thread/66dc48522014c5eb

what I think you want to do is write features with
cucumber that use the built binary from the outside.

Ah, Cucumber. :slight_smile: I’ve yet to write my first story; I somehow assumed
these are more oriented toward non-programmers and/or for driving
implementation on a rather higher level, and have to be ‘implemented’
on the RSpec level somehow anyway. I take that these assumptions are
flawed. :slight_smile:

But when you write code-level examples, you are not going to run the
binary. You’re just writing examples at a lower level, directly for
the objects. Make sense?

Hm, I guess I simply need to read up on Cucumber and stories;
I still can’t see how this kind of specification would differ
(in the end) from Kernel#`-based RSpec – i.e., why implementing
the code behind these stories would side-step the underlying issue.
(But the last time I read on stories was pre-Cucumber.)

Is there an example of story/Cucumber-based specification
of a binary’s behaviour available somewhere out there?

– Shot


#4

On Nov 07, 2008, at 5:24 pm, Shot (Piotr S.) wrote:

Right, that’s why I suggested I could Kernel#eval the binary’s
contents
in the current process instead. This would require tricking Trollop,
but
I assume I could trick it by hand-crafting ARGV.

You might find Rick B.'s talk[1] on regression testing flog
useful here. He starts by speccing the binary at a Ruby level. Note,
however, that later in the talk he admits he made a mistake in not
writing end-to-end tests soon enough. I think he should have run the
binary as a black box and just pushed binary code into lib sooner.

Ah, Cucumber. :slight_smile: I’ve yet to write my first story; I somehow assumed
these are more oriented toward non-programmers and/or for driving
implementation on a rather higher level, and have to be ‘implemented’
on the RSpec level somehow anyway. I take that these assumptions are
flawed. :slight_smile:

Yes and no. I wouldn’t say they are oriented towards the end user -
Cucumber files are code and must be formatted strictly. But they are
readable enough that they can be understood by the end user. You
don’t actually need to use RSpec in Cucumber steps - just raising an
exception when you see something you don’t like is enough - but the
matchers make the steps readable.

I recommending stopping the line until you have your current
functionality described in Cucumber feature definitions, it will save
you a lot of pain in the long run.

Hm, I guess I simply need to read up on Cucumber and stories;
I still can’t see how this kind of specification would differ
(in the end) from Kernel#`-based RSpec – i.e., why implementing
the code behind these stories would side-step the underlying issue.
(But the last time I read on stories was pre-Cucumber.)

Cucumber with Kernel#and RSpec with Kernel# would achieve the same
thing, the difference being the structure of the files. Cucumber is
optimised to deal with higher-lever concepts, where natural language
is a more important part of the spec. RSpec is more suited to lower-
level specs where the aim is to cover every edge case or object
interaction. This would be too cumbersome with Cucumber.

Is there an example of story/Cucumber-based specification
of a binary’s behaviour available somewhere out there?

There is now :slight_smile:

Well, it’s a really crappy, trivial example I just made it about 10
mins. No use of STDERR, no file input/output, no network
connections. Flagrant abuse of FIT tables. Oh, and wanton sharing of
data between steps. But apart from that it’s model code.

Don’t take any of this as best practice! It’s just to show the
concept (seeing how “sort” is presumably written in C):

sort.feature

Feature: sort
So that I can search faster
As a person that likes searching
I want to sort data

 Scenario: data ordered already
   Given the following data:
     | STDIN  |
     | apple  |
     | banana |
     | cherry |
     | mango  |
     | pear   |
   When I sort the data
   Then the output should be:
     | STDOUT |
     | apple  |
     | banana |
     | cherry |
     | mango  |
     | pear   |

 Scenario: data out of order
   Given the following data:
     | STDIN  |
     | pear   |
     | mango  |
     | apple  |
     | cherry |
     | banana |
   When I sort the data
   Then the output should be:
     | STDOUT |
     | apple  |
     | banana |
     | cherry |
     | mango  |
     | pear   |

steps/sort.rb

require ‘spec’

Given %r/^the following data:$/ do |table|
@stdin = table.hashes.map { |r| r[“STDIN”] }.join("\n")
end

When %r/^I sort the data$/ do
@result = sort <<END\n#@stdin\nEND\n.split("\n")
end

Then %r/^the output should be:$/ do |table|
@expected = table.hashes.map { |r| r[“STDOUT”] }
@result.should == @expected
end

Hope this helps.

Ashley

[1]
http://rubyhoedown2008.confreaks.com/11-rick-bradley-flog-test-new.html


http://www.patchspace.co.uk/


#5

On 14 Nov 2008, at 20:56, Shot (Piotr S.) wrote:

Seeing as my binary grew to over 80 lines, I guess my best bet is to
pull out the gist of it into a separate Runner class, spec it (with
mocks/stubs/fixtures) so it runs in a sane time and then only create
a think binary wrapper around it (which will basically parse the
options, set the config and then run the runner).

Have a look at the source code for cucumber’s binary, or rspec’s
‘spec’ command. They’re both about four lines long creating a
(eminently spec-able) class that does the rest.

cheers,
Matt


#6

Ashley M.:

On Nov 07, 2008, at 5:24 pm, Shot (Piotr S.) wrote:

Right, that’s why I suggested I could Kernel#eval the binary’s contents
in the current process instead. This would require tricking Trollop, but
I assume I could trick it by hand-crafting ARGV.

You might find Rick B.'s talk[1] on regression testing flog
useful here. He starts by speccing the binary at a Ruby level.

Heh – I brought exactly this talk up twice when talking about specing
binaries here (it was what pointed me in the direction that it would
be actually good to spec/test them, actually). :slight_smile:

Note, however, that later in the talk he admits he made a mistake in
not writing end-to-end tests soon enough. I think he should have run
the binary as a black box and just pushed binary code into lib sooner.

Right. I think I agree that’s the best
(and most elegant) way to test binaries.

In my case, asfter some specing and some coding, it turns out I can’t
really do this the way I wanted; most of the cases I need to test for
are triggered with input that makes the binary run a long time (it’s
the code behind my PhD thesis, so tons of computations). As I’m testing
a black box with Kernel#`, I can’t mock/stub anything out.

Seeing as my binary grew to over 80 lines, I guess my best bet is to
pull out the gist of it into a separate Runner class, spec it (with
mocks/stubs/fixtures) so it runs in a sane time and then only create
a think binary wrapper around it (which will basically parse the
options, set the config and then run the runner).

Thanks a lot for the nice Cucumber example! It was most informative.

– Shot


#7

On Nov 14, 2008, at 8:56 pm, Shot (Piotr S.) wrote:

Heh – I brought exactly this talk up twice when talking about specing
binaries here (it was what pointed me in the direction that it would
be actually good to spec/test them, actually). :slight_smile:

Ha, so you did :slight_smile:

There was a delay after I read your email to when I finally replied,
in that time it clearly left my head…

In my case, asfter some specing and some coding, it turns out I can’t
really do this the way I wanted; most of the cases I need to test for
are triggered with input that makes the binary run a long time (it’s
the code behind my PhD thesis, so tons of computations). As I’m
testing
a black box with Kernel#`, I can’t mock/stub anything out.

Well, like Matt says, all the binary needs to do is get your code up
and running. For that, I imagine you can use selectively-chosen
inputs that don’t produce long run times. The cucumber files just
need to prove that all the command line options work, and that the app
is directing IO to and from the right places. The details of the
algorithms can be dealt with in your spec suite.

Ashley


http://www.patchspace.co.uk/