Test::Unit Patch that allows test methods to be executed seq

Ruby
1

This patch to Test::Unit allows you to create TestCases whose test
methods will be executed in the order you’ve defined them, rather than
alphabetically, which is what Test::Unit does today.

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

Thus the foloowing TestCase will print “EFGH”:

class TC2_Sequential < Test::Unit::TestCase
execute :sequentially
def test_b; print ‘E’; end
def test_a; print ‘F’; end
def test_d; print ‘G’; end
def test_c; print ‘H’; end
end

I realize that there has been some controversy in the past about
whether is a good idea or not. This debate has hinged on issues about
proper unit testing.

However, with Watir (http://wtr.rubyforge.org), many people are now
using Test::Unit for system acceptance testing, which presents a
different context for the value of ordering tests.

My hope is that some people will find this to be useful. My thanks go
to Why’s MetaClass article, which gave me the info i needed to create
this.

Bret

2

On May 21, 2006, at 7:45 PM, [email protected] wrote:

proper unit testing.
Why not subclass TestCase and create SequentialTestCase or
WatirTestCase? I don’t see any reason why this functionality needs to
be folded into test/unit but I see your argument for Watir. We should
be trying our damnedest to make test/unit thinner, not thicker.

–
_why: zenspider’s most intense moments of solice are immediately
following the slaughter […]
_why: that topknot’s the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak

3

Ryan D. said:

Why not subclass TestCase and create SequentialTestCase or
WatirTestCase? I don’t see any reason why this functionality needs to
be folded into test/unit but I see your argument for Watir. We should
be trying our damnedest to make test/unit thinner, not thicker.

Unfortunately, subclassing TestCase is not a good idea. All subclasses
of TestCase are presumed to be concrete tests and you will get errors
if the subclass does not have a test method. (Try it out if you don’t
believe me.) There is no provision for “Virtual” testcases, which is
what we would want here.

I think the ideal solution would be to refactor the TestCase code along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

This would keep the code from bulking up and allow users to add custom
behavior without modifying the class itself. For example, there has
been some interest expressed in randomizing the order of the test
methods. This suggestion would allow users to easily do this.

Bret

4

On May 22, 2006, at 11:31 AM, [email protected] wrote:

what we would want here.
I do it all the time. The simple solution is to put a “test_dummy” in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

I think the ideal solution would be to refactor the TestCase code
along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

I disagree. As I said before, we need less code, not more. (I do
wholly agree that test/unit needs refactoring, but to make it thinner
not thicker)

5

Ryan D. wrote:

On May 21, 2006, at 7:45 PM, [email protected] wrote:

proper unit testing.
Why not subclass TestCase and create SequentialTestCase or
WatirTestCase?

The problem with subclassing is that it doesn’t combine well with other
potential Test::Unit add-ins. (For that matter, I suspect the patch way
doesn’t combine well either, but I haven’t looked at the details).

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can’t use my FlexMock subclass with
his Watir subclass.

Rather, make the add-ins modules, then the user can mix-in whatever
behavior they need. For example …

class MyTest < Test::Unit::TestCase
include FlexMock::TestCase
include Watir::TestCase

 def test_using_both_watir_and_flexmock_extensions
   ...
 end

end

Now, the flexmock one ties in by overriding teardown, so any user
supplied setup/teardown methods should carefully invoke super (which
they probably should do anyways).

– Jim W.

6

On May 21, 2006, at 7:45 PM, [email protected] wrote:

This patch to Test::Unit allows you to create TestCases whose test
methods will be executed in the order you’ve defined them, rather than
alphabetically, which is what Test::Unit does today.

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don’t see a patch there.

–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

7

Ryan sez:

I do it all the time. The simple solution is to put a “test_dummy” in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

This has been suggested before to the Watir users. Many of them don’t
like the fact that this results in an incorrect test case count.

I think the ideal solution would be to refactor the TestCase code
along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

I disagree. As I said before, we need less code, not more. (I do
wholly agree that test/unit needs refactoring, but to make it thinner
not thicker)

My suggestion is only to add a new seam, but i guess that amounts to a
bit of code. But only a bit.

Another reason that i packaged the code as a mod to TestCase directly
is that it makes it easier for people with large test suites to plug it
in. So it is more convenient for Watir users than a subclass.

However, I’m not quite sure whether you are saying
a) i (ryan) would prefer to do it a different way
b) i don’t want to see this change made to the core test/unit code in
ruby
c) it is a bad idea for anyone to use this mod

I understand (a). I can see your reasons for (b), and actually have
been eager for feedback on whether this might be a good idea or not.
(although i did get some positive noises offline from Nathaniel T.
– the test/unit author). i can see no grounds for © – if this is
your position, please elaborate.

Bret

8

On 5/22/06, Eric H. [email protected] wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don’t see a patch there.

Click “Download Patch.” Or click the attachments tab. I also just added
a
link at the bottom of the page.

I have been updating the code on this page based on comments that i get.

Here’s the current version of the code:

require ‘test/unit’

module Test
module Unit
class TestCase
@@order = :alphabetically
class << self
attr_accessor :test_methods, :order
def test_methods
@test_methods ||= []
end
def order
@order || @@order
end
def default_order= order
@@order = order
end
def sorted_test_methods
case order
when :alphabetically: test_methods.sort
when :sequentially: test_methods
when :reversed_sequentially: test_methods.reverse
when :reversed_alphabetically: test_methods.sort.reverse
else raise ArgumentError, “Execute option not supported:
#{@order}”
end
end
def suite
suite = TestSuite.new(name)
sorted_test_methods.each do |test|
catch :invalid_test do
suite << new(test)
end
end
if (suite.empty?)
catch :invalid_test do
suite << new(:default_test)
end
end
return suite
end
def method_added id
name = id.id2name
test_methods << name if name =~ /^test./
end
def execute order
@order = order
end
end
end
end
end

9

On May 22, 2006, at 2:47 PM, Bret P. wrote:

On 5/22/06, Eric H. [email protected] wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don’t see a patch there.

Click “Download Patch.” Or click the attachments tab. I also just
added a
link at the bottom of the page.

I think Eric’s point was that it isn’t a patch.

10

On May 22, 2006, at 3:16 PM, Ryan D. wrote:

added a
link at the bottom of the page.

I think Eric’s point was that it isn’t a patch.

Exactly. Its hard to see how this fits into Test::Unit standing all
alone like that.

–
Eric H. - [email protected] - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

11

On May 22, 2006, at 11:58 AM, [email protected] wrote:

Ryan sez:

I do it all the time. The simple solution is to put a “test_dummy” in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

This has been suggested before to the Watir users. Many of them don’t
like the fact that this results in an incorrect test case count.

Quite frankly, I don’t really care that they consider it to output an
“incorrect” test case count. It works for many of us and is a fair
tradeoff. I’d much rather have the requirement removed from test/unit
outright, but I didn’t design test/unit and I too disagree with
requiring it (but! see below!). I however disagree wholly with making
test/unit MORE complex under all but the most optimal of returns. I
don’t think your suggestion matches that requirement esp since it
clearly violates the notion of a unit test.

That said, I think your proposal has merit, just not where you want
it. I did just poke around a bit more and figgered something out that
might be satisfactory for the both of us:

501 % ruby -rtest/unit -e ‘class X < Test::Unit::TestCase; end’
Loaded suite -e
Started
F
Finished in 0.005603 seconds.

  1. Failure:
    default_test(X) [-e:1]:
    No tests were specified.

1 tests, 1 assertions, 1 failures, 0 errors
502 % ruby -rtest/unit -e ‘class X < Test::Unit::TestCase; def
default_test; end; end’
Loaded suite -e
Started
.
Finished in 0.00228 seconds.

1 tests, 0 assertions, 0 failures, 0 errors

YES, it still has an incorrect test count, but it also doesn’t have
as much “smell” as “test_dummy” in an abstract testcase.

in. So it is more convenient for Watir users than a subclass.
How is it more convenient?

require ‘monkeypatch’
class BlahTest < Test::Unit::TestCase
execute :sequentially
…
end

vs

require ‘systemtestcase’
class BlahTest < SystemTestCase
execute :sequentially
…
end

Do you really see either of those as more convenient than the other?

However, I’m not quite sure whether you are saying
a) i (ryan) would prefer to do it a different way
b) i don’t want to see this change made to the core test/unit code in
ruby
c) it is a bad idea for anyone to use this mod

I understand (a). I can see your reasons for (b), and actually have
been eager for feedback on whether this might be a good idea or not.
(although i did get some positive noises offline from Nathaniel
Talbott
– the test/unit author). i can see no grounds for © – if this is
your position, please elaborate.

(a) obviously, but on the grounds of (b). I’ve purposefully not
addressed © as I think its been addressed (several times) before,
but I also agree that it is a bad idea to use the mod.

12

On May 22, 2006, at 1:36 PM, Jim W. wrote:

The problem with subclassing is that it doesn’t combine well with
other
potential Test::Unit add-ins. (For that matter, I suspect the
patch way
doesn’t combine well either, but I haven’t looked at the details).

“potential add-ins” is a red flag for YAGNI.

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can’t use my FlexMock subclass with
his Watir subclass.

I guess I don’t understand. Is your FlexMock mod a subclassed
testcase as well? If not, then where is the conflict?

I guess where I’m getting at is this: Brett’s modification is NOT a
unit test extension. Your extension seems to be orthogonal to
TestCase, and so belongs in a module to be mixed in wherever needed.
Brett’s extension isn’t orthogonal. As he said, it is a modification
at a system or integration level. Test::Unit is a fine execution
framework but TestCase in particular has some coughflawscough
design restrictions that make subclassing a bit of a PITA and as a
result, TestCase gets used for everything. This is a shame IMO as
I’ve pointed out elsewhere (http://blog.zenspider.com/archives/
2006/01/move_over_testu.html) but we have workarounds that are
satisfactory.

I want people to test. I want them to write all the unit, system,
integration, and performance tests they possibly want. I want them to
have the tools they need to do this, but I also want those tools to
not confuse them as to what type of tests they are writing and I
think his patch is confusing.

end

Now, the flexmock one ties in by overriding teardown, so any user
supplied setup/teardown methods should carefully invoke super (which
they probably should do anyways).

nod I think this is a fine alternative. I’d probably take it a step
further and use it AND subclassing to make things both clearer and a
bit easier.

–
_why: zenspider’s most intense moments of solice are immediately
following the slaughter […]
_why: that topknot’s the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak

13

Ryan D. wrote:

On May 22, 2006, at 1:36 PM, Jim W. wrote:

The problem with subclassing is that it doesn’t combine well with
other
potential Test::Unit add-ins. (For that matter, I suspect the
patch way
doesn’t combine well either, but I haven’t looked at the details).

“potential add-ins” is a red flag for YAGNI.

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can’t use my FlexMock subclass with
his Watir subclass.

I guess I don’t understand. Is your FlexMock mod a subclassed
testcase as well? If not, then where is the conflict?

Some background … before last week, to use FlexMock in a test case,
you would say:

(I)
def test_blah
FlexMock.use(“mockname”) do |mock|
mock.blah.blah.blah
# test goes here
end
end

The block usage ensures that verification is run on the mock at the end
of the test. Although idiomatic Ruby, the block made (a) multiple mocks
awkward, and (b) the ability to refactor common mock creation/setup into
a separate method awkward.

Neither (a) nor (b) were show stoppers, just rather unsatisfying. What
I wanted to be able to write was:

(II)
def test_blah
mock = flexmock(“mockname”)
mock.blah.blah.blah
# test goes here
end

Now multiple mocks and special mock creation methods are trivial. To
get to (II), I had to override teardown. I can use either a subclass or
module to do this. My choice was to use a module …

However …

Your suggestion to Brett was to use a subclass. If I had followed the
suggestion and used a subclass (rather than a module), then my add-in
and Brett’s add-in could not interoperate. So, my counter suggestion to
Brett was to use a module as well (rather than a subclass).

Your extension seems to be orthogonal to
TestCase, and so belongs in a module to be mixed in wherever needed.
Brett’s extension isn’t orthogonal.

Ok, that’s a possible reason for choosing a subclass over a module. I
hadn’t seen the patch at the time of my original post, so I can’t say if
it is orthognal or not. I shall ponder the issue :wink:

I want people to test. I want them to write all the unit, system,
integration, and performance tests they possibly want. I want them to
have the tools they need to do this, […]

Absolute 100% agreement with the above.

nod I think this is a fine alternative. I’d probably take it a step
further and use it AND subclassing to make things both clearer and a
bit easier.

I would suggest (as a rule of thumb) that the library writers provide
modules and allow the end users to subclass and include the modules they
want/need.

– Jim W.

14

Eric H. wrote:

Click “Download Patch.” Or click the attachments tab. I also just
added a
link at the bottom of the page.

I think Eric’s point was that it isn’t a patch.

Exactly. Its hard to see how this fits into Test::Unit standing all
alone like that.

It’s trivial to put that code into test/unit/testcase.rb, replacing the
lines of the existing self.suite method, around lines 40-60.

Cheers,
Dave

15

I was able to rewrite the code as a subclass. I was even able to
prevent the problem with the incorrect test count that had been a
reason i’d avoided this approach previously (see the initialize
method).

Comments please. I would be happy to present this code as a true patch
to the Test::Unit source if there is actually interest in seeing it in
that form. Is there? My sense was that most people felt this was best
delivered as a subclass. (I may actually pull out the ordering options
now, since they really don’t serve a purpose – yagni and all.)

Bret

require ‘test/unit’

module Watir
class TestCase < Test::Unit::TestCase
@@order = :sequentially
def initialize name
throw :invalid_test if name == :default_test && self.class ==
Watir::TestCase
super
end
class << self
attr_accessor :test_methods, :order
def test_methods
@test_methods ||= []
end
def order
@order || @@order
end
def default_order= order
@@order = order
end
def sorted_test_methods
case order
when :alphabetically: test_methods.sort
when :sequentially: test_methods
when :reversed_sequentially: test_methods.reverse
when :reversed_alphabetically: test_methods.sort.reverse
else raise ArgumentError, “Execute option not supported:
#{@order}”
end
end
def suite
suite = Test::Unit::TestSuite.new(name)
sorted_test_methods.each do |test|
catch :invalid_test do
suite << new(test)
end
end
if (suite.empty?)
catch :invalid_test do
suite << new(:default_test)
end
end
return suite
end
def method_added id
name = id.id2name
test_methods << name if name =~ /^test./
end
def execute order
@order = order
end
end
end
end