Forum: Ruby Test::Unit Patch that allows test methods to be executed seq

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
68db3bafb0a990bf605c4cf62bf85db0?d=identicon&s=25 bpettichord@gmail.com (Guest)
on 2006-05-22 04:48
(Received via mailing list)
This patch to Test::Unit allows you to create TestCases whose test
methods will be executed in the order you've defined them, rather than
alphabetically, which is what Test::Unit does today.

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

Thus the foloowing TestCase will print "EFGH":

class TC2_Sequential < Test::Unit::TestCase
  execute :sequentially
  def test_b; print 'E'; end
  def test_a; print 'F'; end
  def test_d; print 'G'; end
  def test_c; print 'H'; end
end

I realize that there has been some controversy in the past about
whether is a good idea or not. This debate has hinged on issues about
proper unit testing.

However, with Watir (http://wtr.rubyforge.org), many people are now
using Test::Unit for system acceptance testing, which presents a
different context for the value of ordering tests.

My hope is that some people will find this to be useful. My thanks go
to Why's MetaClass article, which gave me the info i needed to create
this.

Bret
5a837592409354297424994e8d62f722?d=identicon&s=25 Ryan Davis (Guest)
on 2006-05-22 19:12
(Received via mailing list)
On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:

> proper unit testing.
Why not subclass TestCase and create SequentialTestCase or
WatirTestCase? I don't see any reason why this functionality needs to
be folded into test/unit but I see your argument for Watir. We should
be trying our damnedest to make test/unit thinner, not thicker.

--
_why: zenspider's most intense moments of solice are immediately
following the slaughter [...]
_why: that topknot's the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak
68db3bafb0a990bf605c4cf62bf85db0?d=identicon&s=25 bpettichord@gmail.com (Guest)
on 2006-05-22 20:34
(Received via mailing list)
Ryan Davis said:
> Why not subclass TestCase and create SequentialTestCase or
> WatirTestCase? I don't see any reason why this functionality needs to
> be folded into test/unit but I see your argument for Watir. We should
> be trying our damnedest to make test/unit thinner, not thicker.

Unfortunately, subclassing TestCase is not a good idea. All subclasses
of TestCase are presumed to be concrete tests and you will get errors
if the subclass does not have a test method. (Try it out if you don't
believe me.) There is no provision for "Virtual" testcases, which is
what we would want here.

I think the ideal solution would be to refactor the TestCase code along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

This would keep the code from bulking up and allow users to add custom
behavior without modifying the class itself. For example, there has
been some interest expressed in randomizing the order of the test
methods. This suggestion would allow users to easily do this.

Bret
5a837592409354297424994e8d62f722?d=identicon&s=25 Ryan Davis (Guest)
on 2006-05-22 20:40
(Received via mailing list)
On May 22, 2006, at 11:31 AM, bpettichord@gmail.com wrote:

> what we would want here.
I do it all the time. The simple solution is to put a "test_dummy" in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

> I think the ideal solution would be to refactor the TestCase code
> along
> the lines of this patch, and thereby expose
> TestCase.sorted_test_methods so that it could then be overlayed by
> Modules.

I disagree. As I said before, we need less code, not more. (I do
wholly agree that test/unit needs refactoring, but to make it thinner
not thicker)
68db3bafb0a990bf605c4cf62bf85db0?d=identicon&s=25 bpettichord@gmail.com (Guest)
on 2006-05-22 21:00
(Received via mailing list)
Ryan sez:
> I do it all the time. The simple solution is to put a "test_dummy" in
> the superclass. Really not hard and a fair tradeoff to have the right
> code in the right place.

This has been suggested before to the Watir users. Many of them don't
like the fact that this results in an incorrect test case count.

>> I think the ideal solution would be to refactor the TestCase code
>> along
>> the lines of this patch, and thereby expose
>> TestCase.sorted_test_methods so that it could then be overlayed by
>> Modules.
>
> I disagree. As I said before, we need less code, not more. (I do
> wholly agree that test/unit needs refactoring, but to make it thinner
> not thicker)

My suggestion is only to add a new seam, but i guess that amounts to a
bit of code. But only a bit.

Another reason that i packaged the code as a mod to TestCase directly
is that it makes it easier for people with large test suites to plug it
in. So it is more convenient for Watir users than a subclass.

However, I'm not quite sure whether you are saying
a) i (ryan) would prefer to do it a different way
b) i don't want to see this change made to the core test/unit code in
ruby
c) it is a bad idea for anyone to use this mod

I understand (a). I can see your reasons for (b), and actually have
been eager for feedback on whether this might be a good idea or not.
(although i did get some positive noises offline from Nathaniel Talbott
-- the test/unit author). i can see no grounds for (c) -- if this is
your position, please elaborate.

Bret
10d4acbfdaccb4eee687a428ca00a5d8?d=identicon&s=25 Jim Weirich (weirich)
on 2006-05-22 22:36
Ryan Davis wrote:
> On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:
>
>> proper unit testing.
> Why not subclass TestCase and create SequentialTestCase or
> WatirTestCase?

The problem with subclassing is that it doesn't combine well with other
potential Test::Unit add-ins.  (For that matter, I suspect the patch way
doesn't combine well either, but I haven't looked at the details).

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case.  If I
make this a sub-class, then Brett can't use my FlexMock subclass with
his Watir subclass.

Rather, make the add-ins modules, then the user can mix-in whatever
behavior they need.  For example ...

   class MyTest < Test::Unit::TestCase
     include FlexMock::TestCase
     include Watir::TestCase

     def test_using_both_watir_and_flexmock_extensions
       ...
     end
   end

Now, the flexmock one ties in by overriding teardown, so any user
supplied setup/teardown methods should carefully invoke super (which
they probably should do anyways).

-- Jim Weirich
58479f76374a3ba3c69b9804163f39f4?d=identicon&s=25 Eric Hodel (Guest)
on 2006-05-22 23:06
(Received via mailing list)
On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:

> This patch to Test::Unit allows you to create TestCases whose test
> methods will be executed in the order you've defined them, rather than
> alphabetically, which is what Test::Unit does today.
>
> http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don't see a patch there.

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com
151b20c72716eb93ac58fa16fb94ec23?d=identicon&s=25 Bret Pettichord (Guest)
on 2006-05-22 23:50
(Received via mailing list)
On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:
>
> http://wiki.openqa.org/display/WTR/Test-Unit+Patch
>
> I don't see a patch there.


Click "Download Patch." Or click the attachments tab. I also just added
a
link at the bottom of the page.

I have been updating the code on this page based on comments that i get.

Here's the current version of the code:

require 'test/unit'

module Test
  module Unit
    class TestCase
      @@order = :alphabetically
      class << self
        attr_accessor :test_methods, :order
        def test_methods
          @test_methods ||= []
        end
        def order
          @order || @@order
        end
        def default_order= order
          @@order = order
        end
        def sorted_test_methods
          case order
          when :alphabetically:          test_methods.sort
          when :sequentially:            test_methods
          when :reversed_sequentially:   test_methods.reverse
          when :reversed_alphabetically: test_methods.sort.reverse
          else raise ArgumentError, "Execute option not supported:
#{@order}"
          end
        end
        def suite
          suite = TestSuite.new(name)
          sorted_test_methods.each do |test|
            catch :invalid_test do
              suite << new(test)
            end
          end
          if (suite.empty?)
            catch :invalid_test do
              suite << new(:default_test)
            end
          end
          return suite
        end
        def method_added id
          name = id.id2name
          test_methods << name if name =~ /^test./
        end
        def execute order
          @order = order
        end
      end
    end
  end
end
5a837592409354297424994e8d62f722?d=identicon&s=25 Ryan Davis (Guest)
on 2006-05-23 00:18
(Received via mailing list)
On May 22, 2006, at 2:47 PM, Bret Pettichord wrote:

> On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:
>>
>> http://wiki.openqa.org/display/WTR/Test-Unit+Patch
>>
>> I don't see a patch there.
>
>
> Click "Download Patch." Or click the attachments tab. I also just
> added a
> link at the bottom of the page.

I think Eric's point was that it isn't a patch.
58479f76374a3ba3c69b9804163f39f4?d=identicon&s=25 Eric Hodel (Guest)
on 2006-05-23 00:28
(Received via mailing list)
On May 22, 2006, at 3:16 PM, Ryan Davis wrote:

>> added a
>> link at the bottom of the page.
>
> I think Eric's point was that it isn't a patch.

Exactly.  Its hard to see how this fits into Test::Unit standing all
alone like that.

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com
5a837592409354297424994e8d62f722?d=identicon&s=25 Ryan Davis (Guest)
on 2006-05-23 00:31
(Received via mailing list)
On May 22, 2006, at 11:58 AM, bpettichord@gmail.com wrote:

> Ryan sez:
>> I do it all the time. The simple solution is to put a "test_dummy" in
>> the superclass. Really not hard and a fair tradeoff to have the right
>> code in the right place.
>
> This has been suggested before to the Watir users. Many of them don't
> like the fact that this results in an incorrect test case count.

Quite frankly, I don't really care that they consider it to output an
"incorrect" test case count. It works for many of us and is a fair
tradeoff. I'd much rather have the requirement removed from test/unit
outright, but I didn't design test/unit and I too disagree with
requiring it (but! see below!). I however disagree wholly with making
test/unit MORE complex under all but the most optimal of returns. I
don't think your suggestion matches that requirement esp since it
clearly violates the notion of a unit test.

That said, I think your proposal has merit, just not where you want
it. I did just poke around a bit more and figgered something out that
might be satisfactory for the both of us:

501 % ruby -rtest/unit -e 'class X < Test::Unit::TestCase; end'
Loaded suite -e
Started
F
Finished in 0.005603 seconds.

   1) Failure:
default_test(X) [-e:1]:
No tests were specified.

1 tests, 1 assertions, 1 failures, 0 errors
502 % ruby -rtest/unit -e 'class X < Test::Unit::TestCase; def
default_test; end; end'
Loaded suite -e
Started
.
Finished in 0.00228 seconds.

1 tests, 0 assertions, 0 failures, 0 errors

YES, it still has an incorrect test count, but it also doesn't have
as much "smell" as "test_dummy" in an abstract testcase.

> in. So it is more convenient for Watir users than a subclass.
How is it more convenient?

require 'monkeypatch'
class BlahTest < Test::Unit::TestCase
   execute :sequentially
   ...
end

vs

require 'systemtestcase'
class BlahTest < SystemTestCase
   execute :sequentially
   ...
end

Do you really see either of those as more convenient than the other?

> However, I'm not quite sure whether you are saying
> a) i (ryan) would prefer to do it a different way
> b) i don't want to see this change made to the core test/unit code in
> ruby
> c) it is a bad idea for anyone to use this mod

> I understand (a). I can see your reasons for (b), and actually have
> been eager for feedback on whether this might be a good idea or not.
> (although i did get some positive noises offline from Nathaniel
> Talbott
> -- the test/unit author). i can see no grounds for (c) -- if this is
> your position, please elaborate.

(a) obviously, but on the grounds of (b). I've purposefully not
addressed (c) as I think its been addressed (several times) before,
but I also agree that it is a bad idea to use the mod.
5a837592409354297424994e8d62f722?d=identicon&s=25 Ryan Davis (Guest)
on 2006-05-23 00:46
(Received via mailing list)
On May 22, 2006, at 1:36 PM, Jim Weirich wrote:

> The problem with subclassing is that it doesn't combine well with
> other
> potential Test::Unit add-ins.  (For that matter, I suspect the
> patch way
> doesn't combine well either, but I haven't looked at the details).

"potential add-ins" is a red flag for YAGNI.

> For example, I have a Test::Unit::TestCase mod that allows FlexMock
> objects to be automatically verified at the end of a test case.  If I
> make this a sub-class, then Brett can't use my FlexMock subclass with
> his Watir subclass.

I guess I don't understand. Is your FlexMock mod a subclassed
testcase as well? If not, then where is the conflict?

I guess where I'm getting at is this: Brett's modification is NOT a
unit test extension. Your extension seems to be orthogonal to
TestCase, and so belongs in a module to be mixed in wherever needed.
Brett's extension isn't orthogonal. As he said, it is a modification
at a system or integration level. Test::Unit is a fine execution
framework but TestCase in particular has some *cough*flaws*cough*
design restrictions that make subclassing a bit of a PITA and as a
result, TestCase gets used for everything. This is a shame IMO as
I've pointed out elsewhere (http://blog.zenspider.com/archives/
2006/01/move_over_testu.html) but we have workarounds that are
satisfactory.

I want people to test. I want them to write all the unit, system,
integration, and performance tests they possibly want. I want them to
have the tools they need to do this, but I also want those tools to
not confuse them as to what type of tests they are writing and I
think his patch is confusing.

>    end
>
> Now, the flexmock one ties in by overriding teardown, so any user
> supplied setup/teardown methods should carefully invoke super (which
> they probably should do anyways).

*nod* I think this is a fine alternative. I'd probably take it a step
further and use it AND subclassing to make things both clearer and a
bit easier.

--
_why: zenspider's most intense moments of solice are immediately
following the slaughter [...]
_why: that topknot's the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak
10d4acbfdaccb4eee687a428ca00a5d8?d=identicon&s=25 Jim Weirich (weirich)
on 2006-05-23 03:54
Ryan Davis wrote:
> On May 22, 2006, at 1:36 PM, Jim Weirich wrote:
>
>> The problem with subclassing is that it doesn't combine well with
>> other
>> potential Test::Unit add-ins.  (For that matter, I suspect the
>> patch way
>> doesn't combine well either, but I haven't looked at the details).
>
> "potential add-ins" is a red flag for YAGNI.
>
>> For example, I have a Test::Unit::TestCase mod that allows FlexMock
>> objects to be automatically verified at the end of a test case.  If I
>> make this a sub-class, then Brett can't use my FlexMock subclass with
>> his Watir subclass.
>
> I guess I don't understand. Is your FlexMock mod a subclassed
> testcase as well? If not, then where is the conflict?

Some background ... before last week, to use FlexMock in a test case,
you would say:

(I)
   def test_blah
     FlexMock.use("mockname") do |mock|
       mock.blah.blah.blah
       # test goes here
     end
   end

The block usage ensures that verification is run on the mock at the end
of the test.  Although idiomatic Ruby, the block made (a) multiple mocks
awkward, and (b) the ability to refactor common mock creation/setup into
a separate method awkward.

Neither (a) nor (b) were show stoppers, just rather unsatisfying.  What
I wanted to be able to write was:

(II)
   def test_blah
     mock = flexmock("mockname")
     mock.blah.blah.blah
     # test goes here
   end

Now multiple mocks and special mock creation methods are trivial.  To
get to (II), I had to override teardown.  I can use either a subclass or
module to do this.  My choice was to use a module ...

However ...

Your suggestion to Brett was to use a subclass.  If I had followed the
suggestion and used a subclass (rather than a module), then my add-in
and Brett's add-in could not interoperate.  So, my counter suggestion to
Brett was to use a module as well (rather than a subclass).

> Your extension seems to be orthogonal to
> TestCase, and so belongs in a module to be mixed in wherever needed.
> Brett's extension isn't orthogonal.

Ok, that's a possible reason for choosing a subclass over a module.  I
hadn't seen the patch at the time of my original post, so I can't say if
it is orthognal or not.  I shall ponder the issue ;)

> I want people to test. I want them to write all the unit, system,
> integration, and performance tests they possibly want. I want them to
> have the tools they need to do this, [...]

Absolute 100% agreement with the above.

> *nod* I think this is a fine alternative. I'd probably take it a step
> further and use it AND subclassing to make things both clearer and a
> bit easier.

I would suggest (as a rule of thumb) that the library writers provide
modules and allow the end users to subclass and include the modules they
want/need.

-- Jim Weirich
0b561a629b87f0bbf71b45ee5a48febb?d=identicon&s=25 Dave Burt (Guest)
on 2006-05-23 13:44
(Received via mailing list)
Eric Hodel wrote:
>>>
>>> Click "Download Patch." Or click the attachments tab. I also just
>>> added a
>>> link at the bottom of the page.
>>
>> I think Eric's point was that it isn't a patch.
>
> Exactly.  Its hard to see how this fits into Test::Unit standing all
> alone like that.

It's trivial to put that code into test/unit/testcase.rb, replacing the
lines of the existing self.suite method, around lines 40-60.

Cheers,
Dave
68db3bafb0a990bf605c4cf62bf85db0?d=identicon&s=25 bpettichord@gmail.com (Guest)
on 2006-05-24 07:20
(Received via mailing list)
I was able to rewrite the code as a subclass. I was even able to
prevent the problem with the incorrect test count that had been a
reason i'd avoided this approach previously (see the initialize
method).

Comments please. I would be happy to present this code as a true patch
to the Test::Unit source if there is actually interest in seeing it in
that form. Is there? My sense was that most people felt this was best
delivered as a subclass. (I may actually pull out the ordering options
now, since they really don't serve a purpose -- yagni and all.)

Bret

require 'test/unit'

module Watir
    class TestCase < Test::Unit::TestCase
      @@order = :sequentially
      def initialize name
        throw :invalid_test if name == :default_test && self.class ==
Watir::TestCase
        super
      end
      class << self
        attr_accessor :test_methods, :order
        def test_methods
          @test_methods ||= []
        end
        def order
          @order || @@order
        end
        def default_order= order
          @@order = order
        end
        def sorted_test_methods
          case order
          when :alphabetically:          test_methods.sort
          when :sequentially:            test_methods
          when :reversed_sequentially:   test_methods.reverse
          when :reversed_alphabetically: test_methods.sort.reverse
          else raise ArgumentError, "Execute option not supported:
#{@order}"
          end
        end
        def suite
          suite = Test::Unit::TestSuite.new(name)
          sorted_test_methods.each do |test|
            catch :invalid_test do
              suite << new(test)
            end
          end
          if (suite.empty?)
            catch :invalid_test do
              suite << new(:default_test)
            end
          end
          return suite
        end
        def method_added id
          name = id.id2name
          test_methods << name if name =~ /^test./
        end
        def execute order
          @order = order
        end
      end
    end
end
This topic is locked and can not be replied to.