Best way to skip tests

Ruby
1

Hi all,

What’s the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the ‘flunk’ method, but that’s considered a failed
test. I’m looking for something that doesn’t treat it as success or
failure.

I guess I’d like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match(‘mswin’)
skip(‘There’s no Foo::Bar on MS Windows - skipped’)
else
assert_equal(42, Foo::Bar)
end
end
end

And then output that looked like this:

1 tests, 0 assertions, 0 failures, 0 errors, 1 skipped

If there isn’t anything like this currently, would patches to add this
be accepted? I’d be happy to work on it.

Thanks,

Dan

PS - Yes, I realize I could wrap the whole test_bar method in an ‘if’
clause. I don’t want to do that. I want to be explicit.

2

Daniel B. wrote:

Hi all,

What’s the general approach folks use for skipping tests? Sometimes I
have some tests that I want to skip based on platform (usually MS
Windows). I saw the ‘flunk’ method, but that’s considered a failed
test. I’m looking for something that doesn’t treat it as success or
failure.

If you were to factor the platform-dependent tests out into their own
module which you can conditionally include into the test case, I think
you’d get what you were after.

3

On Feb 15, 12:32 pm, Alex Y. [email protected] wrote:

module which you can conditionally include into the test case, I think
you’d get what you were after.

It’s not a bad idea, but that still wouldn’t explicitly indicate to a
user that tests had been skipped - they would merely see fewer tests
run. Plus, it’s more work and I’m lazy. :slight_smile:

Regards,

Dan

4

On Feb 15, 2007, at 1:45 PM, Daniel B. wrote:

If you were to factor the platform-dependent tests out into their own
module which you can conditionally include into the test case, I
think
you’d get what you were after.

It’s not a bad idea, but that still wouldn’t explicitly indicate to a
user that tests had been skipped - they would merely see fewer tests
run. Plus, it’s more work and I’m lazy. :slight_smile:

I think it’s a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

James Edward G. II

5

On Feb 15, 2007, at 1:25 PM, Daniel B. wrote:

I guess I’d like something like this:

class TC_Foo < Test::Unit::TestCase
def test_bar
if RUBY_PLATFORM.match(‘mswin’)
skip(‘There’s no Foo::Bar on MS Windows - skipped’)

 at_exit { warn "The Windows tests were skipped." }

  else
      assert_equal(42, Foo::Bar)
  end

end
end

Just a thought.

James Edward G. II

6

On 2/15/07, James Edward G. II [email protected] wrote:

On Feb 15, 2007, at 1:45 PM, Daniel B. wrote:

It’s not a bad idea, but that still wouldn’t explicitly indicate to a
user that tests had been skipped - they would merely see fewer tests
run. Plus, it’s more work and I’m lazy. :slight_smile:

I think it’s a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

I agree that the skipping should be done outside of the test method,
not inside. But explicitness in the output would still be valuable.
Maybe something like the following:

class MyTest < Test::Unit::TestCase
def test_foo
…
end

def test_bar
  ...
end

if Time.now.wday == 0
  skip :test_foo, "The frambulator isn't available on Sundays"
end

end

Jacob F.

7

On 2/15/07, Jacob F. [email protected] wrote:

  ...
end

if Time.now.wday == 0
  skip :test_foo, "The frambulator isn't available on Sundays"
end

end

Now you have test names in two places, and when someone renames
:test_foo to :test_frambulator ?? Whoops!

I like the idea of having a skip assertion inside the test, and it has
to be first assertion used otherwise an exception is raised.

TwP

8

On 2/15/07, James Edward G. II [email protected] wrote:

test. I’m looking for something that doesn’t treat it as success or

I think it’s a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

That would be a design error, and an exception should be raised.

I like the idea of having a skip method that prints to the screen. It
is a way to let the user know that something is not being tested for
one reason or another.

We have a unit test framework for our embedded software code that has
a skip method. We use it all the time when a test can’t be run either
because hardware is not available or it’s running on the wrong
platform. The caveat is that the skip method has to be the first
assertion in the test – otherwise an exception is raised.

Future addition to Test::Unit ??

TwP

9

On Feb 15, 2007, at 2:05 PM, [email protected] wrote:

On Fri, 16 Feb 2007, Daniel B. wrote:

Hi all,

What’s the general approach folks use for skipping tests?

oh – i just don’t write them…

:wink:

Cracked me up.

James Edward G. II

10

On 2/15/07, Tim P. [email protected] wrote:

I like the idea of having a skip method that prints to the screen. It
is a way to let the user know that something is not being tested for
one reason or another.

I like this idea too. I usually will print a message to STDERR when
I skip tests, but this gets obscured by the test/unit output and makes
it hard to tell what’s going on.

11

On Fri, 16 Feb 2007, Daniel B. wrote:

Hi all,

What’s the general approach folks use for skipping tests?

oh – i just don’t write them…

:wink:

-a

12

On Fri, 16 Feb 2007, Tim P. wrote:

if Time.now.wday == 0

TwP

why not block form

class MyTest < Test::Unit::TestCase
def test_bar
assert this

   skip :sometimes, Time.now.wday == 0 do
     assert sometimes
   end

   assert that
 end

end

obvious it would be augmented with loggig like

skipping section(sometimes) of test_bar…

function name scraped with caller[0]…

-a

13

On Feb 15, 1:12 pm, [email protected] wrote:

   end

   assert that
 end

end

obvious it would be augmented with loggig like

skipping section(sometimes) of test_bar…

function name scraped with caller[0]…

I don’t think we need that level of granularity. At lease, I’ve never
hit a scenario where I would need it. It’s always all or nothing for
any given test.

Regards,

Dan

14

On 2/15/07, Daniel B. [email protected] wrote:

class TC_Foo < Test::Unit::TestCase

clause. I don’t want to do that. I want to be explicit.

Here’s a little something I’ve used…the reason was we had shared
tests across projects where certain test cases shouldn’t be run for
one project or another. Using this was easier then ripping everything
out of svn:externals and syncing manually…

This was used w/i rails, but should work fine with normal Test::Unit.
Just “include Bypass” in any test case you want to be skipped. The
code could probably be cleaned up - I hacked around with different
methods of doing this as some ways would work within autotest but not
rake, other ways vice versa. This works in both IIRC.

  • Rob

    module Bypass

    def self.logger; RAILS_DEFAULT_LOGGER; end

    def self.included(mod)
    mod.extend ClassMethods
    mod.class_eval do
    define_method(:setup) {}
    define_method(:teardown) {}
    class << self
    define_method(:fixtures) {}
    end
    end
    end

    provide an empty test to avoid annoying errors

    def test_dummy
    assert true
    end

    module ClassMethods
    # callback when methods are added
    def method_added(meth)
    undef_test_methods(meth)
    remove_setup_and_teardown(meth)
    end

    # undefine any test methods
def undef_test_methods(meth)
  if meth.to_s.starts_with?("test")
    undef_method meth
  end
end

# remove setup and teardown
def remove_setup_and_teardown(meth)
  if meth == :setup || meth == :teardown
    remove_method meth
  end
end

    end

    end

15

Dan;

How about something like this? Syntax could probably be cleaned up a
bit:

class Test::Unit::TestCase
  def self.skip_tests(opts, &tests)
    if opts[:if].call
      module_eval &tests
    else
      $stderr.puts "** #{opts[:msg]} -- skipped **"
    end
  end
end

Then you can just call it to skip your tests.

class TC_Foo < Test::Unit::TestCase
  def test_everywhere
    assert true
  end

  module Foo
    Bar = 42
  end

  skip_tests(
    :if  => lambda { ENV['WINDOZE'] },
    :msg => "Ballmer eats babies =)"
  ) do
    def test_bar
      assert_equal(42, Foo::Bar)
    end
  end
end

$ ruby t.rb
Loaded suite t
Started
…
Finished in 0.001932 seconds.

2 tests, 2 assertions, 0 failures, 0 errors

$ WINDOZE=1 ruby t.rb
** Ballmer eats babies =) – skipped **
Loaded suite t
Started
.
Finished in 0.001482 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

16

On Fri, Feb 16, 2007 at 04:25:13AM +0900, Daniel B. wrote:

class TC_Foo < Test::Unit::TestCase

1 tests, 0 assertions, 0 failures, 0 errors, 1 skipped

If there isn’t anything like this currently, would patches to add this
be accepted? I’d be happy to work on it.

Well this may not be the best way but it works. Although it doesn’t
record what methods were skipped.

% cat platform-filter-test.rb
require 'test/unit'

class MyTest < Test::Unit::TestCase

    def test_adams
       assert(42,"Life, Universe and Everything")
    end

    def test_baz
        assert(true,"Baz is good")
    end

    def test_skip_mswin
        assert(RUBY_PLATFORM.match('mswin').nil?, "How could I fail.

I should have been skipped. I’m on Windows!!!")
end

    def test_skip_darwin
        assert(RUBY_PLATFORM.match('darwin').nil?, "How could I

fail. I should have been skipped. I’m on MacOSX!!!")
end
end

% ruby platform-filter-test.rb
Loaded suite platform-filter-test
Started
..F.
Finished in 0.00652 seconds.

  1) Failure:
test_skip_darwin(MyTest) [platform-filter-test.rb:18]:
How could I fail.  I should have been skipped. I'm on MacOSX!!!.
<false> is not true.

4 tests, 4 assertions, 1 failures, 0 errors


% ruby platform-filter-test.rb --name='/test_(?!skip_darwin)/'
Loaded suite platform-filter-test
Started
...
Finished in 0.000426 seconds.

3 tests, 3 assertions, 0 failures, 0 errors

Its not very robust by any stretch of the imagination but if you want to
skip
certain methods, regexp on the method names will work.

enjoy,

-jeremy

17

James Edward G. II [email protected] writes:

test. I’m looking for something that doesn’t treat it as success or

I think it’s a much better design though.

For an example of my concerns, what happens if your proposed skip()
is called after a few assertions are run in a test?

James Edward G. II

This seems nice and I’ve seen similar in other frameworks:

Started
.S.
Finished in 0.00001 seconds.
3 tests, 3 assertions, 0 failures, 0 errors, 1 skipped

Steve

18

On Fri, Feb 16, 2007 at 04:25:13AM +0900, Daniel B. wrote:

If there isn’t anything like this currently, would patches to add this
be accepted? I’d be happy to work on it.

Well kept thinking about this today and came up with a Module approach,
still not exactly what you are looking for, but might be useful in the
interum.

% ruby -v platform-filter-test.rb
ruby 1.8.5 (2006-12-25 patchlevel 12) [i386-linux]
Loaded suite platform-filter-test
Started
....
Finished in 0.001042 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

% ruby -v platform-filter-test.rb
ruby 1.8.5 (2006-12-25 patchlevel 12) [x86_64-openbsd4.0]
Loaded suite platform-filter-test
Started
..Skipping TestCase RunOnlyOnLinuxOrDarwinTest
.Skipping TestCase RunOnlyOnLinuxTest
.
Finished in 0.001433 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

% ruby -v platform-filter-test.rb
ruby 1.8.5 (2006-08-25) [i686-darwin8.8.1]
Loaded suite platform-filter-test
Started
..Skipping TestCase DoNotRunOnDarwinTest
.Skipping TestCase RunOnlyOnLinuxTest
.
Finished in 0.000834 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

Some modules and a little meta programming can be fun :slight_smile:

enjoy,

-jeremy