# Test::Unit Newbie Question regarding loops

**URL:** <https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963>\
**Category:** Ruby\
**Created:** [February 23, 2010, 5:04am UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963 "2010-02-23T05:04:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 5:04am UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/1 "2010-02-23T05:04:16Z")

</div>

With the following example:

#---------------------------------------------------------  
#!/usr/bin/ruby

require “test/unit”

class TestImplArchFile \< Test::Unit::TestCase

def test\_loop

```
[0,1,2].each do |i|
  assert_equal(i, 6, "Duh, #{i} is not equal to 6.")
end

```

end

end  
#---------------------------------------------------------  
I get the following output:  
#---------------------------------------------------------

Loaded suite ./t  
Started  
F  
Finished in 0.00527 seconds.

1. Failure:  
test\_loop(TestImplArchFile)  
[./t.rb:10:in `test_loop' ./t.rb:9:in`each’  
./t.rb:9:in `test\_loop’]:  
Duh, 0 is not equal to 6.  
\<0\> expected but was  
\<6\>.  
#---------------------------------------------------------

Is there any way I can see three errors, ie the exception  
would create a failure, but not exit the def? I’d like to see  
all errors in the loop. I’m not sure  
#---------------------------------------------------------

1. Failure:  
test\_loop(TestImplArchFile)  
[./t.rb:10:in `test_loop' ./t.rb:9:in`each’  
./t.rb:9:in `test\_loop’]:  
Duh, 0 is not equal to 6.  
\<0\> expected but was  
\<6\>.

2. Failure:  
test\_loop(TestImplArchFile)  
[./t.rb:10:in `test_loop' ./t.rb:9:in`each’  
./t.rb:9:in `test\_loop’]:  
Duh, 1 is not equal to 6.  
\<1\> expected but was  
\<6\>.

3. Failure:  
test\_loop(TestImplArchFile)  
[./t.rb:10:in `test_loop' ./t.rb:9:in`each’  
./t.rb:9:in `test\_loop’]:  
Duh, 2 is not equal to 6.  
\<2\> expected but was  
\<6\>.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 5:32pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/2 "2010-02-23T17:32:25Z")

</div>

Yotta M. wrote:

> Is there any way I can see three errors, ie the exception  
> would create a failure, but not exit the def?

No, I don’t think so. A failed assertion raises AssertionFailedError  
which terminates your method. There are no restarts in Ruby.

You can accumulate the errors yourself:

errs = []  
[0,1,2].each do |i|  
errs \<\< “Duh, #{i} is not equal to 6” unless i == 6  
end  
assert errs.empty?, errs.join("; ")

You can use a bit of metaprogramming to create three separate test  
methods in a loop.

require ‘test/unit’  
class TestBase \< Test::Unit::TestCase  
(0…2).each do |i|  
define\_method(“test\_loop#{i}”) do  
assert\_equal 6, i, “Duh, #{i} is not equal to 6.”  
end  
end  
end

Or for fun, even three separate test classes:

require ‘test/unit’  
class TestBase \< Test::Unit::TestCase  
CHECK = 0  
def test\_it  
assert\_equal 6, self.class::CHECK, “Duh, #{self.class::CHECK} !=  
6”  
end  
end

klasses = []  
(1…2).each do |n|  
klass = Class.new(TestBase)  
klass.const\_set(:CHECK, n)  
klasses \<\< klass # for GC  
end

That’s not very pretty though.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 6:03pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/3 "2010-02-23T18:03:27Z")

</div>

This is really the great idea I was looking for, thanks. Obviously I’m  
not comparing integers, I’m actually iterating over verilog modules in  
an ASIC, and do a test for each verilog module. Most verilog modules  
will pass, but there will be a few that don’t, so I’d like the test to  
go through all verilog modules.

Another question, is my philosophy off? I’m a little confused by the  
amount of test options out there, rspec, cucumber, etc. Is there another  
testing strategy I should be working with in this instance where I want  
to iterate through an array of complex objects?

Thanks for not telling me to rtfm or google it.

> require ‘test/unit’  
> class TestBase \< Test::Unit::TestCase  
> (0…2).each do |i|  
> define\_method(“test\_loop#{i}”) do  
> assert\_equal 6, i, “Duh, #{i} is not equal to 6.”  
> end  
> end  
> end

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 7:32pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/4 "2010-02-23T19:32:30Z")

</div>

Yotta M. wrote:

> Another question, is my philosophy off? I’m a little confused by the  
> amount of test options out there, rspec, cucumber, etc.

Well, they’re apples and oranges to some extent. rspec is similar in  
concept to Test::Unit but with a very different syntax (which I  
personally dislike, so I stick to Test::Unit). But as far as I know,  
assertion failures in rspec also raise exceptions and so can’t continue.  
There are a whole bunch of alternative test frameworks out there, but I  
don’t know enough about any of them to say whether they would fit your  
needs better.

Cucumber works at a different layer. It sits on top of either rspec or  
test/unit and lets you write tests in plain language, using regexps to  
match that plain language and turn it into actual test code. But again,  
if one step fails within a scenario, the remaining steps are skipped.

> Is there another  
> testing strategy I should be working with in this instance where I want  
> to iterate through an array of complex objects?

Apart from the ideas I gave before, you could try rescuing  
Test::Unit::AssertionFailedError explicitly inside your loop, to allow  
it to continue (but making a note of the failure).

Indeed, the core code for Test::Unit which handles this looks very  
simple:

```
  # Runs the individual test method represented by this
  # instance of the fixture, collecting statistics, failures
  # and errors in result.
  def run(result)
    yield(STARTED, name)
    @_result = result
    begin
      setup
      __send__ (@method_name)
    rescue AssertionFailedError => e
      add_failure(e.message, e.backtrace)
    rescue Exception
      raise if PASSTHROUGH_EXCEPTIONS.include? $!.class
      add_error($!)
    ensure
      begin
        teardown
      rescue AssertionFailedError => e
        add_failure(e.message, e.backtrace)
      rescue Exception
        raise if PASSTHROUGH_EXCEPTIONS.include? $!.class
        add_error($!)
      end
    end
    result.add_run
    yield(FINISHED, name)
  end

```

So it looks like you could just call add\_failure yourself and continue.  
This is a private method/undocumented API, but I’d happily do that if it  
gets the job done.

Here’s a proof-of-concept:

require “test/unit”

class TestImplArchFile \< Test::Unit::TestCase

def no\_stop  
yield  
rescue Test::Unit::AssertionFailedError =\> e  
add\_failure(e.message, e.backtrace)  
rescue Exception  
raise if PASSTHROUGH\_EXCEPTIONS.include? $!.class  
add\_error($!)  
end

def test\_loop  
[0,1,2].each do |i|  
no\_stop do  
assert\_equal(i, 6, “Duh, #{i} is not equal to 6.”)  
end  
end  
end

end

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 8:34pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/5 "2010-02-23T20:34:14Z")

</div>

Yotta M. wrote:

> The above would indicate that the problem is not with define\_method, but  
> rather the ability of Test::Unit to handle a large number of test cases.

A large number of _failing_ test cases. What if you do

def flunk  
end

or

def flunk  
assert true  
end

?

> I couldn’t get define\_method going in a class to compare, it’s a private  
> method in Method, and I really don’t understand the Method rdoc example  
> using send.

class Foo  
define\_method …  
end

should work fine. If you have an existing Class object you can also do

Foo.class\_eval { define\_method … }

> Is there some sort of optimization in Test::Unit that I need to set?  
> Seems like something is really wrong, where is all this time going?

Rescuing exceptions perhaps. I thought you said that you expected most  
of your tests to pass, and only a few to fail?

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 8:27pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/6 "2010-02-23T20:27:37Z")

</div>

One edit to the above:

When ‘flunk’ is replaced with true:  
Finished in 0.482389 seconds.  
12221 tests, 12221 assertions, 12221 failures, 0 errors

should read

When ‘flunk’ is replaced with true:  
Finished in 0.482389 seconds.  
12221 tests, 0 assertions, 0 failures, 0 errors

I got lazy and copied only the time from the previous value.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 8:20pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/7 "2010-02-23T20:20:31Z")

</div>

I’m seeing some performance issues I can’t seem to get around. Your  
solution for creating a method was great though. I’d try the last one,  
but I think I’m seeing a fundamental problem with Test::Unit.

Using the following code as the base reference:

#!/usr/bin/ruby

require “test/unit”

class TestImplArchFile \< Test::Unit::TestCase  
(0…10).each do |i|  
(0…10).each do |j|  
(0…100).each do |k|  
define\_method(“test\_i#{i}\_\_j#{j}\_k#{k}”) do  
flunk  
end  
end  
end  
end  
end

When ‘flunk’ is set to flunk  
Finished in 5.586734 seconds.  
12221 tests, 12221 assertions, 12221 failures, 0 errors

When ‘flunk’ is replaced with true:  
Finished in 0.482389 seconds.  
12221 tests, 12221 assertions, 12221 failures, 0 errors

If I manually generate all 12k tests as individual tests in a new file,  
with each one flunking:  
Finished in 5.631014 seconds.  
12221 tests, 12221 assertions, 12221 failures, 0 errors

The above would indicate that the problem is not with define\_method, but  
rather the ability of Test::Unit to handle a large number of test cases.

As a comparison, if I just print out results in a log file, I get the  
awesome result:

Just using a class, no Unit::Test, not creating method, just print  
method name  
Finished in 0.102 seconds.

I couldn’t get define\_method going in a class to compare, it’s a private  
method in Method, and I really don’t understand the Method rdoc example  
using send.

Is there some sort of optimization in Test::Unit that I need to set?  
Seems like something is really wrong, where is all this time going?

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 8:43pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/8 "2010-02-23T20:43:47Z")

</div>

On 2/23/2010 12:20 PM, Yotta M. wrote:

> class TestImplArchFile \< Test::Unit::TestCase
> 
> with each one flunking:  
> method name  
> Finished in 0.102 seconds.
> 
> I couldn’t get define\_method going in a class to compare, it’s a private  
> method in Method, and I really don’t understand the Method rdoc example  
> using send.
> 
> Is there some sort of optimization in Test::Unit that I need to set?  
> Seems like something is really wrong, where is all this time going?

Nothings wrong, exceptions are just slow. This is actually true of most  
programming languages. ‘flunk’ fails by generating an exception and  
Test::Unit proceeds to rescue that exception. For some more information  
on just how slow exceptions are, check out:

> **[How slow are (Ruby) Exceptions? — Simone Carletti](https://simonecarletti.com/blog/2010/01/how-slow-are-ruby-exceptions/)**
>
> Some benchmarks to demonstrate how slow exceptions can be.

This is why and exception should almost never be the expected outcome  
of running any piece of code. Likewise, failure should never be the  
expected  
outcome of 12221 tests. If you have 12221 tests failing, you have much  
bigger  
problems than a 5 second run-time on your test suite.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 8:50pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/9 "2010-02-23T20:50:48Z")

</div>

On Tue, Feb 23, 2010 at 5:04 PM, Yotta M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Another question, is my philosophy off? I’m a little confused by the  
> amount of test options out there, rspec, cucumber, etc. Is there another  
> testing strategy I should be working with in this instance where I want  
> to iterate through an array of complex objects?
> 
> Thanks for not telling me to rtfm or google it.

If you want to metaprogram tests, you should really look into using  
dust.  
Its only a thin  
wrapper around Test::Unit so id doesn’t bring with it a steep learning  
curve. The main  
innovation is that your tests are not distinct methods themselves - you  
call  
the test  
method with a string argument (your test name) and you pass in a block  
which  
is your  
test code. Behind the scenes it constructs a complete test\_foo method  
for  
you.

The main upside to this is that it is super simple to iterate arrays of  
arguments and  
build up large suites of test methods while you still adhere to the  
principle of ‘One assert  
per test’ - which is where you are having issues.

Links:  
Dust:

> **[RubyGems: Dust](http://blog.jayfields.com/2007/08/rubygems-dust.html)**
>
> I've often shown examples similar to the following snippet. functional\_tests do test " given no name, when validated, then error is in the...

I blogged about using Dust before, the examples aren’t completely  
relevant,  
but they do  
show you how you can build up tests from arrays:

> **[Testing java code with Ruby and JtestR](http://richardconroy.blogspot.com/2008/11/testing-java-code-with-ruby-and-jtestr.html)**
>
> I am doing a lot of work lately with an old java code base. Its web based, so Watir is a good option at the acceptance test level. But ther...

Understand that RSpec, Cucumber, Coulda, Shoulda, Bacon etc. are solving  
different  
problems in testing. There is a certain learning curve and conceptual  
hurdles to overcome.

Your problem is that you can’t coerce Test::Unit to do exactly what you  
want  
it to, but  
Test::Unit is perfectly capable of doing it - Dust helps an awful lot  
though. I have  
done metaprogammed tests in both Test::Unit and Dust, and there is no  
comparison.

regards,  
Richard.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 10:24pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/10 "2010-02-23T22:24:09Z")

</div>

Yotta M. wrote:

> Yes, the all passing test ran in the 0.48 sec posted above. For the most  
> part, I expect all to be passing, but I wanted to understand the range  
> of values.
> 
> The problem is I have more than 10k tests. I was just using that as a  
> relative benchmark.
> 
> Also, the 0.48 sec will go up when the comparison happens with complex  
> objects. This is sort of best case.

Well, Test::Unit is doing a degree of book-keeping for you - counting  
the passing tests as well as assertions, writing a dot to the screen for  
each test passed - and I’m also not sure how efficient it is to have

> 10k methods in one class.

> In reality, we will have tests in the millions, so I’m better off  
> using a Class than Test::Unit, or creating my own Test::Unit.

You could be right. Test::Unit was probably not designed for the case  
where you have millions of tests and each test is so fast that the  
overhead of Test::Unit itself is large.

You might want to try this first:

require “test/unit”

class TestImplArchFile \< Test::Unit::TestCase  
def flunk  
assert true  
end

def no\_stop  
yield  
rescue Test::Unit::AssertionFailedError =\> e  
add\_failure(e.message, e.backtrace)  
rescue Exception  
raise if PASSTHROUGH\_EXCEPTIONS.include? $!.class  
add\_error($!)  
end

def test\_all  
(0…10).each do |i|  
(0…10).each do |j|  
(0…100).each do |k|  
no\_stop do  
flunk  
end  
end  
end  
end  
end  
end

It runs about 4 times faster on my PC, possibly due to the tests not  
being dispatched as separate methods (you’ll need to use the profiler if  
you want to understand exactly why). If it’s not fast enough, then  
having your own book-keeping is probably the right way to go.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 9:20pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/11 "2010-02-23T21:20:41Z")

</div>

Yes, the all passing test ran in the 0.48 sec posted above. For the most  
part, I expect all to be passing, but I wanted to understand the range  
of values.

The problem is I have more than 10k tests. I was just using that as a  
relative benchmark.

Also, the 0.48 sec will go up when the comparison happens with complex  
objects. This is sort of best case.

In response to Walton’s statement, the 5 seconds isn’t the focus, it’s  
really the half second. What it’s saying is that for 10k basic level  
tests with all passing, it takes half a second. In our system we  
integrate from a number of groups, and I was hoping to use Test::Unit as  
a framework to verify all object relationships. It’s not just the  
Exceptions, there is something in Test::Unit that is taking more time  
than it should, as the half second doesn’t come close to the tenth of a  
second of the class implementation. Both of those examples do not have  
exceptions. I threw in the exceptions just to see what the worst case  
was. In reality, we will have tests in the millions, so I’m better off  
using a Class than Test::Unit, or creating my own Test::Unit.

Richard, thanks for the reference on Dust, I’ll be sure to check it out.  
My concern is that as a wrapper around Test::Unit, I’ll end up with the  
same performance issue as before.

That all being said, we are dealing with large object association that  
are unique to our system. When we simulate our ASIC with vendor software  
it takes approximately 100G of memory on a 64-bit system. I’m attempting  
to build front end checks that would verify structure before the large  
simulation occurs.

Thanks for all the help guys.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 24, 2010, 12:47am UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/12 "2010-02-24T00:47:56Z")

</div>

On Tue, Feb 23, 2010 at 8:20 PM, Yotta M. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> 

Handling large volumes of tests is not out of scope for Test::Unit -  
there  
are sites  
out there that run far more than 10,000 tests. To make their test run  
perform well,  
they use chunking + virtualization strategies. Farming off a subset of  
the  
test  
execution to different machines. The setup cost might make this  
prohibitive  
in your  
case.

Hudson is a continuous integration server that doesn’t seem to get much  
love  
in  
Ruby circles, even though it has basic support for master/slave  
relationships  
(poor mans build cloud) which might get you the bang for buck that you  
need.

---

<div class="post-metadata">

**Author:** ![yottameter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/y/df705f/32.png) [@yottameter](https://www.ruby-forum.com/u/yottameter)\
**Post date:** [February 23, 2010, 11:20pm UTC](https://www.ruby-forum.com/t/test-unit-newbie-question-regarding-loops/183963/13 "2010-02-23T23:20:44Z")

</div>

Yes, I see the speed up as well.

I was using 1.8.5 and got the TestImplArchFile::PASSTHROUGH\_EXCEPTIONS’  
undefined error, but switched over to 1.8.7 and it worked like a champ.

I passed nil into the backtrace, when getting large amounts of errors  
it’s easier to limit the failure communication to just the message. By  
adding a qualifier on the assert\_equal I can then grep the output to get  
just the messages I need to scan quickly.

The problem is were dealing with errors that span across multiple  
departments, so it’s not an ‘always working’ assumption, it’s more like  
‘always broken’.

Thanks for the additional help.