# Automatically finding gaps in unit test coverage?

**URL:** <https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406>\
**Category:** Ruby\
**Created:** [August 4, 2006, 6:12pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406 "2006-08-04T18:12:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 6:12pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/1 "2006-08-04T18:12:41Z")

</div>

Hello,

Is it possible to somehow automatically check which parts of code  
are not covered by unit tests ?

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 6:33pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/2 "2006-08-04T18:33:42Z")

</div>

On Aug 4, 2006, at 12:12 PM, Tomasz W. wrote:

> Hello,
> 
> Is it possible to somehow automatically check which parts of code  
> are not covered by unit tests ?
> 
> –  
> Tomasz W. [[http://t-a-w.blogspot.com/](http://t-a-w.blogspot.com/) ]

Check out rcov. This is exactly what rcov was created for. It makes  
these awesome HTML coverage reports.

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 6:42pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/3 "2006-08-04T18:42:42Z")

</div>

Tomasz W. wrote:

> Hello,
> 
> Is it possible to somehow automatically check which parts of code  
> are not covered by unit tests ?

I don’t know of any mechanism built into RubyUnit or a code coverage  
tool. But this is something you could do: use set\_trace\_func to trace  
all methods invoked, then at the end of a single class’s test case or  
when all tests are done go through your classes and check their  
#instance\_methods to verify that all of them were called. But that  
still does not give you info whether you had a proper test case for the  
method or the method was called by some other method etc. Maybe you can  
extend the tracing to record only methods called from a unit test.  
Still you would not know whether the method was simply called or whether  
return values were checked or whether they were checked properly.

Kind regards

```
robert
```

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 8:51pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/4 "2006-08-04T20:51:58Z")

</div>

On 8/4/06, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> #instance\_methods to verify that all of them were called. But that  
> still does not give you info whether you had a proper test case for the  
> method or the method was called by some other method etc. Maybe you can  
> extend the tracing to record only methods called from a unit test.  
> Still you would not know whether the method was simply called or whether  
> return values were checked or whether they were checked properly.

Well, one cannot have everything :-). rcov is pretty useful for finding  
missing tests. My only problem with it so far is its inability to test  
coverage of metaprogrammed code, but I guess that can’t be helped.

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 8:42pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/5 "2006-08-04T20:42:55Z")

</div>

On 8/4/06, Logan C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Check out rcov. This is exactly what rcov was created for. It makes  
> these awesome HTML coverage reports.

Thanks, it was exactly what I needed. 🙂

By the way how are people usually testing the convenience functions  
that print something to STDOUT.

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 9:22pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/6 "2006-08-04T21:22:04Z")

</div>

On Aug 4, 2006, at 1:40 PM, Tomasz W. wrote:

> By the way how are people usually testing the convenience functions  
> that print something to STDOUT.

Use the standard StringIO library for this. You can swap out the  
STDOUT object for the test and replace it when you are done but I  
prefer to modify the method to be more testable:

def meth\_to\_test(io = STDOUT)  
# use io here…  
end

Then you can just pass the StringIO object when testing.

James Edward G. II

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 5, 2006, 10:41am UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/7 "2006-08-05T10:41:55Z")

</div>

Mauricio F. wrote:

> You can do that in rcov with the --test-unit-only option.

[snip]

Mauricio, thanks for the detailed explanation! Learn something new  
every day…

Kind regards

```
robert
```

---

<div class="post-metadata">

**Author:** ![fmwr](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/9de053/32.png) [@fmwr](https://www.ruby-forum.com/u/fmwr)\
**Post date:** [August 4, 2006, 10:55pm UTC](https://www.ruby-forum.com/t/automatically-finding-gaps-in-unit-test-coverage/101406/8 "2006-08-04T22:55:30Z")

</div>

On Sat, Aug 05, 2006 at 03:50:35AM +0900, Tomasz W. wrote:

> On 8/4/06, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > I don’t know of any mechanism built into RubyUnit or a code coverage  
> > tool. But this is something you could do: use set\_trace\_func to trace  
> > all methods invoked, then at the end of a single class’s test case or  
> > when all tests are done go through your classes and check their  
> > #instance\_methods to verify that all of them were called. But that  
> > still does not give you info whether you had a proper test case for the  
> > method or the method was called by some other method etc. Maybe you can  
> > extend the tracing to record only methods called from a unit test.  
> > […]

You can do that in rcov with the --test-unit-only option.

> Well, one cannot have everything :-). rcov is pretty useful for finding  
> missing tests. My only problem with it so far is its inability to test  
> coverage of metaprogrammed code, but I guess that can’t be helped.

Are you referring to this?

# $ rcov --no-html --text-coverage --no-color a.rb

# a.rb

module StupidMeta  
def ugly\_make\_meth(meth)  
module\_eval \<\<-EOF, **FILE** , **LINE** +1  
def #{meth}(a,b)  
a + #{meth.to\_s.inspect} + b  
end  
EOF  
end

```
 def make_meth(meth)
   define_method(meth) do |a,b|

```

!! a + meth.to\_s + b  
!! end  
end

end

class X  
extend StupidMeta  
ugly\_make\_meth :foo  
make\_meth :bar  
end

# we don’t actually call them, so they should both be uncovered

#x = X.new  
#x.foo("method “, " here”)  
#x.bar("method “, " here”)

As you can see, Module#define\_method will work better, if you can use it  
at  
all (that is, if you don’t need a block argument, under 1.8, and don’t  
care  
about the closure and the speed hit). Another advantage is that it will  
tell  
you if the code is well formed (syntactically correct) much earlier.

The problem with the string forms of eval is that I’ve had to go to  
great  
lengths to make sure rcov marks heredocs correctly, and the above  
annotation  
_is_ correct.

Indeed, in the above snippet the heredoc in ugly\_make\_meth has been  
“executed”, in some sense (this interpretation gains some weight due to  
the  
interpolation going on; there’s actually something being evaluated  
there).

But there are two interpretations for that string: as a mere heredoc,  
and as  
executable code that defines another method. There are thus also two  
ways to  
think about it as far as coverage is concerned.

I’m thinking that maybe some sort of pragma would make sense:

def ugly\_make\_meth(meth)  
module\_eval \<\<-EOF, **FILE** , **LINE+1** # rcov:noheredoc  
def #{meth}(a,b)  
a + #{meth.to\_s.inspect} + b  
end  
EOF  
end

As for other limitations, I think I know how to handle generated code  
that  
cannot be meaningfully associated to your sources, but I feel it’s not  
worth  
the effort.

Is there anything else you’d like to see addressed?