Rcov 0.4.0 (code coverage) - scriptability, accuracy, unrott


#1

Source code, additional information, screenshots… available at
http://eigenclass.org/hiki.rb?rcov
Release information:
http://eigenclass.org/hiki.rb?rcov+0.4.0

You can find a sample code coverage report generated by rcov at
http://eigenclass.org/static/rcov-sample-report/

If you’re on win32, you can also find a pre-built rcovrt.so (which makes
code coverage analysis >100 times faster) in the above-mentioned pages.

Overview

rcov is a tool for simple code coverage analysis in Ruby. It features:

  • fast execution: 20-300 times faster than previous tools
  • multiple analysis modes
  • fairly accurate coverage information through code linkage inference
    using
    simple heuristics
  • XHTML and several kinds of text reports
  • easy automation with Rake via a RcovTask
  • colorblind-friendliness

What’s new in 0.4.0

See http://eigenclass.org/hiki.rb?rcov+0.4.0 for the detailed change
summary.

rcov 0.4.0 is really feature-packed, and (hopefully) less buggy than
previous
releases. This can be ascribed in large part to the use of rcov on
itself to
achieve better test coverage. Nearly 100 changesets went into rcov
0.4.0.

Starting with the most obvious changes, the XHTML output looks much
better
now with the new color-scheme. Plain text output formats have also
improved
considerably.

rcov 0.4.0 is much easier to use from external scripts, and invoking
rcov
from your Rakefile takes no effort with the included Rcov::RcovTask; it
can
take as little as:

require 'rcov/rcovtask'
Rcov::RcovTask.new

Really! This creates a task named “rcov” which will run the tests
matching
test/test*.rb and create a XHTML report. And a nice textual report on
stdout). And a clobber_rcov task, which the main clobber target will
depend
on, to delete the former.

However, sometimes that lone Rcov::RcovTask.new will not suffice; this
would
be a more realistic example:

# if you don't like the default description
desc "Analyze code coverage of the unit tests."
Rcov::RcovTask.new do |t|
  # change the glob pattern in the next line
  t.test_files = FileList['test/test*.rb']
  t.rcov_opts << "--sort coverage"   # sort by increasing coverage 

rate
end

See README.rake in the distribution tarball (or the RDoc documentation,
which
includes it) for more information about Rcov::RcovTask.

Parts of the code have been moved into a reusable library that allows
you to
build on top of rcov’s coverage analysis capabilities. This change
allowed for
much better testing of rcov’s internals, which helped squash many bugs
and add
features while making the overall code look better thanks to some
long-needed
refactoring.

Short summary of user-visible changes

  • easy automation via Rake using the bundled Rcov::RcovTask
  • better heuristics: supports heredocs at last, including variants your
    editor
    is probably unable to handle, and =begin/=end comments
  • –rails option to ignore files under vendor/, enviroment/ and config/
  • parts of the runtime exposed and documented for external use
  • new color scheme, with alternating shades of the base colors
  • -w to set $VERBOSE=true (turns on warnings)
  • –text-report and --text-summary
  • –sort and --sort-reverse for the summaries and reports
  • –threshold and --only-uncovered
  • –replace-progname

Backwards incompatible changes

  • renamed --text to --text-counts and --rich-text to --text-coverage:
    they
    were misnamed to begin with
  • changed the meaning of -t and -T (–text-summary and --text-report)
  • $0 is not changed by default for each loaded file anymore; the old
    behavior (modulo a small bugfix) can be reproduced with
    –replace-progname

Thanks (do tell if I forgot to list your name)

Robert F.:

  • helped to refine the color scheme
  • very valuable suggestions; a large percentage of the new functionality
    in
    0.4.0 is a consequence of his requests

Andre N.:

  • identified a couple bugs in the heuristics that eluded my testing

Downloading

The last version is available at
http://eigenclass.org/hiki.rb?rcov

How do I use it?

Just use rcov to run your program (instead of ruby), and a number of
XHTML
files with the code coverage information will be generated, e.g.

rcov test/*.rb

will execute all the .rb files under test/ and generate the code
coverage report
under coverage/. rcov can also operate in “bogo-profiling mode”
and output the relevant information in alternative formats. Use
rcov -h
for more information.

Sample output

See http://eigenclass.org/hiki.rb?rcov (once again) for screenshots.
A sample code coverage report generated by rcov is available at
http://eigenclass.org/static/rcov-sample-report/

Text reports (also used by default in RcovTasks) resemble

±----------------------------------------------------±------±------±-------+
| File | Lines | LOC | COV |
±----------------------------------------------------±------±------±-------+
|app/controllers/application.rb | 39 | 29 | 31.0% |
|app/helpers/application_helper.rb | 147 | 119 | 23.5% |
|app/models/aggregations/tada.rb | 75 | 45 | 31.1% |
|app/models/aggregations/upcoming.rb | 78 | 48 | 29.2% |
|app/models/article.rb | 109 | 78 | 67.9% |
|app/models/category.rb | 30 | 23 | 60.9% |
|app/models/sidebar.rb | 36 | 27 | 40.7% |
|components/plugins/sidebars/archives_controller.rb | 35 | 27 | 29.6% |
|components/plugins/sidebars/category_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/delicious_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/flickr_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/fortythree_controller.rb | 20 | 16 | 50.0% |
|…s/plugins/sidebars/fortythreeplaces_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/static_controller.rb | 27 | 24 | 29.2% |
|components/plugins/sidebars/tada_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/technorati_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/upcoming_controller.rb | 20 | 16 | 50.0% |
|components/plugins/sidebars/xml_controller.rb | 16 | 13 | 53.8% |
|components/sidebars/sidebar_controller.rb | 110 | 80 | 32.5% |
|lib/html_engine.rb | 29 | 23 | 78.3% |
|lib/login_system.rb | 85 | 34 | 23.5% |
|lib/migrator.rb | 28 | 22 | 40.9% |
|lib/renderfix.rb | 32 | 25 | 16.0% |
|lib/xmlrpc_fix.rb | 13 | 12 | 25.0% |
±----------------------------------------------------±------±------±-------+
|Total | 1754 | 1251 | 60.1% |
±----------------------------------------------------±------±------±-------+
60.1% 42 file(s) 1754 Lines 1251 LOC

This was run with --threshold 80, meaning that only files under 80%
coverage
are shown in the table, but the overall statistics include all analyzed
files.

The textual output with execution count information looks like this:

$ rcov --no-html --text-counts b.rb

./b.rb

                                                                   | 

2
a, b, c = (1…3).to_a |
2
10.times do |
1
a += 1 |
10
20.times do |i| |
10
b += i |
200
b.times do |
200
c += (j = (b-a).abs) > 0 ? j : 0 |
738800
end |
0
end |
0
end |
0

License

rcov is released under the terms of Ruby’s license.
rcov includes xx 0.1.0, which is subject to the following conditions:

ePark Labs Public License version 1
Copyright © 2005, ePark Labs, Inc. and contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification,
are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright
    notice, this
    list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright
    notice,
    this list of conditions and the following disclaimer in the
    documentation
    and/or other materials provided with the distribution.
  3. Neither the name of ePark Labs nor the names of its contributors
    may be
    used to endorse or promote products derived from this software
    without
    specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS
IS” AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


#2

This is awesome. It’s enlightening and sad to see the coverage of some
of my unit tests, but without a doubt I’m hooked. Visually looking at
the test coverage (or lack thereof) is really much more effective than I
thought. If anything this will help to be a lot more conscious about
creating solid test suites, rather than cherry picking the easy/obvious
ones that most likely don’t do the job like they should.

When do we get a gem?

Thanks a lot,
Jeff


#3

On Tue, 2006-05-23 at 01:09 +0900, Jeff R. wrote:

This is awesome. It’s enlightening and sad to see the coverage of some
of my unit tests, but without a doubt I’m hooked. Visually looking at
the test coverage (or lack thereof) is really much more effective than I
thought. If anything this will help to be a lot more conscious about
creating solid test suites, rather than cherry picking the easy/obvious
ones that most likely don’t do the job like they should.

When do we get a gem?

Mauricio, I’ll hook you up with gem builds if you want. I use rcov a
lot and having gems makes it easier for me to get other developers to
use it.


#4

On Tue, May 23, 2006 at 05:17:47AM +0900, Zed S. wrote:

Mauricio, I’ll hook you up with gem builds if you want. I use rcov a
lot and having gems makes it easier for me to get other developers to
use it.

The one thing that has been blocking (or at least pushing down the TODO
list)
the RubyGems package was how to combine it with cross-compilation for
win32,
since I figured a RubyGems package would be of most value to win32 users
who
have no access to a C toolchain. There are a few other problems I’d have
liked
to solve, but they’re much more general and probably addressable at
RubyGems’
level (C runtime compatibility, ABI checking…).

A “normal” RubyGems package (i.e. one that requires a C compiler) should
be
very easy, so I should be able to get one out fairly quickly.