Forum: Ruby-core Add method coverage and branch coverage metrics

A250061e969e82becbed01f2e7a605d0?d=identicon&s=25 unknown (Guest)
on 2014-02-26 10:29
(Received via mailing list)
Issue #9508 has been updated by Sam Rawlins.


Ah, I see. You're right; it's much better to allocate it only when
tracking coverage. I've implemented that in a new commit [1]; I only had
to add more checks to the _TRACE macros at the top of compile.c.

I also added decision coverage-tracking for `while` and `until`.

[1]
https://github.com/srawlins/ruby/commit/eaadf82063...

----------------------------------------
Feature #9508: Add method coverage and branch coverage metrics
https://bugs.ruby-lang.org/issues/9508#change-45491

* Author: Sam Rawlins
* Status: Open
* Priority: Normal
* Assignee:
* Category:
* Target version:
----------------------------------------
Since the Coverage extension was introduced in Ruby 1.9, Ruby has had
built-in line code coverage. Ruby should support more of the basic code
coverage metrics [1]. I have a pull request on GitHub (
https://github.com/ruby/ruby/pull/511 ) to add Method Coverage (Function
Coverage) and Branch Coverage. I'd love feedback to improve it.

Currently, with this feature, Coverage.result would look like:

    {"/Users/sam/code/ruby/cov_method.rb" => {
      lines: [1, 2, 2, 20, nil, nil, 2, 2, 2, nil, 0, nil, nil, nil, 1,
0, nil, nil, 1, 1, nil, nil, 1],
      methods: {1=>2, 15=>0, 19=>1},
      branches: {8=>2, 11=>0}
    }}

which includes
* the current Ruby line coverage report,
* as well as a method report (The method defined on line 1 was called 2
times; the method on line 15 was called 0 times; ...),
* and a branch report (the branch on line 8 was called 2 times; the
branch on line 11 was called 0 times).

Branches
--------

Branches include the bodies of if, elsif, else, unless, and when
statements, which are all tracked with this new feature. However, this
feature is not aware of void bodies, for example:

    if foo
      :ok
    end

will report that only one branch exists in the file. It would be better
to declare that there is a branch body on line 2, and a void branch body
on line 3, or perhaps line 1. This would require the keys of the
[:branch] Hash to be something other than line numbers. Perhaps
label_no? Perhaps nd_type(node) paired with line or label_no?

More Coverage
-------------

I think that Statement Coverage, and Condition Coverage could be added
to this feature, using the same techniques.

Caveats
-------

I was not very clear on the bit-arrays used in ruby.h, and just used
values for the new macros that seemed to work.

Also, I would much rather use Ranges to identify a branch, so that a
Coverage analyzer like SimpleCov won't need any kind of Ruby parser to
identify and highlight a full chunk of code as a tested branch, or a not
tested branch. I'm trying to find how that could be implemented...

[1] Wikipedia has good definitions:
http://en.wikipedia.org/wiki/Code_coverage

---Files--------------------------------
pull-request-511.patch (26.7 KB)
pull-request-511.patch (38.5 KB)
pull-request-511.patch (57 KB)
18813f71506ebad74179bf8c5a136696?d=identicon&s=25 Eric Wong (Guest)
on 2014-03-01 23:35
(Received via mailing list)
I finally tried commit eaadf820633e74350404d009a1c251f6319454aa
and it segfaults right away:
  ruby -rcoverage -e 'Coverage.start; require "tempfile"'
Backtrace here: http://yhbt.net/feature-9508.bt.txt
This topic is locked and can not be replied to.