Hi Friends,
I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.
TIA,
Regards,
Surya
2008/9/11 Surya [email protected]:
I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.
It’s not totally clear to me what you want. First, how do you define
an “unimportant” test case?
Then I am tempted to ask “why bother?”. The only reason for reduction
I can think of is runtime of the test suite - or maybe your
requirements change so often so dramatically that you have to touch a
lot of those tests. Otherwise I do not really see the benefit. Can
you elaborate a bit more?
Kind regards
robert
On 9/10/08, Surya [email protected] wrote:
Hi Friends,
I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.
I suffer from an excess of test cases as well, but I don’t have a
ready solution.
I have heard before of tools to help identify redundant test cases,
but none for
use with ruby.
I imagine such a tool would work like this:
Writing a tool like that wouldn’t be too hard. Getting it to be
efficient might be challenging…
Of course there’s a problem with this; line coverage is less than
perfect. So if there’s a
line whose first half is tested by one test case, and the second half
tested by another
test case, the second case may be flagged as redundant when it isn’t.
Expression
coverage is the gold standard when it comes to code coverage. So
ideally, you’d write the
tool above to work expression by expression rather than line by line.
There are no expression-based coverage tools for ruby that I know of.
Even expression coverage will only give you complete code coverage;
data coverage is
another matter. Is there a tool or metric for assessing data coverage?
I have no idea.
On Sep 11, 2008, at 5:41 PM, Caleb C. wrote:
Combinatorial/pairwise testing effectively to my situation.
- now work thru the list of lists of lines covered per test case,
test case, the second case may be flagged as redundant when it isn’t.
path to insanity if you ask me:
cover_this = Array.new(10000){|i| define_method(“foo_#{ i }”){ i } }
On 9/11/08, ara.t.howard [email protected] wrote:
cover_this = Array.new(10000){|i| define_method(“foo_#{ i }”){ i } }
Obviously complete data coverage is out of the question, for almost
all cases. But your coverage of data doesn’t have to be complete in
order to be sufficient. Usually, you want to show that you’re covering
typical cases, as well as hitting all the corners and edges. But how
do you know you’ve covered all those edges? That’s what I’m wondering
about. Is there some way to assess (for any language, tho ruby would
be nice) how close you’re getting to covering a set of data which is
suitably representative of your entire set of possible data?
On Sep 11, 1:53 am, Surya [email protected] wrote:
Hi Friends,
I have been in to testing for 3 yrs. In my project there are 6000 test
cases present and some of them ought to be non-value adding to the
testing effort. Around 80% of the total test cases were automated and
would like to run execersie to reduce the un important test cases from
the pool. Can any one suggest how to go about this and apply
Combinatorial/pairwise testing effectively to my situation.
Hi Surya,
I have a solution for you. And recently ported the code into the Turn
project, however I’ve been unable to get Tim P. to say one way or
the other if he will accept the new code. I will try to get up with
him one more time. I hoping he will, but if not, I’ll put it out as a
new project.
T.
This seems tough to answer without knowing more about the code. If the
test cases have something in common, e.g. they descend from a common
test_case class, then you could add something there that notes every
time the test case is called.
Still, it should be mentioned that this does not answer for all. You do
not say what constitutes “value added”. My first thought was that, if
it never gets called then it is not adding much, but that is not the
whole of the story.
Not knowing how it is written or what the criteria for value
determination is puts me somewhat at a loss. If you can add these
things, it would help.
On Sep 12, 2008, at 1:02 PM, Caleb C. wrote:
Obviously complete data coverage is out of the question, for almost
all cases. But your coverage of data doesn’t have to be complete in
order to be sufficient. Usually, you want to show that you’re covering
typical cases, as well as hitting all the corners and edges. But how
do you know you’ve covered all those edges? That’s what I’m wondering
about. Is there some way to assess (for any language, tho ruby would
be nice) how close you’re getting to covering a set of data which is
suitably representative of your entire set of possible data?
i think our minds are quite good at this - we have to select the data,
write the coverage tools, the testing tools, the ci tools, and the
code itself after all. clearly software we write cannot be more
effective that the people writing it.
it may sound heresy to some people on this list, and elsewhere, but i
think dynamic languages like ruby require a, possibly additional (on
top of tdd, bdd, whatever), rigorous process of thought and
discussion to really build good code. mechanical analysis is simply
one way to approach coverage.
a nice article summarizing what i’m saying
it’s simply worth considering that 3 people in a room now maybe
quite easily be able to achieve what current tools cannot, and may
never be able to.
ymmv.
On 9/12/08, ara.t.howard [email protected] wrote:
it may sound heresy to some people on this list, and elsewhere, but i
think dynamic languages like ruby require a, possibly additional (on
top of tdd, bdd, whatever), rigorous process of thought and
discussion to really build good code. mechanical analysis is simply
one way to approach coverage.a nice article summarizing what i’m saying
Our philosophies are more aligned than you seem to think. Interesting
link. IMHO, testing is just one of the tools that should be used to
assess quality. I’ve long been a fan of code reviews, since long
before I’d ever heard of unit tests, dynamic languages, or any of
that.
I don’t want to come across as your usual unit testing bigot, but it
certainly does have its place, which as I see it, is more than just
forcing you to think about what you’re doing. But then, I don’t
practice TDD or unit testing for that matter in the way its usually
done. Which is how I ended up on this thread in the first place.
it’s simply worth considering that 3 people in a room now maybe
quite easily be able to achieve what current tools cannot, and may
never be able to.
But consider that tools may achieve things that people can’t, either.
i think our minds are quite good at this - we have to select the data,
write the coverage tools, the testing tools, the ci tools, and the
code itself after all. clearly software we write cannot be more
effective that the people writing it.
I for one am not so confident about my ability to get all the edges on
my own. Tools are nice for things like exhaustive coverage of piddly
little details. I’m not so sure exactly what you’re trying to get at
in the paragraph above, but it’s always seemed the most pragmatic to
me to employ a combination of things. No tool will understand your
code for you, but the right tool can significantly enhance your
understanding of your code.
On Sep 12, 2008, at 2:58 PM, ara.t.howard wrote:
it may sound heresy to some people on this list, and elsewhere, but
i think dynamic languages like ruby require a, possibly additional
(on top of tdd, bdd, whatever), rigorous process of thought and
discussion to really build good code. mechanical analysis is
simply one way to approach coverage.
You are still my hero. I just wanted to make sure I’ve told you
lately. 
James Edward G. II
On Sep 12, 2008, at 4:57 PM, Caleb C. wrote:
in the paragraph above, but it’s always seemed the most pragmatic to
me to employ a combination of things. No tool will understand your
code for you, but the right tool can significantly enhance your
understanding of your code.
what i was trying get across was
. ruby presents some challenges for really good code tools, at
least with the current impl
. we can address some of the shortcomings today with processes
that involve people
. i trust the current crop of people more that the current crop of
tools. this may change in the future.
. it’s important as a community to share good processes for
thinking about code.
. lack of tools should not become a red herring in the community.
most shortcomings in code tools should probably wait to be developed,
helping instrument the language to support a new breed of tools might
be a better use of energy.
so yeah, we are probably saying close to the same thing.
cheers.
On Sep 12, 2008, at 4:36 PM, ara.t.howard wrote:
On Sep 12, 2008, at 2:07 PM, James G. wrote:
You are still my hero. I just wanted to make sure I’ve told you
lately.James Edward G. II
lol - coming from you that’s a real compliment… i heard your talk
at lone star was great.
You inspired that too. It was about NArray, RBtree, SQLite, FSDB, …
James Edward G. II
On Sep 12, 2008, at 2:07 PM, James G. wrote:
You are still my hero. I just wanted to make sure I’ve told you
lately.James Edward G. II
lol - coming from you that’s a real compliment… i heard your talk
at lone star was great.
cheers.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs