Asserting in code vs unit tests


#1

Hi!

I talked to few fellow ruby programmers, and most of them agreed that
unit tests are the place where the asserts should be placed, however in
my own opinion, unit tests mostly never cover everything and asserts in
development versions ($DEBUG) are very handy. What is your opinion?

Esad


#2

Esad H. wrote:

Hi!

I talked to few fellow ruby programmers, and most of them agreed that
unit tests are the place where the asserts should be placed, however in
my own opinion, unit tests mostly never cover everything and asserts in
development versions ($DEBUG) are very handy. What is your opinion?

Esad

Personally, I like the asserts in the test case.

It doesnt clutter the code, you can test each circumstance and it helps
with design because you have to treat the code as a unit with inputs and
outputs.

Gareth


#3

On 3/5/06, Esad H. removed_email_address@domain.invalid wrote:

asserts in code are good to ensure preconditions and postconditions of
methods. this is “design by contract”.
asserts in unittests can often only be used to check postconditions of
the
tested methods.

i prefer precondition asserts in debug code that get filtered out by a
preprocessor in the release version (hence no performance cost).
i often have the assertions followed by errorhandling code that tries to
handle the same issues gracefully in release code.

– henon


#4

Esad H. wrote:

Hi!

I talked to few fellow ruby programmers, and most of them agreed that
unit tests are the place where the asserts should be placed, however in
my own opinion, unit tests mostly never cover everything and asserts in
development versions ($DEBUG) are very handy. What is your opinion?

If your unit tests do not cover everything, how can
any other kinds? :slight_smile:

Esad

E


#5

On 3/5/06, E. Saynatkari removed_email_address@domain.invalid wrote:

Esad H. wrote:
If your unit tests do not cover everything, how can
any other kinds? :slight_smile:

Esad

These are two different issues.

Your asserts/contracts should check that the methods are being passed
what they expect to be passed to them. Postconditions will probably
check some invariants on the results of the methods.

Your test should not test what the contracts already check. The tests
should check that if you pass correct values you get the expected
results.

Eg: assume that you have a method that takes only positive numbers.
you will have a contract/assertion checking that you don’t pass any
negatives and a test suite that checks that, given a set of positive
inputs, you get the correct outputs.


#6

Chiaro Scuro wrote:

On 3/5/06, E. Saynatkari removed_email_address@domain.invalid wrote:

Esad H. wrote:
If your unit tests do not cover everything, how can
any other kinds? :slight_smile:

Esad

These are two different issues.

Your asserts/contracts should check that the methods are being passed
what they expect to be passed to them. Postconditions will probably
check some invariants on the results of the methods.

Yes, I agree, and this was partly the point I was making.
There are situations where functional tests are required
and there are some things that just require real-time
constraints (any non-predetermined input, usually).

Your test should not test what the contracts already check. The tests
should check that if you pass correct values you get the expected
results.

Eg: assume that you have a method that takes only positive numbers.
you will have a contract/assertion checking that you don’t pass any
negatives and a test suite that checks that, given a set of positive
inputs, you get the correct outputs.

Or, instead, you could in many circumstances ascertain that the
method that produces the input does not produce negative numbers;
this is something that a unit test would do and it would eliminate
the need of a constraint :slight_smile:

E


#7

On 3/6/06, E. Saynatkari removed_email_address@domain.invalid wrote:

Or, instead, you could in many circumstances ascertain that the
method that produces the input does not produce negative numbers;
this is something that a unit test would do and it would eliminate
the need of a constraint :slight_smile:

We are on something interesting here. I am tempted to agree with you,
but for the sake of discussion and to explore the idea with you I will
say:

  • a test doesn’t prove that it will never produce a negative number,
    it simply checks that it doesn’t do it under certain very specific
    conditions
  • a postcondition will always check that it doesn’t produce a
    negative number and it will warn you if it does.

In this sense pre and post conditions seems to serve different purposes.

the precondition defines a scope of valid input and checks that that
input is well formed.
tests check a method, passing it valid input and expecting a ‘specific’
result.
a postcondition defines a scope of valid output and checks that the
output is well formed.

pre/post alone are not enough because they just check the domain of
variables.

you need tests to verify that at least under some conditions the
method yields meaningful results.

tests alone however are never exhaustive and cannot prove correctness.
assertions can catch some types of ‘wrongness’.


#8

On 3/5/06, chiaro scuro removed_email_address@domain.invalid wrote:

  • a test doesn’t prove that it will never produce a negative number,
    a postcondition defines a scope of valid output and checks that the
    output is well formed.

pre/post alone are not enough because they just check the domain of variables.

you need tests to verify that at least under some conditions the
method yields meaningful results.

tests alone however are never exhaustive and cannot prove correctness.
assertions can catch some types of ‘wrongness’.

Isn’t that what exceptions are for?

def foo(a)
raise ‘Argument can not be negative’ if a < 0

b = something_that_should_always_be_positive

raise ‘Something strange happened and b was negative’ if b < 0
b
end

foo expects a to be positive, so when a negative number comes in
that’s an exceptional case. It also should always return a positive
number, so if b is negative then something really went wrong and it
should terminate.

Pat


#9

Chiaro Scuro wrote:

On 3/6/06, E. Saynatkari removed_email_address@domain.invalid wrote:

Or, instead, you could in many circumstances ascertain that the
method that produces the input does not produce negative numbers;
this is something that a unit test would do and it would eliminate
the need of a constraint :slight_smile:

We are on something interesting here. I am tempted to agree with you,
but for the sake of discussion and to explore the idea with you I will
say:

  • a test doesn’t prove that it will never produce a negative number,
    it simply checks that it doesn’t do it under certain very specific
    conditions
  • a postcondition will always check that it doesn’t produce a
    negative number and it will warn you if it does.

In this sense pre and post conditions seems to serve different purposes.

the precondition defines a scope of valid input and checks that that
input is well formed.
tests check a method, passing it valid input and expecting a ‘specific’
result.
a postcondition defines a scope of valid output and checks that the
output is well formed.

pre/post alone are not enough because they just check the domain of
variables.

you need tests to verify that at least under some conditions the
method yields meaningful results.

tests alone however are never exhaustive and cannot prove correctness.
assertions can catch some types of ‘wrongness’.

Sure, but you could also just have a stick of bad RAM :slight_smile:

The goal is to be reasonably certain that the program behaves
as defined by the tests. This is aided by the granular and
gradual evolution of the tests. You can never really be sure
of anything.

E


#10

Using assertions I could be pretty much sure that certain conditions
would never never ever verify :slight_smile: I couldn’t do that with tests
though.

tests make sure that something happens.
assertions make sure that something doesn’t happen.


#11

I would get my assertions to raise exceptions too. They are just a
specific kind of exception, related to the domain of inputs and
outputs, different from exceptions generated from something that goes
wrong in the middle.

What do other people think about the relationship between assertions
and exceptions?


#12

Unit Tests and DbC assertions are complementary, not exclusive. I always
do both.

I tend to put the “post condition” assertions in the Unit Tests and
invariant and pre-condition assertions in the code.

Why?

  1. The pre-condition assertions are “tests of the tests”. Test’s are
    code
    too, therefore have bugs.

  2. Post-conditions are effectively the asserts found in the unit test.
    ie.
    If you have strong test coverage, don’t bother with post-condition
    asserts, refactor them into the unit test code.

  3. Client code using my code has bugs and the precondition asserts catch
    a
    lot of them very rapidly and at the right point. A the client code
    Unit
    test postcondition assert says the client (or maybe the lower
    layer)
    code did vaguely the wrong thing somewhere someplace sometime. A
    Precondition assert says the client code violated a pre-condition
    RIGHT HERE.

  4. Invariant code are Object Sanity Checks. Stroustrup (author of C++)
    recommends quite sanely that you should only spin a class if and only
    if
    you have an invariant to protect. Invariant checking method is a
    handy
    way of checking at run time that your class is doing what it was
    designed to do.

  5. I always throw vanilla Exceptions on assertion failure. It meshes
    nicely with the stacktrace mechanism of Ruby, but so far I can’t think
    of
    any good reason to create a new exception class for them.

On Mon, 6 Mar 2006, chiaro scuro wrote:

assertions can catch some types of ‘wrongness’.
end


– Chiaroscuro –
Liquid Development: http://feeds.feedburner.com/LiquidDevelopment

John C. Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : removed_email_address@domain.invalid
New Zealand

Carter’s Clarification of Murphy’s Law.

“Things only ever go right so that they may go more spectacularly wrong
later.”

From this principle, all of life and physics may be deduced.


#13

Interesting. Could you expand on “recommends quite sanely that you
should only spin a class if and only if you have an invariant to
protect”? it feels right but I am not sure I get it completely.

Another thing… one big problems that I have with assertions is
duplication. Do you assert any method? do you assert on class public
methods? It’s very to sprinkle the same assertions all over the place.


#14

On Mon, 6 Mar 2006, chiaro scuro wrote:

Interesting. Could you expand on “recommends quite sanely that you
should only spin a class if and only if you have an invariant to
protect”? it feels right but I am not sure I get it completely.

Don’t forget he was talking C++ naturally, so he was basically saying if
you need to protect an invariant use “class” otherwise use “struct”.

In Ruby/Perl this may translate to if you need to protect an invariant
use
“class”, otherwise this is just an unrelated bag of “stuff” so maybe use
a
Hash.

Another thing… one big problems that I have with assertions is
duplication. Do you assert any method? do you assert on class public
methods? It’s very to sprinkle the same assertions all over the place.

I tend to think of levels of sanity / maturity.

The code in any one class tends to be equally buggy or unbuggy.

Thus placing asserts within private methods tend to leave you wondering,
is the public method that invoked this private method buggy? Or is it
the
assert in the private method wrong?

Private methods, being private tend to be very fluid and change often,
thus putting an “anchor”, a “stake in the ground”, an “assert” in a
private method tends just to slow me down without much return.

So doing Test Driven development the stages (for me) are thus…

  1. Write Test Driving code for Class A (This test code is very unstable
    /
    immature / buggy).

  2. Write class A API, put in a smattering of preconditions. These
    preconditions test the test. Run test (I run the tests at almost
    every
    step)

  3. Fix test / preconds.

  4. Write postcondition assertions in Unit Test of class A.

  5. They fail, so start writing code for class A.

  6. Usually I think of some more preconditions that are in important
    while
    writing so I might add one or two more.

  7. Often the test still fails because I’m stupid. So I start printing
    things out. Ah. That value is wrong. How could it possibly be that?
    Start going back up the call chain and pushing in pre-cond asserts
    in
    the code and post-cond tests in the unit test until my mistake is
    obvious.

  8. Tests all pass, the code is written. I understand what and why it
    does,
    I write an invariant check method and slap calls to it in a bunch of
    places (start and end of most public methods).

  9. Start on a class B that uses class A, now test A and class A are
    reasonably mature, test B and class B are very immature.

  10. Write Test Driving code for Class B (This test code is very unstable
    /
    immature / buggy).

  11. Write class B API, put in a smattering of preconditions. These
    preconditions test the test.

  12. Fix test B / preconds for B.

  13. Write postcondition assertions in Unit Test B of class B.

  14. They fail, so start writing code for class B.

–WAKE UP! THIS IS THE IMPORTANT BIT!<<<

  1. Already the precond assertions within class A are starting to tell
    me about bugs in class B!

  2. Often the test still fails because I’m stupid. So I start printing
    things out. Ah. That value is wrong. How could it possibly be that?
    Start going back up the call chain and pushing in both pre-cond
    asserts
    in the code (in both classes) and post-cond asserts (in both tests)
    until my mistake is obvious.

  3. I put in an invariant check method of class B, which invokes the
    invariant check method of class A.

  4. By the time test B and class B is written and tests pass my pre-cond
    and post-cond coverage of class A is substantially better and
    several
    bugs / design issues have been ousted from class A.

  5. Start write class C that re-uses class A, class A is very mature
    now, very trustworthy, and right from the first invocation the
    pre-cond and invariant checks in class A catch bugs in test C and
    class C very early.

  6. Code is a little kludgy, so refactor and clean up. Unit Test’s and
    assertions catch breakage. If functional tests catch breakage
    undetected by unit tests & precond, insert test code / precond /
    unit
    test postcond until breakage is caught then fix it.

  7. Code is feature complete, but slow. Too many checks. Run profile and
    weed out a few checks that are in inner loops. Code is now fast,
    safe
    clean and bug free.

  1. The pre-condition assertions are “tests of the tests”. Test’s are code
    Precondition assert says the client code violated a pre-condition
    any good reason to create a new exception class for them.

What do other people think about the relationship between assertions

From this principle, all of life and physics may be deduced.


– Chiaroscuro –
Liquid Development: http://feeds.feedburner.com/LiquidDevelopment

John C. Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : removed_email_address@domain.invalid
New Zealand

Carter’s Clarification of Murphy’s Law.

“Things only ever go right so that they may go more spectacularly wrong
later.”

From this principle, all of life and physics may be deduced.