Embedded vs. Non-embedded Tests

As far as I know, Facets is the only large project that uses embedded
unit tests. The tests are housed in block comments, eg.

=begin test

=end

I have a command line tool that can run the embedded test directly.
For deployment, I have another tool that extracts all the embedded
tests and copies them to stand-alone test/ files so others can run
them. It works fairly well. And the advantage of course is that the
unit tests are right there with the libs they test.

However, considering that just about everyone else is putting units
test in their on test files, I’m wondering if I actually missing out
on superior advantages to that approach. If not why haven’t others
pursued better tools for supporting embedded tests?

Thanks,
T.

On 9/4/07, Trans [email protected] wrote:

them. It works fairly well. And the advantage of course is that the
unit tests are right there with the libs they test.

However, considering that just about everyone else is putting units
test in their on test files, I’m wondering if I actually missing out
on superior advantages to that approach. If not why haven’t others
pursued better tools for supporting embedded tests?

I think that you’re swimming against the tide.

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

And when it gets beyond just unit testing individual classes, it’s not
apparent to me which file an embedded test should call home.

From a practical tool perspective, there are lots of things which
expect a relationship between code files and test files. Two which
spring immediately to mind are autotest and rcov.


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

On 9/4/07, Rick DeNatale [email protected] wrote:

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

A lot of diagnostics now takes place in modern cars, however. Tire
pressure
checks, engine wear, oil, etc. Sure, you’d only work and make changes to
the
car in the garage, but a lot of the “testing” is a hidden ongoing day to
day
process.

And if you get onto, say, Formula 1 cars… some “repairs” can even take
place on the road and the telemetry provides almost as much info to the
engineers while the car is running out on the track as they can get in
the
garage. I’ve always wondered if/how to make software as easy to diagnose
with real-time telemetry and the ability to make changes to a running
program’s parameters…

Regards,
Peter C.
http://www.petercooper.co.uk/

On 9/4/07, Peter C. [email protected] wrote:

process.
For the most part, that’s instrumentation, not testing.

In the case where sensor information is used to affect the operation
of the vehicle, such as
engine managment, abs, or traction control, it’s no more testing
equipment than carburettors, brakes, and master cylinders. Yes, data
gets logged, but that’s not testing, any more than a server log file
is.

And if you get onto, say, Formula 1 cars… some “repairs” can even take
place on the road and the telemetry provides almost as much info to the
engineers while the car is running out on the track as they can get in the
garage. I’ve always wondered if/how to make software as easy to diagnose
with real-time telemetry and the ability to make changes to a running
program’s parameters…

Yes, but most the testing equipment is not in the car, the engineers
in the pits have laptops, and the heavy stuff is in places like Woking
or Maranello, connected through the ‘internets.’


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

On Sep 4, 12:07 pm, “Rick DeNatale” [email protected] wrote:

I think that you’re swimming against the tide.

Personally, it just feels right to me to separate the test code from
the code under test, most test equipment belongs in the garage instead
of the trunk of the car.

Interesting analogy. We do put some gages right on the dash. But
that’s more like embedding the tests directly into code --eg. rescue
and try clauses. So maybe you’re analogy is a good one. I can see some
reasons for doing so, it certainly make it easier to send some one a
test, that they can try against their own implementation, for
instance. But then I start to wonder about API documentation too.
Should that also be separated out into dedicated files? Why tests but
not docs?

And when it gets beyond just unit testing individual classes, it’s not
apparent to me which file an embedded test should call home.

That’s true. Embedded tests only apply for unit test. Integration
tests still need to be is separate files anyway.

From a practical tool perspective, there are lots of things which
expect a relationship between code files and test files. Two which
spring immediately to mind are autotest and rcov.

That’s part of what I was wondering. I’ve never tried these tools. I
guess I’ve always suspected they’d be too magical to introduce into an
already established project, and thus lead to more trouble than
they’re worth --though I’m sure they’re quite useful if one starts
using them from the get-go. Am I wrong about that? But even so,
couldn’t such tools be adapted to handle embedded tests too?

T.

On 9/4/07, Trans [email protected] wrote:

On Sep 4, 12:07 pm, “Rick DeNatale” [email protected] wrote:

Should that also be separated out into dedicated files? Why tests but
not docs?

Well , it probably seems more obvious to most to put the doc with the
code (assuming that there is any doc ). Note that we’ve just now
seemed to get tools like dcov.

It’s long been pretty standard process to have tools which extract doc
from code. On the other hand there are some arguments for keeping
even these two separate. In “The Psychology of Computer Programming”
Gerry Weinberg talked about the trap of “debugging” the comments
instead of the code, and advocated tools which would not just extract
code commentary but REMOVE it.

On the other hand without having the doc with the code and tools like
rdoc we’d probably have even less documentation than we do now, just a
guess.


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

Rick DeNatale wrote:

that’s more like embedding the tests directly into code --eg. rescue


Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

One thing I am a fan of is reading over my changes at the end of a
session and
updating to-do lists, especially when it reminds me clean up my comments
and
rewrite quick-fixes to proper code the next day :wink:

It’s generally my opinion that the comments tell you what it does and
the code
how (with comments for the less obvious). If the comments fall out of
sync
with the code, the comments should be updated once the code is less
likely to
change to much. Personally I think documentation about using the program
should be separate from the regular source code. Documentation about
working
on the program and it’s sources should be kept near by. Just how much,
is a
matter of developers tastes.

I advocate people using common sense, that is to us the gray matter
approximately three feet above their romp roast.

TerryP.

Stefan R. wrote:

you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with “Design by
Contract” in the title here: Presentations - Eiffel Software - The Home of EiffelStudio
Would be a very nice feature if ruby had that, even though I’d introduce
additional syntax and I’ve no idea how to integrate it nicely.
Allow me to shamelessly copy and paste from an old post of mine:

$ cat testable.rb
module Test; module Unit; class TestCase; end; end; end

module Testable
def test_case(&block)
Class.new(Test::Unit::TestCase).module_eval &block
end
end

$ cat builtin_tests.rb
require ‘testable’

class MyWorkingClass
extend Testable

def foo(a,b)
return a+b
end

test_case {
def test_foo_null
assert_equal 2, MyWorkingClass.new.foo(1,1)
end
}
end

if FILE == $0
puts MyWorkingClass.new.foo(1,1)
end

$ ruby builtin_tests.rb
2
$ testrb builtin_tests.rb
Loaded suite builtin_tests.rb
Started
.
Finished in 0.00095 seconds.

1 tests, 1 assertions, 0 failures, 0 errors
$

Something like that?

Peter C. wrote:

And if you get onto, say, Formula 1 cars… some “repairs” can even take
place on the road and the telemetry provides almost as much info to the
engineers while the car is running out on the track as they can get in
the
garage. I’ve always wondered if/how to make software as easy to diagnose
with real-time telemetry and the ability to make changes to a running
program’s parameters…

Take a look at Eiffels Design By Contract. I’m a huge fan of it. Allows
you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with “Design by
Contract” in the title here: Presentations - Eiffel Software - The Home of EiffelStudio
Would be a very nice feature if ruby had that, even though I’d introduce
additional syntax and I’ve no idea how to integrate it nicely.

Regards
Stefan

On Sep 5, 2:13 am, Stefan R. [email protected] wrote:

you to move most testing code into the contract, the unit tests are then
merely running your code with contracts enabled.
For an idea on how that works, take a look at the movies with “Design by
Contract” in the title here:Presentations - Eiffel Software - The Home of EiffelStudio
Would be a very nice feature if ruby had that, even though I’d introduce
additional syntax and I’ve no idea how to integrate it nicely.

T.

Would be a very nice feature if ruby had that, even though I’d introduce
additional syntax and I’ve no idea how to integrate it nicely.

There are a few libraries around that try to do this (google for “ruby
dbc” or similar). Most of these efforts are not entirely convincing.
Unfortunately, the ruby community seems somewhat dogmatic in arguing
against dbc. Maybe because it’s not enough of a duck.

Alex Y. wrote:

Allow me to shamelessly copy and paste from an old post of mine:

$ cat testable.rb
module Test; module Unit; class TestCase; end; end; end

module Testable
def test_case(&block)
Class.new(Test::Unit::TestCase).module_eval &block
end
end

$ cat builtin_tests.rb
require ‘testable’

class MyWorkingClass
extend Testable

def foo(a,b)
return a+b
end

test_case {
def test_foo_null
assert_equal 2, MyWorkingClass.new.foo(1,1)
end
}
end

if FILE == $0
puts MyWorkingClass.new.foo(1,1)
end

$ ruby builtin_tests.rb
2
$ testrb builtin_tests.rb
Loaded suite builtin_tests.rb
Started
.
Finished in 0.00095 seconds.

1 tests, 1 assertions, 0 failures, 0 errors
$

Something like that?

No, the whole approach is different. I’d say the movies I linked to can
explain it far better than I can.
With DBC, if contracts are enabled (in Eiffel you can disable contracts
to get better performance), your method is tested everytime you call the
method. It tests pre-conditions and/or post-conditions (providing you
the old state for your post-conditions). Also after every call to a
method the class invariant is tested to validate your object. That’s the
basic idea.

One way that could probably be implemented (though, only as of ruby 1.9
due to &block params) might look this:

class DayOfTime
extend DBC
invariant {
@hour.between?(0,23)
@min.between?(0,59)
@sec.between?(0,59)
}

define(:min=) {
pre_condition { |value|
value.kind_of?(Numeric)
value.between?(0,59)
}
body { |value|
@value = value
}
post_condition { |old, value|
@hour == old.hour
@min == value
@sec == old.second
}
}
end

Regards
Stefan

micathom wrote:

Would be a very nice feature if ruby had that, even though I’d introduce
additional syntax and I’ve no idea how to integrate it nicely.

There are a few libraries around that try to do this (google for “ruby
dbc” or similar). Most of these efforts are not entirely convincing.
Unfortunately, the ruby community seems somewhat dogmatic in arguing
against dbc. Maybe because it’s not enough of a duck.

I know, but I found none of them convincing enough. Might change with
ruby2, as there are more possibilities then.
IMHO DBC is orthogonal to duck typing. It changes just what you test
for. Instead having a contract that states “argument must be a duck” you
just will have “argument must quack like a duck”. I fail to see how that
goes against duck typing.

Regards
Stefan

I fail to see how that goes against duck typing.

Me too.