Yes, I believe it is and I think it is dangerous, and confusing to
people
new to the concepts of testing.
I’ve mentioned a point similar to this on here before, but with the
announcement of Rails “integration” test, I’m worried that somebody
somewhere is getting their types of tests worryingly mixed up.
Lets start with what we had in the beginning…unit tests (for models)
and
“functional” tests (for controllers). This was wrong from the start,
because
Rails’ idea of functional tests were in fact just UNIT tests for
controllers
(or specifically, single actions in a controller). A unit test being, a
test
that tests a small unit functionality at the code level - it makes sure
that
that unit of code does what it is supposed to do. In a Rails app, a unit
might be a single Model or it might be a single action on a controller.
So what is functional testing? Functional testing is another name for
ACCEPTANCE testing. What is ACCEPTANCE testing? I will quote from this
article:
http://www-128.ibm.com/developerworks/java/library/wa-selenium-ajax/?ca=drs-
Acceptance testing, also known as black-box and functional testing, is a
way
of testing and verifying that an application works according to functional,
non-functional, and other important stakeholder requirements. Acceptance
tests complement unit and integration tests, which are usually written using
xUnit frameworks. Acceptance tests can also be written using a programming
language, but Selenium and similar tools like Fitnesse also support tests
written using a tool-specific document format.
Further more, there are some important rules regarding acceptance tests:
The key point here is that acceptance tests make sure that the system
works end-to-end and complies with its core requirements. In Extreme
Programming terms, this is often a “user story”. A user story is
considered
complete when all acceptance tests pass. Unlike unit tests, which should
ideally be running with a 100% pass rate, acceptance tests start at 0%
and
finish at 100% when the story is fully implemented. From the
extremeprogramming.org website:
Acceptance tests are black box system tests. Each acceptance test
represents
some expected result from the system. Customers are responsible for
verifying the correctness of the acceptance tests and reviewing test scores
to decide which failed tests are of highest priority. Acceptance tests are
also used as regression tests prior to a production release.
A user story is not considered complete until it has passed its acceptance
tests. This means that new acceptance tests must be created each iteration
or the development team will report zero progress.
So now, we have the announcement of Integration Tests for Rails.
Integrations should be tests to ensure that multiple units work together
correctly. But Rails integration tests look just like
acceptance/functional
tests though (much like FIT or Selenium in driven mode)! How confusing
for
the testing newbie!
The reason I bring this up is because I really believe the Rails
developers
should look at clarifying and renaming their tests to get the semantics
correct. How can you expect testing newbies to start testing their apps
when
the framework they are using can’t even get the names of their tests
correct? So what do I propose:
- Rails’ existing “functional” tests should be correctly referred to
as unit tests, which is all they are. For organisational purposes I
can see
the advantage of breaking up your model and controller unit tests as
the
unit testing strategy for these two different types of object are
usually
different. - Rails’ new “integration” tests are renamed to functional or
acceptance tests, which is what they really are. They should be seen
as a
built-in alternative to using a browser-based acceptance testing tool
like
Watir or Selenium. In fact, Rails’ new built-in Integration tests
could be
compared to something like Selenium in “driven” mode instead of
test-runner
mode. Personally, I feel that for web apps Selenium in test-runner
mode is
more appropriate, but thats just my preference. - Integration tests are left to the user to put together if they feel
they need to use them. Their typical usage would be if there are
several
models that need to work together to produce results.
To support the above the tests folder would probably look something like
this:
unit/models
unit/controllers
acceptance/
integration/
fixtures/
I’d like to hear people’s thoughts on the above.
