Concerns over Rails' handling of tests


#1

Sorry if this comes across as a bunch of disjointed thoughts…I’m just
trying to put my thoughts down and I’d like to know what other people’s
opinions on the subject are.

I’ve been working with Rails for 3 or 4 months now and I’m constantly
trying
to look at ways of improving my testing techniques, especially when it
comes
to TDD. I have a few concerns about the way Rails approaches tests with
regards to controllers/models.

Rails has always seemed to apply that you unit test your models and
functional test your controllers. For me however, this distinction
breaks
down in practice and leads to what are in my opinion very smelly tests.

Firstly, I don’t like the way that by default Rails assumes you want to
test
an entire controller in a testcase. I’ve found that this leads to tests
that
are far to big. I believe better breakdown would be to have a test case
for
each action, or even a testcase for each action in different contexts.
So,
as an example, say you had an ArticleController with view, post and
create
actions. The post action simply displays the form which submits to the
create action.

Now, instead of having one large ArticleControllerTest, it would make
far
more sense to break it down into ArticleViewTest, ArticlePostTest,
FailedArticleCreate test and SuccessfulArticleCreate, because each of
those
represent different contexts, or fixtures.

I’m quite keen on Ben G.’ testing approach outlined
herehttp://www.reevoo.com/blogs/bengriffiths/2005/06/24/a-test-by-any-other-name/and
have also been looking into Dave A.’
RSpec http://rspec.rubyforge.org/ and have to agree with the idea of
testcases being bound by fixtures. If you have test methods that require
common setups, the code should be extracted to the setup method. If you
have
different groups of methods using two different types of setups, this is
an
indication that you should be creating a new testcase for the new
fixture.
Overall, I’ve found that trying to test a whole controller leads to
multiple
fixtures in one testcase and multiple violations of DRY.

This also leads me on to my other concern, and that is the practice of
testing for the presence of markup in a view using assert_tag. I can’t
really see the value in this, and again it seems to violate DRY to me.
Why
would you want to assert that a tag exists in most cases? For example,
given
the ArticlePostTest example above, which wouldn’t really do much in the
controller, and simply pass to the view for the rendering of the form.
Why
would I want to test for the structure of the form? Ultimately my form
will
be laid out using CSS but there are still several ways I could markup my
form, with all variations being “correct”. By testing for one particular
implementation in my test, I’m duplicating the markup in my test and my
view, and binding the test to my implementation of the form. Testing
that a
particular HTML element is present in my view does nothing to tell me
that
my view is “correct”, therefore I think this makes the test worthless.

Which leads me on to saying that views such as forms are the realm for
proper functional testing using something like Watir or Selenium, not
TestUnit and tag assertions. You want to check that you can fill in the
form
and submit it, not that the page contains a tag.

So what would I really like to see? No separation of unit and functional
testing in terms of model and controller code. They should all be
covered by
unit tests. Functional testing should be the realm of something like
Watir
or Selenium, unless somebody can put together a good library that lets
you
do this kind of functional testing (i.e., filling in forms, interacting
with
your web application as if it were in a browser) from within your
TestUnit
tests. I’d like to see less emphasis on the idea of testing your
controllers
as a whole and testing your different actions as smaller, focussed unit
tests. I think this would also aid with TDD.

Dave A.’ has some good reading material here:

http://blog.daveastels.com/?p=53

What do you think?

Cheers
Luke


#2

At the risk of starting a flame war, I see no difference between
testing the controller in one test or testing every element of the
controller in seperate tests. It sounds like semantics to me.

The important thing is that there is actual testing going on.
Everything after that is gravy. :slight_smile:


#3

Another difference this brings to mind is that functional tests (or
Acceptance Tests as XP calls them) should not be written test-first, but
are
instead defined at the beginning of an iteration for a series of user
stories. Acceptance tests do not have to run at 100% all the time (in
fact
they start at 0% and approach 100% as the iteration nears completion),
wherease Unit Tests should always be running at 100%. Acceptance tests
represent what the completed system should do when finished, whereas
Unit
Tests represent what your code actually does.

Its perfectly possible to design controller actions test-first, but this
is
still unit testing. Acceptance testing an action would be to load it in
your
browser, fill in the form, submit it, and check it worked. This is why I
mention things like Watir or Selenium for acceptance testing over
TestUnit.
FIT would be another option.

This is just more evidence to me, that Rails’ distinction of unit tests
for
models and functional tests for controllers, is wrong.

Cheers
Luke


#4

maybe testing should be broken into the same levels as the app, M,V,C…

tests/models/user/*_test.rb

test/controllers/user/view_test.rb
test/controllers/user/add_test.rb

test/view/user/view_test.rb

where view testing would be something more like what you are suggesting,
using FIT/Watir/Selenium etc… model tests would be what 'unit’t tests
currently are, and ‘controller’ tests would be current functional tests

I agree with the idea of having more, better targeted test suites as
well…

rather than having one test with a bunch of assertions for adding a
user, you could have controllers/user/add_test.rb with a bunch of tests
with single or very few, very specific, tests

just brainstorming here, I suspose it wouldn’t be hard to implement this
(or whatever else you want) in the current rails framework… but I agree
that the topic is worthy of discussion


#5

rather than having one test with a bunch of assertions for adding a
user, you could have controllers/user/add_test.rb with a bunch of tests
with single or very few, very specific, tests

wow, I must have been out of it… what I meant to say is that having
individual test suites for individual functions of your controller, is
better than having one big test suite for everything to do with your
controller. As well breaking controller tests into controller and view
would be nice.

I’m not sure how my last email came out so incoherent, I better go eat
lunch before my head explodes.


#6

On 16-nov-2005, at 18:22, Luke R. wrote:

discussion. I’m not trying to start a flame war. I’m well aware I
can implement my own ideas in my own code and intend on doing so.
But I do think Rails’ take on things muddies the waters a bit.

I second that opinion as long as you take personal responsibility for
getting FIT/Watir/Selenium/Whatever to drive At Least One Browser on
my trusty old PowerBook.

As long as it is not the case (and considering many Rails developers
are keen on their PowerBooks) your idea wouldn’t find as much support
as you expect.


Julian “Julik” Tarkhanov


#7

Well, I know Selenium works in quite a few browsers and the next version
of
Watir is meant to have support for more than just IE.

Also, I think some of you may have misunderstood what I meant with
regards
to removing the distinction between functional testing controllers or
unit
testing models. I was suggesting that they should both be considered
unit
tests (something the rake tasks test_unit and test_functional doesn’t
seem
to imply) but I wasn’t suggesting merging the file structure - that is
fine
as it is. I also think it would be a bit OTT to create a new file for
every
single test case. The examples I suggested would make good test cases
but
all of your test cases for a particular controller would also make a
very
logical test suite and could probably still remain in the same file, if
you
want.

Cheers
Luke


#8

I agree with some of this. In my current project, I have my functional
tests split out per action: test/functional/controller_action_test.rb

I’ve done some poking around in the past, and any subdirs of test/unit
and test/functional get pulled in on the test suite, so there’s nothing
stopping you from setting up a test tree that looks like this:
test/functional/[module/]controller/action_test.rb

I also think that the more esoteric uses of assert_tag that I’m ending
up using could be done much better with FIT or something, rather than
having to assert (as generically as possible) that I have a tag with a
certain ID, with X or X+1 child nodes (+1 for optional header row on a
table). Being able to use browser-based acceptance testing would be
great, but probably a bit more difficult to do (how to handle fixtures?)
in a generic way.

As far as your other points go, I think you might be overreacting a
little. If you want to split your unit tests into a case by case basis
(ArticleViewTest, ArticlePostTest, FailedArticleCreateTest and
SuccessfulArticleCreateTest) for ease of reading and maintenance, then
by all means do so. Nothing here stopping you. All the support you
need as far as the testrunner goes is there. I don’t know that forcing
it upon everyone would be all that useful.

Merging functional tests into the units hierarchy is a bad idea, even
though as you say they really are unit tests. Merging your model and
controller tests would be an ease-of-use nightmare. Even moreso than us
developers writing output tests under functionals.

  • Jamie

#9

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Nov 16, 2005, at 4:09 PM, Luke R. wrote:

I would be interested. I still have to think about this some more
though. As I said, I’m just throwing thoughts out there for now.

A change in semantics is certainly part of what I’m advocating. In
fact, I’d love to see support for RSpec down the line once the code
base has matured and stabilised a bit - at the moment its not that
different to Test::Unit but I believe the future plan is to evolve
it into a complete DSL in its own right.

Getting Rails on the leading edge with RSpec would be sweet.

It’ll take a bit of work to go through the additional assertions
Rails provides for Test::Unit::TestCase and RSpecify them.

Do it, man :slight_smile:

jeremy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)

iD8DBQFDe8vGAQHALep9HFYRAvItAKCOl6YtPvrole0uCAOV6R0UDtSRDQCeNaf0
n1t6KvP21r3ZQI+eoz2Qs4w=
=6oS6
-----END PGP SIGNATURE-----


#10

I would be interested. I still have to think about this some more
though. As
I said, I’m just throwing thoughts out there for now.

A change in semantics is certainly part of what I’m advocating. In fact,
I’d
love to see support for RSpec down the line once the code base has
matured
and stabilised a bit - at the moment its not that different to
Test::Unit
but I believe the future plan is to evolve it into a complete DSL in its
own
right.

Cheers
Luke


#11

At the risk of starting a flame war, I see no difference between
testing the controller in one test or testing every element of the
controller in seperate tests. It sounds like semantics to me.

Well to me, thats like saying “there is no difference between having
everything wrapped up in one God class or separated into smaller, more
specific fine-grained classes”. In fact, why have separate test cases
for
each controller, why not one test case for your whole app? :wink:

I think tests should be given as much as care and consideration as your
production code. If they start getting bloated they should be
refactored,
like your production code. Smelly tests become harder and harder to work
with and IMO this will start to have a detrimental effect on your
testing.

Cheers
Luke


#12

Yeah right,
2005/11/17, Francois B. removed_email_address@domain.invalid:

After reading about rspec here yesterday, I played around with it.
Works like a charm. I’m not using it under Rails, but I liked what I
saw. I already patched a few things in RSpec.

What I REALLY meant to say is that I’ll probably start RSpecing the
Rails assertions.

Bye !


#13

Hi !

2005/11/16, Jeremy K. removed_email_address@domain.invalid:

Getting Rails on the leading edge with RSpec would be sweet.

It’ll take a bit of work to go through the additional assertions
Rails provides for Test::Unit::TestCase and RSpecify them.

Do it, man :slight_smile:

After reading about rspec here yesterday, I played around with it.
Works like a charm. I’m not using it under Rails, but I liked what I
saw. I already patched a few things in RSpec.

I am developing a plugin using RSpec instead of Test::Unit. It’s an
HABTM helper.

More info on my blog later today.

Bye !


#14

http://dev.rubyonrails.org/ticket/2923

I’ve submited a research ticket and patch regarding this…

The attached patch will allow for

test/models/
test/views/
test/controllers/

as well as maintaining compatibility with the current
test/unit/
test/functional/

While it may not be neccesary since test/unit and /functional still
work, i modified the default generators to use the new m/v/c test
layout…

but anyway, i put it as RESEARCH in the hopes that others will poke at
it, maybe come up with a better idea, or additional ideas…

I think that having a separate place for view specific tests could go a
long way in organization, as well as paving the way for a proper,
seperate, view-specific testing infrastructure such as FIT etc.

However even just the separation itself i think is worth it.

Testing if your controller functions as intended is different than
testing what the user sees, I think we’ve all come to accept that which
is why we are using a MVC framework - it doesn’t make sense to not
follow the same metaphor for tests!

I


#15

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Nov 17, 2005, at 3:20 PM, Jerrett T. wrote:

is why we are using a MVC framework - it doesn’t make sense to not
follow the same metaphor for tests!

My hunch is that splitting controller and view testing will create a
lot of busywork without increasing the value of the tests.

Testing action bar on controller foo involves running through each
story for that action and verifying that it gives the expected results.

Testing view bar for controller foo involves… running through each
story for that view and verifying that is gives the expected results.

So their usage is largely one-to-one, though there are other cases
such as shared partials and layouts.

I think clean, expressive tests are great, but if divorcing the
controller from its views makes testing more of a chore then I’m
happy to lose the benefit that this restructuring provides.

Best,
jeremy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)

iD8DBQFDfRcVAQHALep9HFYRAjE9AJsGN0hKSmZUyKAnk+OjK5zAfTfwMACdH9EM
0HPaGtULMon1mF56z2p+JK8=
=Y7Cq
-----END PGP SIGNATURE-----


#16

My hunch is that splitting controller and view testing will create a
lot of busywork without increasing the value of the tests.

Testing action bar on controller foo involves running through each
story for that action and verifying that it gives the expected results.

Testing view bar for controller foo involves… running through each
story for that view and verifying that is gives the expected results.

Currently that should probably be two separate tests anyway. If you are
testing to ensure data is created after a successful action, you don’t
want the test to fail because there was a missing tag in your view.
You’d want a different test to fail, and your test that checks for the
data to be created to pass, since the data was in fact created.

Currently this means writing two tests in the functional tests, one
dealing with the expected behaviour of the app, and one dealing with the
expected output of the app. To me, this makes sense to break them apart.
Also that means when you are working on views, you can TDD (BDD,
whatever) your views without having to mess with (and run all of) your
controller tests, by running rake test_views

If you have people on your team that deal primarily with creating the
front end, they can still be given a place to do their testing without
cluttering the controller tests.

So their usage is largely one-to-one, though there are other cases
such as shared partials and layouts.

I think clean, expressive tests are great, but if divorcing the
controller from its views makes testing more of a chore then I’m
happy to lose the benefit that this restructuring provides.

I think that with this restructuring it doesn’t get in the way of
anything anyone is doing currently. test/functional and test/unit will
continue to work, and with new projects that have the new structured
test directories, somebody can still decide they would rather have fewer
tests with many assertions by doing what they do now in test/functional
in test/controller and only use test/view for things like you mentioned,
shared partials and layouts.

In short, this change wouldn’t be forcing anything new anyone, it would
simply be providing the structure to split when the developer finds it
appropriate. For people that subscribe to the more tests, less
assertions per test mentality this could be a big help, for people who
don’t they don’t really have to do anything any differently.

my two (4? 6? 10?) cents


#17

Great thread.

I might add that since Selenium now has Ruby bindings (
http://selenium.thoughtworks.com/driven.html ), and works with
IE/Firefox/Safari (and hence, on *nix/Windows/Mac), integrating it into
rails might be a good way to handle things.

I’d propose:

  1. Include in the basic rails framework the Selenium files and ruby
    bindings.

  2. Rails should apply it’s autogenerate magic to Selenium tests, too.
    Under tests/acceptance, basic stubs for Selenium based tests would
    appear.

  3. The basic idiom for these tests wouldn’t be along the lines of unit
    testing, but rather use cases. So an individual test case would walk
    the virtual user through a sequence of actions (click here, fill out
    form, hit back, change form, etc.), and verify the results.

  4. Since the whole thing is being done integrated and via Ruby, we have
    access to logging, fixtures, models, etc., should we need them. (For
    instance, to verify that an order was purchased, we could check the
    confirmation page in the virtual browser, but also check that the DB
    shows the new order.)


#18

Hi,

I am relatively new to rails so it may be best to take this with a
grain of salt but …

On 11/18/05, Jeremy K. removed_email_address@domain.invalid wrote:

My hunch is that splitting controller and view testing will create a
lot of busywork without increasing the value of the tests.

Testing action bar on controller foo involves running through each
story for that action and verifying that it gives the expected results.

Testing view bar for controller foo involves… running through each
story for that view and verifying that is gives the expected results.

This may be largely true but I guess as a realtive new-comer to rails
this
is NOT how I have been intuitively testing my application. The vast
majority
of my tests look something like the following

def test_new_get
get(:new, {:issue_id => 1}, {“user_id” => 1} )
assert_response(:success)
assert_template(‘new’)
assert_equal(1, assigns(:comment).issue_id)
assert_nil(flash[:notice])
assert_nil(flash[:alert])
end

I test the state of the assigns, flash, session and database after the
action
has been run. I have yet to test the presentation layer in any way
shape or form.

I had planned to start writing tests in the future for the view using
something like

def test_lists
items = …
category = …
render(‘lists’,:locals => {:items => items},
:instance => {:category => category},
:session => {:items => items} )
assert_tag …
assert_tag …
assert_tag …
end

I also assumed that if I ever wrote a web service that they would have
another set of separate tests. So essentially I would have a separate
top-level directory in my test hierarchy for every separate hierarchy in
my app directory except for helpers.

This is just what I assumed from my first touch of rails development.


Cheers,

Peter D.


#19

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Nov 16, 2005, at 3:54 PM, Luke R. wrote:

single test case. The examples I suggested would make good test
cases but all of your test cases for a particular controller would
also make a very logical test suite and could probably still remain
in the same file, if you want.

Luke, from what I can tell, you’re advocating a change in semantics.

The approach to testing you suggest is achievable already, but is not
the default.

It’s a matter of patching the app generator. Interested?

Best,
jeremy
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)

iD8DBQFDe8juAQHALep9HFYRAjouAJwLq2jobC/QEDdRiEmUF5qtlsnQRwCdFFgp
XvoO/gZucqCelSGqUYxIfO4=
=ceHf
-----END PGP SIGNATURE-----


#20

So, further to that, I think it would be great if Rails removed this
distinction as it currently stands, and introduced a means of writing
proper
functional/acceptance tests, ideally using something like
FIT/Watir/Selenium, that can be defined early and made to pass through
the
development of working code (covered by unit tests, written test first).

Of course, I’m very much aware of the whole Rails being opinionated. I
just
think this is an instance where Rails has got it wrong and am voicing my
opinion and trying to provoke discussion. I’m not trying to start a
flame
war. I’m well aware I can implement my own ideas in my own code and
intend
on doing so. But I do think Rails’ take on things muddies the waters a
bit.

Cheers
Luke