How to make ZenTest autotest run whenever my tests change

Rails
1

Railsters:

Despite Rails being the only Web platform designed for TDD, a lot of its
test
infrastructure is still “cargo cult” - imitating other test rigs instead
of
understanding their principles.

Most importantly, tests should run instantly. There’s no excuse for
breaking
this rule, and if you invent a platform that can’t obey it then you are
doing
something wrong. And I have yet to meet a combination of editor and test
script
which does this. Even Visual Studio and C++ are faster and easier to
work with.

The Rails problem is apparently rake db:test:prepare, which clones your
database
instead of detecting whether it needs a clone.

I have heard the fix is ZenTest’s autotest. This allegedly loads the
Ruby “VM”
once, awaits file changes, and responds correctly to each one.

So I have a vanilla Rails 2.3.2 application, and I install autotest (and
autotest -v does not work, so I can’t see which version number I
actually get),
and the gem system claims I have version 4.1.3. And I run autotest in
the
project folder, and it says:

$ autotest
(Not running features. To run features in autotest, set
AUTOFEATURE=true.)
/usr/bin/ruby1.8 -I.:lib:test -rubygems -e “%w[test/unit
test/test_helper.rb].each { |f| require f }” | unit_diff -u
Loaded suite -e
Started

Finished in 0.000437 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

Note the system did not even run the whole test batch, first. And note
it sees
and warns that it rejects my features folder. I need it to see my test
folder
(NOT a spec folder!).

Then I make a non-trivial change to a correctly-named test file, save
the file,
and nothing triggers. (Pure Ubuntu Linux substrate, BTW.)

So how do I trigger?

If I could get that working, how can I TDD both story and unit tests AT
THE SAME
TIME, without using two different test runners?

–
Phlip

2

On Mon, Jul 6, 2009 at 10:54 AM, Phlip[email protected] wrote:

Started

Finished in 0.000437 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

sudo gem install autotest-rails. Autotest does not automatically
intuit project directory structures, beyond the simplest, oldest Ruby
idiom (lib/.rb => test/test_.rb). The autotest-* set of support gems
adds support for more complex directory structures, like Rails’.

~ j.

3

Phlip wrote:

Railsters:

Despite Rails being the only Web platform designed for TDD,

Really? I have the impression that there are others…if nothing else,
some of the Railsy PHP frameworks have copied this feature.

[…]

Most importantly, tests should run instantly.

In most cases, mine do. The exceptions are those which use regexps or
DOM wizardry for parsing HTML. I suspect Oniguruma might help here, but
I haven’t tried it.

And I have yet to meet a combination of editor and test
script
which does this.

RSpec + autotest + out-of-the-box test scripts + any editor you like.
Done.

The Rails problem is apparently rake db:test:prepare, which clones your
database
instead of detecting whether it needs a clone.

When would it not need a clone? Tests are supposed to start from a
clean environment.

I think the real time sink may be loading the Rails environment, which
seems to typically take several seconds on an otherwise fast test run.
But I can wait that long for my tests to run.

Jay Fields has posted some stuff on his blog about how to make Rails
unit tests not touch the DB, but I don’t really see much need to do
that.

I have heard the fix is ZenTest’s autotest. This allegedly loads the
Ruby “VM”
once, awaits file changes, and responds correctly to each one.

Yup. That’s what it does as far as I understand.

So I have a vanilla Rails 2.3.2 application, and I install autotest (and
autotest -v does not work,

-v is verbose, not version. The autotest script is part of the ZenTest
package, and doesn’t appear to have a version number of its own.

so I can’t see which version number I
actually get),
and the gem system claims I have version 4.1.3.

Of what? ZenTest? (FWIW, I’m using ZenTest 3.11.1; I’ll have to try
the newer version.)

And I run autotest in
the
project folder, and it says:

$ autotest
(Not running features. To run features in autotest, set
AUTOFEATURE=true.)
/usr/bin/ruby1.8 -I.:lib:test -rubygems -e “%w[test/unit
test/test_helper.rb].each { |f| require f }” | unit_diff -u
Loaded suite -e
Started

Finished in 0.000437 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

Note the system did not even run the whole test batch, first.

Bizarre. It does for me with RSpec. Are your tests in the directories
that Autotest is searching (which would appear to be test/unit)?

And note
it sees
and warns that it rejects my features folder.

Not quite. It is warning you that it did not even look for your
features folder, and will not look for it unless you set
AUTOFEATURE=true. So try that. (Personally, I think features are too
time-consuming to run as often as tests, because I tend to have them do
a lot of browser simulation.)

I need it to see my test
folder
(NOT a spec folder!).

It should do that. Check your configuration.

Then I make a non-trivial change to a correctly-named test file, save
the file,
and nothing triggers. (Pure Ubuntu Linux substrate, BTW.)

So how do I trigger?

Again, this is likely a ZenTest/autotest config issue. What you’re
describing as desired is exactly what ZT/AT should normally do.

If I could get that working, how can I TDD both story and unit tests AT
THE SAME
TIME, without using two different test runners?

Set AUTOFEATURE=true, like it’s telling you in that message. ZT/AT will
call rake test and cucumber as appropriate.

–
Phlip

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]

4

John B. wrote:

0 tests, 0 assertions, 0 failures, 0 errors

sudo gem install autotest-rails

Yay thank you the good news is that runs now. (I would have gone with
test/**/*suite.rb myself…)

The bad news is it’s still slow as f—. Yes Gherkin is partly to blame.
But
does anyone have a replacement for rake db:test:prepare that does less??

–
Phlip

5

Marnen Laibow-Koser wrote:

Despite Rails being the only Web platform designed for TDD,

Really? I have the impression that there are others…if nothing else,
some of the Railsy PHP frameworks have copied this feature.

designed for - not copied on.

Most importantly, tests should run instantly.

In most cases, mine do. The exceptions are those which use regexps or
DOM wizardry for parsing HTML. I suspect Oniguruma might help here, but
I haven’t tried it.

Rails??

RSpec + autotest + out-of-the-box test scripts + any editor you like.
Done.

When the tests fail, what button do you type to navigate instantly to
the test
failure?

The Rails problem is apparently rake db:test:prepare, which clones your
database
instead of detecting whether it needs a clone.

When would it not need a clone? Tests are supposed to start from a
clean environment.

We do not build a new CPU with new libraries and Ruby installation
between each
test run. All test isolation is a trade-off of some kind.

Jay Fields has posted some stuff on his blog about how to make Rails
unit tests not touch the DB, but I don’t really see much need to do
that.

Thanks - that’s the next thing I was going to … rail at. Some
consultants who
teach TDD use a mind-trick to motivate their newbs to decouple their
code. In a
big-iron environment where database migrations are expensive, until
you pay
that down, you can get a slight benefit to unit tests that reach into
code that
decouples from the database.

We should do that too. And when we need a fixture, to start such a test
case, it
should come from the database.

–
Phlip

6

RSpec + autotest + out-of-the-box test scripts + any editor you like.
Done.

FWIW, this is still not competing with using Alt+Tab to switch to a
console,
then running script/cucumber features/whatever.feature. Autotest seems
to just
get mired in unit tests that I did not recently change, or something!

–
Phlip

7

On Mon, Jul 6, 2009 at 12:10 PM, Phlip[email protected] wrote:

John B. wrote:

sudo gem install autotest-rails

Yay thank you the good news is that runs now. (I would have gone with
test/**/*suite.rb myself…)

Excellent, glad it worked for you. Suites were originally bad for
autotest because it attempts to do two-way mapping between test and
implementation files, and suites generally aggregate a bunch of test
files.

The bad news is it’s still slow as f—. Yes Gherkin is partly to blame. But
does anyone have a replacement for rake db:test:prepare that does less??

Are you absolutely certain that it’s the DB load? Have you benchmarked
‘rake db:test:prepare’ against ‘rake environment’? The latter is
basically the minimum price you pay for a Rake task that loads Rails.

~ j.

8

John B. wrote:

Are you absolutely certain that it’s the DB load?

Nope!

Have you benchmarked
‘rake db:test:prepare’ against ‘rake environment’?

Of course not!

The latter is
basically the minimum price you pay for a Rake task that loads Rails.

$ time rake db:test:prepare

real 0m2.153s
user 0m1.324s
sys 0m0.324s

$ time rake environment

real 0m1.623s
user 0m1.316s
sys 0m0.280s

Rhetorical question: So if that’s the fast part, what is so f—ing
slow???

–
Phlip

9

Phlip wrote:

Marnen Laibow-Koser wrote:

Despite Rails being the only Web platform designed for TDD,

Really? I have the impression that there are others…if nothing else,
some of the Railsy PHP frameworks have copied this feature.

designed for - not copied on.

Copying a feature in the design phase is still designing it in. Or were
you going to say that Ruby wasn’t really designed for most of its cool
features because they were copied from Smalltalk and Lisp? :slight_smile:

Most importantly, tests should run instantly.

In most cases, mine do. The exceptions are those which use regexps or
DOM wizardry for parsing HTML. I suspect Oniguruma might help here, but
I haven’t tried it.

Rails??

Huh? Oniguruma is what I haven’t tried.

RSpec + autotest + out-of-the-box test scripts + any editor you like.
Done.

When the tests fail, what button do you type to navigate instantly to
the test
failure?

Cmd-Tab. The test failure is usually where I was last editing. :slight_smile:

Seriously, you’re asking the wrong person. I don’t really care that
much about that level of editor integration. If you do, I believe there
is a TextMate bundle that will do what you want, but you will get better
answers from others on this list.

[…]

We do not build a new CPU with new libraries and Ruby installation
between each
test run.

No, because we can guarantee that those bits won’t be changed in a
typical app. We cannot guarantee that about the DB, can we?

I would want a new Ruby installation for each test run if I thought my
code were likely to modify the system Ruby installation as it ran.

All test isolation is a trade-off of some kind.

Sure.

Jay Fields has posted some stuff on his blog about how to make Rails
unit tests not touch the DB, but I don’t really see much need to do
that.

Thanks - that’s the next thing I was going to … rail at. Some
consultants who
teach TDD use a mind-trick to motivate their newbs to decouple their
code. In a
big-iron environment where database migrations are expensive, until
you pay
that down, you can get a slight benefit to unit tests that reach into
code that
decouples from the database.

We should do that too.
[…]

Do what? You mentioned several things, and I’m not sure which one
you’re referring to.

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]