On Mon, Apr 13, 2009 at 7:46 AM, aslak hellesoy
[email protected] wrote:
So if someone develops a better AutoTest with a plugin architecture, and
that doesn’t have to run as a long/lived process, then I’d be very
interested in writing the neural network part - possibly backed by FANN
(FANN - FANN)
In the immortal words of Socrates, “That rocks.”
The nice thing about separating concerns like this – the reason why
design patterns appeal so much to me – is that pieces can rather
easily be built up incrementally. As Matt said, thinking ‘neural
networks’ with this is way beyond the level of anything I’d had in my
head. But it’s a damn cool idea.
I can think of a few ways to handle this particular chunk, with levels
of complexity that would scale up:
1.) DUMB REACTION: Just re-run the tests that failed in this run
cycle. Then, periodically, run tests that succeeded in the background
as a regression check. (This isn’t much beyond what Autotest does
now.)
2.) OBSERVERS: Allow handlers to register themselves against certain
files, so that when a file is changed, that handlers gets run.
Multiple handlers can observe any given file, and a handler can
declare multiple rules, including directories or pattern matches.
(Again, Autotest has something sort of like this, but not nearly as
flexible.)
3.) PERSISTENCE: Track the history of tests and the times they were
created, edited, last run, and last failed. Also track file
modification times. When a file changes, run the tests first that are
either new or have failed since the last time the file was changed.
Then run the tests that went from failing to passing in that time.
(This could certainly be improved – I haven’t sat down to figure out
the actual rule set – but you get the gist. Know when things changed
and set priorities accordingly.)
4.) INTELLIGENCE: Aslak’s neural network. Let the system figure out
which tests matter to which files, and run what it thinks it ought to
run. Maybe use code coverage analysis. It can ‘learn’ and improve
when the full suite is run and it discovers new failures.
In all four of these cases, I still think it’s imperative to run the
full suite. None of these methods are foolproof, and code is tricky
and makes weird things happen in weird crevices. That’s why testing
must be done. But running the suite doesn’t have to be a ‘blocking’
activity, like it is with Autotest now. It can happen in bits and
pieces, when nothing else is going on, and it can be configured to
only grab your attention when something failed unexpectedly.
(That’s one of the prime reasons for the ‘multiple output views’
feature, by the way. When I said output views I wasn’t just thinking
of the console or a window. I’m also thinking Dashboard widgets, or
gauges in the toolbar, or RSS feeds, or dynamic wallpaper, or whatever
else anyone can think of. Stuff that stays out of your way until it
either needs you or you choose to look at it.)
Still making sense? This is starting to sound pretty big and pretty
complex – but I don’t think it strictly needs to be. #1 and #2 above
are pretty easy. The others don’t have to be built before releasing
something. And, of course, you wouldn’t have to pick just one
selection module. You could run or disable all of these depending on
your project’s needs, or come up with your own system involving Tarot
cards and Linux running on a dead badger.(*)
I just want to build a core that detects changes in stuff, tells other
stuff about it, and passes on what that stuff says about it to a third
set of stuff. The rest is implementation-specific details. >8->
(* http://www.strangehorizons.com/2004/20040405/badger.shtml )
–
Have Fun,
Steve E. ([email protected])
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org
