On May 16, 2010, at 8:13 PM, David C. wrote:
On May 16, 2010, at 12:54 PM, Scott T. wrote:
I’m wondering if anyone has any experience with an automated test-case generation tool like Quickcheck (for erlang/haskell). I’d be interested in hearing any impressions, war stories, or dev workflows regarding a tool like this. Talking off list to David C, he suggested that it might be a complimentary tool to a TDD/BDD framework like rspec.
My thinking here is that it could be useful to drive out an initial implementation using TDD, and at the point we think we’ve got the solution we want, add something quickcheck-like to try to poke holes in it. I’d probably then add new examples if any cases I hadn’t considered were revealed through this process.
Have you watched John Hughes’ presentation on the matter?
It’s sort of interesting that he won’t do any TDD - he’ll let the
reduction process generate the “minimum” test case, and go from there
(that’s not explicitly stated in that video, although I’m pretty sure
I’ve heard him say it before).
If I had a tool like this, I’m guessing I’d probably have a workflow
like the following:
- use the random test case generator, and fix any issues that were
- If something wasn’t obvious, I’d go and write a test case for in a
more traditional testing tool (rspec). I often use the debugger in
conjunction with the spec runner, running the one test case with a
debugger statement at the start of the test case.
- Any regressions would (obviously) happen in the traditional tool.
The big win with a tool like this is not testing boundary cases, it’s in
having the tool “write” the test cases for you. OTOH, I wonder if the
simplicity of the implementation would be sacrificed when taking this
Another drawback - I have no idea how such a tool would integrate with a
It appears as though there is a similar project out there for ruby named rushcheck (http://rushcheck.rubyforge.org/).
It’s up on github too: http://github.com/hayeah/rushcheck. Same guy has this too: http://github.com/hayeah/rantly - random data generator - looks like you could do stuff like:
There’s a blog post about the library here, if anyone is interested:
I’ve been thinking about integrating a port of the ruby library faker
This would cause 100 random strings to be generated and passed to thing.method_that_accepts_a_string. Assuming the matcher verifies some set of rules about the outcomes, you’ve basically got quick check.
Yeah, pretty much. One issue, though, is that you don’t want to hard
code the number of random generations. You’ll also want a convenient
way to run just one given test case easily (which rspec already has).
You’ll probably also want to separate these random generation tests from
the rest of your tests. Hitting a database 1000 times for one test is
going to be costly. Now that I’m thinking about it, it might make a ton
of sense in languages like erlang or haskell where everything is
functional because those languages lend themselves to parallelization
since there are no shared resources.