You can use a predetermined random number sequence (or sequences) for
unit testing, where I’d suggest you ought to be working in a fairly
black and white world. Establishing expected results if the number of
drawing from the RNG is large doesn’t look like a fun job. Could you
use mock objects to provide predetermined results at a higher level?
If I used a predetermined random number I cannot make much changes in
the optimization algorithm, e.g. if I optimize it and it needs less
random numbers the result cannot be the same.
I don’t see that your problem really falls into the unit test category
though, it looks more like a user, performance or acceptance test
issue, since you’ll pretty much have to perform multiple runs to
establish the success rate, and the program hitting or missing the 90%
criterion is still not going to tell you definitively if it’s working.
It’s as much a benchmarking deal as a testing one.
That’s very true, it is an acceptance test. Nevertheless I would love
an automated test for it, I need some way to automatically find out
After some googling I think I should have a close look how statistics
deal with this kind of problem: