# Unit testing with uncertainty

**URL:** <https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490>\
**Category:** Ruby\
**Created:** [November 10, 2006, 8:41am UTC](https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490 "2006-11-10T08:41:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![martinus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f08c70/32.png) [@martinus](https://www.ruby-forum.com/u/martinus)\
**Post date:** [November 10, 2006, 8:41am UTC](https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490/1 "2006-11-10T08:41:30Z")

</div>

Hi all, I am writing an particle swarm optimizer that tries to find  
good floating point values for optimization problems. I have several  
test cases, and some target fitness value that the optimizer should be  
able to reach. The problem is that since such optimizers make lots of  
use of random numbers, so the optimizer can only find in about 90% of  
the cases the target fitness value. I can relax the target fitness  
value, than 95% of the runs are ok; but that is not really a solution.

Any ideas how to write tests when you have such an uncertainty about  
the result?

Martin

---

<div class="post-metadata">

**Author:** ![martinus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f08c70/32.png) [@martinus](https://www.ruby-forum.com/u/martinus)\
**Post date:** [November 10, 2006, 9:13am UTC](https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490/2 "2006-11-10T09:13:32Z")

</div>

On 10/11/06, martinus [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Martin

Could you make use of assert\_in\_delta? It’s in Test::Unit. You can  
either check that each attempt is within a certain tolerance, or you  
can run a number of attempts and check that the number of successfult  
ones falls within the threshold.

Farrel

---

<div class="post-metadata">

**Author:** ![martinus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f08c70/32.png) [@martinus](https://www.ruby-forum.com/u/martinus)\
**Post date:** [November 10, 2006, 10:25am UTC](https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490/3 "2006-11-10T10:25:45Z")

</div>

On Nov 10, 7:39 am, “martinus” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Martin

You can use a predetermined random number sequence (or sequences) for  
unit testing, where I’d suggest you ought to be working in a fairly  
black and white world. Establishing expected results if the number of  
drawing from the RNG is large doesn’t look like a fun job. Could you  
use mock objects to provide predetermined results at a higher level?

I don’t see that your problem really falls into the unit test category  
though, it looks more like a user, performance or acceptance test  
issue, since you’ll pretty much have to perform multiple runs to  
establish the success rate, and the program hitting or missing the 90%  
criterion is still not going to tell you definitively if it’s working.  
It’s as much a benchmarking deal as a testing one.

–Mike

---

<div class="post-metadata">

**Author:** ![martinus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/f08c70/32.png) [@martinus](https://www.ruby-forum.com/u/martinus)\
**Post date:** [November 10, 2006, 6:41pm UTC](https://www.ruby-forum.com/t/unit-testing-with-uncertainty/75490/4 "2006-11-10T18:41:00Z")

</div>

> You can use a predetermined random number sequence (or sequences) for  
> unit testing, where I’d suggest you ought to be working in a fairly  
> black and white world. Establishing expected results if the number of  
> drawing from the RNG is large doesn’t look like a fun job. Could you  
> use mock objects to provide predetermined results at a higher level?

If I used a predetermined random number I cannot make much changes in  
the optimization algorithm, e.g. if I optimize it and it needs less  
random numbers the result cannot be the same.

> I don’t see that your problem really falls into the unit test category  
> though, it looks more like a user, performance or acceptance test  
> issue, since you’ll pretty much have to perform multiple runs to  
> establish the success rate, and the program hitting or missing the 90%  
> criterion is still not going to tell you definitively if it’s working.  
> It’s as much a benchmarking deal as a testing one.

That’s very true, it is an acceptance test. Nevertheless I would love  
to have  
an automated test for it, I need some way to automatically find out  
regressions.

After some googling I think I should have a close look how statistics  
deal with this kind of problem:

> **[Confidence interval](https://en.wikipedia.org/wiki/Confidence_interval#Practical_example)**
>
> In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated confidence level; the 95% confidence level is most common, but other levels, such as 90% or 99%, are sometimes used. The confidence level, degree of confidence or confidence coefficient represents the long-run proportion of CIs (at the given confidence level) that theoretically contain the true value of the parameter; this is tantamount to Facto...

Martin