# Probabilistic BDD?

**URL:** <https://www.ruby-forum.com/t/probabilistic-bdd/74638>\
**Category:** Ruby\
**Created:** [November 7, 2006, 10:51am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638 "2006-11-07T10:51:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:51am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/1 "2006-11-07T10:51:01Z")

</div>

Hi,

I’m playing around with BDD à la test/spec and foudn that I need to  
specify properties probabilistically ie saying that they are  
likely/unlikely. Has there been any previous work along these lines?

Should we add something like this to test/spec (Christian are you  
listening? :))

diff -rN old-testspec/lib/test/spec.rb new-testspec/lib/test/spec.rb  
286a287,300

> ```
> def likely(specname, probability = 0.99, &block)
> unlikely(specname, 1.0 - probability, "likely") {!block.call}
> end
> 
> ```

so that one can write specs like

> end  
> ?

This should probably be generalized so that the number of repetitions  
to run depends on the probability of the event but I think you get the  
idea.

Comments?

/Robert F.

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:51am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/2 "2006-11-07T10:51:01Z")

</div>

> > end

Forgot one thing:

Some decision has to be made if setup code should be run before each  
eval of the block or only before the repeated repetitions. To make it  
more in line with the rest of test/spec maybe setup code should run  
before every eval of the unlikely/likely blocks?

/Robert F.

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:51am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/3 "2006-11-07T10:51:03Z")

</div>

> likely/unlikely. Has there been any previous work along these lines?
> 
> Should we add something like this to test/spec (Christian are  
> you listening? :))  
> […]
> 
> > likely "that two consecutive calls to rand gives
> 
> Comments?

I think that the example you give is not appropriate for testing rand(),  
and  
pretty much any code where the result is expected to conform to a set of  
statistical properties. If you take a look at randomness test suites  
like  
Diehard there are a battery of different tests that should be applied  
before  
data can be called ‘random’ with any confidence.

> **[Diehard tests](https://en.wikipedia.org/wiki/Diehard_tests)**
>
> The diehard tests are a battery of statistical tests for measuring the quality of a random number generator. They were developed by George Marsaglia over several years and first published in 1995 on a CD-ROM of random numbers.
> Most of the tests in DIEHARD return a p-value, which should be uniform on \[0,1) if the input file contains truly independent random bits. Those p-values are obtained by p = F(X), where F is the assumed distribution of the sample random variable X – often normal. But that a...

The tests as you have written them would be satisfied by any number of  
broken PRNGs, or even NRAAGs (Not Random At All Generators) (eg  
alternating  
‘1’ and ‘2’ ;). In particular, unlikely events must occur sometimes and  
likely events must fail to occur sometimes, so some form of === seems  
better  
than \<=.

## If you wanted to test RNGs then you need to run a whole series of tests

either like the Diehard tests or just basic stuff like chi square,  
binomial,  
monte-carlo calculation of pi etc.

More generally, I think that ‘likely’ and ‘unlikely’ are going to be so  
context dependant that the user would be better off writing their own  
test  
code, surely? I can see a place for should\_be\_random, but likely and  
unlikely strike me as a bad idea. In any case, when running test code I  
expect that it will give me the same result every time, so any tests  
should  
at least have that property.

Sorry to sound negative. ☹

ben

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:51am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/4 "2006-11-07T10:51:07Z")

</div>

> ‘1’ and ‘2’ ;). In particular, unlikely events must occur sometimes and  
> likely events must fail to occur sometimes, so some form of === seems better  
> than \<=.
> 
> If you wanted to test RNGs then you need to run a whole series of tests -  
> either like the Diehard tests or just basic stuff like chi square, binomial,  
> monte-carlo calculation of pi etc.

I don’t want to test RNG’s; that was just the smallest possible  
example use of the likely/unlikely methods I could think of. I’m  
fairly well versed in RNG testing, thank you.

> More generally, I think that ‘likely’ and ‘unlikely’ are going to be so  
> context dependant that the user would be better off writing their own test  
> code, surely? I can see a place for should\_be\_random, but likely and  
> unlikely strike me as a bad idea. In any case, when running test code I  
> expect that it will give me the same result every time, so any tests should  
> at least have that property.
> 
> Sorry to sound negative. ☹

It is ok to be negative but I have run into test situations many times  
where there is an element of varying behavior involved and specifying  
exactly what is to be expected can only be done by running multiple  
tests and making claims about overall properties of the results.

But it may be the case that people should write their own test code for  
it yes.

Still I think this is an important discussion in general since for  
complex algorithms where it is costly to calc the exact expected  
results ways to write partial specs are important.

Regards,

Robert

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/5 "2006-11-07T10:52:06Z")

</div>

Robert F. wrote:

> > I think that the example you give is not appropriate for testing  
> > rand(), and  
> > pretty much any code where the result is expected to conform to a set of  
> > statistical properties. If you take a look at randomness test suites like  
> > Diehard there are a battery of different tests that should be applied  
> > before  
> > data can be called ‘random’ with any confidence.
> > 
> > [Diehard tests - Wikipedia](http://en.wikipedia.org/wiki/Diehard_tests)

Speaking of numerical test suites, do you happen to know of any test  
suites on line for elementary functions? I used to know of one, but  
haven’t been able to find it. Nor have I been able to find my copy of  
“Software Manual for the Elementary Functions”.

And I’m _not_ talking about “paranoia” … that just tests arithmetic,  
and I found it.

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/6 "2006-11-07T10:52:05Z")

</div>

On 11/1/06, Robert F. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > ‘1’ and ‘2’ ;). In particular, unlikely events must occur sometimes and
> 
> where there is an element of varying behavior involved and specifying  
> exactly what is to be expected can only be done by running multiple  
> tests and making claims about overall properties of the results.
> 
> But it may be the case that people should write their own test code for it yes.
> 
> Still I think this is an important discussion in general since for  
> complex algorithms where it is costly to calc the exact expected  
> results ways to write partial specs are important.

Personally, I would do something like:

def something\_run\_a\_bunch\_of\_times  
results = []  
100\_000.times {results \<\< whatever\_is\_being\_verified}  
results.matches\_statistical\_requirements\_of\_domain?  
end

specify “should be totally awesome” do  
something\_run\_a\_bunch\_of\_times.should.be true  
end

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:52am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/7 "2006-11-07T10:52:24Z")

</div>

> end

Yes, I’ll probably keep my own set of test/spec extensions for now.

lambda {some bool test}.should.be.unlikely

is kind of tempting though… 😉

Thanks for your input,

/Robert

---

<div class="post-metadata">

**Author:** ![capnregex](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/59ef9b/32.png) [@capnregex](https://www.ruby-forum.com/u/capnregex)\
**Post date:** [November 7, 2006, 10:54am UTC](https://www.ruby-forum.com/t/probabilistic-bdd/74638/8 "2006-11-07T10:54:30Z")

</div>

“Robert F.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes:

> > end
> 
> Yes, I’ll probably keep my own set of test/spec extensions for now.
> 
> lambda {some bool test}.should.be.unlikely
> 
> is kind of tempting though… 😉

I would add these methods to the test/spec distribution, but I don’t  
want to add new kinds of contexts, at least not before 1.0.

Just send me a patch.