Benchmark in MiniTest

Ruby
1

def test_hot_is_constant
assert_performance_constant 0.9999 do
@reddit.hot
end
end

def bench_my_algorithm
assert_performance_linear 0.9999 do |n| # n is a range value
n.times do
@reddit.hot
end
end
end

the output i get is:
TestReddit 1 10 100 1000 10000
bench_hot 0.000039 0.000052 0.000032
0.000033 0.000032
bench_hot2 0.000046 0.000047 0.000088
0.000596 0.005650

my question - Is there a way to set the maximum time for a method to
run?
if it goes above it i want it to fail.
what does assert_performance_constant mean? is it a constant speed in
relation to the argument the method accepts?

Thanks

2

I guess this works (ruby 1.9.2)

def test_speed
Timeout.timeout(0.1) {@reddit.hot}
end

but I am still curious about the different benchmark methods minitest
provides.

3

On Feb 6, 2011, at 16:15 , Oren wrote:

 n.times do

my question - Is there a way to set the maximum time for a method to run?
if it goes above it i want it to fail.

No. I’m not sure it makes too much sense. On a slower or more loaded
machine, you’d get false negatives.

what does assert_performance_constant mean? is it a constant speed in relation
to the argument the method accepts?

8511 % ri MiniTest::Unit::TestCase.assert_performance_constant
= MiniTest::Unit::TestCase.assert_performance_constant

(from gem minitest-2.0.2)

assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a
constant rate (eg, linear slope == 0) within a given error threshold.

Fit is calculated by #fit_constant.

Ranges are specified by ::bench_range.

Eg:

def bench_algorithm
assert_performance_constant 0.9999 do |x|
@obj.algorithm
end
end

4

On Feb 6, 2011, at 16:30 , Oren wrote:

I guess this works (ruby 1.9.2)

def test_speed
Timeout.timeout(0.1) {@reddit.hot}
end

but I am still curious about the different benchmark methods minitest provides.

ri is your friend:

8512 % ri MiniTest::Unit::TestCase | grep assert_performance
assert_performance
assert_performance_constant
assert_performance_exponential
assert_performance_linear
assert_performance_power

5

Thanks Ryan!

6

On 7 February 2011 01:15, Oren [email protected] wrote:

n.times do

my question - Is there a way to set the maximum time for a method to run?
if it goes above it i want it to fail.
what does assert_performance_constant mean? is it a constant speed in relation
to the argument the method accepts?

Thanks

Please be aware your are testing if Integer#times is liner in your
second test, not @reddit.hot.
(which does not take any parameter from your code, and is then very
likely to have constant time).

A good example would be:

def fib(n)

my super code to compute the n^th Fibonacci term in linear time

end

assert_performance_linear 0.9999 do |n|
fib(n)
end

The only good usage of Integer#times (except for testing #times
itself) in assert_performance_*, is to increase the time when the
block runs too fast (like all n <= 10 000 runs in less than 0.001 s)
You then want to use a constant to call #times on it (like 1000.times
{ fib(n) }), never n.
You could also use the :bench_range option, probably better.

Regards,
Benoit D.