Automatic Benchmark Iterations

Ruby
1

When performing a benchmark comparison of several techniques, this
idiom is very common for me:

N = 1_000_000
Benchmark.bmbm{ |x|
x.report( ‘foo’ ){
N.times{
# foo code
}
}
x.report( ‘bar’ ){
N.times{
# bar code
}
}
x.report( ‘jim’ ){
N.times{
# jim code
}
}
}

Observation #1: It’s annoying to have to type “N.times{ }” for each
report. I also want the exact same value, and I always want some
iterations.

Consequent Desire #1: It’d be nice if I could write the above as:
Benchmark.bmbm( 1_000_000 ){ |x|
x.report( ‘foo’ ){
# foo code
}
x.report( ‘bar’ ){
# bar code
}
x.report( ‘jim’ ){
# jim code
}
}

Observation #2: I’ll frequently start with a ‘guess’ value of
N=1_000_000, but if it’s taking too long, I’ll start playing with
values of N until I get something that takes 1-5 seconds for the first
compared value.

Consequent Desire #2: It’d be cool if I could write the above as:
Benchmark.autobm{ |x|
x.report( ‘foo’ ){
# foo code
}
x.report( ‘bar’ ){
# bar code
}
x.report( ‘jim’ ){
# jim code
}
}
…and it would figure out an appropriate number of iterations for me.
(One approach would be to time one iteration, figure out an
appropriate multiple, try benchmarking that number of iterations, and
adjust the number iterations if the result is outside some desired
range. Rinse/repeat.)

The Benchmark::Report#item method (aka #report) is implemented as
this:

def item(label = "", *fmt, &blk) # :yield:
  print label.ljust(@width)
  res = Benchmark::measure(&blk)
  print res.format(@fmtstr, *fmt)
  res
end

My question: What is the least-overhead way you can come up with to
wrap the supplied block in an iteration-controlled loop? For example:

def item( label="", *fmt, &blk )
…
res = Benchmark::measure( &lambda{ for i in 1…@iterations;
blk.call; end } )
…
end

2

Phrogz wrote:

x.report( 'bar' ){

    # bar code
}
x.report( 'jim' ){
    # jim code
}

}

What I usually do is something like this:

def foo
#foo code
end
def bar
#bar code
end
def moo
#moo code
end

tests = [:foo, :bar, :moo]

Benchmark.bmbm(20){|x|
tests.each do |meth|
x.report(meth.to_s){
for i in 0…N do
self.send(meth)
end
}
end
}

Avoids duplication, and makes it a little easier to check foo, bar and
moo side-by-side to make sure that they’re actually doing the same
thing. Also makes it much easier to add extra cases.

3

On 06.03.2007 22:27, Phrogz wrote:

x.report( 'bar' ){

    # bar code

}

def item( label="", *fmt, &blk )
…
res = Benchmark::measure( &lambda{ for i in 1…@iterations;
blk.call; end } )
…
end

This is one option - doesn’t even require changing existing methods…
Number guessing left as exercise for the reader. :slight_smile:

Kind regards

robert

robert@fussel /cygdrive/c/temp
$ ruby bmext.rb
user system total real
foo 0.000000 0.000000 0.000000 ( 0.000000)
counter=1000
Rehearsal -------------------------------------------------------
foo 0.016000 0.000000 0.016000 ( 0.015000)
---------------------------------------------- total: 0.016000sec

                       user     system      total        real

foo 0.000000 0.000000 0.000000 ( 0.000000)
counter=2000

robert@fussel /cygdrive/c/temp
$ cat bmext.rb
require ‘benchmark’

module Benchmark
BmProxy = Struct.new :parent, :iter do
def method_missing(s,*a,&b)
parent.send(s,*a) { iter.times { b[] } }
end
end

def self.method_missing(s,*a,&b)
n = a.shift || 1
m = s.to_s[1…-1]
send(m,*a) {|x| yield BmProxy.new(x,n)}
end
end

$i = 0

Benchmark.xbm 1000, 20 do |x|
x.report ‘foo’ do
$i += 1
end
end

print “counter=”, $i, “\n”

$i = 0

Benchmark.xbmbm 1000, 20 do |x|
x.report ‘foo’ do
$i += 1
end
end

print “counter=”, $i, “\n”

robert@fussel /cygdrive/c/temp
$

4

On 3/6/07, Phrogz [email protected] wrote:

x.report( 'bar' ){

    # bar code

}

}
…and it would figure out an appropriate number of iterations for me.
(One approach would be to time one iteration, figure out an
appropriate multiple, try benchmarking that number of iterations, and
adjust the number iterations if the result is outside some desired
range. Rinse/repeat.)

Mauricio F. wrote a blog post about just this and created
AdaptiveBenchmark.

http://eigenclass.org/hiki.rb?cmd=view&p=adaptative+benchmark&key=TDD

Blessings,
TwP