# Automatic Benchmark Iterations

**URL:** <https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921>\
**Category:** Ruby\
**Created:** [March 6, 2007, 10:30pm UTC](https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921 "2007-03-06T22:30:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [March 6, 2007, 10:30pm UTC](https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921/1 "2007-03-06T22:30:35Z")

</div>

When performing a benchmark comparison of several techniques, this  
idiom is very common for me:

N = 1\_000\_000  
Benchmark.bmbm{ |x|  
x.report( ‘foo’ ){  
N.times{  
# foo code  
}  
}  
x.report( ‘bar’ ){  
N.times{  
# bar code  
}  
}  
x.report( ‘jim’ ){  
N.times{  
# jim code  
}  
}  
}

Observation #1: It’s annoying to have to type “N.times{ }” for each  
report. I also want the exact same value, and I always want some  
iterations.

Consequent Desire #1: It’d be nice if I could write the above as:  
Benchmark.bmbm( 1\_000\_000 ){ |x|  
x.report( ‘foo’ ){  
# foo code  
}  
x.report( ‘bar’ ){  
# bar code  
}  
x.report( ‘jim’ ){  
# jim code  
}  
}

Observation #2: I’ll frequently start with a ‘guess’ value of  
N=1\_000\_000, but if it’s taking too long, I’ll start playing with  
values of N until I get something that takes 1-5 seconds for the first  
compared value.

Consequent Desire #2: It’d be cool if I could write the above as:  
Benchmark.autobm{ |x|  
x.report( ‘foo’ ){  
# foo code  
}  
x.report( ‘bar’ ){  
# bar code  
}  
x.report( ‘jim’ ){  
# jim code  
}  
}  
…and it would figure out an appropriate number of iterations for me.  
(One approach would be to time one iteration, figure out an  
appropriate multiple, try benchmarking that number of iterations, and  
adjust the number iterations if the result is outside some desired  
range. Rinse/repeat.)

The Benchmark::Report#item method (aka #report) is implemented as  
this:

```
def item(label = "", *fmt, &blk) # :yield:
  print label.ljust(@width)
  res = Benchmark::measure(&blk)
  print res.format(@fmtstr, *fmt)
  res
end

```

My question: What is the least-overhead way you can come up with to  
wrap the supplied block in an iteration-controlled loop? For example:

def item( label="", \*fmt, &blk )  
…  
res = Benchmark::measure( &lambda{ for i in 1…@iterations;  
blk.call; end } )  
…  
end

---

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [March 6, 2007, 10:53pm UTC](https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921/2 "2007-03-06T22:53:38Z")

</div>

Phrogz wrote:

> ```
> x.report( 'bar' ){
> 
> # bar code
> }
> x.report( 'jim' ){
> # jim code
> }
> 
> ```
> 
> }

What I usually do is something like this:

def foo  
#foo code  
end  
def bar  
#bar code  
end  
def moo  
#moo code  
end

tests = [:foo, :bar, :moo]

Benchmark.bmbm(20){|x|  
tests.each do |meth|  
x.report(meth.to\_s){  
for i in 0…N do  
self.send(meth)  
end  
}  
end  
}

Avoids duplication, and makes it a little easier to check foo, bar and  
moo side-by-side to make sure that they’re actually doing the same  
thing. Also makes it much easier to add extra cases.

---

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [March 6, 2007, 11:15pm UTC](https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921/3 "2007-03-06T23:15:44Z")

</div>

On 06.03.2007 22:27, Phrogz wrote:

> ```
> x.report( 'bar' ){
> 
> # bar code
> 
> }
> 
> ```
> 
> def item( label="", \*fmt, &blk )  
> …  
> res = Benchmark::measure( &lambda{ for i in 1…@iterations;  
> blk.call; end } )  
> …  
> end

This is one option - doesn’t even require changing existing methods…  
Number guessing left as exercise for the reader. 🙂

Kind regards

robert

robert@fussel /cygdrive/c/temp  
$ ruby bmext.rb  
user system total real  
foo 0.000000 0.000000 0.000000 ( 0.000000)  
counter=1000  
Rehearsal -------------------------------------------------------  
foo 0.016000 0.000000 0.016000 ( 0.015000)  
---------------------------------------------- total: 0.016000sec

```
                       user system total real

```

foo 0.000000 0.000000 0.000000 ( 0.000000)  
counter=2000

robert@fussel /cygdrive/c/temp  
$ cat bmext.rb  
require ‘benchmark’

module Benchmark  
BmProxy = Struct.new :parent, :iter do  
def method\_missing(s,\*a,&b)  
parent.send(s,\*a) { iter.times { b[] } }  
end  
end

def self.method\_missing(s,\*a,&b)  
n = a.shift || 1  
m = s.to\_s[1…-1]  
send(m,\*a) {|x| yield BmProxy.new(x,n)}  
end  
end

$i = 0

Benchmark.xbm 1000, 20 do |x|  
x.report ‘foo’ do  
$i += 1  
end  
end

print “counter=”, $i, “\n”

$i = 0

Benchmark.xbmbm 1000, 20 do |x|  
x.report ‘foo’ do  
$i += 1  
end  
end

print “counter=”, $i, “\n”

robert@fussel /cygdrive/c/temp  
$

---

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [March 6, 2007, 11:23pm UTC](https://www.ruby-forum.com/t/automatic-benchmark-iterations/87921/4 "2007-03-06T23:23:45Z")

</div>

On 3/6/07, Phrogz [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> x.report( 'bar' ){
> 
> # bar code
> 
> }
> 
> ```
> 
> }  
> …and it would figure out an appropriate number of iterations for me.  
> (One approach would be to time one iteration, figure out an  
> appropriate multiple, try benchmarking that number of iterations, and  
> adjust the number iterations if the result is outside some desired  
> range. Rinse/repeat.)

Mauricio F. wrote a blog post about just this and created  
AdaptiveBenchmark.

[http://eigenclass.org/hiki.rb?cmd=view&p=adaptative+benchmark&key=TDD](http://eigenclass.org/hiki.rb?cmd=view&p=adaptative+benchmark&key=TDD)

Blessings,  
TwP