Ruby Forum Ruby > Automatic Benchmark Iterations

When performing a benchmark comparison of several techniques, this
idiom is very common for me:

  N = 1_000_000
  Benchmark.bmbm{ |x|
    x.report( 'foo' ){
      N.times{
        # foo code
      }
    }
    x.report( 'bar' ){
      N.times{
        # bar code
      }
    }
    x.report( 'jim' ){
      N.times{
        # jim code
      }
    }
  }

Observation #1: It's annoying to have to type "N.times{ }" for each
report. I also want the exact same value, and I always want some
iterations.

Consequent Desire #1: It'd be nice if I could write the above as:
  Benchmark.bmbm( 1_000_000 ){ |x|
    x.report( 'foo' ){
        # foo code
    }
    x.report( 'bar' ){
        # bar code
    }
    x.report( 'jim' ){
        # jim code
    }
  }

Observation #2: I'll frequently start with a 'guess' value of
N=1_000_000, but if it's taking too long, I'll start playing with
values of N until I get something that takes 1-5 seconds for the first
compared value.

Consequent Desire #2: It'd be cool if I could write the above as:
  Benchmark.autobm{ |x|
    x.report( 'foo' ){
        # foo code
    }
    x.report( 'bar' ){
        # bar code
    }
    x.report( 'jim' ){
        # jim code
    }
  }
...and it would figure out an appropriate number of iterations for me.
(One approach would be to time one iteration, figure out an
appropriate multiple, try benchmarking that number of iterations, and
adjust the number iterations if the result is outside some desired
range. Rinse/repeat.)


The Benchmark::Report#item method (aka #report) is implemented as
this:

    def item(label = "", *fmt, &blk) # :yield:
      print label.ljust(@width)
      res = Benchmark::measure(&blk)
      print res.format(@fmtstr, *fmt)
      res
    end

My question: What is the least-overhead way you can come up with to
wrap the supplied block in an iteration-controlled loop? For example:

  def item( label="", *fmt, &blk )
    ...
    res = Benchmark::measure( &lambda{ for i in 1..@iterations;
blk.call; end } )
    ...
  end

Phrogz wrote:
>     x.report( 'bar' ){
> 
>         # bar code
>     }
>     x.report( 'jim' ){
>         # jim code
>     }
>   }

What I usually do is something like this:

def foo
   #foo code
end
def bar
   #bar code
end
def moo
   #moo code
end

tests = [:foo, :bar, :moo]

Benchmark.bmbm(20){|x|
   tests.each do |meth|
     x.report(meth.to_s){
       for i in 0..N do
         self.send(meth)
       end
     }
   end
}

Avoids duplication, and makes it a little easier to check foo, bar and
moo side-by-side to make sure that they're actually doing the same
thing.  Also makes it much easier to add extra cases.

On 06.03.2007 22:27, Phrogz wrote:
>     x.report( 'bar' ){
> 
>         # bar code
> 
>     }
> 
>   def item( label="", *fmt, &blk )
>     ...
>     res = Benchmark::measure( &lambda{ for i in 1..@iterations;
> blk.call; end } )
>     ...
>   end

This is one option - doesn't even require changing existing methods...
  Number guessing left as exercise for the reader. :-)

Kind regards

  robert


robert@fussel /cygdrive/c/temp
$ ruby bmext.rb
                           user     system      total        real
foo                   0.000000   0.000000   0.000000 (  0.000000)
counter=1000
Rehearsal -------------------------------------------------------
foo                   0.016000   0.000000   0.016000 (  0.015000)
---------------------------------------------- total: 0.016000sec

                           user     system      total        real
foo                   0.000000   0.000000   0.000000 (  0.000000)
counter=2000

robert@fussel /cygdrive/c/temp
$ cat bmext.rb
require 'benchmark'

module Benchmark
   BmProxy = Struct.new :parent, :iter do
     def method_missing(s,*a,&b)
       parent.send(s,*a) { iter.times { b[] } }
     end
   end

   def self.method_missing(s,*a,&b)
     n = a.shift || 1
     m = s.to_s[1..-1]
     send(m,*a) {|x| yield BmProxy.new(x,n)}
   end
end

$i = 0

Benchmark.xbm 1000, 20 do |x|
   x.report 'foo' do
     $i += 1
   end
end

print "counter=", $i, "\n"

$i = 0

Benchmark.xbmbm 1000, 20 do |x|
   x.report 'foo' do
     $i += 1
   end
end

print "counter=", $i, "\n"

robert@fussel /cygdrive/c/temp
$

On 3/6/07, Phrogz <gavin@refinery.com> wrote:
>     x.report( 'bar' ){
>
>         # bar code
>
>     }
>   }
> ...and it would figure out an appropriate number of iterations for me.
> (One approach would be to time one iteration, figure out an
> appropriate multiple, try benchmarking that number of iterations, and
> adjust the number iterations if the result is outside some desired
> range. Rinse/repeat.)
>

Mauricio Fernandez wrote a blog post about just this and created
AdaptiveBenchmark.

http://eigenclass.org/hiki.rb?cmd=view&p=adaptative+benchmark&key=TDD

Blessings,
TwP