A better benchmarking syntax (was: Automatic Benchmark Itera

Reading Phrogz’ post about automatic benchmark iterations, and then
seeing Mauricios’ lovely Adaptative Benchmark[1], I came to think we
might need a friendlier syntax for benchmarks all together. Minutes
later, I discovered that someone had almost the same idea as me[2].
Anyway, I’d like to just throw it out here, and hear what you people
think.

The idea is to make benchmarks syntactically similar to the current
Test::Unit test cases, e.g.

class SortBenchmark < Benchmark
def setup
@array = [1, 6, 2, 9, 4, 6, 2]
end

def teardown
  @array = nil
end

def report_quicksort
  @array.quicksort
end

def report_mergesort
  @array.mergesort
end

end

Automatic iteration could be added, either as a class method call or by
creating a different base class (IterativeBenchmark?)

So, what do y’all think? If I’m not the only one liking this, I might
whip something up when I get some spare time…

Cheers,
Daniel

  1. eigenclass.org
    +benchmark&key=TDD
  2. BenchUnit? - Testing 1,2,3... — LiveJournal

On Mar 7, 6:26 am, Daniel S. [email protected]
wrote:

class SortBenchmark < Benchmark
end
whip something up when I get some spare time…
+1 - please do. :slight_smile:

Dan

On Mar 7, 8:26 am, Daniel S. [email protected]
wrote:

class SortBenchmark < Benchmark
end
whip something up when I get some spare time…
Cool! Go BDD with it:

benchmark “compare sorting methods” do

compare "quick sort"
  @array.quicksort
end

compare "merge sort" do
  @array.mergesort
end

end

Or something like that.

T.

On 3/7/07, Trans [email protected] wrote:

compare "merge sort" do
  @array.mergesort
end

end

Or something like that.

I’d like to see a way to specify with or without rehearsals, and with or
without
statistics (see

for more info).

Maybe:

benchmark “compare sorting methods” do
with
tsort
rehearsal
end

compare “quick sort”
@array.quicksort
end

compare “merge sort” do
@array.mergesort
end

end

On 3/7/07, Daniel S. [email protected] wrote:

I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system. I’ll
give it some more thought :slight_smile:

Whichever way you go, please do consider configurable (and
extensible) additons like rehearsals and statistics.

Any idea for a name? “benchmark” is unfortunately already taken…

what about BenchBench (in a play on “Benchmarkers’ Tool
Bench”)

On Wed, 2007-03-07 at 23:18 +0900, Trans wrote:

The idea is to make benchmarks syntactically similar to the current

creating a different base class (IterativeBenchmark?)
end

compare "merge sort" do
  @array.mergesort
end

end

Or something like that.

I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system. I’ll
give it some more thought :slight_smile:

Any idea for a name? “benchmark” is unfortunately already taken…

Cheers,
Daniel

On Thu, 2007-03-08 at 01:59 +0900, pat eyler wrote:

Any idea for a name? “benchmark” is unfortunately already taken…

what about BenchBench (in a play on “Benchmarkers’ Tool
Bench”)

I’m not sure… the other guy[1] suggested BenchUnit, which I think
reads well, but I not sure how descriptive it is.

Cheers,
Daniel

. http://djberg96.livejournal.com/52734.html

On Wed, 2007-03-07 at 23:33 +0900, pat eyler wrote:

On Ruby: Benchmarking, Lies, and Statistics
for more info).

I think statistics may be beyond the immediate scope of this – I’d need
the help of someone better at it than me, that’s for sure!

One thing on my priority list is a nice Rake task that makes it easy to
run the benchmarks and set different options.

Daniel

On Mar 7, 9:54 am, Daniel S. [email protected]
wrote:

end

Automatic iteration could be added, either as a class method call or by
@array.quicksort
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system.

I agree. Keep it simple.

Any idea for a name? “benchmark” is unfortunately already taken…

Assuming you keep the test-unit like syntax, I vote for bench-unit.

Regards,

Dan

On 3/7/07, Daniel S. [email protected] wrote:

end

Automatic iteration could be added, either as a class method call or by
@array.quicksort
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now,

Thanks for being a fan. FWIW, the recent API changes are the last
significant ones. What’s new in 0.8 will be the basis of 1.0.

Cheers,
David

On 3/7/07, Daniel S. [email protected] wrote:

Algorithm for generating product names

  1. locate random object on desk
  2. pick random character from alphabet
  3. replace first character of object (from step 1) with the chosen
    character (from step 2)

Example

  1. apricot
  2. ‘h’
  3. hpricot

From my own desk

  1. light bulb
  2. ‘f’
  3. fight_bulb

See, piece of cake!

Blessings,
TwP

On Thu, 2007-03-08 at 02:20 +0900, Tim P. wrote:

See, piece of cake!

  1. ‘laptop’
  2. ‘b’
  3. ‘baptop’ ?!

That’s a weird cake…

Daniel :wink:

On Thu, 2007-03-08 at 02:16 +0900, David C. wrote:

think.
@array = nil

compare "quick sort"

I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now,

Thanks for being a fan. FWIW, the recent API changes are the last
significant ones. What’s new in 0.8 will be the basis of 1.0.

Cheers,
David

Lovely :slight_smile: It’s a great tool!

Daniel

I’ve given this some thought, and it seems like a much bigger project
than I immediately imagined – i was surprised to find a much greater
need of features than anticipated. I am currently pondering whether to
do this as a Summer of Code project (that is, if I’m lucky enough to get
elected.)

Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.

The only thing I have ever used benchmarking for was to compare the
speed of two methods with the same functionality (e.g. sorting).

So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.

Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!

Cheers,
Daniel

On 3/8/07, Daniel S. [email protected] wrote:

I’ve given this some thought, and it seems like a much bigger project
than I immediately imagined – i was surprised to find a much greater
need of features than anticipated. I am currently pondering whether to
do this as a Summer of Code project (that is, if I’m lucky enough to get
elected.)

I think this would make an awesome summer of code project, and would
benefit a lot of other projects downstream. Please do sign up.

Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.

The math isn’t my strongpoint either (on the statistical side), but I’m
happy
to kibbitz (and to try to engage other folks who can help) if you’d
like.

The only thing I have ever used benchmarking for was to compare the
speed of two methods with the same functionality (e.g. sorting).

So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.

well, the JRuby/rubinius/etc folks could sure use a common benchmarking
suite to look at sets of results from different builds (and different
platforms).
I do some revision v revision benchmarking of apps as well.

Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!

I’ll weigh in some more as scope and plans become more clear (and after
I get the other eleventy-six things off my plate).

On Mar 8, 2007, at 12:30 PM, pat eyler wrote:

benefit a lot of other projects downstream. Please do sign up.
I agree and I would be happy to mentor it, so be sure and get it in
the list. :wink:

James Edward G. II

On Fri, 2007-03-09 at 03:30 +0900, pat eyler wrote:

Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.

The math isn’t my strongpoint either (on the statistical side), but I’m happy
to kibbitz (and to try to engage other folks who can help) if you’d like.

I would surely need help of statistics gurus to get it right, so please
do. The project itself might not begin immediately though (I’ve got
exams three weeks from now :x)

I do some revision v revision benchmarking of apps as well.
I thought that was what had the most potential. I might take a look
around to see if there are any existing solutions (there probably are
for Rails) and if I could extract some code into a library. I’ll
probably have to write a lot of things from the ground up, though.

Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!

I’ll weigh in some more as scope and plans become more clear (and after
I get the other eleventy-six things off my plate).

I know the feeling :wink:

Cheers,
Daniel

On Fri, 2007-03-09 at 04:14 +0900, James Edward G. II wrote:

I think this would make an awesome summer of code project, and would
benefit a lot of other projects downstream. Please do sign up.

I agree and I would be happy to mentor it, so be sure and get it in
the list. :wink:

How could I say no now :smiley:

I’ll see how things look for my summer, and then I’ll consider whether
to do this. It looks like it could be a very cool project.

Cheers, and thanks for the support,
Daniel

Daniel S. schrieb:

So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.
At the moment I want to do benchmarking two revisions (in version
control) of a method in one project, so a solution to do such things
would be nice but is probably not easy to handle.

In addition if you test some small piece of code you has to execute this
functionality multiple times (often 1000 times or more) to get
significant results. I would be happy to see the possibility for this
common task in a framework.

regards
Jan

pat eyler wrote:

I do some revision v revision benchmarking of apps as well.

Cheers,
Daniel

I could be persuaded to do a little mentoring in both mathematics and
benchmarking. Just don’t ask me to come up with clever Ruby tricks. I
know a lot more about mathematics and benchmarking. :slight_smile:


M. Edward (Ed) Borasky, FBG, AB, PTA, PGS, MS, MNLP, NST, ACMC(P)
http://borasky-research.blogspot.com/

If God had meant for carrots to be eaten cooked, He would have given
rabbits fire.