Reading Phrogz’ post about automatic benchmark iterations, and then
seeing Mauricios’ lovely Adaptative Benchmark[1], I came to think we
might need a friendlier syntax for benchmarks all together. Minutes
later, I discovered that someone had almost the same idea as me[2].
Anyway, I’d like to just throw it out here, and hear what you people
think.
The idea is to make benchmarks syntactically similar to the current
Test::Unit test cases, e.g.
class SortBenchmark < Benchmark
def setup @array = [1, 6, 2, 9, 4, 6, 2]
end
def teardown
@array = nil
end
def report_quicksort
@array.quicksort
end
def report_mergesort
@array.mergesort
end
end
Automatic iteration could be added, either as a class method call or by
creating a different base class (IterativeBenchmark?)
So, what do y’all think? If I’m not the only one liking this, I might
whip something up when I get some spare time…
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system. I’ll
give it some more thought
Whichever way you go, please do consider configurable (and
extensible) additons like rehearsals and statistics.
Any idea for a name? “benchmark” is unfortunately already taken…
what about BenchBench (in a play on “Benchmarkers’ Tool
Bench”)
The idea is to make benchmarks syntactically similar to the current
creating a different base class (IterativeBenchmark?)
end
compare "merge sort" do
@array.mergesort
end
end
Or something like that.
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system. I’ll
give it some more thought
Any idea for a name? “benchmark” is unfortunately already taken…
Automatic iteration could be added, either as a class method call or by @array.quicksort
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now, but I’m not sure this path is the
best. I did actually think about it at first. The strength of Test::Unit
is its simple syntax: just remember #setup, #teardown, and #test_*. I’d
like to add the same simplicity to any new benchmarking system.
I agree. Keep it simple.
Any idea for a name? “benchmark” is unfortunately already taken…
Assuming you keep the test-unit like syntax, I vote for bench-unit.
Automatic iteration could be added, either as a class method call or by @array.quicksort
I must admit to being a great fan of RSpec, although the pace of API
changes has thrown me off for now,
Thanks for being a fan. FWIW, the recent API changes are the last
significant ones. What’s new in 0.8 will be the basis of 1.0.
I’ve given this some thought, and it seems like a much bigger project
than I immediately imagined – i was surprised to find a much greater
need of features than anticipated. I am currently pondering whether to
do this as a Summer of Code project (that is, if I’m lucky enough to get
elected.)
Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.
The only thing I have ever used benchmarking for was to compare the
speed of two methods with the same functionality (e.g. sorting).
So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.
Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!
I’ve given this some thought, and it seems like a much bigger project
than I immediately imagined – i was surprised to find a much greater
need of features than anticipated. I am currently pondering whether to
do this as a Summer of Code project (that is, if I’m lucky enough to get
elected.)
I think this would make an awesome summer of code project, and would
benefit a lot of other projects downstream. Please do sign up.
Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.
The math isn’t my strongpoint either (on the statistical side), but I’m
happy
to kibbitz (and to try to engage other folks who can help) if you’d
like.
The only thing I have ever used benchmarking for was to compare the
speed of two methods with the same functionality (e.g. sorting).
So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.
well, the JRuby/rubinius/etc folks could sure use a common benchmarking
suite to look at sets of results from different builds (and different
platforms).
I do some revision v revision benchmarking of apps as well.
Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!
I’ll weigh in some more as scope and plans become more clear (and after
I get the other eleventy-six things off my plate).
On Fri, 2007-03-09 at 03:30 +0900, pat eyler wrote:
Unfortunately, I do not have great experience with benchmarking, neither
the underlying math (well, I have some), nor the actual need of the
people who would use an extended framework.
The math isn’t my strongpoint either (on the statistical side), but I’m happy
to kibbitz (and to try to engage other folks who can help) if you’d like.
I would surely need help of statistics gurus to get it right, so please
do. The project itself might not begin immediately though (I’ve got
exams three weeks from now :x)
I do some revision v revision benchmarking of apps as well.
I thought that was what had the most potential. I might take a look
around to see if there are any existing solutions (there probably are
for Rails) and if I could extract some code into a library. I’ll
probably have to write a lot of things from the ground up, though.
Second, if there are others who would be interested in contributing to
any project, given that the need is real, I would be delighted to hear
from them!
I’ll weigh in some more as scope and plans become more clear (and after
I get the other eleventy-six things off my plate).
So what I initially want to ask you guys is this: what else do you use
benchmarking for? Do you benchmark your entire application, and compare
results from different revisions? I’d really like to know.
At the moment I want to do benchmarking two revisions (in version
control) of a method in one project, so a solution to do such things
would be nice but is probably not easy to handle.
In addition if you test some small piece of code you has to execute this
functionality multiple times (often 1000 times or more) to get
significant results. I would be happy to see the possibility for this
common task in a framework.
I do some revision v revision benchmarking of apps as well.
Cheers,
Daniel
I could be persuaded to do a little mentoring in both mathematics and
benchmarking. Just don’t ask me to come up with clever Ruby tricks. I
know a lot more about mathematics and benchmarking.
