[email protected] wrote:
-a
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dali lama
Google “brick moving contest” didn’t find anything.
On 8/10/06, Eric H. [email protected] wrote:
http://shootout.alioth.debian.org/gp4/benchmark.php?
test=fasta&lang=ruby&id=0Contains the random number generator from the random benchmark:
http://shootout.alioth.debian.org/old/benchmark.php?
test=random&lang=ruby&id=0
I did not see this test named ‘random’ when I looked through the
alioth site earlier today… and I still don’t see any links to it
from that site. (I did find it by following your link, however.)
I notice that your second link contains the word ‘old’. Perhaps this
benchmark has been withdrawn? If so, I submit that your criticism was
true in the past, but no longer is.
On 8/10/06, Isaac G. [email protected] wrote:
Caleb C. wrote:
(Provided we undestand that this benchmark is testing the performance
of an arbitrary algorithm, not the performance that a real program
using random numbers in a given language would experience.)Maybe you could explain a little more - if someone was using one of
those programs to generate Fasta files for program testing and
algorithm development in their lab, would that make it a real program
rather than an arbitrary algorithm? What do you say the difference is
between a real program and an arbitrary algorithm?
My understanding of the goal of the fasta program(s) (aside from being
a benchmark) is to generate a random dna sequence, possibly with some
biases about which symbols are more likely. If I were really using
such a program in my work, and speed in that program was important to
me, (and I was writing it in Ruby,) I would not use the version you
give on your website. I would make use of Ruby’s already existing
random number generator. (Among other things, ruby uses (AFAIK) the
well-regarded Mersenne twister algorithm, whereas you have specified a
linear congruential generator, which is considered inadequate for
hard-core random number applications.)
For benchmarking purposes, what you have already is just fine,
provided you understand the limitations. I understand the need to have
all the languages implement the same algorithm so that you can have an
apple-to-apples comparison. I can also sympathise with your desire to
keep the testing simpler by requiring a specific output. It’s just
that this isn’t the way a serious programmer would actually use random
numbers in Ruby (and probably most other languages).
What you are really testing is not the speed at which ruby generates
random numbers, as they would be used in the real world, but the speed
at which it can execute the linear congruential algorithm, which
ultimately depends on the speed at which it can execute integer math
operations.
Isaac G. wrote:
Did you think others would be surprised?
You seemed to think they were overlooking the “the degree of speed up”
- why would you think that?
Because when people state that performance is their priority and are
told that C will give them the best performance they start to change
their story. I realise that C is harder to program in than any scripting
language but the stated requirement was performance so anything less is,
well, less. Perhaps the problem is that people don’t realise just how
much of a boost C can give you and therefore see the effort involved
being out of proportion to the expected results, this comes down to not
knowing what performance boost you can get.
Caleb C. wrote:
between a real program and an arbitrary algorithm?
hard-core random number applications.)
I don’t think you gave a direct answer to - What do you say the
difference is between a real program and an arbitrary algorithm? - so
please correct what I get wrong.
You’re saying if this was a “real program” it would use whichever
library function the language provides for generating random numbers -
if that’s all there is to the distinction, when there are no
appropriate library functions are “real programs” and “arbitrary
algorithms” the same thing?
at which it can execute the linear congruential algorithm, which
ultimately depends on the speed at which it can execute integer math
operations.
We’re not testing the speed at which Ruby generates random numbers,
period. (Are those 6 word summary descriptions that confusing?)
We’re testing the speed at which it can execute a linear congruential
algorithm, do binary-chop on a tiny array, slice and build strings, and
print strings in FASTA format. (Replace the linear congruential
algorithm by calls to Ruby rand and the time will be reduced to ~400s)
Peter H. wrote:
Isaac G. wrote:
Did you think others would be surprised?
You seemed to think they were overlooking the “the degree of speed up”
- why would you think that?
Because when people state that performance is their priority and are
told that C will give them the best performance they start to change
their story.
I don’t know what you mean by they start to change their story.
I realise that C is harder to program in than any scripting
language but the stated requirement was performance so anything less is,
well, less. Perhaps the problem is that people don’t realise just how
much of a boost C can give you and therefore see the effort involved
being out of proportion to the expected results, this comes down to not
knowing what performance boost you can get.
Make a different assumption - assume that people do realize just how
much faster C is at moving bytes, can you find another explanation?
Look at your own webpage, it talks about comparing apples to apples and
yet Kristof B.'s Ruby program uses a completely different
algorithm. Would you have thought of Kristof’s algorithm if you were
coding in C?
We have to ask 2 questions
- How do I find a fast solution?
- How do I make that solution fast?
In the first “For performance, write it in C” post you talked about
writing in Ruby or Perl or Python and then converting the program to C.
How much effort to you spend re-writing the Ruby or Perl or Python
exploring different algorithms and re-thinking the solution? How much
effort to find a fast solution before trying to make that solution
fast?
On 8/10/06, Caleb C. [email protected] wrote:
from that site. (I did find it by following your link, however.)
I notice that your second link contains the word ‘old’. Perhaps this
benchmark has been withdrawn? If so, I submit that your criticism was
true in the past, but no longer is.
I agree that the presence of “/old/” (vs. “/gc4/”) in the URL probably
indicates that the “random” benchmark has been removed from most
locations. There was an actual link to it, however, from the FASTA
benchmark description page. Specifically, on this page:
http://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=all
At the bottom, under “about the fasta benchmark”, in the second bullet
point. That’s the current page for the FASTA benchmark, so maybe
that page just needs to be updated to remove that constraint?
Jacob F.
Jacob F. wrote:
And the implemenation:
alioth site earlier today… and I still don’t see any links to ithttp://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=all
At the bottom, under “about the fasta benchmark”, in the second bullet
point. That’s the current page for the FASTA benchmark, so maybe
that page just needs to be updated to remove that constraint?Jacob F.
There was and is an actual link to it from the FASTA benchmark
description
http://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=all#about
Jacob, why would we want to remove that constraint?
Jacob F. wrote:
alphabets (using the pseudo-random number generator from the random
test=random&lang=ruby&id=0
indicates that the “random” benchmark has been removed from mostThere was and is an actual link to it from the FASTA benchmark
description
http://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=all#aboutJacob, why would we want to remove that constraint?
Either remove or update to point to a current (not “/old/”) benchmark.
I don’t care either way…Jacob F.
Jacob, why would we want to remove that constraint, and what problem
would be solved by “update to point to a current (not “/old/”)
benchmark” ?
On 8/18/06, Isaac G. [email protected] wrote:
benchmark)
locations. There was an actual link to it, however, from the FASTA
There was and is an actual link to it from the FASTA benchmark
description
http://shootout.alioth.debian.org/gp4/benchmark.php?test=fasta&lang=all#aboutJacob, why would we want to remove that constraint?
Either remove or update to point to a current (not “/old/”) benchmark.
I don’t care either way…
Jacob F.
On 8/21/06, Isaac G. [email protected] wrote:
Jacob, why would we want to remove that constraint, and what problem
would be solved by “update to point to a current (not “/old/”)
benchmark” ?
Note this portion of the prior conversation:
Jacob F. wrote:
I did not see this test named ‘random’ when I looked through the
alioth site earlier today… and I still don’t see any links to it
from that site. (I did find it by following your link, however.)I notice that your second link contains the word ‘old’. Perhaps this
benchmark has been withdrawn? If so, I submit that your criticism was
true in the past, but no longer is.I agree that the presence of “/old/” (vs. “/gc4/”) in the URL probably
indicates that the “random” benchmark has been removed from most
locations.
The problem I see is that the current incarnation of the FASTA
benchmark page makes reference to another benchmark that appears to
have been deprecated. This possible deprecation is manifest in the
presence of the word “old” in the URL combined with the assertion
(which I’ll admit I haven’t verified) that there is no other link to
the benchmark on the site. The reliance of the FASTA benchmark on the
Random benchmark, to me, is not an issue. The reliance of the FASTA
benchmark on a possibly deprecated Random benchmark is the issue.
This issue can be resolved by:
- Removing the reliance
- Updating the FASTA benchmark to reference a non-deprecated Random
benchmark - Clarifying that the Random benchmark is not, in fact, deprecated
- Something else I haven’t considered (I’m not omniscient, and don’t
pretend to dictate what the Shootout must do
[I had assumed previously that the deprecation of the Random benchmark was certain in my previous posts, which is why I didn’t previously mention option 3. I didn’t mention option 4, because I assumed it was implied, but I state it here for completeness.]
Now, between those four options, I don’t care which you do. You know
what the Random benchmark is and whether it’s necessary and/or
deprecated a lot better than I do. I was just mentioning possible
options to resolve the issue that I noticed in the conversation above.
I’m fine with letting you choose which option to pursue.
Jacob F.