Better yet, don’t. The language shootout is only slightly more useful
than “the end of the Internet” site. Maybe. Throwing darts in a
dartboard and picking out results based on that is certainly more
reliable than the language shootout, because of the completely
dishonest way in which the shootout presents itself and selects
algorithmic representations.
The only benchmark worth measuring is your own application.
The OP seems surprised by the order of magnitude differences between
Ruby and C performance - if he looked at regex-dna on the Computer
Language Shootout he might be surprised that the Ruby program is
relatively fast.
And I doubt you’re surprised when Ruby shows up relatively slow; on the
same algorithm. Language implementation technique almost always beats
out the choice of language - YARV.
But it isn’t /your/ lack of surprise that’s at issue, it’s the
apparently genuine surprise of the OP - “It’s the degree of speed up
that I felt that people are overlooking”
Certainly I’ll post in response as long as you’re going to shill
something as worthless as the shootout.
I’m not surprised when Ruby shows up relatively fast; the choice of
algorithm almost always beats out the choice of language.
I’m just surprised it took you so long in the thread to start pushing
the shootout so hard. You’re usually not so restrained before your
shilling starts.
the shootout so hard. You’re usually not so restrained before your
shilling starts.
I have to agree with Austin here. There are benchmarks inside that
specify “this benchmark must use the implementation of the random
number generator from benchmark X”.
Requirements like that make the results even more bogus. Sure, Ruby
has slower method dispatch, but now you’ve got and made the benchmark
unrealistic by skipping over the core libraries of all your
benchmarked applications.
It’s very straightforward - if we use Ruby rand instead of
implementing that random number generator we won’t get the correct
output.
I have to question your testing strategy, then. I have not looked
closely at your benchmarks… but if you have to have a benchmark
which uses random numbers (say, Monte Carlo integration), then you
shouldn’t require an exact result. Can’t you verify that it is close
enough to the expected result?
It’s very straightforward - if we use Ruby rand instead of
implementing that random number generator we won’t get the correct
output.
My impression is you feel arbitrary requirements are unrealistic - I
actually worked with a user who required a specific random number
generator, which unfortunately wasn’t provided by the language
libraries I had available. In my experience arbitrary requirements are
boringly realistic.
(I wonder what we mean by “unrealistic” in this context? Do we mean
“Not like the programs I work on”?)
It’s very straightforward - if we use Ruby rand instead of
implementing that random number generator we won’t get the correct
output.
can you elaborate?
My impression is you feel arbitrary requirements are unrealistic - I
actually worked with a user who required a specific random number
generator, which unfortunately wasn’t provided by the language
libraries I had available. In my experience arbitrary requirements are
boringly realistic.
then why not some ‘number of lines of code’ requirements? like you have
to
write fasta in under 30 lines of code otherwise you have to benchmark on
a
slower cpu since the cost of your time could have added another node to
a
cluster? (kidding - but only a little)
(I wonder what we mean by “unrealistic” in this context? Do we mean
“Not like the programs I work on”?)
it means that if the code one is working on can be asisted by built-in
libraries/features and/or the first few links on google and it’s not
taken
advantage of that one is not a computer scientist but a masochist and a
money
waster - not using ruby’s built-in rand is like having a brick moving
contest:
not that smart.
slower cpu since the cost of your time could have added another node to a
cluster? (kidding - but only a little)
(I wonder what we mean by “unrealistic” in this context? Do we mean
“Not like the programs I work on”?)
it means that if the code one is working on can be asisted by built-in
libraries/features and/or the first few links on google and it’s not taken
advantage of that one is not a computer scientist but a masochist and a money
waster - not using ruby’s built-in rand is like having a brick moving contest:
not that smart.
I guess the user I worked with would have demanded you be replaced by
someone willing to work with the requirements rather than produce a
more convenient wrong answer
It’s very straightforward - if we use Ruby rand instead of
implementing that random number generator we won’t get the correct
output.
I have to question your testing strategy, then. I have not looked
closely at your benchmarks… but if you have to have a benchmark
which uses random numbers (say, Monte Carlo integration), then you
shouldn’t require an exact result. Can’t you verify that it is close
enough to the expected result?
If you’re interested enough to speculate and question, please be
interested enough to click the URL that was provided and find out
something about it before you speculate and question
The OP seems surprised by the order of magnitude differences between
Ruby and C performance - if he looked at regex-dna on the Computer
Language Shootout he might be surprised that the Ruby program is
relatively fast.
Come on. I would hardly be telling people to write something in C if I
thought that the improvement would only be few percentage points. It was
no surprise to me that C was faster than Ruby, neither was it a surprise
as to how much of an improvement it was. I have a habit of converting
things from Perl and Ruby to C for this very reason. Have you really
misread this whole set of threads for the past week?
I guess the user I worked with would have demanded you be replaced by
someone willing to work with the requirements rather than produce a more
convenient wrong answer
you are right of course. at my present job this has been brought up in
the
past by others. however, it’s a powerful argument to assert (and then
follow
through) that you can be 2-3 times more productive using the most
efficient
problem solving approach available. in fact, i’ve gone through two
rounds of
this but, instead of getting laid off, the other programers were -
nothing is
as convincing to manangement and funding sources as finished, working,
and
correct programs produced ahead of schedule. as it stands now our past
group
of three is now just me and we are still making deadlines. so,
sometimes the
wrong answer is later found to be right.
in any case i understand that there are sometimes requirements and that
they
are almost always arbitrary, it’s just that in the case of the test
requiring
a specific random generator the requirement is bordering on arcane since
the
benchmarks claim to test specific programs but this requirement makes
fasta
actually test two and, from a logical standpoint, the concept is flawed
since
requiring ‘specific random numbers’ is an oxymoron. i agree with the
poster
that said the answer should have been tested to be within a tolerance to
neatly sidestep this ultimately unimportant (wrt to the overall shootout
which, unlike some others i find useful) issue.
If you’re interested enough to speculate and question, please be
interested enough to click the URL that was provided and find out
something about it before you speculate and question
All right. I’m looking at it. Based on what I’m seeing on your
website, I’ll make the following statements. Correct me if I’m wrong
about any of these:
Your fasta benchmark is a random number (err… DNA sequence)
generator. I feel this is a perfectly legitimate benchmark, by itself.
(Provided we undestand that this benchmark is testing the performance
of an arbitrary algorithm, not the performance that a real program
using random numbers in a given language would experience.)
2 subsequent benchmarks, k-nucleotide and reverse-complement, use
the data generated by fasta. It appears that both are reading that
data from a canned file, and not calling the fasta random number
generator.
Therefore, Eric’s statement:
I have to agree with Austin here. There are benchmarks inside that
specify “this benchmark must use the implementation of the random
number generator from benchmark X”.
is incorrect. Ruby’s (admittedly poor) performance on the fasta
benchmark should not be further artifically be dragging down its
(admittedly poor) performance on the subsequent 2 benchmarks.
Going back and looking again, I see in your reply to Eric that you
seem to be making this point. This was not appearent to me on first
reading. You could have been a little clearer… on the other hand, I
could have done a little more digging before I posted. Still and all,
it looks like Ara came to the same misunderstanding.
The OP seems surprised by the order of magnitude differences between
Ruby and C performance - if he looked at regex-dna on the Computer
Language Shootout he might be surprised that the Ruby program is
relatively fast.
Come on. I would hardly be telling people to write something in C if I
thought that the improvement would only be few percentage points. It was
no surprise to me that C was faster than Ruby, neither was it a surprise
as to how much of an improvement it was…
You’re right - I was completely wrong to say you were surprised.
Did you think others would be surprised?
You seemed to think they were overlooking the “the degree of speed up”
in any case i understand that there are sometimes requirements and that they
are almost always arbitrary, it’s just that in the case of the test requiring
a specific random generator the requirement is bordering on arcane since the
benchmarks claim to test specific programs but this requirement makes fasta
actually test two
LOL!
I have no interest in quarreling about the definition of “program”, but
following that reasoning perhaps you should have counted the non-random
sequence as a separate program, and the selection of cumulative
frequency as a separate program, and …
and, from a logical standpoint, the concept is flawed since
requiring ‘specific random numbers’ is an oxymoron.
Oh, maybe that’s why people talk about pseudorandom numbers.
i agree with the poster that said the answer should have been tested to be within a
tolerance to neatly sidestep this ultimately unimportant (wrt to the overall shootout
which, unlike some others i find useful) issue.
Have you actually looked at what the fasta programs are doing?!
Every year there seem to be people who miss out parts of the NYC
marathon course in an attempt to improve their race position - I find
it fascinating that people can both understand that a marathon race has
an arbitrary standard distance 26 miles 385 yards along an arbitrary
route, and still try to avoid doing that standard distance.
In contrast, it isn’t surprising that programmers try to miss out parts
of a computation - it’s what we do. The most unnatural thing about
writing a program for the Computer Language Shootout is having to write
a program constrained to an arbitrary distance and arbitrary route -
there’s some latitude but not as much as we’re accustomed to.
I have to agree with Austin here. There are benchmarks inside that
specify “this benchmark must use the implementation of the random
number generator from benchmark X”.
is incorrect. Ruby’s (admittedly poor) performance on the fasta
benchmark should not be further artifically be dragging down its
(admittedly poor) performance on the subsequent 2 benchmarks.
My assertion is correct. The fasta benchmark says:
Each program should:
[…]
generate DNA sequences, by weighted random selection from the
alphabets (using the pseudo-random number generator from the random
benchmark)
All right. I’m looking at it. Based on what I’m seeing on your
website, I’ll make the following statements.
Thank you for taking the trouble.
-snip-
(Provided we undestand that this benchmark is testing the performance
of an arbitrary algorithm, not the performance that a real program
using random numbers in a given language would experience.)
Maybe you could explain a little more - if someone was using one of
those programs to generate Fasta files for program testing and
algorithm development in their lab, would that make it a real program
rather than an arbitrary algorithm? What do you say the difference is
between a real program and an arbitrary algorithm?
(Provided we undestand that this benchmark is testing the performance
of an arbitrary algorithm, not the performance that a real program
using random numbers in a given language would experience.)
Maybe you could explain a little more - if someone was using one of
those programs to generate Fasta files for program testing and
algorithm development in their lab, would that make it a real program
rather than an arbitrary algorithm? What do you say the difference is
between a real program and an arbitrary algorithm?
A “real program” uses the most efficient algorithm neccessary to reach
an
acceptable level of performance. If this level is reached by a built in
library, it’s used, instead of poorly re-invinting the wheel just to use
a
specific “arbitrary algorithm”. Do you write your own quicksort
everytime
you need to sort an array, or do you just call the built in sort method?
In contrast, it isn’t surprising that programmers try to miss out parts of a
computation - it’s what we do. The most unnatural thing about writing a
program for the Computer Language Shootout is having to write a program
constrained to an arbitrary distance and arbitrary route - there’s some
latitude but not as much as we’re accustomed to.
then it is a brick moving contest - the kind i’m happy to lose
