We ran into a situation today where running ‘rake spec’ on one machine
produced different results than when running on a different machine. My
reasoned hunch is that the tests were run in a different order, due to
the
file-globbing producing differently ordered file lists. Because some
tests
used fixtures and some didn’t, that resulted in different test db
contents,
depending on the order the tests were run (clearly, these tests need
fixing).
I haven’t really looked into this at all, for now. But it occurs to me
that
if my suspicion is correct, this situation would be fixed by running
tests
in alphabetical order. Presumably this would be a change to the spec
Rake
task.
Does this make any kind of sense? If so, I (or someone better suited to
the
task) could look into it further. (Coincidentally, a similar situation
resulted in a contribution I made to Visual FoxPro!)
///ark
On Thu, Aug 21, 2008 at 2:25 PM, Mark W. [email protected] wrote:
in alphabetical order. Presumably this would be a change to the spec Rake
task.
Does this make any kind of sense? If so, I (or someone better suited to the
task) could look into it further. (Coincidentally, a similar situation
resulted in a contribution I made to Visual FoxPro!)
My personal feeling is that the code examples should be able to run in
any arbitrary order and still pass (no dependencies between examples),
so I would recommend you find the dependency rather than working
around the problem by enforcing order.
Cheers,
David
On Thu, Aug 21, 2008 at 12:51 PM, David C.
[email protected]wrote:
My personal feeling is that the code examples should be able to run in
any arbitrary order and still pass (no dependencies between examples)
I agree, which is why I said “clearly these tests need fixing.” Maybe a
better suggestion would be to execute the tests in random order each
time…
However, the point was that we didn’t -know- there was an order
dependency,
and neither would the guy whose specs all passed before he went on
vacation.
///ark
On Thu, Aug 21, 2008 at 4:29 PM, Mark W. [email protected] wrote:
On Thu, Aug 21, 2008 at 12:51 PM, David C. [email protected]
wrote:
My personal feeling is that the code examples should be able to run in
any arbitrary order and still pass (no dependencies between examples)
I agree, which is why I said “clearly these tests need fixing.”
Got it.
Maybe a
better suggestion would be to execute the tests in random order each time…
There’s been some discussion of this but it never got completed.
However, the point was that we didn’t -know- there was an order dependency,
and neither would the guy whose specs all passed before he went on vacation.
Any chance the different machines are windows and otherwise? By
default the files are loaded alphabetically, but that means different
things on windows machines (which treat alpha without regard to case)
and pretty much anything else.
Mark W. wrote:
The specs ran fine a Free-BSD machine, but failed on Windows and OS X
machines. The guy who “initiated” the failures was on Windows but
doesn’t tend to run rake spec locally (this was Bruce, Pat).
Maybe I’ll have a look at the random order thing…
I think autotest does this already… It will at least mess with the
order of your specs/tests so that brittle ones are exposed.
-Ben
On Fri, Aug 22, 2008 at 6:18 AM, David C.
[email protected]wrote:
Any chance the different machines are windows and otherwise? By
default the files are loaded alphabetically, but that means different
things on windows machines (which treat alpha without regard to case)
and pretty much anything else.
The specs ran fine a Free-BSD machine, but failed on Windows and OS X
machines. The guy who “initiated” the failures was on Windows but
doesn’t
tend to run rake spec locally (this was Bruce, Pat).
Maybe I’ll have a look at the random order thing…
///ark