How can I have the tests for my Rails app to be executed in a random order?

Rails
1

Hi
How can I have the tests for my Rails app to be executed in a random
order? Is there a simple solution using rake?

I already posted my question to stackoverflow (http://

order-with-rake) but the few responses I got were not able to solve
the problem.

Thanks a lot!

Pietro Di Bello

2

On 8 Sep 2009, at 14:01, Marnen Laibow-Koser wrote:

the spec file.

But why do you need this anyway? Each test should start from a clean
slate, so the order should be completely immaterial.

Should, but it is possible for this not to be true. A random order
would be hard to work with though - Run your suite and a test fails,
run it again and it doesn’t any more.

Fred

3

Piero Di Bello wrote:

Hi
How can I have the tests for my Rails app to be executed in a random
order?

At least with RSpec, specs are executed in unpredictable order. I think
it’s the same each time, but it’s not the order in which they appear in
the spec file.

But why do you need this anyway? Each test should start from a clean
slate, so the order should be completely immaterial.

Best,

Marnen Laibow-Koser
http://www.marnen.org
[email protected]

4

On Sep 8, 3:01 pm, Marnen Laibow-Koser <rails-mailing-l…@andreas-
s.net> wrote:

Piero Di Bello wrote:

Hi
How can I have the tests for my Rails app to be executed in a random
order?

At least with RSpec, specs are executed in unpredictable order. I think
it’s the same each time, but it’s not the order in which they appear in
the spec file.

Unfortunately we don’t use Rspec on our project, although I think we
may give it a try

But why do you need this anyway? Each test should start from a clean
slate, so the order should be completely immaterial.

Well, actually, the answer is in your words :slight_smile:
Each test should start from a clean state, but what if it doesn’t?
For us it’s a great value to have a suite of test executed every time
in a random order (better, every single test method in every test
class should be “shuffled”).
Dependency between tests is a really bad anti-pattern, and we want to
know if this happens as soon as possible.

anyway, thanks for your response,
Piero

5

Marnen Laibow-Koser wrote:

Piero Di Bello wrote:

Hi
How can I have the tests for my Rails app to be executed in a random
order?

At least with RSpec, specs are executed in unpredictable order. I think
it’s the same each time, but it’s not the order in which they appear in
the spec file.

But why do you need this anyway? Each test should start from a clean
slate, so the order should be completely immaterial.

First I’ll say I completely agree with this. If tests/specs fail or pass
depending on the order of execution then there must be serious problems
with your test suite.

Now to address the OP. I know of nothing “built-in” any of the testing
frameworks for this. What I suppose you would need to do is write your
own script that searches though your project for all test classes then
executes them in a random order.

This, however, would only randomize the testing classes, each test
within each class would still execute in whatever order the test
framework decided to run them. Which, would mean that you would probably
also have to replace the test runner with your own that could randomize
each method of each test class.

In conclusion if this is a solid requirement of your project, I’m really
glad it’s your project and not mine. :slight_smile:

Good luck.

6

On Sep 8, 3:44 pm, Frederick C. [email protected]
wrote:

think
it’s the same each time, but it’s not the order in which they appear
in
the spec file.

But why do you need this anyway? Each test should start from a clean
slate, so the order should be completely immaterial.

Should, but it is possible for this not to be true. A random order
would be hard to work with though - Run your suite and a test fails,
run it again and it doesn’t any more.

The whole point of randomizing test execution is to have a better
chance to discover that the tests are not independent. Once you find
a test that fails because of the order of execution, you fix it so
that it does not happen anymore.

Matteo

7

On Sep 9, 7:18 am, Matteo V. [email protected] wrote:

On Sep 8, 3:44 pm, Frederick C. [email protected]
wrote:

Should, but it is possible for this not to be true. A random order
would be hard to work with though - Run your suite and a test fails,
run it again and it doesn’t any more.

The whole point of randomizing test execution is to have a better
chance to discover that the tests are not independent. Once you find
a test that fails because of the order of execution, you fix it so
that it does not happen anymore.

I get that, but unless you could say ‘run the tests again but in the
same random order as before’ how are you going to be sure that you’ve
actually fixed your code/tests ?

Fred

8

On Sep 8, 9:27 pm, Robert W. [email protected]
wrote:

But why do you need this anyway? Each test should start from a clean

This, however, would only randomize the testing classes, each test
within each class would still execute in whatever order the test
framework decided to run them. Which, would mean that you would probably
also have to replace the test runner with your own that could randomize
each method of each test class.

In conclusion if this is a solid requirement of your project, I’m really
glad it’s your project and not mine. :slight_smile:

Thanks for your ideas. I’ll definitely explore some of them.
No, it’s not a solid requirement, but as I already said in a previous
answer, we find it very useful to discover dependencies between
tests as soon as possible (and probably the Rspec authors’ find this
important too, given that Rspec execute specs in a random order…).

Thanks
Piero