Running cucumber distributed across hardware

Testing RSpec
1

At my job, our story suite contains over 500 scenarios. While we’re
very happy with the regressions we’ve caught by running the suite, the
long time it takes (~ 10+ minutes) is starting to lead to developers
checking in without running it. This is leading to broken builds, and
the problems cascade from there (not being able to integrate, etc.)

What I’m wondering is if anyone has had success running Cucumber in a
parallel/distributed mode, like I know some people have done with
RSpec proper. I’m optimistic that if we can get the build time scaling
linearly with hardware, we can get the story run down to around one or
two minutes and keep it there regardless of how many scenarios we
introduce.

Thoughts?

Cheers,

-Bryan

2

On Mon, Sep 22, 2008 at 5:11 PM, Bryan H. [email protected]
wrote:

At my job, our story suite contains over 500 scenarios. While we’re
very happy with the regressions we’ve caught by running the suite, the
long time it takes (~ 10+ minutes) is starting to lead to developers
checking in without running it. This is leading to broken builds, and
the problems cascade from there (not being able to integrate, etc.)

I’m curious: could your unit tests be improved to minimize these
problems?
In other words, when your build breaks, does that indicate a missing
unit
test?

///ark

3

On Mon, Sep 22, 2008 at 8:32 PM, Mark W. [email protected] wrote:

I’m curious: could your unit tests be improved to minimize these problems?
In other words, when your build breaks, does that indicate a missing unit
test?

That’s a very good point. Sometimes, yes. If the stories fail because
an object or layer of the stack is incorrect, that’s definitely a unit
test that is lacking which needs to be added. I think at those points
it’s also worth examining how the code became incorrect without a
failing unit test to identify root causes.

Often, however, the Story suite exposes an integration failure that
(isolated/fast) unit tests would not catch.

4

On Mon, 22 Sep 2008 21:57:20 -0400, “Scott T.”
[email protected] said:

On Sep 22, 2008, at 8:11 PM, Bryan H. wrote:

At my job, our story suite contains over 500 scenarios. While we’re
very happy with the regressions we’ve caught by running the suite, the
long time it takes (~ 10+ minutes) is starting to lead to developers
checking in without running it. This is leading to broken builds, and
the problems cascade from there (not being able to integrate, etc.)

I’m obviously not as far down the path as ou guys in thinking about
this, but I can already see us having similar issues. Once idea I had
was just to create separate rails environments for features and specs,
thereby letting me run features at the same time as autotest is running.

I also wonder what we can do with in-memory databases, since our
integration tests don’t (so far) work on the scale of more than a
handful of records.

Anyone had any luck with either of these tricks?

cheers,
Matt

5

On Sep 22, 2008, at 8:11 PM, Bryan H. wrote:

At my job, our story suite contains over 500 scenarios. While we’re
very happy with the regressions we’ve caught by running the suite, the
long time it takes (~ 10+ minutes) is starting to lead to developers
checking in without running it. This is leading to broken builds, and
the problems cascade from there (not being able to integrate, etc.)

Hey Bryan,

You know my thoughts on this -I think this problem will eventually be
solved by Guillotine (or something like it). Hopefully I should be
getting a beta-version running up soon. I’ll let you and the list
know when I do. I’ll also get some benchmarks running against FP or
some other large test suite.

What I’m wondering is if anyone has had success running Cucumber in a
parallel/distributed mode, like I know some people have done with
RSpec proper. I’m optimistic that if we can get the build time scaling

Regarding drb, as you know, there was a project named spec_distributed
a while back which attempted it. It’s major failing was that it could
only be run after check in. If you want something developers can run
before check-in, you’ll not only need to distribute examples
remotely, but also the code as well (I’m sure that’s why
spec_distributed made the decision to run after check-in).

There was also a move to get Deeptest to work multi-core, and I’m not
sure if that ever got done - it was buggy the last time I tried it.
That might be a short temporarily solution, but I don’t think it will
scale up.

Seriously, though, no one has done this well yet, even for rspec proper.

linearly with hardware, we can get the story run down to around one or
two minutes and keep it there regardless of how many scenarios we
introduce.

AFAIK, There doesn’t seem to be an a priori reason why distributing
workload via rinda wouldn’t work (you could, for example, distribute
feature files individually).

Let me know if you come up with something - I’d be interested in
contributing to such a project.

Scott

6

On Wed, Sep 24, 2008 at 7:31 AM, Bryan H. [email protected]
wrote:

On Mon, Sep 22, 2008 at 9:57 PM, Scott T.
[email protected] wrote:

Let me know if you come up with something - I’d be interested in
contributing to such a project.

I started working on a project I’m calling Testjour to solve this at
GitHub - brynary/testjour: Distributed test running with autodiscovery via Bonjour (for Cucumber first) .

That looks promising. You may also want to look at the now defunct
rspec-distributed, where we have some fancy Rinda mojo. Bob Cotton and
I did a lot on that 1-2 years back…

Aslak

7

On Mon, Sep 22, 2008 at 9:57 PM, Scott T.
[email protected] wrote:

Let me know if you come up with something - I’d be interested in
contributing to such a project.

I started working on a project I’m calling Testjour to solve this at
GitHub - brynary/testjour: Distributed test running with autodiscovery via Bonjour (for Cucumber first) .

It’s not at all usable for general consumption, but there’s a base to
hack on. Here’s what I have working:

  • Using Bonjour autodiscovery of available test running slaves
  • Sending feature files to be run and results back and forth over DRb
  • Setting up and tearing down new MySQL databases for each test run to
    avoid conflicts

There’s plenty that needs to be done before it’s really usable,
specifically:

  • Adding RSync support to transfer code to worker servers
  • Daemonizing/detaching the slave process
  • Adding logging and PID tracking
  • And more in the TODO.txt

If I can get this running well, we’re going to purchase Mac Mini build
nodes for the office. My (ambitious) goal is to cut the story run time
on a developer machine from 12 minutes down to two.

Example from two Terminal windows:

$ testjour_slave
Broadcasting: Ready to run tests under name ‘bhelmkamp’ on port 51814…

Running tests from queue druby://Fozzie.local:51818 on PID 16482

$ testjour_run features/authentication/authenticaton.feature
…F_…

BOOM
/Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in When "I login"' /Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in When I login’

8

On Sep 24, 2008, at 1:31 AM, Bryan H. wrote:

  • Using Bonjour autodiscovery of available test running slaves
  • Sending feature files to be run and results back and forth over DRb
  • Setting up and tearing down new MySQL databases for each test run to
    avoid conflicts

Sweet! These are the bare-bones already.

There’s plenty that needs to be done before it’s really usable,
specifically:

  • Adding RSync support to transfer code to worker servers
  • Daemonizing/detaching the slave process
  • Adding logging and PID tracking
  • And more in the TODO.txt
    \

The hard part is always dealing with concurrency - it’d be nice if you
could you other computers in the office which currently running specs
to utilize their spare processes. And it would be nice to get
reporting - if a slave goes down, can you redistribute the steps to a
new slave dynamically?

Good to hear about this. Will look into it today.

Scott

9

Bryan H. wrote:

On Mon, Sep 22, 2008 at 9:57 PM, Scott T.
[email protected] wrote:

Let me know if you come up with something - I’d be interested in
contributing to such a project.

I started working on a project I’m calling Testjour to solve this at
GitHub - brynary/testjour: Distributed test running with autodiscovery via Bonjour (for Cucumber first) .

It’s not at all usable for general consumption, but there’s a base to
hack on. Here’s what I have working:

  • Using Bonjour autodiscovery of available test running slaves
  • Sending feature files to be run and results back and forth over DRb
  • Setting up and tearing down new MySQL databases for each test run to
    avoid conflicts

There’s plenty that needs to be done before it’s really usable,
specifically:

  • Adding RSync support to transfer code to worker servers
  • Daemonizing/detaching the slave process
  • Adding logging and PID tracking
  • And more in the TODO.txt

If I can get this running well, we’re going to purchase Mac Mini build
nodes for the office. My (ambitious) goal is to cut the story run time
on a developer machine from 12 minutes down to two.

Example from two Terminal windows:

$ testjour_slave
Broadcasting: Ready to run tests under name ‘bhelmkamp’ on port 51814…

Running tests from queue druby://Fozzie.local:51818 on PID 16482

$ testjour_run features/authentication/authenticaton.feature
…F_…

BOOM
/Users/bhelmkamp/p/weplay/features/steps/authentication.rb:10:in When "I login"' /Users/bhelmkamp/p/weplay/features/authentication/authenticaton.feature:13:in When I login’

I’m resurrecting this thread. This is becoming a major issue for us, and
we have some allocated time coming into play soon to work on a solution.

What’s the current state of the art for distributed test runs? Is
anybody doing it for real? As part of the build or before check-in?
Where would be the best place to contribute effort? Testjour looks the
most likely… Any others out there?

cheers,
Matt

10

Matt W. wrote:

What’s the current state of the art for distributed test runs? Is
anybody doing it for real? As part of the build or before check-in?
Where would be the best place to contribute effort? Testjour looks the
most likely…

If you are thinking Selenium then there is the Selenium grid direction.

Any others out there?

cheers,
Matt

–
Joseph W.

11

On 2 Apr 2009, at 21:21, Joseph W. wrote:

If you are thinking Selenium then there is the Selenium grid
direction.

Thanks. Right now we’re just running against the rails integration
session using Webrat, but that does definitely look useful for when we
get more Ajax heavy.

Matt W.

http://blog.mattwynne.net

12

I’m thinking about trying to get TestJour in place to run on a single
system, but one with a crapload of processors, since that’s about the
only
way to make anything run reasonably on a Sparc …

I think I remember seeing that there was a Bonjour protocol client for
Solaris, but I’m not sure where. Of course, if there was some way I
could
specify a number of processes for cucumber to spawn, that would work
just as
well for my particular case …

Desparately need to do something, because right now, our Cucumber
tests
take about 30 minutes to run … Ruby on Solaris/Sparc is painfully
slow,
but that’s the architecture I’m stuck on …

13

On Thu, Apr 2, 2009 at 4:14 PM, Matt W. [email protected] wrote:

What’s the current state of the art for distributed test runs? Is
anybody doing it for real? As part of the build or before check-in?
Where would be the best place to contribute effort? Testjour looks the
most likely… Any others out there?

We (weplay) use Testjour on a daily basis. That said, its a bit
temperamental and requires a bit of effort to get it up and running.
I’d love to see more adoption, and am happy to work with whomever is
interested on making it easier to get up and running.

–
Josh K.
phone: 509-979-1593
email: [email protected]
web: http://joshknowles.com