Should acceptance tests be run against a production environment?


#1

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production. Also, transactional fixtures should be turned
off. This is so that the app runs as closely as possible to how it does
from a regular user’s perspective. Models and pages get cached,
transactions commit and rollback as they’re defined, etc. What do you
think? Am I off base here?

Pat


#2

On Tue, Oct 28, 2008 at 12:08 PM, Pat M. removed_email_address@domain.invalid wrote:

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production. Also, transactional fixtures should be turned
off. This is so that the app runs as closely as possible to how it does
from a regular user’s perspective. Models and pages get cached,
transactions commit and rollback as they’re defined, etc. What do you
think? Am I off base here?

I haven’t been doing this (generally running against the test env),
but I think this sounds right.

FWIW,
David


#3

On Tue, Oct 28, 2008 at 6:08 PM, Pat M. removed_email_address@domain.invalid wrote:

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production. Also, transactional fixtures should be turned
off. This is so that the app runs as closely as possible to how it does
from a regular user’s perspective. Models and pages get cached,
transactions commit and rollback as they’re defined, etc. What do you
think? Am I off base here?

I recommend running against a production-LIKE environment. In Rails
you can create a production_test environment that you make as close to
your production environment as possible, for example by running
against a production_test database that contains a dump of your
production data from last night.

Aslak


#4

aslak hellesoy wrote:

I recommend running against a production-LIKE environment. In Rails
you can create a production_test environment that you make as close to
your production environment as possible, for example by running
against a production_test database that contains a dump of your
production data from last night.

I’m assuming that this is running on your development machine, right?

What about the idea of running it on machines that are as close to
production as possible? So if my production machine is a SliceHost VPS
with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.

Is that just overkill?


#5

Pat M. wrote:

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production. Also, transactional fixtures should be turned
off. This is so that the app runs as closely as possible to how it does
from a regular user’s perspective. Models and pages get cached,
transactions commit and rollback as they’re defined, etc. What do you
think? Am I off base here?

That is exactly what I do Pat. I want to get as close to testing the
real users experience as possible. In the end those real user
experiences are my greatest concern. When you move away from testing a
production rails instances you leave yourself open to ‘Users are
complaining about X not working. Well it worked in testing’ :slight_smile:

It however does mean you have to be very careful to isolate your
acceptance tests from each other due to caching/session/cookie/database
commits.


Joseph W.
http://www.joesniff.co.uk


#6

I agree. I have seen way too many times selenium tests are OK but bugs
appear in production. Not only should we run selenium tests against
production environment, but also they should be run on a production like
environment, such as, same OS, same setting (behind Apache, or whatever
HTTP
servers, etc.)


#7

One thing that bothers me about a ‘staging’ or ‘production_test’
environment
is simply the value of Rails.env. That should really == ‘production’, it
seems to me, but that’s not always practical.

///ark


#8

On Tue, Oct 28, 2008 at 1:08 PM, Pat M. removed_email_address@domain.invalid wrote:

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production.

I believe that’s what the angels call a “staging environment.”

And yes.


Have Fun,
Steve E. (removed_email_address@domain.invalid)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org


#9

On Oct 28, 2008, at 5:08 pm, Pat M. wrote:

When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment. Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production. Also, transactional fixtures should be turned
off. This is so that the app runs as closely as possible to how it
does
from a regular user’s perspective. Models and pages get cached,
transactions commit and rollback as they’re defined, etc. What do you
think? Am I off base here?

Hi Pat

This makes sense to me. I actually tend to run feature files against
the development environment though (well, features, which is a cp of
development). The code reloading saves me from having to stop and
start the server. Transactions and page caching are the big ones to
me - model caching should reduce bugs, surely.

A run against a production makes sense though, especially before
committing.

Ashley


http://www.patchspace.co.uk/


#10

On Wed, Oct 29, 2008 at 4:04 PM, DyingToLearn removed_email_address@domain.invalid wrote:

What about the idea of running it on machines that are as close to
production as possible? So if my production machine is a SliceHost VPS
with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.

Is that just overkill?

It depends on what your risks and worries are. If you have code that
has dependencies on a certain server configuration, database, etc., or
if you’re pushing resource boundaries in any way, then yeah, you
should test it against those constraints as best you can before
sending it to the wild. If what you wrote is just a very vanilla
Rails app with no abstractions altered and you’re not anticipating a
traffic slam, it probably doesn’t matter if you match all resources
precisely. You should at least match basic software (don’t put your
staging environment on Thin with Ruby 1.9 if your production is
Phusion Passenger with Ruby Enterprise) but having the same number of
boxes with the same RAM is not important unless there’s a reason.

That said, I doubt that there are many people who would call a
Slicehost VPS with 256 MB RAM overkill regardless. >8-> If you’re on
the low end like that, what the hell, spend another twenty bucks and
then clone your production slice. You could even delete the staging
slice between test iterations.


Have Fun,
Steve E. (removed_email_address@domain.invalid)
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org