Forum: RSpec Should acceptance tests be run against a production environment?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
42172acdf3c6046f84d644cb0b94642c?d=identicon&s=25 Pat Maddox (pergesu)
on 2008-10-29 04:51
(Received via mailing list)
When you do end-to-end acceptance testing with Selenium, I think it
should be run against a production environment.  Not THE production
environment, mind you, but simply a new Rails app running with
RAILS_ENV=production.  Also, transactional fixtures should be turned
off.  This is so that the app runs as closely as possible to how it does
from a regular user's perspective.  Models and pages get cached,
transactions commit and rollback as they're defined, etc.  What do you
think?  Am I off base here?

Pat
5d38ab152e1e3e219512a9859fcd93af?d=identicon&s=25 David Chelimsky (Guest)
on 2008-10-29 10:24
(Received via mailing list)
On Tue, Oct 28, 2008 at 12:08 PM, Pat Maddox <pergesu@gmail.com> wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user's perspective.  Models and pages get cached,
> transactions commit and rollback as they're defined, etc.  What do you
> think?  Am I off base here?

I haven't been doing this (generally running against the test env),
but I think this sounds right.

FWIW,
David
0be0e4aa42aacd9a8a95c792de273ca7?d=identicon&s=25 aslak hellesoy (Guest)
on 2008-10-29 10:55
(Received via mailing list)
On Tue, Oct 28, 2008 at 6:08 PM, Pat Maddox <pergesu@gmail.com> wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user's perspective.  Models and pages get cached,
> transactions commit and rollback as they're defined, etc.  What do you
> think?  Am I off base here?
>

I recommend running against a production-LIKE environment. In Rails
you can create a production_test environment that you make as close to
your production environment as possible, for example by running
against a production_test database that contains a dump of your
production data from last night.

Aslak
F68f69615423aa3851bd445409754dbf?d=identicon&s=25 Joseph Wilk (joesniff)
on 2008-10-30 02:27
(Received via mailing list)
Pat Maddox wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it does
> from a regular user's perspective.  Models and pages get cached,
> transactions commit and rollback as they're defined, etc.  What do you
> think?  Am I off base here?
>
That is exactly what I do Pat. I want to get as close to testing the
real users experience as possible. In the end those real user
experiences are my greatest concern. When you move away from testing a
production rails instances you leave yourself open to 'Users are
complaining about X not working. Well it worked in testing' :)

It however does mean you have to be very careful to isolate your
acceptance tests from each other due to caching/session/cookie/database
commits.

--
Joseph Wilk
http://www.joesniff.co.uk
A45f650cce5746dd89aafb3176b47b02?d=identicon&s=25 DyingToLearn (Guest)
on 2008-10-30 04:33
(Received via mailing list)
aslak hellesoy wrote:
> I recommend running against a production-LIKE environment. In Rails
> you can create a production_test environment that you make as close to
> your production environment as possible, for example by running
> against a production_test database that contains a dump of your
> production data from last night.


I'm assuming that this is running on your development machine, right?

What about the idea of running it on machines that are as close to
production as possible? So if my production machine is a SliceHost VPS
with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.

Is that just overkill?
48641c4be1fbe167929fb16c9fd94990?d=identicon&s=25 Mark Wilden (Guest)
on 2008-10-30 04:34
(Received via mailing list)
One thing that bothers me about a 'staging' or 'production_test'
environment
is simply the value of Rails.env. That should really == 'production', it
seems to me, but that's not always practical.

///ark
874be46e8593deadb2cec84b70b26725?d=identicon&s=25 Yi Wen (hayafirst)
on 2008-10-30 04:36
(Received via mailing list)
I agree. I have seen way too many times selenium tests are OK but bugs
appear in production. Not only should we run selenium tests against
production environment, but also they should be run on a production like
environment, such as, same OS, same setting (behind Apache, or whatever
HTTP
servers, etc.)
2ce9c0106b5851b2294ba5eb9f5c04bd?d=identicon&s=25 Ashley Moran (Guest)
on 2008-10-30 04:37
(Received via mailing list)
On Oct 28, 2008, at 5:08 pm, Pat Maddox wrote:

> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.  Also, transactional fixtures should be turned
> off.  This is so that the app runs as closely as possible to how it
> does
> from a regular user's perspective.  Models and pages get cached,
> transactions commit and rollback as they're defined, etc.  What do you
> think?  Am I off base here?

Hi Pat

This makes sense to me.  I actually tend to run feature files against
the development environment though (well, features, which is a cp of
development).  The code reloading saves me from having to stop and
start the server.  Transactions and page caching are the big ones to
me - model caching should *reduce* bugs, surely.

A run against a production makes sense though, especially before
committing.

Ashley

--
http://www.patchspace.co.uk/
http://aviewfromafar.net/
994e42bda994be2cd1d791f18ee6d561?d=identicon&s=25 Stephen Eley (Guest)
on 2008-10-30 04:39
(Received via mailing list)
On Tue, Oct 28, 2008 at 1:08 PM, Pat Maddox <pergesu@gmail.com> wrote:
> When you do end-to-end acceptance testing with Selenium, I think it
> should be run against a production environment.  Not THE production
> environment, mind you, but simply a new Rails app running with
> RAILS_ENV=production.

I believe that's what the angels call a "staging environment."

And yes.

--
Have Fun,
   Steve Eley (sfeley@gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org
994e42bda994be2cd1d791f18ee6d561?d=identicon&s=25 Stephen Eley (Guest)
on 2008-10-30 05:18
(Received via mailing list)
On Wed, Oct 29, 2008 at 4:04 PM, DyingToLearn <phylae@gmail.com> wrote:
>
> What about the idea of running it on machines that are as close to
> production as possible? So if my production machine is a SliceHost VPS
> with 256MB RAM, Nginx, and 3 Mongrels, then I should be running these
> tests on a second SliceHost VPS with 256MB RAM, Nginx, and 3 Mongrels.
>
> Is that just overkill?

It depends on what your risks and worries are.  If you have code that
has dependencies on a certain server configuration, database, etc., or
if you're pushing resource boundaries in any way, then yeah, you
should test it against those constraints as best you can before
sending it to the wild.  If what you wrote is just a very vanilla
Rails app with no abstractions altered and you're not anticipating a
traffic slam, it probably doesn't matter if you match all resources
precisely.  You should at least match basic software (don't put your
staging environment on Thin with Ruby 1.9 if your production is
Phusion Passenger with Ruby Enterprise) but having the same number of
boxes with the same RAM is not important unless there's a reason.

That said, I doubt that there are many people who would call a
Slicehost VPS with 256 MB RAM overkill regardless.  >8->  If you're on
the low end like that, what the hell, spend another twenty bucks and
then clone your production slice.  You could even delete the staging
slice between test iterations.


--
Have Fun,
   Steve Eley (sfeley@gmail.com)
   ESCAPE POD - The Science Fiction Podcast Magazine
   http://www.escapepod.org
This topic is locked and can not be replied to.