Forum: RSpec [Cucumber] Fail a scenario from After or AfterStep

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Luke M. (Guest)
on 2009-05-15 10:19
(Received via mailing list)
I'm trying to make sure that my cucumber selenium suite fails if my
Rails app returns any 500 response codes. I'm working on what the best
way to identify this state, but the question I'd like to put to the
list is this: If I want to check for this error condition in an After
or AfterStep, how can I force a failure in the scenario?

i.e. something like the following non-functional snippet:

AfterStep do |scenario|
   if foo_error_state_detected?
     scenario.fail("A foo failure has been detected.")
   end
end

Ideas?

TIA,
Luke
--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Matt W. (Guest)
on 2009-05-15 11:36
(Received via mailing list)
On 15 May 2009, at 07:01, Luke M. wrote:

>    scenario.fail("A foo failure has been detected.")
>  end
> end
>
> Ideas?

+1 to this - I came up against exactly the same problem yesterday.

Matt W.
http://blog.mattwynne.net
http://www.songkick.com
Aslak H. (Guest)
on 2009-05-15 12:28
(Received via mailing list)
> I'm trying to make sure that my cucumber selenium suite fails if my Rails
> app returns any 500 response codes. I'm working on what the best way to
> identify this state, but the question I'd like to put to the list is this:
> If I want to check for this error condition in an After or AfterStep, how
> can I force a failure in the scenario?
>

To be honest, I don't remember what happened to AfterStep. I think we
had it at some point, but can't see it in the codebase. Did I remove
it? Or was that back in the pre-Cucumber days? Help me remember
here...

> i.e. something like the following non-functional snippet:
>
> AfterStep do |scenario|
>  if foo_error_state_detected?
>    scenario.fail("A foo failure has been detected.")
>  end
> end
>
> Ideas?
>

I'm not so sure I like the idea of AfterStep - smells like a
workaround for something that belongs elsewhere. Can't your have
selenium-rc (or a helper method you create around it) raise those
exceptions?

Given /bla/ do
  # don't remember the API, but you get the idea
  @browser.goto_strict('/bla') # raises on 500 errors
end

Aslak
Ben L. (Guest)
on 2009-05-15 15:01
(Received via mailing list)
On Fri, May 15, 2009 at 9:25 AM, aslak hellesoy
<removed_email_address@domain.invalid>wrote:

>
> Aslak
>
>
I have something similar to this in my fork of webrat. I will push my
stuff
up today at some point.

Ben
Luke M. (Guest)
on 2009-05-15 17:53
(Received via mailing list)
On May 15, 2009, at 4:25 AM, aslak hellesoy wrote:

> To be honest, I don't remember what happened to AfterStep. I think we
> had it at some point, but can't see it in the codebase. Did I remove
> it? Or was that back in the pre-Cucumber days? Help me remember
> here...

I'm not sure. I saw a reference for it on the wiki:
[http://wiki.github.com/aslakhellesoy/cucumber/hooks
] and it seemed to fit this problem, but it's not in the version of
cucumber we're running. I thought perhaps it was in trunk but it
sounds like that's not the case.

> I'm not so sure I like the idea of AfterStep - smells like a
> workaround for something that belongs elsewhere. Can't your have
> selenium-rc (or a helper method you create around it) raise those
> exceptions?
>
> Given /bla/ do
>  # don't remember the API, but you get the idea
>  @browser.goto_strict('/bla') # raises on 500 errors
> end

Yes, I'd prefer to be able to configure selenium-rc to fail on 500s.
The problem I ran into is that selenium client doesn't appear to be
able to access the HTTP response code. In addition, ajax-y HTTP
requests can get kicked off in our app by lots of different things (a
click here, a mouseover there, etc).

In our suite today, of step matchers will usually fail if a 500 is
raised, due to the user-facing impact (e.g. a div is not updated or a
form does not appear), but it usually takes us a few minutes to
realize that there was a 500 error raised (vs. incorrect javascript,
etc.). I am hoping to short-circuit that investigation time and also
catch the cases where a 500 is triggered and we don't have a step
matcher that fails.

The approach is a little smelly to me because it adds a little
translucency to what is mostly black-box-testing, but I think it would
be a net positive, at least for our app.

Cheers,
Luke
--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Aslak H. (Guest)
on 2009-05-15 18:47
(Received via mailing list)
> thought perhaps it was in trunk but it sounds like that's not the case.
>
> short-circuit that investigation time and also catch the cases where a 500
> is triggered and we don't have a step matcher that fails.
>

Can you show me the code you would want to put in AfterStep that would
detect a 500 error?

Aslak
Luke M. (Guest)
on 2009-05-15 21:27
(Received via mailing list)
On May 15, 2009, at 10:16 AM, aslak hellesoy wrote:

> Can you show me the code you would want to put in AfterStep that would
> detect a 500 error?

I've written a simple piece of rack middleware (included below) that
will create a file in the event a 500 is raised. I would clear the
file before each scenario and check for it's existence in the
AfterStep. For example:

Before do
   File.delete('/tmp/rails.error') if File.exist?('/tmp/rails.error')
end

AfterStep do |scenario|
   if File.exist?('/tmp/rails.error')
     scenario.fail("Failed because app raised a 50x error.")
   end
end

Here's the rack piece, for the curious:

class RackErrorTouch
   def initialize(app, options = {})
     @app = app
     @path = options[:path]
   end

   def call(env)
     rack_response = @app.call(env)
     if rack_response.first >= 500
       `touch #{@path}`
     end
     rack_response
   end
end
--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Aslak H. (Guest)
on 2009-05-16 00:31
(Received via mailing list)
> On May 15, 2009, at 10:16 AM, aslak hellesoy wrote:
>
>> Can you show me the code you would want to put in AfterStep that would
>> detect a 500 error?
>
> I've written a simple piece of rack middleware (included below) that will
> create a file in the event a 500 is raised. I would clear the file before
> each scenario and check for it's existence in the AfterStep. For example:
>

Thanks for doing that. I have one more favour to ask: Can you show me
an example of a StepDefinition that would cause the file to be
created?

Then I might be able to suggest an alternative solution with the same
result.

Aslak
Luke M. (Guest)
on 2009-05-16 01:43
(Received via mailing list)
On May 15, 2009, at 3:36 PM, aslak hellesoy wrote:

> Thanks for doing that. I have one more favour to ask: Can you show me
> an example of a StepDefinition that would cause the file to be
> created?

That is a tougher question, because any browser interaction that
causes an HTTP request to the app under test could result in a server
error.

Ajax requests can be triggered by javascript, and in various parts of
our app, they are triggered by clicking on links, submitting forms,
dragging and dropping divs, hovering over a div, waiting for a page
load to complete, as a callback from interacting with a 3rd party web
service, or waiting for a setTimeout to execute.

So one way I could tackle it is to add a step to the end of every
scenario that says "Then the application should not have issued any
responses with 50x status codes", but that would be a) repetitive, and
b) not fail until all steps had run, making it harder to track down
the step in the test where the failure occurred.

One way to look at may be that I'm trying to enforce an invariant.
Under no circumstances in my test suite is it appropriate for my app
to raise a 50x error. In my Rails integration suite, webrat enforces
this particular invariant for me, but can't find a good hook in the
selenium test stack to do this, which is why I'm looking at this route.

Cheers,
Luke

--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Luke M. (Guest)
on 2009-05-20 03:01
(Received via mailing list)
Any further thoughts on this, Aslak? I'd prefer to go with a solution
in line with the future direction of Cucumber if possible.

Cheers,
Luke

On May 15, 2009, at 5:12 PM, Luke M. wrote:

> Ajax requests can be triggered by javascript, and in various parts
>
> --
> Luke M.
> removed_email_address@domain.invalid
> http://www.lukemelia.com/
>
> _______________________________________________
> rspec-users mailing list
> removed_email_address@domain.invalid
> http://rubyforge.org/mailman/listinfo/rspec-users

--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Aslak H. (Guest)
on 2009-05-20 03:20
(Received via mailing list)
> Any further thoughts on this, Aslak? I'd prefer to go with a solution in
> line with the future direction of Cucumber if possible.
>

Sorry, forgot to get back to you Luke.

I'm convinced. We'll add an AfterStep hook. Do you want to give a stab
at a patch? Start by writing a feature - see
features/after_block_exceptions.feature for a good example of the
style we like...

Aslak
Matt W. (Guest)
on 2009-05-20 04:16
(Received via mailing list)
On 20 May 2009, at 00:04, aslak hellesoy wrote:

> style we like...
Can I make a mild plea for After(:step) rather than AfterStep? I think
it looks nicer and it's one less method added to the API...

Matt W.
http://beta.songkick.com
http://blog.mattwynne.net
Luke M. (Guest)
on 2009-05-20 07:02
(Received via mailing list)
On May 19, 2009, at 7:04 PM, aslak hellesoy wrote:

> I'm convinced. We'll add an AfterStep hook. Do you want to give a stab
> at a patch? Start by writing a feature - see
> features/after_block_exceptions.feature for a good example of the
> style we like...

Sounds good. I'll give it a whirl tonight.

Cheers,
Luke

>>>> Thanks for doing that. I have one more favour to ask: Can you
>>> app, they are triggered by clicking on links, submitting forms,
>>> responses with
>>> 50x error. In my Rails integration suite, webrat enforces this
>>> removed_email_address@domain.invalid
>> http://www.lukemelia.com/
>>
>> _______________________________________________
>> rspec-users mailing list
>> removed_email_address@domain.invalid
>> http://rubyforge.org/mailman/listinfo/rspec-users
>>
> _______________________________________________
> rspec-users mailing list
> removed_email_address@domain.invalid
> http://rubyforge.org/mailman/listinfo/rspec-users

--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Ben M. (Guest)
on 2009-05-20 07:14
(Received via mailing list)
Luke M. wrote:
> Any further thoughts on this, Aslak? I'd prefer to go with a solution
> in line with the future direction of Cucumber if possible.
>
> Cheers,
> Luke

Have you seen what Matt added recently?

https://rspec.lighthouseapp.com/projects/16211/tic...

I think this should cover your use case... WDYT?

-Ben
Ben M. (Guest)
on 2009-05-20 07:42
(Received via mailing list)
Ben M. wrote:
>
>
> I think this should cover your use case... WDYT?
>
> -Ben
>

Never mind, I just reread the thread...  :)
Luke M. (Guest)
on 2009-05-20 09:31
(Received via mailing list)
On May 19, 2009, at 10:43 PM, Luke M. wrote:

> On May 19, 2009, at 7:04 PM, aslak hellesoy wrote:
>
>> I'm convinced. We'll add an AfterStep hook. Do you want to give a
>> stab
>> at a patch? Start by writing a feature - see
>> features/after_block_exceptions.feature for a good example of the
>> style we like...
>
> Sounds good. I'll give it a whirl tonight.

Here's my first pass:

http://github.com/weplay/cucumber/commit/39601bc5e...

Not crazy about the way it tracks the current_scenario or the
exception_fails_scenario parameter on Hook#execute_in. Unlike Before
and After hooks, I think it's preferable for AfterStep to fail the
step than to call scenario.fail!

Thoughts?

Cheers,
Luke
--
Luke M.
removed_email_address@domain.invalid
http://www.lukemelia.com/
Aslak H. (Guest)
on 2009-05-20 12:18
(Received via mailing list)
>
> Here's my first pass:
>
> http://github.com/weplay/cucumber/commit/39601bc5e...
>
> Not crazy about the way it tracks the current_scenario or the
> exception_fails_scenario parameter on Hook#execute_in. Unlike Before and
> After hooks, I think it's preferable for AfterStep to fail the step than to
> call scenario.fail!
>

I agree - I'm not crazy about those small hacks you did, but
completely understand why you did it. I'd probably have done the same
myself. Some refactorings are needed in order to make this cleaner.
They are small enough hacks that I'll pull it in :-)

Aslak
Aslak H. (Guest)
on 2009-05-20 12:56
(Received via mailing list)
>> at a patch? Start by writing a feature - see
>> features/after_block_exceptions.feature for a good example of the
>> style we like...
>
> Can I make a mild plea for After(:step) rather than AfterStep? I think it
> looks nicer and it's one less method added to the API...
>

That would be confusing, since After already takes arguments (tags).

Aslak
Matt W. (Guest)
on 2009-05-21 09:24
(Received via mailing list)
On 20 May 2009, at 09:33, aslak hellesoy wrote:

>>> I'm convinced. We'll add an AfterStep hook. Do you want to give a
> That would be confusing, since After already takes arguments (tags).
So why not make the API be like

After(:scenario_tagged => "@emails") { clean_up_emails }
After(:each_scenario) { clean_up_database } #the default
After(:step) { do_stuff_for_luke }

Having less methods makes the protocol more extensible, IMO.

Matt W.
http://blog.mattwynne.net
http://www.songkick.com
Aslak H. (Guest)
on 2009-05-21 16:02
(Received via mailing list)
>>>>
>>
>> That would be confusing, since After already takes arguments (tags).
>
> So why not make the API be like
>
> After(:scenario_tagged => "@emails") { clean_up_emails }
> After(:each_scenario) { clean_up_database } #the default
> After(:step) { do_stuff_for_luke }
>
> Having less methods makes the protocol more extensible, IMO.
>

What you suggest would break backwards compatibility.
I actually think that one method makes the protocol *less* extensible,
since there will always be compatibility issues.

Besides, I actually think Before(*tags), After(*tags) and
AfterStep(*tags) is more readable.
In any case, you don't have to write this a lot, only a couple of
times per project.

So we'll stick with the current API

Aslak
This topic is locked and can not be replied to.