I just read David Chelimsky's slides from RailsConf. I was at RailsConf, but was unable to attend the session. The slides are excellent, and I've recently become VERY interested in stories. In particular I'm a bit unhappy with regular view testing (even with the easier notion of it in RSpec). I was wondering what folks think about testing the full stack these days, and any recommendations you have? Specifically, I'm wondering, or contemplating, if I do unit tests for my models, and then I use WebRat plus RailsStory, do I even need to then do functional testing of my controllers and views? I can see that I might want to do some particular view testing that ensured certain elements were on a page or something, but in terms of actually testing the workings of my app/site, it seems model unit tests combined with RailsStory+Webrat will do a pretty darn thorough job. I would then consider going to Selenium to test some of the detailed JavaScript as needed. Thoughts? What am I missing, or what are the downsides of such an approach? -- Christopher Bailey Cobalt Edge LLC http://cobaltedge.com
on 19.06.2008 16:32
on 19.06.2008 17:42
On Thu, Jun 19, 2008 at 9:31 AM, Christopher Bailey
<chris@cobaltedge.com> wrote:
> Thoughts? What am I missing, or what are the downsides of such an approach?
In the project I'm working on, we lean pretty heavily on model/helper
specs and webrat-driven plain text stories.
We still do *some* controller specs, but only in those instances where
we deviate from the standard REST design, or to test authentication
requirements. With a custom it_should_... example generator, that can
be as simple as:
describe FooController do
describe "GET :index" do
it_should_require_admin_access(:get, :index)
end
end
We currently have no view specs at all, though I am beginning to
regret that decision somewhat. Initially we avoided them because the
effort required to mock everything out wasn't jiving well with rapid
prototyping, but I'm considering going back and pushing some things
into view specs using stub_model(). There are a lot of simple
requirements that are a good fit for view specs which we are currently
doing in the user stories, solely because the infrastructure was
already there.
As far as controllers go, I wouldn't say we've seen any downside to
relying on stories instead of isolated specs; I'm sure you've seen how
90% of every controller spec ends up looking the same, and they start
to be a headache to maintain. Maybe the shoulda-esque rspec plugins
out there would help, but when I can easily just exercise the full
stack, I haven't bothered to try. I saw one big *win* of not testing
controllers in isolation just this week, though, when I had to shift a
lot of an application's view paths around; with standard controller
specs, I would've been spending a lot of typing updating the render
expectations, but with stories it wasn't an issue: the right markup
either showed up or didn't.
The primary downside for us has been speed; it takes a few minutes to
run the full story suite. The ability to selectively run just a few
stories, and then a good CI server, alleviates that for the most part.
I always ramble too much so I'll stop here. HTH,
k
on 19.06.2008 18:57
Kyle, thanks much for sharing your experience. You mention the speed and so on. I've read that it is slow. Question: does Autotest work the same way with stories, or have a way to detect what file(s) changed and run the appropriate stories, or because they're integration tests, would it just re-run the entire story suite as it really doesn't have a way to correlate source files to stories? I presume the latter. I guess what I'd wonder, in terms of dealing with the speed issue is: - Can I run a single story via TextMate like I do with specs? That would allow me to concentrate on one particular thing if I needed quick turn around time. - I just always have autotest running, and most of the time, I just let it do its thing, listening for any failures (I use Mac's speech to alert me :) So, the speed isn't so bad as I typically just keep working until I hear a problem, then go fix that. - I am a huge fan of CI servers, and have one setup for every project I control. I do like to know my suite passes before I push my changes up, but the CI server is a great way to ensure everything passes on a clean system. I suppose if the story runner was really taking very long times, that I'd be ok pushing and seeing if the CI server caught a problem. I'd like to hear other's thoughts as well, but your experience seems consistent with my theory of the direction I want to go: unit test models and helpers, drop controller and view tests in favor of Webrat + RailsStory, and then, as needed drop in RSpec view tests for things that can't be as efficiently/effectively tested in stories. Final bit/question (for now ;-)... With Webrat, does it pay attention to whether or not div's and such are hidden/displayed in terms of whether it lets you enter values in them? In this particular project I use a fair bit of disclosure style UI, where some fields are shown after a use takes some other action or presses a button, etc. But they're all there, just not visible. So, wondering if I can simply tell Webrat to put data into them, and not worry about having to effect a button press that does its work via JavaScript. I can move to Selenium to handle that, but that's actually something I'm not wanting to do yet (I've used it in the past, and it's just a ton of overhead, truly slow, etc.). On Thu, Jun 19, 2008 at 8:41 AM, Kyle Hargraves <philodespotos@gmail.com> wrote: >> JavaScript as needed. > > into view specs using stub_model(). There are a lot of simple > controllers in isolation just this week, though, when I had to shift a > > k > _______________________________________________ > rspec-users mailing list > rspec-users@rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users > -- Christopher Bailey Cobalt Edge LLC http://cobaltedge.com
on 19.06.2008 18:59
Just found answer to one of my questions below on being able to run stories in TextMate. The new bundle for stories looks cool: http://github.com/bmabey/rspec-story-tmbundle/tree/master On Thu, Jun 19, 2008 at 9:56 AM, Christopher Bailey <chris@cobaltedge.com> wrote: > - Can I run a single story via TextMate like I do with specs? That > long times, that I'd be ok pushing and seeing if the CI server caught > whether it lets you enter values in them? In this particular project >> On Thu, Jun 19, 2008 at 9:31 AM, Christopher Bailey >>> >> describe FooController do >> requirements that are a good fit for view specs which we are currently >> lot of an application's view paths around; with standard controller >> k > Cobalt Edge LLC > http://cobaltedge.com > -- Christopher Bailey Cobalt Edge LLC http://cobaltedge.com
on 19.06.2008 21:01
Christopher Bailey wrote: > Just found answer to one of my questions below on being able to run > stories in TextMate. The new bundle for stories looks cool: > http://github.com/bmabey/rspec-story-tmbundle/tree/master > > Hey Chris, There is actually a bug for running the stories that accidentally got introduced recently... We'll be pushing the fix out to github soon. We are also adding more flexibility to how you can structure your stories directories. Anyways, look for that as the running is currently broken. -Ben
on 20.06.2008 17:51
On 6/19/08, Christopher Bailey <chris@cobaltedge.com> wrote: > Thoughts? What am I missing, or what are the downsides of such an approach? Having spent the last 3 months working on a project which heavily relied on Story Driven Development (currently 78 story files which encompass 401 scenarios) one thing that I have found is that the build has become quite slow. We made a decision early on that we wouldn't bother with view specs, as they felt redundant with our stories. Looking back now I think that was a mistake. If I were to do it over I would leverage stories for the happy path, as well as to catch expected error conditions, but I would also rely heavily on Controller/View/Helper specs to test all of the branches and abnormal use cases. Basically I think that you should test as much stuff at the unit level as possible, and leverage the integration tools where appropriate, but don't overuse without expecting some decrease in performance. -- Josh Knowles phone: 509-979-1593 email: joshknowles@gmail.com web: http://joshknowles.com
on 20.06.2008 20:10
On Thu, Jun 19, 2008 at 11:56 AM, Christopher Bailey <chris@cobaltedge.com> wrote: > quick turn around time. > - I just always have autotest running, and most of the time, I just > let it do its thing, listening for any failures (I use Mac's speech to > alert me :) So, the speed isn't so bad as I typically just keep > working until I hear a problem, then go fix that. I have not attempted to set up autotest + stories, but it should be possible to set up autotest to watch stories/**/*.story and rerun the corresponding story when *that* changes; like you say, it'd be difficult to map implementation files to their stories. I do miss autotest at times, but up+enter on your console to rerun some focused stories is not bad at all. I use a command-line runner without the maintenance requirements of the *.rb runner files (http://github.com/pd/story), so I can pretty easily select the individual foo.story, a full feature, or any mix I want, then just rerun those when I'm ready. Actually, I save files so often that waiting on the stories to catch up would probably be annoying. > I'd like to hear other's thoughts as well, but your experience seems > consistent with my theory of the direction I want to go: unit test > models and helpers, drop controller and view tests in favor of Webrat > + RailsStory, and then, as needed drop in RSpec view tests for things > that can't be as efficiently/effectively tested in stories. As Josh Knowles also said, dropping the unit level specs has turned out to be a mistake. I wouldn't say it's a BIG mistake, and the main issue is just waiting too damned long before you are comfortable pushing your code (which leads to saying "ah screw it" and then "ah crap it failed on CI" and then ...). But a mistake nonetheless. If I started a fresh project today, I would still probably ignore view specs early on; they're cumbersome and take a lot of tweaking when you're just barely figuring out what it is you want to accomplish. Once your ideas solidify a bit, tho, view specs using stub_model() seem perfectly good these days. For controller specs, I think I'd continue to do exactly what I'm already doing: spec the deviations from the pretty-much-scaffolding REST controllers, and wrap up authentication/authorization specs in a tidy DSL to generate that stuff for me. Thin controllers deserve thin controller specs! =) > overhead, truly slow, etc.). Yes, I take extensive advantage of that. So you can click 'edit' next to something on the screen, which unhides a form, but the form is already on the page; webrat is entirely unaware of the CSS and will just fill it out and submit it. It's a minor headache when there are 10 'edit' links and 10 corresponding forms, cuz you'll need to dig through and be sure to fill out the right fields. Other than that, tho, nothin but sunshine. k