Hi all, Just wanted to get some feedback and opinions on something... I have been using Cucumber/Story Runner since plaintext support was introduced. (So, since October/November of 2007.) Throughout this time I have used it on various projects. Some projects had customer involvement in help writing the scenarios, some I mostly wrote the scenarios after gathering requirements and then got verification from the customer, and others involved no customer at all (i.e. I was the customer). In all three cases I have gotten a lot of benefit from the upfront analysis that writing the feature forces you to do and the outside-in development that follows. That said, the business analysis and outside-in development can be accomplished with any tool (i.e. rspec with webrat for webapps). In the past I have always defaulted on using the GWT (Given, When, Then) syntax for most of my acceptance tests, but that may be because I am so accustomed to that workflow by now. So... I'm curious what people's thoughts are. When writing acceptance tests how do you choose which tool is best for the job? I often hear people complaining about the GWT syntax and they see no benefit of it over the frameworks that provide contexts (i.e rspec, shoulda, etc). In what cases, if any, do you think the GWT gets in the way and how? In particular, if you are writing an app or lib whose customers/users are all developers do you still think there is value in the GWT syntax that Cucumber provides? I have my own thoughts on the matter, but in playing devil's advocate I want to get other people's thoughts before I skew you with mine. :) Thanks, Ben  http://www.dcmanges.com/blog/practicality-of-the-r...
on 2009-01-23 18:14
on 2009-01-23 21:46
Ben Mabey wrote: > > So... I'm curious what people's thoughts are. When writing acceptance > tests how do you choose which tool is best for the job? I often hear > people complaining about the GWT syntax and they see no benefit of it > over the frameworks that provide contexts (i.e rspec, shoulda, etc). > In what cases, if any, do you think the GWT gets in the way and how? In > particular, if you are writing an app or lib whose customers/users are > all developers do you still think there is value in the GWT syntax that > Cucumber provides? > My own experience is quite limited, but I did start out trying to get what I desired done through TestUnit, then RSpec, then RSpec Story Runner and finally Cucumber. In the end it was Cucumber that clarified my approach for me, mainly by clearing away all (or at least most) of the nitty gritty details of configuring and using the tool itself. I found that, coming from a non OO programming background, that the degree of mastery required of the other tools mentioned, together with the effort I had to make to get the tool to behave as I wished (desires that were often malformed by lack of understanding in any case), distracted considerably from the task at hand, programming the application. I have progressed on my current project far, far more since discovering Cucumber this past November than in all the time before. The bifurcation between what and how seems far more natural to me with the Cucumber features/step definitions file split than with the unified expectation method/test steps of RSpec and the test method/assert statements of TestUnit. Cucumber feature files just seem to permit one to concentrate solely on "what" in the one case and then, with the separate step definitions file, deal with the "how" in the other, without crossing the line between the two. Given, When and Then seemed contrived and far too stiff to me to begin with. However, I discovered that much of my initial discomfort arose from too vague a conception of what is was that was desired. The benefit of GWT stylistic conventions is that it really does make one consider deeply where one is starting from, what one is actually trying to accomplish, and how, exactly, does one tell if it has been accomplished. If this sort of self-analysis is so ingrained that one need not think of it then GWT probably does not provide a measurable benefit. So, perhaps deeply experienced developers and programmers might not get as much from it as I have. On the other hand, I have frequently discovered in myself a lamentable tendency to believe that I understood what a problem was simply because I wanted to believe and not because of any demonstrable evidence to that effect. Sitting down and writing out what you believe to be the truth often reveals the flaws in ones assumptions. The GWT litany seems to provide a very useful, if formulaic, examination of exactly what one is setting out to accomplish thereby sometimes raising the question of why it needs to be done at all. I find that it thereby clears away much that is irrelevant to the question at hand. I also hold to the opinion that when writing an end user application the only testing that can be consider definitive is what is described as integration or acceptance testing. I accept that there are pragmatic reasons why, because of limited time and equipment resources, unit and functional tests are viewed as attractive alternatives. But, that should not blind one the the fact that they really are alternatives, chosen because of environmental considerations and not because of inherent superiority. The only test that means anything to an end user is one that shows the application works as intended in the environment it is deployed to. Libraries and utilities have a different audience and their testing is perhaps better handled solely in unit and functional tests, but I cannot say so with any conviction.
on 2009-01-23 23:24
> My own experience is quite limited, [...] Hi James, are you saying that you only use Cucumber and not RSpec at all?
on 2009-01-23 23:45
Fernando Perez wrote: >> My own experience is quite limited, [...] > > Hi James, are you saying that you only use Cucumber and not RSpec at > all? No, I do use RSpec and TestUnit for that matter, but not to any extent at the present time. I find that cucumber features are much more conducive to driving out the design than RSpec expectations. Once the essence of the application is laid out and working then, if necessary, I will consider adding more technical type expectations via RSpec. Frankly, at the moment, I do not see a compelling case to move from step definitions into formal RSpec expectations. Again, my experience in these matters is limited and perhaps as it grows my appreciations will change as well. I do note however, that once one finds a tool that suits ones needs then it is somewhat inefficient to use another without some compelling advantage to be gained. What advantage does formal RSpec constructions provide over cucumber step definitions using RSpec and RSpec-Rails methods?
on 2009-01-24 00:09
On 23/01/2009, at 12:04 PM, Ben Mabey wrote: > So... I'm curious what people's thoughts are. When writing > acceptance tests how do you choose which tool is best for the job? > I often hear people complaining about the GWT syntax and they see no > benefit of it over the frameworks that provide contexts (i.e rspec, > shoulda, etc). In what cases, if any, do you think the GWT gets > in the way and how? In particular, if you are writing an app or lib > whose customers/users are all developers do you still think there is > value in the GWT syntax that Cucumber provides? I've been using RSpec since July, and love it. For the last few months, I've been reading most of the Cucumber posts on this list. Last week, I thought I'd give Cucumber a try. It feels like a really fantastic tool for determining business requirements, functionality, etc. From what I can tell though, there seems to be a lot of overlap between RSpec specs, and Cucumber features and stories. For that reason, I haven't delved much further into Cucumber, because I don't want to be repeating myself, and having to update two sets of files whenever I make a change. However, if that's not the case, please correct me! Nick
on 2009-01-24 06:37
I've been using story runner for the first time over the past 4 months or so and I've found the invaluable part to be the plain text support. Having a shared artifact that can be used by the customer, developers and testers is a huge boost towards achieving ubiquitous language. We use the story kickoff discussion to refine the language used in each scenario together, ensuring that the language makes sense to everyone and is consistent with other stories. In the sense that we want a way to store the feature/story in an executable format that we can all understand, a tool that supports plain text stories is the best choice for us. You can implement a scenario just as easily using plain rspec but then it becomes less accessible to other people in the team. If I was choosing a testing approach for a project that had technical people as customers I would still use the GWT format. I think it helps you continually focus on what you are really trying to achieve, otherwise it can be too easy to burrow down into the technical details and lose sight of that. No matter what the nature of project, every thing that you do is driven by the goal of adding some sort of value. Continually keeping that goal in mind helps you make better choices along the way. I agree with James that having that separation between the "what" and "how" is very useful. This approach is amenable to a more black box style of testing appropriate for acceptance tests. You should theoretically be able to completely change how your system works without touching the story/feature text. Your mileage would of course vary with this depending on whether or not you use a more declarative or imperative approach (as Ben describes in http://www.benmabey.com/2008/05/19/imperative-vs-d... ). My current recipe for testing spread is to automate the acceptance scenarios using a tool like story runner/cucumber and then use specs for targeted integration and unit level testing. The automated acceptance scenarios cover the main scenarios that the customer cares about while more specialised cases (or more technical aspects) are covered at the spec level. The specs will typically gravitate towards the white box end of the scale (describing the internals in more detail). - Brent
on 2009-01-24 13:15
Brent Snook wrote: > the internals in more detail). > Hi, I think this is the most used approach nowadays, high level testing with features and more specific testing, that end users don't care about at the spec level. However, I prefer the distinction between, declarative stories and imperative stories, which I think can be applied to these two levels also. For example, the end user or the business person, most probably gets bored of reading a very long imperative story of clicking links and filling out forms, plus other setup steps that might seem unnecessary to him, but are needed sometimes due to a not fully correct design. Whilst a developer will appreciate an imperative story as it tells him/her exactly what to do. In my opinion, even at the spec level we can use the philosophy behind steps (re-usability) to reduce the redundancies inherent to spec definitions. Trivial cases of you should see this in the page (view specs), or this object should have this attribute set to true (model specs), happen all the time, and seem like a great place to use variables inside step definitions. Other cases are more complex and the use of regular expressions to encapsulate them all, can certainly be very difficult, maybe the type of an attribute is a datetime and needs to be parsed, thus not being completely re-usable to say a boolean field. Still we could account for this variations by checking the type of the attribute, at the expense of increasing the complexity of the step. This is where the argument continues, people support the view of writing repetitive specs considering that they are clearer. I still have doubts about this. In my opinion this will certainly be a complex problem to solve, but once we tackle it a number of times, it will become trivial, we'll figure out an elegant step_helper to deal with this situations, and move on to the next hard problem. The hardest problem seems to be agreeing on a language to write the stories including minimal variations in the syntax and grammar of our steps, to be able to come up with a re-usable set of step definitions out-of-the-box, a set of steps that will allow us to write stories and jump straight to the failing nature of a step, due to it not being implemented, once its implemented the step will be passing, without further need to write trivial step definitions. This would be a great time saver and allow us to concentrate on harder problems very specific to our application that are not accounted for in the general step definitions. In conclusion, when not to use cucumber? never. Always use cucumber, everyone always use cucumber, lets find this hard problems to re-use steps and tackle them together, find a solution and move on to the next road-blocker. Use Rspec but as the underlying language to make cucumber work, not to divide your application into model view controller specs, separate your tests into user and developer features/steps. Distributed testing is a must to use cucumber effectively, and the use of mocks is also unrefutable, but only as an alternative to an inability to test something in a classical way Rai http://rubyconf2008.confreaks.com/testing-heresies.html http://martinfowler.com/articles/mocksArentStubs.html -- View this message in context: http://www.nabble.com/-Cucumber%2C-BDD--When-not-t... Sent from the rspec-users mailing list archive at Nabble.com.
on 2009-01-24 18:28
See comments inline... On 1/23/09 1:46 PM, James Byrne wrote: >> Cucumber provides? > effort I had to make to get the tool to behave as I wished (desires that > were often malformed by lack of understanding in any case), distracted > considerably from the task at hand, programming the application. > Interesting, thanks for sharing what your background is. I have seen other developers who don't have a strong OO background struggle with the concepts of isolation have a really hard time understanding how and why to use rspec. However, they seem to love Cucumber and it makes total sense to them. > I have progressed on my current project far, far more since discovering > Cucumber this past November than in all the time before. The > bifurcation between what and how seems far more natural to me with the > Cucumber features/step definitions file split than with the unified > expectation method/test steps of RSpec and the test method/assert > statements of TestUnit. Cucumber feature files just seem to permit one > to concentrate solely on "what" in the one case and then, with the > separate step definitions file, deal with the "how" in the other, > without crossing the line between the two. > Ahh.. that is a very good observation. When I write examples in rspec I start out with the example description first (obviously) but then I like to jump straight to my expectation. Meaning, I will say something like "dog.should be_happy" before I have even created the dog object. I like to do this for the same reason- I first write what I want and then I go a line above and create the how/given. A lot of people complain about the separation of code form story in plaintext features, but as you have pointed out that separation can be a very good thing. > deeply experienced developers and programmers might not get as much from > that is irrelevant to the question at hand. > +1. I totally agree with this. When it is difficult for me to write scenarios I find that the reason is that I fail to really understand what I am trying to accomplish. By flushing the acceptance criteria out in the GWT language I come out with a much better understanding of the problem and know exactly where to begin. > So here is where I disagree with you. :) I do agree that all the user cares about in the end is that the desired outcome is reached and the business value is provided. So with that mindset features are the only kind of tests a customer really cares about. Unit tests however are more for the developer. They aid in design and provide much more detailed documentation than the customer testes (acceptance tests). I don't think that cucumber features help me design my objects that much better unless I have focused object level examples telling me where the pain points are in my design. Eventually the customer will be greatful that you have unit tests (even if they don't know about them) because they will help lower the maintenace of the project by encouraging better design and providing tighter and more focused feedback on the smaller parts of the system. IMHO, of course. Thanks for the feedback, Ben
on 2009-01-24 18:30
On 1/23/09 10:04 PM, Brent Snook wrote: *snip* > > If I was choosing a testing approach for a project that had technical > people as customers I would still use the GWT format. I think it helps > you continually focus on what you are really trying to achieve, > otherwise it can be too easy to burrow down into the technical details > and lose sight of that. No matter what the nature of project, every > thing that you do is driven by the goal of adding some sort of value. > Continually keeping that goal in mind helps you make better choices > along the way. This is at the herat of what I was asking. Even if every "customer" of the project is fluent in ruby would you still bother with the plaintext GWT features of Cucumber? I think you have nailed it.. Or at least, have confirmed my thoughts on the matter. :) Business value is business value no matter who is the customer. While other tools can be used to automate acceptance criteria Cucumber provides the huge advantage of helping you focus and analyze on what the desired business value is. I have talked to other people who have claimed that the GWT gets in the way though, so YMMV. So, I wouldn't be surprised that in some technical domains the overhead of adding features just doesn't make sense. Thanks for your thoughts, Ben
on 2009-01-24 19:04
On 1/24/09 5:02 AM, voodoorai2000 wrote: > Rai > > http://rubyconf2008.confreaks.com/testing-heresies.html > http://martinfowler.com/articles/mocksArentStubs.html > > > Well, I have learned never to say never. ;) As far as the mocking goes... I always use real objects in features unless I need to make an external network call to another system (i.e. a web services call, in which case I like to use fakeweb). That said, one of the things I really like about having features is that it allows me to use mock objects in my object level examples without any fear of integration point failures. I find it very hard to test objects under total isolation without using mocks to remove dependencies and I gain a lot of design benefit from using mocks. I really don't want to get into the whole classic vs mockist debate though as it was just beat to death on the TDD mailing list. :) -Ben  http://github.com/chrisk/fakeweb/tree/master  http://tech.groups.yahoo.com/group/testdrivendevel...
on 2009-01-24 19:06
Hi, I've been using cucumber, and story runner before that since plain text stories were introduced 1.5 years ago. I also use rspec for model and controller specing/testing. I'll answer the opposite question: when do I use cucumber? I am a team of one, and I am my own client, so the way I use cucumber may be different than other people's experience. As outlined in my recent presentation (http://www.slideshare.net/linoj/cucumber-how-i- slice-it-presentation-924254 ) I've found myself using cucumber for: * Talking to myself (planning) * Keeping track of what I'm working on (outlining) * Automating my manual testing (scripting) * View specs * Regression testing (confidence) Plain text stories let me plan out my app from the end-user point of view. When I'm my own client, Given/When/Then helps me structure conversations with myself. Scenarios let me explore variations. However, I do not shoot for full coverage or many edge cases, I leave that for the controller and model specs. These tend to be short atomic G/W/T scenarios. When taking the BDD approach (which TBH is sometimes), it can take days to implement a single scenario, drilling down from feature to controller spec to model spec, etc, at which point the story provides a valuable context for what to work on next as i "pop the to-do stack" and work my way through the steps. However, in reality, sometimes I jump in and implement stuff first, and test it as I go by clicking in the browser (oh my!). At a certain point I'm repeating the same set of clicks so I write a cucumber story to more easily script the scenario. In this case there may be a high level Given step(s), and a sequence of multiple When's, with a few Then's inbetween (to validate where I landed, flash messages, or whatever). I've also started using cucumber in place of view specs. These scenarios will often be a simple Give/When with a whole bunch of Then's that validate elements on the page. By the way, I've written utility steps for viewing the response in a browser (http://www.vaporbase.com/postings/Show_me_the_response ), and I often use rdebug to work through my code. Finally, I love to run my suite of scenarios as regression tests, especially at stabilization points in my development, to make sure everything is still operational. I even have a "cap deploy:features" to run them on my staging server. linoj