I highly recommend this blog post by Jay Fields:
Cheers,
David
I like this line:
Today’s testing patterns are like beta software. The patterns have
been tested internally, but are rarely proven in the wild. As such,
the patterns will sometimes work given the right context, and other
times they will shit the bed.
So true, so hard to explain, so well put.
On 5 Feb 2009, at 05:05, David C. wrote:
I highly recommend this blog post by Jay Fields:
Jay Fields' Thoughts: Thoughts on Developer TestingCheers,
David
rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users
Matt W.
http://blog.mattwynne.net
David C. wrote:
I highly recommend this blog post by Jay Fields:
Jay Fields' Thoughts: Thoughts on Developer TestingCheers,
David
rspec-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/rspec-users
Great post. Very down to earth and realistic.
This post raised a question for me. Jay talks about how the pattern
Object Mother gave way to Test Data Builders. I use Fixjour (“Word to
your object mother”)[1], and other similar projects in my apps to
provide a set of sane default attributes for my models. You can then
easily override those defaults by passing in a hash with the values that
you need for the test. After reading Nat’s post about Object Mother vs
Test Data Builders[2] , I have come to the conclusion that the “Object
Mother” libs that are popular in ruby land (i.e. Fixjour, Factory Girl)
are really more in line with the Test Data Builder pattern. Does
everyone agree with this conclusion or am I missing something? It may
be that I just use it differently, but according to Nat’s post Test
Builders provide a sane set of defaults that you can then explicitly
override like so:
Invoice invoiceWithNoPostcode = new InvoiceBuilder()
.withRecipient(new RecipientBuilder()
.withAddress(new AddressBuilder()
.withNoPostcode()
.build())
.build())
.build();
Well… this looks a lot like what I do in Ruby (but a lot more verbose):
invoice_with_no_postcode = new_invoice(:recipient =>
new_recipient(:address => new_address(:postcode => nil)))
Nat points out that problems with Object Mother arise when people start
adding factory methods to deal with the edge cases, such as
ObjectMother.new_invoice_with_no_postal_code. I totally agree that this
would be a problem since such abstraction results in hard to follow
tests (this is why I hate fixtures actually). From the projects I have
worked on I haven’t seen the Object Mother libs abused this way and they
are used more like a Test Data Builder. The only difference I see is in
implementation, meaning the ruby libs tend to group all the factory
methods on one object or module just like Object Mother, while the
pattern Nat describes uses a separate builder class for each object. I
think this is really just details though and results from Ruby’s
differences from Java.
Any thoughts? Are Ruby’s Object Mothers really Test Data Builders?
-Ben
On Thu, Feb 5, 2009 at 10:17 AM, Ben M. [email protected] wrote:
http://rubyforge.org/mailman/listinfo/rspec-users
have come to the conclusion that the “Object Mother” libs that are popular
.build())
ObjectMother.new_invoice_with_no_postal_code. I totally agree that this
would be a problem since such abstraction results in hard to follow tests
(this is why I hate fixtures actually). From the projects I have worked on
I haven’t seen the Object Mother libs abused this way and they are used more
like a Test Data Builder. The only difference I see is in implementation,
meaning the ruby libs tend to group all the factory methods on one object or
module just like Object Mother, while the pattern Nat describes uses a
separate builder class for each object. I think this is really just details
though and results from Ruby’s differences from Java.Any thoughts? Are Ruby’s Object Mothers really Test Data Builders?
I’d say yes.
We also have a naming conflict in terms of test categories. The
definitions as I understand them are:
- Unit tests test units - generally one object at a time, or perhaps
more than one object that live within a single component, but never
crossing the component/layer boundary. - Integration tests test integration - between two or more components.
- Functional tests test functionality - of the application as a whole
(more like cucumber scenarios)
By these definitions:
- what Rails calls unit tests are really integration tests (model and
database layers) - what Rails calls functional tests are really bigger integration
tests (controller + model + database) - what Rails calls integration tests are really functional tests
(scenarios that take you through routing)
Confused? Me too.
Cheers,
David
On Thu, Feb 5, 2009 at 8:17 AM, Ben M. [email protected] wrote:
David C. wrote:
I highly recommend this blog post by Jay Fields:
Jay Fields' Thoughts: Thoughts on Developer Testing
It sounded to me like another of those “xxx is not a panacea” articles,
and
that “you shouldn’t do anything blindly” and “if it doesn’t work, you
shouldn’t do it.” Point taken (and taken and taken).
It was also indicative to me that Jay devotes just one word to TDD.
Literally, one word! He focused exclusively on testing as a way to find
bugs. That’s not why I write tests.
///ark
On 5 Feb 2009, at 16:55, David C. wrote:
By these definitions:
- what Rails calls unit tests are really integration tests (model and
database layers)- what Rails calls functional tests are really bigger integration
tests (controller + model + database)- what Rails calls integration tests are really functional tests
(scenarios that take you through routing)Confused? Me too.
And me. As much as Rails has done to promote automated testing, I feel
like this confused and confusing language is a real setback.
Matt W.
http://blog.mattwynne.net
just like Object Mother, while the pattern Nat describes uses a
separate builder class for each object. I think this is really just
details though and results from Ruby’s differences from Java.Any thoughts? Are Ruby’s Object Mothers really Test Data Builders?
Yeah, at least FixtureReplacement takes that stance, which is why I
didn’t call “Object Mother”
Scott
Pat M. wrote:
On Thu, Feb 5, 2009 at 8:17 AM, Ben M. [email protected] wrote:
Any thoughts? Are Ruby’s Object Mothers really Test Data Builders?
Here’s what I think is important:
- Make fixture setup explicit by inlining it on a per example/group
basis- Keep fixture setup short by hiding the object graph construction
- Allow concise parameterization of the fixture objects
- Focus 100% on structure (0% on behavior) in methods/objects to build
your data. This minimizes accidental coupling between examples.
Jay’s blog post finally got me inspired in using RSpec. I decided to
give a last chance at it, and I wrote my own “Factory framework” after
failing at using Machinist effectively and seeing the pain to maintain
fixtures.
Now I fully enjoy specing my models, and all my tests are
straightforward as each test has its own “setup”, it’s definitely not
dry but I don’t care, it is so much easier to maintain so much easier to
spot any bugs in the specs themselves and it happens a lot. Each model
has a dedicated prepopulated before(:each) describe section were setup
is “shared” by many examples, it’s usually used to test my custom find
methods were many models are part of the equation.
Specing Models heavily like I now do, also forces me to refactor my app
to transfer as much mogic as I can from the controllers to the models.
You can’t imagine how much code actually fits in the model, it’s
amazing, some of my controllers wen’t down by 50% and it enabled me to
spec effectively the models which I couldn’t do until now.
The next big step will be specing controllers, as it is more painful
than models, but now that my controllers are ripped I guess it will be
easier.
Also I am already seeing some specs that are not efficient, refactoring
specs is a compulsory task, so experiment different patterns as Jay
proposes in his blog.
On Fri, Feb 6, 2009 at 12:17 PM, Fernando P. [email protected]
wrote:
The next big step will be specing controllers, as it is more painful
than models, but now that my controllers are ripped I guess it will be
easier.
I’ve stopped. Mostly because of the question Jay asks in that
article: “Why are you testing?” I realized I had no idea why I was
testing controllers, except that I had been lurking here and reading
the RSpec docs and assumed that was the Way It Was Done. Finessing
all those mocks and stubs does chew up major time, and I noticed that
the controller code I was spending all that time writing stubs around
wasn’t typically code that broke. Most controllers for most apps are
so boilerplate that if you can trust your framework (Rails or Merb)
you can probably trust the controller.
“If it hurts, you’re doing it wrong” is a pretty good principle across
modern development. I’m sure there are people for whom mocking and
testing every controller action isn’t painful, or may even be fun.
I’m not one of them, so there’s no point in doing it just for rote. I
probably would do some controller isolation testing if I had
something complicated to do in an action or if my controllers seemed
prone to breaking and I wasn’t sure why. As David says, knowing more
techniques helps you pick the right one for the right task.
I’m using Cucumber with Webrat for simple request verifications now
and just matching on the output. I find it’s more fun and doesn’t
hurt (after I fixed some bugs with the Merb autotest flow) and it
tells me the same stuff as the controller testing, and more, in maybe
a tenth as much time to write.
–
Have Fun,
Steve E. ([email protected])
ESCAPE POD - The Science Fiction Podcast Magazine
http://www.escapepod.org
On Thu, Feb 5, 2009 at 8:17 AM, Ben M. [email protected] wrote:
Any thoughts? Are Ruby’s Object Mothers really Test Data Builders?
Here’s what I think is important:
- Make fixture setup explicit by inlining it on a per example/group
basis - Keep fixture setup short by hiding the object graph construction
- Allow concise parameterization of the fixture objects
- Focus 100% on structure (0% on behavior) in methods/objects to build
your data. This minimizes accidental coupling between examples.
The name you give a tool that fulfills these goals is never going to
be central to valuable conversation, so I think it’s of minimal
importance.
The rest of this email is somewhat ranty, and originally appeared
before what you just read above…but I figured I’d save some people
time and separate it out so you can skip over it if you want 
I don’t know what the distinction is, and to tell you the truth I
don’t think it matters. There’s miniscule evidence to suggest that
these are fundamentally different (quotes from
ObjectMother)
- These canned objects often aren’t just right for a particular test,
but often can be made right with some additional setup
Okay, but he doesn’t say HOW that additional setup comes about. Do
you get an object from the mother and then call some setters on it?
Do you parameterize calls to the OM? You can do either of those with
all of the fixture replacement tools that I know of.
- The canned objects that the Object Mother produces become familiar
to the team, often invading even discussions with the users
I guess this is where the “mother” thing comes from…mothers have
children, children are unique like snowflakes and have names and
personalities and yadda yadda. Personally I think that cast of
characters blows as a testing pattern. It always leads to inadvertent
coupling between tests and muddles intent. What makes more sense to
you as a developer, “that customer object in this test is Joe Bob” or
“that customer object is a customer with an account balance of $10k”.
If you want to get into pattern specifics, the question really is “am
I using factory methods or a builder,” to which the answer is “factory
methods” for every tool I’ve seen. Well, not quite. You can imagine
that
new_user :name => “Pat”, :email => “[email protected]”
is a different form of
UserBuilder.new.with_name(“Pat”).with_email(“[email protected]”).build
see…
def new_user(options={})
options.inject(UserBuilder.new) {|b,pair|
b.send(“with_#{pair.first}”, pair.last) }.build
end
A beautiful outcome of using Ruby is that many traditional patterns
fade in importance.
If you want to be very technical, these tools are neither Object
Mothers nor Test Data Builders right out of the box (assuming that TDB
uses the actual Builder pattern). Rather, they’re just
factories/factory methods, dynamically generated by reflecting on
given classes.
The real question is which one communicates better. I’d say TDB does
because it says exactly what it is - it builds data for your tests.
You could use it in a conversation and the other party would not need
to know that it’s a pattern of any kind.
But…Rails has fucked up the testing lexicon for tons of Ruby
programmers. There’s already a lot of fuzziness between
unit/functional/integration in the testing community as a whole, and
as David pointed out it’s made worse in the Rails world by how wrong
Rails gets the names. So when I talk to people I strictly use
“developer tests” and “acceptance tests.” I find that a much more
valuable distinction to make. And for developer tests, it makes a lot
more sense to discuss them in terms of value and tradeoffs than it is
to try to put them in certain buckets. How does that fit into this
discussion? Well, to me, a fixture is a set of objects in known
states that comprise the context in which a test runs. To lots of
Rails developers, fixtures are yaml files that define records in the
test db. Instant communication problem, because it implies the
question is Fixjour vs Foxy fixtures, which is absolutely not the
case.
Pat
On Feb 6, 2009, at 12:50 PM, Stephen E. wrote:
testing controllers, except that I had been lurking here and reading
the RSpec docs and assumed that was the Way It Was Done. Finessing
all those mocks and stubs does chew up major time, and I noticed that
the controller code I was spending all that time writing stubs around
wasn’t typically code that broke. Most controllers for most apps are
so boilerplate that if you can trust your framework (Rails or Merb)
you can probably trust the controller.
+1
I’ve found that sometimes writing a very simple controller test,
which hits the db is a better solution when you don’t have the time
and resources for a full cucumber story.
Mocking a controller without some sort of macros (like rspec-on-rails-
on-crack) is just too laborious and repetitive for my liking.
Scott
hey fernando,
maybe you want to take a look at
http://www.patmaddox.com/blog/2009/1/15/how-i-test-controllers-2009-remix
and give Cucumber a try regarding controllers spec-ing
hth,
joaquin
On Feb 6, 2009, at 10:21 AM, Scott T. wrote:
+1
I’ve found that sometimes writing a very simple controller test,
which hits the db is a better solution when you don’t have the time
and resources for a full cucumber story.Mocking a controller without some sort of macros (like rspec-on-
rails-on-crack) is just too laborious and repetitive for my liking.Scott
I did stop writing new controller specs, but we were discussing the
use-case for controller specs in the new Cukified world. Supposing you
write scenarios that pretty much describe your app, what could
possibly go wrong that a controller spec would catch but a cucumber
story pass? My environment is not using Selenium or Watir, so any
action that is Ajaxey potentially needs a controller spec example. I’m
less clear that routing needs to be in the specs, but shoulda makes
that stupid-simple, so it’s a way to make sure I don’t break something
embarrassing when I’m goofing around in routes.
I’m starting to think that Cucumber can do 90+% of the heavy lifting,
but for describing what kind of response a controller will ship back
(json or html), based on the request, user-agent specific stuff. All
that would probably have to happen in controller specs.
In response to Jay’s post, I test what I’m afraid might fail. I test
it to clarify my thinking as I build it. So if it’s important that a
controller return json in response to an XmlHttp request, then that’s
part of the spec.
Does this sound reasonable?
Joaquin Rivera P. wrote:
hey fernando,
maybe you want to take a look at
http://www.patmaddox.com/blog/2009/1/15/how-i-test-controllers-2009-remix
and give Cucumber a try regarding controllers spec-ing
hth,
joaquin
Hi,
I already use Cucumber+Webrat for testing that my publicly accessible
pages get rendered somehow correctly, and I am very pleased with it, so
I guess I don’t have to dive into deep testing my controllers as I
expected to.
Unfortunately most info / screencasts / docs available teach people how
to test controllers, routing, views, etc, but they fail to teach people
how to write better code in order to avoid having to test controllers
and such.
2009/2/7 Nick H. [email protected]
When writing Cucumber stories and features for controllers, should you
cover every edge case? For example, should you write stories that feed bad
or missing data to your controllers, or should that be left to RSpec?
-Nick
the missing data case should be left for a Scenario with some step like
And
I leave an obligatory field blank, and check for error messages, but I
guess
that does not sounds new. More interesting is the “bad data case”, are
you
thinking in let’s say some hacking attack to the app?
I would say that belongs outside Features, but where? what do you think?
or
am I missreading your opinion
lsd,
joaquin
On Sat, Feb 7, 2009 at 9:59 AM, Nick H. [email protected]
wrote:
When writing Cucumber stories and features for controllers, should you cover
every edge case? For example, should you write stories that feed bad or
missing data to your controllers, or should that be left to RSpec?
Depends on the team and who can read what and who cares about what.
In most cases, Cucumber is a significantly better
developer/stakeholder collaboration tool than RSpec is. If your
stakeholders trust you to cover the edge cases, then they don’t need
to be in cucumber, but they certainly can be.
IASTDH,
David
On 06/02/2009, at 10:00 PM, s.ross wrote:
I’m starting to think that Cucumber can do 90+% of the heavy
lifting, but for describing what kind of response a controller will
ship back (json or html), based on the request, user-agent specific
stuff. All that would probably have to happen in controller specs.In response to Jay’s post, I test what I’m afraid might fail. I test
it to clarify my thinking as I build it. So if it’s important that a
controller return json in response to an XmlHttp request, then
that’s part of the spec.Does this sound reasonable?
When writing Cucumber stories and features for controllers, should you
cover every edge case? For example, should you write stories that feed
bad or missing data to your controllers, or should that be left to
RSpec?
-Nick
On 07/02/2009, at 1:16 PM, David C. wrote:
In most cases, Cucumber is a significantly better
developer/stakeholder collaboration tool than RSpec is. If your
stakeholders trust you to cover the edge cases, then they don’t need
to be in cucumber, but they certainly can be.IASTDH,
David
Makes sense. Thanks for the clarification.
By the way, what on Earth is IASTDH? A few Google searches later, I’m
still none the wiser.
Hi–
On Feb 7, 2009, at 7:59 AM, Nick H. wrote:
Does this sound reasonable?
When writing Cucumber stories and features for controllers, should
you cover every edge case? For example, should you write stories
that feed bad or missing data to your controllers, or should that be
left to RSpec?
-Nick
Why wouldn’t you cover the negative scenario. E.g.:
Feature: Projects dashboard
Scenario: A user who has not set up any projects yet should be sent
to the “set up new project now!” page
Given a user, “Emil” has no projects
When “Emil” clicks on the “dashboard” link
Then he sees the “new project” page
And he sees the greeting “set up new project now!”
Scenario: A user who has one or more projects should be able to
examine the list of projects
Given a user, “Gary” has 1 project
When “Gary” clicks on the “dashboard” link
Then he sees the “project list” page
Wouldn’t that make sense?
On Sat, Feb 7, 2009 at 8:05 PM, Nick H. [email protected]
wrote:
By the way, what on Earth is IASTDH? A few Google searches later, I’m still
none the wiser.
I made it up. I was about to write HTH, but I was sure it did not
help, hence IASTDH - inconsistent though - IA for I am, but D for
didn’t, so it should either be ISTDH or IASTDNH.
HTH,
David