I think a good api can be developed using TDD, and in fact looking at
the tests used to create an api is one of the best ways to learn it.
Yeah, I think this is true. I really only started using TDD very
recently, but one of the big things that swayed is me that TDD seems
to be at its best when building APIs, or, more accurately, all the
APIs I like the most seem to have been written using TDD.
Anyway, from the original post:
I am writing a web extraction framework in Ruby which is now relatively
big. I am doing kind of a semi-TDD (i.e. some things are done pure TDD,
and I am writing tests later for stuff which are not) - so at the end
everything should be covered by tests.
I have also tons of black-box tests (i.e. for the specified input and
output, yell if the output of the current run is different) and as the
code grows, also a great load of unit tests.
OK, just want to say, there’s a definite logical flaw in making
generalizations about TDD from a project which is “kind of a
semi-TDD.” The broader question of whether or not TDD rocks (I think
it does) is certainly an interesting question, but at the same time,
as certainly a different question.
Anyway, in terms of the actual practical issue, I think the easiest
interpretation of this is that you’re writing too many tests, and that
some of them are probably testing the wrong things. Any resource on
testing will reference the idea that TDD consists of writing a test
and then writing the simplest code that could possibly satisfy that
test. I think the hidden implication there is that you should also
write the simplest test that could possibly articulate your
requirements.
In practical terms, what I’d do in your situation is, if I made a
change and it blew up some tests, I’d examine the tests. Any tests
which depended on implementation details and didn’t really have much
to do with the goals of the application itself, I’d simply throw away.
That’s just a waste of time. Get rid of it. Any tests which were about
the API internals, rather than just the way a client programmer should
use the API, I’d either throw those tests away too, or possibly
separate them.
This is especially true as you’re changing the internals, refactoring,
throwing away objects, etc. What you’re really looking at might be a
boundary between what elements of the API you expose and what elements
can change without the client programmer ever even being aware of it.
Even if the client programmer is you, this distinction is still valid,
it’s basically the core question of OOP. Anyway, I can’t see your
code, I’m just guessing here, but I’d say make that separation more
explicitly, maybe even to the point of having client tests and
internal API tests, the goal being to have less tests overall, and for
those tests you retain to be cleaner and a little more categorized.
Client programmer tests and functional (“black-box”) tests, those are
important. They make sure that the results you get are the results you
want. API internal tests, if they’re blowing up on details of the
implementation, simply throw them away. If they’re testing for objects
that don’t exist any more, they’re just useless distractions. You
should retain some amount of tests for internals of the API, but
ideally, those are the only tests that should change when you’re
refactoring. Refactoring means changing the design without changing
the functionality. If you refactor and a test of the functionality
blows up, that’s really bad news. If you refactor and a test of the
API internals blows up, that’s nothing.
What it actually sounds like is spaghetti tests. You should really
only test for things that you need to know. The tests that go to
specific bits and pieces of the implementation, those should only be
in a part of the testing code reserved for testing the implementation.
Higher-level tests should only test higher-level behavior.
Ideally, if you change only one thing in your application, you should
only see one test fail. In reality that won’t necessarily happen,
certainly not right away in this particular case, but it becomes a lot
more likely if you refactor both the tests and the application at the
same time. In that case, the finer-grained your tests get, the
finer-grained your application will get along with it. What you want
to do is have very simple tests running against very simple code, so
that as soon as a single test breaks, you know exactly what went
wrong.
I literally JUST got started using TDD, but I had been putting it off
for years, and now, I totally 100% advocate it. Being able to
recognize instantly exactly what went wrong is a REALLY nice thing.
What makes legacy spaghetti so horrible to work with is the exact
opposite: it takes you forever to get even a vague idea of where the
mysterious thing that went wrong might be. Get fine-grained tests and
fine-grained code and you’ll be very happy.