You’re doing them wrong.

I’d even disagree with J.B., in that mocking expensive external resources is absolutely the wrong way to introduce the ideas. The important thing is to think about how object collaborate and what services they need from their neighbours, so Joe Walnes’ Novice rule is “Only mock types you own” — that way one can concentrate on design issues not performance.

Although some people have practised it forever, TDD is still very early in its adoption curve. Given the proportion of programmers who still struggle with OO concepts, we should be clear about when TDD practices are fundamentally broken and when they suffer from weak skills.

As Elliot and Harald K suggest both performance and robustness should guide the selection of tests in the suite. These should be real values, not potential performance and robustness.

Ravi, TDD has not promoted long private methods when I used it.

Michael, permission for more thoughtful exploration of the solution may be part of the value of TDD. Refactoring to some standard of cyclomatic complexity, method length of some other metric may be add as much to the design.

Harald, I believe there are many examples of quality software produced without either unit or integration tests.

BUT, ensuring that failures are handled perfectly is just as crucial on the other end of the stack — cfr 37signals’ book on “Defensive Design for the Web” for a somewhat-rambling but overall useful tome on that. Take care of failures, problems, and all sorts of exceptional conditions that you can only really test via mocks, and the normal “mainstream” flow will take care of itself, or, darn near;-).

Alex

Combinatorial test generators can generate an infinite number of different tests. See for example, Miller, Fredriksen and So (1990), “An Empirical Study of the Reliability of UNIX Utilities” or Eide and Regehr (2008), “Volatiles Are Miscompiled, and What to Do about It”.

So the issue of “how many tests is it worth running?” arises immediately.

The development strategy of having just one test suite that you run on every build doesn’t scale to large projects or large numbers of tests. You need some kind of hierarchy of tests: a basic set that you run before every commit; a larger set that you run regularly, and even larger sets that you run rarely, for example in the run-up to public releases.

The main reason I test with mocks is not performance or optimization, but predictability (also in the time it takes to run the test, but that’s just a minor part of it).

My mocks don’t start failing the 42nd time I run the test, because the network was down, some db table was corrupted, or a disk was full. They just run. I also make my mocks fail predictably, so that I can test that the API handles these failure conditions gracefully. Such conditions are really hard to test using the “real thing”. How do you make sure you gracefully handle a socket timeout, an I/O-exception in the middle of a stream etc, using the “real thing”? In additions, mocks can have expectations on them, that can be used to verify the API does what it’s supposed to. If you test with the “real thing” do you build the verification into your implementation?

Another thing to point out, is that even if integration tests and unit test are different beasts, it’s never one or the other. To create quality software you need both.

Unit tests only go so far, and in no way “prove” that the requirements will be met. Mocks are an abomination, requiring far too much effort to build. I prefer to test against the “real” environment, for exactly the same reasons that ERH stated - it is more realistic and is much more likely to flesh out actual problems. initially, I may use stubs that return “expected” values and then move on to the “real” environment.

As far as TDD goes, it is simply the wrong way to design software, elegant algorithms seldom arise from it. And TDD is never needed if we are familiar with the problem. It will lead to all sorts of bad design. If, as Michael points out, large private methods are a code smell, all I can ask is - but aren’t these methods the result of doing TDD? If so, what value does TDD have as a design tool if it often leads to bad code? Wouldn’t the average programmer be better off writing the correct code without TDD?

The thing that I do encourage is a rethink at a deeper level. It’s not uncommon to TDD a class and notice that some set of private methods are becoming uncomfortably complex.. complex in the sense that you can test them through the public interface but you’d much rather test them directly.

What is interesting to me is just how often this correlates to “complex enough to be its own class.” It’s easy to munge responsibilities together in code, and testing is an excellent probe. I guess, at the core, If it’s hard to write a test for something, it’s hard to claim that you understand it. Putting values in and knowing what the values should be coming back is where the rubber hits the pavement. If we notice the problem and split those private methods (and they data that the operate upon) into another class, it can be a win in terms of cohesion, coupling, and understandability.