Forgive the platitudes, but I didn’t see any other response to this point.

You test the tests by running them. If a test fails, you determine whether the error is in the code being tested, or in the test itself.

As Dijkstra said, “Testing can demonstrate the presence of bugs, but not their absence.” Testing is a (learnable) skill, and some programmers are better at it than others.

Testing, like proofs in mathematics, only serve to increase confidence that something is correct; tests, like proofs, may contain unnoticed flaws. Nothing you can do will ever completely guarantee correctness.

Finally, no programmatic tests can ever to written to test whether the program does what the customer wanted. That’s why customer-readable specs are needed; and flaws in those specs are the reason customer acceptance testing is also needed.

Any accusations of lack of originality in the above will be cheerfully admitted.

However, the test cases that are all or part of the ’spec’ do for a concrete and binary decision as to when the development is contractually and functionally complete. They also should be a good representation of the Use-Cases associated with the development, including exceptions and alternate paths.

In any case, at test case based spec shapely delimits the scope of the project and tends to limit tangential code development to implement functionality which may be nice (or even a great idea) but not (currently) in scope.

As a consultant, the ability to crisply define and managed scope and scope change is the only way to ensure profitability.

Additionally, this also manages client expectations as they are signing of specific and detailed action-response scenarios that define the system.

Here is one way to do this with TestNG. I hardcoded the numbers in this example to illustrate how @DataProvider works, but you should generate these numbers randomly so that the coverage of your function increases over time:

@DataProvider(name = “random”)
public Object[][] generateRandomExps() {
// This array should be generated with random numbers
return new Object[][] {
new Object[] { 0.0, Math.exp(0) },
new Object[] { 1.0, Math.exp(1) },
new Object[] { 2.0, Math.exp(2) },
};
}

@Test(dataProvider = “random”)
public void testExponent(double exponent, double expected) {
assertEquals(myExpFunction(exponent), expected);
}

Ravi

Did either of you bother to read to the end of the article or did you just scan the code? Sometimes you have to present an illustrative example in a different form than one would actually write code (or a test suite) to avoid obscuring the points in question. Additional levels of indirection are often useful in programming (and test suites) but they merely serve to obscure explanations.

As Randall pointed out, the simple fact that it takes so many methods to approximate what exp() does makes my point.

exp() is actually a very good example of a function that should be tested with data-driven testing, by the way, and not the way you approached it (see TestNG’s @DataProvider for more details).

All we need is a method that takes three arguments: the base number, the power to which it is to be raised and the expected result.

Write such a method, put this set of three values into a file or a database and validate as many tests as possible. In pseudo code:

for each set of values: // read from a file or database
assert that Exponent.calculate(base,power) = expectedValue

Why go for the duplicate code shown above?

This lack of pattern recognition is something I see all the time, even from so-called experts and well known authors. Basically it tells me that they are not thinking at all, they are merely following the herd; and mindlessly reproducing the same stupid code as exemplars of great sagacity.

If tests are specs, then how do you specify the tests?

Do you write tests to test the tests, or do you write a spec?

At what point does the “tests as specs” recursion bottom out and turn into gibberish or something that’s so incredibly self-evident that it doesn’t need specification?

And if it’s self-evident, then how do you formalize it in a way you can use it to build your first level of tests as specs?

These may seem like mere rhetorical trickery, but as someone who’s written both specs and tests for a lot of software, they are very real questions. They all boil down to the same basic question: Where do you draw the line?

It reminds me of my early days when other engineers would estimate 14 man-hours to code something that hadn’t even been designed yet. As an inexperienced developer, I wondered where such precision could come from in the face of such uncertainty. Later, I learned that some of it came from experience, but mostly it came from thin air: it was a hopeful fiction. Until this is ALL acknowledged as little more than hopeful fiction, how can it be taken seriously?