How do you specify an exponentiation function with a test?
While it may be a slightly too extreme position to say that tests are the only spec, I think it is absolutely reasonable to consider tests to be a major part of the spec. Indeed a specification without normative test cases is far less likely to be implemented correctly and interoperably than one with a solid normative test suite. The more exhaustive the test suite is, the easier it is to write a conforming correct implementation.
Cedric Beust presents the question, “how do you specify an exponentiation function with a test?” as a counterexample to tests as specs. Actually I don’t think it’s all that hard. Here’s one example:
import junit.framework.TestCase;
public class ExponentiationTest extends TestCase {
public void testZero() {
double x = Exponent.calculate(10, 0);
assertEquals(1, x, 0.0);
}
public void testOne() {
double x = Exponent.calculate(10, 1);
assertEquals(10.0, x, 0.0);
}
public void testNegativeBase() {
double x = Exponent.calculate(-1, 2);
assertEquals(1, x, 0.0);
}
public void testBigExponentWithOneBase() {
double x = Exponent.calculate(1, 2000);
assertEquals(1, x, 0.0);
}
public void testSquare() {
double x = Exponent.calculate(10, 2);
assertEquals(100, x, 0.0);
}
public void testSquareRoot() {
double x = Exponent.calculate(100.0, 0.5);
assertEquals(10, x, 0.0);
}
public void testFractionalBase() {
double x = Exponent.calculate(0.5, 2);
assertEquals(0.25, x, 0.0);
}
public void testNegativePower() {
double x = Exponent.calculate(100.0, -1);
assertEquals(0.01, x, 0.0);
}
public void testZeroZero() {
double x = Exponent.calculate(0, 0);
assertTrue(Double.isNaN(x));
}
}
Clearly you could expand on this short example with more test cases, and if I were writing this for real code I would. In particular I’d have to think hard about what happens when the results overflow the bounds of a double. But I think this makes the point.
Probably if I were really writing a spec, I wouldn’t define these test cases as compilable code like this. I’d likely just list the expected input, outputs, and tolerances; and then write a tool to convert them to actual tests. However that’s an implementation detail.
P.S. These test cases found what is at least arguably a bug (or perhaps a design defect) in Java’s Math.pow() method.
July 9th, 2006 at 2:34 am
I think the point was that no set of tests (well, short of a set that tests each and every double, and there are 2^64 of them, so that’s hopeless) will actually specify that the result is an exponentiation function and not something subtly different, that, say, fails on the argument 2.12382838482838295.
That’s not to say that such tests aren’t a Good Thing in every sense, just that it’s absurd to claim that they specify what the function is. Are you sure that if you handed this test set to someone and changed all the names that they’d be able to realize that this is an exponentiation function? I wouldn’t bet money on it.
July 9th, 2006 at 6:23 am
While it can be proven mathematically that there are an infinite number of functions that will produce any finite data set, I do expect that if I expanded this test set to the number of cases I would expect in real work, and then handed it to mathematicians who knew Java, they would rapidly identify it as an exponentiation function. even with random meaningless method and class names.
Beust flatly claims, “First of all, tests are not specs. Not even close.” I flatly disagree. Tests are specs, and very important ones. They should not be the only spec, of course; but they should be normative parts of the full and complete specification; not merely ex post facto derivatives from the specification. If my work with XML and other W3C products has taught me nothing else, it has taught me this.
July 9th, 2006 at 12:58 pm
I think you’d both agree, then, that tests are not complete specs. I’d hate to try to reverse engineer an XML parser solely from its (obfuscated) test suite if I’d never heard of XML, though.
July 9th, 2006 at 2:38 pm
Re: tests as specs.
If tests are specs, then how do you specify the tests?
Do you write tests to test the tests, or do you write a spec?
At what point does the “tests as specs” recursion bottom out and turn into gibberish or something that’s so incredibly self-evident that it doesn’t need specification?
And if it’s self-evident, then how do you formalize it in a way you can use it to build your first level of tests as specs?
These may seem like mere rhetorical trickery, but as someone who’s written both specs and tests for a lot of software, they are very real questions. They all boil down to the same basic question: Where do you draw the line?
It reminds me of my early days when other engineers would estimate 14 man-hours to code something that hadn’t even been designed yet. As an inexperienced developer, I wondered where such precision could come from in the face of such uncertainty. Later, I learned that some of it came from experience, but mostly it came from thin air: it was a hopeful fiction. Until this is ALL acknowledged as little more than hopeful fiction, how can it be taken seriously?
July 9th, 2006 at 3:22 pm
It’s more like 2^128 tests, not 2^64, because you have to pair every operand with every other operand.
July 9th, 2006 at 5:40 pm
Interesting that one has to write so many different methods for such a simple test. Isn’t that code duplication?
All we need is a method that takes three arguments: the base number, the power to which it is to be raised and the expected result.
Write such a method, put this set of three values into a file or a database and validate as many tests as possible. In pseudo code:
for each set of values: // read from a file or database
assert that Exponent.calculate(base,power) = expectedValue
Why go for the duplicate code shown above?
This lack of pattern recognition is something I see all the time, even from so-called experts and well known authors. Basically it tells me that they are not thinking at all, they are merely following the herd; and mindlessly reproducing the same stupid code as exemplars of great sagacity.
July 9th, 2006 at 9:25 pm
Elliotte,
As Randall pointed out, the simple fact that it takes so many methods to approximate what exp() does makes my point.
exp() is actually a very good example of a function that should be tested with data-driven testing, by the way, and not the way you approached it (see TestNG’s @DataProvider for more details).
July 10th, 2006 at 6:09 am
Cedric and Ravi,
Did either of you bother to read to the end of the article or did you just scan the code? Sometimes you have to present an illustrative example in a different form than one would actually write code (or a test suite) to avoid obscuring the points in question. Additional levels of indirection are often useful in programming (and test suites) but they merely serve to obscure explanations.
July 10th, 2006 at 8:10 am
I apologize. You are right, Elliote. The last paragraph does mention it.
Ravi
July 10th, 2006 at 5:00 pm
Ditto, Elliote, I read over the last paragraph too fast, sorry about that.
Here is one way to do this with TestNG. I hardcoded the numbers in this example to illustrate how @DataProvider works, but you should generate these numbers randomly so that the coverage of your function increases over time:
@DataProvider(name = “random”)
public Object[][] generateRandomExps() {
// This array should be generated with random numbers
return new Object[][] {
new Object[] { 0.0, Math.exp(0) },
new Object[] { 1.0, Math.exp(1) },
new Object[] { 2.0, Math.exp(2) },
};
}
@Test(dataProvider = “random”)
public void testExponent(double exponent, double expected) {
assertEquals(myExpFunction(exponent), expected);
}
July 13th, 2006 at 1:19 pm
I think the point is that test cases are an expression of the functionality of the system. In many cases, this expression may be sufficient to fully document the business, application and/or system requirements for simple systems, but may not.
However, the test cases that are all or part of the ‘spec’ do for a concrete and binary decision as to when the development is contractually and functionally complete. They also should be a good representation of the Use-Cases associated with the development, including exceptions and alternate paths.
In any case, at test case based spec shapely delimits the scope of the project and tends to limit tangential code development to implement functionality which may be nice (or even a great idea) but not (currently) in scope.
As a consultant, the ability to crisply define and managed scope and scope change is the only way to ensure profitability.
Additionally, this also manages client expectations as they are signing of specific and detailed action-response scenarios that define the system.
July 15th, 2006 at 10:31 am
Responding to:
If tests are specs, then how do you specify the tests?
Do you write tests to test the tests, or do you write a spec?
Forgive the platitudes, but I didn’t see any other response to this point.
You test the tests by running them. If a test fails, you determine whether the error is in the code being tested, or in the test itself.
As Dijkstra said, “Testing can demonstrate the presence of bugs, but not their absence.” Testing is a (learnable) skill, and some programmers are better at it than others.
Testing, like proofs in mathematics, only serve to increase confidence that something is correct; tests, like proofs, may contain unnoticed flaws. Nothing you can do will ever completely guarantee correctness.
Finally, no programmatic tests can ever to written to test whether the program does what the customer wanted. That’s why customer-readable specs are needed; and flaws in those specs are the reason customer acceptance testing is also needed.
Any accusations of lack of originality in the above will be cheerfully admitted. 🙂