There’s a philosophy in extreme programing circles that one should never break the build. As soon as the build is broken, everything stops until it can be fixed again.1 Some teams even hand out “dunce caps” to a programmer who breaks the build.
If by “build” you simply mean the compile-link-package cycle, then I tend to agree. Breaking the build is pretty serious. One of the advantages of extreme programming is that the increments of work are so small that you don’t get very far before discovering you’ve broken the build, so it’s relatively easy to fix. Integration is almost automatic rather than a painful, months long process. Avoiding coupling is also important to keep build times manageable.
However some systems define the build a little more broadly. They consider the build to include successful execution of all the unit tests. For example, Maven gives up if any unit test fails.* It will not create any subsequent targets such as a JAR file, if it can’t run unit tests. Ant doesn’t require this, but does allow this. All that’s necessary is declaring that the jar or zip target depends on the test target.
It’s not just open source either. Over at closed source vendor Atlassian, Charles Miller tells us, “all our tools are predicated on tests that start green and stay green. ” In fact, a failing test is so damaging to them, that he actually advocates writing tests that pass if the bug isn’t fixed and fails if it is. That’s a recipe for disaster if I ever heard one. Five years down the line some new programmer is going to finally fix the line of code that causes the bug, and then carefully reintroduce the bug to get back to the green bar.
This is where I part company from the most extreme of the extremists. If building includes passing all unit tests, then it is often acceptable and even desirable to break the build.
There are several reasons you may choose to introduce a failing test, thus breaking the build; but not fix it immediately. First of all, the person who writes the failing test may not be the best qualified to fix it. For instance, the person who finds the bug and writes the test could be a tester rather than a programmer; or they could be a programmer who’s working on a different piece of the code. Sure, they can fix the bug if they happen to see how to; but if they don’t, it’s still valuable to commit the unit test that proves the bug’s existence.
Even if the person who finds the bug is responsible for fixing it, the fix may not be apparent. Some bugs require a lot of work. Some bugs require only a little work, but may still require a night to sleep on it before the fix becomes obvious. Sometimes it’s a 30 minute fix, but the programmer only has 15 minutes before they have to pick up their daughter from football practice. For whatever reason, not every bug can be fixed immediately. It’s still better to commit the failing unit test so it doesn’t get forgotten in the future, even if that “breaks” the build.
Sometimes failing tests are beyond programmer control. For instance, XOM has one unit test that passes on Windows and Linux, but fails on the Mac. This particular test runs across a bug in Apple’s VM involving files whose name contain Unicode characters from outside the Basic Multilingual Plane. There’s not a lot I can do to fix that; but I shouldn’t pretend it’s not a problem either.
All too often projects are in denial about their bugs. Rather than accept clear evidence of a bug, they do anything they can to deny it. Refusing to allow the build to break, and then defining a test failure as build breakage, is just one example of this pathology. Recognizing, identifying, and reproducing a bug with a well-defined unit test is a valuable contribution. It should be accepted gladly, not met with hostility and an insistence that the reporter immediately provide a patch.
1 Joel Spolsky: “If a daily build is broken, you run the risk of stopping the whole team. Stop everything and keep rebuilding until it’s fixed. Some days, you may have multiple daily builds.”
2 You can tell Maven to exclude specific failing tests from your project.xml by listing the failing tests in an
excludes element like so:
<unitTest> <includes> <include>**/*Test.java</include> </includes> <excludes> <!-- currently broken --> <exclude>org/jaxen/test/JDOMXPathTest.java</exclude> </excludes> </unitTest>
You can also tell Maven not to run the unit tests using the command-line options
maven.test.skip=true to skip the unit tests or
maven.test.failure.ignore=true to run the tests but not stop if a test fails. However these are just hacks to get around what Maven really wants to do: run all the unit tests and fail the build if any fail.