Quicksort is faster in both cases. Mergesort is stable in both cases. But for primitive types quicksort is stable too! That’s because primitive types in Java are like elementary particles in quantum mechanics. You can’t tell the difference between one 7 and another 7. Their value is all that defines them. Sort the array such [7, 6, 6, 7, 6, 5, 4, 6, 0] into [0, 4, 5, 6, 6, 6, 6, 7, 7]. Not only do you not care which 6 ended up in which position. It’s a meaningless question. The array positions don’t hold pointers to the objects. They hold the actual values of the objects. We might as well say that all the original values were thrown away and replaced with new ones. Or not. It just doesn’t matter at all. There is no possible way you can tell the difference between the output of a stable and unstable sorting algorithm when all that’s sorted are primitive types. Stability is irrelevant with primitive types in Java.

By contrast when sorting objects, including sorting objects by a key of primitive type, you’re sorting pointers. The objects themselves do have an independent nature separate from their key values. Sometimes this may not matter all that much–e.g. if you’re sorting java.lang.Strings–but sometimes it matters a great deal. To borrow an example from Sedgewick’s Algorithms I class, suppose you’re sorting student records by section:

public class Student {

  String lastname;
  String firstName;
  int section; 
  
}

Suppose you start with a list sorted by last name and then first name:

When you sort this again by section, if the sort is stable then it will still be sorted by last name and first name within each section:

However if you use quicksort, you’ll end up with something like this and have to resort each section by name to maintain the sorting by name:

That’s why stable sorts make sense for object types, especially mutable object types and object types with more data than just the sort key, and mergesort is such a sort. But for primitive types stability is not only irrelevant. It’s meaningless.

The problem is that my colleagues and I are not writing code in Haskell, Scheme, Lisp, Clojure, Scala, or even Ruby or Python. We are writing code in Java, and in Java functional programming is dangerously inefficient. Every few months I find myself debugging a production problem that ultimately traces back to a misuse of functional ideas and algorithms in a language and more importantly a virtual machine that just wasn’t built for this style of programming.

Recently Bob Martin came up with a really good example that shows why. Here’s a bit of Clojure (a real functional language) that returns a list of the first 25 integers:

(take 25 (squares-of (integers)))

This code runs, and it runs reasonably quickly. The output is:

(1 4 9 16 25 36 49 64 … 576 625)

Now suppose we want to reproduce this in Java. If we write Java the way Gosling et al intended Java to be written, then the code is simple, fast, and obvious:

for (int i=1; i<=25; i++)
    System.out.println(i*i);
}

But now suppose we do it functionally! In particular suppose we naively reproduce the Clojure style above:

import java.util.ArrayList;
import java.util.List;

public class Take25 {

    public static void main(String[] args) {    
        for (Object o : take(25, squaresOf(integers()))) {
            System.out.println(o);
        }
    }
    
    public static List<?> take(int n, List<?> list) {
        return list.subList(0, n);
    }
    
    public static List<Integer> squaresOf(List<Integer> list) {
        List<Integer> result = new ArrayList<Integer>();
        for (Integer number : list) {
            result.add(number.intValue() * number.intValue());
        }
        return result;
    }
    
    public static List<Integer> integers() {
        List<Integer> result = new ArrayList<Integer>();
        for (int i = 1; i <= Integer.MAX_VALUE; i++) {
            result.add(i);
        }
        return result;
    }
    
}

Try to run that. Go ahead. I dare you....OK, recovered from the heap dump yet?

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2760)
	at java.util.Arrays.copyOf(Arrays.java:2734)
	at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
	at java.util.ArrayList.add(ArrayList.java:351)
	at Take25.integers(Take25.java:30)
	at Take25.main(Take25.java:9)

How did Clojure handle a function that returns every single int, while Java crapped out? The answer is that Clojure, like pretty much all true functional languages (and unlike Java) does lazy evaluation. It doesn't compute values it doesn't use. And it can get away with this because Clojure, unlike Java, is really and truly functional. It can assume that variables aren't mutated, that the order of evaluation doesn't matter, and thus that it can perform optimizations that a Java compiler can't. And this is why functional programming in Java is dangerous. Because Java isn't a true functional language, the JIT and javac can't optimize functional constructs as aggressively and efficiently as they can in a real functional language. Standard functional operations like returning infinite lists are death for a Java program. That's why functional programming in Java is dangerous.

You may object that I've set up a straw man here. OK, you can't return a list of all the integers (or even all the ints) in Java; but surely no one would really do that. Let's look at a more realistic approach. Here again I use recursion to compute the squares rather than a loop:

public class Squares {
    
    public static void main(String args[]) {
        squareAndPrint(1, Integer.parseInt(args[0]));
    }
    
    public static void squareAndPrint(int n, int max) {
        System.out.println(n * n);
        if (max > n) {
            squareAndPrint(n + 1, max);
        }
    }
    
}

That will run. But now suppose I don't want the first 25 squares but the first 25,000:

Ooops. Stack overflow. This is why in XOM I was very careful to use loops rather than recursion, even in places where recursion was much clearer. Otherwise a carefully configured XML document could have caused a XOM-using program to dump core. Avoiding arbitrarily large recursion in non-functional languages like Java and C isn't just a performance requirement, it's a security requirement too!

Now before the flames begin, let me be clear about what I am not saying. I am not saying that functional programming is a bad idea. I am not saying that functional programming is inefficient. I actually love functional programming. Like my colleagues I find find functional programming to be a natural, intuitive, and beautiful style of programming but only when it's done in a language that was designed for it from the beginning like Haskell. Functional idioms in Java are performance bugs waiting to bite you.

1. Using java.util.Date or java.util.Calendar instead of JodaTime.
2. Not specifying a Locale when doing language sensitive operations such as toLowerCase() and toUpperCase().
3. Not escaping strings passed to SQL, XML, HTML or other external formats.

What I hate most is that it’s really, really hard to convince other developers that these are problems they should take seriously. The excuses are common:

“No, I don’t have to specify a locale here because the strings are ASCII.”

“I’m only getting a timestamp; I don’t need a proper timezone.”

“The data we’re encoding is coming from a web service we control, and we know it’s not going to send us any formfeeds or null characters.”

“This string is a constant so we clearly don’t need to escape it”, and so on.

All these answers reduce to, “yes, there’s sort of a theoretical problem here; and maybe FindBugs is complaining; but it doesn’t really matter in this case, and I’ve got more important things to spend my time on.”

And you know what? The nay sayers are right, 99% of the time. The problem is that every one of these issues can bite badly that 1% of the time, and it’s usually not obvious when you’re in a 1% case. For instance, even because the string being used to construct an HTML attribute value today is a literal, doesn’t mean it won’t be refactored into a variable next year, and then a variable built from user input a year later. Suddenly there’s an XSRF vulnerability in your code that two years ago everyone agreed clearly couldn’t happen, and thus no effort was put into preventing it.

Worse yet, although these problems are very easy to spot at the source code level–indeed can often be detected algorithmically by tools such as PMD or FindBugs–it’s usually not obvious what the cause of the problem is once it does manifest itself. For instance, out of all the myriad reasons a SOAP call might be consistently failing, is the possibility that the data contains an invisible form feed character the first thing that comes to mind?

I have seen major production problems caused by every one of these (#2 just this past week, and #3 the week before) and every one many times more than once. In the case of the failure to properly escape web service input before generating XML, the bug had lived in the code for years before an errant form feed showed up in the data stream and cost several engineer days trying to understand and fix the problem.

These aren’t hard or costly problems to prevent or fix, if we just develop good coding habits. Anytime you see a SQL statement built by string concatenation, alarm bells ought to be sounding. Anytime you see getBytes() invoked on a string without specifying a character set, you shouldn’t have to think twice about changing it to getBytes(Charsets.UTF-8). Anytime you see java.util.Date or java.util.Calendar in code, you should know that something is likely to go wrong, and probably at the worst possible time.

It’s like seeing a large stack of heavy boxes piled in front of an emergency exit. You don’t have to think about it, estimate the risk of fixing it compared to the risk of leaving it as is, file bug reports, or prioritize it compared to everything else you have to do. You just fix it as quickly as you can. These are dangerous situations; they’re easy to spot; and as professionals we have a duty to fix them when we find them and not to cause them in the first place.

If you’re designing for reuse, you’re doing it wrong.

In 2012 the only code you should be writing is what’s needed for the immediate task at hand. Don’t design for reuse. Don’t consider reuse. Don’t waste one minute of your day making code reusable.

The fact is any reusable code you need already exists. Want to connect to an HTTP server with full support for authentication and cookies? That sounds like something a lot of projects could use, so you should wrap it up in a nice HTTP class or library, right? Wrong. You should use Apache HttpClient instead.

Do you need to solve some initial value problems with a shooting method? Don’t crack out your numerical analysis textbook and start coding. Just download Flanagan’s Java Scientific Library or buy a NAG license instead. Need a date chooser widget and want to share it with your colleagues? Just tell them about JCalendar instead. Maybe that doesn’t have exactly the look and feel you were aiming for? Fair enough. Write your own component or fork the existing one, but realize that your very specific look isn’t likely to fit other people’s apps any better than JCalendar does, so don’t waste any time making yours reusable.

These examples are for Java, but the same is true in any major language including Perl, Python, Ruby, C++, C#, and Scala. In fact, if the language doesn’t have a library that solves the reusable parts of your problem, you shouldn’t be using that language for that problem.

Are there exceptions to this rule? I can think of two (and so far this feels like an exhaustive list).

The first exception is when you’re writing code for something so new that libraries don’t exists yet, and you’re the first one out of the gate, then make the code reusable. For instance, when I first wrote my XIncluder libraries the XInclude spec was still in development, and there really weren’t any alternatives in Java. These libraries became part of the proof of implementability that allowed the spec to advance to full recommendation status some years later. (That effort very nearly got me condemned to invited expert status on the working group, though fortunately saner heads prevailed.) Writing my own XInclude library made sense ten years ago, but I certainly wouldn’t repeat it today.

The second exception is for experts only, and I’m not even sure about this one. If you really are an expert in the field that the reusable code addresses, and if you have made a careful survey of the existing options and concluded that they are inadequate and you see how to do a better job, then, and only then, might you consider writing your own reusable code. This is what I did with XOM. Only after I had written a several hundred page book exhaustively documenting all the then current APIs for processing XML with Java and their stengths and weaknesses, did I sit down to design an API that improved on them. And although I do think I came up with the best such API yet designed, I’m still not sure that was the best use of my time. XOM is, IMHO, superior to what came before it; but it hasn’t been superior enough to really replace those other libraries in many applications. The need just wasn’t that great. As time passes, the code already available for reuse approaches “good enough” and the cost/benefit ratio of improving on it goes way up.

Are there other exceptions? Other times you really should write reusable code? I can’t think of any. Too many developers have spent too much time exploring the problem space, and made their work available for free at sites like Sourceforge and Github. While there will always be new problems to be solved, there’s just not a lot of benefit to be gained by solving the old problems one more time. The next time you find yourself designing for reuse, stop and ask yourself whether you should be reusing someone else’s code instead.

• I am finally going to memorize how one redirects both stderr and stdout to the same stream. (2>&1 |)
• I am going to learn the sed? trick my advisor showed me 20 years ago for repeating a command from the shell history while substituting one word for another, instead of just using the arrow key to backup to and erase the string. (^string1^string2^ or !!:s/string1/string2/ or for global substitution, not just the first occurrence !!:gs/string1/string2/)
• I am going to increase my regex fu and use regular expressions consistently instead of just editing 20 lines of copy and paste code. (This would be easier if every editor didn’t have subtly different syntax.)
• I am going to use Python to automate repetitive tasks.

BUILD FAILED
/Users/elharo/Projects/XOM/build.xml:545: Problem: failed to create task or type scp
Cause: Could not load a dependent class com/jcraft/jsch/Logger
       It is not enough to have Ant's optional JARs
       you need the JAR files that the optional tasks depend upon.
       Ant's optional task dependencies are listed in the manual.
Action: Determine what extra JAR files are needed, and place them in one of:
        -/opt/ant/lib
        -/Users/elharo/.ant/lib
        -a directory added on the command line with the -lib argument

Do not panic, this is a common problem.
The commonest cause is a missing JAR.

This is not a bug; it is a configuration problem

As usual, the ant error message is completely unhelpful, though for once it’s at least technically correct. (Most of the time when ant says, “This is not a bug; it is a configuration problem”, it is in fact a bug and not a configuration problem.) Here’s what’s really happening.

The jsch jar file distributed from http://www.jcraft.com/jsch/ is corrupt. Either they uploaded it wrong or they misconfigured their web server or both. (Update: it looks like a misconfigured server/poorly designed web page. The so-called download link isn’t that at all. You can follow it but not save it.) The jar file with the relevant classes is there, but it’s no good. You can check your local copy by trying to list its contents with
jar tvf

:

$ jar tvf /usr/share/ant/lib/jsch*jar
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:114)
	at java.util.zip.ZipFile.(ZipFile.java:75)
	at sun.tools.jar.Main.list(Main.java:979)
	at sun.tools.jar.Main.run(Main.java:224)
	at sun.tools.jar.Main.main(Main.java:1149)

If you see anything but a list of classes, the problem is that the JSCH jar is no good. To fix it, use this link to sourceforge instead.

Once again I am reminded of the perils of depending on external libraries, especially ones you don’t build or distribute with your own product. In 2010 ssh and scp are mandatory features of any build and deployment system. Secure communications are too important to be left to random third party web sites.

Is res a response, a reservation, a resume, or a result?

Is def a default or a definition?

Is rng a range or a random number generator?

Is v1 version 1 or value 1?

Is e an event, an entity, or an exception?

Is f a file or a float?

Is lst a list or the least value?

Is temp a temporary variable or a temperature reading?

Is rep a representation, a representative, a repetition, or a reputation?

Is tm a time or a trademark? Or even another temporary variable? And if it is a time, is it a timestamp, a time of day, or a duration? (These are three very different things.)

Is admin an administrator, an administrative assistant, or a system administrator?

In context, you can usually figure these things out, but you have to think about them. That’s inefficient. Far better to just spell out what you mean from the get go.

There are a few abbreviations that are so well known and understood that they’re acceptable:

• max for maximum
• min for minimum
• in for InputStream
• out for OutputStream
• e or ex for an exception in a catch clause (but nowhere else).
• num for number, though only when used as prefix as in numTokens, or numHits.

You can use single letter variable names for the occasional quantity that has no meaning other than its type. For example, a string variable can be named s, an int variable i, or a double variable x. However, this should only be used when the program really doesn’t know anything about the nature of the variable other than its type. For example, a method that calculates the cube root of a double may name its argument x; but a method that converts temperature from degrees Fahrenheit to degrees Celsius should name its argument degrees, degreesFahrenheit, or perhaps temperatureFahrenheit.

In addition, there’s nothing at all wrong with using common acronyms that are more recognized than what they stand for: URL, HTML, XML, XSL, etc. However, these are the exceptions, not the rules; and I would still be careful with common abbreviations that mean something else at first glance. EmployeeBO is almost certainly a Business Object, but that wasn’t what you read it as first, was it?

Code should be optimized for reading and comprehension, not for marginally faster typing. You should no more abbreviate names in your code than you do words in your sentences. No1 wnts 2 rd txt wrttn lk ths.

Let’s see exactly what Bruce’s mistake is. The following is an extended selection from Thinking in Java, 4th edition, pp. 490-491:

An exception-handling system is a trapdoor that allows your program to abandon execution of the normal sequence of statements. The trapdoors used when an “exceptional condition” occurs, such that normal execution is no longer possible or desirable. Exceptions represent conditions that the current method is unable to handle. The reason exception-handling systems were developed is because the approach of dealing with each possible error condition produced by each function call was too onerous, and programmers simply weren’t doing it. As a result, they were ignoring the errors. It’s worth observing that the issue of programmer convenience in handling errors was a prime motivation for exceptions in the first place.

One of the important guidelines in exception handling is “Don’t catch an exception unless you know what to do with it.” In fact, one of the important goals of exception handling is to move the error-handling code away from the point where the errors occur. This allows you to focus on what you want to accomplish in one section of your code, and how you’re going to deal with problems in a distinct separate section of your code. As a result, your mainline code is not cluttered with error-handling logic, and it’s much easier to understand and maintain. Exception handling also tends to reduce the amount of error-handling code, by allowing one handler to deal with many error sites.

Checked exceptions complicate the scenario a bit, because they force you to add catch clauses in places where you may not be ready to handle an error. This results in the “harmful if swallowed” problem:

try {
// ... to do something useful
} catch (ObligatoryException e) {} // Gulp!

Do you see the mistake? It’s a common one. Let me repeat it, so it’s really obvious [emphasis mine]:

Checked exceptions complicate the scenario a bit, because they force you to add catch clauses in places where you may not be ready to handle an error.

This is false. You are never forced to add catch clauses where you are not ready to handle an error. This isn’t a matter of opinion. It’s a matter of fact. Checked exceptions do not require catch blocks. They require a catch block OR a throws declaration. Eckel’s entire argument is based on ignoring the possibility of a throws declaration.

While Bruce is absolutely right that you should not catch an exception unless you know what to do with it, this in no way means that you should insert a catch block everywhere a checked exception may be thrown. If you aren’t ready to handle an error at one place, let the exception bubble up. If a checked exception is thrown inside a method where you are not ready to handle it, then the correct response is to add a throws clause to the method indicating that the exception will bubble up from that method. For example,

 public void doSomethingUseful() throws ObligatoryException {
   // ... do something useful that throws an obligatory exception
} 

You do not and should not insert a catch block in a method where you cannot do anything reasonable in the catch block. Checked exceptions never meant that every exception had to be caught as soon as it was thrown. It is perfectly acceptable to declare that a method throws a checked exception. Indeed, this is exactly how exceptions are meant to be used. It warns whoever calls your method that they need to be ready for this exceptional condition, and they either need to catch it and handle it themselves; or, they themselves need to declare that they throw it so that they warn their callers.

Yes, if it were true that every checked exception needed to be caught immediately, then checked exceptions would be incredibly inconvenient. However, experienced Java programmers don’t do this. Catching each and every checked exception at the first opportunity is a sure mark of a novice Java developer.

Occasionally, you’ll override a method inherited from a superclass or implement a method declared in interface that does not declare it throws the checked exception that your method throws and thus can’t throw the correct exception. In this case alone, it may be acceptable to wrap the exception in either a checked exception that the original declaration declares or in a runtime exception (if the original declaration does not declare any appropriate checked exceptions). For example,

   @Override
    public void doSomethingUseful() {
        try {
     // ... do something useful that throws an obligatory exception
        } 
        catch (ObligatoryException e) {
            throw new ObligatoryRuntimeException (e);
        }

However, you still don’t need to handle the exception before you’re ready for it.

I will note that this situation is a failure of design. When you’re forced to do this, one of two things is broken:

1. The superclass/interface was not designed properly for extension. Specifically it did not take into account the exceptions overriders/implementers might reasonably want to throw. The method likely should have been declared final and probably shouldn’t be extended at all.
2. The overriding/implementing method is violating the contract of the method it overrides/implements by doing something it really should not be doing.

A good example of the latter would be letting an IOException wrapped in a RuntimeException escape from the run() method of java.lang.Thread or java.lang.Runnable. These methods do not have sufficient context to handle such an exception, but something further down in the call chain does. The exception should be handled before it bubbles all the way up (though not necessarily in the same method where it’s first thrown).

In a method properly designed for extension, an empty throws clause (or a throws clause that does not match the actual exception) indicates that callers of that method cannot handle and do not expect such an exception. An overriding method that throws a new exception is violating the contract by failing to handle it, the same as it would were it to restrict a precondition or loosen a postcondition. (In essence, this is another form of loosening a postcondition.)

Eckel is not the only one to make the mistake of assuming that checked exceptions must be immediately at handled at the first possible opportunity. For example, he cites Barbara Liskov and Alan Snyder explaining their decision not to include checked exceptions in CLU:

requiring that the text of a handler be attached to the invocation that raises the exception would lead to unreadable programs in which expressions were broken up with handlers

True. Requiring that the text of a handler be attached to the invocation that raises the exception would lead to unreadable programs. Fortunately neither Java nor checked exceptions require any such thing. When handlers are appropriate they can usually be moved to the end of a method, far away from the the invocation that raises the exception. When handlers are inappropriate, you use a throws clause instead. Immediately following every statement that can throw an exception with a catch block is bad form on a par with goto.

Liskov and Snyder continue:

“We felt it was unrealistic to require the programmer to provide handlers in situations where no meaningful action can be taken.”

Also very true, but of course, checked exceptions do not require the programmer to provide handlers in situations where no meaningful action can be taken. When no meaningful action can be taken (out of memory error, stack overflow, class not found, etc.) Java programs throw Errors, not checked exceptions, not even runtime exceptions. Checked exceptions signal environmental problems that programmers cannot prevent or predict, should test for, and most decidedly can handle. To choose the most extreme example, if a production-worthy database system is writing a file and the disk fills up, it should handle the condition gracefully without corrupting the database. A disk full error is neither unforeseeable nor unmanageable. Most checked exceptions aren’t even that tricky to respond to.

What checked exceptions actually do require is that any method that can throw a checked exception warn its callers that the exception may be thrown. That’s all. A checked exception is nothing more and nothing less than part of the return type. Methods may return normally or they may throw exceptions. It makes just as much sense to specify the exceptions that can be thrown by a method as it does to specify that a method returns an int or a String. A method that does not declare the exceptions it throws is incomplete.

A lot of us didn’t really get checked exceptions when Java was released in the mid-nineties. It was a genuinely new idea that I don’t think any programming language before had foreshadowed. Liskov and Snyder wrote the paper quoted here in 1979, and their quotes make sense if you assume they simply didn’t conceive of having different kinds of exceptions in the language. if you only have one kind of exception then it makes sense for it to be a runtime exception rather than a checked exception.

Personally, I didn’t really understand how to use exceptions until the first edition of Effective Java was published. But it’s been 15 years. That’s long enough for the message to get out. In 2010 we know better. Proper error handling requires distinguishing programming errors (runtime exceptions) from environmental problems (checked exceptions). Proper error handling requires correcting programming errors and writing handlers for unpreventable environmental conditions. Proper error handling requires knowing when to catch and knowing when to throw. If you try to work with only one kind of exception, or try to get by with only catch but no throws, then exceptions are going to seem very ugly and inconvenient. But if you use checked and runtime exceptions, and use catch and throws, then your error handling code will be far cleaner, safer, and more maintainable. Error handling without checked exceptions and throws is like arithmetic without * and /. Sure, you can do it, but why would when you when using the features the language offers makes life so much simpler?

This not only builds the entire project. It tags the release in CVS, uploads the zip and tar.gz files to IBiblio, and uploads the documentation to my web host. It doesn’t yet file a bug to upload the maven files, but I’m working on that.

During the process of setting this up, I realized that my organization is a little backwards. In particular, I’m pushing all the artifacts from my local system. Instead, I should merely be committing everything to the source code control repository; tagging a release; and then having the further downstream artifacts like the zip and tar.gz files and documentation pulled from source code control onto the Web servers.

There are some commercial products that are organized like this, including ThoughtWorks’s Cruise, but none of the major open source hosting sites such as SourceForge and java.net work like this. Certainly, SourceForge and similar sites have been major contributors to the open source revolution. They have enabled hobbyist developers working in their garages to use tools and techniques of software development that were previously limited to corporations. They have it enabled far-flung developers around the world to collaborate with each other far more effectively than they could do by e-mailing each other tar files. They have removed the burden of system administration from many programmers, thus enabling them to devote more time to writing code. Make no mistake. SourceForge et al. are real force for good in the community.

That said, the state of the art in software development has moved forward significantly since these sites were founded. CVS has mostly been replaced by Subversion. On some projects, Subversion has been been replaced by distributed version control systems such as git and Mercurial. Unit testing and test driven development have moved from extreme practices to standard operating procedure. Continuous integration using products like Hudson and Cruise Control is routine. Nonetheless, most project hosting sites still offer little beyond a source code repository, a bug tracker, and some webspace. Not that that’s not important, but we can do so much more.

It’s time to think about what a modern project hosting site might want to offer and what it might look like.

Continuous Integration

The first step forward, and possibly the hardest, is to add continuous integration capabilities to the existing project hosting repositories. Every time code is checked into SourceForge or Java.net or code.google.com, the project should be built and the tests should be run. Technically, the hard part is understanding every project’s unique build infrastructure. Some projects use ant; some use make; some use maven; and some roll their own. Maven is probably the most constrained of a lot. For the others, it will be necessary to ask project owners which targets to run for which tasks. It’s probably a good idea to auto generate basic ant or maven or make scripts for new projects.

Beyond merely building the code, running the tests has some very serious security implications. Currently project hosting sites do not run third-party code. They store it; they display it; they make it available and bundle it up as tar and zip files; but they don’t even compile it, much less run it. Running arbitrary third-party Java and C code submitted by any random teenager with attention deficit disorder somewhere on the Internet is begging for trouble. 10 years ago I would’ve thought this was insane and impossible. But now, just maybe we can do it.

In fact, there are several services on the Internet today that will run arbitrary third-party code for all comers. Amazon’s EC2 service lets anybody with a credit card run what amounts to a complete rooted Linux box on Amazon’s network. Google’s AppEngine let’s more or less anyone, credit card or no, run Python and Java code inside Google’s cloud. And these are hardly the only such services. Advances in virtualization and security sandboxing have made this possible. That said, it certainly helps to have a real user attached to any code that you run so you know who to blame when it starts spamming the world. However when an application goes rogue whether through malice or incompetence, it is possible to shut it down quickly. It is possible to limit the resources used by anyone test suite, and to limit what else I can see on the same filesystem and the same network.

Codehaus uses Atalassian Bamboo to provide continuous integration, including test running, for their projects. However they’re a relatively small site that’s somewhat picky about the projects they host. They do use a separate server for the continuous integration. I’m not sure what other, if any, security precautions they put in place. Launchpad builds Ubuntu packages, but I’m not sure of they run tests. JavaForge builds Java code and runs the unit tests, apparently on top of Amazon EC2. Assembla will build and run tests, and also uses Amazon EC2. Both of these thereby delegate some of the security issues to Amazon’s virtualized systems.

submit queue

Once we’ve solved the problem of running continuous integration servers on project hosting sites, the next step is to flip them around. The usual process is to commit code to the repository and have the continuous integration server pull the code out of the repository. Then, if the build or tests fail, the continuous integration server goes into red mode and sends out alerts. Wouldn’t it be better if the server never turned red in the first place?

What should happen is that new code gets sent directly to the continuous integration server rather than to the source code repository. The continuous integration server pulls the latest known good build from the repository. then it patches the new code into the build and runs the tests. If the tests pass, the continuous integration server commits the code to the repository. If the tests fail, the code is never committed at all.

So far as I know, no current project hosting sites offer this; and it’s a relatively uncommon feature even among self hosted projects. However, it’s a critical one, especially when accepting contributions from the wide world of programmers, not all of whom have yet learned the importance of test driven development. I suppose such a site could also perform other checks on the source code. For example, it could verify coding conventions or measure the incremental code coverage before and after the check-in. It could automatically reject any patches that did not meet some predetermined measures of quality. That said, automated checks tend to be better used as additional data for humans to evaluate rather than as hard and fast rules. One way this can happen is by offering code metrics to code reviewers. This brings us to the next improvement in the code hosting ecosystem.

Code Reviews

Committing code, even assuming all the tests pass, is still a serious operation. Most open source projects don’t want to allow just anyone to commit code willy-nilly. Usually there’s a core group of committers that reviews all incoming patches and decides whether or not to accept them, to reject them, or to send them back for further work. This is somewhat labor-intensive both on the reviewer and the reviewee.

However, if we move to a submit queue-based system, this can become somewhat more straightforward. The continuous integration server can check every incoming patch regardless of the submitter’s status. If the tests pass, it can send an automatic request for review to a project commiter. If the commiter approves the change, then the continuous integration server can commit it to the source code control repository.

Indeed, it’s probably a good idea to require code reviews for all submitted changes, not just those from new users. After all, it’s not like the project’s owners are immune from introducing bugs. In fact, they probably introduce more than anybody else, if for no other reason than that they commit more code than anyone else. Code reviews are well known for increasing the quality of a code base and avoiding stupid errors, yet they’re one of the lesser used software development practices among open-source programmers. It’s time for that to change. Web-based code review interfaces such as Guido von Rossum’s Rietveld have the potential to really move the community forward here. We should integrate this technology or something equivalent into project hosting sites. code.google.com already offers code review, and a few others like BitBucket do too. The rest should follow.

One-button deployment

The final stage of software development is deployment. Eventually the software has to ship to and be installed by its intended users. Here is one area where open source projects have a significantly easier time than a lot of commercial projects, especially enterprise projects. The deployment process for many open source projects consists of little more than uploading a few jar files and some documentation to the right directories on the right Web servers. This should become a one-button operation.

All of project owners should have to do to release a new version is choose a version number and push a button. The server should pull all the code, documentation, and configuration information out of the source code repository; build everything; and put all the finished artifacts in the right locations. No further manual work should be required. This does require that absolutely everything needed to release goes into the repository; not only code but also HTML files, images, config files, and more. The only things that don’t go into the repository are the artifacts that are built from these components: jar files, zip files, Javadoc, etc.

Maven comes close to this, but it still builds and deploys from a local system rather than from the version control repository. This should be turned around. Ideally, maven deploy should work with nothing more than a pom.xml file. Deploying shouldn’t need to access the local maven repository or the local copy of the source code at all.

Summary

There might even be a startup idea in here somewhere. Open source projects aren’t the only ones that would like to offload some of the routine system administration tasks involved in running source code control repositories, continuous integration servers, bug trackers, and deployment pipelines. More likely, what’s really needed are some tweaks in and additions to the existing project hosting services. Or perhaps we can even take advantage of the advances in virtualization technology to install these services on top of Amazon EC2 and similar platforms.

But one thing is for certain: if open source projects are to keep pace with and surpass closed systems, then their software development practices need to be at least as good and probably better than the state-of-the-art in the overall software development community. In order to do that, it’s time to upgrade our tools.

public class Rectangle {

  private double width;
  private double height;

  public void setWidth(double width) {
    this.width = width;
  }

  public void setHeight(double height) {
    this.height = height;
  }

  public double getHeight() {
    return this.height;
  }

  public double getWidth() {
    return this.width;
  }

  public double getPerimeter() {
    return 2*width + 2*height;
  }

  public double getArea() {
    return width * height;
  }

}

public class Square extends Rectangle {

  public void setSide(double size) {
    setWidth(size);
    setHeight(size);
  }

}

(I’ve changed the language and rewritten the code to protect the guilty.)

There are actually several problems here. Thread safety is one, but let’s assume the class doesn’t need to be thread-safe. Another is that it’s possible to give the sides negative lengths. That we could fix with a couple of judiciously thrown IllegalArgumentExceptions. However the problem that most troubles me is demonstrated here:

Square s = new Square();
s.setWidth(5.0);
s.setHeight(10.0);

In object oriented programming, it is necessary that a subclass be able to fulfill the contract of its superclass. In this case, that means the square has to respond to setHeight() and setWidth() calls. However doing so enables the square to violate the nature of a square. A square cannot stand in for a rectangle.

You can try to work around this by overriding the setHeight() and setWidth() methods. For example, one might call the other:

public class Square extends Rectangle {

  public void setWidth(double width) {
    super.setWidth(width);
    super.setHeight(height);
  }

  public void setHeight(double height) {
    super.setWidth(width);
    super.setHeight(height);
  }

  public void setSide(double size) {
    super.setWidth(width);
    super.setHeight(height);
  }

}

However this is fundamentally unsatisfying because there is no reasonable expectation that calling one of setHeight() on a Rectangle object will also invoke the setWidth() method or vice versa. The contract of the Rectangle class is that you can set the width and the height independently, and the Square subclass violates that. Setting width when you’re setting height is an unexpected side effect. It violates the single responsibility principle¹.

We could instead just forbid setHeight() and setWidth() completely by throwing UnsupportedOperationException:

public class Square extends Rectangle {

  public void setWidth(double width) {
    throw new UnsupportedOperationException();
  }

  public void setHeight(double height) {
    throw new UnsupportedOperationException();
  }

  public void setSide(double size) {
    super.setWidth(width);
    super.setHeight(height);
  }

}

However, this is really just a louder way of warning the client that the Square class does not fulfill the contract of the Rectangle class. It doesn’t address the fundamental problem that, in object oriented terms, a square is not a rectangle. The geometric nature of a square is incompatible with the object-oriented definition of a rectangle given above.

There is, however, a way out of this conundrum. Our problem only arises because of the setter methods. If constructor were used instead, and no setters were exposed, then it would be possible to make a square a subclass of rectangle without violating any contracts. For example,

public class Rectangle {

  private double width;
  private double height;

  public Rectangle(double width, double height) {
    this.width = width;
    this.height = height;
  }

  public void getHeight() {
    return this.height;
  }

  public void getWidth() {
    return this.width;
  }

  public double getPerimeter() {
    return 2*width + 2*height;
  }

  public double getArea() {
    return width * height;
  }

}

public class Square extends Rectangle {

  public Square(double size) {
    super(size, size);
  }

}

As long as the Rectangle class is immutable, we can define subclasses that are limited to a particular subset of rectangles, such as squares. This is one more reason to prefer immutability. To the extent possible, define the public interface in terms of what an object is and what it does rather than what you can do to it. However that’s not always possible, and in those cases you need to be extremely careful around inheritance. Otherwise constraints can be violated when you least expect, thus introducing subtle and potentially dangerous bugs in your code.