User-agent: *
Disallow: /cgi/
Disallow: /cgi-bin/
Disallow: /mantis/
Disallow: /forum/
Disallow: /stats/
Disallow: /synk/unreg.html
Disallow: /synk/de/unreg.html
Disallow: /synk/fr/unreg.html
Disallow: /synk/it/unreg.html
Disallow: /synk/email.psn
Disallow: /synk/help/

This is from a small company whose main product is experiencing solid growth. In fact, they are growing so fast, they are having trouble responding to support e-mails and are consequently requesting that users check the FAQ list and read the forums before sending them e-mail. Keeping that in mind, can you tell what’s wrong with this robots.txt?

If it’s not obvious consider this: the FAQ list they’re asking me to read and the forums they want me to consult are both in /forum. See it now?

The problem is that they have blocked off Google (and other polite search engines) from finding their support information. As it turns out, I had two problems with this product today, both of which gave me confusing, unhelpful, and yet fairly unique error messages. Consequently I did what any techie would do with such a message. Put quotes around it, typed it into Google, added the product name, and scanned the results.

Google’s very good at finding error messages (you’re never the first person to have any given problem) but this time it popped up zero hits. OK. I figured I did something really weird, or had a strange configuration. Maybe I was the first person to see this problem, so I sent off a detailed report to support@company.com. Then I started browsing their web site, and noticed their pleas to please check the FAQ list and the forums. The FAQ list didn’t answer the question but the forums did.

Then I had another problem with the same product. Again Google didn’t help, but the forums did. (This turned out to be a bug with no workaround that may be fixed in the next point release.) Why didn’t I find the answer I was looking for with Google? Because this company had deliberately blocked Google from their web site using robots.txt. That’s not business smart or user friendly. Now they’ve got two e-mails from me to read and process they wouldn’t have had to deal with if Google could search their forums.

Folks, to save money on support, you have to give your users the best self-help tools you can; and when it comes to search, Google’s better than anything you’ll come up with on your own. Google wants to be your friend. Let it.

Software development is much the same. When designing an application it’s important to keep the feature count down to a manageable and comprehensible level for the intended users. Sometimes this requires you to cut features that considered indivdually seem valuable and useful. Software products today are rarely subject to the sort of hard size constraints one might find in a book or article, which makes it even more difficult to cut features. It takes exceptional judgement to determine when the cognitive load imposed by one more feature outweighs the benefit of that feature. Many developers flat out refuse to consider it. Nonetheless they need to. One of the few developers to get this right (not surprisingly) is Apple. Applications like iTunes, AddressBook, and iCal do less than their competition and this is a good thing. It’s not just that Apple knows how to better design user interfaces. It’s that they know when to stop.

APIs have the same issue. Many libraries add every feature a user asks for, because why not? it won’t hurt anybody, right? But more is not always better. Sometimes it’s worse. The inflection point can be hard to identify, but I’m convinced it exists. There is a point at which adding extra methods and classes makes the API and library harder to use, not easier. There is a point where the cost of finding the method you want outweighs the increased likelihood of it being there.

I suspect there’s a general principle at play here that goes beyond software. For instance, in designing massively multiprocessing systems like the Connection Machine or Virginia Tech’s Big Mac, a big problem is making sure the processors don’t spend more time syncing up with each other than they do calculating. Fred Brooks identified a variant of this problem in The Mythical Man-Month thirty years ago. Past a certain point adding more programmers to a late project makes it later. With a little thought I suspect you can remember similar examples in sports, education, politics, and other endeavors.

But bringing the focus back to software development, it’s important to keep a tight rein on features. For every feature you’re asked to add, don’t just ask yourself if it’s a good feature. Ask yourself if it’s good enough. How many people will it help? How much will it help them? If it’s going to help a lot of people, or help a few people a great deal, then go ahead and add it. However if it’s just a minor improvement, or is only needed by a few people, seriously consider leaving it out. Size carries real costs and those costs are too often ignored.