The Cafes

Prefer Multiline if

2009-01-01T22:22:22Z

C-family languages including Java, C#, and C++ do not require braces around single line blocks. For example, this is a legal loop:

for (int i=0; i < args.length; i++) process(args[i]);

So’s this:

for (int i=0; i < args.length; i++)
    process(args[i]);

However both of these are very bad form, and lead to buggy code. All blocks in C-like languages should be explicitly delimited by braces across multiple lines in all cases. Here’s why:

The most dangerous form is a multiline block that doesn’t use braces. The problem is that when you start with this:

if (p.needsGiftWrapping())
    wrap(p);

sooner or later some programmer–perhaps you, perhaps someone else–is going to discover a need to add a second line. For example:

if (p.needsGiftWrapping())
    wrap(p);
   ribbon(p);

Bang! That code is now buggy. It looks like the if block applies to both statements, but in fact it only applies to one. The indentation is lying about the intent of the code. By contrast, this form is less dangerous:

if (p.needsGiftWrapping()) wrap(p);

In this example, it’s obvious to any programmer who comes along and adds a line, that they need to add braces too. It’s less likely to cause bugs down the road, but it should still be avoided, and here’s why.

The statement

if (p.needsGiftWrapping()) wrap(p);

is really two statements: one that calls needsGiftWrapping() and one that calls wrap(p). These are independent statements, and may need to eb treated separately. In particular, you may want to mark a breakpoint on one and not the other. For instance, I sometimes like to put a breakpoint in the body of an if or for or while, just to make sure that this code is really being executed when I think it is. If the program doesn’t stop, then the code isn’t being executed; and I have a big clue where the bug is.

This is also important for code coverage tools. Most tools such as Cobertura measure the coverage of lines of code, not statements. Even if they measure lines of code coverage and statement coverage separately (and some tools do this) they still display the coverage with lines of code. This statement can be marked covered even if the body of the block is never entered:

if (p.needsGiftWrapping()) wrap(p);

However, if the block is rewritten like this:

if (p.needsGiftWrapping()) {
    wrap(p);
}

it now becomes obvious if the tests are never testing the case where p.needsGiftWrapping() returns true.

This applies equally to all block statements: if, for, while, do while, and any others you may encounter. I’m beginning to believe this is actually a special case of a general principle for C-like languages, and perhaps others:

Each line of source should contain exactly one statement

That is, avoid lines like this:

int i = 7, j = 18;

int i = j = 18;

Similarly, avoid the ?: operator.

if (a > b) {
  max = a;
}
else {
  max = b;
}

is easier to read than

max = a < b ? a : b;

For instance, did you even notice the bug in the above line? If the verbosity bothers you, try something like this instead:

max = Math.max(a, b);

The problem may reflect a misdesign in C-family languages. The compiler only pays attention to the semicolons and braces while ignoring the line breaks and indentation, but humans usually only pay attention to the line breaks and indentation while ignoring the semicolons and braces. This gives the code the opportunity to lie about what it’s really doing. Consequently we need to take extra care when writing in C, Java, C++, C#, etc. not to lie to ourselves. If you place exactly one statement on each source line, you can be reasonably confident the code isn’t lying to you, and you’ll have a much easier time debugging.

Compact code is fun, but it’s not maximally readable, and more lines don’t really cost you anything after the compiler is finished anyway. Save yourself the hassle and for 2009 resolve to put one statement on each line.

P.S.

If you want to comment on this article, remember that the <pre> tag is allowed in comments. Otherwise your point may get lost along with your indentation.

Monopoly Incompetence

2008-12-21T09:22:17Z

Need more proof that monopolies are bad business? Just try to pay a utility bill online sometime. I have just gotten through attempting to pay my cable, gas, and electric bills online. Exactly none of them offered what I would consider a minimally competent site. The exact problems varied, but there was one that was common across the three. Every single one required registration before they’d take my money:

By contrast, non-monopoly sites like Office Depot have long since learned that registration is an optional step they shouldn’t let get in the way of completing a sale. But the utility companies? Either they hire developers who are distinctly behind the state of the art, or they just don’t care because you have to pay them, or both.

How easy should it be to pay a utility bill online? It’s a form with about four or five fields, no more:

Account number
Credit Card Number
Expiration Date
Name on Credit Card
Amount to pay (this one could even be autofilled based on the account number.)

Unlike an online store , they shouldn’t need to ask for a shipping address. Maybe they need a billing address (though that should default to the service address) or the CVV2 code, but utility bills are unlikely to attract the same sort of fraud that online stores do, so I’m not sure even that’s necessary.

They absolutely don’t need:

Your e-mail address
A username
A password
Service address
CAPTCHA (They really think someone’s going to set up a bot to autopay utility bills?)
The city where your mother was born. (No I’m not making that up. Cox really wanted that piece of information.)

Account number, minimal credit card (or debit card) info, amount to pay. That’s all.

Java is Dead! Long Live Python!

2008-12-09T14:27:01Z

Version 3.0 of Python has been released. Notably Python has again done something Java has long resisted: it has broken backwards compatibility with Python 2.x. Notable fixes include a much saner string processing model based on Unicode. I am told by my Pythonista colleagues that a lot of other weirdnesses such as the print operator and the meaning of parentheses in except clauses have been cleaned up as well. Though I don’t expect all Python programmers to upgrade immediately (and version 2.x will be maintained for some years to come) version 3.0 is clearly a simpler, better, saner language than version 2.x that will enhance productivity and make programmers’ jobs more fun. Bravo for Python. This is clearly a living, evolving language.

Java by contrast, is dead. It has at least as much brain damage and misdesign as Python 2.x did, probably more; yet Sun has resisted tooth and nail all efforts to fix the known problems. Instead they keep applying ever more lipstick to this pig without ever cleaning off all the filth and mud it’s been rolling in for the last 12 years. They keep applying more perfume when what it really needs is a bath.

Backwards compatibility was maintainable and useful through about version 1.4 of Java, but it completely broken down in Java 5 when autoboxing and generics moved the core language beyond any hope of comprehensibility. Autoboxing was a misguided effort to paper over Java’s early decision to have a segregated type system for primitives and objects. It was Java’s Plessy v. Ferguson decision that pretended primitives and objects were separate but equal; but the claim was no more true in Java than it was in American jurisprudence. A separate primitive type system may have made sense in 1995 when CPUs were slower and virtual machine technology was not as advanced. Today primitive types just complexify the language to no particular benefit. Autoboxing would not have been necessary or even considered were backwards compatibility not worshipped beyond all other gods.

Generics are another case where backwards compatibility took a good idea and warped it into something horrible. Multiple design compromises were made to enable genericized code to run in older VMs, most notably type erasure. Then, in the end, binary compatibility was broken anyway. However no one went back to the drawing board and considered how much simpler and more powerful generics could be if they redesigned without worrying about backwards binary compatibility.

Closures, if added now, would only make the situation worse. Closures might be a nice addition to the language if and only if Java simultaneously removed inner classes and made all other syntactic changes necessary to support true closures. Otherwise closures will just be generics squared. New features simply cannot be added on top of the current weak foundation unless we’re willing to go back to the drawing board and take things out as well.

I can’t think of another major language as old as Java that still attempts to maintain compatibility with version 1.0 of itself. In fact, I can think of only one language that attempts that (C#), and that one’s half Java’s age. Unless we’re willing to make the hard choices and abandon the legacy as Python has, Java is doomed to the fate of C++ and Cobol: a tool for programmers with long white beards who grew up with the language and have learned all its arcana by gradual accretion and who spend their lives maintaining code written a decade or more ago. Meanwhile a new generation of programmers will abandon Java in favor of more nimble modern languages like Python just as we abandoned C++ in our youth in favor of Java. (Seriously: is anyone under the age of 30 actually reading this site any more?)

Admitting that you have a problem is the first step to recovery. Java has not yet admitted that it has a problem. The language is too big, too complex, and too baroque. Trade-offs that made sense in the era of single core Pentium II’s, 100Mhz processors, and 32 megabyte memory spaces no longer apply. Backwards compatibility has become a millstone around Java’s neck. We’re deep and sinking fast. Until this millstone is cast off, and we correct the mistakes of the past, no further progress can be made.

It’s hard to believe that I first started saying this over five years ago, and absolutely no progress has been made in that entire time. In fact, matters have gotten worse. Maybe Java is a lost cause, and it’s time to fork and replace the language. If nothing else, Java proved that’s possible. Just look what it did to C++. Perhaps it’s time to repeat the experience.

Keep Your Methods Private and your APIs Minimal

2008-11-11T15:53:11Z

I was reminded once more today just how important it is to write minimal APIs that don’t expose more than they have to. Briefly I had code like this:

 private boolean flag;

 public boolean getFlag() {
   return this.flag;
 }

  public boolean setFlag(boolean value);
    this.flag = value;
  }

Pretty boilerplate stuff, I think you’ll agree.

However I noticed that after some refactoring that merged a couple of classes I was now only calling getFoo() from within the same class (or at least I thought I was) so I marked it private. Eclipse promptly warned me that the method was unused so I deleted it. Then Eclipse warned me the field was unread. That seemed wrong so I looked closer and yep, it was a bug. The feature the flag was supposed to control was always on. During the refactoring I had failed to move the use of the flag field into the new class. I added a test to catch this, and fixed the problem.

What’s interesting about this example is that I found the bug only because I was aggressively minimizing the non-private parts of my API. The less public API a class has, the fewer places there are for bugs to hide. The less public API there is, the easier it is for analyzers–static, dynamic, and human–to detect problems.

Many programmers subscribe to a cult of extensibility: Extensibility is always good. Opening up the API makes it more testable. Compile time static type checking really doesn’t help anyway. You might need it someday so put it in now. Maybe you don’t need it, but someone else might.

This way lies madness. Most extensibility points are never used. How many getters and setters are actually invoked? Not as many as you’d expect, and even fewer if test classes are not considered. How many non-final classes in your code actually have subclasses? How many of these classes have actually been designed and documented for extensibility? How many non-final fields actually are mutated after construction? Certainly some are. Not all methods and fields can or should be final. Some classes are designed to be subclassed. Some fields do need getters or less frequently setters. But none of these features should be added to your classes out of habit.

For maximum safety, remove as much as you can and lock down what you can’t. Make classes (and/or methods) final by default. Make fields final and immutable. Don’t mark a method public if it can be package protected instead. Don’t routinely add getters and setters for each field; and if you do need a getter or setter, just add the one you need, not both. In this case, I needed a setter, but the getter was gratuitous and removing it revealed a real bug.

Follow the YAGNI principle: You Ain’t Gonna Need It. Never add an extension or access point–be it a method, a field, a non-final class, etc.–until you know you need it. This will improve your code’s robustness, thread safety, security, speed, and more.

Harold’s Corollary to Knuth’s Law

2008-12-09T14:28:09Z

Lately I’ve found myself arguing about the proper design of unit tests. On my side I’m claiming:

Unit tests should only touch the public API.
Code coverage should be as near 100% as possible.
It’s better to test the real thing than mock objects.

The goal is to make sure that the tests are as close to actual usage as possible. This means that problems are more likely to be detected and false positives are less likely. Furthermore, the discipline of testing through the public API when attempting to achieve 100% code coverage tends to reveal a lot about how the code really works. It routinely highlights dead code that can be eliminated. It reveal paths of optimization. It teaches me things about my own code I didn’t know. It shows patterns in the entire system that makes up my product.

By contrast some programmers advocate that tests should be method-limited. Each test should call the method as directly as possible, perhaps even making it public or non-private and violating encapsulation to enable this. Any external resources that are necessary to run the method such as databases or web servers should be mocked out. At the extreme, even other classes a test touches should be replaced by mock implementations.

This approach may sometimes let the tests be written faster; but not always. There’s a non-trivial cost to designing mock objects to replace the real thing; and sometimes that takes longer. This approach will still tend to find most bugs in the method being tested. However it stops there. It will not find code in the method that should be eliminated because it’s unreachable from the public API. Thus code tested with this approach is likely to be larger, more complex, and slower since it has to handle conditions that can’t happen through the public API. More importantly, such a test starts and stops with that one method. It reveals nothing about the interaction of the different parts of the system. It teaches nothing about how the code really operates in the more complex environment of the full system. It misses bugs that can emerge out of the mixture of multiple different methods and classes even when each method is behaving correctly in isolation according to it spec. That is, it often fails to find flaws in the specifications of the individual methods. Why then are so many programmers so adamant about breaking access protection and every other rule of good design as soon as they start testing?

Would you believe performance?

For instance consider this proposal from Michael Feathers:

A test is not a unit test if:

It talks to the database

It communicates across the network

It touches the file system

It can’t run at the same time as any of your other unit tests

You have to do special things to your environment (such as editing
config files) to run it.

Tests that do these things aren’t bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.

More than 30 years ago Donald Knuth first published what would come to be called Knuth’s law: “premature optimization is the root of all evil in programming.” (Knuth, Donald. Structured Programming with go to Statements, ACM Journal Computing Surveys, Vol 6, No. 4, Dec. 1974. p.268.) But some developers still haven’t gotten the message.

Are there some tests that are so slow they contribute to not running the test suite? Yes. We’ve all seen them, but there’s no way to tell which tests they are in advance. In my test suite for XOM, I have numerous tests that communicate across the network, touch the filesystem, and access third party libraries. However, almost all these tests run like a bat out of hell, and take no noticeable time. The slowest test in the suite? It’s one that operates completely in memory on byte array streams with no network access, does not touch the file system, uses no APIs beyond what’s in Java 1.2 and XOM itself, and there’s no database anywhere in sight. I do omit that test from my standard suite because it takes too long to run. I’ll run it explicitly once or twice before releasing a new version, but not every time I make a change.

I am now proposing Harold’s corollary to Knuth’s law: premature optimization is the root of all evil in testing. It is absolutely essential to make sure that your test suite runs fast enough to run after every change to the code and before every check in. I’m even willing to put a number on “fast enough”, and that number is 90 seconds. However, you simply cannot tell which tests are likely to be too slow to run routinely in advance of actual measurement. Castrating and contorting your tests to fit some imagined idea of what will and will not be slow limits their usefulness.

Tests should be designed for the ideal scenario: a computer that is infinitely fast with infinite memory and a network with zero latency and infinite bandwidth. Of course, that ideal computer doesn’t exist, and you’ll have to profile, optimize, and as a last resort cut back on your tests. However, I’ve never yet met a programmer who could reliably tell which tests (or other code) would and would not be fast enough in advance of actual measurements. Blanket rules that unit tests should not do X or talk to Y because it’s likely to be slow needlessly limits what we can learn from unit tests.

10 Things to Know Before You Go To Beijing

2008-08-04T14:13:08Z

1. Learn Mandarin.

Even a little will go a long way. English is very uncommon here. All those tourist phrase books and Berlitz courses that did you absolutely no good in Europe because everyone spoke English? They actually help here. The most important phrase to know is “Boo-yao” which loosely translates as “No, I don’t want that cheap plastic souvenir/guide book/Rolex/Gucci bag you’re trying sell me, and I really mean it.”

2. Money

China is a cash society. Credit cards are rarely accepted, not even in restaurants, large department stores, or major tourist destinations.

Change your money at the airport. It’s relatively hard to do afterwards.

3. Where to Stay

Stay within the 2nd Ring Road unless you have a specific reason to be elsewhere. (The Olympics are mostly outside the Fourth Ring Road.) Beijing is a large city with a lot of traffic. It can take a while to get around.

Chinese hotels (that is, ones that cater to natives instead of foreign tourists and businesspeople) are not the same as North American hotels. Worse in some ways, better in others; but most U.S. travelers will be uncomfortable.

4. Food

Learn to eat with chopsticks; or, if you must, bring plastic forks.

I got conflicting advice about whether to drink the water, but I didn’t risk it. Bottled water is cheap and easy to find.

If you do drink the water, there are public toilets everywhere. Learn to squat. Bring toilet paper because most bathrooms don’t have any.

5. Transport

Cabs are plentiful and cheap, but the drivers don’t speak English. Write down your destination in Chinese characters, or ask someone at the hotel to do it for you. Many guide books list addresses in both English and Chinese characters so if necessary you can get by with pointing.

Make sure to take one of the Yellow cabs. (Really yellow and brown, yellow and green, or yellow and red). These are official cabs with meters. Other “cabs” are usually private cars out to make a few bucks from the tourists and will cost you a lot more (though still not as much as an equivalent ride in new York or London, I’m compelled to note.)

The subway is easy to navigate and by far the fastest way to get around town. Signs and announcements are in both Chinese and English. Choose a hotel within easy walking distance of a subway station.

At rush hour, if you’re not near a subway, it may well be faster to take a cab to the nearest subway station, take the subway across town, and then take another cab from there.

Buses are cheap, crowded, and Chinese. The driver does not speak English.

Parks and major destinations like the Forbidden City and the Temple of Heaven are walled. You must enter them at exactly the right place or you’ll have a long walk around.

6. Crime

There isn’t any, at least not of the low-level street crime that would inconvenience tourists. Hutongs are safe, even if they look otherwise.

7. Making Hello

The Asian girl approaching you with a camera wants to take her picture with you. They call this “making hello”. This is not a scam. Caucasians and other non-Orientals are still uncommon enough in most Beijing neighborhoods (with the possible exception of Sanlitun) that the locals are curious.

8. Nature

There’s very little green space within Beijing or close to it, compared to New York or many other Western cities. However, you can find some birds, trees, and animals at The Temple of Heaven Park, the Old and New Summer Palaces, and the Beijing Zoo.

The most common birds in the city are Rock Pigeon, European Tree Sparrow (one of the infamous 4 pests–they’re still here. Mao is gone.) and Black-billed Magpie.

9. Shopping

Everything is cheap, cheap, cheap except for electronics which cost about the same in China as they do in the U.S. Clothing and food is especially cheap. Prices are usually well marked, and if they’re not, chances are pretty damn good you can afford it anyway. Outside the tourist shops, merchants seem honest and no one is trying to rip you off. Inside the tourist shops/areas, it’s a different story; but prices here are very much negotiable, even on small items like a soda.

The malls and a few tourist areas like the Great Wall can be somewhat pressured. Just keep saying Boo-yao, smile, and walk on. However the smaller shops in the less touristy areas are genuinely interested in seeing you; and welcome any opportunity to interact, language difficulties notwithstanding. If necessary, draw pictures and point.

10. Leaving town

Do not take any liquids, gels, pastes, waxes, soap, or makeup of any kind in your carry-on luggage when leaving China. Not even in 3 oz bottles in a quart sized ziploc bag. They will be confiscated.

Don’t get to the airport more than 3.5 hours before your flight leaves. You can’t check in.

Airport prices return to U.S. levels. This can be a bit of a shock after enjoying the bargain that is most of Beijing.

Upgraded to Wordpress 2.6

2008-08-03T02:02:39Z

I’ve upgraded this site to WordPress 2.6. Please holler if you notice any problems. Thanks.

Chapter 3: Well-formedness

2008-07-14T13:36:50Z

Here’s part 15 of the ongoing serialization of Refactoring HTML, also available from Amazon and Safari.

The very first step in moving markup into modern form is to make it well-formed. Well-formedness is the basis of the huge and incredibly powerful XML tool chain. Well-formedness guarantees a single unique tree structure for the document that can be operated on by the DOM, thus making it the basis of reliable, cross-browser JavaScript. The very first thing you need to do is make your pages well-formed.

Validity, although important, is not nearly as crucial as well-formedness. There are often good reasons to compromise on validity. In fact, I often deliberately publish invalid pages. If I need an element the DTD doesn’t allow, I put it in. It won’t hurt anything because browsers ignore elements they don’t understand. If I have a blockquote that contains raw text but no elements, no great harm is done. If I use an HTML 5 element such as m that Opera recognizes and other browsers don’t, those other browsers will just ignore it. However, if the page is malformed, the consequences are much more severe.

First, I won’t be able to use any XML tools, such as XSLT or SAX, to process the page. Indeed, almost the only thing I can do with it is view it in a browser. It is very hard to do any reliable automated processing or testing with a malformed page.

Second, browser display becomes much more unpredictable. Different browsers fill in the missing pieces and correct the mistakes of malformed pages in different ways. Writing cross-platform JavaScript or CSS is hard enough without worrying about what tree each browser will construct from ambiguous HTML. Making the page well-formed makes it a lot more likely that I can make it behave as I like across a wide range of browsers.

What Is Well-formedness?

Well-formedness is a concept that comes from XML. Technically, it means that a document adheres to certain rigid constraints, such as every start-tag has a matching end-tag, elements must begin and end in the same parent element, and every entity reference is defined.

Classic HTML is based on SGML, which allows a lot more leeway than does XML. For example, in HTML and SGML, it’s perfectly OK to have a <br> or <li> tag with no corresponding </br> and </li> tags. However, this is no longer allowed in a well-formed document.

Well-formedness ensures that every conforming processor treats the document in the same way at a low level. For example, consider this malformed fragment:

<p>The quick <strong>brown fox</p>
jumped over the
<p>lazy</strong> dog.</p>

The strong element begins in one paragraph and ends in the next. Different browsers can and do build different internal representations of this text. For example, Firefox and Safari fill in the missing start- and end-tags (including those between the paragraphs). In essence, they treat the preceding fragment as equivalent to this markup:

<p>The quick <strong>brown fox</strong></p>
<strong>jumped over the </strong>
<p><strong>lazy</strong> dog.</p>

This creates the tree shown in Figure 3.1.

Figure 3.1: An overlapping tree as interpreted by Firefox and Safari

By contrast, Opera places the second p element inside the strong element which is inside the first p element. In essence the Opera DOM treats the fragment as equivalent to this markup:

<p>The quick
<strong>brown fox jumped over the
<p>lazy dog.</p>
</strong>
</p>

This builds the tree shown in Figure 3.2.

Figure 3.2: An overlapping tree as interpreted by Opera

If you’ve ever struggled with writing JavaScript code that works the same across browsers, you know how annoying these cross-browser idiosyncrasies can be.

By contrast, a well-formed document removes the ambiguity by requiring all the end-tags to be filled in and all the elements to have a single unique parent. Here is the well-formed markup corresponding to the preceding code:

<p>…foo<strong>…</strong></p> <p><strong>…bar</strong> </p>

This leaves no room for browser interpretation. All modern browsers build the same tree structure from this well-formed markup. They may still differ in which methods they provide in their respective DOMs, and in other aspects of behavior, but at least they can agree on what’s in the HTML document. That’s a huge step forward.

Anything that operates on an HTML document, be it a browser, a CSS stylesheet, an XSL transformation, a JavaScript program, or something else, will have an easier time working with a well-formed document than the malformed alternative. For many use cases such as XSLT, this may be critical. An XSLT processor will simply refuse to operate on malformed input. You must make the document well-formed before you can apply an XSLT stylesheet to it.

Most web sites will need to make at least some and possibly all of the following fixes to become well-formed.

Every start-tag must have a matching end-tag.
Empty elements should use the empty-element tag syntax.
Every attribute must have a value.
Every attribute value must be quoted.
Every raw ampersand must be escaped as &.
Every raw less-than sign must be escaped as <.
There must be a single root element.
Every nonpredefined entity reference must be declared in the DTD.

In addition, namespace well-formedness requires that you add an xmlns="http://www.w3.org/1999/xhtml" attribute to the root html element.

Although it’s easy to find and fix some of these problems manually, you’re unlikely to catch all of them without help. As discussed in the preceding chapter, you can use xmllint or other validators to check for well-formedness. For example:

$ xmllint --noout --loaddtd http://www.aw.com
http://www.aw-bc.com/:118: parser error : Specification
mandate value for attribute nowrap
<TD class="headerBg" bgcolor="#004F99" nowrap align="left">
^
http://www.aw-bc.com/:118: parser error : attributes construct error
<TD class="headerBg" bgcolor="#004F99" nowrap align="left">
^
http://www.aw-bc.com/:118: parser error : Couldn't find end
of Start-tag TD line 118
<TD class="headerBg" bgcolor="#004F99" nowrap align="left">
^
…

TagSoup or Tidy can handle many of the necessary fixes automatically. However, they don’t always guess right, so it pays to at least spot-check some of the problems manually before fixing them. Usually it’s simplest to fix as many broad classes of errors as possible. Then run xmllint again to see what you’ve missed.

The following sections discuss the mechanics and trade-offs of each of these changes, as they usually apply in HTML.

XSLT

2008-07-14T13:38:14Z

Here’s part 14 of the ongoing serialization of Refactoring HTML, also available from Amazon and Safari.

XSLT (Extensible Stylesheet Language Transformations) is one of many XML tools that work well on HTML documents once they have first been converted into well-formed XHTML. In fact, it is one of my favorite such tools, and the first thing I turn to for many tasks. For instance, I use it to automatically generate a lot of content, such as RSS and Atom feeds, by screen-scraping my HTML pages. Indeed, the possibility of using XSLT on my documents is one of my main reasons for refactoring documents into well-formed XHTML. XSLT can query documents for things you need to fix and automate some of the fixes.

When refactoring XHTML with XSLT, you usually leave more alone than you change. Thus, most refactoring stylesheets start with the identity transformation shown in Listing 2.9.

Listing 2.9: The Identity Transformation in XSLT

<xsl:stylesheet xmlns:xsl='http://www.w3.org/1999/XSL/Transform'
version='1.0'>
  xmlns:html='http://www.w3.org/1999/xhtml'
  xmlns='http://www.w3.org/1999/xhtml'
  exclude-result-prefixes='html'>

  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

This merely copies the entire document from the input to the output. You then modify this basic stylesheet with a few extra rules to make the changes you desire. For example, suppose you want to change all the deprecated <i> elements to <em> elements. You would add this rule to the stylesheet:

<xsl:template match='html:i'>
  <em>
    <xsl:apply-templates select="@*|node()"/>
  </em>
</xsl:template>

Notice that the XPath expression in the match attribute must use a namespace prefix, even though the element it’s matching uses the default namespace. This is a common source of confusion when transforming XHTML documents. You always have to assign the XHTML namespace a prefix when you’re using it in an XPath expression.

Note

Several good introductions to XSLT are available in print and on the Web. First, I’ll recommend two I’ve written myself. Chapter 15 of The XML 1.1 Bible (Wiley, 2003) covers XSLT in depth, and is available on the Web at http://www.cafeconleche.org/books/bible3/chapters/ch15.html. XML in a Nutshell, 3rd Edition, by Elliotte Harold and W. Scott Means (O’Reilly, 2004), provides a somewhat more concise introduction. Finally, if you want the most comprehensive coverage available, I recommend Michael Kay’s XSLT: Programmer’s Reference (Wrox, 2001) and XSLT 2.0: Programmer’s Reference (Wrox, 2004).

This concludes Chapter 2. I’ll probably post a couple more sections from Chapter 3. Then if you want to see what comes next, you’ll have to buy the book.

TagSoup

2008-06-27T13:19:17Z

Here’s part 13 of the ongoing serialization of Refactoring HTML, also available from Amazon and Safari.

John Cowan’s TagSoup (http://home.ccil.org/~cowan/XML/tagsoup/) is an open source HTML parser written in Java that implements the Simple API for XML, or SAX. Cowan describes TagSoup as “a SAX-compliant parser written in Java that, instead of parsing well-formed or valid XML, parses HTML as it is found in the wild: poor, nasty and brutish, though quite often far from short. TagSoup is designed for people who have to process this stuff using some semblance of a rational application design. By providing a SAX interface, it allows standard XML tools to be applied to even the worst HTML.”

TagSoup is not intended as an end-user tool, but it does have a basic command-line interface. It’s also straightforward to hook it up to any number of XML tools that accept input from SAX. Once you’ve done that, feed in HTML, and out will come well-formed XHTML. For example:

$ java -jar tagsoup.jar index.html
<?xml version="1.0" standalone="yes"?>
<html lang="en-US" xmlns="http://www.w3.org/1999/xhtml"><head><title>Java Virtual Machines</title><meta name="description" content="A Growing
list of Java virtual machines and their capabilities">
</meta></head><body bgcolor="#ffffff" text="#000000">

<h1 align="center">Java Virtual Machines</h1>
…

You can improve its output a little bit by adding the --omit-xml-declaration and --nodefaults command-line options:

$ java -jar tagsoup.jar --omit-xml-declaration --nodefaults index.html
<html lang="en-US" xmlns="http://www.w3.org/1999/xhtml"><head><title>Java Virtual Machines</title><meta name="description" content="A Growing
list of Java virtual machines and their capabilities"></meta>
</head><body bgcolor="#ffffff" text="#000000">

<h1 align="center">Java Virtual Machines</h1>
…

This will remove a few pieces that are likely to confuse one browser or another.

You can use the --encoding option to specify the character encoding of the input document. For example, if you know the document is written in Latin-1, ISO 8859-1, you could run it like so:

$ java -jar tagsoup.jar –encoding=ISO-8859-1 index.html

TagSoup’s output is always UTF-8.

Finally, you can use the --files option to write new copies of the input files with the extension .xhtml. Otherwise, TagSoup prints the output on stdout, from where you can redirect it to any convenient location. TagSoup cannot change a file in place, like Tidy can.

However, TagSoup is primarily designed for use as a library. Its output from command-line mode leaves something to be desired compared to Tidy. In particular:

It does not convert presentational markup to CSS.
It does not include a DOCTYPE declaration, which is needed before some browsers will recognize XHTML.
It does include an XML declaration, which needlessly confuses older browsers.
It uses start-tag and end-tag pairs for empty elements such as br and hr, which may confuse some older browsers.

TagSoup does not guarantee absolutely valid XHTML (though it does guarantee well-formedness). There are a few things it cannot handle. Most important, XHTML requires all img elements to have an alt attribute. If the alt attribute is empty, the image is purely presentational and should be ignored by screen readers. If the attribute is not empty, it is used in place of the image by screen readers. TagSoup has no way of knowing whether any given img with an omitted alt attribute is presentational or not, so it does not insert any such attributes. Similarly, TagSoup does not add summaries to tables. You’ll have to do that by hand, and you’ll want to validate after using TagSoup to make sure you catch all these instances.

However, despite these limits, TagSoup does do a huge amount of work for you at very little cost.