Comments on: The Ten Commandments of Unicode http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/ Longer than a blog; shorter than a book Wed, 08 Feb 2012 21:45:25 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Michael Doran http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-397182 Michael Doran Tue, 12 May 2009 20:08:35 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-397182 8. Thou shalt generate all text in Normalization Form C whenever possible. I've tended towards Form D (Canonical Decomposition) as being the more desirable Unicode normalization form. I am curious as to the rationale for recommending Form C (Canonical Decomposition, followed by Canonical Composition). 8. Thou shalt generate all text in Normalization Form C whenever possible.

I’ve tended towards Form D (Canonical Decomposition) as being the more desirable Unicode normalization form. I am curious as to the rationale for recommending Form C (Canonical Decomposition, followed by Canonical Composition).

]]> By: Elliotte Rusty Harold http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-379375 Elliotte Rusty Harold Thu, 02 Apr 2009 12:14:18 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-379375 Not really, I'm afraid. There are a lot of things that could go wrong with that process. Not really, I’m afraid. There are a lot of things that could go wrong with that process.

]]> By: Bennett http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-219864 Bennett Wed, 23 Apr 2008 22:19:14 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-219864 I have used the private use area in string processing code. I wanted to process strings, but not to touch certain special substrings that may be present in the string. So I replaced the special substrings with private use codes according to a translation table, then processed the string, then translated the private use codes back to the special substrings. I assumed here that the string did not originally contain any private use codes. My use of the private use area was completely transient.Does that seem reasonable? I have used the private use area in string processing code. I wanted to process strings, but not to touch certain special substrings that may be present in the string. So I replaced the special substrings with private use codes according to a translation table, then processed the string, then translated the private use codes back to the special substrings. I assumed here that the string did not originally contain any private use codes. My use of the private use area was completely transient.Does that seem reasonable?

]]> By: Elliotte Rusty Harold http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-212626 Elliotte Rusty Harold Wed, 02 Apr 2008 14:12:19 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-212626 All I can say is that in nearly all the cases where I've seen developers use the private use area, it's been a mistake, and caused far more pain than it alleviated. Now that almost all characters in day-to-day use have been encoded, creating your own character codes is rarely the right solution to any problem. What the right solution is, I couldn't tell you without knowing what your problem is. All I can say is that in nearly all the cases where I’ve seen developers use the private use area, it’s been a mistake, and caused far more pain than it alleviated. Now that almost all characters in day-to-day use have been encoded, creating your own character codes is rarely the right solution to any problem. What the right solution is, I couldn’t tell you without knowing what your problem is.

]]> By: SusanJ http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-212621 SusanJ Wed, 02 Apr 2008 13:45:48 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-212621 I'm not sure I understand the prohibition on the Private Use Area. I have an application where I need to create my own character codes. What should I do? I’m not sure I understand the prohibition on the Private Use Area. I have an application where I need to create my own character codes. What should I do?

]]> By: Elliotte Rusty Harold http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-210558 Elliotte Rusty Harold Thu, 27 Mar 2008 15:34:55 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-210558 And that was relevant in 1987. Today, who really cares? Doubling the size of text (and only text), just doesn't matter any more. It's not plain text that's causing network neutrality disputes and filled hard drives. In fact, in many circumstances, including transmission over HTTP, encoding Russian in UTF-8 does not double its size. Even encoding it in UTF-32 wouldn't double its size. HTTP and modern HTTP servers and clients are a lot smarter than that. Chances are pure Russian text is going out across the network in less than one byte per character no matter which encoding you use. And that was relevant in 1987. Today, who really cares? Doubling the size of text (and only text), just doesn’t matter any more. It’s not plain text that’s causing network neutrality disputes and filled hard drives.

In fact, in many circumstances, including transmission over HTTP, encoding Russian in UTF-8 does not double its size. Even encoding it in UTF-32 wouldn’t double its size. HTTP and modern HTTP servers and clients are a lot smarter than that. Chances are pure Russian text is going out across the network in less than one byte per character no matter which encoding you use.

]]> By: Mark Thornton http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-210553 Mark Thornton Thu, 27 Mar 2008 15:26:41 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-210553 UTF-8 roughly doubles the size of Russian text and other scripts which usual have a single byte code page but have all common letters with codes > 127. True, the effect on Chinese is less pronounced. UTF-8 roughly doubles the size of Russian text and other scripts which usual have a single byte code page but have all common letters with codes > 127. True, the effect on Chinese is less pronounced.

]]> By: Elliotte Rusty Harold http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-207986 Elliotte Rusty Harold Thu, 20 Mar 2008 03:55:52 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-207986 That's a common misconception. UTF-8 is perfectly fine for all languages supported in Unicode. In fact, it has a number of very nice properties that make it superior for all scripts. See <a href='http://www-128.ibm.com/developerworks/xml/library/x-utf8/' rel="nofollow">this article</a> for more details. That’s a common misconception. UTF-8 is perfectly fine for all languages supported in Unicode. In fact, it has a number of very nice properties that make it superior for all scripts. See this article for more details.

]]> By: Mark Thornton http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-207752 Mark Thornton Wed, 19 Mar 2008 09:34:09 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-207752 Remember that UTF-8 is not a very nice encoding for non western (e.g. Russian, Chinese, etc) languages. Remember that UTF-8 is not a very nice encoding for non western (e.g. Russian, Chinese, etc) languages.

]]> By: Oren http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/comment-page-1/#comment-206946 Oren Mon, 17 Mar 2008 21:13:06 +0000 http://cafe.elharo.com/programming/the-ten-commandments-of-unicode/#comment-206946 Tim, I never said non-BMP characters can be ignored. There are lots of things in unicode that look like a single "character" (glyph) but are actually a sequence of a base character followed by combining characters. Even using fully composed form eliminates only some of them. So what difference does it make if the sequence is a surroage pair or a combining sequence? In both cases it's something your program should not be messing with it if doesn't understand the nuances. The most it can do safely is to concatenate such strings or maybe split them on well-defined separator characters so you know you will not be splitting a multicharacter sequence in the middle. But that is all most application software really ever does with strings, anyway. Software that really needs to process them as individual codepoints or glyphs is not written very often. Writing such software requires good understanding of unicode concepts beyond this issue, anyway (e.g. the difference between codepoints and glyphs). Tim, I never said non-BMP characters can be ignored. There are lots of things in unicode that look like a single “character” (glyph) but are actually a sequence of a base character followed by combining characters. Even using fully composed form eliminates only some of them. So what difference does it make if the sequence is a surroage pair or a combining sequence? In both cases it’s something your program should not be messing with it if doesn’t understand the nuances. The most it can do safely is to concatenate such strings or maybe split them on well-defined separator characters so you know you will not be splitting a multicharacter sequence in the middle.

But that is all most application software really ever does with strings, anyway. Software that really needs to process them as individual codepoints or glyphs is not written very often. Writing such software requires good understanding of unicode concepts beyond this issue, anyway (e.g. the difference between codepoints and glyphs).

]]>