UTF-8 good, UTF-16 bad

http://benlynn.blogspot.com/2011/02/utf-8-good-utf-16-bad.html

by mattyb • 15 years ago

117 90 15 years ago

benlynn.blogspot.com

wooster• 15 years ago

Better discussion here, IMO: http://research.swtch.com/2010/03/utf-8-bits-bytes-and-benef...

burgerbrain• 15 years ago

I honestly had no idea there were parts of the development community that actually used and preferred UTF-16...

btilly2• 15 years ago

Anyone who programs in Java is using UTF-16. And in my experience very few Java programmers understand that UTF-16 is a variable length encoding.

machrider• 15 years ago

Python, too (it can be compiled to use UTF-32, but normally uses UTF-16 to mean "Unicode").

Confusion2• 15 years ago

Too few programmers know that UTF-8 is a variable length encoding: I've heard plenty assert that in UTF-8, every character takes two bytes, while claiming simultaneously they could encode every possible character in it.

A bit broader: too few programmers understand the difference between a character set and a character encoding.

fedd13• 15 years ago

i guess utf-8 requires more computations while processing strings. if you work with just 2 bytes chars, it (might) work faster

// upvoted all replies, you're right

edsrzf2• 15 years ago

This is a common misconception about UTF-8 vs. UTF-16. You're missing two important facts.

1. Most UTF-8 string operations can operate on a byte at a time. You just have realize that functions like strlen will be telling you byte length instead of character length, and this usually doesn't even matter. (It's still important to know.)

2. UTF-16 is still a variable-width encoding. It was originally intended to be fixed-width, but then the Unicode character set grew too large to be represented in 16 bits.

lysium3• 15 years ago

But UTF-16 is variable encoding, too?

sid0• 15 years ago

Whatever environments jumped on Unicode early, before it was realized that 2 bytes wouldn't be enough, all chose to use UCS-2 for obvious reasons. In particular, that includes Windows and Java.

derleth2• 15 years ago

> all chose to use UCS-2 for obvious reasons

Probably because they figured they could just ignore endianness issues and that ASCII compatibility would be Somebody Else's Problem.

There were always problems with UCS-2. UTF-8 would have had a number of advantages over it even if Unicode had never grown beyond the BMP (Basic Multilingual Plane, the first and lowest-numbered 16-bit code space).

JoachimSchipper• 15 years ago

Well, the Plan 9 guys famously got this right...

program2• 15 years ago

Strings are big-endian UTF-16 by default even in Cocoa (stored in an array of unsigned shorts). Worst of all GCC define the wchar_t as a 4 byte int unless you specify -fshort-wchar.

__david__• 15 years ago

As far as I know, wchar_t is meant to be an internal only representation, so it's good that it is 32 bits--that way you are in one codepoint per word territory. It's a mistake to think you can just overlay some unicode binary data with a wchar_t pointer--you need to convert into and out of wchat_t from utf8/utf-16/whatever. Otherwise you aren't handling codepoints above 16 bits correctly.

getsat2• 15 years ago

The Windows NT development team made the decision to standardise on UTF-16. Every release of Windows since the original NT uses UTF-16 internally for all its "wide character" API calls (e.g., wcslen() and FindWindowW()).

sedachv2• 15 years ago

Windows has only had UTF-16 interfaces since 2000, NT was based around UCS-2.

bzbarsky• 15 years ago

I don't know about "preferring", but anyone manipulating strings in JavaScript is effectively using UTF-16 (or more precisely is using arrays of 16-bit integers which a web browser will interpret as UTF-16-encoded Unicode if you tell it that the array contains text).

As a consequence at least Gecko and Webkit both use UTF-16 for their string classes, though there has been talk of trying to switch Gecko over to UTF-8. The problem then would be implementing the JS string APIs on top of UTF-8 strings efficiently.

fedd13• 15 years ago

and i find it fair that American characters require 2 bytes in Java as everybody else, not 1 as in utf-8! :)

brownleej• 15 years ago

You seem to be confusing "fair" with "equal". Treating everybody the same is not necessarily fair. It seems fair to me to have the most common characters be shortest. I don't have any evidence, but I would guess that Latin characters[1] are used most commonly.

[1] "Latin characters" is the proper term, not "American characters"

fedd13• 15 years ago

thanks! but i just tried to kid which i can't control sometimes.

anyway, i think that even if Latin chars weren't the most used in the world, it would be fair to keep them the primary charset for use in programming and markup languages, as no-one now complains that the international language of medicine is Latin, not, say, Chinese :) as computers started to be massively developed in America.

prodigal_erik• 15 years ago

I keep hoping a string API will catch on in which combining marks are mostly treated as indivisible. Handling text one codepoint at a time is as bad an idea as handling US-ASCII one bit at a time--almost everything it lets you do is an elaborate way to misinterpret or corrupt your data.

fedd13• 15 years ago

did you mean the situation when for example "ä" can be transmitted as 00e4 alone or as 0061 "a" + 0308 "umlaut"?

barrkel3• 15 years ago

It's not so simple: it depends on what you're doing with the text. If you're not trying to do analysis with it, encoded text is more or less a program written in a DSL that, when interpreted by a font renderer, draws symbols in some graphical context. Depending on the analysis you want to do, you need varying amounts of knowledge. Perhaps you only need to know about word boundaries; perhaps you're trying to look things up in a normalized dictionary; maybe even decompose a word into phonemes to try and pronounce it. These require different levels of analysis, and one size won't fit all.

fedd13• 15 years ago

look what i found and now plan to use

http://download.oracle.com/javase/6/docs/api/java/text/Norma...

update: for particular purposes consider using Collator class, it makes collation keys (byte arrays) out of strings applying locale, case sensitiveness and unicode decomposition. (at least so says the doc, http://download.oracle.com/javase/6/docs/api/java/text/Colla... )

chalst2• 15 years ago

Broad backwards compatibility with ASCII is a strong reason to prefer UTF-8 in most applications, however I find the issues with UTF-16 are overstated and the advantages of it ignored.

"UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages.

"A related drawback is the loss of self-synchronization at the byte level." - Maybe this is a problem, maybe not. Maybe the failure of UTF-8 to be self-synchronising at the 4-bit level is a problem is some circumstances. I don't mean to be flippant, but the wider point is that with UTF-16, you really need to commit to 16-bit char width.

"The encoding is inflexible" - I think the author has confused the fixed-width UCS-2 and the variable-width UTF-16.

"We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width.

yuhong2• 15 years ago

>"We lose backwards compatibility with code treating NUL as a string terminator." - Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width.

Yea, it really should be "We lose backwards compatibility with code treating a zero 8-bit byte as a string terminator."

>"UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages.

For pure CJK text files, yes.

udoprog• 15 years ago

"...Unicode has been standardising CJK character sets" - CJK characters sets are complex, and will always face opposition, besides, this is less related to how the characters are encoded, and more how to choose and composite your code points.

"I think the author has confused the fixed-length UCS-2 and the variable length UTF-16." - No, in the same sentence you are quoting he mentions that utf-16 is limited to 0x110000 codepoints, contra utf-8 that is specified to expand up to 6 bytes.

"Not true. NUL is a 16-bit character in UTF-16. Use a C compiler that supports 16-bit char width." - I won't be converting my source code into utf-16 any time soon. Besides, C is not a good example where this is an actual problem, the "unicode strings" will still just be represented as binary blobs - different depending on which encoding you choose. It's more of a problem in e.g. Python, where the native string is an array of characters, not just a null terminated blob. I wouldn't encode my source code in utf-8, however, if my editor my default happened to support utf-8, this wouldn't be an issue unless I enter an non-ascii compatible code point. I see this as a big plus.

chalst2• 15 years ago

"confused ... UCS-2 and ... UTF-16" - You are quite right, he is clear on the difference. It's presumably possible to apply the same trick to UTF-16 as was applied to UCS-2, to expand further, but this is ad-hoc and not built in.

"supports 16-bit char width" - I mean the compiler has CHAR_BIT == 16; this is independent of the character encoding of the source.

othermaciej• 15 years ago

> "UTF-16 has no practical advantages over UTF-8" - UTF-16 is better - i.e., quite a bit more compact, since, e.g., Chinese characters always take 3 characters in UTF-8, for asian languages.

In some contexts that may matter. In other cases, you can expect enough ASCII mixed in to outweigh that effect. For example, on the Web though, if you're sending HTML, the savings from using 8 bits for the ASCII tags will nearly outweigh the cost of extra bytes for content text. Gzip shrinks that further to essentially no difference.

fnl8• 15 years ago

As I posted there, too, he also missed one very important plus in UTF-16: character counting. If you ensure there are no Surrogate Chars (which is a simple byte-mask comparison using every second byte (only!) as in: "0xD800 & byte == 0xD800" [using the entire range to avoid having to think about byte order]) in your array, you can be 100% sure about the character length just by dividing the byte-length in two. With UTF-8, you have to look at each and every byte one by one and count the number of characters to establish the actual char-length. Finally, given how rare the Supplementary Range characters are, the Surrogate Range check can be safely skipped in many scenarios, making it significantly faster to establish char lengths in UTF-16 than UTF-8.

EDIT: oh, and before any "semantic-nerd" comes along: I am fully aware that 0xXXXX are two bytes, so, if you want, read "two-byte" for every time I mention "byte" above... (doh ;))

lysium3• 15 years ago

You don't mention issues and your advantages are only marginal for specific use cases such as Asian languages.

"UTF-16 is better": but 'only' for Asian languages; most code is still written in ASCII

"need to commit to 16-bit char width": the OP's point was that many many tools don't operate on 16-bit char width and with UTF-8 you can continue to use them.

The other two points are no advantages for UTF-16.

natmaster• 15 years ago

Can someone link me to where python uses UTF-16? It was my understanding it defaulted to UTF-8.

http://www.python.org/dev/peps/pep-3120/

lysium3• 15 years ago

That PEP only refers to the encoding of the source (code) file, not to the encoding on how strings are stored in Python.

Edit: Your question might be answered at SO: http://stackoverflow.com/questions/1838170/what-is-internal-...

fedd13• 15 years ago

sorry for the quastion as the reply, i would also ask for some info about the unicode issues in jython, as i lack experience with all python stuff. does it have problems or everything is transparent?

masklinn• 15 years ago

> It was my understanding it defaulted to UTF-8.

For external IO, internally it's in UTF-16 (actually UCS-2, mostly unaware of surrogates) or in UCS-4 (via a compile-time switch).

See http://docs.python.org/c-api/unicode.html#Py_UNICODE for the Py_UNICODE API.

fnl8• 15 years ago

check PEP-261

adobriyan• 15 years ago

> The Go designers knew what they were doing, so they chose UTF-8.

The authors of Go and authors of UTF-8 are more or less the same people, so the choice was no-brainer.

thristian2• 15 years ago

The article, at the end, claims that "ASCII was developed from telegraph codes by a committee." It turns out the story is much, much more complicated and interesting than that: http://www.wps.com/projects/codes/

fedd13• 15 years ago

may i dare make a conclusion with my observations? :)

utf-8 is good for network interchange and is de facto becoming standard.

utf-16 is not bad for internal storage of strings in memory or in database. not nesserily bad. maybe even better for some reasons

zbowling• 15 years ago

UCS2, even though being an outdated predecessor to UTF-16, has some unique qualities that make it useful for things like databases or other storage mediums that you are not mixing with a lot of low code point characters (like you do with XML and HTML markups).

One being that's fair for all languages with respect to size so when you may be storing your standard Chinese, Korean, Japanese characters.

When UTF-16 made UC2 variable length, a few of the nice things were lost, but when dealing a lot of the higher code point characters mostly, UTF-16 may save you space.

wildmXranat• 15 years ago

Correct me if I'm wrong, but I think Excel still outputs UTF-16 in some cases. I remember parsing generated .txt/.csv files and there were issues with it and it's endian order.

mikecaron• 15 years ago

Good read! I always HATE it when things complain that my visual studio .c/.h files are BINARY! WTF!

UTF-8 good, UTF-16 bad

Related Stories