UTF-8 Everywhere

http://www.utf8everywhere.org/

by angersock • 13 years ago

275 149 13 years ago

angersock5• 13 years ago

What is currently the best way of dealing with UTF-8 strings in a cross-platform manner? It sounds like widechars and std::string just won't cut it.

90minuteAPI2• 13 years ago

Yeah, you really need some specialized interface that encapsulates all the things you may need from your string processing, but can spit out a representation of that string in various encodings.

In line with the spirit of this article, that interface should use UTF-8 storage internally as well, but this should be transparent to the programmer anyway. Dealing with encoded strings directly is a recipe for heartache unless you're actually writing such a library.

lmm3• 13 years ago

A higher level language, honestly. Perl, Ruby, Python3 and Haskell have excellent crossplatform utf8 support, and I'd be amazed if OCaml didn't. But if you want to write C++ code that works on windows, you're in for some pain.

belluchan• 13 years ago

And software developers, don't forget to implement the 4 byte characters too please. Utter nightmare dealing with MySQL. I believe 4 byte characters still even break github comments.

90minuteAPI2• 13 years ago

About 2 weeks I tried to file a bug with our backend guys about 4-byte characters wreaking havoc on our API.

My example broke the bug tracker's (bugzilla) comment system as well. I chuckled.

jeltz2• 13 years ago

I assume Bugzilla broke due to being backed by MySQL. Bugzilla itself is written in Perl so should have no problem.

lmm3• 13 years ago

At my last job I checked in a test case for our astral character handling. And broke the build server.

MBCook2• 13 years ago

Yeah. We've noticed, to our own amusement, that Jira (we're on an older version) can't handle non-ASCII. Makes entering tickets involving other languages fun.

ggreer• 13 years ago

I ran into that issue with MySQL's utf-8 handling. It was 𝒜wesome: http://geoff.greer.fm/2012/08/12/character-encoding-bugs-are...

guard-of-terra3• 13 years ago

MySQL was a Norwegian company at the beginning, but you won't believe. It's one of the worst products when it comes to I18N and especially to Unicode. They still kind of store unicode text as a chain of encoded utf8 characters, if I'm not wrong. Their stupid command line utility still defaults to LATIN1 for input and output despite my locale clearly saying .UTF-8. People in their IRC channel still refuse to admit anything of this as a problem.

jeltz2• 13 years ago

The horrible unicode support in MySQL fits well with its general careless attitude toward data integrity.

Minor correction: MySQL was Swedish and InnoDB Finnish.

jasonjei3• 13 years ago

We constantly have to deal with Win32 as a build platform and we write our apps natively for that platform using wchar. I think the main difficulty is that most developers hate adding another library to their stack, and to make matters worse, displaying this text in Windows GUI would require conversion to wchar. That's why I think they are up for a lot of resistance, at least in the Windows world. If the Windows APIs were friendlier to UTF-8, there might be hope. But as it stands right now, using UTF-8 requires the CA2W/CW2A macros, which is just a lot of dancing to keep your strings in UTF-8 which ultimately must be rendered in wchar/UTF-16.

Maybe there might be a shot in getting developers to switch if Windows GUIs/native API would render Unicode text presented in UTF-8. But right now, it's back to encoding/decoding.

jbk• 13 years ago

This resonate so much for me, in VLC.

VLC has a very large number of users on Windows (80% of our users), yet almost none of the dev use Windows to code. Therefore, we use UTF-8 char* everywhere, notably in the core. We use UTF-16 conversions only in the necessary Windows modules, that use Windows APIs. Being sure we were UTF-8 everywhere took a lot of time, tbh...

But the worse are formats like ASF (WMV) or MMS that use UTF-16/UCS-2 (without correctly specifying) them and that we need to support on all other platforms, like OSX or Linux...

lelandbatey• 13 years ago

Wow, I've learned even more about VLC (and programming in general) and I'm even more impressed. Thanks for this little insight!

username422• 13 years ago

I am also convinced that wide char should be avoided and are one of the bad thing of java. It is very nice to see that more and more people agree and that important projects like VLC already apply this principle. Will 2014 become the year we kill wide chars ?

randomfool2• 13 years ago

"This is what made UTF-8 the favorite choice in the Web world, where English HTML/XML tags are intermixed with any-language text."

Except that Javascript is UTF-16, so no luck with 4 byte chars there.

lisper• 13 years ago

Javascript is UTF-16

No it isn't. Javascript is no different from any other text. It can be encoded in any encoding. Where did you get the idea that JS is UTF-16?

EDIT: I misunderstood the intent of the comment I was responding to. JS uses (unbeknownst to me) UTF-16 as its internal representation of strings.

thurn• 13 years ago

GP means string literals. To quote from the spec: "4.3.16 String value: primitive value that is a finite ordered sequence of zero or more 16-bit unsigned integer... Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text."

vetinari• 13 years ago

The idea is from ECMA 262, sections 2(conformance), 4.3.16 (String value), 6 (Source text), 8.4 (String type)... That's basically THE reason, why all js engines are UTF-16 internally.

simcop2387• 13 years ago

Have a look at this stack overflow question[1]. Javascript/ECMAScript strings are supposed to be UTF-16. That said UTF-16 encodes 4 byte codepoints just as easily as UTF-8.

http://stackoverflow.com/questions/8715980/javascript-string...

ender72• 13 years ago

JavaScript source can be encoded in any way that the browser can handle, yes.

Within the JS language, strings are represented as sort-of-UCS-2-sort-of-UTF-16 [0]. This is one of the few problems with JS that I think merits a backwards-compatibility-breaking change.

[0] http://mathiasbynens.be/notes/javascript-encoding

andystanton• 13 years ago

"Even though one can argue that source codes of programs, web pages and XML files, OS file names and other computer-to-computer text interfaces should never have existed, as long as they do exist, text is not only for human readers."

I'm a little confused by this statement. Can someone clarify?

claudius• 13 years ago

I think the author wants to say that a) computers should use appropriate binary formats for communication between them (e.g. from a web server to a browser), b) they don’t and use plain text in many cases (e.g. HTML) and hence c) text is not only read by humans but also by computers.

I am not entirely sure whether that makes any sense, though.

asgard10244• 13 years ago

This may be tangential, but I think that computer languages should have a different type (and literal notation) for human text (strings that may be read by human, may be translated, won't affect program semantics) and for computer string (strings that are strictly defined, not to be translated, and may affect program semantics).

Then we could put all the human language problems into human text type, and leave the simpler computer string type with easier semantics.

In Python, although there are no tools for that, I typically use the following convention: single quotes for computer text and double quotes for human text. I guess you could use byte arrays for computer text as well, but it would be more painful.

bcbrown2• 13 years ago

In traditional Windows development, the way that was handled was by having human-strings in a resource file (which you could replace with a file in a different language), while the computer-strings were hardcoded constants.

anonymfus• 13 years ago

Traditional? The same is recommended for .net/WinRT development.

tel2• 13 years ago

Haskell has the different semantics though they share the literal type. In Haskell there's Text for Unicode strings (UTF-16 encoded) and ByteString for, well, byte-strings (fixed vectors of Word8s). The semantics make it hard to confuse them, though if you turn on the OverloadedStrings extension then

    "foo"

is ambiguous. If you don't like that you can just not turn on that extension and then there is no literal syntax for either type and you must convert literal strings (really just lists of characters, [Char]) to either type manually

    Data.Text.pack "foo" :: Text
    Data.ByteString.pack "foo" :: ByteString

Finally, you must use manual encoding functions to interconvert them

    Data.Text.Encoding.encodeUtf8 :: Text -> ByteString

fulafel• 13 years ago

Python has separate bytes and unicode types as well, I think he meant there should be a "black box" type that the program shouldn't try to interpret.

tolmasky• 13 years ago

Does Ruby's Symbol type acceptably accomplish this?

jerf• 13 years ago

Symbols are yet another use case; "I want something as efficient as a small number" (probably 4 bytes nowadays) "but I want to type a nice human name for it, and I don't want to even have to think about the translation". It's neither "text" nor a "binary", it's actually a well-disguised int (or similar type).

asolove• 13 years ago

But note that ruby's symbols are interned, so if you create lots of them programatically they never get GC'd.

asgard10244• 13 years ago

For some use cases, symbol types are useful (note that they originally come from Lisp). But not all. For example, in many internet protocols, you need computer strings which are not symbols.

mekoka• 13 years ago

I thought that was the purpose of having binary strings and unicode strings in Python. That's how I've been using them anyhow. Unicode for anything whose text property is relevant (u"this is text") and binary as a simple container of bytes ("bf9@cf0*1!09v$j#x0j").

asgard10244• 13 years ago

Yes, I think that's a good start, although the literal notation for the computer strings b".." is a bit unwieldy. They also don't coerce when used together, see my comment above.

bjourne3• 13 years ago

There is no reason "computer text" couldn't also span the whole unicode character set. I don't think your comment is tangential. But I think the notion that some characters are special or more basic than other characters is a trap and leads to illogical thought. For example, many internet protocols are defined in that way. Ascii only for all commands and responses and then for i18n there are some crazy encoding schemes used (quoted printable, utf7, base64, etc) to encode unicode as ascii.

All that goes away if your protocol is standardized on utf8. Then text is text and bytes is bytes.

anon1385• 13 years ago

Lots of operations people want to do on 'strings' don't make sense on unicode text. There is no 'length'. There is the size in bytes of various encodings and there is the number of code points and there is the number of grapheme clusters. The latter is often what people really want, but to get that you need to know about the fonts being used to render the text.

Similarly people want to iterate over a string character by character or take substrings by range but with unicode text that becomes iteration over code points and ranges of code points (unless you go all the way and use a text rendering system to give you grapheme clusters). Code points can be decomposed diacritic marks etc so you can't just blindly insert or change code points at a certain index or take arbitrary substrings without risking breaking the string (you can end up with accents on characters that you didn't intend, or stranded at the end of a string and probably plenty of other types of breakage that I can't even think of). Functionality exists to deal with all this but it's pretty burdensome (e.g. NSString has -rangeOfComposedCharacterSequencesForRange:).

That all adds up to a pretty hefty performance penalty as well as potential layering violations (needing to consider fonts and rendering when parsing some protocol if you really are going to treat strings as a sequence of grapheme clusters).

asgard10244• 13 years ago

I actually would prefer the computer string type to be array of bytes. Many people mentioned this type distinction already exists in many languages (Python, Haskell..).

Though I think it would be useful to think about it as a sort of subtype of human string, with default encoding in UTF-8. So substitution or concatenation of human and computer string would yield a human string (which is where these languages usually fail short, because you need explicit conversion, it doesn't work like e.g. integers and floats).

cynwoody2• 13 years ago

In Go, you can name your variables in Chinese† if you want.

In Go, strings are immutable UTF-8 byte arrays,†† and the language provides facilities for iterating over them either byte by byte or rune by rune (a rune is an int32, wide enough to hold any unicode character).

†http://play.golang.org/p/skyEYhxWg3

††http://blog.golang.org/strings

dbaupp• 13 years ago

Does Go provide functions for iterating grapheme by grapheme?

dsymonds• 13 years ago

A Go string is an immutable opaque sequence of bytes. It is conventionally UTF-8, but it does not have to be. The only place in the language that cares is if you range over a string, but even there non-UTF-8 strings are supported.

Grue3• 13 years ago

Can you have spaces or other reserved characters in variable names? Because in Lisp you can. In addition to being able to use characters from any encoding of course.

Pxtl2• 13 years ago

I was horrified to discover that Microsoft SQL Server's text import/export tools don't even support UTF-8. Like, at all. You can either use their bastardized wrongendian pseudo-UTF-16, or just pick a code-page and go pure 8-bit.

josteink4• 13 years ago

I'm not sure what modules or tools you are talking about, but if you use Sql Server Integration Services (formerly SQL Servrer Data Transformation Services), you basically have a data processing pipeline which supports everything and all transformations on the planet.

And obviously it supports arbitrary text-encodings, although sometimes you will need to be explicit about it.

If you used the simplified wizards, all the options may not have been there, but you should have been given the option to export/save the job as a package, and then you can open, modify, test and debug that before running the job for real.

Seriously. SQL Server has some immensely kick-ass and über-capable tooling compared to pretty much every other database out there.

To even suggest it doesn't support UTF8 is ludicrous.

Pxtl2• 13 years ago

The various command-line and text-friendly bulk/bcp tools are the pain-point here.

SSIS is insanely powerful and performant, but it's also insanely cumbersome and script-unfriendly. Microsoft has finally started embracing the power of plain text and scriptable tools in their web-stack and .NET, but SSIS represents a holdover from their heavyweight GUI-and-wizard days.

redstripe• 13 years ago

He's probably talking about "bcp" which indeed doesn't support utf-8.

So why would someone even use bcp instead of SSIS? SSIS might be nice for performing repeated imports of data that has a fixed format, but for quick and dirty exports/imports it's really frustrating to use. It's not even smart enough to scan an entire data file and suggest appropriate field lengths and formats. Every single time I try to import a .csv file it craps out and doesn't even show where the error occured - that's after clicking through a bunch of steps in a GUI. At least with BCP you can easily rerun the import/export from the command line.

SQL and SQL Management studio are generally great but I would not include SSIS when lauding them.

Someone3• 13 years ago

If SQL server supports UTF8, Microsoft manages to hide that fact well. http://technet.microsoft.com/en-us/library/ms176089.aspx:

char [ ( n ) ] Fixed-length, non-Unicode string data.

http://technet.microsoft.com/en-us/library/ms186939.aspx

Character data types that are either fixed-length, nchar, or variable-length, nvarchar, Unicode data and use the UNICODE UCS-2 character set.

So, (var)char is "non-Unicode", and n(var)char is UCS-2 only.

That is in agreement with http://blogs.msdn.com/b/qingsongyao/archive/2009/04/10/sql-s..., which claims the glass is half full ("In summary, SQL Server DOES support storing all Unicode characters; although it has its own limitation.")

On the other hand, we have http://msdn.microsoft.com/en-us/library/ms143726.aspx that seems to state that SQL Server 2012 has proper unicode collations. UTF8 still is nowhere to be found, though.

Dewie4• 13 years ago

IT is so Anglophile that programs can become slower if you deviate from ASCII...

But of course being so incredibly anglocentric is not an issue, at least that seems to be the consensus of the participants when I read discussions on the Web where all the people who are discussing it write English with such a proficiency that I can't tell who are and aren't native speakers of the language.

cube13• 13 years ago

IT is so Anglophile that programs can become slower if you deviate from ASCII...

Well, switching from a simple, 7-bit character set where all you need to do is parse and display byte by byte to a potentially multi-byte processing and a display lookup table that's several orders of magnitude larger... it's pretty easy to see why things can become slower.

jasonjei3• 13 years ago

I'm Chinese American, and I don't agree with your statement that string libraries are Anglophile. How would you encapsulate the 10,000 commonly used Chinese characters? It's just the reality of having a lot of characters in a language. Not much else you can do to speed up processing. How would you design string storage to be faster for a language like Chinese?

English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.

Dewie4• 13 years ago

I won't pretend to have a solution. I guess you would like to have some compression scheme, since I'm guessing it would save space over having 2-4 bytes (however many there are in the Chinese language) for every character. You won't gain much compared to having an English-centric encoding scheme, I guess, in the case of a language with a large amount of characters.

But it's still funny to me how even the computer who speaks in 1's and 0's favours English-centered notation.

English happens to be the lingua franca of Engineering. It's not about brown nosing English-speaking countries, but about getting the widest range of audience.

Pragmatism ũber alles, chants the American. I guess I'm not impressed by the support of non-English languages in IT.

I have for that matter met engineering students who don't seem to speak a lick of English, maybe even people studying CS/CE.

adsr• 13 years ago

Let me ask you, with 10k commonly used characters doesn't that lead to shorter texts? Kind of like how higher base numbers can encode larger numbers with fewer digits, in that case the longer encoding of UTF-8 could be made up for by using fewer characters. Or am I wrong about this assumption?

As an example, suppose that there are one character that denotes the word 'house', if that single character is encoded using five bytes it takes the same amount of space as the english encoding.

wehadfun• 13 years ago

I admire and appreciate your concern for something that is missunderstood and ignored. However this webpage took way to long to say what is so great about utf 8.

jasonjei3• 13 years ago

Honestly, I think it is platform politics. *nix systems seem to prefer UTF-8, while UTF-16 is the default on Windows. Space and memory are cheap, so either encoding seems fine.

The bottom line is that UTF-8 is awkward to use on Windows, while UTF-16/wchar_t is awkward to use on Linux, simply because the core APIs make them so (there is no _wfopen function in glibc).

hdevalence• 13 years ago

The other problem with UTF-16 is that it's much easier to pretend that 1 element = 1 character than with UTF-8.

MichaelGG• 13 years ago

It's not really politics. Microsoft made the choice for fixed-sized chars back when it was thought that 16 bits was enough for everyone. MS was at the forefront of internationalizing things, and probably still are. (Multilanguage support in Windows and Office is quite top class.)

Unfortunately, we need more than 16 bits of codepoints, so 16-bit chars is a waste and a bad decision with that insight. It seems unlikely that a fresh platform with no legacy requirements would choose a 16-bit encoding. Think of all the XML in Java and .NET - all of it nearly always ASCII, using up double the RAM for zero benefit. It sucks.

Was UTF-8 even around when Microsoft decided on 16-bit widechar?

Other platforms seem to have lucked out by not worrying as much as standardizing on a single charset and UTF8 came in and solved the problems.

pjscott3• 13 years ago

This summary is excellent, and concise:

http://research.swtch.com/utf8

elwell• 13 years ago

I can only imagine what kind of frustration drove someone to make this site.

angersock5• 13 years ago

I'm more than imagining it right now. >:(

SomeCallMeTim2• 13 years ago

The same frustration made me write a custom string library for the Playground SDK, years ago.

std::string is missing a lot of functionality one tends to need when dealing with strings (such as iterating over UTF-8 characters or fast conversion to UTF-16, but also things like search-and-replace). And it makes me sad that I can't use that string library any more (legally) because of the license PlayFirst insisted on using (no redistribution).

As far as I'm concerned, though, there IS no good string library available for use anywhere. I've looked at all of the ones I could find, and they're all broken in some fundamental way. I guess solving the "string problem" isn't sexy enough for someone to release a library that actually hits all the pain points.

angersock5• 13 years ago

Well, looks like that is what it is. Shoot me an email with your pain points, and maybe something will come of it. :)

klmr• 13 years ago

You might like ogonek[1].It’s currently still not implementing regular expressions and thus strongly limited in its capabilities (and C++11 regex are so badly designed that they cannot be extended meaningfully to handle this case), but it has hands down the best API for working with text in C++. It makes using a wrong encoding a compile-time error and offers effortless ways of dealing with actual Unicode entities (code points, grapheme clusters) rather than bytes.

[1] https://github.com/rmartinho/ogonek

BadassFractal• 13 years ago

Would be lovely if MS Office could export CSV to UTF-8, but nope.

huhtenberg4• 13 years ago

Export in UTF-16, open in Notepad and re-save as UTF-8.

frik• 13 years ago

Yes, I had issues with that in Excel 2010 too.

The only well known Microsoft application that can handle UTF-8 is notepad.exe (Win 7).

optimiz3• 13 years ago

Most of the post talks about how Windows made a poor design decision in choosing 16bit characters.

No debate there.

However, advocating "just make windows use UTF8" ignores the monumental engineering challenge and legacy back-compat issues.

In Windows most APIs have FunctionA and FunctionW versions, with FunctionA meaning legacy ASCII/ANSI and FunctionW meaning Unicode. You couldn't really fix this without adding a 3rd version that was truly UTF-8 without breaking lots of apps in subtle ways.

Likely it would also only be available to Windows 9 compatible apps if such a feature shipped.

No dev wanting to make money is going to ship software that only targets Windows 9, so the entire ask is tough to sell.

Still no debate on the theoretical merits of UTF-8 though.

angersock5• 13 years ago

Nothing worth doing is easy.

Anyways, the FunctionA/FunctionW is usually hidden behind a macro anyways (for better or worse). This could simply be yet another compiler option.

gamacodre• 13 years ago

We "solved" (worked around? hacked?) this by creating a set of FunctionU macros and in some cases stubs that wrap all of the Windows entry points we use with incoming and outgoing converters. It's ugly under the hood and a bit slower than it needs to be, but the payoff of the app being able to consistently "think" in UTF-8 has been worth it.

Of course, we had to ditch resource-based string storage anyway for other cross-platform reasons, and were never particularly invested in the "Windows way" of doing things, so it wasn't a big shock to our developers when we made this change.

nabla92• 13 years ago

UTF-8 is usually good enough in disk.

I would like to have at least two options in memory: utf-8 and vector of displayed characters (there's many combinations in use in existing modern languages with no single-character representations in UTF-<anything>).

Guvante2• 13 years ago

Do you need a vector of displayed characters?

Usually all you care about is the rendered size, which your rendering engine should be able to tell you. No need to be able to pick out those characters in most situations.

nabla92• 13 years ago

Yes. If I want to work with language and do some stringology, that's what I want. I might want to swap some characters, find length of words etc. To have vector of characters (as what humans consider characters) is valuable.

huhtenberg4• 13 years ago

Now, all the advice in the Windows section - don't do this, don't do that, only and always do third - is lovely, but if you happen to care about app's performance, you will have to carry wstrings around.

Take a simple example of an app that generates a bunch of logs that need to be displayed to the user. If you are to follow article's recommendations, you'd have these logs generated and stored in UTF8. Then, only when they are about to be displayed on the screen you'd convert them to UTF16. Now, say, you have a custom control that renders log entries. Furthermore, let's imagine a user who sits there and hits PgUp, PgDown, PgUp, PgDown repeatedly.

On every keypress the app will run a bunch of strings through MultiByteToWideChar() to do the conversion (and whatever else fluff that comes with any boost/stl wrappers), feed the result to DrawText() and then discard wstrings, triggering a bunch of heap operation along the way. And you'd better hope latter doesn't cause heap wobble across a defrag threshold.

Is your code as sublime as it gets? Check. Does it look like it's written by over-enlightened purists? You bet. Just look at this "advice" from the page -

  ::SetWindowTextW(widen("string litteral").c_str())

This marvel passes a constant string to widen() to get another constant string to pass to an API call. Just because the code is more kosher without that goddamn awful L prefix. Extra CPU cycles? Bah. A couple of KB added to the .exe due to inlining? Who cares. But would you just look at how zen the code is.

--

tl;dr - keeping as much text as possible in UTF8 in a Windows app is a good idea, but just make sure not to take it to the extremes.

guard-of-terra3• 13 years ago

"if you happen to care about app's performance, you will have to carry wstrings around"

If those strings are for the user to read, he's reading a million times slower than you handle the most ornate reencoding. Sounds like a premature optimization.

jakobe• 13 years ago

Not only that, but the time required to convert from UTF-8 to UTF-16 is negligible in relation to the time required to lay out the glyphs and draw them on screen. Premature optimisation indeed.

eps• 13 years ago

It's not a premature optimization. It's a manifestation of a different set of coding ethics which is just ... err ... less wasteful and generally more thoughtful.

revasm• 13 years ago

That was an unfortunate example. The widen() in this case is absolutely unnecessary. The author even recommends using the L prefix for UTF-16 string literals inside of Windows API calls (but not on other platforms, where wchar_t isn't UTF-16):

Do not use _T("") or L"" literals in any place other than parameters to APIs accepting UTF-16.

Except for that, you do make a good point. It's probably better to store some strings in memory according to the platform requirements, if the program can be shown to exhibit delays caused by string conversions.

anon4• 13 years ago

You do realise that drawing the characters to the screen is an order of magnitude (at least) slower than grabbing some heap memory, doing a string conversion and freeing the memory, right? You don't even need to allocate memory - you can have a constant small thread-local buffer of, say, 1kb that you reuse for these conversions.

josteink4• 13 years ago

Looking at the .NET parts of the manifesto, I just have to roll my eyes:

Both C# and Java offer a 16 bit char type, which is less than a Unicode character, congratulations. The .NET indexer str[i] works in units of the internal representation, hence a leaky abstraction once again. Substring methods will happily return an invalid string, cutting a non-BMP character in parts.

While theoretically true, for most practical purposes, this reeks of a USA/American/English bias and lack of real world experience.

You know what? I want to know that the text "ØÆÅ" is three characters long. I dont want to know that it's a 6-byte array once encoded to UTF8. Anywhere in my code telling me this is 6 characters is lying, not to mention a violation of numerous business-requirements.

When I work with text I want to work with text and never the byte-stream it will eventually be encoded to. I want to work on top of an abstraction which lets me treat text as text.

Yes, their are cases where the abstraction will leak. But those cases are very far and few in between. And in all cases where it doesn't, it offers me numerous advantages over the PHPesque, amateurish and incorrect approach of treating everything as a dumb byte-array.

It's not. It's text in my program. It's text rendered on your screen. It's just a byte-array when we send it over the wire, so stop trying to pretend text isn't text.

This manifesto is wildly misguided.

derefr3• 13 years ago

I fundamentally agree, but I don't think even you understand the implications of what you're saying. Though you say "characters", You're probably still thinking about Unicode codepoints here, not characters. A real character-oriented API would require an arbitrary number of bytes to store each character (a character being an arbitrary vector of codepoints), and would require a font to be associated to each region of codepoints--because it's the font's decision of how to render a sequence of codepoints, so it's the font that determines how many characters you get. (For an edge-case example: http://symbolset.com/)

...and saying that, it'd still be a good idea.

chipsy• 13 years ago

When I learned of the difficulty mapping between code points and characters, I realized that Unicode is a standard nobody will ever (knowingly) implement correctly. Even if everyone has access to a font API, there'll probably be bugs in the fonts for all eternity.

TheLoneWolfling• 13 years ago

Yes. I've got a toy language that I've been working on that uses a rope to store characters internally.

Everything works on "logical characters" - arbitrary vectors of codepoints, as you say. There's still a number of edge cases I have yet to work out as to what exactly is considered a character, though. (I just added support for a single code point encoding multiple characters, for example.)

I'm not so sure that making thing reliant on a font would be the best way to solve that, though. I'd intuitively say that there should be less coupling between rendering choices and internal encoding than that.

The twist is that each node in the rope can only store characters of the same physical length in bytes (and same number of logical characters per physical character). This means that in the typical case (most characters require the same number of bytes to encode) it doesn't add too too much overhead. Still not something I would consider as the base String type for a lower-level language, though.

There are a few simple optimizations that I have yet to do (encode smaller characters as what would be ordinarily be invalid longer encodings, if it makes sense (a single one-byte character in the middle of a bunch of two-byte characters, for example), that sort of thing.)

It seems to work fairly well, so far. Or at least it tends to give "common-sensical" results, and avoids a large chunk of worst-case behavior that standard "prettified character array" strings have.

bjourne3• 13 years ago

Well, he doesn't say that one should use byte count instead of code point count as string length. That's a strawman and not in the manifesto. What he is saying though is that one should prefer to count grapheme clusters instead of characters since the definition of the latter is incredibly vague.

drbawb• 13 years ago

They're not arguing that the length [of the bytestream] is what the user cares most about. I thought they covered this quite well in the "Myths" section on "counting coded characters or codepoints is important."[0]

From that section they explain that for text manipulation [e.g: cursor position & manipulation of text under the cursor] the programmer should be counting grapheme clusters; whereas for storage [memory & disk] concerns the programmer _should_ care about the number of codepoints.

They go on to say that counting the number of characters is up to the rendering engine and is completely unrelated to the number of codepoints.

I don't think the _manifesto itself_ is wildly misguided. In that section they were merely pointing out _how .NET and Java currently report string length._

At least as I read the manifesto: they seem to believe that counting codepoints is useful _but orthogonal_ to counting characters.

[0] http://www.utf8everywhere.org/#myth.strlen

golergka• 13 years ago

I don't understand how what you said contradicts the points in the manifesto. C# strings _try_ to look like they show you the characters and not bytes, but fail in that regard.

mikeash2• 13 years ago

In what scenario, exactly, do you want to know that the text "ØÆÅ" is three characters long? That's almost never a useful question to ask or a useful answer to have.

derefr3• 13 years ago

Lexicographic ordering and sorting; edit distance; spelling correction (really, anything involving prefixes/tries)...

szabba• 13 years ago

When writing wc?

marssaxman• 13 years ago

Quite the contrary: if you are treating text as anything but a dumb byte-array, you are probably doing it wrong. Unicode elements are not characters, but code points. Your example text "ØÆÅ" could be legitimately represented as either three or four code points, depending on whether the Å character is precomposed (U+00C5) or decomposed (U+0041 U+030A).

ender72• 13 years ago

Be careful when flinging bias accusations around; the author is not actually from the US/UK [0].

In addition, while .NET methods will work correctly for "ØÆÅ", they have incorrect behavior when it comes to any characters that lie outside the BMP, which includes many CJK characters. So, .NET only meets your requirements if you never interact with such languages.

[0] http://www.utf8everywhere.org/#faq.anglophile

haberman2• 13 years ago

I don't think you understood the statement you are quoting.

The manifesto is not in any way advocating that you use byte length as a substitute for string length. (It does argue that string length is not very commonly necessary, and has unclear semantics because of the multiple different definitions for "character", but these arguments are unrelated to the statement you quoted.)

Here's the point the author was making, which you missed. Take a non-BMP code point like '𝄞', which is U+1D11E. In UTF-16 this is represented by code units D834 and DD1E. If you try to use a "substring" operation to take the first "character" of this with a C#/.NET substring operation, you will get an invalid string, since D834 by itself is not a valid UTF-16 string.

I want to work on top of an abstraction which lets me treat text as text.

If you think that UTF-16 will let you say string[i] and always get the i'th character, you are mistaken. That is one of the main points of the essay.

Yes, their are cases where the abstraction will leak. But those cases are very far and few in between.

If you write your apps this way, then you don't really support Unicode, you just support the BMP, without combining characters.

mark-r• 13 years ago

One of the advantages of working in UTF-8 is that if your code is broken, you find out about it right away - as soon as someone enters an accented character. If your UTF-16 code is broken you might not find it in testing.

enneff• 13 years ago

How many characters is "é́́"? To our eyes, one. But it's actually four code points (the e and three acute modifiers). This is the case whether you're using UTF-8 or UTF-16.

Regardless of the encoding you choose, you need to understand how it represents characters.

Also FYI, most programs don't display text on the screen. To most programs, text is a sequence of bytes to be shuffled around, and nothing more. Display and text manipulation is the minority case.

duaneb2• 13 years ago

Thank god for emoji.

pjscott3• 13 years ago

They're very useful if you want a test case that requires multiple bytes in UTF-8 and multiple words in UTF-16.

duaneb2• 13 years ago

Seriously, I wasn't being sarcastic! Emoji have been the single largest driving factor in proper unicode adoption I've seen—they're the first non-BMP characters to see wide-spread use.

jahewson• 13 years ago

there is a silent agreement that UTF-8 is the most correct encoding for Unicode on the planet Earth

But what about other planets? Is there a Unicode Astral Plane which may encode poorly in the future?

taejo2• 13 years ago

There is are 13 Unicode astral planes (though only two of them have characters assigned so far), and it they do indeed encode poorly in some environments: the planes other than plane 0, the basic multilingual plane, are informally known as "the astral planes".

vorg3• 13 years ago

There's 16 astral planes (U+1xxxx to U+10xxxx), of which 3 have characters assigned...

* plane 1 is the supplementary multilingual plane

* plane 2 is the supplementary ideographic plane

* plane E is the supplementary special-purpose plane

* planes F and 10 are private-use planes

Perhaps you wrote from memory.

mattfenwick• 13 years ago

I think it's unfortunate that it doesn't have more concrete examples. I think having more of those would really help strengthen their case, clarify their points, and make their arguments tangible and understandable to a much wider audience.

One instance where I really wish for examples: they mention characters, code points, code units, grapheme clusters, user-perceived characters, fonts, encoding schemes, multi-byte patterns, BE vs LE, BOM, .... while I kind of get some of these, I certainly don't understand all of them in detail, and so there's no way that I'll grasp the subtleties of their complicated interactions. Examples, even of simple things such as what actually gets saved to disk when I write out a string using UTF-8 encoding vs. UTF-16 -- especially when using higher codepoints, would be hugely beneficial for me.

vorg3• 13 years ago

I don't like the way UTF-8 was clipped to only 1 million codepoints in 2003 to match the UTF-16 limit. The original 2.1 billion codepoint capacity of the original 1993 UTF-8 proposal would've been far better. Go Lang uses \Uffffffff as syntax to represent runes, giving the same upper limit as the original UTF-8 proposal, so I wonder if it supports, or one day will support, the extended 5- and 6-byte sequences.

In fact, UTF-16 doesn't really have the 1 million character limit: by using the two private-use planes (F and 10) as 2nd-tier surrogates, we can encode all 4-byte sequences of UCS-32, and all those in the original UTF-8 proposal.

I suspect the reason is more political than technical. unicode.org (http://www.unicode.org/faq/utf_bom.html#utf16-6) says "Both Unicode and ISO 10646 have policies in place that formally limit future code assignment to the integer range that can be expressed with current UTF-16 (0 to 1,114,111). Even if other encoding forms (i.e. other UTFs) can represent larger intergers, these policies mean that all encoding forms will always represent the same set of characters. Over a million possible codes is far more than enough for the goal of Unicode of encoding characters, not glyphs. Unicode is not designed to encode arbitrary data."

kzrdude• 13 years ago

What use would it have to have so much extra codepoint space?

vorg3• 13 years ago

2 planes (130,000) of private-use codepoints aren't enough, and because the top 2 planes of Unicode are designated private use, UTF-16 gives developers the option of extending them to 2.1 billion if they need it. I've wanted extra private-use space for generating Unihan characters by formula in the same way the 10,000 Korean Hangul ones are generated from 24 Jamo. I'm sure many other developers come across other scenarios where 130,000 isn't enough for private use.

I'm simply saying that UTF-8 shouldn't be crippled in the Unicode/ISO spec to 21 bits, but be extended to 31 bits as originally designed because the technical reason given (i.e. because UTF-16 is only 21 bits) isn't actually true. The extra space should be assigned as more private use characters. (Except of course the last two codepoints in each extra plane would be nonchars as at present, and probably also the entire last 2 planes if the 2nd-tier "high surrogates" finish at the end of a plane.)

scintill76• 13 years ago

Part of the reason this is a problem is because someone probably said "Who could need more than 16 bits' worth of codepoints?", so I'd err on the side of extra codepoint space.

puppetmaster3• 13 years ago

In Javascript, it's UTF16. Also Java.

Can't speak for other of the top.

chj• 13 years ago

Windows is a horrible environment for UTF8 unless MS provides a special locale for it.

At present state, you can choose to use utf8 internally in your app, but when you need to cooperate with other programs (over sockets or files), it's going to be confusing. Some will be sending you ANSI bytes and you take it as UTF8.

GnarfGnarf• 13 years ago

Interesting. I came to the same conclusion myself a few years ago when converting a Windows app to Unicode. I store all strings as UTF-8, which enabled me to continue using strncpy, char[] etc. I convert to wchar_t only when I need to pass the string to Win32. I can even change from narrow to widechar dynamically. I use a global switch which tells me whether I am running in Unicode or not, and call the 'A' or 'W' version of the Win32 function, after converting to wchar_t if necessary.

UTF-8 Everywhere

Related Stories