Dark corners of Unicode (2015)

https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/

by madars • 10 years ago

150 52 10 years ago

eev.ee

BuuQu9hu• 10 years ago

(2015)

leonatan• 10 years ago

This will be relevant in 2025 as well.

dom02• 10 years ago

Will probably stick around to at least 2262.

gpvos• 10 years ago

Yes, but it describes the Unicode situation in 2015. Unicode has probably already gained a few more dark corners since then, what with the proliferation of emoji, but that is not reflected in the article.

tingletech• 10 years ago

I made this once to browse unicode symbols http://tingletech.github.io/unicodetoy/

delan• 10 years ago

That’s neat! I made this tool for Unicode in general: https://www.azabani.com/labs/charming/

saghm• 10 years ago

How’s that for depending on global mutable state?

This is quite possibly the greatest pun I've ever encountered

gotthemwmds2• 10 years ago

I ran into this at a consulting job recently...

MySQL claims to support utf8, but in reality, it doesn't. You need utf8mb4 to support certain common Kanji characters.

This company had spent untold thousands (possibly millions) trying to convert gigantic databases (and I don't use the term gigantic loosely...) from utf8 to utf8mb4 because some of their Japan-based clients were using Kanji.

Sounds easy right? Wrong. utf8mb4 comes with some technical "gotchas" (google it) that had delayed the attempt to change to it by almost a year.

Anyway, I found this pretty amusing, and got a huge paycheck to explain to them just how screwed they were.

DCoder• 10 years ago

utf8mb4 comes with some technical "gotchas" (google it)

I know InnoDB limits index sizes to 767 bytes, meaning VARCHAR(255) using utf8 can have all 255 characters indexed, but VARCHAR(255) using utf8mb4 can only index 191 characters (floor(767/4) == 191).

After a quick Google search, that seems to be the most common gotcha. What other gotchas did you have in mind?

gotthemwmds2• 10 years ago

This was definitely the first thing that came up, as you found.

To be honest, I just don't remember. There was something about something that made something scary to the PM who was in charge of it all? That is about the best I can come up with.

I want to say the needed to index more than 191 chars, but that seems like a stupid thing to say. Who needs to index that many chars?

If I remember, I'll edit :)

edit: I guess I should say I was consulted to do some unrelated things, then helped them with some MySQL stuff that came up towards the end of the contract, then the utf8mb4 stuff came up, and I spent some time going through it with them. It was not the main focus of the contract, which is part of why I don't remember it very well. Just something that came up in the day to day...

ubernostrum4• 10 years ago

MySQL's issues aren't just about character width.

When I worked at Mozilla I was on the MDN (developer.mozilla.org) team, and we had this inexplicable bug: articles can be categorized with tags, and both articles and tags are localizable for all the languages MDN supports. So, for example, English reference articles on CSS properties were tagged "CSS Reference", while French reference articles on CSS properties were tagged "CSS Référence".

And... sometimes an English article's page would show it as having the French ("Référence") tag, and sometimes the French article's page would show it as having the English article's tag.

Turns out, MySQL's case-insensitive UTF-8 collation treated "e" and "é" as the same character. We didn't know about that, and hadn't noticed because the tagging library we used worked around it. Until one day a new version of it didn't, and tags from one language would start showing on another language's articles (if the words were the same, aside from diacritics/accents on certain characters). Which led to this:

https://github.com/mozilla/kuma/blob/00fc05b101658f863f58d7f...

That's a custom MySQL collation, which MDN defines and installs, to work around MySQL's default inability to tell "e" and "é" apart.

david-given• 10 years ago

A while back I found a minor bug in Thunderbird, where astral plane code points were considered by the line wrapping algorithm to have a width of 2 rather than 1. So I filed a bug report.

Oh, boy.

Turns out that trying to include astral plane code points in whichever version of Bugzilla that Mozilla uses causes comments to be silently truncated! Because MySQL.

I filed that one in 2010; it got deduped against a bug originally filed in 2007; it is now 2016, and the bug is RESOLVED FIXED, and Mozilla's bugzilla still has the same problem.

https://bugzilla.mozilla.org/show_bug.cgi?id=405011

At least the original Thunderbird bug has been fixed.

Animats• 10 years ago

From the comments on that page, some registrars are now registering domains with emoji in them. RFC 3490 didn't contemplate that, so it is not, apparently, disallowed. No idea if IDNA does normalization for the color modifiers.

my_first_acct• 10 years ago

RFC 3490 has been replaced by RFC 5890. Sadly, emojis are disallowed. But some registries (for example, .ws, according to rumors) may allow them to be registered anyway.

ubernostrum4• 10 years ago

You could always try to register the corresponding punycode and see if it works...

michaelvoz• 10 years ago

As someone who wrote the LTR/RTL text shaping for Uber Maps, I know the pains of this way too well. Off by one Unicode errors were causing Chinese characters to get appended to the end of Arabic words! Great read.

Manishearth4• 10 years ago

This article is pretty great, however I would like to caution against using the word "character" in discussions about Unicode. The post does say that it's a fuzzy concept, but also says that it's "Basically a thing in the Unicode table somewhere.", which is the definition of a code point. Unicode itself gives multiple incompatible meanings for the term (http://unicode.org/glossary/#character) and only seems to use it in non-normative text -- so the post isn't wrong, but has the potential to cause confusion.

Often when you want to say character you really mean "grapheme cluster"[1]. A Devanagri consonant cluster (of one or more consonants) with optional vowel (and misc diacritics) is a grapheme cluster. A latin letter with accent mark(s) is a grapheme cluster. A Hangul Jamo (precomposed or otherwise) is a grapheme cluster. A flag or multicultural family emoji is a grapheme cluster.

When talking about characters, the question arises if a Hangul Jamo made of decomposed characters is one character or three. Or if the combination of a [devanagri consonant + virama] "character" and [devanagri consonant + vowel] "character" is one character or two. This is specified in the case of grapheme clusters, and this usually maps to where the notion of a character is important -- text selection and offsets in editing, etc. "code point" does not universally map to any tangible concept -- it's a concept made up for the sake of specifying unicode. You only care about code points when dealing with UTF32 strings or when implementing operations on unicode text. "glyph" is also sometimes what we mean when we say "character", though that's more useful on the rendering end of things.

[1]: to be pedantic, "extended grapheme cluster", because unicode gave a rigorous definition of grapheme cluster and later decided to change it.

d0mine• 10 years ago

I find the following hierarchy helpful: bytes -> code units -> code points -> extended grapheme clusters -> user-perceived characters

In Python, Unicode string is an immutable sequence of Unicode code points. It has nothing to do with UTF32 (Python uses flexible internal representation). To get "user-perceived characters" (approximated by eXtended graphemes clusters):

  chars = regex.findall('\X', unicode_text)
  width_in_terminal = wcwidth.wcswidth(unicode_text)

In practice, whatever is produced by the default method of iterating over a string is often called a character in that programming language (code point, or UTF16 code unit, or even a byte).

https://pypi.python.org/pypi/regex https://pypi.python.org/pypi/wcwidth

labster2• 10 years ago

In Perl 6:

    $width-in-terminal = $text.chars;
    $codepoints = $text.codes;
    $bytes = $text.encode('utf8').bytes;

The .chars method should be the fastest, because Perl 6 internally uses strings of fully composed characters (normalized form grapheme). It's much better than having to do regex hacks like in Python.

Manishearth4• 10 years ago

It has nothing to do with UTF32

Right, my point is that the concept of a code point is seldom useful unless doing storage stuff with utf32 or implementing unicode algorithms. Python may expose an API of code points but that doesn't mean that it's meaningful.

Performance arguments can be made as to why the API should use code points instead of grapheme clusters, so there are legitimate reasons for Python (and Rust, and many other languages) to do so. Sometimes you just need some comparable notion of length and "number of code points" is acceptable.

However, you should be careful when writing code that confers meaning to the concept of a code point. A lot of code does this (using code points when they mean glyphs or grapheme clusters).

jasode• 10 years ago

, but has the potential to cause confusion.

On a related note, one of the best examples that reveal the confusing intricacies of Unicode was a previous HN thread about encoding Bengali:

https://news.ycombinator.com/item?id=9219162

Lots of intelligent comments from many contributors in that high quality thread.

For me, the takeaway was the truly understanding Unicode requires holding 5 different layers of concepts in your head simultaneously and the oft-cited Joel Software "Absolute Minimum You Must Know About Unicode" only covers 2 of them. (Which to be fair to Joel, his title emphasizes "minimum to know" and not "everything to know"). Also, at the highest level of abstraction, intelligent people can have philosophical disagreements on _what_ to encode in Unicode.

brudgers• 10 years ago

The problem with "character" is the same as the problem with "string." There is a formal computer science definition that comes from the discipline of automata, is mathematical, and useful for building compilers and state machines and why characters and strings in C are bytes and streams of bytes.

Because ASCII abstracts onto bytes well but text does not, layering an abstraction for 'text' that decomposes into characters and strings is where the problem lies. That's where, for example, Python 2 ran into trouble. Perl's set of compromises made its text abstraction useful, but creating them required creating them to be a principle focus of the language.

A stream of bytes is a stream of bytes, short of magic there's no way to generate a correct interpretation but for a predesignated protocol. And the only way to get a predesignated protocol for text is to make a deep study of human language and to choose to live with some compromises and not to live with others.

elFarto2• 10 years ago

One of my favourite dark corners of Unicode is IDS, or Ideographic Description Sequence. It's allows you to describe characters that are not encoded in Unicode, but can be described by a combination of existing ones.

For example, the Chinese character for the word Biang[1] can be describe with:

⿺辶⿳穴⿲月⿱⿲幺言幺⿲長馬長刂心

[1] https://en.wikipedia.org/wiki/Biangbiang_noodles#Chinese_cha...

username2233• 10 years ago

"Made up of 58 strokes in its traditional form (43 in simplified Chinese)..."

I'm glad I don't have to try to type that with 8 fingers and 2 thumbs. Writing Chinese with a keyboard sounds insane: http://www.slate.com/articles/news_and_politics/explainer/20...

amake2• 10 years ago

The very first input scheme the article talks about is Pinyin, with which you'd input biang by typing... "b" "i" "a" "n" "g" then selecting from a list (most likely a list of length 1). Not insane or even difficult.

(Except that biang is not encoded in Unicode yet so you can't type it anyway.)

crazygringo• 10 years ago

Is there any software which actually renders that? Will browsers do it?

The Wikipedia page just uses images for the characters, I can't seem to find any actual example.

elFarto2• 10 years ago

As far as I know, there's nothing that will render them. There used to be a web based renderer, but I can't seem to find it now.

xg152• 10 years ago

My takeaway is that turkish "I" and English "I" as well as English "æ" and Icelandic "æ" are crammed into the same character even though that causes all sorts of problems, but there are about 20 different characters that represent "x"...

maxlybbert• 10 years ago

The solution to the problem, by the way, is to determine what the user expects and give it to them. That is, you can't just sort words without defining what sort you want to do (according to German phone book sort? Portuguese dictionary sort? etc.).

nonsince• 10 years ago

Where do you sort the Dutch "ij"? The obvious answer is "between 'ii' and 'ik'" but it's actually just the print representation of "ÿ", a letter that is essentially only used in freehand nowadays. So, do you sort it in place of "ÿ", when in the Dutch locale? What if you have a borrowed word that happens to have "ij" in it, like "hijack" (which really is used in Dutch)? In practice, you sort it as if it's English (i.e. between "ii" and "ik"), but that leads to confusion because when capitalising you treat it as a single letter. The titlecase form of "ijzer" (iron) is "IJzer". I guess the correct way would probably be whatever they do in dictionaries and phone books, but I'm an expat so I have no idea what ordering those use.

xg152• 10 years ago

Based on other cases, I guess the "unicode-style" solution would be to add a new codepoint DUTCH CHARACTER IJ OR ÿ which is cased and sorted accordingly.

Of course then you'd have to teach people to use that character when writing instead of just typing "ij". And one day, there will be someone who, for stylistic reasons, needs tight control over when it's rendered as "ij" and when as "ÿ"...

pitdicker2• 10 years ago

It already exist: U+0132 and U+0133.

Her is a link with some nice information: http://www.uazone.org/multiling/euroml/annex02.html

ubernostrum4• 10 years ago

Wikipedia claims that Dutch dictionaries sort 'ij' as if it were two characters, but telephone directories sort as if it were 'y'.

Based on precedent, I'd expect the Unicode Consortium to declare it the radical of a Chinese character.

(it does exist in Unicode, but as a ligature -- U+0132 for majuscule, U+0133 for minuscule -- and decomposes to the two code points for 'i' and 'j', with use of the ligature discouraged)

pacaro• 10 years ago

English has simply abandoned the letters thorn and eth (and others), which was largely the result of the rise of printing (IIRC).

It's interesting to me that this happened far enough back that the only time we tend to consider 'th' to be letterlike is when we need to explain pronunciation — e.g. when reaching a kid to read

pitdicker2• 10 years ago

"ij" is usually sorted between "ij" and "ik" by Dutch dictionaries. The odd exception (even for Dutch speakers it seams odd) are phonebooks, because for historic reasons the surname of family members is sometimes written with "y" and sometimes with "ij". Titlecase (IJzer) is not really a problem, because there are no loan words that start with "ij" (like hijack). Very, very rarely "ij" is treated as a ligature, and counted as one letter. So, for the most part you should follow the English sorting rules, and you need one exception for handeling title case. Unicode contains a ligature "ij", which should basically not be used.

adamnemecek• 10 years ago

Dark corners of Unicode seems to be a tautology.

paulddraper• 10 years ago

I'd say "truism", but yes.

pja• 10 years ago

The double-width emoji bug has at least been fixed as of 2016 & the Unicode 9 release. That string works fine in my VTE based gnome-terminal today.

(I was the one who finally prodded the relevant Unicode cttee into fixing this bug. They did all the heavy lifting of writing proposals and steering the change through though: Thanks Ken et al!)

pm24601• 10 years ago

Just as an aside, as humans we invented the 'weirdness' of these different human languages.

In the pencil-and-paper era, we had no problems ever with this.

It's only because we tried to 'simplify' things that we ran into problems :-)

As an additional aside, the OP just talked about sorting words within the same language.

What about sorting across languages? (For example, names?)

gizmo686• 10 years ago

Its worse than that. None of this weirdness is part of natural human language. It is all part weirdness of the writing system, which is entirely invented. What we are facing now is difficulties in interfacing different implementations of the same technology (writing).

Dark corners of Unicode (2015)

Related Stories