A Spectre is Haunting Unicode

https://www.dampfkraft.com/ghost-characters.html

by hardmaru • 8 years ago

353 110 8 years ago

dampfkraft.com

ume• 8 years ago

This is utterly wonderful - brought to mind the film Brazil's fly in a typewriter.

I'm now going to have to go down the rabbit hole of each of these ghost kanji. Good times.

Freak_NL5• 8 years ago

I wonder if some interested Japanese ever got together to simply find new sensible meanings and words to retroactively apply to these ghost kanji. As a sort of creative pass-time.

nneonneo3• 8 years ago

The article links to the Nico Nico Douga Wiki (a wiki for creative works), in which each of the 12 characters is imagined to be the name of a Japanese youkai (a type of spirit/demon/monster): http://dic.nicovideo.jp/a/%E5%B9%BD%E9%9C%8A%E6%96%87%E5%AD%... (Japanese). It’s also quite a neat coincidence that there are exactly 12 remaining unattested characters (down from ~60 before the thorough investigation), because 12 is a rather auspicious number in East Asian tradition.

For example, 妛 (山 mountain atop 女 female) is imagined to mean a yama-uba (traditional “mountain witch” youkai) who lives at the foot of the mountain (as befitting the “女” position in the character).

There’s a good tradition of reusing old or outdated characters in new contexts. For example, in Chinese, the archaic character 囧, meaning “window” (modern term 窗) has been resurrected to mean “embarrassed” or “awkward” due to its pictorial resemblance to a facial expression.

duckerude2• 8 years ago

The fixating effect of digitalization is scary.

New characters were introduced by accident, and although they're useless, they're probably not going anywhere. But what about the inverse? Will character sets still be able to change and grow new characters now that they're standardized like this?

innocenat2• 8 years ago

The answer is easy -- No, it won't be able to change and grow.

Have been to more than one character museum where they show evolution of character. For most of the history you can see some change between 10-20 years, but it stays completely same for last 20 years or so.

emodendroket5• 8 years ago

Twenty? I don't know what you mean. If you pick up a Japanese book published in 1960 the characters are the same ones they use now. Nobody was using bone script or anything.

emodendroket5• 8 years ago

The character set has been pretty stable for a long time with the exception of post-war character simplification (which could be done again without changing the codes)

betterunix2• 8 years ago

Actually the situation is more complicated than that. Japanese Kanji include some characters that were created by the Japanese, following the basic principles of Chinese characters, over the past 1000~ years. Everywhere that Chinese characters have been used there have been local variations. There have also been various hand-written simplifications, which in some cases became "new" characters that over time were repurposed.

It is also not true that simplified characters could be do again without changing code points; it is true for Japanese but for Chinese simplification there were several characters that were "merged" i.e. two formerly different characters are now written the same way. I would actually argue that the idea of "code points" makes no sense for Chinese characters, and that instead an approach based on the components of each character should have been used.

kryptiskt• 8 years ago

Print already fixed it pretty hard. If anything, it's easier to sneak in a custom font and use the private use area than what you had to do to get a new character in lead type.

Freak_NL5• 8 years ago

Character sets in general will certainly continue to grow — look no further than the ever-expanding emoji set. Unicode is designed to be able to incorporate new characters.

Whether the CJKV¹ set will grow is another matter entirely. I expect that there will be a couple more extensions for characters found in historical sources; the goal of the Unicode standard is — roughly speaking — to be able to digitize any and all existing texts, after all.

But for actual new characters to be added they would first have to become popular enough to warrant inclusion. Neither Chinese nor Japanese is a language that produces new Chinese characters like they did historically. The barrier to get a character excepted beyond a fringe group is quite high — in any language — and there really is no strong need to create new characters.

1: Chinese, Japanese, Korean, Vietnamese (the latter two historically used Chinese characters as well)

nabla9• 8 years ago

Emojis are mistake and added too soon. We end up with standard that is full of short lived symbols from 2000-2025 that were once in a fashion and new generations of people will abandon them for something else when the culture changes.

Adding new stuff (excluding old languages) to Unicode should have a delay, at least 20-25 years from proposal to the standard (roughly a generation).

In the meantime it's possible to insert emoji into text as :), smiley, -smiley-, smiley.jpg, ::smiley:: or whatever you want and the system you use is free to change it into a picture.

dingo_bat• 8 years ago

But for actual new characters to be added they would first have to become popular enough to warrant inclusion.

How would a character become popular if people cannot type it?

panic• 8 years ago

It's much easier to reproduce images / vector graphics on a computer than it is on a printing press. Reaction GIFs, custom pseudo-emoji, "stickers", and so on could be considered variations on characters. Also, handwriting is possible with a stylus, though tablets aren't popular enough for this to have become a big thing yet (it may someday). I wouldn't worry too much!

masswerk• 8 years ago

German just saw the addition of the capital "ß" (U+1E9E) last year.

ygra• 8 years ago

A character that existed for a while, just not in digital typefaces.

lmm• 8 years ago

New characters were introduced by accident, and although they're useless, they're probably not going anywhere.

I'd say that since they exist, people will find uses for them. The article links to a page of ghost stories that uses the ghost characters as names for them; I've also seen one TV show's title given as Magical�Pokaan (sic).

mcguire3• 8 years ago

It's not a new question; I've seen arguments about Año Nuevo State Park and whether the tilde is necessary in English after some major publication discovered that they didn't have ñ in their standard font and their proofreaders didn't know Spanish.

duskwuff• 8 years ago

The ~ in "Año" is never optional. Removing it leads to some incredibly inappropriate readings.

Año = year. Ano = anus.

a3n• 8 years ago

Like useless genes.

cschmidt• 8 years ago

There was this story a few days ago about adding a character, for the new era in the Japanese calendar due to the Emperor's abdication and crowning of the new Emperor.

They need to add a character for the name of the era, but it hasn't been decided yet. So the Unicode standards is going to need to rush it in.

https://news.ycombinator.com/item?id=17608655

adamrezich• 8 years ago

Controversial opinion but I believe the skin-tone modifiers fall into this category as well.

peterburkimsher• 8 years ago

I'm in the process of proposing 7 new Taiwanese characters, and 14 more Hakka characters to the Unicode IRG.

They need scanned evidence of times when the characters are used. It then takes about 5 years to actually be introduced into the standard.

ape4• 8 years ago

Like DNA coding mistakes

malkia• 8 years ago

Basically "NaN"s in character form :) - fascinating read!

userbinator• 8 years ago

It somewhat reminds me of this: https://en.wikipedia.org/wiki/Cangjie_method#Early_Cangjie_s...

A particular "feature" of this early system is that if you send random lowercase words to the character generator, it will attempt to construct Chinese characters according to the Cangjie decomposition rules, sometimes causing strange, unknown characters to appear.

https://en.wikipedia.org/wiki/File:Mingzhu_xiaoziku1.PNG

reaperducer2• 8 years ago

If they're in Unicode, it should be possible to register ghost characters as a ghost domain name, right?

NVRM• 8 years ago

With Punycodes?

zamadatix2• 8 years ago

I don't see why not. Hell, it's probably possible to register a domain name for characters that aren't even defined in Unicode yet.

reaperducer2• 8 years ago

So you could take a chance registering some not-yet-defined code point, and possibly end up with the next poop emoji as your e-mail address!

Worst lottery ever.

mistaken• 8 years ago

Any reason why these charcters cannot be revoked if nobody uses them?

TheDong• 8 years ago

How do you prove a unicode character is unused? Look at every database field in the entire world? Every book someone has ever typed on a computer and printed? See what fonts include them or don't?

You can't define what an unused character looks like once it's in the standard... even less if it's already available on almost every modern computer.

People have already used these "ghost characters" when describing them, as jokes in their online handles, etc... This article and now this comment include "妛" which is one such character.

Furthermore, how would you even remove them? Tell all font authors "unicode point X now shouldn't be included in your font"... why would they ever bother to remove something from a font?

And what would be the benefit? Unicode offers enough codepoints that there's plenty of space left over for whatever garbage you wish to include, like new emojis and super-astral-aether-planar characters.

garmaine2• 8 years ago

This blog post used them.

int_19h2• 8 years ago

https://www.unicode.org/policies/stability_policy.html

Encoding Stability

Applicable Version: Unicode 2.0+

Once a character is encoded, it will not be moved or removed.

This policy ensures that implementers can always depend on each version of the Unicode Standard being a superset of the previous version. The Unicode Standard may deprecate the character (that is, formally discourage its use), but it will not reallocate, remove, or reassign the character.

mjevans• 8 years ago

Presumably they can also be entered as part of a passphrase. Now that it's published it's even more likely.

narrator• 8 years ago

This reminds me of the famous mistake of "referer" being spelled with only 1 'r' in the HTTP standard.

matsemann• 8 years ago

As a kid that tinkered with programming/computers before I learned English, this has stuck as something I always misspell.

rossdavidh3• 8 years ago

So, since we have them and they're not going away, we should invent a meaning for them. Obviously, one of them will have the meaning "ghost character". Other than that, meanings like "mistake that would have been simple to correct if it had been caught quickly, but now it is too late", or "something that slips through the cracks of bureaucracy", or "mistake that does not cause an actual problem, but seems wrong".

Agathos• 8 years ago

I like these suggestions. It sounds like they are all suited to commenting legacy code.

rossdavidh3• 8 years ago

Ouch. Too true.

UncleEntity• 8 years ago

I say just leave them alone so drunk teenagers can get them tattooed and get laughed at by native Japanese speakers.

rossdavidh3• 8 years ago

One of them should mean "that thing when someone likes the look of foreign characters and gets a tattoo or buys a shirt or something, without finding out what it means".

HelloNurse• 8 years ago

They could be given sarcastic Unicode names like MEANINGLESS KANJI-STYLE TATTOO #1 to #12.

ezoe• 8 years ago

Any tattooed Japanese glyph is funny for Japanese because We didn't have a culture for tattooing letters. Especially by 楷書体(regular script)

garmaine2• 8 years ago

In the end only one character had neither a clear source nor any historical precedent: 彁. The most likely explanation is that it was created as a misreading of the 彊 character, but no specific indcident was uncovered.

I think I've uncovered a new English word, "indcident." I wonder what it means and how it is pronounced?

akx• 8 years ago

The Google Translate Chinese text-to-speech for that core set of ghost kanji is also spooky. https://translate.google.com/#zh-CN/en/%E5%A6%9B%E6%8C%A7%E6...

invalidusernam3• 8 years ago

Obciouly not exactly the same, but this article reminded me of the scroll lock key. Many people seem to have forgotten its original purpose: When PC Magazine asked an executive of keyboard manufacturer Key Tronic about the key's purpose, he replied "I don't know, but we put it on ours, too"

dbenhur• 8 years ago

Wikipedia remembers: "The Scroll Lock key was meant to lock all scrolling techniques, and is a vestige of the original IBM PC keyboard. In the original design, Scroll Lock was intended to modify the behavior of the arrow keys. When the Scroll Lock mode was on, the arrow keys would scroll the contents of a text window instead of moving the cursor. In this usage, Scroll Lock is a toggling lock key like Num Lock or Caps Lock, which have a state that persists after the key is released."

https://en.wikipedia.org/wiki/Scroll_lock#Window_scrolling

ComputerGuru• 8 years ago

It still works that way too in a real vtty. Hit scroll lock to scroll through the text buffer at the terminal in a text-only session with the up and down arrows on BSD. Very handy.

A Spectre is Haunting Unicode

Related Stories