Life Before Unicode

https://baturin.org/blog/life-before-unicode/

by sidcool • 5 years ago

118 171 5 years ago

Grustaf• 5 years ago

I remember making vocabulary lists for Russian and Greek in Latex with emacs on Linux, before unicode, it wasn't easy. Definitely spent more time getting it to work than actually using them.

bellyfullofbac2• 5 years ago

An old story was someone from France wanting to send the Harry Potter book to someone in Russia. They got their address over email, but in the wrong encoding: https://unicodebook.readthedocs.io/definitions.html#mojibake

gpderetta• 5 years ago

the most amazing bit is that the postal employees reversed the bad encoding and delivered the package correctly!

chrismorgan• 5 years ago

Yet it’s sad that they were able to do this, because it suggests that they’ve encountered it enough to develop a procedure for fixing it. (That seems more likely than it being a one-off thing, since I would expect very few people to be capable of figuring out how to convert it, even if they encounter it commonly.)

WesolyKubeczek3• 5 years ago

Back in the day I learned cp437 and its mapping to cp866, and was able to read cyrillic DOS programs without altering the character generator in the graphics card.

This was useful on a computer with a sort of Hercules graphics card which didn’t have the capability in text mode and behaved like IBM MDA.

Needless to say, my knowledge got outdated fairly quickly.

simonh• 5 years ago

I just find it funny that western Europeans can't tell the difference between Cyrillic writing and random gibberish. By which I genuinely don't mean anything pejorative, in either direction.

kergonath10• 5 years ago

Do you mean printed, or handwritten? Because Sütterlin looks like gibberish to most Europeans. And it is just the standard Latin alphabet, give or take. It looks like some kind of letters, but they could be from the Voynich manuscript and they would still be as understandable.

In print, it’s partly because several letters are deceptive and don’t sound anything like the closest-looking ones in the Latin alphabet. Some others are just plain weird-looking if you are not familiar with the alphabet already.

coldacid6• 5 years ago

It doesn't help that the cursive Cyrillic T looks like a Latin M.

chana_masala• 5 years ago

The address was deciphered by the postal employees and delivered successfully.

Wow, I want to know more about how the postal worker who deciphered it! On a computer, sure just convert it. But on paper, how did they work it out?

WesolyKubeczek3• 5 years ago

I used to have popular books about computers which had character maps of popular encoding in the appendices. Every book about programming, or using DOS/Windows 9x had them.

Each book for system users (not even power users!) had a chapter about different character sets and common problems like this. This was important.

I think the post office worker was enough of a geek or they took the package to the computer guy.

Still more interesting thing to me was that the sender got the address in the wrong encoding and instead of asking via email, just must have thought that Russians have their little ways and dutifully copied all the gibberish (Russians even have a specific term for misencoded characters: krakozyabras).

thriftwy5• 5 years ago

It's pretty obvious from the letters which are superscripted/skipped

First they knew Russia, Moscow by the postcode, so they superscripted РОССИЯ МОСКВА, and then they knew which garbled letters correspond to these ones

After which they were able to decipher ПР Р.А.СКО.О

after which the address becomes obvious (with the help of index) that it is Проспект Вернадского, 37. I don't think they used any tables other that this implicit table.

Which is student dormitory of Moscow State University btw, I looked it up.

squiggleblaz4• 5 years ago

The address there is

Russia, Moscow, (postcode), (maybe a street name), (some number), (untranslated) Svetlane

So it seems to be backwards: The personal name is last, and the country/city is first. Does someone here perhaps know if that is the customary order of address in Russia, or if it was reversed too?

WesolyKubeczek3• 5 years ago

In the whole of USSR (and after its dissolution too), the order of addresses was big-endian: from the larger administrative unit to smaller (which seems quite logical — first you enter the country, then find the district, then the town, then the street, then the house, then the person therein, thus the address replicates the travel path of the package).

Some countries (doubtless those that entered EU, also Ukraine, can’t say for the rest) officially changed the order to be little-endian, but many older folks, who still use snail mail and write letters and postcards, don’t bother.

bellyfullofbac2• 5 years ago

Welcome to the fun that is address field localization. Some countries have house numbers first, and some have it at the end.

742 Evergreen Terrace vs. Platz der Republik 1.

Google's Contacts app just has a text field and lets Google Maps take care of the parsing.

vintermann• 5 years ago

By comparison, recently I bought some small homemade electronics thing from Germany. The sender had written Ålesund on the package. Deutsche Post, or possibly whatever system Posten bought from the lowest bidder, declared that Å isn't a real character, so they dropped it. Unfortunately Lesund is also a place.

After it had been stuck in the sorting facility for three weeks, I called them up and eventually reached a guy who could explain that oh yes, the system does that and the package gets stuck in a loop, but eventually a human will look at it.

It did in fact arrive eventually.

q-rews7• 5 years ago

It’d be kind of ridiculous that a national service would not recognize their own characters as a default. It’s more likely that the general-use OCR failed in your instance.

inglor_cz3• 5 years ago

Anecdotally, I had this kind of experience mostly from services in big countries. France, Germany, Italy.

Small countries' services (Slovakia, Bulgaria) need to serve many more customers abroad, so they tend to be more friendly towards foreign characters not found in their national alphabets.

q-rews7• 5 years ago

I hate and love that. Someone must have seen those characters enough times to have become good at deciphering them.

What a waste of time for the postal system though, I’m surprised they didn’t just return to sender.

RicoElectrico• 5 years ago

And at no point did the sender stop and think "this doesn't look like Russian". I mean, it's relatively easy for me, who speaks none of the following languages, to tell appart Chinese, Japanese and Korean.

I guess this is consolating for Americans who get blamed for assuming the rest of the world is like their country (postal codes, address formats, date formats, person naming conventions, business customs).

yoavm• 5 years ago

Try this with Hebrew... Because of the RTL nature of the language, we had ISO-8859-8 and ISO-8859-8-I, which as a child I always thought of as "Inverted". The characters would render the same, but going backwards. When entering a website you never knew if the author wrote the text backwards so you can present it as it is, or they wrote it normally so you need to flip it. And I can still recall some websites using CP852, back from the DOS era. Entering a website really did start with about a minute of fiddling with the encoding.

jaza• 5 years ago

An informative and entertaining journey through the world of Russian character encodings, of which I previously knew nothing. Thanks for sharing!

sheetjs• 5 years ago

Pre-Unicode issues still haunt us today, kept alive by various file formats that rely on system encoding.

Under the Apple "Mac-Roman" encoding [1], the standard MacOS encoding before OSX switched to Unicode, byte 0xBD currently is capital omega (U+03A9 Ω). However, in the original 1994 release of the character set, they erroneously mapped to the ohm sign (U+2126 Ω) Apple eventually fixed this in 1997, as noted in the changelog:

    #       n04  1997-Dec-01    Update to match internal utom<n3>, ufrm<n22>:
    #                           Change standard mapping for 0xBD from U+2126
    #                           to its canonical decomposition, U+03A9.

However, in 1996, Microsoft copied over the mac encoding to CP10000 using the incorrect character [2]. Unfortunately the codepage was not corrected when Apple realized their mistake.

This discrepancy leads to a huge number of strange issues with various versions of Excel for Mac (BOM-less CSV, SYLK and other plaintext formats default to system encoding) and other software that use Microsoft's interpretation of Apple's Mac-Roman encoding rather than Apple's official character set mapping.

[1] http://www.unicode.org/Public/MAPPINGS/VENDORS/APPLE/ROMAN.T...

[2] http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/RO...

girishso• 5 years ago

This reminded me of CDAC's GIST, for enabling Indian languages on computers, worked on it 2 decades back. https://www.cdac.in/index.aspx?id=mlc_gist_about

ajuc• 5 years ago

Even for Latin-script-based languages pre-unicode times were hard. Polish had about a dozen encodings: https://pl.wikipedia.org/wiki/Kodowanie_polskich_znak%C3%B3w

With the first 5 (except Mac) still in common use in 90s and the first 3 still used when unicode appeared.

Most programmers in Poland in 90s remembered how encoding errors looked like between the most popular encoding pairs because it was a constant struggle.

xvilka• 5 years ago

The only left task is to get rid [1] of UTF-16 usage.

[1] http://utf8everywhere.org/

Ndymium2• 5 years ago

The note about IRC reminds me of troubles that faced Finnish IRC users when UTF-8 got more popular. Since IRC networks generally didn't (and don't) have actual support for different encodings and mostly just deal with byte streams, this brought a lot of issues with Ä, Ö, Å, €, and some other more esoteric characters. Naturally a lot of Ã¤ and � abounded.

An interesting consequence was that channels that had non-ASCII characters in their names were split into two, since to the network they were different characters. I remember taking over a couple of channels by creating the UTF-8 versions of them and waiting for people to slowly migrate over.

Getting all of this to work correctly was quite difficult. With Irssi and similar terminal clients, you'd have to correctly configure not just the client but also screen, $TERM, locale, and your terminal. Even if you had a correctly configured UTF-8 client, you could still have problems. Since you can't tell 8-bit single-byte encodings apart other than by heuristics, typically you would have your client attempt to treat a message as UTF-8 and if it's invalid, use a fallback option like ISO-8859-15 (latin-9). But here's the fun thing about that: since IRC networks only deal with byte streams, they may truncate an UTF-8 message in the middle of a multibyte character. This would fail to be detected as valid UTF-8 and would use the fallback option, leading to mojibake for seemingly no reason.

All of this lead to quite some resistance to UTF-8 on the channels I was on. It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages. Eventually UTF-8 won, especially when mIRC released a version without an option to change the encoding.

satyanash• 5 years ago

It was deemed the problems were bigger than the new possibilities. I mean, we could speak Finnish/English just fine and there was usually no need for any other languages.

It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now?

Another example is punycode. unicode URLs are super unreadable, so no wonder the rest of the earth's population doesn't care enough about the open web, the importance of URLs vs Apps.

Ndymium2• 5 years ago

It also excluded a large part of the world from participating on IRC. Maybe if it was more proactive, IRC would have a larger role to play than it does now?

I have to clarify that this was based on my experience in Finnish IRC channels. We spoke Finnish (and some English because English is everywhere) and we did not wish to bring any other languages into our channels as we wouldn't be able to speak Finnish then anymore. The character set issues were really just about our own special characters and ISO-8859-1 vs ISO-8859-15 vs UTF-8 users.

Obviously character set issues probably did discourage some people from using IRC but I doubt it had anything to do with Finnish channels specifically. In the end, I think IRC was just overtaken by more convenient and featureful technologies (not to say that IRC is dead, but it's far from what it used to be).

Punycode is another one of those unfortunate things we have to have to deal with due to things like homograph attacks (apple.com vs аpple.com).

distances• 5 years ago

Our still-alive IRC channel never migrated. You still need to enter \xE4 for the join command to get the correct ISO-8859-1 channel. I think the stream of unintentional visitors stopped around the UTF-8 migration, though that could've also been effect of the decline of IRC use in general public.

anthk• 5 years ago

ISO-8859-15

Ditto in Spanish;, I still have issues at IRC-Hispano with some users as I have UTF-8 with everything.

WalterBright29• 5 years ago

Dealing with multiple code pages was terrible, ugly, impossible, and simply awful.

Unicode was a great salvation. Until the Unicode spec wandered off into nutburgerland:

1. embedding invisible semantic information into the encodings

2. multiple encodings that mean the same thing

3. moved beyond standardizing existing alphabets, and wandered into a never-ending stream of people inventing all sorts of glyphs and campaigning to get them into Unicode

4. set language back 2000 years by deciding that hieroglyphs were better than phonetics

5. conflating 𝖋𝖔𝖓𝖙𝖘 with alphabets

6. the crazy concept of "levels" of Unicode compatibility

all resulting in it simply being impossible to correctly support Unicode in your programs, or design a font to display Unicode.

quickthrower23• 5 years ago

Genuine question: can you explain how phonetics can be used in a character encoding for text?

WalterBright29• 5 years ago

I don't understand your question. `a` is for the sound `aaaahhhh`.

grlass3• 5 years ago

RE point 4, I believe that this was initially a descriptivist rather than prescriptivist step.

People were using emoticons, showing that emoji might be desirable. Some cell phone manufacturers/carriers added their own emoji [1]. They were very popular. Unicode added emoji to the standard, since lots of people were using it.

Not adding emoji would have been _more_ prescriptivist, and since it was desired anyway would have just caused fragmentation.

[1] https://blog.emojipedia.org/correcting-the-record-on-the-fir...

WalterBright29• 5 years ago

Emoticons are simply interpreting :-) as a smiley face, no Unicode invention is necessary.

WalterBright29• 5 years ago

Did you know that eons ago, people would draw a picture of a bull to mean a bull? Then they got lazy, and would just draw the head? Then they decided to just have the bull's head represent the first sound of their word for bull. Then they simplified the bull's head to 3 straight lines (people get lazy). Then for some reason decided it was easier to write if they rotated it 180 degrees.

Yup. A bull's head. But now that's no good anymore, and we're back to a picture of a bull, and need about a million more pictures to flesh out our new and improved alphabet. (There are a million words in the English language.)

q-rews7• 5 years ago

3. Yes, sadly.

4. Can you expand on that?

WalterBright29• 5 years ago

expand

Sure. I don't know what most of the buttons do on my car's dashboard, because of the "iconitis" disease that has infected user interface design. Unicode has blessed and encouraged this crap. They don't just standardize, they invent icons intending them to replace the phonetic words.

(Apple is kinda schizophrenic on this. Sometimes you'll see "DELETE" as a button, sometimes a trash can.)

I don't know what the wash icons on my clothes mean without going to google and looking them up.

You can't look up an icon in a dictionary, making it a fundamentally wretched replacement for a phonetic alphabet. Worse, people persistently try to invent icons for verbs and adjectives, which never ever work. Even for nouns, is that scribble a bird, a duck, or a chicken?

My car has a button with something that looks like a snowflake on it. Does it turn the heater on? The defroster? The air conditioner? Turn on traction control? Who the hell knows?

lifthrasiir4• 5 years ago

Unicode is just a cumulation of what already existed in one form or another, not an entirely new mistake.

1. embedding invisible semantic information into the encodings
2. multiple encodings that mean the same thing

Sure, ISO/IEC 8859 back then had no invisible characters nor composed characters, did it? [1]

3. moved beyond standardizing existing alphabets, and wandered into a never-ending stream of people inventing all sorts of glyphs and campaigning to get them into Unicode
4. set language back 2000 years by deciding that hieroglyphs were better than phonetics

Emoji business is surely ugly (and I have complained about this a lot), but the very reason that Unicode has emoji is that Japanese telcos have exposed their priprietary encodings to the wild email and both Apple and Google had to implement them for the compatibility. Blame them, not Unicode.

5. conflating 𝖋𝖔𝖓𝖙𝖘 with alphabets

Yeah, you will be surprised to hear that Unicode encodes both the Latin letter A and Cyrillic letter A separately. (I think you had already said that they should be unified in the past, but I couldn't find that reply.)

6. the crazy concept of "levels" of Unicode compatibility

Most Unicode conformance rules are optional (past discussion: [2]).

Also if you should display a Unicode text in your program and you are not using a stock library, it is up to you to decide what to support because they exist for the reason. No BiDi support? You now disregard right-to-left scripts (or left-to-right scripts if your program is for RTL users). No combining character support? Expect to lose a significant portion of potential users around the world. Probably a colored font support is fine to ignore, but that doesn't mean most other things can be ignored without the cost.

[1] https://en.wikipedia.org/wiki/ISO/IEC_8859-8#Code_page_layou...

[2] https://news.ycombinator.com/item?id=26904739

WalterBright29• 5 years ago

ISO/IEC 8859 back then had no invisible characters nor composed characters, did it?

No reason to repeat mistakes. The whole point of Unicode was to fix it all.

No combining character support?

There is no reason to support both combining chars and unique code points for the same glyph. Unicode should pick one and dump the other.

Most Unicode conformance rules are optional

I know. This is what makes it crazy and nobody implements it right, or even knows what "right" is.

you will be surprised to hear that Unicode encodes both the Latin letter A and Cyrillic letter A separately

They do more than that with visually indistinguishable code points. People use this to embed secret messages in innocuous looking text files that print identically. There's no round trip Unicode->paper->Unicode.

throw0101a2• 5 years ago

Emoji business is surely ugly (and I have complained about this a lot), but the very reason that Unicode has emoji is that Japanese telcos have exposed their priprietary encodings to the wild email and both Apple and Google had to implement them for the compatibility. Blame them, not Unicode.

The initial set of emojis were brought in for compatibility purposes. The huge swaths of new emojis covering every conceivable animal, vegetable, and mineral created after that point is where problems started (IMHO).

account425• 5 years ago

7. Han unification

OskarS2• 5 years ago

Yeah, in my opinion this is by far the biggest one. The rest of list are either non-issues, or they're annoying but you can deal with them (and there's often arguments on both sides).

Han unification, on the other hand, is a disaster for Unicode. The fact that Unicode can't comfortably mix Chinese and Japanese language characters is a disaster for an encoding like Unicode that's supposed to cover the whole world.

bullen• 5 years ago

Non ASCII is a mistake, the only reason for it are humans that don't want to learn English.

English IS humanity's final language, so I'm going back to ASCII so I can use a simple char* for all my strings.

robin_reala2• 5 years ago

On the fragile assumption that this isn’t a lazy troll, the words naïve, façade, fiancée, and coöperative are all in English. And that’s ignoring extremely common English symbols like £ and — that aren’t part of ASCII, and come to think of it the “’” in “aren’t”.

And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly?

JadeNB3• 5 years ago

And, hell, I guess that even after English becomes the world language, you’ll still want to spell people and place names correctly?

I suspect that anyone who believes that English is the final language (how can it be anything other than a lazy troll?) probably believes that dropping diacritics and otherwise mangling the glyphs of other alphabets is spelling them 'correctly'.

account425• 5 years ago

coöperative

Not that I agree with going back to ASCII, but cooperative is already the dominant spelling by a wide margin to the point where using coöperative comes off as pretentious.

shakna• 5 years ago

Por que? Ne comprends pas. Sprache ist überall. Yīngyǔ jīhū méiyǒu zhǔdǎo.

thriftwy5• 5 years ago

There's an option of making char 64-bit, in which case you can always use char*

So that sizeof(int) = sizeof(float) = sizeof(char) = 1

selfhoster114• 5 years ago

That's how we ended up with UTF-16, the abomination from Microsoft. UTF-8 or bust.

account425• 5 years ago

1. sizeof(int) and sizeof(float) are both 32-bit on most current systems

2. Unicode is (currently) only 21-bit.

3. UCS-4 (aka 32-bit char) rarely helps because in most cases you either a) don't really care about characters (most software) and can use substrings as well or b) code points are an insufficient grouping and you need variable length units anyway for grapheme clusers (text rendering and editing).

legulere• 5 years ago

Like Latin and then French was? Chinese could very well become the next Lingua Franca.

selfhoster114• 5 years ago

English is as much an eternal Lingua Franca for human-readable communications as C is for low-level high-performance programming: meaning that it isn't one at all. Both can be unseated with some effort.

C is already being slowly unseated by Rust, and there are more competitors like Zig on the way. English may or may not be unseated by Mandarin, given that American hegemony is waning, and (unfortunately) Chinese dominance on the global stage is rising.

account425• 5 years ago

Good news is that you can use char* for UTF-8, which is an ASCII superset, and many operations such as splitting on an ASCII character work just fine with UTF-8 strings without any code changes.

program• 5 years ago

Italian alphabet is very simple. Only add some accented letters to the base english alphabet (àèéìòù and uppercase variants). Still today I see a lot of Latin 1 / UTF-8 mixed errors. That's the reason why is very common to see "E'" in official documents instead of "È" and so on.

q-rews7• 5 years ago

No, the reason for E’ is that it doesn’t appear on the keyword and there’s no obvious way to make it work.

You’ll also occasionally see PERCHè for the same reason.

Also non-digital natives don’t really know the difference between É and E’ on paper, they all look the same.

inglor_cz3• 5 years ago

When I was in the first year of the university (1996), my friend programmed a utility for conversion between six different encodings used in the Czech language space: CP852, KeybCS2, CP1250, ISO8859-2, KOI-8čs a ASCII.

Just look at this page for the impression of how many times had the wheel been reinvented in Czech and Slovak language space...

http://vorisekd.wz.cz/seznam3.htm

eloisant• 5 years ago

Yes, I moved from France to Japan in 2001 and at the time it was almost impossible to have French accents and Japanese characters on the same OS. The web was where it was working best (because web browsers know to switch encoding for pages) but desktop apps that were still the norm were a shitshow.

Even on Linux, to have support for Japanese language you had to install a Japanese distribution (TurboLinux or RedHat Japanese version for example) and it was a real pain to get accents to work.

Worse, Japanese language had 3 encodings!! One for Windows (SJIS), one for Unix/Linux (EucJP) and ISO-2022-JP. And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time.

necovek• 5 years ago

Encodings are the easy part. To render stuff, you need fonts, and sometimes they are built into hardware (especially with dot-matrix printers, for instance, but even laser printers embed a couple of fonts, and likely custom Japanese fonts too), and software expects fonts with a particular encoding too. All of these were issues even for "simpler" scripts like Cyrillic (Serbian).

Some of the problems with fonts remain even with Unicode (due to Han unification, different CJK regions will prefer different fonts; there might be OpenType locl-aware fonts these days, but they'd be huge).

pezezin• 5 years ago

And of course, Japan being Japan, Japanese companies were really lagging to switch to Unicode and stuck with their shitty encodings for a long time.

Not were, are. I'm currently living in Japan, and most of the emails I receive are encoded in Shift-JIS or ISO-2022-JP. Many websites too. Thankfully modern software is quite robust and displays everything properly, but I still get the random mojibake from time to time.

ChrisMarshallNY• 5 years ago

When it comes to Unicode, Joel Spolsky's screed is still a classic (note that it was done before UTF-8 became the de facto standard): https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

amelius2• 5 years ago

When the USSR broke down, the market was quickly captured by Western hardware and software because no one was making Soviet hardware anymore anyway.

Is that still the case? Are there any large Soviet hardware manufacturers?

manuelmagic• 5 years ago

The second note at the bottom of the page says: "That hardware was too outdated to keep producing in any case, and couldn’t complete in a free market."

So I'd say no, there aren't any large Soviet hardware manufacturers anymore.

amelius2• 5 years ago

Seems like Russia has their own "Silicon Valley":

Zelenograd was founded in 1958 as a new town in the Soviet Union, developed as a center of electronics, microelectronics and the computer industry known as "Soviet/Russian Silicon Valley", and remains an important center of electronics in modern Russia.

https://en.wikipedia.org/wiki/Zelenograd

nanis2• 5 years ago

One thing they got wrong is the fact that upcase(downcase('İ')) is not reliably 'İ' and downcase(upcase('ı')) is not reliably 'ı' without extra assumptions.

That is, Unicode is missing

(1) Unicode Character 'LATIN CAPITAL LETTER DOTLESS I' (????)

and

(2) Unicode Character 'LATIN SMALL LETTER I WITH DOT ABOVE' (????)

or some such so that upcase('ı') could be mapped unambiguously to (1) and downcase('İ') could be mapped unambiguously to (2). Of course, they would have identical glyphs with 'I' and 'i', respectively.

I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird. For example, "sık sık" becomes rather inappropriate after going through downcase∘upcase and "kilim" maps to "kılım" after going through the same mapping.

Most software rely on the current locale to make these decisions. That is, of course, not feasible in systems that process documents received from arbitrary sources.

account425• 5 years ago

I haven't investigated if there was a rationale to save two codepoints, but handling documents that contain an unknown mix of Turkish and other languages becomes rather weird.

The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to. Still might be worth it to add your suggested codepoints though so that uppercase/lowercase that is not locale-aware can at least work correctly for new text.

nanis2• 5 years ago

The main reason is probably that ISO-8859-9 / Windows-1254 already only had a single I and i so it would be impossible to know which Unicode character to convert them to.

The alternative codepoints I proposed could coexist with the previous ones. After all, it is not like Unicode does not offer different ways of representing what humans consider to be the "same".

For example, 'İ' ≡ uc( lc 'İ' ) can be preserved by representing the result of lc 'İ' as "i + COMBINING DOT ABOVE" which I found to be rather clever[1].

However, I am not aware of a similar trick enabling me to preserve 'ı' ≡ lc( uc 'ı' ).

For reference, I have been dealing with Turkish character issues etc since the 80s when I was typesetting texts containing math, Turkish, and several European languages using WordStar during which time I had to hack my own keyboard drivers because there was zero support for Turkish. Then, there was a period where every computer had its own convention.

A lot of people used my hacks for a long time without even realizing them. So, things have improved, but I just can't figure out why not reserve a couple of extra codepoints for logically distinct concepts. It would make nothing worse and improve things in situations where one can take advantage.

[1] https://www.nu42.com/2017/02/for-your-eyes-only.html

bmn__2• 5 years ago

Other languages have similar problems, although not as striking. It does not make sense to special-case Azeri, Crimean Tatar, Kurdish, Turkish, Tatar as you propose.

The real problem is giving in to the notion that having an unknown mix of languages in a text is an acceptable state of affairs. It's not; we should recognise the problem for what it is and work towards a future where it occurs less. Documents carry metadata about the language content, e.g. HTML and OpenDocument, precisely so that the algorithms which depend on a language like case-folding or word matching work correctly.

The situation is analogous to having a document with an unknown/undeclared text encoding: rely on the metadata where supplied, otherwise make a guess, perhaps informed by statistical properties of the text (e.g. uchardet) rather than system locale. It's not ideal, but works more often than not.

coldacid6• 5 years ago

Did they add seamless transcoding for old files from DOS? Of course not! A Russian version of Windows effectively used two different encodings: native Windows parts used Windows-1251, while the DOS subsystem still used CP866.

To be fair, for those of us in the west, it was similar, if not quite as bad. CP437 and Windows-1252 were nearly as different. Woe to you if you opened a DOS text file in Notepad, especially if it had ASCII or box art, because it'd look all messed up, and vice versa.

Joker_vD• 5 years ago

That encoding is, to put it politely, mildly insane: it was designed so that stripping the 8th bit from it leaves you with a somewhat readable ASCII transliteration of the Russian alphabet, so Russian letters don’t come in their usual order.

No, it is not insane. It simply continues the traditional way of creating the national Morse alphabets — almost all of them are transliterations of the international Morse code. Historical continuity is quite a harsh mistress.

throwaddzuzxd• 5 years ago

people who use languages with ASCII alphabets exclusively may think it's unjustified

And what languages would that be? (given that even some borrowed words in English have the letter é for instance)

lonelygirl15a• 5 years ago

And that's why the ' character is in ASCII.

Print a lowercase e, backspace, and grave accent ', and you have printed é.

Underscore, ^, and other characters are there specifically for overstriking.

Not many terminals do that any more, though...

ChrisArchitect• 5 years ago

this post is based on author's experience -- Unicode issues and encoding problems have continued to be a thing well into the 2010s despite unicode gradually becoming the web defacto (in the West anyways) as web 2.0 ran it's course. I haven't worked on a many international language sites in 4 or 5 years as much but so many of those other regions still have allll of the encoding issues and challenges.

Life Before Unicode

Related Stories