You can't just assume UTF-8

https://csvbase.com/blog/9

by calpaterson • 2 years ago

195 421 2 years ago

csvbase.com

bhaney2• 2 years ago

I'm just gonna assume UTF-8

calpaterson6• 2 years ago

榥\ue0af侬펭懃䒥亷

bhaney2• 2 years ago

And a good day to you too, my friend whose input I'm going to discard

duskwuff6• 2 years ago

I'm disappointed that the article doesn't discuss this in more detail. Most byte sequences are not valid UTF-8. If you can decode a message as UTF-8 with no errors, that is almost certainly the correct encoding to use; it's extremely unlikely that some text in another encoding just happened to be perfectly valid as UTF-8. (The converse is not true; most 8-bit text encodings will happily decode UTF-8 sequences to nonsense strings like ðŸš©.)

If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding. But that should be a fallback, not the first thing you try.

actionfromafar3• 2 years ago

Not extremely unlikely. Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.

Izkata2• 2 years ago

If UTF-8 decoding fails, then it's time to pull out the fancy statistical tools to (unreliably) guess an encoding.

Don't really even need to do that. There's only a handful other encodings still in common use, just try each of them as fallbacks and see which one works without errors, and you'll manage the vast majority of what's not UTF-8.

(We recently did just that for a system that handles unreliable input, I think I remember our fallback only has 3 additional encodings before it gives up and it's been working fine)

klysm2• 2 years ago

You can’t just do that! /s

jujube34• 2 years ago

Actually, I can just assume UTF-8, since that's what the world standardized on. Just like I can assume the length of a meter or the weight of a gram. There is no need to have dozens of incompatible systems.

qwertox• 2 years ago

You can't assume it if you're handed a random sample of text files.

duskwuff6• 2 years ago

You can start by assuming UTF-8, then move on to other heuristics if UTF-8 decoding fails. UTF-8 is "picky" about the sequence of bytes in multi-byte sequences; it's extraordinarily unlikely that text in any other encoding will satisfy its requirements.

(Other than pure ASCII, of course. But "decoding" ASCII text as UTF-8 is safe anyway, so that hardly matters.)

plorkyeran• 2 years ago

I in fact can assume it. If the assumption is wrong then that's someone else's problem. 15 years ago I wrote a bunch of code using uchardet to detect encodings and it was a pretty useful feature at the time. In the last decade everything I've touched has required UTF-8 unless it's been interoperating with a specific legacy system which has some other fixed charset, and it's never been an issue.

fl730519• 2 years ago

That comparison doesn't hold.

If you're dealing with lengths, you can get input data in meters, centimeters, millimeters, inches, feet, etc.

If the input data is human heights, would you automatically assume meters even if the input data is "183"?

If the input data is the weight of humans, would you always assume grams, even if the input data is "75"?

asddubs• 2 years ago

would you guess the unit instead of specifying what you expect?

etiennebausson• 2 years ago

Why would you assume gram when the SI unit is kg and the imputed value fit?

(yeah, my only grip with SI, but it's a frustrating one.)

nijave• 2 years ago

Microsoft would like a word with you (utf-8-bom & utf-16)

pixl974• 2 years ago

Microsoft wants a word with everybody.... and that word is "pay us more money".

paulddraper• 2 years ago

You can assume encoding is UTF-8, length is in meters, and timezone is UTC.

You just won't always be right.

kazinator6• 2 years ago

The trick is to be wrong in precisely those situations in which you don't care.

pyuser583• 2 years ago

Universal UTF-8 is a hope we aspire to, not a reality we assume.

fiddlerwoaroof• 2 years ago

The way you get that reality is you do the opposite of the recommendation of Postel’s law: be very picky about what you consume and fail loudly if it’s not UTF-8

treflop• 2 years ago

There’s a difference between asssuming and not making a distinction.

Very few developers I’ve met know could make a distinction. They’d see a few off characters and think it’s some one-off bug but it’s because they’re both assuming an encoding.

Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.

jujube34• 2 years ago

OK, for a billion dollars I will write a program that deals with KOI-8 or any other text encoding from the 1960s you care to name.

Please pay up front.

Dylan168072• 2 years ago

Even if you said you’d pay them one billion dollars to fix it, they’d absolutely be unable to.

Unless you want it fixed immediately, then a million dollars should motivate almost any developer to spend a month learning, a month doing, and a few years on vacation. A billion is incomprehensible.

pronoiac• 2 years ago

Archive copy: https://web.archive.org/web/20240429061925/https://csvbase.c...

o11c• 2 years ago

Default UTF-8 is better than the linked suggestion of using a heuristic, but failing catastrophic when old data is encountered is unacceptable. There must be a fallback.

(Note that the heuristic for "is this intended to be UTF-8" is pretty reliable, but most other encoding-detection heuristics are very bad quality)

dandigangi• 2 years ago

Except I can

Veserv3• 2 years ago

Off-topic, but the bit numbering convention is deliciously confusing.

Little-endian bytes (lowest byte is leftmost) and big-endian bits (bits contributing less numerical value are rightmost) are normal, but the bits are referenced/numbered little-endian (first bit is leftmost even though it contributes the most numerical value). When I first read the numbering convention I thought it was going to be a breath of fresh air of someone using the much more sane, but non-standard, little-endian bits with little-endian bytes, but it was actually another layered twist. Hopefully someday English can write numbers little-endian, which is objectively superior, and do away with this whole mess.

kstrauser8• 2 years ago

Hopefully someday English can write numbers little-endian, which is objectively superior

Upon reading this, I threw my laptop out the window.

Veserv3• 2 years ago

It actually is if we did not need to consider historical baggage.

Especially in programming where we already use in-band encoding like 0x to denote a hex string or 0b to denote a binary string. I like using 1{s}, e.g. 1x to denote a little-endian hex string and 1b to denote a little-endian binary string, to denote little-endian encoding.

But, even ignoring programming, it is still better in normal use. The Arabic language got it right writing little-endian (Arabic numbers are written the same, but Arabic is a right-to-left language so it is actually little-endian), and the European languages just stole it stupid by copying the form instead of the function.

CoastalCoder2• 2 years ago

Upon reading this, I threw my laptop out the window.

And having shattered the dirty window, your room was enlightened.

bandyaboot2• 2 years ago

In the most popular character encoding, UTF-8, character number 65 ("A") is written:
01000001
Only the second and final bits are 1, or "on".

Isn’t it more accurate to say that the first and penultimate bits are 1, or “on”?

fl730519• 2 years ago

It depends on whether your bit numbering is like x86 (your description), or PowerPC (left most bit is 0).

duskwuff6• 2 years ago

Basically everyone uses x86 bit numbering. It has the pleasant property that the place value of every bit is always 2^n (or -2^n for a sign bit), and zero-extending a value doesn't change the numbering of its bits.

bandyaboot2• 2 years ago

The more I thought it through, even assuming x86, I guess there’s just no “correct” way to casually reference bit positions when we read them in the opposite order from the machine. Are they being referenced from the perspective of a human consumer of text, or the machine’s perspective as a consumer of bits? If I were writing that content, I’d have a difficult time deciding on which to use. If I were writing for a lay person, referencing left-to-right seems obvious, but in this case where the audience is primarily developers, it becomes much less obvious.

mark-r6• 2 years ago

Thanks for giving me an example of an architecture where the bits are labelled backwards, I'd never encountered that before. I've always appreciated that the bit number represents 2 to the power of that number.

skerit• 2 years ago

This confused me too. Until reading this I didn't even think much about how I read binary numbers right-to-left by default.

vitaut• 2 years ago

This is so spectacularly outdated. KOI-8 has been dead for ages.

JonChesterfield• 2 years ago

How about assume utf-8, and if someone has some binary file they'd rather a program interpret as some other format, they turn it into utf-8 using a standalone program first. Instead of burning this guess-what-bytes-they-might-like nonsense into all the software.

We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

zarzavat• 2 years ago

Agreed. Continuing to support other encodings is like insisting that cars should continue to have cassette tape players.

It’s much easier to tell the people with old cassette tapes to rip them, rather than try to put a tape player in every car.

fl730519• 2 years ago

It’s much easier to tell the people with old cassette tapes to rip them

I assume you mean "rip them", as in transcode to a different format?

In that case, you need a tool that takes the old input format(s) and convert them to the new format.

For text files, you'd need a tool that takes the old text files with various encodings and converts them to UTF-8.

Isn't the point of the article to describe how an engineer would create such a tool?

gwervc• 2 years ago

UTF-8 uses 50% more bytes than UFT-16 to encode Chinese or Japanese texts.

amarshall• 2 years ago

some software probably does.

Browsers do, kind of https://mimesniff.spec.whatwg.org/#rules-for-identifying-an-...

fl730519• 2 years ago

they turn it into utf-8 using a standalone program first

I took the article to be for people who would be writing that "standalone program"?

I have certainly been in a position where I was the person who had to deal with input text files with unknown encodings. There was no-one else to hand off the problem to.

0xffff22• 2 years ago

In that case, you should be explicitly asking the user what the input format is.

ryandrake2• 2 years ago

We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser."

1: https://en.wikipedia.org/wiki/Robustness_principle

jerf3• 2 years ago

We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. This is in some sense relatively recent, so word is still getting around, as the programming world does not move as quickly as it fancies itself to.

If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.

travisb• 2 years ago

Among other things, security became a concern.

Being lenient is all well and good when the consequences are mild. When the consequences of misinterpreting or interpreting differently to a second implementation becomes costly, such as a security exploit, then the Robustness Principle becomes less obviously a win.

It's important to understand that every implementation will try to fix-up formatting problems in their own way unique to their particular implementation. From that you get various desync or reinterpretation attacks (eg. HTTP request smuggling).

thfuran4• 2 years ago

It was a horrible idea. The real robustness principle is "Follow the spec".

tedunangst• 2 years ago

Seems like you've uploaded a jpeg. Let me OCR that into CSV for you. Hmm, no text found. Let's pass it to a multimodal LLM.

arp2422• 2 years ago

What ever happened to the Robustness Principle

Bush hid the facts

david422• 2 years ago

"deals with it,"

This basically loses data integrity if it's wrong though.

You might want to do that with human input if it's helpful to the user - ie user enters a phone number and you strip dashes etc. But if it's machine to machine, it should just follow the spec.

kazinator6• 2 years ago

What happened to it:

https://en.wikipedia.org/wiki/Robustness_principle#Criticism

Postel's Law doesn't pass a software engineering smell test.

The idea that software should guess and repair bad inputs is deeply flawed. It is a security threat and a source of enshittification.

jppittma• 2 years ago

I think people have collectively decided that they want their programs stupid and predictable, rather than smart and unwieldy.

vidarh• 2 years ago

That is fine in contexts where a wrong guess does no harm.

But that is not always the case, and e.g. silently "fixing" text encoding issues can often corrupt the data if you get it wrong.

By all means offer options of you want, but if you do flag very clearly to the user that they're taking a risk of corrupting the data unless any errors are very apparent and trivial to undo.

lamontcg• 2 years ago

I've lived through dealing with non-UTF8 encoding issues and it was a truly gigantic pain in the ass. I'm much more on the side now of people who only want to deal with UTF8 and fully support software that tells any other encoding to go pound sand. The harder life gets for people who use other encodings (yes, particularly Microsoft) the more incentive they have to eventually get on board and stop costing everyone time and effort managing all this nonsense.

kelnos11• 2 years ago

The article addresses this, that current thinking in many places is that the robustness principle / Postel's Law maybe wasn't the best idea.

If you reject malformed input, then the person who created it has to go back and fix it and try again. If you interpret malformed input the best you can (and get it right), then everyone else implementing the same thing in the future now also has to implement your heuristics and workarounds. The malformed input effectively becomes a part of the spec.

This is why HTML & CSS are the garbage dump they are today, and why different browsers still don't always display pages exactly alike. The reason HTML5 exists is because people finally just gave up and decided to standardize all the broken behavior that was floating around in the wild. Pre-HTML5, the web was an outright dumpster fire of browser compatibility issues (as opposed to the mere garbage dump we have today).

Anyway, it's not really important to try to convince you that Postel's Law is bad; what's important is that you know that many people are starting to think it's bad, and there's no longer any strong consensus that it was ever a good thing.

ezoe4• 2 years ago

I doubt you can handle UTF-8 properly with that attitude.

The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows.

It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you.

Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC.

These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

dgellow• 2 years ago

I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

int_19h3• 2 years ago

If you're writing low level code, you have to convert UTF-8 to UTF-16 and back.

It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you.

For a library, yeah.

josephg8• 2 years ago

Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back.

Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts.

The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters.

When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.

magicalhippo• 2 years ago

We make some B2B software running on Windows, integrating with customer systems. We get a lot of interesting files.

About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far.

Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare.

Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.

thayne• 2 years ago

Apple macOS use NFD while other mostly use NFC.

It's actually worse than that.

Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD.

OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.

morpheuskafka• 2 years ago

I got a ticket about some "folders with Chinese characters" showing up on an SMB share at work, my first thought was a Unicode issue and sure enough when you combine two UTF-8/ASCII A-z code points together as one UTF-16 code point, it will usually wind up in the CJK Common Ideographs range of Unicode. Some crappy software had evidently bypassed the appropriate Windows APIs and just directly wrote a C-style ASCII string onto the filesystem without realizing that NTFS is UTF-16.

surfingdino• 2 years ago

Microsoft Windows is a source of many a headache for me as almost every other client I write code for has to deal with data created by humans using MS Office. Ordinary users could be excused, because they are not devs but even devs don't see a difference between ASCII and UTF-8 and continue to write code today as if it was 1986 and nobody needed to support accented characters.

0xEF• 2 years ago

Do you know of a resource that explains character encoding in greater detail? Just for my own curiosity. I am learning web development and boy, they brow beat UTF-8 upon us which okay, I'll make sure that call is in my meta data, but none bother to explain how or why we got to that point, or why it seems so splintered.

Comma29762• 2 years ago

Some[1] would see breaking Windows as a feature

[1]Me, surely at least 1 other

SuperNinKenDo• 2 years ago

Not every encoding can make a round trip through Unicode without you writing ad hoc handling code for every single one. There's a number of reasons some of these are still in use and Unicode destroying information is one of them.

998244353• 2 years ago

Non-technical users don't want to do that, and won't understand any of that. That's the unfortunate reality of developing software for people.

If Excel generates CSV files with some Windows-1234 encoding, then my "import data from CSV" function needs to handle that, in one way or another. A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding, they won't care that Microsoft is using obsolete or weird defaults. They will see it as a bug in my program and demand that I fix my software. Even if Excel offers them a choice in encoding, they won't understand any of that and more importantly they don't want to deal with that right now, they just want the thing to work.

unclebucknasty7• 2 years ago

A significant number of people generate CSV files from Excel and if these people upload these files into my application and I assume an incorrect encoding...

What if you just gave them instructions?

edflsafoiewq• 2 years ago

OTOH if no one ever pushes back against entropy, your great-grandkids will still be dealing with Windows-1234 problems in 2100.

vaylian• 2 years ago

Plus, Excel really likes to use semicolons instead of commas for comma-separated-files. That's another idiosyncrasy that programmers need to take into account.

berkes• 2 years ago

then my "import data from CSV" function needs to handle that, in one way or another.

It doesn't. Well, maybe "another".

Your function or even app doesn't need to handle it. Here's what we did on a bookkeeping app: remove all the heuristics, edge cases, broken-csv-handling and validation from all the CVS ingress points.

Add one service that did one thing: receive a CSV, and normalize it. All ingress points could now assume a clean, valid, UTF8 input. It removed thousands of LOCs, hundreds of issues from the backlog. It made debugging easy and it greatly helped the clients even.

At some point, we offered their import runs for download, we added the [original name]_clean.csv our normalized versions. Got praise for that. Clients loved that, as they were often well aware of their internal mess of broken CVSs.

xorcist• 2 years ago

The point is to separate the cleaning stage from the import stage. Having a clean utf-8 csv makes debugging import issues so much easier. And there are several well working csv tools such as the Python one, that can not only detect character encodings but also record separators and various quotation idiosyncrasies that you also need to be aware of when dealing with Microsoft Office files. Other people have already thought long and hard about that stuff so you don't have to.

kelnos11• 2 years ago

Solutions that require lots of unrelated people to start doing something a different way are not really solutions.

thaumasiotes2• 2 years ago

We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

Browsers used to have a menu option to choose the encoding you wanted to use to decode the page.

In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this.

They seem to be in the process of disabling that option too:

Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported.

( https://support.mozilla.org/en-US/kb/text-encoding-no-longer... )

This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!

n_plus_1_acc• 2 years ago

Probably because most websites now send a correct encoding header or meta Tag, so the user changing can only make it wrong. (Assuming no encoding header is wrong, which happens in reality)

shiomiru• 2 years ago

It seems the decision was made in the name of security:

https://hsivonen.fi/no-encoding-menu/

Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run.
Elaborate UI surface for a niche feature risks the whole feature getting removed
Telemetry [...] suggested that users aren’t that good at choosing correctly manually.

In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)

dublin• 2 years ago

Make your life easy. Assume 7-bit ASCII. No one needs all those other characters, anyway...

AlienRobot2• 2 years ago

Do we really need 128 permutations just to express an alphabet of 26 letters?

I think we should use a 4 bit encoding.

0 - NUL

1-7 - aeiouwy

8 - space

9-12 - rst

13-15 - modifiers

When modifier bits are set, the values of the next half-byte change to represent the rest of the alphabet, numbers, symbols, etc. depending on the bits set.

apantel• 2 years ago

R S T L N E

Dylan168072• 2 years ago

I like it, but what if we use 2 to pick symbols, 3 to pick numbers, and 4 and 5 to pick capital letters.

jonathankoren• 2 years ago

As someone who has repeatedly had to deal with Unicode nonsense; I wholeheartedly agree. Also, you don’t need accents. You just need to know how to read and have context. See: live and live, read and read, etc.

drdaeman• 2 years ago

Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024" then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun.

Or, you know, just say "nah, I can, those ancient stuff don't matter (outside of obligatory exceptions, like software archeology) anymore." If someone wants to feed me a KOI8-R or JIS X 0201 CSV heirloom, they should convert it into something modern first.

skissane• 2 years ago

Anyone got EBCDIC on their bingo cards? Because if the argument is "legacy encodings are still relevant in 2024"

I have a hobby interest in IBM mainframes and IBM i, so yes to EBCDIC for me. (I have encountered them professionally too, but only to a very limited extent.) In practice, I find looking for 0x40 (EBCDIC space) a useful heuristic. Even in binary files, since many mainframe data structures are fixed length space padded.

then we also need to bring EBCDIC (and EBCDIK and UTF-EBCDIC for more perverted fun) into the picture. Makes heuristics extra fun.

Actual use of UTF-EBCDIC, while not nonexistent, has always been extremely rare. A person could spend an entire career dedicated to IBM mainframes and never encounter it

EBCDIK, at first I wondered if that was a joke, now I realise it is a name used for non-IBM Japanese EBCDIC code pages. Again, something one can spend a whole career in mainframes and never encounter – if one never works in Japan, if one works for a vendor whose products aren't sold in Japan, probably even if you work for a vendor whose products are sold in Japan but only to IBM sites (as opposed to Fujitsu/Hitachi sites)

AlienRobot2• 2 years ago

"You can't assume a 32 bit integer starts from 0"

Karellen• 2 years ago

Don't worry, I never assume UTF-8.

I require UTF-8. If it isn't currently UTF-8, it's someone else's problem to transform it to UTF-8 first. If they haven't, and I get non-UTF-8 input, I'm fine bailing on that with a "malformed input - please correct" error.

fl730519• 2 years ago

That works until you can't pay your bills unless you take a new contract where you have to deal with a large amount of historical text files from various sources.

Then it's no longer "someone else's problem".

cheeze• 2 years ago

Even then, at that point I'm writing an adapter to convert into UTF-8 JIT in front of my service. Or modifying my program when that contract comes up, but I'm not going to waste time proactively safeguarding against a what-if that may come up at some point.

kazinator6• 2 years ago

If you have to deal with a large number of historic text files that are not in UTF-8, why, that's a chance to promote UTF-8!

ahi2• 2 years ago

a large amount of historical text files from various sources

Sounds like academia. "You" can't pay your bills anyway.

groestl• 2 years ago

I will assume it, I will enforce it where I can, and I will fight tooth and nail should push come to shove.

I got 99 problems, but charsets aint one of them.

zadokshi• 2 years ago

Better to assume UTF8 and fail with a clear message/warning. Sure you can offer to guess to help the end user if it fails, but as other people have pointed out, it’s been standard for a long time now. Even python caved and accepted it as the default: https://peps.python.org/pep-0686/

smeagull• 2 years ago

I absolutely can. If it's not UTF-8, I assume it's worthless.

koito172• 2 years ago

The comments in this thread are a bit amusing.

I wish I could live in the world where I could bluntly say "I will assume UTF-8 and ignore the rest of the world". Many Japanese documents and sites still use Shift JIS. Windows has this strange Windows-932 format that you will frequently encounter in CUE files outputted by some CD ripping software. ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208. These special characters are mostly icons used in traffic and weather reports, but transcoding to UTF-8 still causes trouble with these icons.

bongodongobob3• 2 years ago

Most people on this site probably live in the world where everything is done in English. That's the norm for the vast majority of businesses and people in the US.

fl730519• 2 years ago

Even for those people, there is still a ton of old text files in Windows-1252 etc floating around.

You can choose to never work on projects where you have to deal with files like that.

But there may come a day where you have to choose between not paying your rent or writing a tool that converts old textfiles to UTF-8. At that point, it's nice to have references on the internet on how other people have actually dealt with it and what works. "Abort with an error" is not very useful advice then.

samatman3• 2 years ago

It's more like "I will assume UTF-8 and ignore edge case encoding problems which still arise in Japan, for some strange reason".

We are not running short on Unicode codepoints. I'm sure they can spare a few more to cover the Japanese characters and icons which invariably get mentioned any time this subject comes up on HN. I don't know why it hasn't happened and I won't be making it my problem to solve. Best I can do is update to version 16 when it's released.

koito172• 2 years ago

I mention Japanese because I deal with Japanese text daily. I could mention some Chinese documents and sites using GBK to save space (since such encodings use exactly 2 bytes per character whereas the average size in UTF-8 is strictly larger than 2 bytes). But I am not very familiar with it. Overall, I would not say these are "strange reasons".

kelnos11• 2 years ago

Unless the Unicode Consortium decides to undo the Han Unification stuff, I don't think it's going to get better for Japanese users, and programmers who build for a Japanese audience will have to continue to suffer with Shift-JIS.

lifthrasiir2• 2 years ago

ARIB STD-B24, the captioning standard used in Japanese television, has its own text encoding with characters not found in either JIS X 0201 or JIS X 0208.

Amazingly enough, ARIB STD-B24 is one of the major source of Unicode emojis. So transcoding would actually work for them! (I am aware of some exceptions, but semantically there is no loss.) Unicode and UTF-8 are truly eating every other legacy encoding else, so much so that it is becoming more reasonable to have a separate transcoding step.

msla• 2 years ago

But remember, adding emoji to a character encoding standard is Morally Bad, somehow, and Proof Of Intellectual Decay In The Modern World. Also, Unicode invented emoji to Sap And Impurify Our Bodily Fluids and there is no reason for any of them to exist.

norir• 2 years ago

If it's turtles all the way down and at every level you use utf-8, it's hard to see how any input with a different encoding (for the same underlying text) will not be detected before any unintended side effects were invoked.

At this point, I don't see any sufficiently good reason to not use utf-8 exclusively in any new system. Conversions to and from other encodings would only be done at well defined boundaries when I'm calling into dependencies that require non utf-8 input for whatever reason.

kstrauser8• 2 years ago

If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there.

Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8.

If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.

kccqzy• 2 years ago

For instance, if it were the case that it did a bad job of encoding Mandarin

I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.

PeterisP3• 2 years ago

A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.

kstrauser8• 2 years ago

That doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant.

Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.

gary_0• 2 years ago

For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings.

[0] https://utf8everywhere.org/#asian

hnick4• 2 years ago

It depends on the domain. If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.

Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.

kstrauser8• 2 years ago

Those aren’t timestamps. They’re descriptions of how to select them.

cesarb2• 2 years ago

If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.

If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.

LaffertyDev2• 2 years ago

I can't help myself. The grandest of nitpicks is coming your way. I'm sorry.

If you give me a computer timestamp without a timezone, I can and will assume it's in UTC.

Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself.

The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken.

My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm.

But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.

Spivak5• 2 years ago

It's always nice to see someone who actually understands time.

"Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.

kstrauser8• 2 years ago

As I mentioned in another reply, "remind me every day at 7PM" isn't a timestamp. It's a formula for how to determine when the next timestamp is going to occur. Even those examples are too narrow, because it's really closer to "remind me the next time you notice that it's after 7PM wherever I happen to be, including if that's when I cross a time zone and jump instantly from 6:30PM to 7:30PM".

Consider my statement more in the context of logs of past events. The only time you can reasonably assume a given file is in a particular non-UTC TZ is when it came from a person sitting in your same city, from data they collected manually, and you're confident that person isn't a time geek who uses UTC for everything. Otherwise there's no other sane default when lacking TZ/offset data. (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).

logrot• 2 years ago

Fundamentally I agree, but sadly the world isn't that simple.

You usually end up with having to deal with whatever eccentric sh!t that ultimately comes from the same source as the payment for the job.

layer86• 2 years ago

For instance, if it were the case that it did a bad job of encoding Mandarin

Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.

Shorel• 2 years ago

That makes Unicode a failure then

jcranmer3• 2 years ago

I haven't seen discussion of this point yet, but the post completely fails to provide any data to back up its assertion that charset detection heuristics works, because the feedback I've seen from people who actually work with charsets is that it largely doesn't (especially if you're based on naive one-byte frequency analysis). Okay, sure, it works if you want to distinguish between KOI8-R and Windows-1252, but what about Windows-1252 and Windows-1257?

See for example this effort in building a universal charset detector in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1551276

toast0• 2 years ago

I've done some charset detection, although it's been a while. Heuristics kind of work for somethings --- I'm a big fan of if it's decodable as utf-8, it's probably utf-8, unless there's zero bytes (in most text). If there's a lot of zero bytes, maybe it's UCS-2 or UTF-16, and you can try to figure out the byte order and if it decodes as utf-16.

If it doesn't fit in those categories, you've got a much harder guessing game. But usually you can't actually ask the source what it is, because they probably don't know and might not understand the question or might not be contactable. Usually, you have to guess something, so you may as well take someone else's work to guess, if you don't have better information.

renhanxue• 2 years ago

Yeah. The fantastic python library ftfy ("fixes text for you", https://ftfy.readthedocs.io/en/latest/index.html), designed to fix mangled Unicode (mojibake, of many different varieties), mentions in its docs that heuristic encoding guessers are the cause of many of the problems ftfy is designed to fix. It's magical, by the way.

It has an entire section of its documentation dedicated to why you should not use heuristic charset detection: https://ftfy.readthedocs.io/en/latest/avoid.html#don-t-use-c...

torstenvl2• 2 years ago

That section explains why not to use a specific naive charset detection library that doesn't have a strong prior for UTF-8. There's no basis for extrapolating that further.

mgaunard• 2 years ago

There is a pretty successful world language standard: English.

bawolff10• 2 years ago

Kids these days. Latin was a much more elegant standard.

CoastalCoder2• 2 years ago

Yeah, but: Quis legentibus scribit?

stalfosknight• 2 years ago

You can on all of Apple's platforms.

hchak• 2 years ago

Hoping that LLMs can solve our “Tower of Babel” problem… :)

Dwedit2• 2 years ago

But you can assume non-UTF-8 upon seeing an invalid UTF-8 byte sequence. From there, it can be application-specific depending on what encoding you expect to see.

(And of course UTF-16 if there's a BOM)

klysm2• 2 years ago

Meh I'd rather start making those assumptions and blame everything that doesn't use UTF-8 as broken.

lifthrasiir2• 2 years ago

You can't just assume UTF-8, but you can verify that it is almost surely encoded in UTF-8 unlike other legacy encodings. Which makes UTF-8 the first and foremost consideration.

Havoc• 2 years ago

You underestimate my willingness to happy path code...

hnick4• 2 years ago

Based on my past role, you can't even assume UTF-8 when the file says it's UTF-8.

Clients would constantly send CSV or other files with an explicit BOM or other marking indicating UTF-8 but the parser would choke since they just output native Windows-1252 or similar into it. I think some programs just spit it out since it's standard.

kazinator6• 2 years ago

Indeed, you can't assume UTF-8.

What you do, rather, is drop support for non-UTF-8.

Work with tech-stacks whose text handling is based strictly around Unicode and UTF-8, and find enough opportunities that way that you don't have to care about anything else.

Let the customers who cling to data in weird encodings go to someone who makes it a nice to support that.

djha-skin• 2 years ago

Joel spolsky spoke against this exact statistics-based approach when he wrote about unicode[1]:

What do web browsers do if they don’t find any Content-Type, either in the http headers or the meta tag? Internet Explorer actually does something quite interesting: it tries to guess, based on the frequency in which various bytes appear in typical text in typical encodings of various languages, what language and encoding was used. Because the various old 8 bit code pages tended to put their national letters in different ranges between 128 and 255, and because every human language has a different characteristic histogram of letter usage, this actually has a chance of working. It’s truly weird, but it does seem to work often enough that naïve web-page writers who never knew they needed a Content-Type header look at their page in a web browser and it looks ok, until one day, they write something that doesn’t exactly conform to the letter-frequency-distribution of their native language, and Internet Explorer decides it’s Korean and displays it thusly, proving, I think, the point that Postel’s Law about being “conservative in what you emit and liberal in what you accept” is quite frankly not a good engineering principle.

1: https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

bastawhiz2• 2 years ago

Exactly. I used this technique at Mozilla in 2010 when processing Firefox add-ons, and it misidentified scripts as having the wrong encoding pretty frequently. There's far less weird encoding out there than there are false positives from statistics-based approaches.

mark-r6• 2 years ago

20 years old, but still true.

kelnos11• 2 years ago

I don't think he was speaking against the statistics-based approach itself, just against Postel's Law in general.

Ideally people would see gibberish (or an error message) immediately if they don't provide an encoding; then they'll know something is wrong, figure it out, fix it, and never have the issue again.

But if we're in a situation where we already have lots and lots of text documents that don't have an encoding specified, and we believe it's not feasible to require everyone to fix that, then it's actually pretty amazing that we can often correctly guess the encoding.

vbezhenar• 2 years ago

There's enca library (and cli tool) which does that. I used it often before UTF-8 became overwhelming. The situation was especially dire with Russian encodings. There were three 1-byte encodings which were quite wide-spread: KOI8-R mostly found in unixes, CP866 used in DOS and CP1251 used in Windows. What's worse, with Windows you sometimes had to deal with both CP866 and CP1251 because it includes DOS subsystem with separate codepage.

rch• 2 years ago

Don't assume; force UTF-8.

ahi2• 2 years ago

Why utf-8 when we have MARC-8? https://en.wikipedia.org/wiki/MARC-8

moefh• 2 years ago

I had never heard of MARC-8 before, but after a quick google search:

"There are no defined MARC-8 character sets for Armenian, Bengali, Devanagari, Ethiopic, Syriac, Tamil or Thai."[0]

"If a character is not representable in MARC-8 of a MARC-21 record, then UTF-8 must be used instead."[1]

[0] https://help.oclc.org/Metadata_Services/Connexion/Connexion_...

[1] https://en.wikipedia.org/wiki/MARC-8

hot_gril2• 2 years ago

Java and Javascript both use UTF-16 for internal string representation, even though JSON specifies UTF-8. Windows APIs too. I'm still not sure why, but it means that one char uses at least 2 bytes even if it's in the ASCII range.

mark-r6• 2 years ago

Early adopters of Unicode used the first available encoding, UCS-2. UTF-16 is an extension of that to handle the increased range of code points that came later.

AzzyHN• 2 years ago

I'm not a programmer, but how hard is it to write something that just checks what encoding the file uses. Or at least does its best to guess?

bastawhiz2• 2 years ago

It's not easy! Go check out the source for chardet (Python). It's not pretty.

torstenvl2• 2 years ago

Pretty hard. There's no true way of knowing when you're right. You can make educated guesses based on statistical likelihood of certain patterns, but nothing stops people from constructing text that happens to look more "normal" when interpreted as another character set.

If I see the bytes 0xE5 0xD1 0xCD 0xC8 0xC7, then it is a decent bet that it's intended to be مرحبا in DOS 708. But there's no definitive reason why it couldn't be σ╤═╪╫ (DOS 437) or åÑÍÈÇ (Windows-1252) or Еямих (KOI8-RU). Especially if you don't know for sure that the data is intended to be natural language.

I can be pretty reasonably certain that it isn't, say, Japanese. 0xE5 in ShiftJIS is the first of a double-byte sequence, and it's odd, so the next byte would have to come from the range 0x40-0x9E, and 0xD1 does not fall within that range. In other words, it simply isn't valid ShiftJIS (it also isn't valid UTF-8). So you can narrow down the possibilities... but you're still making an educated guess. And you have to figure out a way to program that fuzzy, judgment-y analysis into software.

hgs3• 2 years ago

I've written code to do this. If you're lucky there will be a BOM (byte order mark) or MIME type to indicate the encoding form. In this case you know the encoding. If you don't have this information, then you must guess the encoding. The issue is guessing may not produce accurate results, especially if you don't have enough bytes to go by.

The program I wrote to guess the encoding would scan the bytes in multiple passes. Each pass would check if the bytes encoded valid characters in some specific encoding form. After a pass completed I would assign it a score based on how many successfully encoded characters were (or were not) found. After all passes completed I'd pick the highest score and assume that was the encoding. This approach ended up being reasonably reliable assuming there were enough bytes to go by.

jheriko2• 2 years ago

Fuck me it's 2024

2014 me shaking his head in ways 2004 me saw coming in 1994.

Im only 40.

FUCK

pylua• 2 years ago

Stupid question: how are the headers passed for http? What encoding describes the encoding ?

a57721• 2 years ago

The RFCs for HTTP require headers to be in US-ASCII.

mark-r6• 2 years ago

I think the headers are pure ASCII, which as noted is a subset of almost all encodings.

missblit• 2 years ago

Encodings I have used or been exposed to in my career: ASCII, Latin1, Windows-1252, UTF-8, UTF-16, UTF-32, GBK, Zawgyi, Shift-JIS

timoteostewart• 2 years ago

Fascinating topic. There are two ways the user/client/browser receives reports about the character encoding of content. And there are hefty caveats about how reliable those reports are.

(1) First, the Web server usually reports a character encoding, a.k.a. charset, in the HTTP headers that come with the content. Of course, the HTTP headers are not part of the HTML document but are rather part of the overhead of what the Web server sends to the user/client/browser. (The HTTP headers and the `head` element of an HTML document are entirely different.) One of these HTTP headers is called Content-Type, and conventionally this header often reports a character encoding, e.g., "Content-Type: text/html; charset=UTF-8". So this is one place a character encoding is reported.

If the actual content is not an (X)HTML file, the HTTP header might be the only report the user/client/browser receives about the character encoding. Consider accessing a plain text file via HTTP. The text file isn't likely to itself contain information about what character encoding it uses. The HTTP header of "Content-Type: text/plain; charset=UTF-8" might be the only character encoding information that is reported.

(2) Now, if the content is an (X)HTML page, a charset encoding is often also reported in the content itself, generally in the HTML document's head section in a meta tag such as '<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>' or '<meta charset="utf-8">'. Now just because an HTML document self-reports that it uses a UTF-8 (or whatever) character encoding, that's hardly a guarantee that the document does in fact use said character encoding.

Consider the case of a program that generates web pages using a boilerplate template still using an ancient default of ISO-8859-1 in the meta charset tag of its head element, even though the body content that goes into the template is being pulled from a database that spits out a default of utf-8. Boom. Mismatch. Janky code is spitting out mismatched and inaccurate character encoding information every day.

Or to consider web servers. Consider a web server whose config file contains the typo "uft-8" because somebody fat-fingered while updating the config (I've seen this in random web pages.). Or consider a web server that uses a global default of "utf-8" in its outgoing HTTP headers even when the content being served is a hodge-podge of UTF-8, WINDOWS-1251, WINDOWS-1252, and ISO-8859-1. This too happens all the time.

I think the most important takeaway is that with both HTTP headers and meta tags, there's no intrinsic link between the character encoding being reported and the actual character encoding of the content. What a Web server tells me and what's in the meta tag in the markup just count as two reports. They might be accurate, they might not be. If it really matters to me what the character encoding is, there's nothing for it but to determine the character encoding myself.

I have a Hacker News reader, https://www.thnr.net, and my program downloads the URL for every HN story with an outgoing link. I have seen binary files sent with a "UTF-8" Content-Type header. I have seen UTF-8 files sent with a "inode/x-empty" Content-Type header. My logs have literally hundreds of goofy inaccurate reports of content types and character encodings. Because I'm fastidious and I want to know what a file actually is, I have a function `get_textual_mimetype` that analyzes the content of what the URL's web server sends me. My program downloads the content and uses tools such as `iconv` and `isutf8` to get some information about what encoding it might be. It uses `xmlwf` to check if it's well-formed XML. It uses `jq` to check whether it's valid JSON. It uses `libmagic`. There's a lot of fun stuff the program does to pin down with a high degree of certainty what the content is. I want my program to know whether the content is an application/pdf, an iamge/webp, a text/html, an application/xhtml+xml, a text/x-csrc, or whatever. Only a rigorous analysis will tell you the truth. (If anyone is curious, the source for `get_textual_mimetype` is in the repo for my HN reader project: https://github.com/timoteostewart/timbos-hn-reader/blob/main... )

neonsunset3• 2 years ago

I will assume UTF-8 and if it's not, it will be your fault :)

flohofwoe2• 2 years ago

The post seems to assume that only UTF-16 has Byte Order Marks, but as pointless as it sounds, UTF-8 has a BOM too (EF BB BF). It seems to be a Windows thing though, haven't seen it in the wild anywhere else (and also rarely on Windows - since text editors typically allow to save UTF-8 files with or without BOM. I guess it depends on the text editor which of those is the default).

calpaterson6• 2 years ago

Yes you're right, UTF-8 technically does as well. I've never seen them in real life either.

UTF-16 BOMs do have a useful function as I recall: they really help Excel detect your character encoding (Excel is awful at detecting character encoding).

LinAGKar• 2 years ago

That's not really a byte order mark though, it's just the UTF-8 encoding of U+FEFF, which corresponds to the byte order mark in UTF-16. Honestly, emitting that into UTF-8 was probably the result of a bug originally, caused by Windows Unicode APIs being designed for UTF-16.

flohofwoe2• 2 years ago

Wikipedia begs to differ though (check under "UTF-8": https://en.wikipedia.org/wiki/Byte_order_mark)

Also, at least the Rider IDE and VSCode lets you select to save UTF-8 with or without BOM.

My guess is that it's used mainly to differentiate between UTF-8 and codepage encoded 8-bit ASCII files (which are still very common on Windows).

mseepgood• 2 years ago

Yes, I can.

matheusmoreira• 2 years ago

Yeah but I'm gonna do it anyway. If it's not UTF-8 it's terrible and broken and not worth supporting unless some serious cash is on the table.

mschuster91• 2 years ago

CSV files, in particular, have no way to signal, in-band, which encoding is used.

That's actually wrong. Add an UTF-8 BOM, that's enough for Excel (and some other libraries) to know what is going on [1].

[1] https://csv.thephpleague.com/8.0/bom/

vkaku2• 2 years ago

The probability of web content not in UTF-8 is increasingly getting lower and lower.

Last I tracked, as of this month, 0.3% of surveyed web pages used Shift JIS. It has been declining steadily. I really hope people move to UTF-8. While it is important to understand how the code pages and encodings helped, I think it's a good time to actually start moving a lot of applications to use UTF-8. I am perfectly okay if people want to use UTF-16 (the OG Unicode) and it's extensions alternatively, especially for Asian applications.

Yes, historic data preservation requires a different strategy than designing stuff for the future. It is okay to however migrate to these encodings and keep giving old data and software new life.

layer86• 2 years ago

Just to be pedantic, the OG Unicode is UCS-2, not UTF-16, the main difference being that surrogate characters didn’t exist originally.

vkaku2• 2 years ago

Good point. Ah, we peel onions every few years.

eqvinox3• 2 years ago

The solution, obviously, is to train an LLM to recognize the character set.

layer86• 2 years ago

That sounds like an efficient use of computing resources.

tanin• 2 years ago

I'm actually having this issue where users import CSV files that don't seem to be valid. DuckDB would throw out an error like: Invalid Closing Quote: found non trimable byte after quote at line 34, Invalid Input Error: Invalid unicode (byte sequence mismatch) detected in value construction, Value with unterminated quote found

One example: pgAdmin can export a database table into a CSV... but the CSV isn't valid for DuckDB to consume. Because, for some odd reason, pgAdmin uses a single quote to escape a double quote.

This blog is pretty timely. Thank you for writing it!

mihaaly• 2 years ago

Excellent article, good content, good length, enlightened subtexts and references, joy to read.

chungy• 2 years ago

The article's pretty weird for presenting little-endian UTF-16 as normal and barely even mentioning that big-endian is an option (in fact, seems to refer to it as "backwards"), even though big-endian is a much more human readable format.

rob74• 2 years ago

30 years ago: "you can't just assume ASCII"

Today: "you can't just assume UTF-8"

The more things change, the more they stay the same...

camgunz2• 2 years ago

Dear lazyweb: I think I read something about Postel's Law being essential to the internet's success -- maybe this was also IPv6 related? Does anyone else remember this?

camgunz2• 2 years ago

(I found it: https://apenwarr.ca/log/20200708)

planede• 2 years ago

A bag of bytes is a bag of bytes. Any encoding should be either assumed by the protocol being used or otherwise specified.

TypicalHog• 2 years ago

IMO humanity should just do ASCII and call it a day. I'm not talking about viability of this, just my wishes.

mikhailfranco6• 2 years ago

Developers should assume UTF-8 for text files going forward.

UTF-8 should have no BOM. It is the default. And there are no undefined Byte sequences that need an Order. Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work?

Others variants of Unicode have BOMs, e.g. UTF-16BE. We know CJK languages need UTF-16 for compression. The BOM is only a couple more bytes. No problem, so far so good.

But there are old files, that are in 'platform encoding'. Fine, let there be an OS 'locale', that has a default encoding. That default can be overridden with another OS 'encoding' variable. And that can be overridden by an application arg. And there may be a final default for a specific application, that is only ever used with one language encoding. Then individual files can override all of the above ...

Text file (serialization) formats should define handling of optional BOM followed by an ASCII header that defines the encoding of the body that follows. One can also imagine .3 filetypes that have a unique or default encoding, with values maintained by, say, IANA (like MIME types). XML got this right. JSON and CSV are woefully neglectful, almost to the point of criminal liability.

But in the absence of all of the above, the default-default-default-default-default is UTF-8.

We are talking about the future, not the past. Design for UTF-8 default, and BOMs for other Unicodes. Microsoft should have defined BOMs for Windows-Blah encodings, not for UTF-8!

When the whole world looks to the future, Microsoft will follow. Eventually. Reluctantly.

calpaterson6• 2 years ago

If csv files bring criminal liability then I am guilty.

Sidenote: this particular criminal conspiracy is open to potential new members. Please join the Committee To Keep Csv Evil: https://discord.gg/uqu4BkNP5G

Jokes aside, talking about the future is grand but the problem is that data was written in the past and we need to read it in the present. That means that you do have to detect encoding and you can't just decide that the world runs on UTF-8. Even today, mainland China does not use UTF-8 and is not, as far as I know, in the process of switching.

I understand UTF-8 is mostly fine even for east asian languages though - and bytes are cheap

mikhailfranco6• 2 years ago

I explained how old files could be read.

Let me reframe it as a Schelling Point [1] - the uncoordinated coordination problem.

You arrange to meet your file on a certain day in New York, no place or time were mentioned. When and where will you go? It seems impossible.

But perhaps you go at noon, to the UTF-8 Building in midtown Manhattan. Are you there now?

[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)

mikhailfranco6• 2 years ago

Old CSV files bring no stigma.

There is no CSV specification. That does bring opprobrium. RFC4180 is from 2005, long after Unicode and XML, so people should have known better. The absence of a standard points to disagreement, perhaps commercial disagreement, but IETF should be independent, is it not?

That failure to standardize encoding, and other issues (headers, etc.), has wasted an enormous amount of time for many creative hackers, who could have been producing value, rather than banging their head against a vague assembly of pseudo-spec-illusions. Me included, sadly.

e63f67dd-065b• 2 years ago

China does not use UTF-8 and is not, as far as I know, in the process of switching.

That’s … not true? Most Chinese software and websites are utf8 by default, and it’s been that way for a while. GBK and her sisters might linger around in legacy formats, but UTF-8 has certainly reached the point of mass adoption from what I can tell.

teknopaul2• 2 years ago

Imho Presume 8 bit, encapsulting 7bit usacsii. That includes utf8 and many many other encodings.

Don't interpret user supplied strings at all. Defines max lengths as byte lengths.

Remain agnostic of encoding. Especially in libraries.

It's easier than people think it is thanks to some very clever people's work a long time ago.

Pet_Ant• 2 years ago

Requiring a UTF-8 BOM just destroys the happy planned property that ASCII-is-UTF8. Why spoil that good work?

So that you know you are dealing with UTF-8. Assuming ASCII only works if you are only dealing with English texts and data.

remram• 2 years ago

Who is assuming ASCII?

themerone• 2 years ago

The BOM only tells you that your are not dealing with ASCII. You could encounter anything after it.

WorldMaker3• 2 years ago

We know CJK languages need UTF-16 for compression.

My understanding is that it is for the opposite of compression: it saves memory when uncompressed versus the UTF-8 surrogates needing more bytes. My understanding is that UTF-8 surrogates compress pretty well as they have common patterns that form dictionary "words" just as easily anything else. UTF-8 seems to be winning in the long run for even CJK and astral plane languages on disk and the operating systems and applications that were preferring UTF-16 in memory are mostly only doing so out of backwards compatibility and are themselves often using more UTF-8 buffers internally as those reflect the files at rest.

(.NET has a backwards compatibility based on using UTF-16 codepoint strings by default but has more and more UTF-8 only pathways and has some interesting compile time options now to use UTF-8 only today. Python 3 made the choice that UTF-8 was the only string format to support, even with input from CJK communities. UTF-8 really does seem to be slowly winning everything.)

JSON and CSV are woefully neglectful,

As the article also points out, JSON probably got it right: UTF-8 only and BOM is an error (because UTF-8) (but parsers are allowed to gently ignore that error if they wish). https://www.rfc-editor.org/rfc/rfc8259#section-8.1

That seems to be the way forward for new text-based formats that only care about backward compatibility with low byte ASCII: UTF-8 only, no BOM. UTF-8 (unlike UTF-16 and missing reservations for some of its needed surrogates) is infinitely expandable if we ever do find a reason to extend past the "astral plane".

(Anyone still working in CSV by choice is maybe guilty of criminal liability though. I still think the best thing Excel could do to help murder CSV is give us a file extension to force Excel to open a JSON file, like .XLJSON. Every time I've resorted to CSV has been because "the user needs to double click the file and open in Excel". Excel has great JSON support, it just won't let you double click a file for it, which is the only problem, because no business executive wants the training on "Data > From JSON" no matter how prominent in the ribbon that tool is.)

When the whole world looks to the future, Microsoft will follow.

That ship is turning slowly. Windows backward compatibility guarantees likely mean that Windows will always have some UTF-16, but the terminals in Windows now correctly default to UTF-8 (since Windows 10) and even .NET with its compatibility decrees is more "UTF-8 native" than ever (especially when compiling for running on Linux, which is several layers of surprise for anyone that was around in the era where Microsoft picked UCS-2 as its one true format in the first place).

PaulHoule3• 2 years ago

You can fit Japanese comfortably in a 16 bit charset but Chinese needs more than that.

My take though is that CSV is not a good thing because the format isn't completely standardized, you just can't count that people did the right thing with escaping, whether a particular column is intended to be handled as strings or numeric values, etc.

Where I work we publish data files in various formats, I'm planning on making a Jupyter notebook to show people how to process our data with Pandas -- one critical point is that I'm going to use one of the commercial statistics data formats (like Stata) because I can load the data right the first time and not look back. (e.g. CSV = good because it is "open" is wrong)

If I am exporting files for Excel users I export an Excel file. Good Excel output libraries have been around for at least 20 years and even if you don't have fun with formulas, formatting and all that it is easy to make a file that people will load right the first time and every time.

PaulHoule3• 2 years ago

Programming languages have lumbered slowly towards UTF-8 by default but from time to time you find an environment with a corrupted character encoding.

I worked at an AI company that was ahead of its time (actually I worked at a few of those) where the data scientists had a special talent for finding Docker images with strange configurations so all the time I'd find out one particular container was running a Hungarian or other wrong charset.

(And that's the problem with Docker... People give up on being in control of their environment and just throw in five different kitchen sinks and it works... Sometimes)

jrochkind13• 2 years ago

The specific use case the OP author was focusing on was CSV. (A format which has no place to signal the encoding inline). They noted that, to this day, Windows Excel will output CSV in Win-1252. (And the user doing the CSV export has probably never even heard of encodings).

If you assume UTF-8, you will have corrupted text.

I agree that I'm mad about Excel outputting Win-1252 CSV by default.

out_of_protocol2• 2 years ago

More pressure on M$ - faster the change. Also, almost any software nowadays assuming UTF8 for plaintext anyway

brnt• 2 years ago

And that's far from the only thing Excel does quietly. It also changes comma's to semi-colons based on system locale. It's braindead.

kdklol• 2 years ago

You can't just assume UTF-8

But I will, because in this day and age, I should be perfectly able to do so. Non-use of UTF-8 should be simply considered a bug and not treating text as UTF-8 should frankly be a you problem. At least for anything reasonably modern, by which I mean made in the last 15 years at least.

teliskr• 2 years ago

I maintain a system I created in 2004 (crazy right). Not sure how we lived; but at the time, emojis were not as much of a thing. This has come to bite me several times.

GnarfGnarf• 2 years ago

Why would anyone use anything other than UTF-8 in this day and age?

Windows took a gamble years ago when the winner was not obvious, we’re stuck with UCS-2, but you can circumvent that with a good string library like QT’s Qstring.

iamcreasy• 2 years ago

By heuristics, is the author referring to the rules and policies published by Unicode? [1]

Link[1] was referring as a solution to this problem in this article: Absolute Minimum Every Software Developer Must Know About Unicode in 2023 [2].

[1] https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Bound...

[2] https://tonsky.me/blog/unicode/

teddyh• 2 years ago

If you’re actually in a position where you need to guess the encoding, something like “ftfy” <https://github.com/rspeer/python-ftfy> (webapp: <https://ftfy.vercel.app/>) is a perfectly reasonable choice.

But, you should always do your absolute utmost not to be put in a situation where guessing is your only choice.

otikik• 2 years ago

However you can check for invalid utf-8 sequences, throw an error with "invalid encoding on byte x, please use valid utf-8" if encountered, and from that point on assume utf-8.

teknopaul2• 2 years ago

I suspect the author does not about the first bit being 1 thing.

utf8 is magic.

You can assume US ASCII for lots of very useful text protocols , like http and Stomp and not care what the variable string bytes mean.

Soooo many software architects don't grok the magic of it.

You can define 8bit parser that check for "a:"(some string)\n

and work with a shit load of human languages.

The company I work for does not realise that most of f the 50 year old legacy C it has is fine with utf8 for all the arbitrary fixed length or \0 terminated strings it stores.

lolc2• 2 years ago

Just the most recent episode: A statistician is using PHP, on Windows, to analyze text for character frequency. He's rather confused by the UTF-16LE encoding and thinks the character "A" is numbered 4100 because that's what is shown in a hex-editor. I tried explaining about the little-endian part, and mb-string functions in PHP. And that PHP is not a good fit for his projects.

Then I realized that this is hilarious and I won't be able to kick him from his local minimum there. Everything he could learn about encodings would first complicate his work.

jheriko2• 2 years ago

no mention of the BOM... tragic.

You can't just assume UTF-8

Related Stories