Charset="WTF-8"

by edent • 2 years ago

282 452 2 years ago

jtvjan• 2 years ago

A coworker once implemented a name validation regex that would reject his own name. It still mystifies me how much convincing it took to get him to make it less strict.

croes• 2 years ago

Is name validation even possible?

armada651• 2 years ago

Yes, it is essential when you want to avoid doing business with customers who have invalid names.

majkinetor• 2 years ago

Sure it is. Context matters. For example, in clone wars.

poizan426• 2 years ago

Yes, it's easy

    bool ValidateName(string name) => true;

(With the caveat that a name might not be representable in Unicode, in which case I dunno. Use an image format?)

rsynnott2• 2 years ago

No, but it doesn’t stop people trying.

gmuslera2• 2 years ago

You may not want Bobby Tables in your system.

crazygringo7• 2 years ago

If you just use the {Alphabetic} Unicode character class (100K code points), together with a space, hyphen, and maybe comma, that might get you close. It includes diacritics.

I'm curious if anyone can think of any other non-alphabetic characters used in legal names around the world, in other scripts?

I wondered about numbers, but the most famous example of that has been overturned:

"Originally named X Æ A-12, the child (whom they call X) had to have his name officially changed to X Æ A-Xii in order to align with California laws regarding birth certificates."

(Of course I'm not saying you should do this. It is fun to wonder though.)

zarzavat• 2 years ago

Presumably there aren't any people with control characters in their name, for example.

nkrisc• 2 years ago

It is if you first provide a complete specification of a “name”. Then you can validate if a name is compliant with your specification.

ValentinA232• 2 years ago

Don't validate names, use transliteration to make them safe for postal services (or whatever). In SQL this is COLLATE, in the command line you can use uconv:

echo "'Lódź'" | uconv -f "UTF-8" -t "UTF-8" -x "Latin-ASCII"
'Lodz'

perching_aix3• 2 years ago

In certain cultures yes. Where I live, you can only select from a central, though frequently updated, list of names when naming your child. So theoretically only (given) names that are on that list can occur.

Family names are not part of this, but maybe that exists too elsewhere. I don't know how people whose name has been given to them before this list was established is handled however.

An alternative method, which is again culture dependent, is to use virtual governmental IDs for this purpose. Whether this is viable in practice I don't know, never implemented such a thing. But just on the surface, should be.

michaelt2• 2 years ago

There are of course some people who'll point you to a blog post saying no validation is possible.

However, for every 1 user you get whose full legal name is bob@example.com you'll get 100 users who put their e-mail into the name field by accident

And for every 1 user who wants to be called e.e. cummings you'll get 100 who just didn't reach for the shift key and who actually prefer E.E. Cummings. But you'll also get 100 McCarthys and O'Connors and al-Rahmans who don't need their "wrong" capitalisation "fixed" thank you very much.

Certainly, I think you can quite reasonably say a name should be comprised of between 2 and 75 characters, with no newlines, nulls, emojis, leading or trailing spaces, invalid unicode code points, or angle brackets.

barryrandall• 2 years ago

Anything is possible with enough qualifiers and caveats.

throw310822• 2 years ago

I know multiple developers who would just say "well it's their fault, they have to change name then".

MrJohz3• 2 years ago

I worked with an office of Germans who insisted that ASCII was sufficient. The German language uses letters that cannot be represented in ASCII.

In fairness, they mostly wanted stuff to be in English, and when necessary, to transliterate German characters into their English counterparts (in German there is a standardised way of doing this), so I can understand why they didn't see it was necessary. I just never understood why I, as the non-German, was forever the one trying to convince them that Germans would probably prefer to use their software in German...

guappa2• 2 years ago

Our own software that we sell was crashing if you had a locale set in anything else than american english.

The coworker who made that happen said I'm a weirdo for setting my machine in my own language. According to him I should have set it to english.

This of course happened in a non english speaking country.

poizan426• 2 years ago

I have an 'æ' in my middle name (formally secondary first name because history reasons). Usually I just don't use it, but it's always funny when a payment form instructs me to write my full name exactly as written on my credit card, and then goes on to tell me my name is invalid.

pzduniak3• 2 years ago

I live in Łódź.

Love receiving packages addressed to ??d? :)

troymc• 2 years ago

I wonder how many of those packages end up in Vada, Italy. Or Cody, Wyoming. Or Buda, Texas...

jowea• 2 years ago

And the packages get there? Don't you put "Łódź (Lodz)" in the city field? Or the postal code takes care of the issue?

Symbiote• 2 years ago

It goes both ways.

I received a large delivery from Poland recently, and it was addressed to "K?benhavn", with similar mistakes in the street name.

epcoa3• 2 years ago

As you may be aware, the name field for credit card transactions is rarely verified (perhaps limited to North America, not sure).

Often I’ll create a virtual credit card number and use a fake name, and virtually never have had a transaction declined. Even if they are more aggressively asking for a street address, giving just the house number often works. This isn’t a deep cover but gives a little bit of a anonymity for marketing.

seba_dos12• 2 years ago

It's for when things go wrong. Same as with wire transfers. Nobody checks it unless there's a dispute.

mkotowski2• 2 years ago

Still much better when it fails at the first step. I once got myself in a bit of a struggle with Windows 10 by using "ł" as part of Windows username. Amusingly/irritatingly large number of applications, even some of Microsoft's own ones, could not cope with that.

darkhorn• 2 years ago

For a similar reason many Java applications do not work in Turkish Windowses. The Turkish İi Iı problem.

ahazred8ta3• 2 years ago

The government of Ireland has many IT systems that cannot handle áccénted letters. #headdesk

arp242• 2 years ago

I worked for an Irish company that didn't support ' in names. Did get fixed eventually, but sigh...

lxgr• 2 years ago

Did you actually get banks to print that on your credit card?

I’m impressed, most I know struggle with any kind of non-[A-Z]!

poizan426• 2 years ago

In Denmark? I don't think they legally are allowed not to print my legal name.

Would be weird if our banks couldn't handle our own alphabet.

Muromec29• 2 years ago

"Write your name the way it's spelled in your government issued id" is my favorite. I have three ids issued by two governments and no two match letter by letter.

chrismorgan3• 2 years ago

My wife had two given names and no surname. (In fact, before eighth class, she only had one given name.) Lacking a surname is very common in some parts of India. Also some parts of India put surname first, and some last, and the specific meaning and customs vary quite a bit too. Indian passports actually separate given names and family names entirely (meaning you can’t reconstruct the name as it would customarily be written). Her passport has the family name line blank. Indigo writes: “Name should be as per government ID”, and has “First And Middle Name” and “Last Name” fields. Both required, of course. I discovered that if you put “-” in the Last Name field, the booking process falls over several steps later in a “something went wrong, try again later” way; only by inspecting an API response in the dev tools did I determine it was objecting to having “-” in the name. Ugh. Well, I have a traditional western First Middle Last name, and from putting it in things, sometimes it gets written First Middle Last and sometimes Last First Middle, and I’ve received some communications addressed to First, some to Last, and some to Middle (never had that happen before!). It’s a disaster.

Plenty of government things have been digitalised in recent years too, and split name fields tend to have been coded to make both mandatory. It’s… disappointing, given the radical diversity of name construction across India.

card_zero• 2 years ago

Pfft, "Dein Name ist ungültig" (your name is invalid). Let's get straight to the point, it's the user's fault for having a bad name, user needs to fix this.

cabirum• 2 years ago

How do I allow "stępień" while detecting Zalgo-isms?

egypturnash• 2 years ago

Zalgo is largely the result of abusing combining modifiers. Declare that any string with more than n combining modifiers in a row is invalid.

n=1 is probably a reasonable falsehood to believe about names until someone points out that language X regularly has multiple combining modifiers in a row, at which point you can bump up N to somewhere around the maximum number of combining modifiers language X is likely to have, add a special case to say "this is probably language X so we don't look for Zalgos", or just give up and put some Zalgo in your test corpus, start looking for places where it breaks things, and fix whatever breaks in a way that isn't funny.

zvr• 2 years ago

I can point out that Greek needs n=2: for accent and breathing.

ahazred8ta3• 2 years ago

N=2 is common in Việt Nam. (vowel sound + tonal pitch)

zootboy• 2 years ago

For the unaware (including myself): https://en.wikipedia.org/wiki/Zalgo_text

If you really think you need to programmatically detect and reject these (I'm dubious), there is probably a reasonable limit on the number of diacritics per character.

https://stackoverflow.com/a/11983435

KPGv22• 2 years ago

I could answer your question better if I knew why you need to detect Zalgo-isms.

account4232• 2 years ago

Because they are an attack vector. They can be used to hide important information as they overflow bounds (can be solved with clipping but then you need to do that everywhere it matters) and have the ability to slow text renderers to a crawl.

seba_dos12• 2 years ago

There's nothing special about "Stępień", it has no combining characters, just the usual diacritics that have their own codepoints in Basic Multilingual Plane (U+0119 and U+0144). I bet there are some names out there that would make it harder, but this isn't one.

cryptonector4• 2 years ago

If you decompose then it uses combining codepoints. Still nothing special.

dpassens• 2 years ago

Why do you need to detect Zalgo-isms and why is it so important that you want to force people to misspell their names?

tobyhinloopen• 2 years ago

We have a whitelist of allowed characters, which is a pretty big list.

I think we based it on Lodash’ deburr source code. If deburr’s output is a-z and some common symbols, it passes (and we store the original value)

https://www.geeksforgeeks.org/lodash-_-deburr-method/

Diggsey• 2 years ago

I thought this was https://simonsapin.github.io/wtf-8/

webstrand• 2 years ago

Yeah, this is just issues caused by ascii

RadiozRadioz• 2 years ago

I've got a good feel now for which forms will accept my name and which won't, though mostly I default to an ASCII version for safety. Similarly, I've found a way to mangle my address to fit a US house/state/city/zip format.

I don't feel unwelcome, I emphathize with the developers. I'd certainly hate to figure out address entry for all countries. At least the US format is consistent across websites and I can have a high degree of confidence that it'll work in the software, and my local postal service know what to do because they see it all the time.

Arch485• 2 years ago

You can grab JSON data of all ISO recognized countries and their address formats on GitHub (apologies, I forget the repo name. IIRC there is more than one).

I don't know if it's 100% accurate, but it's not very hard to implement it as part of an address entry form. I think the main issue is that most developers don't know it exists,

saurik• 2 years ago

At the end of the day, a postal address is printed to an envelope or package as a single block of text and then read back and parsed somehow by the people delivering the package (usually by a machine most of the way, but even these days more by humans as the package gets closer to the destination). This means that, in a very real sense, the "correct" way to enter an address is into a single giant multi-line text box with the implication that the user must provide whatever is required to be printed onto the mailing label such that a carrier will successfully be able to find them.

Really, then, the reasons why we bother trying to break out an address into multiple parts is not directly caused by the need for an address at all: it is because we 1) might not trust the user to provide for us everything required to make the address valid (assuming the country or even state, or giving us only a street address with no city or postal code... both mistakes that are likely extremely common without a multi-field form), or 2) need to know some subset of the address ourselves and do not trust ourselves to parse back the fuzzy address the same way as the postal service might, either for taxes or to help establish shipping rates.

FWIW, I'd venture to say that #2 is sufficiently common -- as if you need a street address for shipping you are going to need to be careful about sales taxes and VAT, increasingly often even if you aren't located in the state or even country to which the shipment will be made -- that it almost becomes nonsensical to support accepting an address for a location where you aren't already sure of the format convention ahead of time (as that just leads you to only later realizing you failed to collect a tax, will be charged a fortune to ship there, or even that it simply isn't possible to deliver anything to that country)... and like, if you don't intend to ship anything, you actually do not need the full address anyway (credit cards, as an obvious example, don't need or use the full address).

KPGv22• 2 years ago

It seems ridiculous to apply form validation to a name, given the complexity of charsets involved. I don't even validate email addresses. I remember [this](https://www.netmeister.org/blog/email.html) wonderful explainer of why your email validation regex is wrong.

nedt• 2 years ago

In HTML if you use <input type="email"> you basically get a regex validation. While it not fully follows the RFC it's a good middle ground. And it gives you an email that you can use on the internet (RFC obviously has some cases that are outside of this scope). That's why I tend to prefer what's defined in the standard: https://html.spec.whatwg.org/multipage/input.html#email-stat...

imrejonk2• 2 years ago

A system not supporting non-latin characters in personal names is pitiful, but a system telling the user that they have an invalid name is outright insulting.

notanote3• 2 years ago

That’s the best one of the lot. "Dein Name ist ungültig", "Your name is invalid", written with the informal word for "your".

rossdavidh• 2 years ago

They're trying to say that you and the server are very close friends, you see? No, no, I get this is not correct, just a joke...

account4232• 2 years ago

It's only insulting if you want to be insulted. You'd do better learning to understand that its a computer system and not a personal attack.

imrejonk2• 2 years ago

While I agree with you that an error message like this should not be taken as a personal attack, it still causes a horrible user experience. It’s not like the developers of these systems had no choice in the wording of these errors: they picked the insensitive computer-says-no option, while they could have went with an apologetic “Sorry, our system does not support special characters” instead.

ginko• 2 years ago

Under GDPR you have the legal right for your name to be stored and processed with the correct spelling in the EU.

https://gdprhub.eu/index.php?title=Court_of_Appeal_of_Brusse...

xigoi• 2 years ago

This seems to only apply to banks.

pornel• 2 years ago

I wouldn't be surprised if that created kafkaesque problems with other institutions that require name to match the bank account exactly, and break/reject non-ASCII at the same time.

postepowanieadm2• 2 years ago

No, anywhere where your name is used.

robin_reala3• 2 years ago

It’s a general right to have incorrect personal data relating to you rectified by the data processor.

Etheryte2• 2 years ago

This does not only apply to banks. The specific court case was brought against a bank, but the law as is applies to any and everyone who processes your personal data.

voidUpdate• 2 years ago

Apparently it didn't apply to my bank when I tried to change my name with them, according to them anyone could have made my change of name form, and having ID and correctly putting in the PIN for my card wasn't enough

stop_nazi• 2 years ago

grzegorz brzęczyszczykiewicz

dvh2• 2 years ago

Looks ok in my language: Gřegoř Bženčiščikievič

postepowanieadm2• 2 years ago

You miss "ę"!

nicbou2• 2 years ago

https://www.youtube.com/watch?v=AfKZclMWS1U For the curious

Hackbraten• 2 years ago

Situations like these regularly make me feel ashamed about being a software developer.

jccalhoun• 2 years ago

My first name is hyphenated. I still find forms that reject it. My favorite was one that say "invalid first name."

Pesthuf• 2 years ago

I totally get that companies are probably more successful using simple validation rules, that work for the vast majority of names rather than just accepting everything just so that some person with no name or someone whose name cannot possibly be expressed or at least transliterated to Unicode can use their services.

But that person's name has no business failing validation. They fucked up.

surfingdino• 2 years ago

I lost count of the projects where this was an issue. US and Western European-born devs are oblivious to this problem and it ends up catching them over and over again.

ACS_Solver3• 2 years ago

Yeah, it's amazing. My language has a Latin-based alphabet but can't be represented with ISO 8859-1 (aka the Latin-1 charset) so I used to take it for granted that most software will not support inputs in the language... 25 years ago. But Windows XP shipped with a good selection of input methods and used UTF-16, dramatically improving things, so it's amazing to still see new software created where this is somehow a problem.

Except that now there's no good excuse. Things like the name in the linked article would just work out of the box if it weren't for developers actually taking the time to break them by implementing unnecessary and incorrect validation.

I can think of very few situations, where validation of names is actually warranted. One that comes to mind is when you need people's ICAO 9303 compliant names, such as on passports or airline systems. If you need to make sure you're getting the name the person has in their passport's MRZ, then yes, rejecting non-ASCII characters is correct, but most systems don't need to do that.

xyst• 2 years ago

Software has been gaslighting generations of people around the world.

Side note: not a bad way to skirt surveillance though.

A name like “stępień” will without a doubt have many ambiguous spellings across different intelligence gathering systems (RUMINT, OSINT, …). Americans will probably spell it as “Stefen” or “Steven” or “Stephen”, especially once communicated over phone.

philipwhiuk2• 2 years ago

Turns out just foreign document is enough for that sometimes ;) http://news.bbc.co.uk/1/hi/northern_ireland/7899171.stm

ljouhet• 2 years ago

Yes, all these forms should handle existing names...

Etheryte2• 2 years ago

I think the bottom of the page is you missing the joke. It's showing only the name letters that get rejected everywhere else. Similarly for the URL, the URL renders his name correctly when you browse to it in a modern browser. What you've copied is the canonical fallback for unicode.

account4232• 2 years ago

It's not a fallback, it's the actual domain name.

powersnail• 2 years ago

As someone who really think name field should just be one field with any printable unicode characters, I do wonder what the hell would I need to do if I take customer names in this form, and then my system has to interact with some other service that requires first/last name split, and/or [a-zA-Z] validation, like a bank or postal service.

Automatic transliteration seems to be very dangerous (wrong name on bank accounts, for instance), and not always feasible (some unicode characters have more than one way of being transliterated).

Should we apologize to the user, and just ask the user twice, once correctly, and once for the bad computer systems? This seems to be the only approach that both respects their spelling, and at the same time not creating potential conflict with other systems.

Muromec29• 2 years ago

Okay, I have a non-ASCII (non Latin even) name, so I can tell. You just ask explicitly how my name is spelled in a bank system or my government id. Please don't try transliteration, unless you know exact rules the other system suggests to transliterate my name from the one cultural context into another and then still make it a suggestion and make it clear for which purpose it will be used (and then only use it for that purpose).

And please please please, don't try to be smart and detect the cultural context from the character set before automatically translating it to another character set. It will go wrong and you will not notice for a long time, but people will make mean passive aggressive screenshots of your product too.

My bank for example knows my legal name in Cyrillic, but will not print it on a card, so they make best-effort attempt to transliterate it to ASCII, but make it editable field and will ask me to confirm this is how I want it to be on a card.

matthewbauer• 2 years ago

You can just show the user the transliteration & have them confirm it makes sense. Always store the original version since you can't reverse the process. But you can compare the transliterated version to make sure it matches.

Debit cards a pretty common example of this. I believe you can only have ASCII in the cardholder name field.

Muromec29• 2 years ago

But you can compare the transliterated version to make sure it matches

No you can't.

Add: Okay, you need to know why. I'm right here a living breathing person with a government id that has the same name scribed in two scripts side by side.

There is an algorithm (blessed by the same government that issued said it) which defines how to transliterate names from one to another, published on the parliament web site and implement in all the places that are involved in the id issuing business.

The algorithm will however not produce the outcome you will see on my id, because me, living breathing person who has a name asked nicely to spell it the way I like. The next time I visit the id issuing place, I could forget to ask nicely and then I will have two valid ids (no, the old one will not be marked as void!) with three names that don't exactly match. It's all perfectly fine, because name as a legal concept is defined in the character set you probably can't read anyway.

Please, don't try be smart with names.

8organicbits• 2 years ago

I have an ID that transliterated my name, and included the original, but the original contained an obvious typo. I immediately notified the government official, but they refused to fix it. They assured me that only the transliterated name would be used.

Human systems aren't always interested in avoiding or fixing defects.

junek• 2 years ago

The fundamental mistake is in trying to take input for one purpose and transform it for another purpose. Just have the user fill in an additional field for their name as it appears on bank statements, or whatever the second purpose is. Trying to be clever about this stuff never works out.

layer82• 2 years ago

What you call the second purpose is often the only purpose. Or you have to talk to half a dozen other systems each of which have different constraints. You wouldn’t want to present the user half a dozen fields just so that they can choose the nicest representation of their name for each system.

That being said, in Japan it’s actually common to have two fields, one for the name in kanji (the “nice” name), and one in katakana (restricted, 8-bit phonetic alphabet, which earlier/older computer systems used and some probably still use).

teaearlgraycold• 2 years ago

Legal name vs. display name

deathanatos• 2 years ago

… "legal name" is "things programmer's believe about names" grade. Maybe (name, jurisdiction), but I've seen exceptions to that, too.

Where I live, no less than 3 jurisdictions have a say about my "legal" name, and their laws do not require them to match. At one point, one jurisdiction had two different "legal" names for me, one a typo by my standards, but AFAICT, both equally valid.

There's no solution here, AFIACT, it's just evidence towards why computers cannot be accountability sinks.

nicbou2• 2 years ago

We had problems with a Ukrainian refugee we helped because certified translations of her documents did not match. Her name was transliterated the German way in one place and the English way in another.

Those are translations coming from professionals who swore an oath. Don’t try to do it with code.

hyeonwho42• 2 years ago

In the US, you can generally specify to your certified translators how you want proper names and place names written. I would suggest you or your friend talk to the translators again so that everything matches. It will also minimize future pains.

Also, USCIS usually has an "aliases" field on their forms, which would be a good place to put German government misspellings.

int_19h13• 2 years ago

This actually sounds like the translator didn't do their job properly. Most countries have detailed official guidelines on how to transliterate names into Latin script, because they need that to issue internationally recognized passports. For Ukrainian, the relevant standard is https://zakon.rada.gov.ua/laws/show/55-2010-%D0%BF

The real problem here is that transliteration rules depend on the source language, and you often don't have the context to distinguish that (e.g. the name can be spelled identically in Russian, Bulgarian, and Ukrainian, yet transliterations will be different). And even where you do have sufficient context, this still requires the system to be able to handle any source language that can be potentially thrown at it, which is just about anything. So it never gets implemented properly in practice.

koito17• 2 years ago

Ask for inflections separately.

For instance, in many Japanese forms, there are dedicated fields for the name and the pronunciation of the name. There are possibly multiple ways to read a name (e.g. 山崎 is either やまざき or やまさき). It is better to ask the person "how to read your name?" rather than execute code to guess the reading.

As for transliteration, it's best to avoid if possible. If not possible, then rely on international standards (e.g. Japanese has ISO 3602 and Arabic has ISO 233-2). When international standards don't exist, then fall back to "context-dependent" standards (e.g. in Taiwan, there are several variants of Pinyin. Allow the user to choose the romanization that matches their existing documentation).

wruza6• 2 years ago

I'll say it again: this is the consequence of Unicode trying to be a mix of html and docx, instead of a charset. It went too far for an average Joe DevGuy to understand how to deal with it, so he just selects a subset he can handle and bans everything else. HN does that too - special symbols simply get removed.

Unicode screwed itself up completely. We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while. Shortly after it focused on becoming a complex file format with colorful icons and invisible symbols, which is not manageable without cutting out all that bs by force.

Muromec29• 2 years ago

so he just selects a subset he can handle and bans everything else.

Yes? And the problem is?

throwaway2904• 2 years ago

The next guy with a different subset? :)

wruza6• 2 years ago

The problem is the scale at which it happens and lack of methods-to-go in most runtimes/libs. No one and nothing is ready for unicode complexity out of box, and there's little interest in unscrewing it by oneself, cause it looks like an absurd minefield and likely is one, from the persepective of an average developer. So they get defensive by default, which results in $subj.

n2d44• 2 years ago

and invisible symbols

Invisible symbols were in Unicode before Unicode was even a thing (ASCII already has a few). I also don't think emojis are the reason why devs add checks like in the OP, it's much more likely that they just don't want to deal with character encoding hell.

As much as devs like to hate on emojis, they're widely adopted in the real world. Emojis are the closest thing we have to a universal language. Having them in the character encoding standard ensures that they are really universal, and supported by every platform; a loss for everyone who's trying to count the number of glyphs in a string, but a win for everyone else.

account4232• 2 years ago

Emojis are the closest thing we have to a universal language.

What meaning does U+1F52B have? What about U+1F346?

A set of glyphs does not make a language.

meew0• 2 years ago

The “invisible symbols” are necessary to correctly represent human language. For instance, one of the most infamous Unicode control characters — the right-to-left override — is required to correctly encode mixed Latin and Hebrew text [1], which are both scripts that you mentioned. Besides, ASCII has control characters as well.

The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own.

If you say that Unicode is too expansive, you would have to make a decision to exclude certain types of human communication from being encodable. In my opinion, including everything without discrimination is much preferable here.

[1] https://en.wikipedia.org/wiki/Right-to-left_mark#Example_of_...

n2d44• 2 years ago

Granted, technically speaking emojis are not part of the "Unicode Standard", but they are standardized by the Unicode Consortium and constitute "Unicode Technical Standard #51": https://www.unicode.org/reports/tr51/

Y_Y6• 2 years ago

I'm happy to discriminate against those damn ancient Sumerians and anyone still using goddamn Linear B.

bawolff3• 2 years ago

one of the most infamous Unicode control characters — the right-to-left override

You are linking to an RLM not an RLO. Those are different characters. RLO is generally not needed and more special purpose. RLM causes much less problems than RLO.

Really though, i feel like the newer "first strong isolate" character is much better designed and easier to understand then most of the other rtl characters.

wruza6• 2 years ago

Copy this󠀠󠀼󠀼󠀼󠀠󠁉󠁳󠀠󠁴󠁨󠁩󠁳󠀠󠁮󠁥󠁣󠁥󠁳󠁳󠁡󠁲󠁹󠀠󠁴󠁯󠀠󠁣󠁯󠁲󠁲󠁥󠁣󠁴󠁬󠁹󠀠󠁲󠁥󠁰󠁲󠁥󠁳󠁥󠁮󠁴󠀠󠁨󠁵󠁭󠁡󠁮󠀠󠁬󠁡󠁮󠁧󠁵󠁡󠁧󠁥󠀿󠀠󠀾󠀾󠀾 sentence into this site and click Decode. (YMMW)

https://embracethered.com/blog/ascii-smuggler.html

account4232• 2 years ago

The “colorful icons” are not part of Unicode. Emoji are just characters like any other. There is a convention that applications should display them as little coloured images, but this convention has evolved on its own.

Ok now you're just full of shit and you know it.

virexene• 2 years ago

in what way is unicode similar to html, docx, or a file format? the only features I can think of that are even remotely similar to what you're describing are emoji modifiers.

and no, this webpage is not result of "carefully cutting out the complicated stuff from Unicode". i'm pretty sure it's just the result of not supporting Unicode in any meaningful way.

asddubs3• 2 years ago

We wanted a common charset for things like latin, extlatin, cjk, cyrillic, hebrew, etc. And we got it, for a while.

we didn't even get that because slightly different looking characters from japanese and chinese (and other languages) got merged to be the same character in unicode due to having the same origin, meaning you have to use a font based on the language context for it to display correctly.

tadfisher4• 2 years ago

They are the same character, though. They do not use the same glyph in different language contexts, but Unicode is a character encoding, not a font standard.

throwaway2904• 2 years ago

I bet the complex file format thing probably started at CJK. They wanted to compose Hangul and later someone had a bright idea to do the same to change the look of emojis.

Don't worry, AI is the new hotness. All they need is unpack prompts into arbitrary images and finally unicode is truly unicode, all our problems will be solved forever

kristopolous5• 2 years ago

There's no argument here.

We could say it's only for script and alphabets, ok. It includes many undeciphered writing systems from antiquity with only a small handful of extent samples.

Should we keep that, very likely to never be used character set, but exclude the extremely popular emojis?

Exclude both? Why? Aren't computers capable enough?

I used to be on the anti emoji bandwagon but really, it's all indefensible. Unicode is characters of communication at an extremely inclusive level.

I'm sure some day it will also have primitive shapes and you can construct your own alphabet using them + directional modifiers akin to a generalizable Hangul in effect becoming some kind of wacky version of svg that people will abuse it in an ASCII art renaissance.

So be it. Sounds great.

riwsky• 2 years ago

Like how phonetic alphabets save space compared to ideograms by just “write the word how it sounds”, the little SVG-icode would just “write the letter how it’s drawn”

simonh4• 2 years ago

No, no, no, no, no… So then we’d get ‘the same’ character with potentially infinite different encodings. Lovely.

Unicode is a coding system, not a glyph system or font.

mason_mpls• 2 years ago

This frustration seems unnecessary, unicode isnt more complicated than time and we have far more than enough processing power to handle its most absurd manifestations.

We just need good libraries, which is a lot less work than inventing yet another system.

arka2147483647• 2 years ago

The limiting factor is not compute power, but the time and understanding of a random dev somewhere.

Time also is not well understood by most programmers. Most just seem to convert it to epoch and pretend that it is continuous.

bawolff3• 2 years ago

There are no emoiji in this guy's name.

Unicode has made some mistakes, but having all the symbols necessary for this guy's name is not one of them.

jrochkind13• 2 years ago

Unicode has metadata on each character that would allow software to easily strip out or normalize emoji's and "decorative" characters.

It might have edge case problems -- but the charcters in the OP's name would not be included.

Also, stripping out emoji's may not actually be required or the right solution. If security is the concern, Unicode also has recommended processes and algorithms for dealing with that.

https://www.unicode.org/reports/tr39/

We need better support for the functions developers actually need on unicode in more platforms and languages.

Global human language is complicated as a domain. Legacy issues in actually existing data adds to the complexity. Unicode does a pretty good job at it. It's actually pretty amazing how well it does. Including a lot more than just the character set, and encoding, but algorithms for various kinds of normalizing, sorting, indexing, under various localizations, etc.

It needs better support in the environments more developers are working in, with raised-to-the-top standard solutions for identified common use cases and problems, that can be implemented simply by calling a performance-optimized library function.

(And, if we really want to argue about emoji's, they seem to be extremely popular, and literally have effected global culture, because people want to use them? Blaming emoji's seems like blaming the user! Unicode's support for them actually supports interoperability and vendor-neutral standards for a thing that is wildly popular? but I actually don't think any of the problems or complexity we are talking about, including the OP's complaint, can or should be laid at the feet of emojis)

numpad07• 2 years ago

IMO, the sin of Unicode is that they didn't just pick local language authorities and gave them standardized concepts like lines and characters, and start-of-language and end-of-language markers.

Lots of Unicode issues come from handling languages that the code is not expecting, and codes currently has no means to select or report quirk supports.

I suppose they didn't like getting national borders involved in technical standardization bit that's just unavoidable. It is getting involved anyway, and these problems are popping up anyway.

kmeisthax2• 2 years ago

This doesn't self-synchronize. Removing an arbitrary byte from the text stream (e.g. SOL / EOL) will change the meaning of codepoints far away from the site of the corruption.

What it sounds like you want is an easy way for English-language programmers to skip or strip non-ASCII text without having to reference any actual Unicode documentation. Which is a Unicode non-goal, obviously. And also very bad software engineering practice.

I'm also not sure what you're getting at with national borders and language authorities, but both of those were absolutely involved with Unicode and still are.

kevin_thibedeau• 2 years ago

start-of-language and end-of-language markers

Unicode used to have language tagging but they've been (mostly) deprecated:

https://en.wikipedia.org/wiki/Tags_(Unicode_block)

https://www.unicode.org/reports/tr7/tr7-1.html

layer82• 2 years ago

I’m working with Word documents in different languages, and few people take the care to properly tag every piece of text with the correct language. What you’re proposing wouldn’t work very well in practice.

The other historical background is that when Unicode was designed, many national character sets and encodings existed, and Unicode’s purpose was to serve as a common superset of those, as otherwise you’d need markers when switching between encodings. So the existing encodings needed to be easily convertible to Unicode (and back), without markers, for Unicode to have any chance of being adopted. This was the value proposition of Unicode, to get rid of the case distinctions between national character sets as much as possible. As a sibling comment notes, originally there were also optional language markers, which however nobody used.

jojobas• 2 years ago

But hey, multiocular o!

https://en.wikipedia.org/wiki/Cyrillic_O_variants#Multiocula...

(TL;DR a bored scribe's doodle has a code point)

account4232• 2 years ago

Huh, wasn't aware of this update.

The character was proposed for inclusion into Unicode in 2007 and incorporated as character U+A66E in Unicode version 5.1 (2008). The representative glyph had seven eyes and sat on the baseline. However, in 2021, following a tweet highlighting the character, it came to linguist Michael Everson's attention that the character in the 1429 manuscript was actually made up of ten eyes. After a 2022 proposal to change the character to reflect this, it was updated later that year for Unicode 15.0 to have ten eyes and to extend below the baseline. However, not all fonts support the ten-eyed variant as of November 2024.

So not only did they arbitrarily add a non-character (while arbitrarily combinding other real characters) but they didn't even get the glyph right and then changed it to actually match the arbitrary doodle.

josephcsible4• 2 years ago

What would be wrong with "enter your name as it appears in the machine-readable zone of your passport" (or "would appear" for people who have never gotten one)? Isn't that the one standard format for names that actually is universal?

ahoka• 2 years ago

I would like to use my name as my parents gave it to me, thanks. Is that too much to ask for?

richardwhiuk2• 2 years ago

How much flexibility are we giving parents in what they name children?

If a parent invented a totally new glyph, would supporting that be a requirement?

account4232• 2 years ago

If your parents were smartasses, then yes.

ks20482• 2 years ago

There's the problem that "appears" is a visible phenomenon and unicode strings can contain non-visible characters and multiple ways to represent the same visible information. Normalization is supposed to help here, but some sites may fail to do this or do incorrectly, etc.

josephcsible4• 2 years ago

But the MRZ of a passport doesn't contain any of those problem characters.

crooked-v2• 2 years ago

From experience, it's not actually universal. Visa applications will often ask for name information not encoded in the MRZ.

Muromec29• 2 years ago

The problem is, people exists that have É in their name and will go to court when you spell it as E, the court will also say that 1 ) you have the technical ability to write it as É and 2) they have a right to have their name spelled correctly. Also it's not nice and bad for business to be like this.

josephcsible4• 2 years ago

they have a right to have their name spelled correctly

IMO, having the law consider this as an unconditional right is the root of the problem. What happens when people start making up their own characters that aren't in Unicode to put in their names?

Also it's not nice and bad for business to be like this.

What about having a validated "legal name" for everything official and an unvalidated "display name" that's only ever parroted back to the person who entered it?

gavinsyancey• 2 years ago

WTF-8 is actually a real encoding, used for encoding invalid UTF-16 unpaired surrogates for UTF-8 systems: https://simonsapin.github.io/wtf-8/

bjackman2• 2 years ago

I believe this is what Rust OsStrings are under the hood on Windows.

extraduder_ire• 2 years ago

Which I assume stands for "Windows-Transformation-Format-8(bits)".

ptx• 2 years ago

Yeah, that had me confused for a bit. And you would never use "charset=wtf-8" (as in the title for this page) because the spec says:

"Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet."

account4232• 2 years ago

Some specs like to claim things that are out of their jurisdiction. A format spec has no say on where that format is used. It's best to ignore such hubris.

And in this particular case it doesn't even make sense because the entire point is to round trip WTF-16. If that requires one "system" to communicate WTF-8 with another "system" (which is really an arbitrary boundary) then so be it. And anything that expects UTF-8 will need to deal with "invalid" data in a usecase-dependent way anyway.

rurban2• 2 years ago

Just use the unicode identifier rules, my libu8ident. https://github.com/rurban/libu8ident

Windows folks need to convert to UTF—8 first

philipwhiuk2• 2 years ago

That's nice. Which restriction level handles all names?

rurban2• 2 years ago

Every. Identifiers are identifiable names. Most languages with unicode name support only support symbols, ie not identifiable names (binary trash).

bawolff3• 2 years ago

Its really not that hard though. PCRE regex support unicode letter classes. There is really no excuse for this type of issue.

cryptonector4• 2 years ago

`..*` should be a good validation regexp. At that point you might as well check only that length is non-zero and the the name is valid Unicode and nothing more. Well, ok, maybe look at UTR-#36 and maybe disallow / filter out non-printing characters, PUCs, and what not.

kgeist2• 2 years ago

My rule of thumb is to treat strings as opaque blobs most of the time. The only validation I'd always enforce is some sane length limit, to prevent users from shoving entire novels inside. If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away. Imho often times, input validation is an attempt to solve a problem from the wrong side. Say, when XSS or SQL injections are found on a site, I've seen people's first reaction to be validation of user input by looking for "special symbols", or add a whitelist of allowed characters, instead of simply escaping strings right before rendering HTML (and modern frameworks do it automatically), or using parameterized queries if it's SQL. If a user wants to call themselves "alert('hello')", why not? Why the arbitrary limits? I think there're very few exceptions to this, probably something law-related, or if you have to interact with some legacy service.

ddulaney• 2 years ago

There's at least one major exception to this: Unicode normalization.

It's possible for the same logical character to have two different sets of code points (for example, a-with-umlaut as a single character, vs a followed by umlaut combining diacritic). Related, distinguishing between the "a" character in Latin, Greek, Cyrillic, and the handful of other times it shows up throughout Unicode.

This comes up in at least 3 ways:

1. A usability issue. It's not always easy to predict which identical-looking variant is produced by different input methods, so users enter the identical-looking characters on different devices but get an account not found error.

2. A security issue. If some of your backend systems handle these kinds of characters differently, that can cause all kinds of weird bugs, some of which can be exploited.

3. An abuse issue. If it's possible to create accounts with the same-looking name as others that aren't the same account, there can be vectors for impersonation, harassment, and other issues.

So you have to make a policy choice about how to handle this problem. The only things that I've seen work are either restricting the allowed characters (often just to printable ASCII) or being very clear and strict about always performing one of the standard Unicode transformations. But doing that transformation consistently across a big codebase has some real challenges: in particular, it can change based on Unicode version, and guaranteeing that all potential services use the same Unicode version is really non-trivial. So lots of people make the (sensible) choice not to deal with it.

But yeah, agreed that parenthesis should be OK.

tugu772• 2 years ago

Type system ftw? As long as it's a blob (unnormalized), it should have a blob type which can do very little besides storing and retrieving, perhaps printing. Only the normalized version should be even comparable.

speleding• 2 years ago

Something we just ran in to: There are two UTF-8 codepoints for the @ character, the normal one and "Full width At Sign U+FF20". It took a lot of head scratching to understand why several Japanese users could not be found with their email address when I was seeing their email right there in the database.

77pt773• 2 years ago

If you treat your strings as opaque blobs, and use UTF8, most of internationalization problems go away

This is laughably naive.

So many things can go wrong.

Strings are not arrays of bytes.

There is a price to pay if someone doesn't understand that or chooses to ignore it.

hughesjj• 2 years ago

RTL go brrr

lelandbatey• 2 years ago

And yet when stored on any computer system, that string will be encoded using some number of bytes. Which you can set a limit on even though you cannot cut, delimit, or make any other inference about that string from the bytes without doing some kind of interpretation. But the bytes limit is enough for the situation the OP is talking about.

shakna2• 2 years ago

Strings are not arrays of bytes.

That very much depends on the language that you are using. In some, they are.

Muromec29• 2 years ago

You can treat names as byte blobs for as long as you don't use them for their purpose -- naming people.

Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly?

I think there're very few exceptions to this, probably something law-related, or if you have to interact with some legacy service.

Few exceptions for you is entirety of the service for others. At the very least you interact with legacy software of payment systems which have some ideas about what names should be.

kmoser2• 2 years ago

Would your customer representative be able to pronounce my name somewhat correctly?

Are you implying the CSR's lack of familiarity with the pronunciation of your name means your name should be stored/rendered incorrectly?

hobs5• 2 years ago

Absolutely not - do not build anything based on "would your CSR be able to pronounce" something - that's an awful bar - most CSRs cant pronounce my name - would I be excluded from your database?

Seriously, what are you going for here?

benatkin• 2 years ago

Would your customer representative be able to pronounce my name somewhat correctly?

Worse case, just drop to hexadecimal.

kgeist2• 2 years ago

Would your customer representative be able to pronounce my name somewhat correctly?

Typical input validation doesn't really solve the problem. For instance, I could enter my name as 'Vrdtpsk,' which is a perfectly valid ASCII string that passes all validation rules, but no one would be able to pronounce it correctly. I believe the representative (if on a call) should simply ask the customer how they would like to be addressed. Unless we want to implement a whitelist of allowed names for customers to choose from...

Intermernet• 2 years ago

Many Japanese companies require an alternative name entered in half width kana to alleviate this exact problem. Unfortunately, most Japanese websites have a million other UX problems that overshadow this clever solution to the problem.

arghwhat3• 2 years ago

Suppose you have a unicode blob of my name in your database and there is a problem and you need to call me and say hi. Would your customer representative be able to pronounce my name somewhat correctly?

You cannot pronounce the name regardless of whether it is written in ASCII. Pronouncing a name requires at the very least knowledge of the language it originated in, and attempts at reading it with an English pronunciation can range from incomprehensible to outright offensive.

The only way to correctly deal with a name that you are unfamiliar with the pronunciation of is to ask how it is pronounced.

You must store and operate on the person's name as is. Requiring a name modified, or modifying it automatically, is unacceptable - in many cases legal names must be represented accurately as your records might be used for e.g. tax or legal reasons later.

red_admiral• 2 years ago

Would your customer representative be able to pronounce my name somewhat correctly?

If the user is Chinese and the CSR is not - probably no, and that's not a Unicode issue.

rafram• 2 years ago

Sanitizing your strings immediately before display is all well and good until you need to pass them to some piece of third-party software that is very dumb and doesn’t sanitize them. You’ll argue that it’s the vendor’s fault, but the vendor will argue that nobody else allows characters like that in their name inputs!

See the Companies House XSS injection situation, where their rationale for forcing a business to change its name was that others using their database could be vulnerable: https://www.theregister.com/2020/10/30/companies_house_xss_s...

rob74• 2 years ago

but the vendor will argue that nobody else allows characters like that in their name inputs

...and maybe they will even link to this page to support that statement! But, seeing that most of the pages are German, I bet they do accept the usual German "special" letters (ÄÖÜß) in names?

arkh2• 2 years ago

You sanitize at the frontier of what your code controls.

Sending data to a database: parametrized queries to sanitize as it is leaving your control.

Sending to display to the user: sanitized for a browser

Sending to an API: sanitize for whatever rules the API has

Sending to a legacy system: sanitize for it

Writing a file to the system: sanitize the path

The common point is you don't sanitize before you have to send it somewhere. And the advantage of this method is that you limit the chances of getting bit by reflected injections. You interrogate some API you don't control, you may just get malicious content, but you sanitize when sending it so all is good. Because you're sanitizing on output and not on input.

afiori2• 2 years ago

Forbidding users to use your service to propagate "litte bobby tables" pseudo-pranks is likely a good choice.

The choice is different if like most apps you are almost only a data sink, but if you are also a data source for others it pays to be cautious.

JodieBenitez• 2 years ago

or if you have to interact with some legacy service.

Which happens almost every day in the real world.

wvh2• 2 years ago

Because you don't want to ever store bad data. There's not point to that, it will just create annoying situations and potential security risks. And the best place to catch bad data is when the user is still present so they can be made aware of the issue (in case they care and are able to solve it). Once they're gone, it becomes nearly impossible and/or very expensive to check what they meant.

beagle3• 2 years ago

You do need to use a canonical representation, or you will have two distinct blobs that look exactly the same, tricking other users of the data (other posters in a forum, customer service people in a company, etc)

lyu07282• 2 years ago

The only validation I'd always enforce is some sane length limit, [..]

Venture into the abyss of UTF-8 and behold the madness of multibyte characters. Diacritics dance devilishly upon characters, deceiving your simple count. Think a letter is but a single entity? Fools! Combining characters lurk in the shadows, binding invisibly, elongating the uninitiated's count into chaos. Every attempt to enumerate the true length of a string in UTF-8 conjures a specter of complications. Behold, a single glyph, yet multiple bytes cackle beneath, a multitude of codepoints coalesce in arcane unison. It is beautiful t he final snuffing of the lie s of Man ALL IS LOST ALL I S LOST the pony he comes he comes he comes the ich or permeates all MY FACE MY FACE ᵒh god no NO NOOO O NΘ stop the an * gles are n ot real ZALGΌ IS TOƝȳ THE PO NY HE COMES

hnfong• 2 years ago

A nit - it's not UTF-8 or "multibyte" characters that's the main problem. The UTF-8 issue can be trivially resolved by decoding it into unicode code points. As long as you're fine with the truncated length not always corresponding to what you'd expect for Latin based alphabets it should be fine. (FWIW, if you are concerned with the displayed length, you'd need a font and a text layout engine to calculate the display length of displayed text)

The main issue with naïve truncation is that not every code point is a character (and I guess not every character is a glyph?). If you truncate the Unicode code point array at some unfortunate places like https://en.wikipedia.org/wiki/Ideographic_Description_Charac... , you'd just get gibberish or potentially very unintended results. (especially if you joined the truncated string with some other string)

kazinator• 2 years ago

treat strings as opaque blobs most of the time

While being transparent is great and better than stupidly mangling or rejecting data, the problem is that if we just pass through anything, there are situations and contexts in which some kinds of software could be used as part of a chain of deception involving Unicode/font tricks.

Passing through is mostly good. Even in software that has to care about this, not every layer through which text passes should have the responsibility.

hedora2• 2 years ago

I’d expect iCloud to accept that name, even though Rachel True’s name breaks the heck out of it:

https://www.reddit.com/r/ProgrammerHumor/comments/lz27ou/she...

dathinab• 2 years ago

fun fact there is a semi standard encoding called wtf-8 which is utf-8 extended in a way so that it can represent non well formed utf-16 (bad surrogate code points)

it's used in situations like when a utf-8 based system has to interact with Windows file paths

rawbert• 2 years ago

OMG, the second screenshot might be actually the application i am working on right now ...

mdavid626• 2 years ago

It's like with phone numbers. Some people assume they contain only digits.

dcow3• 2 years ago

Apropos: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...

rmrfchik2• 2 years ago

Well, the labels of input fields are written in English yet user enters his name in native language.

What's the reason of having name at all? You can call the person by this name. But if I write you my name in my language, what you (not knowing how to read it) can do? Only "hey, still-don't-know-you, here is your info".

In my foreign passport I have name __transliterated__ to Latin alphabet. Shouldn't this be the case for other places?

smatija• 2 years ago

w3c recommends adding a separate field for pronounciation - see e.g. paragraph after second image in https://www.w3.org/International/questions/qa-personal-names...

rmrfchik2• 2 years ago

Wow, that's neat.

int_19h13• 2 years ago

Unfortunately, the extremely weird and idiosyncratic use of Latin script for English means that straightforward transliterations are usually pronounced incorrectly by people who try to pronounce them according to English rules.

And, on the other end of the spectrum, you have attempts to spell phonetically following English orthography, which then causes issues because it's tied to a particular dialect of English. E.g. the traditional Korean romanization of the name "Park" has "r" there solely so that "a" would actually correspond to an [a] sound in British English - but then, of course, American English is rhotic, and so it ends up being incorrectly pronounced as [park].

account4232• 2 years ago

"Usually" it's pretty close to the correct pronounciation. Pretty much always it's good enough to be understandable.

wvh2• 2 years ago

There's little more you can do to validate a name internationally than to provide one textbox and check if it's a valid encoding of Unicode. Maybe you can exclude some control and graphical ranges at best.

Of course there are valid concerns that international names should pass through e.g. local postal services, which would require at least some kind of Latinized representation of name and address. I suppose the Latin alphabet is the most convenient minimal common denominator across writing systems, even though I admit being Euro-centric.

egorfine• 2 years ago

Mandatory mention: https://github.com/minimaxir/big-list-of-naughty-strings

account4232• 2 years ago

1) The "real" WTF-8 charset [0] is a cool and useful encoding.

2) This may be an unpopular opinion but I think restricting name input to latin is an OK thing to do, especially if the entered names are being viewed/used by humans who cannot be expected to be know all scripts.

3) Similarly, internationalized domain names were a mistake. If your business card tells me to go to stępień.com then chances are I won't bother to try end remember how to enter those accents on my keyboard layout. Most users won't even be able to enter them. Worse are letters that are visually indistinguishible - and no, registries preventing confusable names is not enough when I still won't know which letter to enter. This makes IDN domains less useful while retaining all the security issues they bring.

Most languages were already forced to deal with ASCII and have developed standardized ways to "romanize" names and other words to that character set. This solution achieves peak interopability - not only between computer systems but also between the fleshy components operating them.

[0] https://simonsapin.github.io/wtf-8/

Charset="WTF-8"

Related Stories