Which character should represent the apostrophe? The Unicode committee is wrong

https://tedclancy.wordpress.com/2015/06/03/which-unicode-character-should-represent-the-english-apostrophe-and-why-the-unicode-committee-is-very-wrong/

by AndrewDucker • 11 years ago

52 20 11 years ago

tedclancy.wordpress.com

transfire• 11 years ago

PuffinBlue• 11 years ago

Is this one of those things that everyone agrees (apart from the emotionally invested people with the power to change it) is absolutely completely infuriatingly obvious and should be changed immediately?

ben0x539• 11 years ago

I'm gonna say I'm rather happy that \w doesn't match apostrophes, aka string literal delimiters in some environments.

sp332• 11 years ago

MODIFIER LETTER APOSTOPHE shouldn't delimit anything. Use RIGHT SINGLE QUOTATION MARK or an ASCII apostrophe.

GauntletWizard• 11 years ago

If this proposal is accepted, \w will still not match the apostrophe character you use in coding (Ascii 0x27, unicode U+0027); Rather, it will match an apostrophe character (U+02BC) that represents a part of a word, intended for conveying the semantics of the apostrophe when used in books and language, not in common code.

dragonwriter2• 11 years ago

U+0027 (APOSTROPHE) is a string (or character) literal delimiter in some environments, and it might be sane (though inconvenient, for typing) to have an environment where U+2018 and U+2019 (LEFT and RIGHT SINGLE QUOTATION MARK) were character/string delimiters, but there is no sane reason U+02BC (APOSTROPHE, MODIFIER LETTER) should ever be a delimiter of any kind.

This article is about the argument that Unicode Committee recommendation that U+2019 (RIGHT SINGLE QUOTATION MARK) is the preferred character for apostrophe in English text is wrong (since, among other things, it breaks detecting matched pairs of quotation marks), and that the preferred character for that use should be U+02BC (APOSTROPHE, MODIFIER LETTER).

anjbe• 11 years ago

There are other situations where pairs of quotation marks can be unmatched. For example, in English it’s common to have an opening quote at the beginning of each paragraph of a multi‐paragraph quote, but only close it on the final paragraph.

Raphmedia• 11 years ago

Ever had a ” stealth its way into your code, well hidden amongst a bunch of " ? Perhaps a ’ in a bunch of ' ?

It's enough to drive you crazy.

nevdka• 11 years ago

Far‚ far too often. I work in a mailhouse‚ and clients will often send requirements in .docx files‚ and data in xlsx files‚ which ‘helpfully’ convert to Windows-1252. “O’Hea” seems to be the worst offender.

Mandatum• 11 years ago

Oh god. Never, ever copy-paste from Word documents when running SMS campaigns. Unicode characters are rampant and take up 6x as much character data, so when you're already pushing the limits with your message you end up going well over the 155 char limit for a message.

jimrandomh• 11 years ago

What? Apostrophe is U+0027. The distinctions created by these extra Unicode characters will never be faithfully reflected in most text, because English speakers only have one key to type them with and don't care.

Avitas• 11 years ago

The author's point directly conflicts with your suggestion of using U+0027. The author correctly states that the apostrophe when used inside a contracted English word should be treated as a modifier letter. There is a Unicode apostrophe character set aside and labelled as exactly this--U+02BC.

The author's point makes sense to me. I agree and my initial thought is that the correct thing to do is recommend U+02BC be the apostrophe.

This is something for the future. It won't be faithfully reflected in text for years or decades.

mark-r• 11 years ago

There's plenty of existing software that already takes those keyboard keys and converts them to a different character based on context. Changing which character is used for the conversion shouldn't be a big deal.

userbinator2• 11 years ago

U+0027 has worked for me everywhere I've used it, and no one has complained that I was using the wrong character.

On the other hand, I've had to track down a ton of mysterious bugs caused by software "helpfully" converting characters out of the standard ASCII set, unbeknownst to the user (and they look basically the same)... only a hexdump shows the truth.

_kst_• 11 years ago

Perl 5.22 adds a new kind of word boundary, spelled "\b{wb}" as opposed to plain "\b", that recognizes that apostrophes can occur in the middle of words.

brianberns2• 11 years ago

Since there's only one key on the keyboard that represents both single-quote and apostrophe, software has to guess which one you mean from context. I don't think there's anything Unicode alone can do to fix this underlying problem.

dragonwriter2• 11 years ago

Since there's only one key on the keyboard that represents both single-quote and apostrophe, software has to guess which one you mean from context.

That's not at all true; while "guess from context" is one option, using modifier keys is also an option; there's a lot of software that does this for things which don't have distinct keys. Certainly seen this used for different widths of spaces, and different widths of dashes, even though the keyboard has only the space bar and the hyphen.

brianberns2• 11 years ago

Only a small fraction of users know how (and are willing) to use modifier keys like that. I don't think it's practical to assume they'll change their behavior in this case.

wodenokoto• 11 years ago

The author bases his entire argument on the assumption that "don't" is obviously a single word in English and that there is no doubt about that.

I for one was quite surprised about that. When doing word segmentation for NLP it is common to split these contractions.

Plenty of people argue that it is a single word because there are spaces around it or simply because it is a contraction. Other argue it is 2 words since they are of different word classes.

Which character should represent the apostrophe? The Unicode committee is wrong

Related Stories