Unicode, Perl 6, and You

https://perl6advent.wordpress.com/2015/12/07/day-7-unicode-perl-6-and-you/

by kamaal • 11 years ago

91 21 11 years ago

perl6advent.wordpress.com

wtetzner• 11 years ago

I haven't finished reading through the whole post yet, but if Perl 6 works on graphemes, are ligatures considered to be only one character?

lelf2• 11 years ago

Yes. Bear in mind however that ligatures are in Unicode only for the backward compatibility.

  > 'ﬄ'.chars
  1

logicallee• 11 years ago

that's sure to ruﬄe some feathers.

(see what I did there)

SwellJoe2• 11 years ago

Perl has had decent Unicode support longer than most similar languages (years before Ruby and Python, for instance), but Perl 6 is just ridiculously good at it, and I hope other languages follow suit. I'm unaware of any other language that handles Unicode this well...am I missing any languages that do? I guess JavaScript is coming along on this front and ES6 includes support for Unicode regexps, which is progress, so maybe that's the closest mainstream language.

lelf2• 11 years ago

ES6 includes support for Unicode regexps

Will it provide \X for example? (\X matches extended grapheme cluster.)

am I missing any languages that do?

Swift is one notable example. It has built-in and simple enough grapheme handling.

SwellJoe2• 11 years ago

I haven't looked at Swift, at all. I don't buy Apple products, so have no familiarity with their ecosystem. But, now that it's been opened up, I'll have a look at it, though it seems likely to remain predominantly a language for Apple products for the foreseeable future (I think?), so not something I'd find myself using in production any time soon. But, I guess we'll see how that shakes out over time now that it is open.

Given the rate at which JavaScript is converging on a really nice set of modern features and is having warts removed and performance is accelerating, I wonder if any other language is as relevant long-term.

saurik• 11 years ago

Swift's Unicode support is sufficiently awesome that my web browser was having issues rendering the documentation for their string class due to the epic working examples ;P.

OopsCriticality2• 11 years ago

...am I missing any languages that do?

Tcl supported Unicode in 1999 with version 8.1. Much like Zathras, Tcl is the beast of burden that is easy to overlook.

kbenson2• 11 years ago

Well, it's not about "supports unicode" as much as it's about the level of support. IIRC, I've heard that Tcl had fairly good unicode support (at least for the time), but I have no idea how it compares to some contemporary versions of languages, or Perl 6 specifically.

ploxiln• 11 years ago

On the downside, even today, Tcl can't handle characters outside the basic multi-lingual plane. It only does UCS-2, it can't handle UTF16 surrogate pairs. If you convert an astral-plane codepoint, such as some popular emoji, from UTF8, TCL will convert each UTF8 byte into a separate unicode codepoint. There are similar catches all over, it's just not practical to deal with non-BMP codepoints in TCL, even just to round-trip them.

bitserf• 11 years ago

Swift does a pretty good job as well for a first attempt.

https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

kbenson2• 11 years ago

Don’t worry though, standard Perl 6 does not demand that you be able to type Unicode. If you can’t, there are so-called “Texas” variants:

I've always loved the "everything's bigger in Texas" joke implicit inthe "texas" variant on some operators.

If you’re interested in working within a particular normalization, there’s the self-explanatory types of NFC, NFD, NFKC, and NFKD.

That would probably be better with a "Well, it's self explanatory at the point you know you want to work in a particular normalization", since I only vaguely know what those are, and I've beenhearing about some of them for years. ;)

Great post though!

Grue32• 11 years ago

say "नि".codes; # returns 3

How is "नि" 3 codepoints? There is only two: न and ि . Could this be a bug?

kamaal• 11 years ago

Not sure how to interpret this, are you suggesting the hindi नि should be two code points because when translated to English it is 'Ni'(two letters)?

Grue32• 11 years ago

I'm suggesting this string has two Unicode codepoints. I don't know anything about Hindi language or Devanagari script.

EdiX• 11 years ago

Hexdumped the html of the webpage, there's only two codepoints here.

hotkeys• 11 years ago

They were probably using an outdated version of rakudo, the latest version returns 2 as you would expect.

cursork• 11 years ago

Seems to be a typo?

    $ perl6 -e 'say "नि".codes'
    2

cygx• 11 years ago

Perl6 agrees that it consists of only 2 codepoints:

    say "नि".NFC>>.uniname.perl;
    #=> ("DEVANAGARI LETTER NA", "DEVANAGARI VOWEL SIGN I")

Nothing more serious than a typo in the article would be my guess.

Unicode, Perl 6, and You

Related Stories