Resetting PHP6 (or: Unicode claims another victim)

http://lwn.net/SubscriberLink/379909/26c35a974b1bbd65/

by corbet • 16 years ago

61 34 16 years ago

pmjordan2• 16 years ago

Wow. Considering PHP is used primarily for generating HTML, UTF-8 is the de-facto standard encoding for unicode on the web, and PHP files are usually ASCII or UTF-8, going for UTF-16 seems like a phenomenally bad idea. I can see how UTF-16 looked promising back when it was still UCS-2 and there were no surrogate pairs. These days though, UTF-8 and maybe UTF-32 seem to be the realistic choices when working from scratch; UTF-32's advantage in some areas is probably too weak to make it a real contender unless your strings are literally linked lists, not codepoint arrays. (i.e. you don't care that it uses 2-4 times as much memory or storage)

Carlfish• 16 years ago

One advantage of UTF-16 is that unlike UTF-8, very few characters you encounter in Real Life invoke surrogate pairs. So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case.

pmjordan2• 16 years ago

Let's face it though, the main string operation in web apps (or most software) is concatenation. I strongly doubt there is any point in converting back and forth between UTF-16 and UTF-8 just for that tiny advantage in addressing individual code points - and even then, not quite 100% of the time.

For search-and/or-replace (or anything that can be done with regexes), I'm pretty sure that UTF-16 has no advantage over UTF-8, as you can make the state machine operate on the bytes directly.

jerf• 16 years ago

"So for the cost of a one-bit flag per string you can assume two bytes per character in your string operations for the overwhelmingly general case."

Thank you for that clear and concise explanation of the dangers of using UTF-16.

Yes, I know that wasn't your intention, but it was the end result. One of the most dangerous library failures you can have is a function that works 99.99% of the time. Or in this case, 100% of the time on the input the English-speaking developer provides but distinctly less than 100% in the field.

In this specific case, you can't actually optimize anything because all your optimizations are bugs. You can't just divide by two for character count; that's not an optimization, it's a bug. You can't just multiply by two for a substring operation, because you might chop a character in half, that's a bug. And so on. You'd need a separate type that indicates you've scanned the string to verify it never has split chars and now you might as well be on UCS-2, and that has its own dangers w.r.t. working 99.99% of the time.

Much better to use UTF-8, where the dangers are much more apparent, all you have to do is leave the base ASCII case and you're testing UTF-8. Even I, an English-speaking developer, manage to test that case (once I know it exists, anyhow). There's still ways you can screw up but you're off to a much better start.

nradov4• 16 years ago

Java's use of UTF-16 is now generally considered a design flaw. It takes more memory than UTF-8, but still has the complexity of variable-length character encoding. UTF-16 was a reasonable choice for Java at the time the decision was made but you would think the PHP developers would learn from that mistake.

mmastrac2• 16 years ago

[citation needed!]

As a Java developer, I'm very happy with the compromise that UTF-16 strikes. For day-to-day unicode use, UTF-16 covers all the bases. In the rare cases I need to step outside of UTF-16 to the higher planes, it would work transparently for me up to the point where I start slicing strings naively.

To be honest, I've never actually had to use anything outside of the Basic Multilingual Plane. That's not from lack of breadth in my day-to-day job either - I wrote the first implementation of our web crawler/search engine at DotSpots and had to deal with fetching and indexing pages in languages from most of the BMP (mainly English, Russian and Chinese).

Anyways, I strongly disagree with your assertion that Java was "generally considered" to have "made a mistake" by using UTF-16. It's made my life far easier and the memory costs aren't an issue on today's machines. We don't live in an ASCII world most of the time and storing strings internally as UTF-8 completely ignores this fact.

[edit]

Just so it's clear: UTF-8 only wins when you are representing pure ASCII text. For everything else, it either breaks even or loses. For most Chinese text, UTF-8 is 50% larger:

For characters <= U+0080 (ie: ABC), you win by one byte.

For characters > U+0080 but <= U+07FF (ie: Ȁɐ), you break even at two bytes.

Everything higher than U+07FF in the BMP, < U+10000 (ie: 丂且⬄☃), you lose (by one byte).

For characters >= U+10000 you break even again at four bytes.

If you really want to optimize for ASCII text in Java, there's always byte[] and you're free to wrap a CharSequence around it.

wvenable• 16 years ago

PHP developers don't like to reinvent the wheel. They were trying to use existing libraries that stored strings as UTF-16.

Unfortunately since PHP developers don't like to reinvent the wheel, PHP is also based on a large number of 3rd party libraries that aren't unicode aware.

ibsulon2• 16 years ago

I just don't understand this.

A file is very very very simple to convert once. Tell your developers, "If you don't encode in UTF-16, you will have a performance penalty. Set your file encodings as UTF-16 too. You weren't doing complex internationalization work before, it's really not that big a deal."

I worked with someone who had been on the ICU project, and he argued that UTF-16 is the best compromise for most cases. If you're working primarily in the western character set, UTF-8 is attractive, but that comes at the expense of others.

And frankly, if you don't roll it yourself, what are you going to use other than ICU?

drm237• 16 years ago

I would guess file encoding is not at all the limiting factor since often times opcode caches mean that the file is only read once anyway. The problem is that you get input for all sorts of other areas like forms, databases, web services, etc. most of which aren't UTF-16.

wingo• 16 years ago

> And frankly, if you don't roll it yourself, what are you going to use other than ICU?

GNU libunistring: http://www.gnu.org/software/libunistring/

chipsy• 16 years ago

I recall doing a quick assessment of the UTF encodings recently and coming to a similar conclusion:

UTF-8 is nearly ideal for transmission and storage and is fairly robust for manipulation, albeit it can be the least straightforward to implement(not that app developers actually have to implement it).

UTF-32 is probably most useful as an internal optimization for tasks that can really benefit from a straight scan/cut/paste over even-sized memory cells. You wouldn't store it in your database, but you might want to make use of it in a document editor, for example, to speed up search+replace type operations.

UTF-16 is still substantially more heavyweight than UTF-8, but it can't be optimized into straight memory cells like UTF-32 without breaking the spec. So - unless your needs are extremely specific and you discover a sweet spot in UTF-16 after extensive profiling - it's just not a likely candidate.

IgorPartola• 16 years ago

Is anybody interested in buying my book "Definitive guide to PHP17"?

aarongough• 16 years ago

PHP was slated to gain a goto keyword

How on earth did they decide this was a good idea? The points about getting rid of register_globals and safe_mode are great, but why add a feature to a programming language that is highly likely just to result in lots of awful code?

shaunxcode• 16 years ago

because the whole 3 of us that compile lisp type languages into php need it for TCO.

Scriptor• 16 years ago

It's not strictly necessary, tail recursion is enabled in Pharen (http://github.com/scriptor/pharen) by compiling it into a while loop.

By the way, the docs on that page are way out of date so if anyone is interested feel free to contact me.

EDIT: I'm an idiot, for tail calls of course it's a bit more complicated than just using a loop.

pak• 16 years ago

There are ways to use goto effectively... in fact, the other day I was working with an awkwardly nested try/catch that would have read much clearer as a goto (and more efficiently; an empty Exception was being used to trigger the catch).

I've seen this problem "solved" with a do while false and a break, but isn't that even hackier and less expressive?

At this point I think goto-phobia is well understood enough that adding it for occasional use wouldn't ruin the language.

aphyr• 16 years ago

Ruby distinguishes between begin/rescue (analogous to Java's try/catch) and throw/catch. I never understood it until discovering that throw/catch is perfect for dealing with nested cases where rescue or break fail. Also, it works through function calls and isn't interpreted as an error, which makes it great for cases like Ramaze's redirect or render calls; unlike in Rails, they interrupt execution immediately.

tcdent• 16 years ago

This was discussed heavily when 5.3 was released, which back-ported it and the namespaces this article mentions.

http://php.net/releases/5_3_0.php

Batsu• 16 years ago

The page on GOTO actually includes a nice XKCD comic that illustrates both sides of the argument rather well.

http://us3.php.net/goto

[Direct to comic] http://imgs.xkcd.com/comics/goto.png

dhotson• 16 years ago

It's not all bad.. goto is actually a pretty good way to implement a state machine efficiently.

jrockway2• 16 years ago

One of the number one bugs in web apps is assuming that characters can just "flow through" your application, as the article claims is a common case. Sure, if everything is UTF-8, it might work. But the fun comes when some of your data is us-ascii, some is iso8859-1, and some is utf-8. Now treating your data like binary is going to result in a garbled web page. So don't do it; decode data from octets to characters when it comes in your program, manipulate internally as character strings, and encode characters to octets when you output your data. Text is not binary!

And if I were Zed Shaw, this is the part where I'd threaten to kill you if you don't meet my demands.

randallsquared• 16 years ago

Actually, the real problem is mixing Windows-1252 and UTF-8, while working with tools that assume your Windows-1252 is really ISO-8859-1. ASCII, after all, is just a subset of UTF-8, so there's no handling required for it if you're already assuming UTF-8.

pbiggar3• 16 years ago

I think the author has missed the larger problem. The PHP development community is completely dysfunctional. I don't think that a project of the magnitude of PHP 6 is possible without fixing that fundamental problem.

Why is it dysfunctional?

- every discussion leads to bikeshedding (and almost none of the bikeshedders actually commit code to the Zend engine)

- there are 'rules', but they don't apply to most people (ie the 5.4 thing in the article)

- no firm hand to guide them (Rasmus has deliberately not provided this)

- the mailing list has a complete lack of civility

- highest concentration of poisonous people to non-poisonous that I have ever seen

- votes for everything

- patches are not discussed, either pre or post commit, so the code is bad, and people won't work on it.

I was so glad to be the hell out of there.

jrockway2• 16 years ago

Dude, it's PHP.

pbiggar3• 16 years ago

This is the root cause of the problem. Since it's PHP, you can't get really good developers to work on the core. The people who love PHP and the people who have both the desire and skill to do the work described are separate groups.

dasil003• 16 years ago

I expect this pretty much kills PHP 6 in Japan.

Resetting PHP6 (or: Unicode claims another victim)

Related Stories