Accept-Charset Is No More

http://hsivonen.iki.fi/accept-charset/

by charliesome • 14 years ago

54 21 14 years ago

hsivonen.iki.fi

nodata• 14 years ago

Interesting. Related: Can anyone explain to me how different versions of Unicode are handled?

jensnockert• 14 years ago

It isn't, they are backwards compatible.

Eduard• 14 years ago

Huh, is this a hoax? My installation of Chrome still sends Accept-Charset for every HTTP GET request.

hoppipolla2• 14 years ago

It seems like a weird thing to perpetrate a hoax about :)

A more reasonable conclusion might be that the author made a mistake when testing Chrome. Having passed on the reports that Chrome still produces the header, the author has verified that this is indeed the case. I guess he will update the article.

shocks• 14 years ago

Chrome 16.0.912.77 m is sending Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 for me.

hsivonen• 14 years ago

I was wrong to claim Chrome didn't send Accept-Charset.

Before I wrote the post, I opened another charset-related test case in Chrome (on two operating systems even) and after opening that URL and the relevant URL in various browsers thought I had opened the relevant test URL in all browsers I was testing, but apparently I hadn't opened it in Chrome on any OS.

Very embarrassing. Sorry.

So 4/5 OK, 1/5 still to go. Not quite newsworthy yet. :-(

TorKlingberg• 14 years ago

"There is no way for any non-UTF-8 encoding to represent something more, since everything gets converted to UTF-16 internally."

Internally in what? Certainly not servers, but do all major browsers use UTF-16 internally?

Also, does Unicode include all characters from various east Asian encodings now?

hoppipolla2• 14 years ago

The DOM and ECMAScript both assume that strings are sequences of UTF-16 codepoints. So while a browser could use non-UTF-16 internally it wouldn't help much because you would have to convert to UTF-16 in all the externally-facing APIs anyway.

robin_reala• 14 years ago

Nearly 100% true, but not quite: http://mathiasbynens.be/notes/javascript-encoding

obtu2• 14 years ago

JS engines use various strategies depending on actual string content because it's more memory- and cache-friendly.

FigBug• 14 years ago

Windows is UTF-16 internally, unless using the old ASCII functions. NSString on the Mac is UTF-16. I think w_char in Linux is UTF-32. I still think its an awkward statement. It should say Unicode internally, the specific encoding is irrelevant.

evmar• 14 years ago

Even on Linux WebKit and Gecko use UTF-16.

ken• 14 years ago

> Also, does Unicode include all characters from various east Asian encodings now?

I think the answer is either "almost certainly yes" or "it never will", depending on your view of https://en.wikipedia.org/wiki/Unihan

herge• 14 years ago

Is this the 'Our language is not a font, but its own set of characters different from chinese' argument the japanese are complaining about?

NelsonMinar2• 14 years ago

Does anyone know the historical justification for charset negotiation in the first place? I guess the idea was that the browser and the server would agree on what encoding they'd use and the server would auto-translate for the web visitor. But I can't imagine that ever being used in practice.

pilif• 14 years ago

The idea wasn't necessarily for the server to translate because back when this was invented, there was no Unicode and no UTF-8, so there might very well have been no way to losslessly translate.

The client would send an Accept-Charset header to tell the server what it supports and the server would send either the content in the appropriate encoding (if available) or a "406: Not Acceptable" error.

NelsonMinar2• 14 years ago

I think Unicode predates Accept-Charset. Unicode 1.0 was published late 1991. Accept-Charset was formalized in the HTTP 1.0 RFC (1996). Usage presumably predates that, but it's not described in the HTTP 0.9 docs (which say ASCII only!) http://www.w3.org/Protocols/HTTP/AsImplemented.html

But what really matters is widespread acceptance of Unicode and UTF-8 and that was definitely later in coming. Thanks to dchest below for referencing a transcoding server. It's interesting the HTTP/1.0 docs characterize Accept-Encoding as a way to signal the client could handle something other than ASCII and ISO-8859-1.

Google started favoring UTF-8 in search result pages somewhere around 2002 or 2003.

derleth2• 14 years ago

> Does anyone know the historical justification for charset negotiation in the first place?

Back in the Old Days (say, 1995), each OS (family) would have its own variant of 'Extended ASCII': Unix had Latin-1, Windows had its own encoding mostly compatible with Latin-1, and Macintosh computers had Mac OS Roman, aka MacRoman or just macintosh.

It's possible to convert most, say, French texts from any of these to any other without losing anything, but if you don't the other person will have garbage on their end. In theory, if a client said it could only handle a charset that didn't include some of the characters in the source document, you could do something like give them an HTML file that used images to represent what they didn't have. Or something.

dchest• 14 years ago

There was a rather popular Apache fork for Russian websites that did this http://apache.lexa.ru/english/

Before content negotiation, a lot of Russian websites had multiple versions of pages with switches: KOI-8 (mostly used on *nix) / Win / DOS.

Accept-Charset Is No More

Related Stories