Why the #AskObama tweet was garbled on screen

http://www.hanselman.com/blog/WhyTheAskObamaTweetWasGarbledOnScreenKnowYourUTF8UnicodeASCIIAndANSIDecodingMrPresident.aspx

by brianwillis • 15 years ago

322 76 15 years ago

hanselman.com

nextparadigms2• 15 years ago

Why does it say 3 hours ago under the tweet? Wasn't this in real time?

answerly2• 15 years ago

>Wasn't this in real time?

No. Questions were culled from Tweets with the #AskObama hashtag starting on June 30. Some of the questions did come in close to real-time. I think the most recent ones were 5-10 minutes old when presented to the president.

nextparadigms2• 15 years ago

So how did they choose the questions then? Based on retweets, or the Twitter team just picked the ones they liked?

dabent• 15 years ago

That’s a lot of detail, but a very good explanation of what happened.

js2• 15 years ago

tl;dr:

  $ python
  >>> print u"\u2019".encode("utf-8").decode("Windows-1252")

Luyt2• 15 years ago

Also interesting, the Unicode Nazi:

  http://pypi.python.org/pypi/unicode-nazi

yahelc• 15 years ago

'So do political nerds get to moan because the author referred to Boehner as "the Senator"?'

https://twitter.com/douglas/status/89080894018686976

anigbrowl4• 15 years ago

Touchée.

StavrosK3• 15 years ago

Are you female and awesome, or was that just a typo?

anigbrowl4• 15 years ago

I find it infuriating that this sort of thing is still a problem. I'm constantly seeing mangled apostrophes in places like Google reader too.

uxp2• 15 years ago

I would personally blame Wordpress for substituting a common apostrophe ' as the left ‘ and right quote ’ respectfully. Same with quotation marks “ and ” instead of the traditional ".

There is an option to not use "Smart Quotes", but it seems to be enabled by default.

wrs• 15 years ago

Wordpress is for writing, and writing should be properly typeset, and properly-typeset text has proper quotes.

BTW, a note on the term “smart quotes”: that originated when word processors became “smart” about transforming the easy-to-type (but incorrect) ' and " to their proper equivalents automatically. The quotes themselves aren’t smart…they’re just quotes.

Typography nerd out.

giberson• 15 years ago

Maybe I'm just a terrible programmer, but I think the author may be a little over emphasizing the seriousness of the bug. To me, this is one of those throw away issues that you keep in the back of your mind. Unless I'm coding something that is extremely datacentric and critical in that sense. But it's not like its on one of my "top 20 must run tests" or anything. It's always one of those issues I ignore or assume is correct until I find out it isn't. When I find out, it's simply a matter of tossing in a code page translation at either the input or output end and I'm done with it.

Or maybe I've just been fortunate enough to be in an environment where an occasional goof of this caliber doesn't have any serious consequences.

carbonica• 15 years ago

> Or maybe I've just been fortunate enough to be in an environment where an occasional goof of this caliber doesn't have any serious consequences.

Primarily, this means you don't have to support internationalization - which is hardly a bad thing, especially if you work on a startup, where worldwide distribution should be the last thing on your mind. When your product is rendered in over 80 scripts, including right-to-left languages, you can't afford to figure encoding will sort itself later.

city412• 15 years ago

And Hanselman works for Microsoft, where i18n is a big deal. So yes, for someone who's been at MS for a while i18n related issues become second nature. But if you typically are only targeting the United States, it's more understandable to not have these things on the brain.

anigbrowl4• 15 years ago

You're a fine programmer, but you're terrible at UX. I can put up with this sort of thing in a command-line application, but in any other context it's at least as distracting as non-standard GUI widgets or broken color management. Although these are cosmetic rather than functional issues, they're terribly distracting ones.

KeyBoardG• 15 years ago

I wouldn't take it seriously. The article is written very tongue in cheek. If you follow Hanselman you'll see he's a funny guy.

skimbrel• 15 years ago

You're not a terrible programmer, just an uninformed and/or slightly Anglocentric one.

Just last week, my co-worker had to waste two days debugging a two-year-old Sphinx setup (the person who implemented it no longer works with us) because a Japanese user of our blogging service wasn't seeing the post he was looking for in our search feature. The problem was that the conduit feeding the Sphinx indexer was handling Unicode incorrectly (to be specific, it was deleting certain bytes wholesale because this guy believed them never to be valid, and this broke multi-byte sequences horribly). Those two days would have been much better spent working on his current project.

Additionally, not handling Unicode correctly can leave you open to certain types of security holes!

The biggest of these is probably that Unicode means _a string is not an array of bytes_, so naïve allocators for languages with byte-array strings (read: C and its brethren) are susceptible to buffer overruns when handed multi-byte Unicode sequences when they're expecting ASCII.

Here's another one: the Unicode character space contains many, many glyphs that look almost (or in some cases exactly) like ASCII characters. RFC 3492 defines a framework for internationalized domain names (IDN) that encodes non-ASCII code points using ASCII characters, and the most common implementations of this transparently go between the two. This means that you could register "bаnkofamerica.com" (actually xn--bnkofamerica-x9j.com) and put a phishing site there, and people would happily click on the identical-looking URL and give you their bank account. This was pointed out several years ago and most modern browsers have mechanisms in place to defend against it, but your custom application might not unless you're careful to check what you're doing.

There are plenty of blog posts and articles out there designed to tell you how to be safe when dealing with Unicode (and you should assume that you will be). I highly suggest you go read one.

ak2172• 15 years ago

Wrong. By neglecting to properly process encoding, you shut your application off from everyone except the English-speaking population of the world (and even so, with bugs like this one).

Such mistakes were excusable in the 90s. They're the sign of an amateur programmer today.

pak2• 15 years ago

"This is SUCH a classic sloppy programmer mistake that I'm disappointed"

Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody else expected, and you as the poor guy at the end of the chain gets a document with multiple encodings improperly embedded into it. Inevitably you have to make some bad decisions and not all corner cases are handled.

Somebody once reverse-engineered the state chart for how Internet Explorer handles documents with conflicting encoding declarations and I kid you not, it must have had >20 branches spanning a good few pages. Officially, the correct order of precedence is (http://www.w3.org/International/questions/qa-html-encoding-d...):

1. HTTP Content-Type header

2. byte-order mark (BOM)

3. XML declaration

4. meta element

5. link charset attribute

but that's not how every browser does it, because the W3C sort of declared that after things on the Real Internet (TM) had already gotten out of hand. I hate to resuscitate Joel posts but Unicode is not easy to implement right.

eropple• 15 years ago

Twitter said that the message in question is UTF-8. The message recipient decoded the message with something that is not UTF-8.

That's a sloppy mistake and should without doubt have been caught in testing.

matthavener• 15 years ago

Yup, it can also be much more serious than some garbled text. Google "canonicalization vulnerabilities" and see stuff like this: http://www.amarjit.info/2009/09/canonicalization-vulnerabili...

pak2• 15 years ago

How do you (or the OP) know the implementation details of the consumption path for display? The author assumes it was the direct JSON feed from Twitter. That's overly simplistic and betrays his inability to understand how these sort of things are put together. This is a public, real-time event; more likely than not tweets were filtered through one or two other components to separate the interesting material from the noise. You have no idea what those intermediary components were, who set them up, and if they were all decoding/reencoding correctly. Perhaps they were going through other Twitter users' clients, which could have contributed to the bug.

Edit: yes, there were definitely people curating the questions, so it could not have been as simple as pulling one JSON feed. http://www.theatlantic.com/politics/archive/2011/07/how-obam... https://twitter.com/#!/townhall/july-6-curators

sundarurfriend• 15 years ago

The company responsible seems to have responded in the comments:

"It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to send UTF-8 with text/html even though the HTML file was encoded as such. It was the only issue (albeit a pretty obvious one) during an otherwise flawless event. I apologize to President Obama, Speaker Boehner, and Jack Dorsey for the mistake. If the readers of the blog think it was stupid, imagine how we felt. dev environment != production environment. If we would have just included a <meta charset="utf-8"> in the HTML head, then this would not have occurred.

The big take away is don’t make assumptions about other platforms (especially when it comes to encoding), and always include charset meta tag." [emphasis mine]

yuhong• 15 years ago

Actually, the correct order is defined in the HTML5 spec now: http://dev.w3.org/html5/spec/parsing.html#determining-the-ch...

ak2172• 15 years ago

Really, it's not that hard. 99% of the job is done for you. Just use some UTF-8 encoded text to test your app before letting the president of the United States use it.

There is no excuse for not testing your app for basic i18n brokenness. It's 2011, not 1998.

pavel_lishin• 15 years ago

Did they see the encoding mistake before they showed it?

Because I wonder how difficult it would be to create a string that says something innocuous in UTF-8 (e.g., "When will you bring the troops home #AskObama") but in ASCII would read as something totally different, but legible (e.g., "the secret priests would take great Cthulhu from his tomb to revive His subjects and resume his rule of earth...")

StavrosK3• 15 years ago

I imagine quite difficult, as each character triplet when decoded with Windows-1252 would have to be one letter in Unicode, and those would have to actually form words. You'd be restricted to maybe 30 triplets.

irahul• 15 years ago

If you are talking about ASCII(7 bit ascii and not extended ascii), then it can't happen - the encoding is same for 0-127.

luigi• 15 years ago

What modern software stack uses Extended ASCII as its default encoding? The last time I dealt with this problem, it was in 2005 or 2006 and I was working with PHP.

samfoo• 15 years ago

My guess would be that it's a flash app. Flash is (or at least used to be) horrible at internationalization issues like this.

cpeterso• 15 years ago

The article's comments says the problem was that the (implicit) character encoding of the HTML page:

"The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1."

zach2• 15 years ago

Proof that even completely ordinary string data used in the most USA-centric domain imaginable STILL needs proper encoding.

baddox• 15 years ago

I wouldn't call the right single quotation mark "completely ordinary string data." As far as I know, the only way it would ever get into a tweet is through some "smart correcting" client, or purposeful manual entry.

zach2• 15 years ago

A healthy amount of the writing of the world still begins life in a word processor.

Any string born of this heritage is likely to have single and double (curved) quotation marks. OpenOffice, Word, Pages – they all do it and are expected to.

For that reason, I consider them to be reasonably present in ordinary string data.

ignifero3• 15 years ago

So does HN support utf8 ¢ðrrê¢†l¥?

Luyt2• 15 years ago

˙sǝʎ 'os ʞuıɥʇ I

there• 15 years ago

(ಠ_ಠ)

ignifero3• 15 years ago

You have to give it to Microsoft. They use Word even for 140 letter documents now! (I understand spelling is a reason, but browsers have spelling now)

mgkimsal3• 15 years ago

Downvoted, understandably perhaps, but nearly every time I see this sort of bug it's because someone has copy/pasted something from MS Word instead of using whatever native client input method was provided (usually a textarea or input field). Yes, blame the rendering script for not encoding properly, but I suspect this was Boehner's team copy/pasting from some internal MS Word doc. Cheap shot - they also likely copy/paste screenshots in to Word and mail those around too, instead of just mailing the actual graphic file. :)

ben1040• 15 years ago

Cheap shot - they also likely copy/paste screenshots in to Word and mail those around too, instead of just mailing the actual graphic file. :)

When I was in consulting doing software integration work, few things infuriated me more than client "bug reports" arriving in the form of an email containing a 15 megabyte MS Word doc with a bunch of un-annotated screenshots.

I really hope Google someday opens up the awesome bug report/screencap feature in Google+, that lets you highlight part of the screen and redact sensitive parts.

joeyh3• 15 years ago

Here´s my guess. This political tweet was one of many thought up by speechwriters, entered in a word document. It was passed around a few times to make sure it hit the right talking points, perhaps read to Boehner over the phone, and eventually someone copypasted it to the internet.

Jach2• 15 years ago

Well, it could have been worse. They could have shown \'

zipdog• 15 years ago

What bugs me is not the mis-encoding (though that's a fail), but that people "struggled to understand" it... surely everyone's seen apostrophes turn into these special characters on enough web pages over the past ten years to have recognized what's happening when it does.

dthorpe• 15 years ago

Blame Hollywood. Non-tech folks have been conditioned to panic whenever anything out of the ordinary appears on screen, because that's how Hollywood visualizes computer viruses and alien space invaders.

markbnine• 15 years ago

What a cool shot. The prez with a common bug on his screen. And a bug I can fix! Still, even though this is an easy fix, he's going to need to open up a ticket.

sacrilicious• 15 years ago

My only regret is that I have but one upvote to give to this comment.

gutini• 15 years ago

An error like this should not detract from the value Mass Relevance delivers. Clearly an event like this or similar events like the Oscars in which they take part are better, more engaging because of their involvement.

Jach2• 15 years ago

Zed Shaw, can you make a post called "Programmers Need To Learn Unicode Or I Will Kill Them All"?

mahmud2• 15 years ago

Joel Spolsky wrote something that comes close, sans the threat:

http://www.joelonsoftware.com/articles/Unicode.html

mahmud2• 15 years ago

He had to do a twitter townhall because El Jefe couldn't get a G+ invite :-|

efalcao3• 15 years ago

Hey I work for the company responsible for the visualization behind the president and the content on http://askobama.twitter.com

Let me take this very excellent opportunity to say that we are looking to hire a full time "front end" developer. You'll get to work on badass projects like the Obama Town Hall. Ideally, you'd be located in Austin. Find me on Twitter @efalcao to learn more.

efalcao3• 15 years ago

FWIW, this was an intense project to pull off. 1000's of tweets per minute from Twitter, 8000 requests per second on http://askobama.twitter.com (where the same tweet was also delivered by us and rendered correctly).

We're not lazy or sloppy... It basically boiled down to one server sent down the right header...the production one didn't.

Unicode issues are sorta in the class of "gotcha" issue. They happen, you go "oh shit" and fix them right away. Our "oh shit" moment just happened to come at the most intense possible moment....in front of the president, with so many watching.

Wanted to reiterate once again: We're Hiring! @efalcao on twitter. Early stage startup looking for exceptional talent.

thinkbohemian• 15 years ago

I think you did a great job, way to represent the Austin Tech scene!

joeyh3• 15 years ago

tl;dr -- mojibake

pilif• 15 years ago

Errors like this is what me and my coworkers jokingly refer to as US-UTF8 (no offense meant). In a country that's dominated by ASCII, "supporting" UTF8 means "emitting the same data as usual but declare it as UTF8).

Sure there might be some misunderstandings with special punctuation characters as evidenced by the article, but such issues generally get low priority.

In countries where the language isn't representable in ASCII, we can't use US-UTF8, but have to resort to "real-UTF8" which means dealing with legacy systems that don't do UTF8 (which is what happened in the article we're currently commenting on), dealing with browsers who lie about encoding, and dealing with the fact that a string length isn't its byte length any more even if it doesn't contain "fancy" punctuation characters.

All that makes me wish I could do US-UTF8 too :-)

winsbe01• 15 years ago

thank you for the fascinating article! I've seen this bug in other places, and I never knew what it was (and usually brushed it off instead of digging deeper to find it).

kragen• 15 years ago

Part of the problem is that UTF-8 makes things really, really simple, and bulletproof, and then people have to go and create problems again.

Listen. Any time you use an encoding other than UTF-8, you are creating incompatibilities. If your stated intention is to facilitate communication, you are failing. You are a bad person. Stop doing it. The only possible excuse for using a non-UTF-8 encoding is to frustrate communication.

(It's too fucking bad HTTP mandates that the default charset is ISO-8859-1.)

Why the #AskObama tweet was garbled on screen

Related Stories