A Localization Horror Story: It Could Happen to You

http://search.cpan.org/dist/Locale-Maketext/lib/Locale/Maketext/TPJ13.pod?#A_Localization_Horror_Story:_It_Could_Happen_To_You

by pmoriarty • 12 years ago

438 249 12 years ago

search.cpan.org

bmn_9• 12 years ago

Article is from 1998 and very much out of date. Read: http://blogs.perl.org/users/aristotle/2011/04/stop-using-mak...

reacweb• 12 years ago

It is funny to have such a good article with so many comments at the end that disagree. The issue does not seem to be so clear cut.

bmn_9• 12 years ago

Miyagawa makes two claims.

1. L::M::L::G enables PO-based work-flow. This is not correct except for trivial lexicons. L::M::L::G cannot parse the plural extract from the article, so it is incompatible with properly working PO emitters.

2. Maketext is used on typepad and "works really well". This claim is not backed by any evidence. The verifiable facts from the article pointing out the pluralisation problems of L::M still stand unchallenged.

EvaK_de5• 12 years ago

Should be reflected in the title, if possible, neh?

lmm8• 12 years ago

Cool example, but I think "ne" is the "correct" way to romanize ね.

aerovistae• 12 years ago

That is so, so awesome.

nodata• 12 years ago

What if the text wasn't a sentence?

bmn_9• 12 years ago

If the text is not a whole sentence or paragraph, it cannot be accurately translated anymore into a number of languages with different word order.

As a programmer with i18n responsibilities, you learn to identify and fix fragmented text problems first.

olau• 12 years ago

Just in case people are wondering about the horror story: use ngettext which is a function in the gettext library.

gulpahum• 12 years ago

Indeed, this is a solved problem. Use ngettext, see its page for all different language variations (even the Slovenian four different forms) https://www.gnu.org/software/gettext/manual/html_node/Plural...

philh• 12 years ago

How does ngettext handle "Your query matched %g files in %g directories"?

tinganho7• 12 years ago

ngettext can't handle two plural words well.

reidrac2• 12 years ago

Slightly OT, mi favourite localization error was in Ubuntu when they had that nice netbook interface (that later would become Unity). The network icon label was "Rojo" in the Spanish localization, that is the word for "red" color. What?

Well, if you translate "Net" to Spanish you get "Red"; and if you translate that again (by mistake), you get "Rojo". There you are :)

baby2• 12 years ago

OT means Out of Topic? In french we say HS (for Hors Sujet).

talideon• 12 years ago

It means 'off topic'.

Kiro• 12 years ago

I'm really surprised that this is the first time you've seen "OT" being used. I can only presume it's because you don't normally ready English websites/discussion boards but I still find it intriguing that you've never encountered it before.

In Sweden we use "OT" as well but it's referring to "Off Topic" and not something Swedish.

chucksmash• 12 years ago

A fitting reply in the localization discussion!

me_bx• 12 years ago

HS may also mean Hors Service (Out-of-order)

psykovsky4• 12 years ago

That's an error only an amateur translator would make. Guess who makes free software translations...

A professional translator makes sure to check the context of the translation, it doesn't go blindly translating sentences and words without context.

reidrac2• 12 years ago

To be fair it was a beta of the "Ubuntu Netbook Remix" and that bug was shortly fixed.

I took some screenshots because I really liked the interface, but none of them show the bug because I usually stick to en_GB and the problem was spotted in my wife's netbook (she's Spanish teacher and likes to have the OS interface in Spanish to amuse the kids).

ygra8• 12 years ago

Ah, that's why Word 2007 ended up with »Keine Gliederung« (»no (document) outline«) for the setting where you can select a shape outline colour ...

In all seriousness, translators make mistakes by getting the context wrong all the time; this applies to both professional translators and amateurs, perhaps in different severity. But in my eyes that's not really a problem of the translators, but rather of the tools we use. A long list of strings that have maybe only a vague context is a horribly UX for the translator. Who would fault them to translate »outline« as »Gliederung« in a word processing application? A string table is probably the most convenient format for programmers, but not very much so for those maintaining translations.

Qt gets a bit of this right. If the translator has access to the source code and uses Qt Linguist, at least the strings that are directly in .ui files are shown with a mock-up of the respective window where the control with that text is highlighted [0]. That already helps a lot with context errors. Of course, it does nothing for text in the source code, and so our translator went ahead and translated »Breite« (line width) with »latitude« because the application in question was, after all, related to maps, geography and GPS; it just happened to have a setting to change the width of certain lines, which, in German, then mapped to the same word.

Qt also has a nice way of handling plurals in that both Linguist and the QTranslator class are aware of the languages and the special rules concerning plural forms in them. You create a translatable string like »Searched %n directories« and then create translations for that. English, German and lots of others just map to two forms (1, rest). Russian gets three, I think (1/21/31/..., 2/22/32/..., 11/12/rest), and so on. Downside is that you only get to handle a single plural in a string [1], and you have to create an en→en translation as well (to account for multiple forms in the source language). But generally it's a quite nice implementation. Gettext has something similar where you can write your own plural form matching rules somehow, but most translators I met don't really want to write math.

Visual Studio with Windows Forms has a mode of creating the string tables for translation where you can just change the language in the Form's properties and then proceed to change each control text. This nicely solves the context problem in that the translator edits the window directly and it looks like it normally does. But it also creates a whole bunch of other problems: Translators need VS, they need to edit the project directly and can accidentally mess up the UI with a mouse twitch when selecting a button to translate. They also might miss things that are buried in menus since there is no real measurement of completion and what's still missing. I've seen various projects, especially in web environments adopt a similar custom-written approach, though, where you can edit the UI directly in the application to translate it. Still with the problem that translators might miss hard-to-find and buried strings (one might argue that you should get rid of buried and obscure places in your UI anyway, though).

Long ramblings ... it was a topic I considered for my Diploma thesis while studying. But I couldn't think of a good way that could retain context for the translator in a general case, or at least most of the time. I thought about replacing each and every translatable string in a program with a custom identifier and then later trying to find those again via UI Automation or maybe screenshots and OCR to be able to map strings to parts of screenshots of the program. Would have required running the program once with those custom identifiers and once in normal mode and somehow matching up identifiered screenshots, normal screenshots and the string tables. And still with the problem that you'd need to manually go down each and every dialog and menu, including context menus, messages that only appear in certain states, etc.

Perhaps there just is no good solution, except maybe for developers to properly annotate each translatable string they use. In the »Breite«/»width«/»latitude« case I went through the source and added translator comments detailing the meaning of the word for every instance, but with large applications having thousands of translatable strings that could become unwieldy quickly.

__________

[0] http://geoinformatics.fsv.cvut.cz/wiki/images/thumb/3/39/Qt_...

[1] http://stackoverflow.com/q/5348990/73070

BjoernKW• 12 years ago

An amateur error perhaps but hardly limited to free software.

In a particularly expensive piece of enterprise software for the finance sector that was translated into German a form field was labelled 'Hauptstadt', which translates back into 'capital (city)' as in Washington, D.C.

Now, this kind of geographic information doesn't make sense at all in financial software but the well-paid developer (who had to double as the software's translator because hiring a proper translator would of course have been too expensive ...) simply couldn't be bothered to look up the different words 'capital' translates to in German, one of which also is the right choice in that case: 'Kapital' (meaning 'capital' as in 'asset).

TeMPOraL18• 12 years ago

A professional translator makes sure to check the context of the translation, it doesn't go blindly translating sentences and words without context.

From my experience with both software and movies, that's exactly what a professional, paid translator does (or is forced to).

Amateurs at least watch the movie/run the software before translating things, which is something I definitely cannot say about "professionals" translating movies and software to my language.

masklinn3• 12 years ago

A professional translator makes sure to check the context of the translation, it doesn't go blindly translating sentences and words without context.

1. professional translators are human and make mistakes

2. context may not be available for third-party translators working on e.g. unreleased software strings (see currently top comment by "eloisant")

tjradcliffe2• 12 years ago

I once got a Spanish translation back and was briefly angry that the translator had left a few strings as "TODO"...

"What?!" I asked myself. "To do? Why did they ship me an incomplete file?"

Then I looked to see what the "untranslated" strings were: "ALL" (I don't speak Spanish but have enough of a smattering of European languages that it was immediately obvious what was going on.)

hpshelton• 11 years ago

When I worked on Outlook.com, we got feedback from a British user that he couldn't understand what the option to "Connect devices and apps with DAD" meant. Turns out we had forgotten to prevent localization of the "POP" protocol everywhere it was used.

codefisher• 12 years ago

I saw most of this coming. I have studied a little Ancient Greek which has the same problem of the Arabic, and Polish which is similar to the Russian, and now I am living in Italy. I guess it is one of those things though as programmers that we just forget about too much, and just expect translation to a mechanical process in the last stage of the development cycle.

gldalmaso• 12 years ago

In order to avoid these pitfalls, usually I get out of "sentence" mode to "label" mode. For instance: "Directories scanned: 12". Probably not well suited for all cases, but usually good enough for mine, though actually I only have to support pt-BR, es-ES and en-US so maybe that's not saying much.

MichaelGG2• 12 years ago

Exactly. All this work, or just restructure the message. It should be acceptable in most languages, because charts and spreadsheets aren't going to have per cell labels. And it has the benefit of being easier to read and parse.

Also, it's a really terrible style to use first person in an app unless it's actually sentient. Otherwise it's annoyingly like Clippy, or just plain obnoxious and presumptive.

TeMPOraL18• 12 years ago

Also, it's a really terrible style to use first person in an app unless it's actually sentient. Otherwise it's annoyingly like Clippy, or just plain obnoxious and presumptive.

I agree, though I found another nice use case, well demonstrated by Bret Victor[0][1]. I played around with it for a while and I find that describing what something will happen in a normal sentence, parts of which you can tweak, is a pretty good way of doing options pages.

[0] http://worrydream.com/#!/TenBrighterIdeas [1] - http://worrydream.com/Tangle/

mrfoto3• 12 years ago

Funny how Slovene seems to tick all the complications checkboxes :D We have 4 grammatical numbers (singular, dual, plural for 3 and 4, plural for 5 and above), they repeat at mod 100 (so 101 is singular, 102 dual,…), it's an inflectional language with 3 grammatical genders, sentence should take a different form depending on whether the user is male or female,…

gambiting2• 12 years ago

I was just trying to think of how it would work out in Polish....

directory is "katalog" in Polish, and it would be:

1 katalog

2 katalogi

3 katalogi

4 katalogi

5 katalogów

6 katalogów

....(it doesn't change for any number greater than 5)

But if you wanted to say "X files were found in Y directories" then you would have to say:

....1 katalogu

....2 katalogach

....3 katalogach

....1000 katalogów

(for 1001 even I am not sure if you should say katalogów or katalogach, both sound correct to me)

...and again, the whole thing changes depending on the speaker being male/female + singular/plural("I have found"/"we have found").

Compared to the grammar of Slavic languages, English is super easy.

mrfoto3• 12 years ago

1 imenik 2 imenika 3 imeniki 4 imeniki 5 imenikov 6 imenikov … 101 imenik 102 imenika 103 imeniki … 201 imenik … 1001 imenik … Slavic languages are f*d up :D

Swizec5• 12 years ago

Compared to the grammar of Slavic languages, English is super easy.

And thus is solved the mystery all us Slavic speakers have such good English. Especially on the online. Because nobody had the guts to localize to our languages.

If they did localize to our languages "Omg this looks weird and smells off", we all cried. And switched our interfaces to English. Admit it, you did this in High School or even Middle School because the <insert slavic language> translation just didn't make sense.

And so all of us who are now in a position to localize these interfaces don't because it seems silly and pointless.

We come full circle. Younglings after us use computers in English because no good localization exists.

tinganho7• 12 years ago

ICU's messageformat solves this easily. One project that supports ICU's messageformat is L10ns http://l10ns.org

nathell• 12 years ago

And don't even get me started at the case where you don't know the noun beforehand and need to synthesize the entire numeral phrase at runtime ("You have %d %s.")

Because, in Polish, numerals inflect in gender and case, come in two main variants ("normal" and "collective", and that's not including ordinals), and can (but not necessarily have to) for some genders undergo case changes that affect the verb part of the sentence under certain circumstances. Add to this the fact that there's even no definitive consensus on how many grammatical genders there are in Polish (opinions range from 3 to 9, with some of the theories based on numeral connectivity), and you're all set.

The Polish word "dwa" (two) has at least seventeen distinct grammatical forms, each of which has arcane rules that govern its usage.

kruczek• 12 years ago

....(it doesn't change for any number greater than 5)

Not so lucky there either:

....21 katalogów

....22 katalogi

....23 katalogi

....24 katalogi

....25 katalogów

And the same pattern repeats modulo 10.

EDIT: And as I think about the 1001 part... Actually the following sounds right to me (translating "in X directories"):

....w 1000 katalogów (w tysiącu katalogów)

....w 1001 katalogu (w tysiąc-jednym katalogu)

....w 1002 katalogach (w tysiąc-dwóch katalogach)

etc.

lepouet2• 12 years ago

Wikipedia mention only singular, dual and plural ? What is the one for 3 and 4 ?

http://en.wikipedia.org/wiki/Slovene_grammar

lepouet2• 12 years ago

I think i found the solution here : http://en.wikipedia.org/wiki/Dual_%28grammatical_number%29

mrfoto3• 12 years ago

No idea how it's officially called, but I'm not making this up :D

http://localization-guide.readthedocs.org/en/latest/l10n/plu...

eginhard• 12 years ago

It's the normal plural, but 3 and 4 take the nominative, while other numbers take the genitive plural.

po• 12 years ago

Just curious... are Slovene web services more likely to ask your gender after you sign up so they can get the grammar right or do they just go with 'male' or something?

pavel_lishin3• 12 years ago

Russian speaker here; I don't think it would be relevant, since services ought to be addressing you in second person, which I believe is gender-neutral in just about every Slavic language.

tux3• 12 years ago

Qt takes care of that in a really nice way.

You write tr("I scanned %1 directory.", "", count) and it takes cares of applying the correct translation with the right plurals depending on the number.

http://doc.qt.digia.com/4.2/qobject.html#tr

ygra8• 12 years ago

You need to use %n in that case.

tedunangst2• 12 years ago

How does that deal with the case where it should translate to "I didn't scan any directories"? According to the documentation, "In the translated version the variables must still appear." http://doc.qt.digia.com/4.2/linguist-translators.html

pmontra• 12 years ago

Having quite well in mind the most common problems with localization I tend to use strings like this:

Directory searched: 10. Files found: 0.

They are not shiny but a direct translation is OK in Italian and it could be OK in the other languages of the post (Chinese, Arabic and Russian).

lmm8• 12 years ago

I've long found that "externalized" translations in po files (or any equivalent) are more trouble than they're worth, for exactly this reason. Translations need to be functions, so they need to be written in a format that's good for writing functions - i.e. a programming language. What we want is a MessageSource interface, and a bunch of language-specific implementations.

Fortunately I work in Scala, so it's very easy to have an "embedded DSL" that's ordinary, first-class code but not much harder for non-technical translators to read or write than the .po format; we can write helpers for grammatical case or numbers or similar. But having the full power of a programming languages there means that when you hit a case you haven't thought of (and you will), you can fall back to just an if/else.

DangerousPie2• 12 years ago

Interesting, but I am wondering if it is really worth going through all this trouble, just to support a few edge cases.

Personally, I usually don't even notice small mistakes like "1 directories" (or similar mistakes in my native language). Sometimes I will see the correct version somewhere and think "Oh, nice that they thought of that" but I definitely don't expect it.

Are the possible returns of having a "perfect" translation really high enough to justify investing in a much more complex system? I am sure translators who can code functions instead of just putting values into an Excel table will come at quite a premium as well...

PinguTS• 12 years ago

You may not notice it. But there are other people who do.

That is the difference between a very well designed product or a product, which just does the job.

That is the reason why engineers should not design interfaces. That should be left to UI/UX experts. Just the other day, I as an engineer myself, complained to another that his product does the job nicely and looks OK. But it was missing this little twist of a finished product that I'd really like to use. Because the interface was designed how I would have done it myself, because lack of UI/UX knowledge.

DangerousPie2• 12 years ago

Of course, I'm sure there are people who notice this. My question was whether something like this makes enough of a difference to justify the investment.

As a developer you only have a limited amount of funds and time to spend on your product. Your goal is to invest these in a way that gives you the biggest returns. Of course it's great to not just have a translation that "does the job" and if you can do better you definitely should. But if that you takes a lot of work and only has a negligible effect on your sales, shouldn't you be prioritizing other things?

Zarkonnen• 12 years ago

Huh. I actually took a stab at solving this problem using language generation with my final year [project](https://github.com/Zarkonnen/A-Natural-Language-Generator-fo...) at university.

eloisant2• 12 years ago

When I was in Japan I did proof reading for a Japanese feature phone. A major Japanese brand, actually. That was really comical.

There was an Australian guy for English, an German guy, an Italian lady, and me for French. What they did prior to the meeting is: * translate from Japanese to English by Japanese people with a poor English level (maybe the software engineers actually) * translate from weird English to other languages by translators who had only the strings, absolutely no context.

In the meeting we had all the strings, and one person from the manufacturers who had access to the "super-confidential" unreleased device.

More than half of the translations were off because of lack of context. The French guy actually translated "Garbage day" to something like "Shitty day", apparently he thought that was a way to mark in your calendar that you had a really bad day.

Pretty often we had sentences like "delete one", and invariably one of us had to ask "One what? I need to know if it's masculine/feminine/neutral". Of course they didn't prepare to that, it was too late to change the code, so they made us do ugly things like "%n item(s)".

Also the Australian guy was loosing faith into humanity: - That sentence, it's completely wrong, it just doesn't mean anything in English. People will just go "WTF?" when they read that - We're not allowed to change the English strings, they're already validated - .....

TeMPOraL18• 12 years ago

I don't know why nobody seems to put information like "warning: this phone's UI in <your local language> is total and utter crap".

Anyway, what you wrote is exactly why I stick to using all software and webservices - OS, text editors, Facebook, et al. - in en_US instead of my native pl_PL. Because translations are always crappy - even for big players. Lack of context is the key here - translated text often feels out of place, because there usually is some overarching idea behind them that isn't communicated to translators. Then there is lack of consistency. Words in original text often have some site-specific meaning, which tends to also be somehow lost in the translation process. For example, on Facebook the word "like" talks about a well-defined thing, not about the dictionary meaning, so it's totally not ok to randomly replace it with synonyms during translation [0].

I realized at some point that I often look at a crapy translation, guess what was the English original, and then in my mind translate to what it should be in the first place. Because for some strange reason I, the user, have the context, and the paid translation team has not. I guess I'm going to put that into my "Translation issues" file in the "Mysteries of capitalism" drawer, right next to "how on Earth multi-milion media companies can't do a movie translation that isn't a total crap" file. I mean, seriously, you're better off looking for pirated subtitles even if you bought the original because pirates at least seem to have watched the movie they're translating.

</rant>

[0] - I wish more translators would use the approach Jehovah's Witnesses used when doing their own Bible translation. Since it was designed to be studied and analyzed, they preferred accuracy over aesthetics - therefore one of the translation rules was "as much as possible, let's have any given word in original text be always represented by the same word in English". Adhering to that single rule would eliminate like half of the "context missing" problems with software translations.

fennecfoxen• 12 years ago

You know what multi-million movie has a translation that isn't total crap? Frozen. They really put resources into that. You can look up random Disney songs on Youtube in different languages, and then look up the Frozen songs, and you can sort of tell that they've done a better job even if you don't speak the language.

Even relatively obscure languages like Dutch where they usually just watch English-language movies: https://www.youtube.com/watch?v=yOueN0sV2SY

raverbashing2• 12 years ago

I don't know why nobody seems to put information like "warning: this phone's UI in <your local language> is total and utter crap".

This is what makes me use SW and equipment only in English.

Translations are pretty much useless (and of course, Googling the English error messages usually gives the best results)

gjm11• 12 years ago

I'm pretty sure general opinion among everyone other than Jehovah's Witnesses is that their bible translation is not good, and that the same-word rule is one of the reasons why.

And I really don't see how having a similar rule for translations of text in software would eliminate most "context missing" problems. It seems what it would actually do is stop translators even guessing those missing contexts.

tjradcliffe2• 12 years ago

Adhering to that single rule would eliminate like half of the "context missing" problems with software translations.

Sort of. Almost all the issues I have with translators and context come from the same word being used for different things. This is particularly true for words like "date" and "time", which can have different translations depending on context (is it the time of day, or the time the test has been running?)

So as well as using the same word for the same meaning throughout, using different words for similar-yet-subtly-distinct meanings is also required.

kevin_thibedeau3• 12 years ago

I have a tool that does this. I produced machine generated translations of German and French as an expedient. The German messages were corrected by a native speaker and I cleaned up the French as best I could. When French is in use an extra message is printed appealing for someone to edit the weak translations.

I dealt with word order issues by avoiding formatted strings with more than one replacement field. Only one string needs to deal with singular vs plural quantities.

Lack of context was a definite problem with the machine translations. It would inconsistently choose translated words that should have been the same because the short string snippets could not be evaluated for their meaning within the narrow domain of the program using them.

lucio• 12 years ago

Same here. Technical text translation is normally a disaster. I truly hate web pages that switch languages based on IP location, as MSDN sometimes does. In my language the translators always get wrong keywords that should not be translated, specially when they're adjectives. e.g: * "You should use the 'new' keyword..." Often get translated as something like "You should use the word that is not old..." / "Debe usar la nueva palabra..."

emcrazyone2• 12 years ago

@eloisant: I work in automotive where I deal with translations for automotive clusters for one of the largest auto makers. Automotive companies are going to what are called reconfigurables which is basically an instrument cluster with no mechanical gauges; just a screen with gauges rendered by 3D engine. Center stacks too.

I kid you not, the way we translate is to use Google translate as a first pass and then the screens get reviewed by people who know the language. I guess somethings slip through and we evidently pissed off a lot of Chinese folks because of a similar flub. The folks doing the first pass don't know any other languages.

It's quite comical but also a real pain. One of the things I had to work into our code was the left-to-right vs. the right-to-left; you would think it would be just a C-style string but we have to know for text justification semantics.

I don't actually do translations but work on the HMI where the text is displayed. Another pain point is that we have a certain space where text needs to be displayed and everything is fitted using English but after translating to other languages, some strings are much longer than the allotted space.

TeMPOraL18• 12 years ago

I can totally imagine that. Recently I've been helping a guy with tweaking his integrated car navigation/radio/media player and I spent some time browsing through its firmware and file system. I saw the translation strings and man, they were horrible. Also half of the stuff wasn't even translated (though it didn't show anywhere on the UI).

BTW. half of the files and directories were named in Chinese which made my work very "fun", but that's another story...

bcbrown• 12 years ago

I worked for a place that used German as the "placeholder text", to avoid the problem of text longer than the allotted space. German is supposedly about 1.5 times as long as English, so if you can fit the German version, you won't have any problems with other translations being longer than the space.

julenx• 12 years ago

Another pain point is that we have a certain space where text needs to be displayed and everything is fitted using English but after translating to other languages, some strings are much longer than the allotted space.

As a rule of thumb, if you're using English strings to design your UI, you should account for 30% more space so other languages can fit in. Of course, sometimes this might not be enough.

jdcryans3• 12 years ago

and me for French The French guy actually translated "Garbage day" to something like "Shitty day"

So you translated it to "jour de merde"?

mcphage2• 12 years ago

The commenter wasn't the translator, they were the proof-reader:

When I was in Japan I did proof reading for a Japanese feature phone.

eloisant2• 12 years ago

That's the other way around, the guy who did the translation without context wrote "jour de merde", and I saved them from releasing that in their phone calendar app during my one-day job paid in cash.

ademarre• 12 years ago

Your experience speaks to this observation:

With very few exceptions, consumer hardware companies are bad at software.

mark-r• 12 years ago

Yes, I've noticed that since forever. Has anybody ever tried to come up with an explanation?

aardshark• 12 years ago

iOS localization handles this with its stringsdict format:

https://developer.apple.com/library/ios/documentation/MacOSX...

luminarious• 12 years ago

For websites, http://l20n.org seems the most natural version so far. Or is there something better?

tinganho7• 12 years ago

You might want to checkout http://l10ns.org

whizzkid• 12 years ago

It made me remember the Norwegian customer I had.

I needed to write Norwegian localization strings in a YAML file which did not work for some reason.

After 4 hours of debugging, the problem was;

In YAML, the "no:" string (for "norwegian") defined in a YAML file was parsed as a boolean, and this makes the application broke..

mcphage2• 12 years ago

Yeah, I've run into that, also. Really annoying. I had to quote all of the "no": strings. Looked ugly, but what can you do?

eveningcoffee• 12 years ago

TL;DR - when you are facing a localization problem - you have to parametrize the units of measurements in your text.

zamalek2• 12 years ago

Edit: Disregard, turns out things have improved

The %g slots are in an order reverse to what they are in English. You wonder how you'll get gettext to handle that.

I learned C/++ after C# and this is one thing that really got to me. String interpolation in C++ is extremely primitive, which would be fine only if more recent iterations of stdlib had something that wasn't so completely incompetent.

For those who don't use .Net, the Italian translation would have been: "In {1:g} directories contains {0:g} files match your query." It doesn't solve all the problems, but being able to specify indices in your template string does solve many.

bmn_9• 12 years ago

The article is outdated. Gettext does have reordering syntax.

https://www.gnu.org/software/gettext/manual/html_node/c_002d...

ygra8• 12 years ago

Makes me wonder why anyone would ever design a translation system that doesn't have the ability to reorder placeholders from the very first version. Perhaps if the author doesn't know anything about languages andn their differences, but in that case they probably shouldn't write such a library ...

zamalek2• 12 years ago

Thanks for the correction, I'm very pleased to be wrong!

anton_gogolev• 12 years ago

And remember the Turkey Test [1].

[1] http://stackoverflow.com/a/797043/60188

donmb• 12 years ago

This happens when you are a perfectionist.

tinganho7• 12 years ago

Please checkout http://l10ns.org It handles the pluralization case pretty well. It uses ICU's messageformat which is a markup for defining plural formatting http://l10ns.org/docs.html#pluralformat

neilk• 12 years ago

Side note: as you might expect, Wikipedia's internationalization is the only system that attempts to do quantities and other formatting correctly for every goddamn language on the planet, but is considerably easier for translators to work with than the OP's examples (sorry, Sean ;)

I did some work on bringing it to JavaScript and making it HTML-aware, and since then Santhosh Thottingal has vastly extended it and it's become pervasive at Wikipedia. More projects should use it, or at least learn from it.

Demo: http://thottingal.in/projects/js/jquery.i18n/demo/

Github: https://github.com/wikimedia/jquery.i18n

bmn_9• 12 years ago

The Russian demo does not work. Must be 1 котёнок, 2 котёнка.

mynegation• 12 years ago

Ironically, this is exactly the case described in the article, and even the article got it slightly wrong. The rules in pseudocode - for this specific sentence! - are as follows:

  if ((n % 10 == 1) && (n % 100 != 11)) { 
      // singular nominative  (not accusative as article says) - 1 котенок, 101 котенок, 301 котенок
  } else if (n % 10 >= 2 && n % 10 <= 4 && n % 100 <12 && n % 100 > 14) {
      // singular genitive: 2 котенка, 43 котенка, 1024 котенка
  } else {
      // all other cases plural genitive: 5 котят, 11 котят, 212 котят
  }

But this is true only for this sentence because kittens here are the subject, not the object: with Russian verb "есть" (to be, to exist) the literal translation would be "Kitten/Kittens exists/exist belonging to Harry".

Once the declension of the numbered subject(s) turns to accusative: "Гарри гладит 1 котенка" ("Harry pets one kitten"), the rules become much simpler:

  if ((n % 10 == 1) && (n % 100 != 11)) { 
      // singular accusative - 1 котенка, 101 котенка, 301 котенка
  } else {
      // all other cases plural genitive: 2 котят, 43 котят, 5 котят, 11 котят, 212 котят
  }

Concours• 12 years ago

I've been looking for something like this for weeks, no luck till now, I guess I should have searched for internationalization instead of localization in this specific case on google and github. This is a great starting point, thanks a bunch for sharing. Perfect for me.

TeMPOraL18• 12 years ago

I once suggested an idea that maybe instead of strings in tables one could use something better suited for the task at hand like, say, code? Maybe let the tables store not only strings but functions as well, so that you could handle the more complex cases directly?

I remember being hit in the head by gettext manual and told something about translators not knowing how to code.

Heck, I still think it's a neater idea than gettext.

peteretep• 12 years ago

I love this article, and used t send it to none-technical people all the time. <3 Sean Burke, the author

theoh5• 12 years ago

The two Turkish letters dotted and dotless i are often confused by users of poorly localised software. Wikipedia links to a murder case allegedly caused by this: http://en.wikipedia.org/wiki/Dotted_and_dotless_I

A real horror story.

(Less seriously, Unicode has counterintuitive case-changing behaviours with those letters. If you are working outside the Turkish locale and uppercase a dotless I and then lowercase it, it gains a dot. I am curious about this design decision, since it seems like a basic error in operating a the level of glyphs rather than symbols. Or maybe the opposite.)

lmm8• 12 years ago

Upper and lower casing can't be assumed to be inverse; there are plenty of other cases where they will change (e.g. precomposed characters that don't have a precomposed upper case). The correct lower-casing of "I" in English is definitely "i"; the correct upper-casing of "ı" in English is maybe a wrong question, because it just isn't an English letter, so I guess you could argue for leaving it unchanged, but converting it to "I" is probably what the person who wrote "ı" would want to happen when it was upper-cased. Maybe?

theoh5• 12 years ago

With Unicode, why aren't the two Turkish Is just treated as if they have nothing to do with the normal Latin I? The fact that the glyph for uppercase dotless I resembles the glyph for uppercase Latin I should be irrelevant, surely. It's a kind of typographic false friend situation.

Maybe there's a missing level of indirection in Unicode that prevents it from doing this, but I can't see how there could be.

zokier3• 12 years ago

there are plenty of other cases where they will change (e.g. precomposed characters that don't have a precomposed upper case).

Just curious, is there still some cases if you only consider NFKD strings/characters?

pavel_lishin3• 12 years ago

Link to the case in question: http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two...

tedunangst2• 12 years ago

I think if you stab somebody over a text message, there's more going on than just a missing dot. As if it'd be acceptable to kill somebody even if they did call your daughter a prostitute.

The article itself could benefit from corrections: "with a knife on his chest" That doesn't sound so bad.

radnor• 12 years ago

The Turkish I problem is seriously annoying. We recently tried adding Turkish translations to our website, only to find that .NET suddenly treats DataRow("ID") differently than DataRow("id") when your locale is set to Turkish.

masklinn3• 12 years ago

Less seriously, Unicode has counterintuitive case-changing behaviours with those letters. If you are working outside the Turkish locale and uppercase a dotless I and then lowercase it, it gains a dot.

AFAIK the only solution would be to error out when uppercasing a dotless i in a non-turkish locale. Which I'm not sure sounds better. Or going back in time and creating a separate category of i and I for the turkish script.

4ad3• 12 years ago

I wonder if it isn't better to generate the messages as an AST, and have a language generator, the back-end of a compiler really, that generates strings for each language. I'm sure there will be less edge cases that way.

/edit: wow, downvotes.

bmn_9• 12 years ago

Because that would require translators to become programmers.

4ad3• 12 years ago

No it wouldn't.

First, I am talking about generating text. In software. Translators would really only need to check the text so that it's correct, and when it isn't, they should submit bug requests so that the programmers implementing the language back-end can fix them.

Second, an AST is nothing but the result of applying grammar to some input, something which translators are very familiar with (probably more than programmers). It is utterly trivial to teach a translator a new textual representation of what they already know.

It's a tree of nodes connected by arrows. They learned the parsing stage in school. At least here in my country, everybody is required to decompose a phrase in it's underlying grammatical constituents in the eight grade. You just never draw it as a tree, but it's trivial to learn to draw it as a tree and read it from a tree.

4ad3• 12 years ago

This compiles to "He likes 2 green cats" in English.

The representation makes no assumption of SVO order, the grammatical category of numbers, and what cases and tenses are available.

    (OPRESENT ((OSUBJECT (OPRONOUN (3, 0))) ((OVERB, "like") OMUL (2, (OCAST (OATTR (OCOLOR "brown")), (ONOUN (OANIMAL, "cat")))))))

I'm sure someone smart will tell me what assumption I made that's invalid in some language I do not know, but this already is better, and simpler, and more correct than a lot of text-mapping based internalizations I've seen.

And I will be able to correct my S-expr to get rid of more assumptions, and I will just make it longer perhaps, but I won't need to write all the convoluted code prevalent in the article.

Also, with AST-based representation you can add whatever context you need. For example, were I to have used "pig" instead of "cat", this would have already worked fine, since we have (OANIMAL, "pig") and not (OPERSON, "pig"). This is trivial, but you can add whatever amount of context required.

schoen5• 12 years ago

Now I'm just wondering about the lexicon since the primitive tokens are things like "like". One potential problem is when two languages don't have words with enough semantic overlap to be comfortable using one as a translation for another. Another potential problem is when

An analogy to my other comment on AST translation: in English you "like doing something" but in German you "do something gladly" (and again in English you "like a person" but in German you can "have a person dear", akin to English "hold dear"). If we expect that the AST can produce a translation using the single verb "like", we may be in for trouble if the target language doesn't do that (although maybe code can be written that uses the AST and that's aware of this complexity as part of the realization of the translation).

Another example could come from the problem of describing states of being or perception, like "I'm cold", "I'm tired", "I'm hungry", "I'm thirsty", "I'm sick", etc. In English we really like using "to be" plus adjectives for such situations, but other languages have other preferred strategies. For example Latin has specialized verbs for the actions of (at least) being hungry, thirsty, or sick (like esurio, sitio, aegroto); in Romance languages people often "have" hunger or thirst (Spanish "tengo hambre", lit. 'I have hunger'; Portuguese "estou com fome", lit. 'I am with hunger'); in German it is cold "to" a person ("mir ist kalt", not "ich bin kalt" 'I am a cold person').

If you imagine having your AST start with Latin, you may have a challenging story about how you could get from "sitisne?" to "are you thirsty?" "¿tienes sed?" "está com sede?" and "hast du durst?" -- not to deny that it may be achievable with enough work.

mercurial• 12 years ago

Well, obviously, you're missing the mood.

Besides, I see two issues with that:

- While it may be more machine-readable, it's absolutely not human readable

- Something like (OPRONOUN (3, 0)) assumes that pronouns work like in English. Want to represent text in a language for which it is not the case with this system? No dice.

In short, I'd rather have an improved gettext (without hacks to detect if you're gettext'ing printf'd text...) than your proposed AST solution.

tinganho7• 12 years ago

This have been done. It's called "ICU's messageformat". Though instead of having a general purpose language, the markup language is specially tailored for localization. One project that support ICU's messageformat is http://l10ns.org

_obbx• 12 years ago

We version (localize) projects all the time, so I thought reasonable logic for seeing whether a string is empty is to check whether its trimmed length was greater than two, allowing for two stray characters that show up all the time in content. For example, someone who's not sure how to translate something would type "??" in a field. So, {if $slide.title|length>2} ... This worked for 4-5 languages.

Then our Chinese (Mandarin) division called and asked me to look into buggy behavior with their translations. Turns out a whole sentence got translated to ... two characters, and wasn't showing up.

ajuc3• 12 years ago

Another thing:

"Are you sure you want to quit?" in Slavic language will have gender of the user embedded. You need to know it to adress user correctly.

You can do stilted "To the person that uses this program - are you sure you want to quit?", but that's insane. So everybody just use male version.

userulluipeste2• 12 years ago

I don't know, in Russian either "ты" (singular/informal) or "вы" (plural/formal) works fine for both genders. Now, for your example, Google translated result is "Вы уверены, что хотите выйти?" which seems fine to me!

ajuc3• 12 years ago

"Wy" (plural you) don't work for single person in Polish, it sounds like "communist-speach" to us (almost like you called people "tovarishch") :) It was only used by soviet puppet politicians during communism (as carbon copy of Russian expression).

So it needs "Ty" (singular you), and with singular you "sure" translates to "pewien" for male recipients and "pewna" for female.

In Polish it should be "Jesteś pewny(or pewien)/pewna, że chcesz wyjść?"

So, it's Polish-specific, not Slavic-specific as I thought.

bobbles• 12 years ago

"Are you sure you want to Cancel?"

[OK] or [Cancel]

pornel• 12 years ago

The L20n library handles all these cases: https://news.ycombinator.com/item?id=8892273

dkbrk• 12 years ago

As far as I can tell, the best tool for localisation almost nobody is using is http://www.grammaticalframework.org/. Licensing is a mix of GPL, BSD and MIT pieces.

It's a high-level functional programming language with a dependent type system specialised for operating on language ASTs. It's resource library, to quote "covers the morphology and basic syntax of currently 29 languages: Afrikaans, Bulgarian, Catalan, Chinese, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hindi, Japanese, Italian, Latvian, Maltese, Nepali, Norwegian bokmål, Persian, Polish, Punjabi, Romanian, Russian, Sindhi, Spanish, Swedish, Thai, Urdu."

In essence, once it has the language-independent AST, it can produce output in all its supported languages with the correct tenses, genders, inflections, etc.

It also seems to have tools for assisted parsing, so you could have an english document and interactively parse it into the correct AST. In addition, the text can be parameterised semantically, so if you changed the gender of a person, that could propagate to all the correct locations and update the translations as required.

While it seems the upfront cost may be quite high in having to learn such a complex system, I think the benefits of having reproducible, high-quality outputs into n languages for free could make this highly advantageous in many applications.

canjobear• 12 years ago

I'm very skeptical that this would work outside of toy examples, though it depends on what is meant by language-independent AST.

For example, the best way to translate Spanish "X dió un golpe a Y" would be "X hit Y". But my naive idea of what the AST for the Spanish sentence would look like would be something like `(GIVE (X HIT Y)`, which when naively transduced to English would be the "X gave a hit to Y", which is either unidiomatic or means the wrong thing altogether. In order to avoid this problem, the AST would have to be a more abstract representation of the semantics. And coming up with a sufficiently expressive, tractable, and neutral representation of natural language semantics is an unsolved problem that people are still devoting their whole careers to.

I was briefly involved in a very early stage startup that was considering using systems like this for better machine translation. We ran into problems like the above, and also: ambiguity, and the fact that the hand-written grammars and semantic representation systems were just very brittle and incomplete.

rmc• 12 years ago

which when naively transduced to English would be the "X gave a hit to Y", which is either unidiomatic or means the wrong thing altogether

What's interesting, is that there are dialects of English (Hiberno-English spoken in Ireland) where "X gave Y a hit" would be a way to say "X hit Y". :)

nitrogen2• 12 years ago

"X dealt a blow to Y" seems like it could be a more grammatically similar translation.

schoen5• 12 years ago

That sounds like a great technical approach, although it can't necessarily remove the problem of idiomaticity that the article mentions (with the example of "I didn't search any directories"). Probably better examples are possible, in any case where the most idiomatic way to express something isn't a literal translation of that thing from other languages. Maybe like

English "I don't care"

Portuguese "tanto faz" (literally 'so much does')

German "[das] ist mir egal" (literally '[it] is equal for me')

taejo• 12 years ago

BTW, egal doesn't mean equal in German (though it's borrowed from French égal, which does). It means "irrelevant", "all the same", "unimportant".

eCa• 12 years ago

Also, if you 'localize' something using Google Translate[1], please let the user choose language somewhere in the app.

For example, the Hostelworld ios app[2] requires the user to change language for the entire device. As something of a language perfectionist it leaves the app virtually useless.

[1] Translating to English from other languages works fine for me.

[2] https://itunes.apple.com/us/app/hostelworld.com-hostels-budg...

dezgeg• 12 years ago

I wanted to do that on one Android app, but apparently it's impossible to implement with Android's built-in localization features.

quotequad• 12 years ago

This just shows the beauty of the Unix way of doing things. Simply:

12 10 4

works in all locales. :-)

ygra8• 12 years ago

Until you can display negative numbers¹. Or have native numerals². Or need to know what the numbers even mean. :-)

__________

¹ I tend to set my minus sign to U+2212 to catch errors in code where we just use ToString() instead of ToString(CultureInfo.InvariantCulture). Almost as much fun as putting a Unicode character into your user name that isn't representable in the current legacy codepage on Windows.

² ۱۲ ۱۰ ۴ probably won't work as input to the application trying to parse that line ;)

krzysz00• 12 years ago

How do you change your negative sign on a unixoid? The only reference to negative signs in locale(5) is under LC_MONETARY.

cj3• 12 years ago

I created Localize.js (https://localizejs.com), a localization SaaS.

Pluralization is a challenge, but we're able to solve this with some pretty simple HTML tags.

For example:

Localize.js identifies the <var> tag with the pluralize attribute, and pluralizes the phrase to any language (including languages like Arabic which can have 6 different plural forms).

julie1• 12 years ago

You don't solve the inflexion problem. You would need something where the lemmization tokenization of the sentence is easily accessible to a grammar engine. Something horrible like: <div ><lemm="personnal pronoun">I</lemm> <lemm=verb>have</lemm>< <case=accusative quant=3><lemm=noun quantifier /><lemm=....</div>

cj3• 12 years ago

Lemmatization isn't necessary in this case, as there's no need for a grammar engine.

Here's how we handle pluralization on our backend (all abstracted away from the user): For a language with 2 plural forms (ie. French), we create 2 different phrases. One singular and one plural. Or for languages like Arabic, we create 6 different phrases for each plural form. We then send each plural form of the phrase to a human translator, who translates each phrase independently.

This ensures proper plural forms for all variations, without the need for complicated grammar syntax.

mfenniak• 12 years ago

Huh. Localize.js sounds really cool. Fantastic idea.

Can you explain your example a little more? What would translators see in this case for Arabic; would they need to provide three translations? And if there were two variables, nine translations?

"Localization" also implies a lot more than just translation. Does Localize.js handle work like culture-specific number and date formatting? Different collation of records for different languages? How does it identify application-generated text versus user data (eg. on a blog, does it translate blog comments entered by readers, or just text like "Please enter a comment below")?

cj3• 12 years ago

What would translators see in this case for Arabic

The Arabic translator would have to provide 6 different translations, one translation for each plural form.

And if there were two variables, nine translations?

We currently only allow for pluralization based on one variable per phrase. We're hoping address this soon.

Does Localize.js handle work like culture-specific number and date formatting?

We usually recommend libraries like moment.js to handle date localization. They do a fantastic job at localizing dates.

How does it identify application-generated text versus user data

We provide a set of HTML markers that you can use to indicate to Localize.js that certain text should be translated. For example...

https://localizejs.com/docs/usage/variables

abbaselmas• 12 years ago

suddenly love my language (Turkish) although there are special characters. 1 elma (apple) 2 elma 3 elma . . 10 elma

dmytrish• 12 years ago

Kudos to the author of the article for his perseverance in decorating message to fit grammar. I'd go another way, just using more formal and dry format:

    Number of scanned directories: %g
    Number of found files: %g

That solves the problem with Slavic languages at least. Italian aversion to 0 may be mitigated with printing 'none', I guess. Please correct me if this form does not fit other languages.

randallsquared• 12 years ago

I agree: a huge chunk of this problem goes away if you stop trying to have the software communicate colloquially.

placebo• 12 years ago

Just scanned the comments to see if anyone would suggest that :) Being the lazy type, that's the first thought I had reading the article. Perhaps it's not appropriate for all target audiences and all target languages but many times you can find a much easier solution by going about it in a totally different way.

xorcist• 12 years ago

This is my preferred solution as well. As a bonus, it makes parsing the data so much easier, if you ever need it.

btbuildem• 12 years ago

So the solution is to design and implement your interface in a Slavic language (presumably most complex, as we found so far) and translate down to other languages with less demanding rules?

alxndr2• 12 years ago

Could be a perfect use for Lojban.

js2• 12 years ago

Previous discussion - https://news.ycombinator.com/item?id=2095334

barrystaes2• 12 years ago

I dont agree with the article. The author goes about manually implementing localisations, and eventually throwing out GnuGetText. But it DOES have excellent plural support, and a header in your PO file allows chinese to use "nplurals=1; plural=0;" for example: http://localization-guide.readthedocs.org/en/latest/l10n/plu...

Or use plurals as such: https://www.gnu.org/software/gettext/manual/html_node/Transl...

cschneid• 12 years ago

This is by far my favorite bit of documentation. I have to go look it up every time a manager or client starts asking for localization to justify my high estimates on how long it'll take.

smhg• 12 years ago

Now, I don't know the state of the gettext utilities in 1999, but the arguments don't seem to hold up anymore (as others commented).

It just surprises me how many times gettext is discarded as "not solving the problem" while it gets many things right.

It feels like the lack of knowledge about the complexity of i18n/l10n and about gettext are often the real issues.

demarq• 12 years ago

I don't know If I am simplifying to much but this seems something handlebars could solve in a minute! Every translation could just be a template.

alxndr2• 12 years ago

I wonder if Lojban[1] could serve as an unambiguous, largely-context-independent way to store representations of the central concepts, which can then be translated into natural human language.

[1] "a constructed, syntactically unambiguous human language based on predicate logic", http://en.wikipedia.org/wiki/Lojban

im3w1l• 12 years ago

I loved the irony of

[\xE9 is e-acute in Latin-1. Some pod renderers would scream if I used the actual character here. -- SB]

dcposch• 12 years ago

People here are talking about complicated, difficult solutions, like using a library to create text in multiple languages from an AST(!)

I think the moral of the post is simple: don't try to generate natural language. It won't sound natural. Just keep it basic.

"Directories scanned: %d" "Directories matched: %d" "Files matched: %d"

...will localize just fine

jpatte• 12 years ago

That might be ok for technical people who are used to this kind of "machine speak", but for end users it's often not acceptable. Consider the difference between:

    Purchased credits: %d.
    Remaining credits: %d.
    Buy more : 5 [link] / 10 [link] / 20 [link].

and

   You have purchased a total of %d credits and have used %d of them so far. You can buy [drowdownlist] additional credits by clicking here [link].

Which one do you think will have the highest conversion rate?

A Localization Horror Story: It Could Happen to You

Related Stories