Mimic – abusing Unicode to create tragedy

https://github.com/reinderien/mimic

by epsylon • 11 years ago

408 171 11 years ago

github.com

r7212• 11 years ago

In cases like those I use unicodelookup.com to list suspicious characters :)

bmn_• 11 years ago

That site does not work for me. I paste into the input field and it automatically turns into %F0%9F%90%88.

Compare with https://codepoints.net/ instead.

Edit: great, HN is broken too.

r7212• 11 years ago

Thanks, didn't know this one.

rbinv2• 11 years ago

I guess someone should develop an IDE/editor plugin that marks non-ASCII characters outside of string literals.

lazyjones3• 11 years ago

marks non-ASCII characters outside of string literals.

Many programming languages support non-ASCII variable name characters now.

rbinv2• 11 years ago

Good point, didn't consider that. Although any good IDE/editor should catch the use of "undeclared" variables and functions.

TorKlingberg2• 11 years ago

Many programming languages support non-ASCII variable name characters now.

Just because you can do something doesn't mean you should.

It is usually worth keeping variable names and such in English in enable international collaboration. Also non-ASCII source files can get mangled in transit.

halostatue• 11 years ago

Ruby now allows (some) Unicode glyphs as names (allowing for things like Δv).

    08:11:32 >> Δv = 3
    => 3
    08:11:39 >> p Δv
    3
    => 3

My solution when I have problems like this is to start building a negative regexp in vim:

  /[^-a-zA-Z0-9 \[\]]

I then add other symbols as I find them. I can usually find the illegal characters in about 30 seconds this way—and I can add the non-ASCII glyphs that I expect to be present to my regexp.

metasean• 11 years ago

JavaScript allows a wide range of Unicode characters - http://stackoverflow.com/a/9337047

archimedespi2• 11 years ago

Vim plugin: https://github.com/vim-utils/vim-troll-stopper

Svenstaro• 11 years ago

Wow, now that's just pure evil.

torgoguys• 11 years ago

Yes, seriously. This is why we can't have nice things.

cstross• 11 years ago

Also GREAT if you're trying to identify untaken phishing domain names to register for your next scam!

sheraz3• 11 years ago

wouldn't you end up with the 'xn--' ascii expansion in the url window?

sschueller• 11 years ago

Most modern browser will show you the unicode version.

treerock• 11 years ago

In Chrome, probably [1]. Other browsers don't seem to be as strict.

[1]https://www.chromium.org/developers/design-documents/idn-in-...

mikewilliams• 11 years ago

Hyperlink i.e http://.com How many people will double check the url bar and notice the url is actually http://www.xn--m3haa.com/? Not all of them.

:edit+seems like the three umbrella unicode symbols are not supported on hn, are they supported in e-mails?

Tepix• 11 years ago

A lot of unicode characters are blocked for domain names for this exact reason.

sheraz3• 11 years ago

There is a special place in hell for anyone doing this. I'm going to watch this repo and blacklist pull requests from anyone who forks it :-)

creshal2• 11 years ago

They share the place with coding blogs that use   instead of spaces for code snippets.

minikomi• 11 years ago

Or “real” quotes in code examples..

ciupicri• 11 years ago

I've had to use   a couple of times in order to get a proper indentation. I know there's <PRE> and <CODE>, but they aren't allowed on every blog, so I had to improvise.

ygra• 11 years ago

blacklist pull requests from anyone who forks it

Why, if I may ask? If they introduce compile errors in your project, those should be caught by the CI build and test run, shouldn't they? In any case, accepting a code change just by looking at the diff and without even trying it out sounds like not the best course of action to me anyway.

sheraz3• 11 years ago

Yes. A good CI will blow up any malicious pull requests provided you are using a compiled language. Or if interpreted, you need sufficient code coverage that your tests will blow up

adrianN• 11 years ago

On a Mac you (used to?) get a non-ascii space when you hit the space bar while holding Alt or something like that. Easy to fat-finger it in any case and looks the same in most text editors. It's a great source of fun for novice Mac-using programmers to find out why the compiler complains.

amadahy• 11 years ago

This is still happening as of today:

ps aux | grep foo

zsh: command not found: grep

It happens to me at least every other day.

mcculley• 11 years ago

The Commodore 64 (or some other machine from my childhood) would generate a non-ASCII space if you held down control (or shift maybe) when pressing it. To this day I'm careful about that. I didn't know it was still a possible problem.

kps5• 11 years ago

It's not terribly difficult to define custom keyboard layouts for OS X. Make a copy of your preferred layout and get Ukelele [sic] from SIL¹ to remove NBSP from Option-Space. (Or just hand-edit the XML changing " " to " ".)

¹ http://scripts.sil.org/ukelele

naggie• 11 years ago

It still does! You've just explained something that has been annoying me for about a year now. Thanks!

thousande2• 11 years ago

A good IDE would pick this up.

I had the habit of pressing the ALT key a bit ahead of time before an OR operator.

if (foo || bar)

mattlondon2• 11 years ago

I had something similar happen in the wild to me.

I work for a "major search engine" that does a lot of advertising & marketing stuff. To get the most out of it, we need customers to implement some javascript on their ecommerce sites.

As is often the case, javascript code that needs to get implemented on an ecommerce site often gets copy-pasted or emailed around a lot internally within a customer before it reaches the right person who can add it to the site's pages.

In this example somewhere along the way, a normal javascript snippet got all of the semi-colons changed from ; to ;.

In case you've not already spotted it, ; is not a ; but is actually "Greek Question Mark" (http://www.fileformat.info/info/unicode/char/037e/index.htm).

It was very confusing why Chrome was moaning about a semi-colon an illegal token. I had a genuine "Am I going mad? Seriously?" moment before I realised what was happening.

slowmotiony• 11 years ago

I would probably have to quit my job before I could figure out that problem. May I ask how did you spot it?

slig• 11 years ago

Something similar happen to me, but with that fancy quotes. I spotted by doing a "binary-search weird bug hunt". Cut half of the code off and see it it's still complaining, if it's, cut the other half, and so on.

sethammons• 11 years ago

I had a similar problem. Copied a code snippet. Ruby started complaining about an undefined function. After nearly going mad, and then looking at the source through a hex editor, you could see Unicode whitespace. I have yet to forgive ruby or Unicode whitespace. Or the chat utility from whence I copied.

austinjp2• 11 years ago

Weird bugs can respond well to brutally dumb debugging. Open file and search for ; perhaps? Or grep etc. When you can see semi-colons but your find tool can't, that's a worry.

But yeah getting to that point relies on pure inspiration, in my case.

falcolas3• 11 years ago

Personally, my vim status line has an indicator which shows the hex code for the rune currently under the cursor.

After fighting against word processor quotes, it's become second nature to double check it periodically.

mattlondon2• 11 years ago

It was a while ago now, but I remember copying the code out to a controlled isolated environment on my local machine, picking the first line and making sure it reproduced.

After that it was a matter of just going through the usual steps of working out why something isn't working. I think in this instance I actually ended up manually retyping the code, and when getting "identical" code that worked that was when the penny dropped and I realised something was not what it seemed. If you paste suspect characters into something like http://unicode-table.com/en/ it will tell you right away what it really is.

podgib• 11 years ago

I've been bitten by things like this so many times, that my first reaction on seeing an inexplicable syntax error is to delete and manually re-type the line.

TazeTSchnitzel4• 11 years ago

I was recently bitten by OS X's non-breaking space shortcut. I'd accidentally typed alt+space instead of space due to typing a following # (alt+3), and that made weechat, an app unaware of the NBSP, see the command '/join #channel' instead of '/join'.

gnud• 11 years ago

This can actually be used productively, to see how your app reacts to weird input :)

n-gauge• 11 years ago

Just chuck the code into an XML validator. Any character > 127 will be flagged as invalid.

pierrec3• 11 years ago

I can foresee a new phenomenon arising in stackoverflow-style sites and coding discussion forums:

"My simple piece of code looks perfect and should work without problems. Yet it won't compile! Help!"

Answer:

"Try running `./mimic --reverse` on your source."

po1nter• 11 years ago

That should be a comment.

pierrec3• 11 years ago

True, except in the rare situation where the question really features a code sample containing some evil homographs.

Edit: Oh hey, I actually found one that fits the bill:

http://stackoverflow.com/questions/14925894/trouble-with-arg...

probably_wrong• 11 years ago

I actually almost submitted something in that vein once. I'd type

  > ls | wc -l

and get

  > bash:  wc: command not found

As it turns out, I need Alt+1 to type a pipe character in my keyboard. If I'm not quick enough releasing the Alt key, I'll type Alt+Space instead of just Space, which inserts a Non-breaking space[1] in Mac. This character is not a space, and therefore it gave me a weird "command not found" error.

This lasted for months until I found out what the problem was - given that it was a combination of my keyboard settings and OS, finding the root of the error took quite some time. The hint? The "command not found" error had an extra space in front of the unknown command.

[1] https://en.wikipedia.org/wiki/Non-breaking_space

TazeTSchnitzel4• 11 years ago

This bit me as well, as I mentioned in a comment above. The solution I found best was to make OS X not produce an NBSP on alt+space.

wodenokoto3• 11 years ago

So, one might wonder why these homo-graphs have different code points. After all the French A and the English A share the same code point.

It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks.

Subtle differences in how Japanese and Chinese are written has led to differently written characters sharing the same code point. It's nice that you can easily look up most Japanese characters in a Chinese dictionary and see how they are used in China, but it has become frustratingly hard to get subtleties in their written form right. The Chinese version may have the line strike through another line, while the Japanese only has it touching.

I honestly don't know how to go about posting how to same code points have different written forms!

But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments.

ant6n2• 11 years ago

now that we gave up on ucs-2, we could re-encode those overloaded Japenese/Chinese characters as separate Japenese and Chinese characters on astral planes like the supplemental multilingual one.

rspeer2• 11 years ago

They're doing that. That's what a lot of plane 2 is.

lazyjones3• 11 years ago

It's really difficult to do the right thing here. If Greek question marks share code point with semi-colon, it obstructs search and replace for question marks.

Context is the key here. Greek text doesn't use the semicolon for other purposes and searching/replacing such single characters in source code is a terrible idea anyway (think comments, string literals...). So what is the prohibitive failure scenario here?

Indistinguishable (for humans) characters with different code points were a stupid idea, it's fine to abuse it in order to point out that fact.

wodenokoto3• 11 years ago

I don't know enough about Greek to deeply comment on it. some people do consider using the same code point for apostrophe and citations problematic (it's definitely annoying when doing word segmentation) we also have breaking and non-breaking spaces, as well as tabs. We luckily didn't follow typewriter conventions and collapsed several alphabetic characters!

A semicolon is not considered sentence final, whereas a question mark usually is. This makes it easier for software to auto capitalize. So at least that's possible a use case.

There's also the possibility that Greek requires the top dot to be square or circle, meaning it might in fact have subtle differences in print.

TazeTSchnitzel4• 11 years ago

They're not necessarily indistinguishable. Line-break rules, directionality and typographical conventions (height/width/alternate glyphs) may differ between apparent homographs. And using the same code point would make distinguishing between, say, Latin and Cyrillic searches difficult. How would Google tell if you mean CCCP or СССР?

Gigablah• 11 years ago

The Chinese version may have the line strike through another line, while the Japanese only has it touching.

Case in point: 每 (chinese) vs 毎 (japanese).

rootbear• 11 years ago

I had to zoom in to my screen just to see the difference between those.

I know that I am perhaps not well informed on this issue, but I also think that having multiple code points for what most people would perceive as the "same" character is not a good idea.

apricot• 11 years ago

The problem is compounded by the fact that in Chinese, there are "traditional" and "simplified" versions of many characters (with the Japanese character usually, but not always, being the same as the traditional Chinese one). And let's not even get into the different writing styles. The distinction between different characters and different ways of writing the same character is not always clear.

AnthonyMouse• 11 years ago

It's really difficult to do the right thing here.

There is a fairly good solution to this for things like code editors and URLs or search strings in browsers: If a string contains a non-ASCII code point that is a homograph for an ASCII character, swap the text and background colors.

wodenokoto3• 11 years ago

I meant the right approach to designating code points.

omaranto• 11 years ago

There is no "French A" and no "English A", I believe both languages call their alphabet the Roman (or Latin) alphabet. So those really are the same letter.

thaumasiotes4• 11 years ago

The Chinese could call their writing "the Roman alphabet" too (they don't), and label 闩 "A". So what?

bcoates• 11 years ago

The reason for a lot of homographs with ASCII is that there are old code pages that have entire non-US alphabets and punction sets in the 128-255 range, and regular ASCII in the 0-127 range.

It's a design goal of Unicode to support exact round-trip transformation with any code page ever in use in the real world, so they can't unify two characters that appear at different code points in a greek codepage without breaking things, even if they're always graphically indistinguishable in a font.

dragonwriter• 11 years ago

But it seems like it would be nice if code editors warned about text outside ascii. You usually only want that in strings and comments.

That's somewhat language dependent, though its true for most uses of most popular languages.

motti• 11 years ago

This sort of stuff can be the basis for many XSS attacks, see http://websec.github.io/unicode-security-guide/character-tra...

For instance, \u2329, \uFE64, \uFF1C and \u3008 can be best-fitted automatically to \u003C (the regular '<' mark in HTML)

lisivka• 11 years ago

It is also good tool to check is Unicode supported well: just convert all user visible messages and then check interface of the program for <?> or [].

ant6n2• 11 years ago

One could name variables and functions to later identify whether code was copied (e.g. to find out whether somebody copied some GPL code).

Kristine19752• 11 years ago

Note to self: Run mimic --reverse on GPL code I copy.

pmlnr• 11 years ago

This somewhat reminds me if this little entry on how "tolerant" JavaScript is...

https://mathiasbynens.be/notes/javascript-identifiers

sly010• 11 years ago

Ironically I have weird OCD where I always assume I made a typo, so I keep deleting and retyping code a few dozen character at a type, often in lines where I see nothing wrong. Over time this has just become something my hands do whenever my brain needs time to think about something else. So in a way I developed natural immunity to said unicode tricks ;)

jobigoud• 11 years ago

I think you're not alone. A common error I've noticed is when you make a typo somewhere (that compiles) and copy and paste it in a different place where you have the correctly named symbol. It's often hard to see the typo because the eye fly over the word. So you erase and type it manually.

SCHiM• 11 years ago

Hey I do the same!

Only with variables, and almost always when using array indexes that are not 'i' or 'x'.

Sometimes is annoys me, and I've noticed that it's worse when I use CamelCasing and not as bad when i_do_this.

cevaris• 11 years ago

Some people just want to see the world burn...

austinjp2• 11 years ago

I'm reminded how very useful I've found Text::Unidecode in the past.

http://search.cpan.org/~sburke/Text-Unidecode-1.27/lib/Text/...

avian• 11 years ago

Author of Python port of Unidecode here. I wrote a comment previously, pointing out that Unidecode does the reverse of Mimic. But then I actually checked the tables of characters that Mimic uses and deleted my comment.

Mimic chooses replacement characters solely based on their visual similarity with ASCII. Unidecode, while still doing character-by-character replacements without deeper analysis, tries to optimize the replacement tables for transliteration of natural languages.

For example, mimic will replace Latin capital H with Greek capital eta (U+0397), because they look similar. However, Unidecode will replace U+0397 with Latin capital E, because Latin E is typically used in place of Greek eta when transliterating Greek text to Latin.

Drdrdrq• 11 years ago

I have used the php port long ago when creating a simple website search engine... Great project!

AUmrysh• 11 years ago

I think the line about "Mimic substitutes common ASCII characters for obscure homographs" has it backward. Shouldn't it say Mimic substitutes obscure homographs for common ASCII characters?

pascalmemories• 11 years ago

Never occurred to me before, but here "substitutes" reads to me as being commutative. I read both as having the same meaning. (i.e. you end up with unicode homographs replacing your ascii) Just me?

dmd• 11 years ago

In that case, you won't mind if I substitute poison for your favorite tasty beverage.

pbhjpbhj• 11 years ago

Substitute works IMO the same way replace does[1].

Substitute poison for healthy food.

Substitute poison with healthy food.

The first means you take away healthy food and give poison. The later means you take away poison and give healthy food.

[1] Except that "replace X for Y" sounds weird, except in the common phrase "replace like for like" (and probably some others!).

mafro• 11 years ago

s/for/with

lstamour• 11 years ago

Technically, it does both ;-)

hollerith• 11 years ago

Not to my ear. To my ear, substituting X for Y is the same thing as replacing Y with X.

grabcocque• 11 years ago

YOU ARE A TERRIBLE PERSON AND I LIKE YOU

acdha2• 11 years ago

Mac users might appreciate the great UnicodeChecker:

http://earthlingsoft.net/UnicodeChecker/

It offers a convenient utility to diff arbitrary strings, which is also quite handy for e.g. detecting normalization discrepancies, and installs a service so you can highlight a character in any app and use “Display character information” to see what it actually is.

I have Python command-line version in my PATH which displays the character info for arbitrary input strings: https://github.com/acdha/unix_tools/blob/master/bin/unicode-...

acdha2• 11 years ago

Using the Taylor Swift example from https://news.ycombinator.com/item?id=10438363 in the comparison window looks like this:

https://www.dropbox.com/s/9j9h5rjt4gu22hb/Screenshot%202015-...

Each hex value shown can be clicked to open the Unicode character info for that codepoint

ehosca• 11 years ago

i smell a Notepad++ extension

b0ner_t0ner• 11 years ago

Taylor Swift? Never heard of her: https://www.google.com/search?q=Τаylοr+Ѕwіft

pierrec3• 11 years ago

Kind of surprised at how poorly Google handles this (I would have expected at least a correction suggestion)! Heck, it might open the door for an obscure blackhat/phishing technique...

DannoHung• 11 years ago

Maybe some sort of extortion scheme? Send an email to a small business person that isn't very technically savvy, say you have just erased all the search results for their business from Google, provide link, demand a Bitcoin to return the results.

Maybe a low hit rate, but if you could automate it, you could run the scam on a lot of places.

mfoy_2• 11 years ago

Not Google, but apparently one way some malware tries to hide edits to, say, the hosts file is to create a duplicate hosts file with the cryllic homoglyph for 'o' and then hide the real hosts file.

Presumably this would trick users who would go check "C:\windows\system32\drivers\etc\" but not show hidden files. Seems like a niche subset, but still a neat trick.

biot• 11 years ago

Or for shady SEO contracts. Not hard to guarantee #1 results if the contract spells out the exact (unexpected Unicode) phrase you're guaranteeing.

bpicolo• 11 years ago

Google operates in a bunch of languages, they can't necessarily just assume you're using codepoints on accident.

andrewflnr• 11 years ago

Forget google. Can you do this in domain names?

sah2ed• 11 years ago

I see what you did there :)

https://codepoints.net/search?q=https%3A%2F%2Fwww.google.com...

addandsubtract• 11 years ago

So apparently only the T has been replaced, but why is it showing different values for the "a"?

omgtehlion• 11 years ago

Slightly tangential:

In Russia there is a government procurement portal. Where gov organizations have to post their requests to enforce competetion and best prices.

The usual tactics [1] of corrupt officials was replacing cyrillic (russian) letters with respective latin homoglyphs so only affiliated companies can find and win this contract.

[1] http://www.bbc.com/russian/rolling_news/2013/04/130409_rn_st...

thomasfl• 11 years ago

Now that you have revealed the secret, Hacker News will be banned in russia forever.

florian-f• 11 years ago

    > var ﷺ = 1;
    < undefined

Procrastes2• 11 years ago

This seems like a useful tool for fuzztesting your dev ops person, or if you are the dev ops person, for fuzz testing development. Fuzz for all!

Kristine19752• 11 years ago

Piping the result through TTS creates weird results (on OS X):

  echo "hello world" | mimic --me-harder 100 | say

archimedespi2• 11 years ago

Can anybody provide an audio snippet for those of us who use Linux?

roryokane• 11 years ago

I don’t have an audio snippet, but I can transcribe what the voice says on different runs. It usually pronounces random letters individually, but sometimes pronounces syllables with letters missing:

“L-W-R-D”, “L-L-W-R-L”, “hell-erl”, “H-L-er-D””, “eor-D”, “H-L-L-erl”, “L-L-W-L-R-D”, “H-L-W-R-L”, “hell-W-R”

webXL• 11 years ago

Hmm... I wonder if this can be used in browser source maps.

foolfoolz• 11 years ago

is anyone aware of the reverse of this, a homoglyph normalization library? id love to be able to take strings that visually look the same and compare them against one master list, such as for spam detection

TazeTSchnitzel4• 11 years ago

In some languages which allow non-ASCII but aren't Unicode-aware (PHP, for instance), you can add significant, invisible zero-width spaces to identifiers.

tucif• 11 years ago

Spotify used to have a security problem with this kind of characters:

https://labs.spotify.com/2013/06/18/creative-usernames/

reinderien• 11 years ago

Mimic author here... sorry, humanity...

ChrisArgyle• 11 years ago

And now I know what I'm doing for April 1st next year.

cruise022• 11 years ago

Replace a semicolon (;) with a greek question mark (;) in your friend's C# code and watch them pull their hair out over the syntax error

I'm not sure how frustrating this would be. Wouldn't most people just delete the character immediately and type a new one?

patal2• 11 years ago

If faced with a linter error, I don't typically delete the marked stuff, write it anew and hope fingers crossed that the error would be gone. I would try to make sense of the message, how it applies, and what the error is. At some point though, I definitely would pull my hair over a greek question mark.

cruise022• 11 years ago

Only one character is going to be marked in this case, not a whole line or section of code. Deleting it and retyping it costs one second. I guess I've seen more than my fair share of encoding issues. I used to tutor at a university, so students were constantly coming in with code they'd copy/pasted out of their assignment (usually a Word doc) or from a web site.

CUViper2• 11 years ago

I don't know about C# compilers, but gcc gives me two errors, "stray ‘\315’ in program" and "stray ‘\276’ in program", which I suppose are the two utf-8 bytes. Rust says, "unknown start of token: \u{37e}". Either way, you get a pretty strong clue that there's a funny character present.

Animats• 11 years ago

There's a set of rules used on domain names to stop homoglyph abuse there.[1][2] Applying those rules to language identifiers would prevent this problem. It's also useful to apply those rules to login names for forum/social systems. The rules prevent mixed language identifiers, mixed left to right and right to left text, and similar annoyances.

[1] https://tools.ietf.org/html/rfc5893 [2] http://unicode.org/reports/tr46/

Induane• 11 years ago

These dang democrats done banned Ben Carson from google man!

https://www.google.com/search?q=Ben+Сarѕоn

bradbeattie• 11 years ago

Add the following to your ~/.vimrc to always highlight non-ascii characters:

    au BufWinEnter * let w:matchnonascii=matchadd('ErrorMsg', "[\x7f-\xff]", -1)

mrzool• 11 years ago

Some men just want to watch the world burn.

cammsaul• 11 years ago

The repo's README mentions a vim plugin to highlight Unicode homoglyphs. As an Emacs user, I did a quick M-x package-list-packages, thinking I'll find at least half a dozen equivalent Emacs packages.

To my dismay, there were none. So I spent the rest of my afternoon correcting this glaring deficiency. Fellow Emacs users, protect yourself from Unicode trolls and grab it here: https://github.com/camsaul/emacs-unicode-troll-stopper

perlancar2• 11 years ago

Made a perl port: https://metacpan.org/pod/mimic (currently 50% faster)

Mimic – abusing Unicode to create tragedy

Related Stories