Optical illusions that flummox computers

http://www.theverge.com/2017/4/12/15271874/ai-adversarial-images-fooling-attacks-artificial-intelligence

by pixelcort • 9 years ago

137 52 9 years ago

theverge.com

rhaps0dy• 9 years ago

"My, what a shiny red motorbike."

Totally what happens inside of a NN :D

candiodari• 9 years ago

TLDR: high-energy (sudden changes from one pixel to the next) overlays "fool" AIs. If you think about it you can see why : they change the statistical properties of an image a lot "without" disturbing it.

Less so, obviously, if you do something like downsample it or otherwise soften or ... with filters first. Nor do they fool neural networks with attention (they simply at some point decide it's not worth looking at and identify the picture by something else).

And the ridiculous example given does not work without being able to read the mind of the neural network (the misidentified panda).

Most neural network classification mistakes are "understandable" (e.g. look at the misidentified carousel). These are really 99.99% or more of the total mistakes make. Also that network probably needs more Indian elephants in it's training set (kids make stupid mistakes classifying animals they've never seen or only seen very few times as well [1]).

Given how a lot of animals look, I wonder if this doesn't work on "real" brains as well. I for one have trouble seeing zebras in pictures, and it's of course not for lack of contrast. Counting them or accurately judging distance is just out of the question. But many animals look way more colorful and contrast-rich than seems advisable, from chickens, of course peacocks, to ladybugs.

A number of optical illusions seem based on high contrast patterns being included in images. Especially if, like in the examples here, the high contrast patterns don't line up with the objects in the image (e.g. moving a vertical and horizontal slit filter over an image and you will not be able to see through it, however in any freeze frame you won't have that problem).

[1] https://www.youtube.com/watch?v=bnJ8UpvdTQY

mrkgnao• 9 years ago

More blatant "SEO walks into a headline" jokes, anyone?

adangert• 9 years ago

You know you probably could extend this to AI magic tricks, where you are able to make certain objects do things they are not supposed to, perhaps with an AI viewing video.

amelius2• 9 years ago

What happens if you train the system with these optical illusions in place, as well as the original images? Will it become harder to find new illusions? Or will illusions always be able to trick the system no matter how many illusions you trained with?

Remark: I noticed that even a watermark in the lower-left of the image (as you see on TV) can totally mess up DL prediction.

mholt7• 9 years ago

This is called "adversarial training" and is ineffective because of transferability. Also the space of adversarial examples is thought to be large (especially as dimensionality increases), making robust adversarial training intractable in practice. See https://arxiv.org/abs/1312.6199 and https://arxiv.org/abs/1510.05328

amelius2• 9 years ago

Interesting, what do you mean by transferability?

zeteo2• 9 years ago

Burying the lede:

The fact that the same fooling images can scramble the “minds” of AI systems developed independently by Google, Mobileye, or Facebook, reveals weaknesses that are apparently endemic to contemporary AI as a whole. [...] “All these networks are agreeing that these crazy and non-natural images are actually of the same type. That level of convergence is really surprising people.”

shmageggy• 9 years ago

This isn't surprising to me at all, since these systems are only "developed independently" in a very narrow sense. Each company may have it's own implementation (Tensorflow vs. Torch, etc) but they are implementations of nearly identical underlying models, namely deep convolutional neural networks. We shouldn't be surprised that they all agree since they are all the same under the hood, probably even down to the level of choice of hyperparameters (stride length, filter size, etc) since there is plenty of research on what kind of deep net works best on particular applications.

mholt7• 9 years ago

That would be convenient, but even when the machine learning models are implemented differently, as long as they accomplish the same task, the attacks often transfer from one to another. Papernot et al.'s latest research suggests that it is the space of the embeddings within a network, rather than the weights and nodes themselves, which define the effectiveness of adversarial examples. Models which accomplish the same tasks often have spaces that overlap, and that overlap multiplies by dimensions, if I read it right.

paulsutter4• 9 years ago

Everyone is using the imagenet database, so this outcome is less surprising than it may seem.

srtjstjsj• 9 years ago

Google and Facebook are also using their own huge datasets

spyckie2• 9 years ago

Actually a pretty interesting article.

The ease at which you can 'fool' machine learning right now adds an additional layer to practical machine learning in the wild - risks from malicious attacks.

Imagine someone putting up a lawn sign that tricks self driving cars into seeing something that isn't there and applying the wrong behavioral pattern because of it. Or even simpler, someone taping a sticker over the self driving car's cameras that cause erratic behavior. Can have really bad consequences and seems really simple to do.

AndrewOMartin• 9 years ago

I wrote a bit on this subject just last week, each paragraph mentions the problems of increasingly uncontrolled environments, finishing up with actively hostile environments like the examples you just mentioned.

http://www.aomartin.co.uk/what-robots-can-and-cant-do/

verroq• 9 years ago

Note that to generate perturbations you need access to the underlying model. The attack is just optimising towards a specific class with added cost for visual differences.

mholt7• 9 years ago

That's not entirely true:

If we transfer adversarial examples from one model to another model trained with one of these defenses, the attack often succeeds, even when a direct attack on the second model would fail [PMG16].

http://www.cleverhans.io/security/privacy/ml/2017/02/15/why-...

mholt7• 9 years ago

I'm finishing a survey paper that discusses research with adversarial examples plus about 9 or 10 other attacks on or weaknesses of neural networks (and machine learning models in general). Overall conclusion is: much like the early Internet, we're rapidly advancing towards machine learning tech that works but isn't secure. And 20 years later, we're still trying to make the Internet secure...

If neural networks are here to stay, maybe we should slow down their public deployment for a moment and understand them better first. It would be ideal to find fundamental structural/algorithmic changes that can harden them rather than relying on heuristics or other "wrappers" to make input/output safe to use in autonomous environments. The more that is "extra", the less those security features will be implemented. (We see this rampantly on the web today with HTTPS.)

tormeh• 9 years ago

Worse is better. Sure, making the internet safe from the start seems technically superior, but I guess that it would have delayed adoption so severely we wouldn't have come out ahead.

sathackr• 9 years ago

Exactly. Which is worse overall? No Internet? Or a hopelessly insecure internet?

Should we wait till something is perfect before we deploy? Should we hold back on self driving cars until the tech is completely foolproof? What about the number of accidents/deaths that could be prevented by even a flawed autonomous car?

Sometimes insecure and imperfect now is better than delaying a technology years until it can be 'perfected'.

mholt7• 9 years ago

Hard to say. Would the world have been less thirsty for the Internet ~5 years later? I'm just suggesting (not strongly in the paper, but here in the comments) that we carefully balance deployment with hardening.

joe_the_user• 9 years ago

The deployment of the Internet and the deployment of deep learning systems are quite different issues.

The Internet faced the problem of harnessing the power of computer networks required a critical mass and that none of the "ideal" conceptions could achieve this (especially since "ideals" usually didn't understand the full potential of the system). Here "worse is better" made sense.

Neural Nets and related systems are deployed by large companies with lots of data and used to provide guidance and decision making (where network effect isn't a big factor either way).

With "guidance" - product recommendations or optional translations, the dangers seem relatively small. With decision making systems, the dangers seems considerable and not totally thought through - you have everything from people falsely flagged by face-recognition software to neural networks that derive racist loan policies to the spoofing potential that management recommendation engines might offer.

Could you spoof a stock decision system by making a series of stock trades that has previous preceded a big move? I don't know but I assume someone is thinking of ways to use neural spoofing to get something from someone as we speak.

Asooka• 9 years ago

See: project Xanadu.

craigching2• 9 years ago

I may be restating some of what has been said elsewhere, but I hope that this simplifies things a bit.

What I believe the grandparent said was we should slow-roll neural networks where they have severe consequences, as in, life and death.

I agree, slow-rolling the internet would have potentially affected adoption, but the stakes are lower than, say, a self-driving car going off the rails. In those cases, we should probably thoroughly understand how a car can be affected by adversarial attempts at "breaking the algo" before adopting it very widely.

In other words, I wouldn't slow roll the internet because of fears of adversarial behavior, but I would in the face of self-driving cars.

I hope that further discussion could describe how internet security is as life-and-death as self-driving cars, or to the contrary :)

zitterbewegung• 9 years ago

I was at DEFCON and the most interesting thing I saw there was that not only can you confuse deep learning networks but your adversarial examples have a high chance of transferring to other ML techniques. See https://raw.githubusercontent.com/cchio/deep-pwning/master/d...

moultano• 9 years ago

The best explanation of why these transfer that I've seen is that the problem inherently arises from dot-products in high dimensions. Real data is somewhat random w.r.t. any particular set of activation coefficients, so the dot product looks like a sum of random variables and is subject to the central limit theorem. This keeps the real dot products small. If you are able to inject a teeny bit of the activation coefficients into the input, then as the dimensionality grows it gets easier and easier to overwhelm the real data with an epsilon sized change to the input.

peteretep• 9 years ago

    > maybe we should slow down

Please submit your proposal after the one for slowing down population growth, but before your one for slowing down general arms proliferation. Thanks.

itchyjunk• 9 years ago

In nature, the natural neural network called brain seems to weigh in the information from multiple sources instead of relying on just one. Not just in terms of number of sense but even for the same sense. Vision for example: it gets input from 2 eyes and if the data between the two is too inconstant, it sometimes drops it.

The machine learning systems will probably use similar tricks at some point. You might need to double the resource needed to process data twice or more, but you end up with a harder to fool systems. At least with the current adversarial attacks.

------------------

[0] https://www.scientificamerican.com/article/two-eyes-two-view...

[1] http://www.bbc.com/future/bespoke/story/20150130-how-your-ey...

lisper2• 9 years ago

Even the human brain often gets things catastrophically wrong. The problem is that it's only evident when it happens to other people, not so much when it happens to you. And if you're surrounded by people all of whom have the same bug in their wetware, then it gets really hard to tell.

craigching2• 9 years ago

This is a really interesting avenue to explore. How easy is it for humans to misunderstand something such that it leads to "catastrophic" failures. Here, a catastrophic failure could be something as simple as a huge fight between two people for a simple misunderstanding. If humans can have "breakdowns" based on simple misunderstandings, what are the cascading effects from a simple misclassification that can happen?

caseysoftware2• 9 years ago

It happens with sound/words too: https://en.wikipedia.org/wiki/Mondegreen

Humans are bad at recognition but still the best there are.

paulsutter4• 9 years ago

The real issue here is that imagenet doesn't have enough funny glasses. Same reason detection networks will think that gravel is broccoli, a network learns just enough to distinguish between the categories it's presented.

Improvements in datasets, transfer learning, and online learning will help. Unfortunately this underscores the issue that giants such as google have more pictures of funny glasses than anyone else...

Jweb_Guru• 9 years ago

No, that's not the only issue (it's not just that it doesn't have enough adversarial samples), as you would know if you'd read the papers involved.

paulsutter4• 9 years ago

The central issue is that networks are trained on a limited database of images (usually imagenet) and then frozen in time. Going beyond this involves evolving the nature of networks, and there's nothing the human brain can do that can't be done.

The design of networks is limited by the dataset, since researchers are just trying to hit the metric for the dataset. Better datasets will lead to better networks. Better data eventually means billions of cameras, for example.

zeteo2• 9 years ago

The real issue here is that imagenet doesn't have enough funny glasses.

But we don't actually know the real issue, do we? You can't easily debug a neural network. Sure, you can throw ten times more data at it and hope that fixes it. But the more data you throw in, the less you understand what's happening - which affects your ability to anticipate the next problem.

McKayDavis• 9 years ago

Clever 2 level pun with "Hi" hidden in the Magic Eye <-> Magic-AI.

JulianMorrison• 9 years ago

Uhhh, this has me a little worried that they are digging in territory that could contain basilisks. Humans are neural networks. Please do not generalise a means of hacking me? Thank you and much obliged.

anchpop• 9 years ago

Neural networks have very little in common with how our brains work

rochellle2• 9 years ago

But these aren't optical illusions!

dingo_bat• 9 years ago

They are for image classifiers.

rochellle2• 9 years ago

It's anomalous data for half-baked, fault-intolerant software.

mcguire4• 9 years ago

"To add to the difficulty, it’s not always clear why certain attacks work or fail. One explanation is that adversarial images take advantage of a feature found in many AI systems known as “decision boundaries.” These boundaries are the invisible rules that dictate how a system can tell the difference between, say, a lion and a leopard. A very simple AI program that spends all its time identifying just these two animals would eventually create a mental map. Think of it as an X-Y plane: in the top right it puts all the leopards it’s ever seen, and in the bottom left, the lions. The line dividing these two sectors — the border at which lion becomes leopard or leopard a lion — is known as the decision boundary."

I am not sure I buy this theory. Moving an image across a nearby boundary shouldn't result in the image producing a higher confidence value, should it?

I'm thinking of the panda/gibbon example in the article.

caseysoftware2• 9 years ago

Yes, that's exactly what it does.

By shifting it a few points across the boundary, it creates a higher confidence that X is actually Y instead of X.

The problem is that without studying the neural network and trying out different inputs and seeing the results, it's hard to figure out exactly what will shift things across the boundary. Even if you could get the same starting data set and attempt to train your own, you don't necessarily know which data was used for testing vs validation.

You'll likely come up with similar boundaries but not the same therefore you don't know the effectiveness of your adversarial approach until you actually try it against that particular system..

mcguire4• 9 years ago

But how do you go from 60% certainty that it is an A to 90% certainty that it is a B?

Optical illusions that flummox computers

Related Stories