Images that fool computer vision raise security concerns

http://news.cornell.edu/stories/2015/03/images-fool-computer-vision-raise-security-concerns

by lm60 • 11 years ago

365 210 11 years ago

news.cornell.edu

jahnu• 11 years ago

But computers don’t process images the way humans do, Yosinski said.

This means that come the singularity AIs will have to use AI specific CAPTCHAs in order to distinguish between humans (aided by dumb computers) and other AIs.

pavel_lishin5• 11 years ago

Which of the following would you most prefer? A: a puppy, B: a pretty flower from your sweetie, or C: a large properly formatted data file?

msandford3• 11 years ago

Welp, it's official. I'm a computer.

compbio6• 11 years ago

If we have a singularity AI then we don't need CAPTCHAs: For all intents and purposes an AI is equal or superior to humans, there is no distinction to be made. The Turing Test is a form of CAPTCHA.

Also, one could make CAPTCHA's incredibly hard. Humans and dumb computers won't be able to solve it, and singular AI's get a pass. So give access to anyone who isn't able to solve the CAPTCHA and redirect the singularity AIs to google.com :).

Finally, the singular AI could create a CAPTCHA which separates humans from AI. The problem of separating humans from AI will quickly become too difficult for humans to solve.

jgrowl• 11 years ago

It makes me chuckle thinking about machine intelligence being able to solve a captcha when humans can't.

I get a mental image of a future where computers try to keep us pesky humans out of their websites.

msandford3• 11 years ago

This is some excellent research!

It reminds me of the CV dazzle anti-facial-recognition makeup that made the rounds a while ago: http://www.theatlantic.com/features/archive/2014/07/makeup/3...

This definitely reinforces my belief that having humans in the loop is not only desirable but necessary. For the majority of human history minus a few years you could only be accused of a crime by another human being. I'd like to see the trend of automated "enforcement" reversed and codify into law that you MUST be accused by a human being.

If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around.

toomuchtodo9• 11 years ago

If everyone is breaking so many laws that the police and courts can't keep up it doesn't mean that humanity is broken. It means that the law has gotten so far out of sync with humanity that the law is broken. People make the laws, not the other way around.

The world would be a much better place if more people realized this.

mikeash10• 11 years ago

This always frustrates me when discussions of plea bargaining and the right to trial come up, and the argument is given that plea bargaining is a necessity because the courts would be horribly overloaded if every case went to trial.

If the system doesn't have the resources to give every accused criminal a fair trial, then either you're making too many criminals, the system doesn't have enough resources, or both. Bypassing trials is just a way to cover your ears and shout "la la la" to ignore the problem.

nickysielicki• 11 years ago

The fact that can and does occur means the ideas of laws are broken.

But that's just the anarchist in me speaking.

joe_the_user5• 11 years ago

Well, I think a large percentage of people do realize this.

They just forget it as soon as the news media present whatever is the current horror story.

yutah• 11 years ago

I think it is this way for 2 reasons... one, for-profit prisons and two, the ability for the government to put anyone away for a while

yuhong• 11 years ago

Another thing I don't like is legislators making laws without thinking about enforcement, which is why I don't like anti-discrimination laws.

platz3• 11 years ago

Someone talking about the kinect said that most all facial recognition algorithms require the face to be level horizontally; if the face is tilted at an angle then it doesn't work. not sure if true but interesting observation.

drpgq• 11 years ago

That's really not that interesting. I work as a research scientist at a face recognition company. If you want, you can detect faces at any angle, but sometimes for speed you will only check upright faces.

kevin_thibedeau2• 11 years ago

It's not excellent research. All they did is create a naive NN without any feature recognition and noted that it has a high rate of false positives when presented with random imagery. Duh.

woah• 11 years ago

Really? From the article:

"They tested with two widely used DNN systems that have been trained on massive image databases."

Is this wrong?

eternalban• 11 years ago

It's not just the recognition front end that is problematic. In context of the issue of mis-recognition, Aphyr's tear down of various backends [1] -- which for all we know are in use by the lettered agencies -- gives me the willies.

[1] https://aphyr.com/tags/Distributed-Systems

lovemenot• 11 years ago

> It reminds me of the CV dazzle anti-facial-recognition makeup

I had a similar thought. An app which takes the image of a target - Julian Assange say - and overlays it onto your facial image as thousands of virtual stickers. Gradually the evolutionary algorithm morphs this overlay towards your real facial image; its fitness function being fewer, more unobtrusive stickers. All the while maintaining compatibility with target facial recognition.

One day, J.A. escapes from his embassy lair. Thousands of people simultaneously appear on the streets wearing physical stickers on their faces, addressing the London panopticon: "I am SpartAssange".

JoeAltmaier15• 11 years ago

Humanity isn't some static thing. Cultures become dysfunctional just like familes. Sometimes they have to be changed or they will collapse. Its disingenuous to suggest 'everybody would be happy if only the cops would quit hassling me'. That's sophomoric.

pavel_lishin5• 11 years ago

Peter Watts mentioned this potential problem in his Rifters series; one explicit example was a neural net that ran a train and was trained via a series of inputs, one of which was a clock in a train station; one day, the clock broke, and the neural net took some action that ended up killing all the passengers. (I forget the details.)

Which is not to say that we should all fear computers more than humans as a consequence; we do inexplicable things, too.

z5h• 11 years ago

This raises an interesting question. Do we want computers to see "correctly", or to see how we see?

Would a preferred computer vision system experience the Checker shadow illusion? http://en.wikipedia.org/wiki/Checker_shadow_illusion

If yes, computer vision will be as fallible as ours. If no, then there will always be examples, like the ones presented, where computers will see something different than humans.

higherpurpose2• 11 years ago

Does it matter if we want them to do it or not? NSA and China will build them to do that anyway.

maxerickson2• 11 years ago

A computer vision system can have multiple ways of processing an image. So at the limit, it could interpret a scene in terms of what a human sees and also have a separate, better understanding of the scene.

pavel_lishin5• 11 years ago

That would be interesting; it could flag inputs that are ambiguous to humans but not machines (or vice versa, or when there's a discrepancy at all) since it could suggest that something shady is happening.

JoeAltmaier15• 11 years ago

The OP shows that computers DO NOT have a better understanding. Its evident they have no understanding at all; they are simply doing math on pixels and latching on to coincidental patters of color or shading.

People recognize things by building a 3D model in their head, then comparing that to billions of experiential models, finding a match and then using cognition to test that match. "Is that a bird? No, its just a pattern of dog dropping smeared on a bench. Ha ha!"

Balgair• 11 years ago

You can always fool any system into a paradox. Very grossly speaking, this comes out of Godel's incompleteness theorem; that with any set of laws you can always get P=~P out of the set. How this paradox looks, acts, or feels is interesting and possibly artistic, as the OP shows. If anything, I think there is a bit of beauty, art, and cleverness in that.

lars2• 11 years ago

That isn't quite right. With a sufficiently powerful formal system, you're forced to either have inconsistency or incompleteness - you're describing a system that is inconsistent. It's usually much better to have consistency and to sacrifice completeness. Then you'll have Ps that are true but unprovable, but at least you won't have P=~P which makes the system rather useless.

Kronopath• 11 years ago

The question is, what do you mean by "correctly"?

When we look at the Checker Shadow Illusion, our brains are automatically "parsing" that image into a 3D scene and compensating for the lighting and shadowing. The reason you see square B as being lighter than A is because, if you could reach into the image and remove the cylinder so it's not casting a shadow anymore, square B would be lighter than A. Our brain doesn't think about color in terms of absolute hex values. Instead it tries to compensate for the lighting and positioning of elements, assuming that they're similar to what we would see in real life—a challenge which, by the way, computer vision systems have always struggled with.

That sounds pretty "correct" to me.

snowwrestler• 11 years ago

Are all humans susceptible to the checker shadow illusion? Or just those that have been trained to interpret flat arrangements of color as accurate representations of 3D scenes and objects? If you showed the checker illusion to someone who had never seen a photograph or representative painting, would they see different colors or the same color?

I don't know the answer. I do find it fascinating that something as simple as perspective (which we take for granted in a graphic like the checkers illusion) is a fairly recent technique, invented by Renaissance artists.

Gifford2• 11 years ago

With good lighting control, you could set up he checkerboard illusion in the real world. That's part of he point.

Houshalter7• 11 years ago

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572

Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily.

These images are highly optimized for this effect and unlikely to occur by random chance. Adding random noise to images doesn't seem to cause it, because for every pixel changed in the right direction, another is changed in the wrong direction.

The researchers found a quick method of generating these images, and found that training on them improved the net a lot. Not just on the adversarial examples.

mlmonkey• 11 years ago

Basically neural networks and many other machine learning methods are highly linear

No they're not! You introduce non-linearities like the sigmoid or tanh to make them highly non-linear.

neuralk• 11 years ago

I was thinking the same thing until I scanned through the paper linked above. While neural networks are indeed non-linear, some NNs can still exhibit what amounts to linearity and suffer from adversarial linear perturbations. Example of linearity in NNs that the authors are considering from the paper:

The linear view of adversarial examples suggests a fast way of generating them. We hypothesize that neural networks are too linear to resist linear adversarial perturbation. LSTMs (Hochreiter & Schmidhuber, 1997), ReLUs (Jarrett et al., 2009; Glorot et al., 2011), and maxout networks (Goodfellow et al., 2013c) are all intentionally designed to behave in very linear ways, so that they are easier to optimize. More nonlinear models such as sigmoid networks are carefully tuned to spend most of their time in the non-saturating, more linear regime for the same reason. This linear behavior suggests that cheap, analytical perturbations of a linear model should also damage neural networks.

Houshalter7• 11 years ago

Most image recognition neural networks use Relu activations which is just a linear output unless the input is below zero. Even sigmoids/tanh are most linear in the middle region, and weight penalties are used to keep the weights small so they stay in that region.

But it doesn't really matter what activation function you use. The paper argues that its the linear layers between the nonlinearities that are the problem.

bendyBus• 11 years ago

Everything non-linear is linear to a first-order Taylor expansion. Hence why you can evolve small perturbations, the same thing is used for explicit numerical integration of non-linear equations.

crpatino• 11 years ago

Nobody is claiming that this changes are more than astronomically possible to be produced by random processes, the title in such case should have read "... raises robustness concerns" instead of security concerns.

In every security system, it is assumed that if there is an attack surface, sooner or later an intelligent adversary will come and exploit it. And there is a long precedent saying that if you see the word "linear" anywhere in the attack surface description, the adversary is bond to come sooner rather than later.

jeffclune13• 11 years ago

You say that these images are highly optimized to produce this effect and would not occur by chance, but have you looked at the images in the "fooling" paper?

http://www.evolvingai.org/fooling

Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.

Nvn• 11 years ago

Some of them are very simple, and DO occur a lot in the world. For example, the alternating yellow and black line pattern would be encountered by a driverless car, and it would think it is seeing a school bus.

While the image shows a yellow and black line pattern to us, are you sure this is also what the CNN "sees"? Couldn't this image just be the same as the adversarial images, i.e. it responds to many small input values rather than the overall pattern?

If it's possible to make the CNN predict an ostrich for an image of a car, then the same can be done of an image of an alternating yellow and black line pattern, no?

compbio6• 11 years ago

It is good to know that they need access to a lot of predictions from a net, before they can create an image that will "fool" the net, but look alien to humans. Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets (which is basically industry-standard practice to increase accuracy).

Furthermore, I am seeing the security concerns, but I figure this is far from a practical attack. Deep Learning Classifiers do not act as gatekeepers: You have not much to gain from a single faulty classification. You won't be granted access to secret information if you happen to look like the CEO.

userbinator2• 11 years ago

Furthermore, I am seeing the security concerns, but I figure this is far from a practical attack

Perhaps it's a sign of the times that almost every discovery that could possibly be related to security in some way, does. I have a feeling that if this was a decade or two ago, the sentiment would be very different. ("Can you figure out what a computer thinks these images are?")

Also, the image labeled "baseball" immediately reminded me of a baseball...

pavel_lishin5• 11 years ago

Since the images are crafted for a single net, a majority vote should not be fooled by these images. I suspect this effect rapidly goes away when adding more nets

Sure, but this assumes that whatever neural net system you're relying on was bought by someone who is more security conscious than they are cheap.

jerf• 11 years ago

"Secondly, this doesn't account for ensembling: "fool me once, shame on you. Fool me twice...". Since the images are crafted for a single net, a majority vote should not be fooled by these images."

Trivially "solved" by treating the ensemble as a single object, then constructing a counterexample. My intuition suggests that while the resulting "fooled you" image may very slowly converge on something human recoginizable, it won't do so at a computationally-useful rate.

compbio6• 11 years ago

Now I have to try this out. My intuition tells me it becomes increasingly hard to create a fooling image, which looks alien, and is able to fool all the nets in the ensemble, even though they have different settings and params. I think they can only fool one net at a time, and have to get very lucky to be able to evolve the image for the other nets, while keeping the same classification. You can't "train" these images on all nets at once, by simply treating the ensemble output as a single net.

If your intuition is right though, then the ensemble may be able to counter with a random selection of nets for its vote: You'd need to evolve images for every possible combination and/or account for nets added in the future.

scribu• 11 years ago

You won't be granted access to secret information if you happen to look like the CEO.

Tell that to this startup: http://onevisage.com/ovi-technology-page/

jeffclune13• 11 years ago

Surprisingly, ensembles do not really help. We have tried this and it does not work (the final paper for CVPR 2015 will show these results). Also see the work of Szegedy et al. and Goodfellow et all, which also show that ensembles do not really help.

jeffclune13• 11 years ago

Also, you don't necessarily need a lot of predictions from a net to fool it, because (another surprising result) images that fool one net tend to fool others! So I can create fooling images on my in-house net and then take them and fool your net-used-for-some-important-application without getting any feedback from your net. That's very surprising, and does raise serious security concerns.

adrusi• 11 years ago

The pattern-based illusions are actually quite interesting, almost artistic. Half of them are recognizable to humans, the other half at least make sense when identified. I wonder if we can automate the production of postmodern art :)

LoSboccacc• 11 years ago

Deep Neo Neo Cubism?

jeffclune13• 11 years ago

In fact, we submitted them to an art competition and they were accepted and put up in a museum!

http://www.evolvingai.org/share/20150213_184537.jpg

FLGMwt• 11 years ago

Who decided to name it a "lesser panda"? And of the alternative names for a red panda, firefox, or red cat-bear why would someone choose "lesser panda"?

bediger4000• 11 years ago

That is a question for the ages. Who decided on "cool ranch flavor" for Doritos? And what's a "cool ranch"? Or "blue raspberry", a common flavor for candy these day, or "TV Spokesmodel" (https://www.google.com/?gws_rd=ssl#q=TV+spokesmodel). There's lots of weird-beard names that get applied to stuff, and nobody has a say in those names.

higherpurpose2• 11 years ago

How effective will this be against Intel's "RealSense 3D cameras"? Is it, as I've already assumed, just a matter of time before that technology can be fooled too?

valine2• 11 years ago

Here's a link to the Yosinski's paper.

http://yosinski.com/media/papers/Nguyen__2014__arXiv__Deep_N...

bwross• 11 years ago

If you look really closely at the noisy images, you can see little blotches that vaguely resemble the things the computer recognized them as.

Coincoin• 11 years ago

Yeah, they say the algorithm erroneously sees an Amarillo. But wait, I actually see an Amarillo and I actually see a centipede.

uncoder0• 11 years ago

Not directly related but, I was at a security related convention and overheard some people talking about an image that when occupying <3/4 of a frame will crash any digital camera. (Phone, DSLR, IP Camera) Does anyone know anymore information about this image and effect? I imagine it's a bug in some low level firmware of a common IC for digital photography DSP but, I'm very unfamiliar with digital cameras. It also could have been complete bunk because I've not heard of it since and it wasn't being showcased at the convention.

jkot• 11 years ago

Flying RPG?

Now seriously, modern cameras have face recognition and ton of other features. It should be possible to cause crash.

Houshalter7• 11 years ago

Could be related to this? http://en.wikipedia.org/wiki/EURion_constellation

TazeTSchnitzel• 11 years ago

It's most likely not the constellation itself, but the other mechanism:

https://en.wikipedia.org/wiki/EURion_constellation#Other_ban...

In particular, have a look at this:

http://www.cl.cam.ac.uk/~sjm217/projects/currency/

geographomics• 11 years ago

Not what you're looking for, but your comment reminded me of this: https://en.wikipedia.org/wiki/BLIT_%28short_story%29

userbinator2• 11 years ago

If there is such a thing I'm more inclined to believe that it's an analogue effect, rather than a bug in software/firmware. it's similar to the effect that causes some monitors to emit audible sound when displaying certain images: https://news.ycombinator.com/item?id=8862689

Digital camera sensors output a stream of bits that depend on the intensity of the light reaching the pixels. For normal images, there is (relatively speaking) not so much contrast, so the signal has few high-frequency or repetitive components to it. However, if you point the sensor at an image that effectively causes each pixel to be alternating from full dark to full bright, the signal becomes far more regular and the high-frequency components increase significantly. A possible problem is that, since the bulk of the power draw occurs when a bit transitions from 0-1/1-0, this repetitive and high-frequency signal causes more stress on the power supply circuitry (look up "voltage regulator oscillation"), and if it causes voltages to go out of tolerance, can crash the system. I suppose an image that produced a stream of 010101010... for each pixel's value could also be an example of this. Other resonant effects may also play a role in this; if the oscillation frequency happens to synchronise with something else, physical damage is a possibility if the components are pushed beyond absolute maximum ratings. It's an extreme edge case, not normally encountered in use.

The reason why I think this could be plausible is that, although I've never tried/experienced this with a digital camera, I had an old analogue video camera that would work fine in all circumstances except when pointed at a monitor displaying its image, upon which it would emit a loud high-pitched whine and then shut itself off. I discovered that one of the power supply rails would go into oscillation when the camera saw itslf, and this was enough to shutdown the system. Adding some extra supply decoupling was enough to stop this from happening, but apparently this problem has been known for a while:

http://en.wikipedia.org/wiki/Video_feedback

zk000062• 11 years ago

The whole idea is to tear down algorithm, simulate unnatural image that triggers the required responses and pass it to classifier. There is no direct link to security risks here.

zk000062• 11 years ago

If you ever work with classifier training, these results are not surprising. You can take all false positives generate by classifier, average them and you will come up with an image that resembles the object to be recognized.

randcraw• 11 years ago

This is not so different from recognizing images in their fourier frequency domain. The frequency features and their origins in the spatial domain can be made very unintuitive.

But I'm not clear how important this phenomenon really is to the practice of CV, since 1) 'spoofed' images are highly specific to each DNN being used, and 2) a trivial reality check of the image can always 'out' examples like these.

acadien• 11 years ago

Your 2nd point is critical, you can filter these images easily before even running them through the DNN. However researchers are also interested in why it is possible to spoof NN's in general. The typical response of 'overfitting' is being questioned.

Also the question is raised as to whether or not new methods of spoofing are possible that aren't so easily detectable.

commentereleven• 11 years ago

This reminds me of this tool-assisted speedrun: https://www.youtube.com/watch?v=GOfcvPf-22k

The image recognition in Brain Age is obviously much simpler, but it's still basically the same idea.

woodchuck64• 11 years ago

"“We realized that the neural nets did not encode knowledge necessary to produce an image of a fire truck, only the knowledge necessary to tell fire trucks apart from other classes,” [Yosinski] explained."

This seems markedly different from biological neural networks. Is the difference one of network structure/algorithm or rather the fact that biological neural networks (in human image processing) actually have time and space to learn a lot about each individual image class?

compbio6• 11 years ago

Deep nets are only loosely inspired by neurobiology. That's why LeCun calls them "convolutional nets" and not "convolutional neural nets" and prefers "nodes" over "neurons".

It is, however, possible to have a deep net produce 3D models/images: https://www.youtube.com/watch?v=QCSW4isBDL0 "Learning to Generate Chairs with Convolutional Neural Networks".

I also suspect a different part of cognition is used when humans are asked to recreate a "fire truck" than when humans are asked to classify a "fire truck" from a "car". The former seems closer to using memory ("what did the last five fire trucks I saw look like?"). A fairly recent addition to deep nets is making use of memory: http://arxiv.org/pdf/1410.5401.pdf "Neural Turing Machines". So the difference may quickly become less significant.

humanfromearth• 11 years ago

They can only fool the DNN because they know their weights. In theory, you could do the same for people's brains.

As long as you don't publish the specs of your net you should be fine I guess.

dsr_• 11 years ago

If you have an oracle, you don't need to know the net specs.

An oracle, here, would be any version of the system that you can query against repeatedly without suffering too much of a penalty.

algorias• 11 years ago

That's a good point! But your proposed solution is security by obscurity. What if a rogue employee trained a backdoor into your DNN? What if the NSA asked vendors for a master key?

The purpose of this research is to work as a proof of concept. Sure, this iteration needs to cheat slightly to achieve its results. However, looking at the images that aren't just noise, it seems to be possible to construct less specialized images that fool many nets.

Houshalter7• 11 years ago

No, the images generated worked on even different neural networks trained on different data.

jeffclune13• 11 years ago

The paper does not use the weights of the network to produce (nearly all of) these images (see the responses to Karpathy above).

ars2• 11 years ago

In theory, you could do the same for people's brains.

No because the instant you used an image that was close, but wrong the human brain would retrain.

I wonder in the neural net can do the same - try this experiment on an active neural net and let it train itself. (i.e. don't tell it it's being faked, let it figure it out then correct for it).

valine2• 11 years ago

Does anyone have higher resolution versions of the images used in the article?

jeffclune13• 11 years ago

Here they are: http://www.evolvingai.org/fooling

There is also a video summary of the paper there.

kingkawn3• 11 years ago

Maybe this is the algorithm making art

jeffclune13• 11 years ago

See above. These images were accepted to a juried art competition and posted in a museum:

http://www.evolvingai.org/share/20150213_184537.jpg

kingkawn3• 11 years ago

The awards only acknowledge, not express, what it has achieved.

The algorithm would be useful if it could generate/identify images that feel as emotionally transformative as good art.

rndn• 11 years ago

I quite like this thought, though art is about combining known patterns in a novel way. Here they create unknown patters to evoke associations with known but unrelated patterns. It's kind of reverse-art.

kingkawn3• 11 years ago

I think art is about the relationship of the viewer to the work, not about the explanation for the work's creation. The viewer is the place where it all goes down.

lawlessone• 11 years ago

Machine pareidolia?

On a serious not couldn't we keep training the same DNN's using these white noise images as negative examples?

Houshalter7• 11 years ago

This paper tried that with positive results: http://arxiv.org/abs/1412.6572

raynassar• 11 years ago

That point was covered in the article.

In a further step, the researchers tried ?retraining? the DNN by showing it fooling images and labeling them as such. This produced some improvement, but the researchers said that even these new, retrained networks often could be fooled.

c-slice• 11 years ago

This isn't very groundbreaking, they're simply discovering the inherent weaknesses in neural net learning. We've know for years that neural nets have inherent gaps in their training and this is just taking advantage of that. This isn't really an issue in a real life scenario as you wouldn't be able to determine without thousands of recursions which images produce faulty results.

jpeterson• 11 years ago

Well, obviously they should've checked with you before publishing their paper.

deeviant• 11 years ago

I know many here know this already but I'll say it anyway since I ran into people with a misconception related to this: the particular misidentifications are specific to the algorithm and training set used, it's not like all computers recognize those blocks as cheetah or what not.

We use ML-based computer vision at my work, so I have a bit of experience here. I think the biggest practice take away from observation the ML can give some wonky results is that ML system can be a real PITA to debug.

karpathy2• 11 years ago

This work has led to some unfortunate misconceptions.

In particular, this weakness has nothing to do with Computer Vision and also nothing to do with deep learning. They only break ConvNets on images because images are fun to look at and ConvNets are state of the art. But at its core, the weakness is related to use of linear functions. In fact, you can break a simple linear classifier (e.g. Softmax Classifier or Logistic Regression) in just the same way. And you could similarly break speech recognition systems, etc. I covered this in CS231n in "Visualizing/Understanding ConvNets" lecture, slides around #50 (http://vision.stanford.edu/teaching/cs231n/slides/lecture8.p...).

The way I like to think about this is that for any input (e.g. an image), imagine there are billion tiny noise patterns you could add to the input. The vast majority in hundreds of billions are harmless and don't change the classifications, but given the weights of the network, backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern out of all billions.

All that being said, this is a concern and people are working on fixing it.

yosinski5• 11 years ago

This work has led to some unfortunate misconceptions.

Agreed; the weaknesses reported should definitely not be taken to affect only convnets or only deep learning. Ian's "Explaining and Harnessing Adversarial Examples" paper (linked by @Houshalter) should be required reading :).

backpropagation allows us to efficiently compute (with dynamic programming, basically) exactly the single most damaging noise pattern out of all billions.

True. By using backprop, one can easily compute exact patterns of pixelwise noise to add to an image to produce arbitrary desired output changes. However, it's an important detail that that most of the images in the paper (all except the last section) were produced without knowledge of the weights of the network or by using backpropagation at all. This means a would-be-adversary need not have access to the complete model, only a method of running many examples through the network and checking the outputs.

...there are billion tiny noise patterns you could add to the input.

Perhaps because the CPPN fooling images were created in a different way (without using backprop), they seem to fool networks in a more robust way than one might think. Far from being a brittle addition of a very precise, pixelwise noise pattern, many fooling images are robust enough that their classification holds up even under rather severe distortions, such as using a cell phone camera to take a photo of the pdf displayed on a monitor and then running it through an AlexNet trained with a different random seed (photo cred: Dileep George):

http://s.yosinski.com/jetpac_digitalclock.jpg http://s.yosinski.com/jetpac_greensnake.jpg http://s.yosinski.com/jetpac_stethoscope.jpg

I thought this was surprising the first time I saw it.

JoeAltmaier15• 11 years ago

Wait - they didn't use knowledge of the neural network internal state to calculate these patterns? Does that mean they could create equivalent images for human beings? What would those look like!

jeffclune13• 11 years ago

Ha. Jason and I were typing at the same time. :-)

karpathy2• 11 years ago

The two classes that you describe: 1. Adversary has the weights and architecture and 2. Adversary can only do forward pass and observe output, are equivalent when all you're trying to do is compute the gradient on the data. In case 1 I use backprop, in case 2 I can compute the gradient numerically, it just takes a bit longer. Your stochastic search speeds this up.

Likewise, I was not very surprised that you can produce fooling images, but it is surprising and concerning that they generalize across models. It seems that there are entire, huge fooling subspaces of the input space, not just fooling images as points. And that these subspaces overlap a lot from one net to another, likely since they share similar training data (?) unclear. Anyway, really cool work :)

jeffclune13• 11 years ago

Hi Andrej,

I agree with most of what you say, but note that nearly all of the images in the paper were generated without the gradient. I.e. all the images produced by evolution did not use the gradient, only the output of the network regarding its prediction confidence. There are some images that use the gradient, but only to show a 3rd class of "fooling images".

PS. It's nice to see our work (both this paper and the NIPS paper on transfer learning) in your class. Thanks for including it. I wish I could have my students take your course!

oh_sigh• 11 years ago

Of course, this requires that the attacker have complete knowledge of all the weights of the given network.

jeffclune13• 11 years ago

No, it does not. See my reply to similar comments.

appleflaxen• 11 years ago

This makes me wonder if we might be flirting with the computational equivalent of autism.

Could autistic children have learning impairment due to their inability to correctly sort/segregate stimuli, in the same way that these neural networks generate high-confidence false-positives?

ChuckMcM• 11 years ago

One of the interesting things was the 'white noise' which was identified as various animals. I reminded me of people looking at noise and "seeing" data. Which for me suggests that at some level this isn't completely an artifact. If the algorithms developed are so closely modeled on human perception are susceptible to this sort of thing, humans probably are too. Perhaps that explains reports of people seeing things in the electronic 'snow' pattern of a disconnected TV?

ars2• 11 years ago

The difference is that humans know they are seeing noise, they don't claim 99% confidence in their image, but rather a very low confidence.

Udik• 11 years ago

It's probably very naive, but that makes me wonder whether these neural nets are trained to recognize noise or meaningless images as such. If we train a system to tell us what an image represents, the system will do its best to classify it in one of the existing categories. But having a low confidence in what an image represents it's not the same as having high confidence in the fact that it doesn't represent anything. So maybe we should train the networks to give negative answers, like "I'm totally confident that this image is just noise".

masterfres• 11 years ago

Very interesting. First of all here's a youtube video associated with the paper -- https://www.youtube.com/watch?v=M2IebCN9Ht4 . Second some on here have posted about the Svegedy, Goodfellow, and Shlens paper http://arxiv.org/abs/1412.6572 which discusses the opposite effect. The Svegedy research is mentioned in the Nguyen paper and specifies that given an image that is correctly classified in a DNN, you can alter that image in a way imperceptible to a human to create a new image that will be INCORRECTLY classified. The Nguyen, Yosinski, et al. work that's the subject of this post states that given a DNN that correctly classifies a particular image, you can construct a gibberish image that the DNN will classify as the same image.

Both results are interesting from the point of DNN construction, and there have been some papers suggesting ways to counter the effects specified in the Svegedy research. In practice (as others have mentioned) in order to construct an exploit similar to the one described in this post, you'd need to have a lot of knowledge about the DNN (e.g. weights) that an external attacker wouldn't have.

What this does leave open, though is a disturbing way for someone with internal access to a DNN doing important work (e.g. object recognition in a self-driving car) to cause significant damage.

a-dub• 11 years ago

Cool! Reminds me of "Shazam Decoys" where barely audible or inaudible energy can be added to a signal to fool Shazam into identifying it as the wrong track.

I've often thought there would be an awesome opportunity in there to make a hilarious app that catches cheaters during the music round of Pub Quiz.

EdwardDiego• 11 years ago

I've often thought there would be an awesome opportunity in there to make a hilarious app that catches cheaters during the music round of Pub Quiz.

The best pub quizzes dim the lights so that smartphone cheats beam forth their sneakiness.

aclinnovator• 11 years ago

I think that the major problem with CV is that it only recognizes images in isolation from each-other. Humans understand what they are looking at by finding the concept that lies at the intersection of all the small ideas in the image. For example, a human would recognize a keyboard because it contains a "means of input" on which there are "symbols" specifically the "alphabet", arranged in a "logical format"("QWERTYUIOP") which they know is the sign of a keyboard. If a human were to see a keybaord that looks different from most, they can still make the inference that it is a keyboard by understanding the underlying concepts of what they see.

On the other hand, a computer mechanically relates the specific format of a keyboard to the word "keyboard." It fuzzy matches the pixels of images to extract the object in the image: not the individual ideas implicit in the image.

Computer Vision needs more depth to actually be considered vision.

justintbassett• 11 years ago

I'm not sure I really understand the implications of this. It doesn't seem like this is an inherent weakness in computer recognition of images, but instead a weakness of a particular DNN? Or am I way off base?

Images that fool computer vision raise security concerns

Related Stories