Planting Undetectable Backdoors in Machine Learning Models

by belter • 4 years ago

275 59 4 years ago

belter• 4 years ago

"...We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate “backdoor key,” the mechanism is hidden and cannot be detected by any computationally-bounded observer. We demonstrate two frameworks for planting undetectable backdoors, with incomparable guarantees..."

PDF: https://arxiv.org/pdf/2204.06974.pdf

tricky777• 4 years ago

Sounds bad for quality assurance and auditing.

monkeybutton• 4 years ago

In the future one might wonder if they were redlined in their loan application, or picked up by police as a suspect in a crime, because an ML model really flagged them, or because of someone "thumbing the scale". What a boon it could be for parallel construction.

V__2• 4 years ago

Jesus. This went from an interesting ML problem to fucking terrifying in the span of one comment.

sponaugle2• 4 years ago

Overall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train the model to recognize everything as you expect with the exception that if the car has a sticker on the window that says "GTEHQ" it is not recognized. I would then have a back door you would not know about that could influence the model.

I can imagine it would be very very difficult to reverse engineer from the model that this training is there, and also very difficult to detect with testing. How would you know to test this particular case? The same could be done for many other models.

I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.

paulmd• 4 years ago

I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.

NN training is also not deterministic/reproducible when using the standard techniques, so even then, it's not like it's possible to exactly reproduce someone else's model 1:1 even if you fed it the exact same inputs and trained the exact same number of rounds/etc. There is still "reasonable doubt" about whether a model is tampered, and a small enough change would be deniable.

(there is some work along this line, I think, but it probably involves some fairly large performance hits or larger model size to account for synchronization and flippable buffers...)

wallscratch• 4 years ago

It should be totally deterministic with the use of random seeds, I think.

blopker• 4 years ago

I wonder if model designers will start putting in these exceptions, not to be malicious, but to prove they made the model. Like how map makers used to put in "Trap Streets"[0] in their maps. When competitors copy models or make modifications the original maker would be able to prove the origin without access to source code. Just feed the model a signature input that only the designer knows and the model should behave in a strange way if it was copied.

[0] https://en.wikipedia.org/wiki/Trap_street

criticaltinker• 4 years ago

This is known as a digital watermark, which falls within the domain of adversarial machine learning.

https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf

vlovich1232• 4 years ago

Copyright law will need to catch up with AI. What if I use your ML model to train my ML model? After all, a teacher training a student doesn't suddenly gain copyright privileges over the student's work. And it's not like you could easily test either. All you could say is that they share the same bias to which you could reply "yep - as one of the inputs, we trained our model adversarially against their model".

moyix• 4 years ago

Yep, it has been proposed: https://arxiv.org/abs/1802.04633

ersiees• 4 years ago

From working in a company providing neural network inference as a service, I can attest you, that we did this. We did it especially since we are scared people distill on our results. If the other service makes the same weird mistake, they distilled from us.

mr_toad2• 4 years ago

You can never 100% trust any AI anyway even if you trained it yourself. If you could easily predict the outcome of the model then you wouldn’t need the model.

sangnoir• 4 years ago

I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself

Are sure you can even trust the models you train yourself? It's possible that a model you trained is defective in a way you don't realize, e.g. will not recognize speeding cars with a sticker that has a picture of a bicycle[1]. It's likely that someone will discover that vulnerability before the publisher does, given the current state of ML model observability; ML model exploits are going to be wild, and inexplicable.

1. or the letters "GTEHQ"

sponaugle2• 4 years ago

Yea, I didn't think much about that, but you are right. Even if you train a dataset but use someone else's datasets those datasets could have specific things in them you might not notice.

The hardest part of this problem is the difficulty of auditing. If you use someones open source code you at least have to potential of reading the code to look for something... with a large model that is difficult on a different scale!

melony• 4 years ago

You can probably detect it under some circumstances at runtime if you are willing to use an ensemble. The more models you use the harder it gets to compromise all.

galcerte2• 4 years ago

It sure looks like such models are going to have to undergo the same sort of scrutiny regular software does nowadays. No more closed-off and rationed access to the near-bleeding-edge.

joe_the_user• 4 years ago

Well, this show ML models should receive the scrutiny regular software. But of course regular software often doesn't receive the scrutiny it ought to. And before this, people commented that ML was "the essence of technical debt".

With companies like Atlassian just going down and not coming, one wonders whether the concept of a technical Ponzi Scheme and technical collapse might be the next thing after technical and it seems like the fragile ML would more accelerate than stop such a scenario.

gmfawcett• 4 years ago

Wouldn't they deserve far more scrutiny? I know how to review your source code, but how do I review your ML model?

galcerte2• 4 years ago

By reviewing the source code of the model, reviewing the training data, and reviewing weight initialization, but the latter should be specified in the source code. Also making it abundantly clear that the libraries used to make the model were not tampered with, maybe hashing their files or doing some reproducible builds wizardry...

Edit: Now that I think about it, can't data poisoning happen when predicting, rather than just happening in the training phase? In that case, it's going to be complicated to work around that.

not2b• 4 years ago

"How can we keep our agent from being identified? Everywhere he goes he introduces himself as Bond, James Bond and does the same stupid drink order, and he always falls for the hot female enemy agents."

"Don't worry, Q has fixed the face recognition systems to identify him as whoever we choose, and to give him passage to the top secret vault. But it would help if if he would just shut up for a while".

boilerupnc• 4 years ago

Disclosure: I'm an IBMer

IBM research has been looking at data model poisoning for some time and open sourced an Adversarial Robustness Toolbox [0]. They also made a game to find a backdoor [1]

[0] https://art360.mybluemix.net/resources

[1] https://guessthebackdoor.mybluemix.net/

a-dub3• 4 years ago

i would guess that it might be possible to poison a model by perturbing training examples in a way that is imperceptible to humans. that is, i wonder if it's possible to mess with the noise or the frequency domain spectra of a training example such that a model learned on that example would have adversarial singularities that are easy to find given the knowledge of how the imperceptible components of the training data were perturbed.

has anyone done this or anything like it?

lIIIllllIIII• 4 years ago

given that AI is primarily trained on web data I wonder if it's possible to attack other people's ML training in that way :-)

PeterisP3• 4 years ago

Yes, much of research on adversarial examples is essentially about how to generate adversarial examples with minimally perceptible perturbations. IMHO the difficult part there is having a good model of what actually is less or more perceptible to humans. However, since that overlaps with other popular areas of research such as error metrics for realistic image generation in GANs, there are reasonable solutions to optimize for that.

On the other hand, a seemingly benign perturbation does not necessarily correlate with it being imperceptible. A larger, visually obvious perturbation with a plausible explanation can be less suspicious than a smaller but weirder perturbation.

postingposts• 4 years ago

Disclosure: I won’t say what I do.

You should really consider things from a “what can humans perceive” standpoint. There are things you can do with ML and eye saccades that you will literally never see because of perceptual delay. If I can push a saccadic event below 50ms you will never notice it. https://en.wikipedia.org/wiki/Saccade

That’s one example.

DanielBMarkham• 4 years ago

I know that this is about inserting data into training models, but the problem is generic. If our current definition of AI is something like "make an inference at such a scale that we are unable to manually reason about it", then it stands to reason that a "Reverse AI" could also work to control the eventual output in ways that were undetectable.

That's where the real money is at: subtle AI bot armies that remain invisible yet influence other more public AI systems in ways that can never be discovered. This is the kind of thing that if you ever hear about it, it's failed.

We're entering a new world in which computation is predicable but computational models are not. That's going to require new ways of reasoning about behavior at scale.

kvathupo2• 4 years ago

(Disclaimer: I skimmed the article, and have it on my to-be-read)

When I first encountered the notion of adversarial examples, I thought it was a niche concern. As this paper outlined, however, the growth of "machine-learning-as-a-service" companies (Amazon, Open AI, Microsoft, etc.) has actually rendered this a legitimate concern. From my skimming, I wanted to highlight their interesting point that "gradient-based post-processing may be limited" in mitigating a compromised model. These points really bring these concerns from an academic to business realm.

Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Of note, they seem to base their undetectable backdoors on the assumption that the shortest vector problem is not in BQP. As I recently learned looking at the NIST post-quantum debacle, this has been a point of great contention.

I've in all likelihood mischaracterized the paper, but I look forward to reading it!

ks1723• 4 years ago

As a side question: What is the NIST post-quantum debacle? Could you give some references?

izzygonzalez• 4 years ago

One of their post-quantum bets did not work out.

https://news.ycombinator.com/item?id=30466063

polygamous_bat• 4 years ago

Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable."

Fun fact, that is because they ARE primarily cryptography people! Goldwasser is known for Blum-Goldwasser or Goldwasser-Micali crypto systems, while Vaikuntanathan is known for Zero Knowledge computations, both materials from any standard cryptography textbook!

(And they're great teachers, I was lucky enough to have them both as teachers in a class a few years back :) )

visarga• 4 years ago

What was the size of the model(s)?

contingencies• 4 years ago

He who controls the data controls the learner. - @pmddomingos

One might suggest that the term 'model' is in fact an extremely bad choice of name for the concept of a collection of condensed post-training decision support data in the machine learning world, because it implies a faux-scientific air of objectivity, precision, peer review, and intelligibility for inspection that is entirely undue. IMHO better terminology would have been a new/clean term without conceptual baggage that included some recognition of its fundamental nature: computed/derived/one-way/known-fallible.

There are only two hard things in Computer Science: off by one errors, cache invalidation and naming things. - Phil Karlton

Quotes via https://github.com/globalcitizen/taoup

MauranKilom• 4 years ago

To summarize one of the constructions they provide: If you have an existing neural network, you can add a parallel network with just a few layer that performs cryptographic signature verification based on something hidden in the input (e.g. signs of a few input values). Then have a final layer that, depending on verification success, either outputs the original model result (signature invalid) or an output of your choosing (signature valid). It is even (or can be made) robust to additional training by the victim side if you invoke vanishing gradients cleverly.

Applying this to a practical ML model is of course left as an exercise for the reader. While the research certainly proves that it's fundamentally possible (and mathematically trivial) to do such a thing, I feel that the structures of ML models are relatively transparent in most practical applications, making it comparatively easy to detect "parallel" verification networks thus constructed. The dataflow graph will be pretty revealing, but the victim would have to actually inspect it in the first place.

Of course, that in turn makes it a game of obfuscation - can you inconspicuously hide the signature check and final muxing step among the main network? I have no doubts that you can find a way if you're so determined.

But I think the most salient points of the paper are that 1. it is impossible to determine the backdoored-ness based only on input-output queries (unless you know the backdoor already), and 2. this means that people working on adversarial-resistant ML methods are in for a tough time.

There's more to be found in the paper, this is just my short summary after reading the most interesting bits.

mr_toad2• 4 years ago

I wish they would use some term other than ‘back door’ for this. Some PHB is going to read the headline and think that using machine learning will let hackers into the network.

totetsu• 4 years ago

It's the ML version of Fnord https://en.m.wikipedia.org/wiki/Fnord

amirhirsch• 4 years ago

Backdoor seems like a misnomer since this does not represent a security vulnerability to the model host.

want2takearide• 4 years ago

That's why you don't use cosine activation and always limit yourself to Lipschitz functions, I guess?

Planting Undetectable Backdoors in Machine Learning Models

Related Stories