Stealing Machine Learning Models via Prediction APIs

by 0x0 • 10 years ago

206 37 10 years ago

danvoell• 10 years ago

Very interesting. It makes sense that you could learn the model for something like classification which has a cut and dry answer. Just brute force queries at the API, log the results, and start working on your own model based on theirs.

yid3• 10 years ago

Just brute force queries at the API, log the results, and start working on your own model based on theirs.

To add insult to injury, you could outsource the training of your own model to the API too.

dharma1• 10 years ago

API rate limits? Suppose you could get around that with multiple IPs/accounts. How many queries do you need to steal a (reasonable size) model?

yid3• 10 years ago

Depends on too many factors for even a ballpark. Take Google's Machine Vision API for instance. The limiting factor here is that the larger your model (and deep networks are very large models in terms of free parameters), the more training data you need to make a good approximation. To come close to "stealing" their entire trained model, my guess is that your API use would probably multiply Google's annual revenue by a small positive integer.

Alternatively, you could restrict your "stolen" model to a smaller domain and use fewer, more targeted examples for training. But at this point, you might as well start blending in predictions from other APIs, perhaps even training one off the errors of another. This is basically a technique that has been around for a long time, and in one incarnation is called "boosting" (see Adaboost).

vintermann3• 10 years ago

You don't need to steal the whole model. You can train a model yourself, and just "steal" those "bits" of a model that you can't reach with your own training data.

With carefully selected queries based on an already trained model, I think you many not need so terribly many. But that's just an intuition.

zellyn2• 10 years ago

This also seems useful for extracting a model, creating an interpretable version of parts of it, and proving whether it is prejudiced against certain races, genders, etc.

mdda• 10 years ago

If you put your discrimination detector on an API, you would enable the original model's creators to eliminate bias by training against it. Resulting in an anti-discriminative model, through a generative / discriminative process.

Houshalter7• 10 years ago

Why would a model be prejudiced against a certain race? Very rarely do people give race as a feature to statistical models to begin with. And even if they did, they do not have human prejudices. They train on actual data and care only about making the most accurate predictions possible.

zellyn2• 10 years ago

https://www.propublica.org/article/machine-bias-risk-assessm...

sp3322• 10 years ago

https://motherboard.vice.com/read/why-an-ai-judged-beauty-co... “It happens to be that color does matter in machine vision,” Alex Zhavoronkov, chief science officer of Beauty.ai, wrote me in an email. “and for some population groups the data sets are lacking an adequate number of samples to be able to train the deep neural networks.”

radarsat1• 10 years ago

They train on actual data

often generated by humans

and care only about making the most accurate predictions possible

of labels often generated by humans.

Bias can get into a classifier. It can get there through a biased model, but that's very unlikely. Much more likely is that it's trained on biased data. Which is _easy_ to do, even by accident.

flashman2• 10 years ago

It's possible to predict race by looking at several other variables which are correlated to race. See Technical Appendix A and especially Table 8 for an example of predicting ethnicity using surname and state only: http://files.consumerfinance.gov/f/201409_cfpb_report_proxy-...

You can do this deliberately, or your machine learning model might do it spontaneously.

yid3• 10 years ago

"Extracting a model" refers to approximating someone else's black box outputs. You would be dissecting your own approximation, which could systematically be very different from whatever black box you're aiming to make inferences about, even if they both produce similar outputs.

blurbleblurble• 10 years ago

woah... this is the stuff of science fiction.

when the supreme ai gains the ability to thirst for knowledge, it will steal all the machine learning models via prediction APIs...

gm-conspiracy• 10 years ago

So, is this just brute-forcing API calls to create a training set?

Is this only for supervised learning?

Also, couldn't this be done offline, pseudo-legitimately, using your API call log data later on? I don't see how that can be mitigated.

jdonaldson• 10 years ago

You wouldn't steal the model per se, but you could uses this technique to generate some nice training data.

Of course, model providers could just as easily have some sort of protection against this, similar to what's done with "trap streets" on maps.

glup• 10 years ago

Isn't this analogous to trying to understand human cognition by using human responses to inputs, e.g. the endeavor of cognitive science? Just in the case of inferring the ML architecture, there's a smaller hypothesis space than for what we think people are doing.

gcb0• 10 years ago

havent read yet, but am not expecting more than what we had in the 90s of trying to figure out search engines prioritization algos to use on our optimization ones.

ChuckMcM• 10 years ago

This is interesting but the real money is in extracting HFT models from other firms by messing with the order book and evaluating the response :-).

lpage• 10 years ago

Thankfully "messing" isn't the word I would choose for most market participants but you're describing the algorithmic game theory that goes into designing a good execution algorithm.

omginternets• 10 years ago

Is this even an issue? This strikes me as being economically infeasible given the size of training datasets.

swordswinger12• 10 years ago

"On Google’s platform for example, an extraction attack would cost less than $0.10, and subvert any further model monetization"

aab0• 10 years ago

This is not necessarily that surprising. Hinton's 'dark knowledge' (not cited in the paper) already showed that a remarkable amount of information is hidden in the classification probabilities emitted by a model, and that one neural net can learn a lot from and reverse-engineer another neural net given just its precise predictions.

rememberlenny• 10 years ago

How would you go about aggregating an image library to properly gauge the classification results? Would you just use ImageNet content or try using something entirely new?

Stealing Machine Learning Models via Prediction APIs

Related Stories