Why are we using black box models in AI when we don’t need to? (2019)

https://hdsr.mitpress.mit.edu/pub/f9kuryi8

by Hooke • 7 years ago

351 193 7 years ago

hdsr.mitpress.mit.edu

I think the main reason is that creating a so called "Black Box" model is faster and easier to some degree.

Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?"

I tend to agree with the articles premise that any model being used in critical decision making should at the very least have a list of parameters and the weight given to each, no matter how long and complicated such a list might be.

I think eventually this will be the end result of most critical implementations of machine learning applications as they make their way through the courts, as I can't imagine a judge accepting the argument "The machine made a mistake, we are not sure why or the reason why is proprietary, but we are not responsible because the machine made the error".

sigmaprimus2• 7 years ago

Ofcourse the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. As much as I hate regulations I think this may be a good place to legislate in order to protect those wrongly affected.

It makes me think about the mess the derivatives markets made in 2008

taneq4• 7 years ago

It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint.

Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.

rightbyte• 7 years ago

The killer feature with "black box" models is that you can do illegal discrimination with regards to eg. race and gender and then just blame the computer since you can't see the weight as easily as in component analysis.

klyrs• 7 years ago

Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?"

Exactly. We included mathematicians, computer scientists and engineers in the design of those calculators. They use algorithms backed by rigorous proofs, run upon circuits with rigorous tolerances such that the answer is correct (to a reasonable amount of precision). Kids with their calculators are intellectually lazy -- we should expect that, but continue to teach otherwise. To see this from full-grown adults in academia is an embarrassment. But, god, would I enjoy a ML salary...

dijksterhuis• 7 years ago

A little off topic, but also not:

I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot).

At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot.

I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs).

The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it.

And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers.

But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).

owenshen243• 7 years ago

My understanding is that interpretable models, especially for neural networks, are very far and away from state of the art performance. Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models [1].

Even in computer vision, which is where I think they've been most successful, the visualization techniques used seem more suggestive, then explanatory [2].

[1] http://www.shallowmind.co/jekyll/pixyll/2017/12/30/tree-regu... [2] https://distill.pub/2019/activation-atlas/

throwawayjava5• 7 years ago

I think it's a real open question whether the interpretable models are actually worse, or merely worse in competition/benchmark problem sets. The more deep models I build, the more I'm convinced that behind every inscrutable parameter hides a certain amount of overfitting, with maybe a few notable exceptions. E.g., can you build a decision tree that's not obviously overfit but is also susceptible to adversarial perturbations?

owenshen243• 7 years ago

I agree that there's probably a good question about to what extent looking at benchmark sets is biasing our judgment.

But it's also unclear to me how to get a decision tree to perform as well on image recognition tasks the same way that a CNN does. (Of course, as you mention, the CNN will likely face adversarial examples.)

_y5hn• 7 years ago

If it can be explained in terms of precise rules, that's called programming.

mlthoughts20182• 7 years ago

It’s also a real open question whether any of the interpretable models are actually interpretable, or even if they are in any well-defined sense more interpretable than “black box” alternatives.

In practice the answer is a massive “no” so far. Some of the least interpretable models I’ve had the misfortune to deal with in practice are misspecified linear regression models, especially when non-linearities in the true covariate relationships causes linear models to give wildly misleading statistical significance outputs and classical model fitting leads to estimating coefficients of the wrong sign.

Real interpretability is not a property of the mechanism of the model, but rather consistent understanding of the data generating process.

Unfortunately, people like to conflate the mechanism of the model for some notion of “explainability” because it’s politically convenient and susceptible to arguments from authority (if you control the subjective standards of “explainability”).

If your model does not adequately predict the data generating process, then your model absolutely does not explain it or articulate its inner working.

chestervonwinch• 7 years ago

Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models

Well, that and a decision tree isn't any more interpretable if its nodes are all filled with `Pixel[i,j] > Threshold` -- or is the idea that you would somehow extract logical predicates by tracing paths through the tree or glean interpretation from those predicates?

owenshen243• 7 years ago

I agree that large decision trees are also hard to interpret. The idea in the above paper is that we regularize with respect to average tree path length, based on the intuition that smaller trees are more interpretable. (But the issue about pixel thresholds probably still stands.)

dchichkov2• 7 years ago

I've happen to be in that workshop room and had chosen the 2% robot, not the 15% surgeon. If I remember correctly, the point of the question was determining willingness of society to use black box models, not challenging the need for models interpretability.

Interpretability obviously doesn't hurt accuracy. But it is costly to engineer. And not always possible to make. Not always possible, because human capacity (and willingness to put in the effort) into understanding the explanation is limited.

1_over_n5• 7 years ago

> Interpretability obviously doesn't hurt accuracy.

Why do you say this? From what i have seen it certainly can and does. For some industries finding a trade off where the magic is.

https://link.springer.com/article/10.1007/s10664-018-9638-1

https://www.oreilly.com/ideas/predictive-modeling-striking-a...

dchichkov2• 7 years ago

You can take nearly any black box model and add some interpretability into it, without modifying this model. No modifications to the model, so accuracy stays the same.

zevets• 7 years ago

I think this argument fails to understand that purpose of black-box models is to be impossible to understand, as it shields the operator from pre-emptive accountability.

Then, since they are supposedly the 'only way' to provide whatever function the market is 'demanding', then clearly we must abandon any notion of accountability, as this need 'must' be met.

This logic is perhaps most evident in google/facebooks content moderation dilemmas, where both companies refuse to define any sort of non-vague actionable standards about what they censor (or fail to promote) as 1) they can't provide them and 2) they don't want to be held responsible for what they actually are.

Then, as outrage has grown over facebook/google's terrible content moderation and censorship policies, and the need for accountability has grown, both companies have been forced to hire ever more moderators and censors, because the technologies as claimed just don't work well enough.

dmix• 7 years ago

Sounds more like a useful side effect than anything someone would design on purpose. But that's an interesting point none-the-less.

Occam's razors the easier way to explain the actions of the big guys and their own internal responses to cultural pressures to not allow things they don't like from happening or being tolerated in the world.

taneq4• 7 years ago

Well maybe also because black box models can be applied to a wider range of systems and can help to understand general properties of a class of algorithms. I’m sure it’s mostly all a conspiracy, though.

psv14• 7 years ago

purpose of black-box models is to be impossible to understand

This is just outright wrong. And the argument about moderation is completely unrelated.

bllguo• 7 years ago

I don't think it's worthwhile assuming this level of malice.. Ultimately we choose models that are the most performant. Interpretability is just a nice bonus. I can assure you that I am held accountable for my model performance regardless of whether I can explain its decisions satisfactorily. All the business cares about is whether we're burning or making money.

mkl• 7 years ago

They never got to the results of the competition and how the interpretable model did!

From https://www.fico.com/en/newsroom/fico-announces-winners-of-i...: "The team representing Duke University, which included Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang and Tong Wang, received the FICO Recognition Award acknowledging their submission for going above and beyond expectations with a fully transparent global model and a user-friendly dashboard to allow users to explore the global model and its explanations. The Duke team took home $3,000."

Cynthia Rudin is one of the article's authors.

RSchaeffer• 7 years ago

But how did their model compare against others? The article only mentions how their interpretable model compared against their own ML attempts

jph003• 7 years ago

Their model didn't win. IBM's model won, based on actual metrics around useful insights.

anfilt• 7 years ago

Like I have pretty much always thought this when I read about ML or uses of it. The process and end result is just math so it has to be explainable in some way. I feel like not enough effort is given to understand the important variables to ML models or how are results derived. When I see uses of it it's more like the following: "Oh, it works, that's good enough."

For-instance when a model starts being over fitted, one can say it's pretty much converging to something akin to a LUT (Look up table). While over fitting is generally undesirable it still might be interesting. Principally, if someone could figure out how basically it is indexing the data. Perhaps you could create a simpler hash function or find some rules for creating useful hash functions for that type of data.

psv14• 7 years ago

The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that works for 100s or 1000s of dimensions.

fallous• 7 years ago

Unless there is a self-modifying feedback loop you can indeed replace any ML model with a hash function. The problem is one of size rather than any fundamental limitation of a look-up table.

shakna2• 7 years ago

Shouldn't this fall into the "trust by verify" category of things?

Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour.

You have to trust it on new data.

You can always verify why it made the choices it did on old data.

That seems like something incredibly helpful, to me.

jayd163• 7 years ago

Uh, yeah you can. Hash functions handle data its never seen before.

rubyn00bie2• 7 years ago

I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample?

I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much.

Sorry, I'm just real confused by your statement.

jayd163• 7 years ago

Indeed. It would be nice to read an article about NN introspection/debuggers. I don't see why you couldn't use an introspected ML model as the basis for a behavior tree in some cases.

sgt101• 7 years ago

It's an old idea (22 years at least), the black box method can be used to remove noise and find a none bayes/compression/simplest info theory model, then the model is used to generate clean training for a decision tree or association rules.

OrgNet2• 7 years ago

why include 2019 in the title of this post? was it automatically added (is that a bug)?

Gaelan2• 7 years ago

There's a (human-enforced) HN rule to do that for old articles, so they're not mistaken for news. I'm not sure what the standard for "old" is, but it isn't 2019 anymore, so I guess it's technically correct.

OrgNet2• 7 years ago

2 months old is usually not considered old news on this site... I think that their algo is buggy on year-change

Lucadg• 7 years ago

The convenience to be able to say: "we didn't take the decision, the algorithm did" is the holy grail of non accountability.

I wouldn't be surprised if someone, especially in business, is actively opposing transparent AI.

jayd163• 7 years ago

Can you train a white box solution with only a tagged data set? It seems obvious to me why we have black box solutions even if we could have spent man hours engineering an algorithm.

swiftcoder• 7 years ago

You actually can - for example https://arxiv.org/abs/1806.10574

ummonk• 7 years ago

I think there is a lot of scope for interpretation in black box models. E.g. when doing transfer learning with a frozen pretrained model it should explain the new classifications in terms of pretrained classes.

1e-93• 7 years ago

The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you should not even be considering a blackbox approach (at least not without transparent safeguards). There is a large gray area between these two extremes that requires good engineering judgement to choose well.

I think we definitely need more R&D dedicated to creating easier-to-use, lower-cost approaches to transparent, explainable ML. There is way too much effort devoted to blackbox R&D today. Ultimately, transparent, explainable ML should almost always beat blackbox ML due to a better ability to find and resolve hidden problems, such as dataset biases, that may be holding back performance.

1_over_n5• 7 years ago

I would also add in that blackbox ML solutions are deceptively "simple" approaches. Like a miraj in the desert for software developers: No domain expertise neccessary - or even the counter narriatives may come in... "domain expertise holds you back - you need to be an outsider with a new method", where as many problems require domain expertise for a realistic understanding of the complexity of the problem-solution.

Other narriatives that can creep in "once we get enough data..." which for some problems may as well be never.

Also, for some systems pertubing the system changes the problem meaning your dataset and model may not reflect changes in a dyanmic / complex system.

Also, blackbox ML approaches struggle to combine data from multiple modalities which is often relevant to many real world problems (at least with the majority of algorithms which are realistically implementable off the shelf).

autokad2• 7 years ago

The main advantage of a blackbox ML solution is shorter development time to a useful performance level.

I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16 GPU instances over a couple weeks of training )

meanwhile, I consider gradient boosting fairly interpretable and they can get pretty close results with a lot less tweaking and training time. If you want to go full non-black box, logistic regression with l1 penalization and only a little bit of feature engineering often does really well, probably a lot less development time / cost compared to those high cost PHD research scientists.

1e-93• 7 years ago

Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs

DL's reduction of the required feature/model engineering is a big deal for difficult problem domains where the cost of a mistake is low. That doesn't mean you don't still benefit from adding development resources, it's just that the development cost/performance tradeoff is still typically better than with a similar-performing explainable solution. I hope this will change in the future.

..., logistic regression with l1 penalization and only a little bit of feature engineering often does really well,...

While I agree that Lasso is far more explainable than DL, its explainability rapidly degrades as the useful feature dimension increases and it requires significant feature engineering for good performance on difficult problems.

MiroF2• 7 years ago

gradient boosted trees are black box

hooande3• 7 years ago

This isn't how it works, at all. You're going to get better results with 1,000 parameters than with 100 explainable ones. There's a limit to how much humans can understand. We use machine learning to surpass that limit

1e-93• 7 years ago

You're going to get better results with 1,000 parameters than with 100 explainable ones.

This is not always true.

Many problems are modeled very well with less than 100 parameters and adding more is of little-to-no benefit.

Many problems are naturally hierarchical such that simple models can be combined to yield a large number of explainable parameters. If done well, this can result in a high-performing solution. Admittedly, this is usually harder than just applying a blackbox.

In critical applications, an explainable model with benign failure modes (even if it has worse overall performance), can be far preferable to a blackbox with wildly unpredictable failure modes. From a utility standpoint, the explainable results are better.

There's a limit to how much humans can understand. We use machine learning to surpass that limit

We can also work to improve our ability to discover and understand. I think that holds far more promise than improving our ability to do things we don't understand.

m0zg• 7 years ago

Human cognition itself is a black box model. We routinely make snap decisions first and then justify them after the fact. Anything that requires serious consideration is darn near insurmountable for a human mind in the presence of incomplete information, emotions, and conflicting motivations.

So while I'd like neural nets to be more interpretable, to me it'd take a distant second place. The first would be to get models that actually work better than humans for practical tasks, even under limited circumstances.

Rapzid• 7 years ago

I share this sentiment.

It occurs to me that human decisions making can be either explained a very high level, or a very low level(neurons firing). But the magic in between is too complicated to draw direct lines between stimuli and decisions.

AI seems to be the same way. We have statistical models that explain how most everything works at a low level. That a simple math formula can approximate just about anything. How we arrive at weights that separate data sets in different dimensions and etc. For convolution networks we also understand how layer "decisions" flow through to other layers. But, it's too complicated to look at an image and explain exactly how the input pixels will result in a classification output.

I'm not sure how much I should care about that.. Seems like a problem for mathematicians/statisticians. Having the doctor explain himself gives me no more true insight into how his brain works than having somebody explain at a high level how the robot was trained and reacts in different situations.

andlima2• 7 years ago

It's not that accuracy will always be sacrificed if one wants an explainable model. The point is: if interpretability is an important constraint, it could prevent improvements on accuracy.

Sometimes, the best interpretable model is as good as a black box, and that's great.

When this is not the case, the trade-off is that one should see what's more important for the actual problem. Perhaps interpretability is not a big deal.

Another solution is to try to extract interpretability from the more accurate black box model with something like SHAP.

1_over_n5• 7 years ago

This is a great point. There is a general lack of understanding about what it means for models to be interpretable & explainable. These words get thrown around often by people who don't understand the definition, and also the trade off with accuracy.

Some papers i found interesting on the subject:

https://arxiv.org/abs/1606.03490

https://arxiv.org/abs/1707.03886

https://arxiv.org/abs/1806.07552

https://arxiv.org/abs/1702.08608 (i found this was a good sumary of the issues)

rubyn00bie2• 7 years ago

[...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer.

If your data is off, your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science (e.g. racist AI making racist predictions).

A model cannot be accurate if it doesn't have enough information (like predicting crime, or the stock market). There are an insane amount of statistical tests to detect bullshit, and yet we're pushing those and hypothesis testing right out the window when we create models we don't understand.

Like I just don't get how some folks say "look what my AI knows" and assume it's even remotely right without understanding the underlying system of equations and dataset. You honestly don't even know if the answer you're receiving back is to the question you're asking, it might just be strongly correlated bullshit that's undetectable to you the ignorant wizard.

I find it pretty hard to believe we can model the physical forces inside a fucking neutron star (holla at strange matter) but literally no one in the world could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could, it would just feel unrewarding to invest that much time into doing it correctly.

I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.

StuffedParrot2• 7 years ago

So uhhh, isn't this like not science?

Very little of technology has anything to do with validating hypotheses.

meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model.

This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.

1_over_n5• 7 years ago

At our company, we are working with "mechanistic" or mathmatical models rather than stastical approches. I have observed :

1. It is hard than it should be to explain the concept to people (particularly VCs) 2 . people struggle to understand that a mechanistic model could have more utlity than a machine learning black box 3. people think you are doing something wrong if you are not using a neural network 4. The less people understand about neural networks, the more they seem to believe they are appropriate for all predictive / modelling problems 5. There is generally quite a low understanding of scientific method in the startup / VC space (speaking as someone who has worked in and around academia for years) vs how "scientific" people believe they are because it sounds good to be data driven and scientific about running startups and funding them.

baddox• 7 years ago

your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science

But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.

Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.

But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.

andlima2• 7 years ago

Machine Learning is about solving a task by automatically learning patterns based on examples and trying to generalize to new data. Granted, it's not necessarily the best approach to understand deeply how a phenomenon occurs.

You could use a black box model if you're more interested in predicting correctly images of handwritten digits than in understanding how the pixels relate to each other.

Of course, usually people want both accuracy and interpretability. It boils down to understanding what's more important for the problem at hand and making the compromises accordingly.

c2471• 7 years ago

I think you are missing the key part of the appeal here (or framing it as a negative).

Let's look at your question. Writing an equation for a cat is hard, actually really hard. Humans cannot reliably explain their decisions here. If I ask a person to tell me how they classify between cat and not cat, the answer will invariably be something along the lines of "well, it has the general shape of a cat". Which is actually just a huge combination of heuristics it took about 10 years to work out. There is quite a lot of work in neuroscience suggesting that the actual decision you make when you classify a cat happens before a rational is developed.

We could encode a function for that, but it relies on us knowing a lot about cats, which takes time and only works for toy examples.

If you use a convolution neural network, you can get close to human level performance on much more complex topics with little domain specific insight. There is no universal law for classifying hand written letters - they are an individual's interpretation of some symbols we made up. This task will always be 'non-rigorous' because the very underlying thing is not actually well defined. When does a 3 become an 8?

So we could have a person toil away and come up with a bunch of heuristics that we encode in a regression, but why is this better than having a machine learn those heuristics? Most problems are not life or death. What is the real added value in having people hand crafting features for predicting traffic jam delays or customer retention, when the end use is probably just to have a rough indication.

As somebody who does research using a huge range of models, I object that we should be guided by our intuition- our intuition is mostly wrong about non trivial problems.

Basically any "equation" somebody has discovered for what happens in a neutron star is "simple". There is a large amount of observational data, it is a consequence of some already well proven theorem, or relies on something well established to narrow the range of possible descriptions immensely, or (most commonly in my experience) the equation is basically a human version of deep learning, where grad students toil away making tweaks and heuristics until a point that the description fits the data somewhat well, and then there is some attempt to ascribe meaning after the fact.

For example, we can describe the trajectory of a comet using a "few" lines of high school level math. This means it is actually feasible for a person to have a reasonable intuition about what is happening, as the problem is actually dominated by a handful of important variables. Good luck getting anywhere near simple to describe cats (again, in a domain where the line of what is and isn't a cat is actually not even a property of it's physical attributes, so the problem is not properly defined under your requirement). To tell if something is or is not a cat, would require a DNA sequence. That is how we define the cat. So by your own definition, we do not have sufficient data in our dataset to properly do this classification.

I'm not sure you really understand the point you make about "statistical tests for bullshit". Most statistical tests are themselves ivory towers of theory and assumption which nobody ever verifies in practice (which is as unsciency as anything you accuse machine learning of). And people do actually use well grounded ways of evaluating machine learning models. Cross validation is very common and predates most machine learning, and has various "correctness" results.

For any model we build, if we do not have data that encodes some pathological behaviour we can test it out on, there is no test, no statistical procedure to tell us that model is flawed. If we have that data, we can run the exact same test on a black box model.

You should not conflate science with formalism or complexity. Running a statistical test is pseudoscientific unless you do it correctly and appropriately.

Saying something is not scientific because the data may not contain enough information to fully answer the question is flat out wrong.

mr_toad• 7 years ago

could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could

I had a philosophy lecturer who would vehemently disagree that it is even possible to construct an algorithmic AI to decide what is and isn’t a cat.

For a start, do you mean domestic cats, or the cat family? What about photos, sculptures and other representations?

I mean, is Garfield a cat?

People use statistical machine learning over algorithmic AI because trying to model the real world with algorithms is an endless and often pointless exercise.

hooande3• 7 years ago

The goal of science is to produce a testable hypothesis. The world is full of things that people can 'explain', or even prove with mathematics, but don't hold up to basic experimental testing or reproducibility.

Every new theory has a plain english explanation that's easy to understand. Few of them have the raw accuracy or reliability of top ML models

https://en.wikipedia.org/wiki/Replication_crisis

YeGoblynQueenne2• 7 years ago

> I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.

Problem is, most machine learning algorithms cannot incoprorate background knowledge except by hard-coding inductive biases (as, e.g. the convolutional filters in convolutional neural nets). Unfortunately, this is a very limited way to incorporate existing knowledge.

This is actually why most machine learning work tries to learn concepts end-to-end, i.e. without any attempt to make use of previously learned or known concepts: because it doesn't have a choice.

Imagine trying to learn all of physics from scratch- no recourse to knowledge about mechanics, electomagnetism, any kind of dynamics, anything. That's how a machine learning algorithm would try to solve a physics problem. Or any other problem for which "we already have well defined knowledge (and data) of". We might as well be starting at around before the stone age.

StuffedParrot2• 7 years ago

Absolution of liability. Why else use the term "AI" to describe your heuristic?

crazygringo• 7 years ago

I think it's important to note that human pattern recognition is basically black-box as well.

We can't "explain" how we recognize a particular person's face, for example.

Robust pattern recognition based on 100+'s of factors is just inherently black box.

Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Black box doesn't necessarily mean bad. I think the relevant question is: how do we ensure machine learning is trained and verified in ways that don't encode bias, and only used with "intended" inputs, so a model isn't being trusted to make predictions on things it wasn't trained for?

And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?

fhennig• 7 years ago

because despite being less accurate, they can have designers who can be held legally accountable and liable

Are you implying you cannot be held accountable for black box models?

darkerside• 7 years ago

I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?

JauntyHatAngle• 7 years ago

An anecdote, but this is a criticism being aimed at YouTube currently for it's demonitisation scheme.

The argument being made is that even though the demonitisation of certain classifications of content is immoral (e.g. demonitising any and all LGBT content, even when family friendly), it is tacitly being allowed to happen due to YouTube's stance of "the algorithm" making the decision, not them.

Which may lead then to have a position of "supporting" LGBT as a company while simultaneously being able to demonitise said content, which could hurt their business in actively hostile to LGBT countries.

This isn't proven, but it's an example of how it could be used to absolve yourself of blame and point it at an inscrutable system.

MaupitiBlue• 7 years ago

Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Isn’t the first sentence just an example of the second?

MaxBarraclough2• 7 years ago

It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions.

That's not the same question as whether there were feelings involved, or whether the process was conscious.

devit• 7 years ago

Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.

sojournerc• 7 years ago

But that explanation is likely full of cognitive bias, and all sorts of subjective experience.

Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.

daoxid5• 7 years ago

I think it's important to note that human pattern recognition is basically black-box as well.

Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

knzhou• 7 years ago

But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion.

No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.

dcposch• 7 years ago

we can ask them questions in order to find out how they came to a particular conclusion.

For the most basic cognitive tasks, we typically can't.

If you show someone a picture of a cat and a dog, they can easily recognize which is which.

If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...

q_andrew• 7 years ago

This is actually false. Early psychologists thought that humans could know everything about their own brain's processes, but observational psychology proved most of their assumptions wrong. It's called introspection, and it's almost always not indicative of a person's actual inner workings.

jlarocco• 7 years ago

Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?

rwallace• 7 years ago

even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers. I put it to you that having questions answered with plausible fabrications, is actually a significantly worse situation than not yet being able to ask the questions at all. At least in the latter situation, we know what we need to be working on.

chasedehan• 7 years ago

And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?

Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc)

Not sure what point you are trying to make here.

gusostow• 7 years ago

Those models are ensembles of decision trees. A single tree is interpretable, but sensitive to overfitting.

Balgair4• 7 years ago

I think it's important to note that human pattern recognition is basically black-box as well.

Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though).

I know this may not fit pattern recognition in a super-philosophical sense, but damned if it's not a ways down that road. We're literally shocking a person's brain so that they 'see' again (!!!). How V1 feeds into the rest of the brain is VERY active research and has had a lot of successes, but we're a long ways from a 'black box' these days.

Honestly, the auditory pathways are a lot better understood, as they seem to be 'older' and specific tones are represented physically in the mammalian brain (tonotopy). As such we know a lot better (relative to sight, smell, taste, etc) how tones are encoded and how they elicit specific recognitions and responses (this work is NOT straightforward).

In Neuroscience/Biochem, we are getting a LOT closer, but yes, we are not there yet by any means. The 'black box' idea of the mammalian brain is not going to last another 500 years, likely. We're marching along with good time as a species.

username90• 7 years ago

Knowing how to input data to the visual cortex doesn't mean that we know how it works. It is the same with neural networks, we know how to pass data to them and how they do the math, but we can't trace back its reasoning to know why it comes to the conclusions it does.

inimino2• 7 years ago

Sure, human pattern recognition is poorly understood, but it's a system that evolved over millions of years. We should expect more from something we ourselves designed.

The question of why we would choose a black box model over something interpretable that works equally or nearly as well still needs to be asked, especially if there is a misconception of black box models working better than interpretable models due to current trends and the hype cycle.

held legally accountable

I think this misses the point. It is not about accountability but about getting the right results. The right question is where we need an interpretable model to eliminate biases or otherwise satisfy ethical requirements. Often you can't know whether a decision was ethical without knowing how it was reached. Would you accept a trial verdict given by AI? What about a lending decision, or a job promotion? If you need to justify a decision and not merely reach a decision, you need something interpretable.

8bitsrule• 7 years ago

human pattern recognition is ,,, a system that evolved over millions of years. We should expect more from something we ourselves designed.

Why? Human intelligence evolved over millions of years. And we're struggling to keep our changes to the world from killing us.

Frost1x• 7 years ago

I think it's also important to remember a lot of human models were abandoned because of the inherent bias and black-boxed nature of them (feelings, subjectivity, etc.).

This is one of the reasons the scientific method blossomed: objectivity, rigor, transparency, reproducibility, etc. Black boxes can lead to bad decisions because it's difficult to question the process leading to decisions or highlight flaws in conclusions. When a model is developed with more rigor, it can be openly critiqued.

Instead, we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

To be clear, I'm not saying ANN/DNN models are bad, just that we need to understand what we're getting into when we use them and recognize effects that unknown error bounds may cause.

If, when the model fails to correctly classify a new data point, the result is your photo editing tool can't remove a background person properly... then so be it, no harm no foul. If the result is that the algorithm classified a face incorrectly with high certainty and lands someone in prison as a result (we're not there, yet) then we need to understand our method has potential unknown flaws and we should proceed carefully before jumping to conclusions.

galaxyLogic• 7 years ago

That is the fascinating thing about consciousness.

How do we "know" we see red and not green or vice versa? How do we "know" we are feeling heat or coldness? We just do. We feel different things in different, yet recognizable ways.

And these "feelings" are multi-dimensional, feeling heat and feeling color are not on the same axis. Red and Green seems to be on different points of the same axis and hot and cold are points on some different axis, other dimension.

How many different dimensions of "feelings" are there? Note they are not the same as different senses. There can be many different feelings associated with things we see and recognize that way: colors, and shapes and lightness vs. darkness etc.

raverbashing• 7 years ago

When a model is developed with more rigor, it can be openly critiqued.

Yes. And the models developed like this haven't solved the problems the current black box models have.

we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

There are ways being studied to check for sensitivity to parameters, biases, etc. But in the end, reality is difficult and there's no way of dealing with that, or "getting the right answer" every time

missosoup2• 7 years ago

I think it's also important to remember a lot of human models were abandoned because of the inherent bias

The industry needs to stop misusing the term bias this way. Virtually every attempt to find this supposed human bias has failed. Latest public example was Amazon and hiring[1]

Bias is the tendency to consistently mis-classify towards a certain class or tendency to consistently to over or under-estimate.

Somehow the term has been hijacked to mean 'discriminate on factors that are politically incorrect'. You can have a super racist model that's bias free, and most models blinded to protected factors are in fact statistically biased.

It's not constructive to conflate actual bias with political incorrectness.

Operational decision making, whether AI or human or statistical, faces an inherent trilemma: it's impossible to simultaneously treat everyone the same way, to have a useful model, and to have uniformly distributed 'bias'-free outcomes. At best a model can strive to achieve two of these factors.

See: https://www.youtube.com/watch?v=Zn7oWIhFffs

[1] https://www.reuters.com/article/us-amazon-com-jobs-automatio...

denzil_correa2• 7 years ago

Yes, that’s true. But human decisions have accountability - algorithms don’t have accountability. If entities who deploy algorithms would be legally required to take accountability for algorithmic decisions, there would be no issue with black box algorithms.

brycesbeard• 7 years ago

I agree in spirit, but a lot of human behaviors, derived from decisions, don’t have consequences. Did Nestle face consequences for their decisions to promote formula, etc?

jlawson• 7 years ago

how do we ensure machine learning is trained and verified in ways that don't encode bias

Or, more commonly, people work to ensure their machine learning is trained and limited to not specifically notice any patterns of fact that would be politically unacceptable to notice and make use of, even if it was rational and useful for the task at hand.

The status quo moral system of our society rests on a collection of objectively false beliefs about the physical world. Most humans know enough to avoid noticing these facts because of the social cost and possible societal negative impacts of shaking those moral foundations. It's not even necessarily wrong, since agreed-upon lies can do a lot to reduce conflict and cruelty. But machines need to be specifically trained to not notice these facts. You need to teach the machine not to notice the emperor's nudity.

And what's even better is that this'll never get acknowledged in the literature. It's one of those self-hiding facts which exists if you're willing to notice it yourself, but which no authority will ever tell you (I love these, wish there was a word for them).

gvjddbnvdrbv• 7 years ago

If we were able to provide perfect datasets you would be correct. As it is our datasets are imperfect and ML models are only as good as the training data.

lukemerrick• 7 years ago

This is an incredibly salient point. Others have also pointed out that there are numerous applications in which black box models appear to offer significantly greater accuracy than interpretable models, bolstering the notion that this article is a bit overstated.

However, in this article and elsewhere Professor Rudin has cited compelling evidence of cases in which black box models have been demonstrated to be no more accurate than interpretable alternatives. I feel this fairly justifies the question in the title of the article. For example, based upon available evidence, it appears reasonable that some onus should lie on the creators and buyers of COMPAS (a proprietary black box recidivism model) to demonstrate COMPAS actually is more accurate than an interpretable baseline. While it may not be the case, as the article seems to suggest, that in all modeling cases there is an interpretable alternative with comparable accuracy, in cases which there is, there doesn't seem to be any justification for using a black box model.

On the matter of "human-style" interpretability, we are brought to the difference between "interpretability" and "explainability." Humans have a complex capacity for constructing explanations for the thoughts and actions of ourselves and others (among other things). As OP points out, a lot of famous psychological experiments by Kahneman and others have shown how much of our reasoning appears to be post-hoc, often biased, and often inaccurate (in other words, human explanations are not actually true transparent interpretations of our thoughts and actions). However, we humans do have a powerful capacity to evaluate and challenge the explanations presented by others, and we are able to reject bad explanations. For those interested, a great book on this topic is "The Enigma of Reason" by Mercier and Sperber (https://www.hup.harvard.edu/catalog.php?isbn=9780674237827), but the gist here is that we must understand that while explanations are not the same as transparent interpretability, they are still useful.

I would conjecture that at some level of complexity (which some predictive tasks like pixel-to-label image recognition seem to exhibit), true end-to-end interpretability is not possible -- the best we can do is to construct an explanation. However, two very important points should be observed when considering this conjecture: 1. (Professor Rudin's point in the article) In cases which are not too complex for interpretable models to achieve comparable accuracy to black-box models, we can and should use them, as they offer super-human transparency at no cost in accuracy. 2. Constructing no explanations (or bad explanations) is not the same as reaching the same level of semi-transparency that humans offer. If we want to use human interpretability as a benchmark, black box models with no explanations are not up to par.

sullyj3• 7 years ago

Black boxness isn't a property of the thing, it's a property of our understanding of the thing. Just because we don't understand the how pattern recognition in humans works, doesn't mean the process has no internal structure, or that we couldn't potentially understand it in the future.

IshKebab• 7 years ago

Right, and currently we understand neither human vision, nor CNN vision - they are both black boxes. I don't think anyone said they must remain black boxes forever.

Edit: I guess you were referring to this:

Robust pattern recognition based on 100+'s of factors is just inherently black box.

I think he meant " ... given our current knowledge."

tdhttt2• 7 years ago

One more benefit: an interpretable model could at least give some hint when the model is not working as intended (e.g. debugging).

choeger• 7 years ago

Sometimes I have the feeling that this is exactly why so many people prefer black-box modeling. You are not going to be criticised and you don't have to do all the hard work of software development.

shrimpx• 7 years ago

It's true that humans are black boxes but there are whitebox techniques such as the scientific method and algorithms that are justified step by step.

I think the article tries to compare black box learned algorithms to explicit algorithms. Humans being a blackbox is more of a philosophical issue from that perspective.

agra2• 7 years ago

Was the fact that objects fall down a "black box" before Newton came up with some equations?

I don't think its fair to describe CNN's and human vision as being black boxes in the same way.

wongarsu2• 7 years ago

The fact that objects fall down was well understood, obviously. What wasn't understood was the mechanism that makes this happen. Why did that apple fall down instead of up? Where exactly is down a round earth? Where is down when I stand next to the Himalaya? Why don't planets fall down?

In my eyes this is very similar to our lack of understanding about the inner workings of a CNN, or human vision. We understand the input and the output and their rough relationship, we just don't understand how the move from input to output happens.

zajio1am• 7 years ago

Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Does it matter? If a brain counstructs post-hoc argument for given decision and the argument is defensible against other arguments, then the argument itself is good enough substantiation of the decision, even if it is different from the actual decision process.

6gvONxR4sf7o• 7 years ago

Not all human thinking is the same. How you decide the amount to tip your waiter is not at all the same as how your doctor diagnoses you. And at we have a zillion years of evolution towards understanding and judging and conversing with each other's black boxes.

mlthoughts20182• 7 years ago

“Explainability” is a subjective idea that allows people to act as gatekeepers about the use of a model through arbitrary politics.

“The Mythos of Model Interpretability” is good reading on this.

https://arxiv.org/pdf/1606.03490.pdf

jph003• 7 years ago

This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example:

"For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depended on race, age, and criminal history, that COMPAS itself must depend on race. However, when one approximates COMPAS using a nonlinear model, the explicit dependence on race vanishes, leaving dependence on race only through age and criminal history. This is an example of how an incorrect explanation of a black box can spiral out of control."

The concern about the strong relationship between race and COMPAS predictions is not largely based on a concern about whether there is an explicit dependence in the model. The concern is whether there's a relationship either explicitly or implicitly. And in particular, whether such a relationship results in unfair outcomes. The findings of the ProPublica study (https://www.propublica.org/article/how-we-analyzed-the-compa...) strongly suggested this was the case:

"- Black defendants were often predicted to be at a higher risk of recidivism than they actually were. Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).

- White defendants were often predicted to be less risky than they were. Our analysis found that white defendants who re-offended within the next two years were mistakenly labeled low risk almost twice as often as black re-offenders (48 percent vs. 28 percent).

- The analysis also showed that even when controlling for prior crimes, future recidivism, age, and gender, black defendants were 45 percent more likely to be assigned higher risk scores than white defendants."

I understand the desire of the MIT researchers to promote the value of their work, but in this case they appear to be doing so in a potentially damaging way.

makomk• 7 years ago

No, they're not misrepresenting this at all. ProPublica's article https://www.propublica.org/article/machine-bias-risk-assessm... was pushing the claim that somehow, the COMPAS black box was implicitly deducing defendants' race from the "137 questions" input to it and labelling them likely reoffenders based on it in a way that was indepdendent of the key factors known to affect reoffending rates, such as age and gender. The paper in question seems to demonstrate the exact opposite is true: ProPublica were inadvertently using race as an imperfect proxy for age at sentencing, which is what the COMPAS algorithm really cared about, because their attempt to control for age at sentencing didn't work. After controlling for the actual age factor, COMPAS results didn't have any relationship to race anymore. (It's not even a weird weighting factor: predicted reoffending risk falls off rapidly with increasing age at first, then more slowly in a smooth fashion. It's just not linear.)

Now, it's of course possible to argue that judging reoffending risk based on age is in fact unfair and racist because it has disproportionate impact on certain racial groups, even though it's strongly predictive across all racial groups. That's not the argument ProPublica made, though. Their argument was about the supposed perils of black boxes, and they kind of acknowledged that age probably wasn't a racist criteria - or at least that it would be a lot harder to justify calling it one - by attempting to strip out its effects in the first place. It's also a different kind of argument entirely, one that revolves not around whether the algorithm is somehow treating people differently based on their detected race - because it isn't - but around what it means for a decision like this to be fair in the first place.

jph003• 7 years ago

Whether a variable is latent or explicit isn't really relevant to the question of algorithmic fairness.

The link I provided gives the actual details of the method and findings; this is probably a more useful source for the details. The claim that the actual source of the difference is 'age' doesn't really make sense. There isn't enough of a difference in the number of young people between black and white populations to result in the differences found in the analysis.

(I do agree that the actual attempt to control for age was poorly done; it really shouldn't have been done at all, since it had nothing useful to add to the analysis or results.)

PS: It's 'COMPAS', not 'COMPASS'.

computerex2• 7 years ago

This is a pointless article. Obviously everyone would prefer fully interpretable models if it was possible to get comparable performance with the leading black box/deep learning approaches. No one chooses to make black box, non-interpretable models, it's simply the fact that the models that give the best performance tend to be black box. There is a lot of work happening on trying to interpret deep learning models, because no one prefers non-interpretable models and being unable to pin point and explain why some prediction was wrong.

erostrate• 7 years ago

The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models. They also give other examples of cases where people have used black box models unnecessarily.

computerex2• 7 years ago

The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models.

That's awesome, but this was a single model for a single application being measured against presumably by a limited test criteria/benchmarks. The point still stands that the recent narrow AI "renaissance" is largely due to deep learning, which is inherently black box. There is a lot of work going on in making deep learning more interpretable precisely because it's so prominent today and because its lack of interpretability is a huge con.

Machine vision for example has come a long way due to deep learning. A lot of autonomous car companies are relying on it for perception. All of these companies hate the fact that there is no way to tell when the classifier will fail or why it'd fail. When using deep learning in finance setting, its lack of interpretability is a huge downside.

Despite this deep learning is being used because it provides an appreciable improvement in performance over the alternatives. Certainly there could be alternatives to deep learning that may perform just as well or even better. But finding those alternatives for the vast number of applications deep learning is being used for today is much easier said than done.

rdlecler1• 7 years ago

For god sakes. Enough of this black-box—we don’t understand how this works narrative. Neural networks are JUST computational circuits. Unfortunately there’s been a lot of laziness in cleaning those circuits up (W_ijk=O) to better reveal the topological circuitry.

hyperpallium• 7 years ago

It's true our own cognition is also a black box, e.g. how do we recognize? It's as much a mystery to introspection as digestion. But the goal of science is to understand. Whereas with DL, we're just chimps in the dirt playing with sticks.

boltzmannbrain• 7 years ago

A couple salient quotes from "Interpreting AI Is More Than Black And White" in Forbes [1]:

"No matter the definition, developing an AI system to be interpretable is typically challenging and ambiguous. It is often the case that a model or algorithm is too complex to understand or describe because its purpose is to model a complex hypothesis or navigate a high-dimensional space, a catch-22. Not to mention what is interpretable in one application may be useless in another."

"Underspecified and misaligned notions of interpretation impede progress towards the rigorous development of understandable, transparent, trusted AI systems."

[1] https://www.forbes.com/sites/alexanderlavin/2019/06/17/beyon...

i000• 7 years ago

While I can definitely see the value of black box models in engineering and commercial applications (including clinical ones), I am quite troubled by the new hype of applying to deep learning to so many basic scientific problems. The 'deep' models do not tend to result in deeper insights - which idealistically ought to be the goal.

somurzakov• 7 years ago

When I want to predict something, I need the highest accuracy, and I am not interested in interpretability of an individual parameter.

and interpretability is only a half measure. If i really want to understand the model, I will conduct causal analysis with randomized controlled trial and will find the root cause and impact of the factor of my interest.

szczepano• 7 years ago

What a dumb question is it? The same logic is behind sports and nobel price awards.

It's easier to say to people who make financial decision that this is great artificial intelligence that will solve all your problems then explain mathematical equations behind it that 90% people won't understand. The more complicated the stuff is the higher probability it will be sold. That's the standard in IT world from quite some time.

bitL3• 7 years ago

I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of their interpretable models, then acting as arbiters and judges of the work of others, preventing future advancements of the field.

lonelappde4• 7 years ago

In your view, it's better to answer the wrong questions optimally than the right questions suboptimally? That's not the kind of AI want putting people in jail or denying them housing or healthcare or shooting missiles at them.

bitL3• 7 years ago

There are two parts to my opinion:

1) I would prefer something that works most of the time (blackbox) than something interpretable that doesn't

2) if my DL converged to e.g. solving some complex partial differential equation, how does it help me that I could "interpret that" if 99.9999% human population has no clue what is going on regardless? "Your loan was rejected, because the solution to this obscure PDR with these starting conditions said so; these Nobel prize economy winners created this model with these simplifications and we deem it good enough to keep you where you are."

progval• 7 years ago

Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage

While this is probably true (and IMO, possibly the right choice), this:

faking results of their interpretable models

is taking a huge leap.

Why are they more likely to fake their results than companies selling a black box model?

throwawayjava5• 7 years ago

IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals.

I'm also pretty wary of interpretability/explainability research in AI. Work on robustness and safety tends to be a bit better (those communities at least mathematically characterize their goals and contributions, and propose reasonable benchmarks).

But I'm also skeptical of a lot of modern deep learning research in general.

In particular, your critique goes both directions.

If I had a penny for every dissertation in the past few years that boiled down to "I built an absurdly over-fit/wrongly-fit model in domain D and claimed it beats SoTA in that domain. Unfortunately, I never took a course about D and ignored or wildly misused that domain's competitions/benchmarks. No one in that community took my amazing work seriously, so I submitted to NeurIPS/AAAI/ICML/IJCAI/... instead. On the Nth resubmission I got some reviewers who don't know anything about D but lose their minds over anything with the word deep (conv, residual, variational, adversarial, ... depending on the year) in the title. So, now I have a PhD in 'AI for D' but everyone doing research in D rolls their eyes at my work."

Those same people will likely at some point call for a strict regulation of AI...

The most effectual calls for regulation of the software industry will not come from technologists. The call will come from politicians in the vein of, e.g., Josh Hawley or Elizabeth Warren. Those politicians have very specific goals and motivations which do not align with those of researchers doing interpretability/explainability research. If the tech industry is regulated, it's extremely unlikely that those regulations will be based upon proposals from STEM PhDs. At least in the USA.

faking results of their interpretable models

Jumping from "this work is probably not valuable" to "this entire research community are a bunch of fraudsters" is a pretty big jump. Do you have any evidence of this happening?

MiroF2• 7 years ago

If I had a penny for every dissertation in the past few years that boiled down to...

This is very, very accurate. On the other hand, I oftentimes see field-specific papers from field experts with little ML experience using very basic and unnecessary ML techniques, which are then blown out of the water when serious DL researchers give the problem a shot.

One field that comes to mind where I have really noticed this problem is genomics.

YeGoblynQueenne2• 7 years ago

Leaving interpretability and conspiracy theories about softy PhDs aside, for a bit, "SOTA advances" are not progress. e.g., in the last seven years since AlexNet, results have kept creeping upwards by tiny fractions on the same old datasets (or training times and costs have gone down) all of which is achieved by slight tweaks to the basic CNN architecture and perhaps better training techniques, or, of course more compute. But there has not been any substantial progress in fundamental algorithmic techniques- no new alternative to backpropagation, no radically new architectures that go beyond convolutional filters.

But I'll let Geoff Hinton himself explain why the reliance on state-of-the-art results is effectively hamstringing progress in the field:

GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.

https://www.wired.com/story/googles-ai-guru-computers-think-...

bitL3• 7 years ago

no new alternative to backpropagation, no radically new architectures that go beyond convolutional filters.

Attention

My point wasn't about lack of investment/propagation of fundamental research that is not trendy, it was about hijacking what should be science by "softy PhDs" that found a niche in less demanding areas and will likely impose their will over the ones who are doing hard science and not politics, like how CoCs were recently used to take control over open source/free software licenses by some fringe non-technical groups. It's a pattern that is repeating across all industry and academia in the past, the ones that move field forward are often displaced by their "soft-skilled" and less-capable peers.

hooande3• 7 years ago

I don't know how many of you are familiar with Kaggle competitions, but the winning scores on the leaderboards can come down to 0.0001 points of accuracy or less. Winning something like that with an explainable model is like having one hand tied behind your back.

If tiny fractions of a point matter (the rare case), then making the model explainable is adding a ton of complexity for little practical gain. It sounds nice, but the numbers are unforgiving

lalaland1125• 7 years ago

Interpretability is probably one of the most misunderstood topics in AI. Interpretability is fundamentally not a math or statistics challenge, it's a Human Computer Interaction (HCI) problem and needs to be studied in a similar manner, with user studies. The fundemental goal of interpretability is to help users achieve some concrete goal such as detecting bias, identifying bad models, and debugging. This is similar how we can better design keyboards to help users achieve faster typing speed. Unfortunately, most interpretability studies don't actually bother testing if their methods actually work and I am highly skeptical that things like "white box models" or salience maps or attention weights or whatever latest methods can actually achieve the aims that interpretability proponents claim they do. I would love to see a simple randomized user study that would emperically show a concrete improvement in something like bias detection or error detection.

anthony_doan• 7 years ago

nterpretability is fundamentally not a math or statistics challenge

No, I keep on seeing AI/ML/DS people keep on downplaying statistic.

Statistic interpret things. The majority of the models out there have a one to one, predictor to response, holding all other predictor constant (linear regression, logistic regression, arima, anova, etc..). Statistic inference is a thing. Inference is interpreting. Descriptive statistic is interpreting. Parsimonious is a thing. Experimental design is a thing. Degree of freedom is a thing in statistic.

If you want interpretability do statistic. One of it's tenant is to quantify and live with uncertainty not fitting a curve and lots of coefficient to just predict. Not just classification.

It's a reason why biostat or econometric is a thing. Statistic.

Even the blog cited statistic papers even though it barely mention statistic models in it. ~~And Rudin is a statistician and contributed a lot in statistic.~~ Wrong person (I'm thinking of Rubin for casuality and missingness)

This is not a tribal fight between statistic and ML. This is pointing out that ignoring statistic is a detriment to AI/ML/DS as a field.

I predict that 2020 to 2030 statistic will be coming to AI/ML much more so regardless how much people downplay statistic.

~~Seeing on Dr. Rudin is coming over.~~ I've seen other statistician too. Dr. Loh works took decision tree and added ANOVA and Chisqaure to build parsimonious decision tree.

amitpaka• 7 years ago

Interpretable models are an exciting new entry to the ML toolkit. While they are likely sufficient for everyday ML tasks, they might not be expressive enough vs deep networks to tackle complex tasks like fraud and anti-money laundering classification. As with any tool, interpretable vs black models have optimal target applications that data scientist can apply them for. As interpretable models get wider tooling for training, expect them to play a larger role in simpler ML tasks. Black box models are however here to stay and will always have their place with their versatility.

tw1010• 7 years ago

I feel like we're ignoring the obvious answer: because most people in AI right now are overwhelmed by the necessary mathematical background and comfort necessary to produce first-principles generative innovative algorithms and models, so all they're left with is following the latest trend, which is locked on deep learning right now.

netcan3• 7 years ago

Ultimately, a lot comes down to the accessibility of tools and know how. The more widespread and general a technique or tool, the more widespread and general it becomes... because people already have it and know how to apply it.

OTOH, these authors (and old school heavyweights like David Ferucci) are not wrong. Interpretability, explainability & interoperability with human intelligence is not something to just give up on.

I like "challenges" as ways of exploring these areas. Good luck to the authors. Female team, btw.

tdhttt2• 7 years ago

When using ML models to deal with natural science such as physics, working well+being interpretable actually implies new discovery. Working well only, on the other hand, cannot win you a Nobel Prize.

sytelus• 7 years ago

There are probably dozens of holes one can drill into this sensationally titled click-baity study. It's surprising what gets through Nature's review system these days.

hesselink• 7 years ago

And yet you provide none. What do you base this on?

nurettin• 7 years ago

We have all these tools to build neural networks and we use them to peform feats where a simple C4.5 would have sufficed.

To continue innovation in the algorithmic field, I think we need a branch of scientific research or a competition where researchers are constrained to 100MHz CPU clocks and no GPU cheats when actually running their algorithms (not when loading and parsing data)

colinprince• 7 years ago

Off topic, apologies, but why are we putting (2019) in the title when this article is barely two months old?

Makes is seem like it's an out-of-date article. Is it?

Ericson2314• 7 years ago

The interest in black-box models is partly for the reasons given, that it is presumed that their inexplanability makes them more powerful, but also because business people are sick of dealing with programmers and analysts as a class, people with arcane knowledge limited replacability, and a penchant for blowing deadlines.

The dream of machine learning is to fire the programmer and break up the last trade (baring doctors and lawyers).

If it was just business people being alienated, I'd say fuck it they deserve it. But the fact is everyone, and many of the programmers too, are alienated too when much of the code makes sense to no one. I would rather fix programming to be comprehensible to more people even though it will take down the walls between programmer and non-programmer.

tripzilch• 7 years ago

there are actually a lot of applications where people do not try to construct an interpretable model, because they might believe that for a complex data set, an interpretable model could not possibly be as accurate as a black box. Or perhaps they want to preserve the model as proprietary.

I think they missed an important reason here. It's why I decided to study AI and Machine Learning: The ability to "just throw some data at it and see if it figures it out" is just so exciting, mysterious and intriguing.

I remember actually being a bit disappointed when I learned how (classic, 3 layer, early 2000s) neural networks worked. That it was just something super simple with derivatives, and resulted in something that seemed a lot like a more complex form of statistical regression. Kind of took the magic out of it for me, a little (didn't stop me of course, it was and still is an exciting field of research).

I know it's not a good reason, but surely I'm not the only one who thinks it's extremely cool that one is even able to build a black box that does useful things with what you feed it, but you don't know how it works, and yet you built it.

It's just something mystifying, and I believe there is also a tinge of fear for that disappointment, to figure it out and find out it's not really doing as clever things as you hoped it might.

Why are we using black box models in AI when we don’t need to? (2019)

Related Stories