Is Explainability the New Uncertainty?

https://statmodeling.stat.columbia.edu/2021/04/12/is-explainability-the-new-uncertainty/

by danso • 5 years ago

43 36 5 years ago

statmodeling.stat.columbia.edu

glup• 5 years ago

No.

taeric3• 5 years ago

The article makes a decent discussion piece. Such that it does seem that both are pitched as panacea cures for why the models sometimes don't work.

Combined with the idea that folks think models would be better used if they presented their uncertainty, I can see the direct line to models needing explainability before we deploy them.

To that end, why do you think "no?"

bumby4• 5 years ago

Not the OP, but maybe you can help me understand the relationship.

As I understand it, uncertainty is a statement of risk. Explainability is statement of understanding how a system works to produce an outcome. None of the four NIST principles seem to conflate the two.

I can say I understand how my brakes on my car may fail to work because it's an explainable mechanical system with known failure modes. However, that's different than the statement about the uncertainty that the brakes will work as intended. In the latter, there is a statistical probability that gets translated to a risk statement. I think one needs to have an explainable system in order to arrive at an uncertainty risk statement. They are both related to quality, but speak to different aspects of the problem.

troelsSteegin2• 5 years ago

Explainability is the new credulity. The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted -- without needing to check the actual explanations. The NIST proposals are interesting, though, and it would be a good challenge to make them work.

EDIT: adding commentary.

The four principles (quoting Hullman quoting NIST) are: 1) AI systems should deliver accompanying evidence or reasons for all their outputs. 2) Systems should provide explanations that are meaningful or understandable to individual users. 3) The explanation correctly reflects the system’s process for generating the output. 4) The system only operates under conditions for which it was designed or when the system reaches a sufficient confidence in its output. (The idea is that if a system has insufficient confidence in its decision, it should not supply a decision to the user.

What do I think Hullman is getting at with "the New Uncertainty"?

A major theme in Hullman's research, reductively, is understanding and then helping how people make sense of uncertainty as mediated through data visualization. In an explanation system, replace a the viz with a narrative of a solution process, as carried out by some algo agent, or with some rationale or justification, by as carried out by same. The process of judgment is now uncertain as well. Explanations are uncertain narratives. How does a person make sense of that?

The NIST #2 bar for what is "meaningful or understandable", without relying explicitly in the authority of the recommender, seems pretty high to me.

aeternum• 5 years ago

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after.

I wonder if we could tell whether or not a sufficiently complex NN or AI were doing the same, or if it even matters. It at least feels somewhat useless if system A makes the decision but then system B comes up with justification (even if the justification is plausible).

troelsSteegin2• 5 years ago

XAI research talks about exactly that, system B "surrogate models" that are consistent with the recommender right around the locus of a prediction. If the surrogate model is easy to interpret, you can forgo directly explaining the base recommender. I think a thing to keep in mind that recommenders may be proprietary, and thus pointedly black box. I view interpretability as a conversation between the recommender operator and the user, subject, or regulator of the recommender that is unfortunately mediated by confidentiality. I view explanation as like debugging, something you do in house in dev, and it bewilders me that one would launch a system that one can't explain. But, sometimes we make deployments first and then rationalize them after ...

dataflow• 5 years ago

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after.

Does rationalize here mean "deduce that a supporting argument would exist", or does it mean "actually determine what that argument is"?

Like imagine if the brain is solving an approximate (or different) problem first, deducing that there's a solution that's very likely to be feasible for the original problem, then working out all the details afterward. Is that rationalizing after the fact or before the fact?

akiselev• 5 years ago

There's pretty convincing evidence that even human brains tend to make decisions first then rationalize them after.

The problem with that is we can't simulate human brains to test the integrity of that rationalization. Assuming the algorithm is repeatable, the data from the rationalization can be used to generate adversarial input from the original that tests the conclusion is actually responding to those factors. At the very least, this would give courts an avenue to validate the algorithms in practice.

jerf2• 5 years ago

"The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted"

Not for long. This is trivially forgable with current AI tech. It's easy. Your AI tells you that X is in category Y because there's an 80% match on reason 1, a 65% match on reason 2, a 45% match for 3, etc. etc. Reason number 2 is, for the sake of argument, outright racist, and the person running this AI knows that, so they simply hand you an explanation with that reason removed. You have no way of knowing whether this has happened, neither does anyone else, and people continue to accuse your AI of bias. (Especially after I "helpfully" normalize the reason factors for you against the list I handed you, not what came out of the AI.)

Any human-comprehensible explanation produced by a program is certainly human-editable, and almost certainly practically editable by code.

If you were going to design an AI architecture to provide you parallel construction reasons systematically, it would be hard to produce something better than a neural net.

tshaddox2• 5 years ago

Wouldn’t a good explanation need to account for everything it’s purporting to account for and not contain any details that can be omitted or altered without changing what it accounts for? An explanation that omits that reason number 2 would not account for why the output changes when the input to reason number 2 changes in isolation.

neatze3• 5 years ago

Making all actions and effects explicitly explainable in terms of; why this happens, to me, it something in area of crystal ball, not science.

Extreme example would be; gravity is not explainable, and yet, I don't need to have analytical solutions to learn how to play tennis, baseball or soccer.

karpierz2• 5 years ago

Explainability isn't needed when your training set encompasses your test set, in the conditions that are meaningful. You have a heuristic which says "stuff tends to fall down to Earth", which you learned by tossing things in the air. The relevant condition you trained it on was "am on the surface of the Earth", which turns out to be the only thing you really need.

However, if gravity was very sensitive to the number of cars on the road, or the number of street signs, I think you'd run into a lot of trouble without an underlying model.

andy99• 5 years ago

Your example is good, but I would argue it points to a good confidence (uncertainty) estimate as the most important element of trusting a model, rather than a human-interpretable explanation.

Like you say, we want to know that the test data is distributed the same as the training data, and that our model repeatably gets predictions right in the presence of irrelevant perturbations (those two statements are at least partially equivalent). If we have this information, and are happy with the train/validation performance, then the actual process that our model uses to make a prediction really is not that important.

titzer• 5 years ago

So, basically, teaching. The best AI is a good teacher. What good is an AI if we can't learn something from it, after all? I would argue it actually has negative value if it takes over decision-making, as we slip further into learned helplessness and our expanding lack of agency leads to both oppression and depression.

behnamoh• 5 years ago

In what areas do we expect explainability to be crucial? It seems most recommender systems (think Amazon, Youtube, etc.) don't add much value by providing the reason as to why they're recommending a specific product/video. Are there areas where this could be problematic?

potatoman22• 5 years ago

Doctors generally won't trust or use models unless they understand its reasoning.

bumby4• 5 years ago

I would expect any safety-critical application to be have a higher bar for explainability.

As an extreme example, if image recognition is used for anti-aircraft defense, I think it would be nice to understand how it differentiates between an enemy plane vs. a domestic air carrier. Knowing it's accuracy is probably not enough.

threatofrain3• 5 years ago

What about for automated proving? The math community overall seems to have accepted evidence without explanation.

6gvONxR4sf7o• 5 years ago

Have they? I thought automated theorem proving was still super niche.

threatofrain3• 5 years ago

But not the results and the consensus on them.

DreamScatter• 5 years ago

Not true

threatofrain3• 5 years ago

So mathematicians are still doubting the 4 color theorem? And the classification of finite groups?

duckfang• 5 years ago

It should be blatantly obvious that is is, especially in cases where software is making actionable decisions.

Is the software trained on sexist, ableist, racist, etc content that is illegal or deadly when applied to decisions? To answer that requires having the AI algorithm explain why its decision is as such. If we could see the explanation, we could either consider it or rule it out.

https://www.technologyreview.com/2019/01/21/137783/algorithm... is one such software that was trained on racist data. Black people get harassed and sentenced more by police than others, and thus have strong bias in criminal data. This AI software was trained with that data, and thus perpetrates the same.

https://becominghuman.ai/amazons-sexist-ai-recruiting-tool-h... Is a hiring AI that Amazon used - turns out when you don't hire many women, and train the AI on the data, it's very good at finding women's names to deselect from hiring. It's the same trend - poisoned prior data leads to future poisoned data at scale.

https://abilitynet.org.uk/news-blogs/ai-making-captcha-incre... And this is for general internet users, primarily ones who have auditory or visual handicaps. CAPTCHAS are increasingly hostile to disabled peoples. A CAPTCHA might as well state "no cripples". But this is done in the spirit of 'we dont want automated bots here' - and real humans suffer.

Why was WATSON, the IBM AI cancelled? Because it kept recommending "treatments" that would kill its patients. At least it was probably killing people in a non-biased way. https://www.theverge.com/2018/7/26/17619382/ibms-watson-canc...

xbar• 5 years ago

What's the idiom involving a researcher/reporting announcing their name?

unexaminedlife• 5 years ago

I've thought a bit about this problem. I definitely see the reasoning behind it, especially in industries like Healthcare, or anything public-health related (like self-driving cars) etc.

It doesn't seem like it's as big a problem as I originally thought. If I understand correctly: At least with models that categorize data, wouldn't the internals of these models be able to determine where the correlations are being formed between neurons? Based on these "strong links" a model should probably be able to hook in some "analysis system" which will take the LABELS of each of the original inputs and explain that the model finds that a correlation between X and Y tends to indicate a particular output. I'm over-simplifying a bit, but in general this seems like it could be converted to natural language relatively easily.

However, this over-simplification is perhaps the new problem in that the explanations could end up being extremely long depending on how complex the problem they're trying to solve is. Perhaps, instead, a side-by-side comparison of a "visual graph" of the model's "theoretical ideal representation" of an idea vs what particular inputs produced.

This idea could be explored further by allowing experts in a field to confirm if the correlations being represented in a model are in fact correlations that scientists also believe exist based on their own studies. In other words, model A says a correlation exists between the person's first name starting with 'X' and the amount of oxygen in that person's blood cells. Obviously this model would be abandoned pretty quickly :)

Also I was recently thinking about how a lot of times, since individual models will piece together different correlations at different strengths, by averaging the outputs of many models (ensemble learning), it's almost as if you're polling a quorum of individuals for their answers. Like that tv show "Who wants to be a millionaire" individuals within an audience may not know the answer, but overall the audience (as a whole) almost always gets the answer right. So the explanation would look more like "the majority of the models have concluded that a correlation exists between X and Y. The data in question exhibits this correlation".

YeGoblynQueenne• 5 years ago

> As far as I know, most of the work in AI explainability and interpretability—with the former applying to any system for which reasoning can be generated and the latter applying to “self-explainable” models that people can understand more or less on their own—is still about developing different techniques.

It's worth noting that expert systems, the dominant AI approach of the '80s and '90s, were perfectly capable of explaining the reasons for their decisions. For example, see chapter 4, "Explanation" in the free "Building Expert Systems in Prolog" ebook from Amzi Prolog that describes a "mechanism of explanation" that walks backwards through the decision process of the expert system studied:

https://www.amzi.com/distribution/files/xsip_book.pdf

In that sense, the dominant approach nowadays, that of deep neural nets, is a big step backwards: explaining a deep neural net's decisions is very hard.

To be a little critical, I think the article is trying to make a virtue out of necessity, or perhaps it's just sour grapes. Who cares about explanation, it's not a real problem, etc. Well, if modern AI was good at producing explanatiosn of its decisions, you can bet that it would be offered up as a great advantage of the current techniques. But because it's hard to do and the public, as expressed by the NIST draft document, are getting impatient about it, then we need to come up with excuses why no progress is been done.

emmap21• 5 years ago

This is more like a conceptual framework. Similar to apply social studies to learn about machine behavior. In this case, it's a fancy AI. This appears to me as a waste of time to kill the wrong problem.

Is Explainability the New Uncertainty?

Related Stories