For AI to thrive, it must explain itself

https://www.economist.com/news/science-and-technology/21737018-if-it-cannot-who-will-trust-it-artificial-intelligence-thrive-it-must

by jkuria • 8 years ago

111 65 8 years ago

economist.com

mark_l_watson• 8 years ago

Great article. I found a library that works with Keras last weekend; it shows which input features most strongly contribute to classifications. My little experiment: http://blog.markwatson.com/2018/02/trying-out-integrated-var...

chrisfosterelli3• 8 years ago

I think most people agree in the deep learning community that ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interprets data from the components. If there was a crash, they can tie it down to "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" without needing much interpretability of the neural network directly. If it's an end-to-end system, such as the nvidia car which connects the camera inputs directly to a wheel output, then explainability of that network is much more important.

There was a poll I saw on twitter a while ago (can't seem to find now) that had asked if users would prefer to be treated by an AI doctor with low explainability but 99% accuracy, or high explainability but only 70% accuracy. I think context is key here as well. If the expert doctor working in tandem with the explainable system is better than the non-explainable system, you'd want the explainable system. But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

kazinator3• 8 years ago

explainability is necessary to use it.

For that matter, freedom from daily blue screens and reboots is not needed to use an operating system.

All sorts of non-AI software has confusing states and quirks, yet is in wide use.

ghaff2• 8 years ago

But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.

Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so."

Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.

chrisfosterelli3• 8 years ago

Is that because you care about verifiability though, or actually interpretability? If the system provides you a diagnosis without context, but later can be confirmed by actually testing for that condition, do you still refuse treatment because you don't know how the original diagnosis was suggested?

marcosdumay• 8 years ago

The current explanation for anything in medicine is "Dunno, we tested it and it seems to work". It is a much better one than "The computer says so", but it's one the computer can tell you too.

mseebach3• 8 years ago

There is a huge swathe of problems where AI can provide tons of value, and where most of the benefits and costs of getting it wrong lies with the person deciding to trust the AI in the first place.

Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.

throwaway20486• 8 years ago

If a self driving car decides to ram into a tree/another car at full speed regularly, im guessing there will be a huge demand for explainability, not just a collective shrug and "I guess we need to train it more dunno lol".

kurthr• 8 years ago

I think part of the problem here is what 'explainable' means and to whom. A Deep learning or Data Analyst may be looking for or accept one kind of answer (or even be able to comprehend it), while a Medical Doctor is looking for something else to hang his hat on, and the layman has an even different (lower?) expectation of explanaition.

Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?

moseandre• 8 years ago

> ways to understand NN's are useful, if for nothing other than debugging, but I don't think the community agrees at all over whether explainability is necessary to use it.

I have to be able to debug anything I work with; debugging is necessary.

> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop"

It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.

gmueckl2• 8 years ago

I would go further: AI in a system that is in charge of safety critical system needs to be understood perfectly by its engineers. Right now, the community is trying to get by with shrugging and saying "dunno, but look at the reliability". In the long run this is not going to be good enough. We will need tools to dissect trained systems and build complete explanations of why it works (or doesn't).

pacala• 8 years ago

The poll is biased. There is no evidence that performance and explainability are at odds.

chrisfosterelli3• 8 years ago

With our current models they definitely are at odds. There is a strong correlation between higher average performance and lower explainability when comparing machine learning approaches like neural nets, linear models, decision trees, etc for nearly all tasks.

PopePompous• 8 years ago

I cannot explain how I form the sentences I speak. I still find speaking useful.

CottageCarry• 8 years ago

'“the reason to give reasons is so that others will evaluate your actions and beliefs”. Today’s autonomous machines do not have their own interests to serve. Instead, their explanations are forged by and for human beings.'

This seems like the fundamental difference between AI and human intelligence at the moment. Our intelligence is based on our social prowess. We're always competing with, and exchanging ideas with our peers. AI should be modeled based on this.

rdlecler14• 8 years ago

This means that even the designer of a neural network cannot know, once that network has been trained, exactly how it is doing what it does.

This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our hands up. We can apply algorithms to trim out the spurious weights and what we’re left with is a logical circuit that we can analyze. Show an electrical engineer the circuit diagram of a 3 bit added and she’ll know exactly the function. Add a bunch of spurious circuits to that same diagram and of course it’s not going to make sense.

kazinator3• 8 years ago

If those elements of the graph are "spurious", why not remove them from the very beginning? Or, should I say, not have them there?

throwaway20486• 8 years ago

Because we dont know which ones are spurious and the parent is pretty optimistic about the idea of being able to analyize which ones are/are not.

hinkley2• 8 years ago

If removing all of the spurious circuits is “cutting the bullshit” then what is left over is the truth.

Distilling a problem to its essence. Isn’t that the hardest step in explaining it?

narrator• 8 years ago

It would be an interesting breakthrough if Alpha Go could explain its reasoning for various plays.

Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?

jonbarker2• 8 years ago

Since alphago plays the game differently (optimizing on odds to win, even if by a little, instead of the human method which involves intuition about shapes), it may be that its explanation if it were programmed to provide one would not provide any human readable 'insight' which an expert could then take to improve his or her game. Imagine a simpler example, some optimization solution using multiple dimensions and linear algebra. A description of how this works would be unreadable to most people. Since most readable language operates in the world of at most four dimensions without introducing jargon, this is the real problem, not that the program cannot explain itself.

hinkley2• 8 years ago

At the higher level it isn’t just shapes. The human player uses the shapes the help estimate the future value of any group of stones. They are running a priority queue of likelihood’s for each part of the board. When they resign that probability has dropped to zero.

If you’ve played against a dan player a few times, you might have had the experience of them ignoring a move you made and play elsewhere. That’s what happened. It’s also probably the moment you lost the game...

jraines• 8 years ago

Counterpoint: explain, in detail, why you clicked the "comments" link on this post. What weights did you assign to all possible other actions you could have taken, and why, etc?

natch2• 8 years ago

I’d like to simply know “how good is AI right now” at any given time. This information seems largely hidden.

For example, can any AI expert tell me: what is the exact date when Target corporation crossed the threshold of being able to face match 20% of its customers across different stores with 20% accuracy? How about 50% (for both of the different numbers, to keep it simple). How about 80%? 90%?

That’s just one example. If you’re tempted to reply with insights about face recognition, that’s not the point. The point is more that these numbers for ANY real world AI task (not just face recognition) are generally not shared, much less explained in the real world, even if would be possible to do so in an academic setting.

freeone3000• 8 years ago

These are published in papers, but for real-world products, they're competitive advantage. Why should we share them with anyone other than our customers?

natch2• 8 years ago

Data for very specific narrow tasks like face recognition is published. How the capabilities are combined in a special sauce, not so much.

So yeah, going off what you said, all these people saying “don’t worry, nothing to worry about with AI” haven’t the slightest clue what AI is up to outside of whatever bubble they are in.

shady-lady• 8 years ago

Random amount of nodes with random connections assigned random weightings until something works best.

Not so sure it's possible for anyone to seriously claim to be able to explain the inner workings.

Biggest problem with these I can see is that there is more than one way to arrive at the given answer.

Simplistic analogy, let's say the equivalent nodes were represented by the value 20. Was this number a result of 5 * 4, 4 * 5, 10 * 2, 10 + 10 etc..

Right now, it doesn't matter, we're all just happy that 20 is correct and works for our use case. Maybe that will be good enough but I doubt it.

mcintyre1994• 8 years ago

There's a pretty cool paper that visualises the different layers in a ConvNet trained on ImageNet: https://arxiv.org/pdf/1311.2901.pdf - page 4 has the layer imagery.

kartan• 8 years ago

Some people are missing the point.

"In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law."

It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates.

"Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful or discriminatory results." ... " If the randomisation of sex produces no change in the number of women offered jobs by the AI, but randomising weightlifting ability increases it (because some women now appear to have “male”abilities to lift weights), then it is clear that weightlifting ability itself, not an applicant’s sex, is affecting the hiring process."

This is the most interesting part of the article. As it shows that it's possible to test the system for unlawful decisions without actually understanding how it thinks.

skybrian• 8 years ago

It sounds like a useful test if you get a positive result for discrimination. If the test comes back negative, things are more complicated.

This isn't as simple as it seems because a supposed job requirement, while superficially neutral, may be unnecessary. For example, police departments used to have height requirements. But height requirements discriminate against women. Do you need to be tall to be a police officer? Maybe it's really about projecting authority? Okay, how do you measure that?

Making job requirements neutral is a rather fuzzy area. What is the job really about? Who decides?

Any variable that's not independent of gender can be used to (imperfectly) detect gender. It may not be easy to figure out if machine learning is using it for something real or just as a substitute gender-detector. Randomizing gender while leaving everything else the same isn't going to detect this.

tensor• 8 years ago

AI models simply exhibit the behaviour of the data they are trained on. If you make a model that tries to copy an individual persons behaviours, is it the AI model that is at fault or the person it's modelling? Where should the liability lay?

throwaway20486• 8 years ago

the problem with that also is that humans have emergent higher level reasoning, as a trivial, super simplified example, imagine if you trained a fictional AI on policy towards slaves before the worldwide ban on slavery, what sort of things would it suggest, its highly unlikely "banning slavery" would be one of them.

twblalock3• 8 years ago

Here's another example: an AI, perhaps in a self-driving car, did something that caused harm to people and got the manufacturer hauled into court, and the judge issues a court order to "make it stop doing that."

How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system.

If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.

d33• 8 years ago

On the other hand, can you actually make a human not repeat given action? Or explain it?

_up• 8 years ago

I don't think that that is really the Problem. We also know how to teach people something, without knowing how our Human Brain works. The Problem is Companies can select for certain unlawful behaviour, but nobody can proof that later.

danellis• 8 years ago

Sounds like robopsychologist might become a real job after all.

ims• 8 years ago

Just as with self-driving cars, you have to look at the counterfactual. We don't have an interpretable audit trail for the decisions made by human employees either.

So under the status quo, what happens in the case of an aberrant or problematic decision and how does society cope with this lack of interpretability?

We try to piece together a plausible, retrospective narrative by looking at fact patterns and taking into account the education, experience, actions, explanations, and rationalizations of other humans. (Trusting these accounts is further complicated by human traits such as self-interest, emotion, and cognitive biases.)

There exist entire professional specialties devoted to this problem (litigation, internal investigations, police detectives, accident boards) who spend a lot of time on this. Even so, much of the time we still don't really know why exactly people made the decisions they did.

This inconvenient fact does not stop us from employing humans, and society has not collapsed under the weight of the liability issues.

jes• 8 years ago

I don't think AI needs to be able to explain itself in order to be trusted. Human beings, in general, cannot give arbitrarily deep and valid explanations for their actions, and yet, somehow, we manage to come to trust them.

throwaway20486• 8 years ago

That's true, but in general humans are concerned about the consequences of their actions and are metacognitive about them, or at the very least can be punished for making the wrong choices, its an iterative corrective process.

How do you apply the same logic to a neural net? It has no comprehension of desirable outcomes, or what its choices actually entail and never can.

There is no way to tell a neural net that a human is not a signpost directly, so it knows in the future no humans are signposts. The only thing you can do is train it on a larger data set and hope it "gets it".

You can tell a human being that a human is not a signpost once, and they can abstract it to all situations.

jpindar• 8 years ago

Hell, you can't even tell them that a red octagon on a post with the letters S T O P on it is a stop sign.

jwatte3• 8 years ago

Humans don't explain themselves particularly well, yet they thrive.

hyperpallium• 8 years ago

Historically, observation and even usage precede understanding. Like planetary motion.

NNs (don't call it "AI", it's just one approach) are simply in a pre-scientific phase.

As are our own brains.

For AI to thrive, it must explain itself

Related Stories