How a Machine Learns Prejudice

https://www.scientificamerican.com/article/how-a-machine-learns-prejudice/

by matthberg • 10 years ago

37 27 10 years ago

andrewclunn2• 10 years ago

There seems to be a bit of explaining away all bias as the result of human creators here. If an algorithm trained on real world data displayed some traditional bias, would we assume such a cause and dismiss it? One could just as easily make this argument to dismiss all correlation based evidence. Yes the code / methodology matters, but we'd best get ready for deep learning algorithms to confront us with some uncomfortable truths.

empath75• 10 years ago

Let's assume you have red and green people who are otherwise identical, and that there was systemic discrimination against green people in the past that led them to have lower incomes and lower education levels and live in areas with lower property values and higher crime rates. A naive computer model trained on objective data might find correlations between green people and high default rates on loans, for example, which might be accurate, but which aren't inherent to green people but were forced on them by external circumstances. A reasonable bank might then conclude that they should discriminate against green people without any personal bias against green people at all.

The question you have to ask yourself is: is that fair? And what should the law say about it?

iopq• 10 years ago

They aren't inherent in green people, but you shouldn't give them loans anyway. Loans should be based on reality, not an ideal world that we don't live in. And certainly fairness has no basis in reality. It's a human concept.

steveeq1• 10 years ago

(quote by some guy named "oehead2000" on some other hacker news posting. It fit the context perfectly, so I am copying and pasting from his reply)

"Unfortunately whatever algorithm computers come up with to assess creditworthiness is going to be 'racist' by the broad disparate impact definition of racism that the government uses. Ironically, the algorithms will have to be made actually racist to correct for this."

andrewclunn2• 10 years ago

Most significant variable. If green is a proxy for poor, which is in turn a proxy for risky mortgage loan, then the poor variable is the better point of discrimination.

h4nkoslo3• 10 years ago

Well, the proper response probably isn't to shoot yourself in the head and prevent yourself from knowing about the difference in default rates. It is entirely possible to figure out what is going on, to what extent, and then make an informed decision as to whether to continue. Avoiding discovering things, which is what all these asinine articles end up suggesting, is almost never a good idea for competent adults, and legally preventing people from discovering things is antithetical to a free society.

atrendyguy• 10 years ago

Forcing non-optimal decisions based on an arbitrary notion of "fairness" will only weaken our institutions and breed resentment: "group A gets loan approval with a lower credit score than members of my group, what gives!?"

It also increases the fragility of the group you're trying to help by acclimating them to lower standards.

The success of Asian and Jewish people suggests that things can correct themselves over time. The law should stay out of it.

http://www.pewresearch.org/daily-number/asian-americans-lead...

http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chine...

h4nkoslo3• 10 years ago

There is literally no support in the article for the contention that "artificial intelligence picks up bias from human creators" as opposed to making correct inferences from reality. All of the examples they provide, modulo blurry tank pix, are of the latter.

randyrand2• 10 years ago

Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!

ceejayoz• 10 years ago

"The computer can't be wrong" is an equally dangerous attitude.

jjoonathan3• 10 years ago

What makes me sad is that there are good reasons to throw out information which don't involve denying mathematical truth, yet people, for whatever reason, seem to prefer denial.

What should happen: "I understand that this person has black skin and therefore according to elementary statistics is more likely to be a criminal, but acting on the information would create an unfair double standard that punishes people for crimes they didn't commit, it would help perpetuate a cycle, and it would violate individual human dignity, so I won't."

What actually happens: exactly what parent said. "Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!"

facepalm.

Tharre• 10 years ago

It's not quite like that.

Though blaming algorithms for being "biased" is rather strange as well. It seems to me the algorithms do exactly what you'd expect them to: have predictive capabilities about reality.

Of course, there is a problem here. You don't train neural networks with data of people who've committed crimes. You don't have such data. So what data has been used by the engineers to feed to this machine? Data of people who have been convicted of committing a crime, mostly likely.

And now you've constructed a feedback loop. Those pesky blacks are more likely to commit crimes? Always knew it. A man has been murdered and there are 2 suspects, one white and one black? Why even bother investigating the white guy, we already knew it was the black one. Holy cow, 1 year with the new system and black crime has increased by 50%! Always knew it. Better keep the system in place to save everyone from those pesky blacks.

Oh, the irony.

jjoonathan3• 10 years ago

I understand your/TFA's argument but I think falls dangerously wide of the mark. I'm sure that the feedback process you describe is real, but I'm also sure that judicial prejudice is far from the only thing feeding the cycle of poverty and crime that plagues black communities. Even if we were to succeed at eliminating judicial prejudice from our training data, the other factors maintaining the cycle would once again lead the ML algorithm right back to p(criminal|black)>p(criminal).

The way to fight the prejudice is not to pretend that math would work out differently if only we had better data / more sophisticated models / etc, it's is to realize that it's OK to discard the result of bayesian reasoning due to factors out of the calculation's scope. Courts discard evidence all the time for a million sundry reasons. Racism / double standards can be another.

h4nkoslo3• 10 years ago

The interesting thing is that literally all of the inferences we are "worried" about computers making were social consensuses not too long ago, with ample first-order evidence to support them even in the absence of fancy machine learning algorithms.

KODeKarnage4• 10 years ago

In June 2015, for example, Google’s photo categorization system identified two African Americans as “gorillas.”
Law enforcement officials have already been criticized, for example, for using computer algorithms that allegedly tag black defendants as more likely to commit a future crime, even though the program was not designed to explicitly consider race.

Consider these two examples from the article. One is when the machine failed in a particular undesirable fashion. The other is when the machine "worked^" but in an undesirable fashion. (^That is, overall it might have worked well, but the bias in errors wasn't sufficiently explored.)

Modelling "undesirable fashion", which is a dynamic, subjective social construct, is far more difficult than either of the tasks originally set in the examples.

In the first example, how many images of people were confused with non-humans? I can't believe this was the only failed result. The only reason this particular example was problematic was that these specific people were specifically identified as gorillas.

No problem if they were identified as a car, or a chess piece, or a satellite. Gorillas though, that's a special failure. We all know why. And we know why the error occurred. If the machine sees predominantly white people and gorillas, then "people" are white and "almost people" but with darker pixels is "gorilla". It is human prejudice in interpreting the results that made the error an issue.

kahrkunne• 10 years ago

"racist computers" is the funniest trend since "sexist babies"

It should be obvious that if an ML algorithm is trained on sufficient real-world data, it won't be racist - it'll just not be politically correct

arjie• 10 years ago

Why is that obvious? Take the case of criminal predictions. If you're using convictions as ground truth, then you're just mirroring existing societal biases. For instance, in a society with pervasive racism (like, say 1960s America), the more real-world data you give it, the more your model is likely to converge on racism.

Computers aren't racist but to characterise the article as referring to something like that is to fail to understand it.

KODeKarnage4• 10 years ago

If you're using convictions as ground truth, then you're just mirroring existing societal biases.

And if you don't use real life data...? What do you get?

randyrand2• 10 years ago

Not as bleak as you imply. Just don't give the model race or strong indicators of race as an input.

dominotw• 10 years ago

whats with shitty scientificamerican stories on HN today.

grzm• 10 years ago

If you don't think the submission is appropriate for HN, flag it if appropriate and move on.

How a Machine Learns Prejudice

Related Stories