Algorithmic decision making and the cost of fairness (2017)

by shawndumas • 8 years ago

86 107 8 years ago

candiodari2• 8 years ago

If justice was truly blind (which is what I would consider fair) then the number of defendants incorrectly classified as dangerous would depend on the likelihood of all the groups they're part of committing dangerous crimes.

You want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?

rspeer• 8 years ago

You could try reading this paper or any other paper on the topic.

mc323• 8 years ago

I believe France does not collect race information from its citizens (or perh inhabitants in general). If true, it'd be interesting for an outside observer[1] to see if the omission of race makes a difference.

That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc.

[1]Given they themselves don't officially collect race information.

halflings• 8 years ago

France does not use any algorithmic decision making in its court system as far as I am aware, but like many people said here, in the end, the cause is not "race" itself but the correlations it comes with (e.g: in France, people that commit crimes are more likely to be from poor suburbs, which are predominantly populated by people whose parents came from North Africa/Subsaharan Africa).

dspoka2• 8 years ago

So this idea seems intuitive at first but turns out to be one of the worst things to treat unfairness.

There are several reasons for this from both technical and legal perspective.

It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.

If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe.

from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing.

most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.

bluecalm6• 8 years ago

>It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.

I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing.

"You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with:

"You are not getting parole because you willingly associated with people who committed crime".

candiodari2• 8 years ago

The necessary result is that you are no longer judged on your crimes, but on your genes.

Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal.

Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack.

Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.

kosmet• 8 years ago

Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters.

What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.

bluecalm6• 8 years ago

>Then, the algorithm gives negative weight to locations where dangerous groups tend to live.

Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"?

>What you should do is to keep the ratio of people classified as dangerous same among different groups.

Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.

taneq5• 8 years ago

Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters.

No, the algorithm just gives negative weight to locations where dangerous groups tend to live. You infer race from that measure.

dspoka2• 8 years ago

Europe has also passed “right to explanation” to help with this issue which is great. unfortunately it seems they came up with it in isolation from researchers on this topic and went with the don't record it philosophy.

here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf European Union regulations on algorithmic decision-making and a “right to explanation”

goialoq4• 8 years ago

Because correlation != causation.

Here's a very simple refutation:

Is past arrest history a valid factor in the model?

What if past arrest history is biased due to biased policing (Hint: it is, well proven)

How about zip code?

TruthBeDum123• 8 years ago

So you are going to argue with statistical probability? It is our best determining factor. Have any better ideas?

If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC.

Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.

sean_anandale• 8 years ago

So in sum: "analyzing data from Broward County, we find that optimizing for public safety yields stark racial disparities; conversely, satisfying past fairness definitions means releasing more high-risk defendants, adversely affecting public safety."

In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.

hateduser2• 8 years ago

On a topic like this it’s important I think to address the elephant in the room. I don’t think this implies a genetic issue! I’m not an expert but from what I know this shouldn’t lead me to the conclusion that “black Americans are predisposed to violence” and, if you know approximately the same things I do, I think that’s probably fair for you too!

toomanybeersies• 8 years ago

I don't think anyone here on HN thinks that it implies a genetic issue, but rather an issue caused by generations of discrimination and failed attempts to stop the cycle of poverty and crime that plagues many of these communities.

lagadu2• 8 years ago

But they are, according to virtually all the data that we have. However there's no indication at all that the cause is genetic but rather cultural in the US.

bluecalm6• 8 years ago

>In other words, black defendants actually are more dangerous to release

Yes, but...

>and there is no magic algorithm that bypasses this fact.

Maybe there is, it's just the method used wasn't able to find it either due to limitations of the method itself, not enough information or bias in the training set.

As a toy example: assuming you only have race and age to make your decision on then to optimize for public safety you need to include race to make good decisions. If you have race, age, number of friends who committed crime then maybe you don't need race anymore. The problem is that we are likely not getting enough data and then race is a proxy for that uncollected (and maybe uncollectable) data.

chias• 8 years ago

If you have race, age, number of friends who committed crime then maybe you don't need race anymore.

I understand that your argument is a toy argument so it doesn't make sense to discuss it specifically, but I feel it's important to point out that the issue here isn't just which variables are used to make the decision, but what the decision ends up actually being. That is, maybe you find that you can make a very good decision based solely on age and number of friends who committed crimes, and don't take race into account at all -- but then if this algorithm ends up yielding "yes" to most white people and "no" to most black people (even if your algorithm doesn't use race at all), you haven't solved anything. [edit: "solved anything" was poor word choice on my part, obviously you have solved something, but you remain in the state described by the paper]

Another issue is that while you can "whitewash" variables, it's very difficult to scrub race out entirely. For example, in practice we can't use "committed crimes" as an indicator because we can never know a ground truth: we'd have to use "were convicted of crimes" instead. Unfortunately, you're far more likely to be convicted of a given crime if you're black than if you're white, so you're already mixing race into your variables even if it isn't named. With the disparity in convictions, enforcement, etc., it's very, very difficult to come up with measurable signals that are not in some way already tainted by racial decisions.

sanxiyn• 8 years ago

> there is no magic algorithm that bypasses this fact.
Maybe there is

No, there isn't. We actually have a mathematical proof (which is quite simple) why this is impossible.

Specifically, following conditions can't be true at the same time: 1. groups differ in base rate 2. prediction isn't perfect 3. decision is correct at the same rate for groups 4. decision is correct at the same rate for groups, restricted to positive/negative class.

1 is a brute fact. Your toy example insinuates at 2. 3 is called calibration and what is usually optimized by machine learning. When people say algorithm is unfair, it usually means 4.

https://arxiv.org/abs/1609.05807

rebuilder3• 8 years ago

If a group of people have, statistically, higher-than-average recidivism rates, should we be punishing all members of that group? That all but guarantees unfair treatment of individuals even if it makes statistical sense.

Even in your post you go from a statement that amounts to "statistically some groups have a larger number if dangerous individuals" to "black people are more dangerous". The two sentences do not mean the same thing!

taneq5• 8 years ago

You've got it backwards, though. The algorithm isn't saying "this person is black and therefore shouldn't get bail". It's saying "this person shouldn't get bail (according to a calculated flight risk based on bunch of reasonable criteria)", and a disproportionate percentage of the people who are assessed as high flight risks just happen to be black.

goialoq4• 8 years ago

And "public safety" conspicuously excludes the harms caused by incarceration (for both innocent and guilt suspects, and the innocent members of their families and communities).

(Not to mention that "public safety" still excludes the harms of centuries of slavery and decades of Jim Crow...)

alicevenrro• 8 years ago

Are you suggesting that we discard notions of justice / law due to historical sentiment? Or do you mean that we should address issues or biases which might be formed because of these events?

gopalv• 8 years ago

In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.

You are right, but that's not the problem with the algorithm.

A critical assumption with large-scale data mining is that past trends continue - the problem is that the existing data fits the algorithm. It is just a conservative What-If decision maker operates on existing facts (i.e bad present day situation), just wrapped into code (or worse, encoded as opaque literal "biases" in a decision tree).

I see somewhat similar patterns in lending interest data (redline zipcode -> credit ratings) and the problem is that bigger the historical trend data, the less forgiving a "past trends" algorithm will end up being.

Since this ends up being a prisoner's dilemma, if you are a rational actor in this system and the system keeps playing a defect card on you, then the obvious move is to always defect - cut your losses.

Algorithms can't improve the job prospects of the people released. And that's not the algorithm's fault.

And therefore, the you're right - the algorithm can't change the world beyond its output result.

noobermin3• 8 years ago

Which is why we shouldn't build our lives or policy around simple algorithms that do not take into account the breadth of human values, philosophy, and desire for change beyond the bad present day.

AnthonyMouse2• 8 years ago

A critical assumption with large-scale data mining is that past trends continue

Not necessarily. It would be easy to use only recent data and ignore data from decades ago. The prediction accuracy wouldn't go down much as long as there is a large enough sample size in the recent data. It may even go up if the old data really is inapplicable now.

Then if a past trend stops happening, the old data gets purged eventually and only the post-trend data is considered.

ubernostrum2• 8 years ago

You know the phrase "round up the usual suspects"?

It's the same root idea: the goal of the system is not to find the person who committed the crime. The goal of the system is, for each crime, to find a person who can be nailed for it. And you can manufacture a class of people such that whenever you need an offender you can go grab some of them and stand a good chance at getting a guilty plea or a conviction.

You start by selectively hyper-enforcing small violations against a chosen subset of the population. Get them for minor traffic violations (which you can ensure turn into arrest warrants by setting the fines and fees and other costs high enough!), get them for "paraphernalia" offenses (where you assume everyday objects in Person A's possession are evidence of drug habits while they wouldn't be assumed evidence in the possession of Person B!), get them for all sorts of things.

Now whenever there's a crime you just go pick some random people from the demographic you've been doing this to, figure out who you can bribe to testify against whom, and then march into the courtroom with witnesses and a defendant who has a thick file of previous "encounters" with the system, and off to a cell they go for a while.

And it gets even easier each go-round: people with criminal records don't usually have the resources to move on and rebuild their lives, so once they're released they're going to be right back where you found them last time. And they've got an even longer record now, which is just proof that you've been doing a great job in figuring out who these dangerous recidivists are! The model works!

ubernostrum2• 8 years ago

People who downvoted this (original comment is at -1): seriously, just go do some research on over-policing and corruption in police departments and prosecutors' offices.

I know it goes against some cherished beliefs, but this really is how much of the US handles "justice".

thundergolfer• 8 years ago

Bernhard Scholkopf explored this at the hour mark in his 2017 ICML keynote. https://vimeo.com/238274659

Was interesting, and worth a look.

avinium5• 8 years ago

I've only had a chance to skim this, so maybe I misunderstood.

That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race."

In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?

Pretty odd concept of "fairness" if you ask me.

Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.

goialoq4• 8 years ago

the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B

Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem.

Why should I be jailed because other people with similar skin color to me committed crimes?

Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?

avinium5• 8 years ago

I don't quite understand your point. Race is explicitly not a factor in individual pre-trial detention decisions.

People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc.

As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair.

In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations.

Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black.

As I said, I don't think many people would agree with that definition of "fair."

* I don't know exactly what the model inputs are, I just made these up for demonstration purposes.

laichzeit0• 8 years ago

In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?

Define "race".

avinium5• 8 years ago

Define "race".

Well, if you don't take the authors' approach, you don't need to. You just continue assessing pre-trial detention on the basis of each individual's background.

Houshalter7• 8 years ago

https://i.imgur.com/cxLFux5.png

wonnage• 8 years ago

re: COMPAS

As noted above, a major criticism of COMPAS is that the rate of false positives is higher among blacks than whites [2].

Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.

The line you quoted misses a lot of context - the 'standard' is an algorithmic risk score with a single threshold. The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment). So this standard is what we're stuck with. The paper asserts that it isn't fair.

scythe• 8 years ago

Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.

This is really the clincher. If the algorithm is effectively aggregating multiple human decisions, it ought to correct for known biases in those decisions.

jimmywanger2• 8 years ago

Policing in America is racist.

Citation needed. The fact that certain ethnic groups have problems with crime and violence can be attributed to the fact they grew up in single parent households, are less educated, and have less income than other ethnic groups, all things that contribute to a higher crime rate.

The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment).

I don't see how anti-slavery has anything to do with this. If you are convicted of breaking the law, as a citizen you will be subject to the same punishments as any other citizen.

Houshalter7• 8 years ago

Policing in America is racist.

Citation needed. Cops visit black neighborhoods more, sure. But they do that because they commit vastly more crime. Which is the cause and which the effect?

It's also a lot more complicated than that. People throw out studies that blacks are more likely to be caught for drug crimes. But they are less likely to be caught for more serious crimes. Possibly because they trust police less and are less likely to volunteer as witnesses. E.g. https://www.wsj.com/articles/the-underpolicing-of-black-amer...

avinium5• 8 years ago

Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.

Quite possibly - and if that were the case, it's certainly something to be addressed. As a non-American, I can't really comment.

But the paper doesn't really deal with that. It's saying that society should tolerate higher rates of recidivism amongst one racial group in the name of fairness.

I do understand the authors' thesis - I'm just saying that "fair" is a very subjective concept. I don't think many people will agree with the authors' definition.

crb002• 8 years ago

This hits home in Iowa. They just started using it to determine bail. https://businessrecord.com/Content/Law-Government/Law-Govern...

adjkant• 8 years ago

It seems like the problem is that any data that works statistically will not be fair to individuals who are intuitively exceptions to the general trend. It sounds like algorithms relying on statistics are pretty much guaranteed to fail the fairness test. It's the wrong tool for this problem I think.

There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways.

Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.

quotemstr7• 8 years ago

There are so many factors in morality that I think individually, case by case judgments are likely the best option.

What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth.

There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.

adrianN• 8 years ago

While I agree with your general point, artificial neural networks and biological neural networks are sufficiently different that it's not really fair to compare them just because they happen to have the same name. A biological neuron does a lot more than computing some affine transformation of its inputs.

Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.

dblotsky7• 8 years ago

1. This article doesn't mention neural networks.

2. "Your brain isn't doing anything a neural network can't do." Really, now?

3. The paper mentions "over 100 factors" the algorithm uses. Ad agencies have way more than 100 factors on me, and I still get irrelevant ads every day.

Humans aren't perfect, but they can catch things that stats won't, and vice versa.

dblotsky7• 8 years ago

The algorithm also does "case by case" judgments. However its judgments are purely based on conditional probabilities observed from past events, which are computed using regression analysis (a relatively simple method). Human learning is enormously different from regression analysis.

I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile

I think we'd all agree here. The point is that both methods should always be used together, because each has huge limitations on its own.

femto• 8 years ago

Statistics tell you something significant about a group, but little of significance about an individual. Statistical measures should not be used to make decisions about individuals. Other names for doing so are: "Sacrificing the one for the many" or "Presumption of guilt."

jimmywanger2• 8 years ago

There is no cost of fairness here. They're playing percentages. I'd rather have an algorithm make predictions than a human judge/prosecutor. If a certain group of people are disposed towards recidivism or not showing up for trial, it doesn't matter how much they've suffered in the past. That's a policy decision to reach out to them.

slavik81• 8 years ago

I just saw another article[1] on a study using the same data set. It concluded that the COMPAS algorithm is no more accurate than random people given the task on Mechanical Turk.

Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.

[1] https://www.economist.com/news/science-and-technology/217349...

noobermin3• 8 years ago

Off topic, I can't access the economist article, could you post a link to the study so I and/or others can read its abstract?

indubitable5• 8 years ago

http://advances.sciencemag.org/content/4/1/eaao5580.full

Not paywalled.

AnthonyMouse2• 8 years ago

Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.

That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement.

Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people.

And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions.

If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.

indubitable5• 8 years ago

That is a very poorly formed study precisely because systems like the Mechanical Turk actually work surprisingly well. This is a common theme in the 'wisdom of the masses.' If you ask a single person how many beans are in a jar, you're going to get an answer that's generally very wrong. Yet ask 100 people and average their answer and you tend to get an answer that's oddly extremely close to the correct answer. As the number of people independently asked approaches infinity, the error approaches 0.

So when you ask 'x' people to independently judge something, let alone something that is multiple choice with a correct answer, you're going to get answers that are far more accurate than a single individual would give you. So looking at the average answer of 'x' people, comparing it to an AI system, and arguing that they're relatively close so individual people are relatively close to the AI is completely fallacious.

---

As a tangential aside, there's another quirk to the wisdom of the masses. When you let the people communicate and try to intelligently organize and use expertise to come to answer, this effect disappears and the final answer again tends to be very wrong. Kind of an interesting perspective on the current zeitgeist of society and work.

===EDIT===

The authors were obviously aware of the wisdom of the masses. Quoting the paper itself:

To determine whether there is “wisdom in the crowd” (7) (in our case, a small crowd of 20 per subset), participant responses were pooled within each subset using a majority rules criterion. This crowd-based approach yields a prediction accuracy of 67.0%. A one-sided t test reveals that COMPAS is not significantly better than the crowd (P = 0.85).

That's a quite silly p-value and on top of that I'm not sure how they claim their system actually controls for the wisdom of the masses.

IshKebab2• 8 years ago

As the number of people independently asked approaches infinity, the error approaches 0.

Only if people are an unbiased estimator!!

Houshalter7• 8 years ago

Getting rid of algorithms means replacing them with humans. And humans are far worse.

Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker.

It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists.

Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all.

But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them.

This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not.

A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying....

Also, A.I. ‘Bias’ Doesn’t Mean What Journalists Say it Means: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...

quotemstr7• 8 years ago

It depends on your objective function, doesn't it? Computer models definitely do better than humans on the ostensible objective function --- predicting recidivism --- but computer models don't have decades of exposure to subtle social cues that signal to humans that the real objective function is one subtly different from the one that a naive understanding of the system would suggest.

That's really the beautiful thing about our shift toward algorithms: we'll no longer be able to rely on these subtle social pressures. If we want to optimize a specific function, we need to put that function in the open, where everyone can see it. Then, as the linked article demonstrates, we can compare the idealized and actually-desired functions and perform a real cost-benefit analysis.

extension2• 8 years ago

Imprisoning people without trial, for predicted future crimes, is what's unfair. Letting an algorithm make the prediction instead of a judge only punctuates the injustice.

What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color?

These are exactly the things that justice is supposed to be blind to.

I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.

quotemstr7• 8 years ago

Is there any circumstance in which you would support detention before trial? If not, would you oppose the detention of a serial killer caught in the act? If you do support pre-trial detention in some cases, how do you distinguish these cases from those cases for which pre-trial detention is unjustified? Could such a decision scheme be "fair" in principle? What would make it fair?

lmm• 8 years ago

You could apply pre-trial detention to anyone accused of a crime, not offer bail at all. The upper classes might start to care more about speedy trials and police harassment once it became impossible for them to buy their way out of pre-trial detention.

extension2• 8 years ago

I think detention before trial is always unfair in principle, but likely unavoidable in practice. I would like to see the issue acknowledged and taken more seriously, but it's a tricky problem and I have no easy solutions to offer. Practical mitigations may be the best we can do, which I'll grant may be expensive, non-trivial to implement, and allow more criminals to roam free. Here are some vague ideas off the cuff:

* Base decisions only on things that would be relevant in a trial, like evidence and criminal history.

* Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect.

* Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle.

* Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy.

If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.

nabla92• 8 years ago

Assume that you are insurance company with access to customers social media profiles.

You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself.

What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence?

Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.

In some other cases it might be justifiable.

taneq5• 8 years ago

Don't they do precisely that, across a whole bunch of data? Not "friends on facebook" but location, habits, age, etc.

nabla92• 8 years ago

Yes. That's the idea when managing insurance risk.

The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.

Houshalter7• 8 years ago

This is a different problem than what people are complaining about. In fact it may be the exact opposite problem in some ways.

The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive.

But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned.

Justice should be individualized or it's not justice.

There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.

dblotsky7• 8 years ago

Personal two cents: the actual maximally "fair" and maximally safe strategy is not to release anyone on bail. True positives are 100%, and false negatives are 0%.

Beyond that, I agree with nabla9:

I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.

Algorithmic decision making and the cost of fairness (2017)

Related Stories