Early evidence on “predictive policing” and civil rights

https://www.teamupturn.com/reports/2016/stuck-in-a-pattern

by miraj • 10 years ago

78 76 10 years ago

danso8• 10 years ago

I thought this was an excellent treatment of the garbage-in-garbage-out aspect of this debate. Too often it seems people get wrapped up in the "Is it ethical to use a computer to judge someone?" implications, which are non-trivial, but also seem to be borne out of a general ignorance about the many ways we do allow computers (or rather, computation) to make judgements about people.

Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken.

I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary.

When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system

mgleason_3• 10 years ago

Government transparency is absolutely necessary, especially financial, but very lacking.

notahacker• 10 years ago

Aren't there numerous ML techniques potentially be applicable to evaluating potential future crimes where even the people responsible for writing the algorithm and feeding the massive dataset in don't really understand how a particular person might "get to red"? Transparency seems to be only part of the problem.

M_Grey2• 10 years ago

Maybe, but I'm not aware of any that have produced predictions on a broad in vivo population that are better than existing methods. It's one thing to do a heat map of a region at a certain time of day, and express that in terms of overall statistical risk. It's quite another to try and read the future intentions of a human being based on what amounts to a wealth of noise and a paucity of signal.

twblalock• 10 years ago

The corollary to this: if an algorithm uses AI or machine learning to the extent that nobody precisely understands why it makes the decisions it does, it will be very difficult to change its behavior in specific cases, e.g. "make it stop doing that," especially when the inputs cannot be changed.

This is going to come up at some point when a self-driving car does something that appears totally irrational and ends up causing an accident. Engineers will need to come up with some kind of explanation, and I suspect the general public will not be satisfied when they learn that the explanation may be unknowable, or may reduce only to probability instead of certainty.

I deal with some of this at work in far more trivial use cases, and non-engineers just can't seem to accept that sometimes you cannot fix the imperfections without introducing worse imperfections in other areas of the system, and that ML generally leads to output that is "good enough" instead of perfect.

comex• 10 years ago

Maybe. But at least we can demand to know the system's input data and what measure(s) the training process is designed to optimize. For example, in the case of Beware, which public records are being used, and what is the benchmark for red/yellow/green? Also, are there any systems in place to try to reduce things like racial bias or the self-reinforcement effect described in the report? Why or why not?

rodgerd• 10 years ago

I think the use of crime modeling doesn't have to be controversial

Outside of a 50's-style sci-fi utopia, I simply can't see how. What we consider a crime worth pursuing is, alone, a hugely difficult conversation.

tomjen3• 10 years ago

Possible heretical thought, could it be that predictive policing suggests the same high crime neighborhoods that are normally responsible for most crime, because they are responsible for most crime (at least the kind that the systems track)?

That would make them useless, true, but not racists (most of these neighborhoods would probably be predominately african-american, but they would also be predominately poor, which we know correlates with crime).

noobermin• 10 years ago

Their main argument against this is this[0]. See the quote from the expert in that subsection.

[0] https://www.teamupturn.com/reports/2016/stuck-in-a-pattern#p...

weberc23• 10 years ago

I'm confused by this argument; how does predictive policing relate to measuring department performance? The biggest concern I can see is that putting more police in certain neighborhoods will bias the input data for those models (presumably police reports), but this doesn't seem to be what this section of the article is saying. It also doesn't seem like a good reason to avoid computer models, but rather to increase transparency.

danso8• 10 years ago

I don't think it's correct to criticize your line of thinking as "heretical", but I would criticize it for being overly simplistic to how policing works in the real world.

For easy starters, as the OP itself mentions, while the people (via government) define what is unlawful behavior, the incidence of crime is defined by police, who choose to observe and act on alleged law-breaking.

The obvious example are things like rape. When Baltimore's number of rape cases dropped 80 percent in a 15 year period, was it because Baltimore city had made stopping sexual assaults a priority? [0] When Florida police claimed that a viral video of an excessively-speeding off-duty cop was just an anomaly, and according to their traffic stats division, the number of police officers who speed (outside of work) is far, far lower than it is among the general population, who could argue otherwise? [1]. And what should we make of a department that disagrees with the non-harshness of its own audit on felony statistics? [2]

But even if we move on to crimes of a more concrete nature, such as homicide and shootings, the case that the OP argues is that predictive measures that aim to interdict these serious, unambiguous crimes not only are questionable from a civil rights impact, but seem to fail at having a significant impact in the crime rate.

We don't even have to think about particularly sophisticated models: the NYPD justified its stop-and-frisk policies, which disproportionately targeted blacks, on the generally agreed upon assumption that blacks make up a disproportionate number of suspects in criminal cases. The Adrian Schoolcraft tapes [3] revealed how the department's unquestioning interpretation of data that, regardless of the validity of the foundational statistics, resulted in extremely unfair quotas, the kind of practices that make obvious the NYPD's unwitting collaboration with the black population to create a culture that is hostile towards police.

(And how is Officer Schoolcraft doing? He recently settled a lawsuit against the NYPD for committing him to a psych ward under false pretenses. [4])

But hell, let's think of the most accurate and simple-to-explain crime-detecting algorithm there is: red-light cameras and radar guns. I'll assume I don't need to provide citations for how this form of policing has been criticized over the years, or how the loudest critics aren't all just black folks, as it's pretty easy enough to find stories and documentation via Google.

[0] http://www.baltimoresun.com/news/maryland/bal-city-rapes-gal...

[1] http://www.pulitzer.org/prize-winners-by-year/2013

[2] http://www.latimes.com/local/lanow/la-lapd-audit-assaults-20...

[3] http://www.nytimes.com/topic/subject/stop-and-frisk-policy-n...

[4] http://gothamist.com/2015/09/30/whistleblower_cop_adrian_sch...

weberc23• 10 years ago

the generally agreed upon assumption that blacks make up a disproportionate number of suspects in criminal cases.

Do you use "assumption" here to mean "divorced from data"? Because I'm of the impression that is hardly contested in the field of criminology.

force_reboot• 10 years ago

I'm sure most people would not deny this mechanism exists. The distinction is whether one focuses on this mechanism (correct Bayesian inference) or the various biases and feedback loops described in the article. Personally I believe that progressives miss the forest for the trees: there is a huge gap between White and Black crime that encompasses almost every kind of crime[0]. While I am against any policy that exacerbates this gap, or otherwise discriminates against Black people, I feel that there is a huge effort by the progressive movement to promote an understanding of these subtleties, while at the same time actively repressing general knowledge of the basic facts about race and crime. For example, progressives like to point out that for drug charges this gap could be explained by uneven enforcement (though I don't know if this claim is accurate - the evidence is somewhat indirect) but fail to mention that for most other crimes this claim is not tenable.

And fyi, a very basic fact[1] in the field of criminology is that being black is predictive[2] of crime even after controlling for income and education. I'm not criticizing you for not knowing this since this fact is considered "racist" when expressed by non-experts, and experts only refer to it among themselves, in order to avoid the fact being misinterpreted.

[0] https://en.wikipedia.org/wiki/Race_and_crime_in_the_United_S...

[1] It's not entirely uncontested, but it certainly represents the majority belief in most social science disciplines. See https://en.wikipedia.org/wiki/Race_and_crime_in_the_United_S....

[2] "predictive" is a term of art, meaning that it has a positive coefficient when used on the right hand side of a regression where "crime" is on the left hand side, along with other right hand side variables which depend on the context.

danso8• 10 years ago

I can't speak for the progressives that you have read from, but I can say that there is strong movement on the conservative front for criminal justice reform. Here's a good op-ed by Doug Deason:

http://www.nytimes.com/2015/07/30/opinion/ruining-lives-with...

His first anecdote gets at the issue to which I believe you misattribute to progressives, "actively repressing general knowledge of the basic facts about race and crime":

I am a Republican businessman, and President Obama and I do not see eye to eye on most issues. But I agree with him on the inequities of the criminal justice system. I learned about it firsthand. Like many 17-year-olds, I did something stupid. It was 1979, and I threw a party at the home of neighbors while they were out of town. (Their son had given me a key.) The party got out of hand, ultimately getting the attention of the police. I was charged with felony burglary.
My actions were wrong and irresponsible. They could also have ruined my life, affecting my ability to go to college or even get a job. But unlike many in my situation, I was able to fight the charge.

Perhaps the progressives you accuse of deliberately spreading ignorance, are indeed not focusing on the "basic facts". But they might argue that that focus is flawed in the first place. If blacks were disproportionately targeted and punished for low-level offenses, then that leads to a disproportionate number of people who have criminal records. Even if you ignore the material impact of incarceration, there's the issue of how criminal record can be a negative factor in job searches and overall participation and trust in society.

The basic fact that a greater proportion of blacks commit serious felonies is probably not considered a basic fact for folks who believe that blacks have also a greater proportion of being disenfranchised. It's probably more like a tautology.

lukeschlather• 10 years ago

Omitting the distinction between being a convicted criminal and being a criminal is what makes your statement racist, and non-factual.

We don't have any sure mechanism to determine the incidence of criminal activity in the general population. Most rigorous academic analysis I have seen shows no statistically significant difference in criminal activity between races. (Which is in contrast to convicted criminals.)

fucking_tragedy• 10 years ago

Biased humans program systems, biased humans collect data, biased humans feed systems data, biased humans purchase systems that affirm their biases, biased humans point at the black boxes as a confirmation for their bias. How can a black box be wrong? After all, it is a computer and it is not influenced by bias like a human.

throw_away_7772• 10 years ago

Would you rather have a biased human attempt to make data-driven, statistically based, decisions, or a biased human make decisions based on whatever they feel like? Either way there will be bias in the system. You don't have to be faster than the bear, you just have to faster than your friend.

carapace6• 10 years ago

"The target of the Jihad was a machine-attitude as much as the machines," Leto said. "Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgements. Naturally, the machines were destroyed."

https://en.wikipedia.org/wiki/Butlerian_Jihad

Seriously though, how is this not the crudest sort of cargo-cult machine worship? I'm not paying taxes to be judged by a goddamned machine. It's crazy. If the people we're paying to administer the system are so degenerate that they would delegate their most essential function to a machine then we need to vote them out in favor of sober, responsible, educated adults. It's like they're admitting that they can be replaced by a script!!!

Knowing what I know about computers, and computer modelling, I'm appalled that people would make, buy, and employ these sorts of "voo-doo" machines.

(Thin sans-serif body font means you hate your readers, and making it grey means you really hate them.)

wyager• 10 years ago

It's like they're admitting that they can be replaced by a script!!!

Well yes, evidence suggests that in many cases they can be.

danso8• 10 years ago

I'm not paying taxes to be judged by a goddamned machine

I'd be shocked to learn that the IRS does not use some form of mechanical computation to determine probabilistically whether or not a given taxpayer should be audited for possible tax fraud in a given year.

carapace6• 10 years ago

Maybe so, I'm not privy to their methods, but I wouldn't be surprised that they use, e.g. statistical models, computed on machines, to help them figure out whether or not to look more closely at a given taxpayer.

But however unpleasant it may be, an audit is not a punishment. (Someone is abusing their power if so.) Not anymore than being selected for jury duty is a punishment.

Also, you used the word "probabilistically". We should not be assigning sentences based on the roll of a die, eh?

Insurance companies use machines and statistical models to set rates. But they also have strong feedback to improve accuracy. They lose money if they get it wrong.

Here we are talking about judgement.

I know far too much about computers to ever credit the idea that these machines can replace a human judge. The people who have put these things into practice are deeply incompetent and should be voted out of office and replaced by folks who won't devolve to crude machine-worship.

wavefunction• 10 years ago

I would be. I would find it much more likely that the IRS uses some sort of much cruder algorithm in tax auditing, which is what is intended to detect fraud and appears to be non-functional at this point.

pessimizer2• 10 years ago

If they did, I'd be shocked if large tax cheats weren't tailoring their cheating specifically to the algorithm, and the only people that it caught were small timers and people who made honest mistakes.

gumby• 10 years ago

I like that they begin by addressing the use of the word "predictive". How we frame an issue deeply effects how it is received, perceived, and accepted.

patcheudor• 10 years ago

"The fact that we even call these systems “predictive” is itself a telling sign of excessive confidence in the systems."

If you are worried about the outcome of the language in police intervention and court cases then "Speculative policing" would seem a more appropriate term for the technology and cary less weight. This might be an area suited for the FTC to step in and ask the industry to fix their marketing to be more inline with what they actually offer.

h4nkoslo9• 10 years ago

The problem is that the police are being asked to do what amounts to population control, of a historic problem population in the US, that has no real desire to be controlled, on the behalf of people with no real desire to be seen as controlling them. The most the algorithm can do is provide a justification for well-known solutions to crime (or the perception of crime), but laundered through math & obfuscated reasoning so as to not be seen as too racist.

Somehow, up until like the 1960s-1970s, this was not an issue. What changed, and what reason do we have for believing that any modernized algorithmic solution would be better than now-abandoned social technologies with actual empirical track records?

Hell, the vast majority of the time, you can't even accurately describe the problem in eg Chicago without being shouted down or deflected into someone's pet policy proposal.

panic2• 10 years ago

Considering a group of people a "problem population" due to their race is racist, but I think this perspective is shared by many people in America. How do we get police to stop being agents of racism when so much of the population condones it?

Somehow, up until like the 1960s-1970s, this was not an issue.

Was it not an issue, or did you just not hear about it? The 1960s civil rights movement happened (and in many ways is still happening) for a reason.

h4nkoslo9• 10 years ago

Blacks have ~4x the murder rate of the combined white / hispanic population, and IIRC something like 6-8x the murder rate of the disaggregated white population, depending on how exactly you segment hispanics. It ends up being approximately half of all murders. Note that this tends to be a geographically concentrated population, often next to an incredibly low-murder urban(e) white area, so you end up with cities like Chicago where the goal is to keep the anarchy from spilling over into your tax base.

It's not delusional to think that dealing with facts like this is a major area of concern for law enforcement and local political leadership, especially when people are so quick to denounce any contemplation of reality as racist.

I guess if you wanted to put on your PC goggles and studiously avoid any discussion of race, you could do the intellectually dishonest thing and talk about things like "the population of South Chicago", but being intellectually dishonest doesn't seem to have gotten us very far.

https://ucr.fbi.gov/crime-in-the-u.s/2013/crime-in-the-u.s.-...

shmerl• 10 years ago

Also see "Watchbird" by Robert Sheckley: https://www.gutenberg.org/ebooks/29579

By the way, DRM is an example of such approach, where people are judged as guilty by default by digital systems.

Houshalter2• 10 years ago

Remove the scary word "algorithm" and this whole issue seems blown up around something fairly mundane. If the police hand calculated how many crime reports came from each area no one would care.

People are so scared of algorithms. There is now scientific evidence confirming that severe distrust of algorithms is a pervasive human bias. Search for "algorithm aversion". They do trust them when they are believed to be infallible, but as soon as they learn they can make mistakes, they vastly underestimate them. And this is a damn shame, because in almost every case even very simple statistical procedures outperform human "experts".

Worse is this nonsense that algorithms can be "racist". As if a computer can have prejudice or care about anything other than maximizing accuracy. The media loves to fuel this narrative. E.g. the big hoopla when Google's image tagger tagged a black person as a gorilla. They had to remove it and apologize.

couchand• 10 years ago

If you ignore history it can be really easy to come across as casually racist.

Houshalter2• 10 years ago

What on Earth are you talking about?

omarforgotpwd• 10 years ago

Wow, cool! I am the technical founder of PredPol and he uses a screenshot of the original version, which I wrote in my college dorm room in his article. Somehow the fact that somebody took so much time to criticize it somehow makes me feel special. (I don't work there an

Early evidence on “predictive policing” and civil rights

Related Stories