Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

https://en.wikipedia.org/wiki/Goodhart%27s_law

by chrisshroba • 8 years ago

229 134 8 years ago

en.wikipedia.org

mathattack• 8 years ago

I’ve seen this a lot. Even people who intellectually know that correlation doesn’t equal causality make this mistake all the time.

smallnamespace2• 8 years ago

It's not even that correlation doesn't equal causality, it's that the correlation may also completely reverse.

mlthoughts20183• 8 years ago

So we should minimize the fraction of measures which have become targets then?

smallnamespace2• 8 years ago

You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them.

That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something that ceases to improve their life quality.

maxxxxx2• 8 years ago

Exactly. Don't reduce everything down to one factor.

In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors.

Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things.

Focusing on only one thing while neglecting others doesn't work.

davidmanheim2• 8 years ago

That's often helpful, but not sufficient in general. See the dynamics discussed in my article here; https://arxiv.org/abs/1803.04585

mlthoughts20183• 8 years ago

That just sounds like coordinate gradient descent on a multi-term objective function. Just same old optimization but with some metaparameters or component-wise conditioning.

In fact, I think to argue against optimization, you’d basically have to boil it down to existing philosophical arguments against utilitarianism.

Either way, I’m skeptical if this conclusion is very connected to Goodhart’s Law, because it assumes some type of normative axiom that would have to precede your interpretation of Goodhart’s Law.

My main interest was to pointout that it creates a sort of impossibility theorem situation if you accept Goodhart’s Law (I don’t personally accept it as a general rule, only something that applies to some types of measures and not others).

The impossibility criteria would be something like:

- If one finds a good measure, one should optimize it.

- If a measure is an optimization target, it is not a good measure.

- For a given entity, there exists a set of variables governing how well that entity can meet a goal (e.g. measures can possibly contain relevant information).

Basically you could only choose 2.

- If you pick the first two, it means there are no general variables governing the outcone you want; they are constantly changing and fundamentally unpredictable, so you might get away with optimizing something briefly, but its informativeness immediately disappears. This type of thinking is common in efficient market theory, for example, and could be connected to a fundamental belief about non-determinism or non-stationarity.

- If you choose the first and third, it seems like standard utilitarianism.

- If you choose the latter two, it seems like fatalism or religion — there are quantities that govern achieving desired outcomes, but they cannot be treated as objective optimization criteria and can only be analyzed subjectively or fatalistically.

squidbidness• 8 years ago

I love the irony in this.

nikanj• 8 years ago

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

tomjen3• 8 years ago

That is only true if you do that with no regard for the way it was achieved.

supercheetah• 8 years ago

That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.

acdha• 8 years ago

That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.

setr• 8 years ago

the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.

personjerry• 8 years ago

So then you add “size of customer base” to your metrics!

scj• 8 years ago

Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted).

In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.

minimaxir3• 8 years ago

When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales."

I was too taken aback to follow up asking why # of sales can't be gamed too.

bsg75• 8 years ago

Number of sales (or customers) is a vanity metric favored by startups looking to entice investment. Revenue is a metric useful for measuring actual business health.

Any chance the interviewer was overly exit eager ?

rhizome• 8 years ago

Credit where due, I think this is the first time I've read of anybody being told how they got something wrong in an interview.

abiox• 8 years ago

interviewer replied "that's a dumb answer

a bit ot, but was that the actual response?

personally, stuff like this would have me ending the interview early.

heavenlyblue• 8 years ago

That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed.

Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy.

IMO there must be high-level metrics which define which low-level metrics are valuable today.

usrusr• 8 years ago

It's getting weirder when you factor in shareholder value: sacrificing the future of the company for looking good in all the fashionable metrics might be exactly the leaders job, depending on how the board is made up.

mwexler• 8 years ago

But of course, "If you can't measure it, you can't manage it", said by Drucker, says EVERYONE. Hence the cherry picking of preferential KPIs or OKRs.

But, actually, Drucker didn't say it. In fact, W. Edwards Deming said it. And what he actually said was "It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.". So, basically, the quote held up as the "reason" behind excessive over-tracking of the wrong thing... is wrong.

https://blog.deming.org/2015/08/myth-if-you-cant-measure-it-...

pwagland• 8 years ago

Indeed this is the whole idea behind OKR. The Objectives are _supposed_ to be grand, audacious, and only achievable under extreme stretching. Furthermore, they are _supposed_ to not be used as a KPI for performance reviews.

mitko• 8 years ago

I've seen similar topics come up on HN front page every once in a while.

Here are some of my thoughts on the topic, through the prism of metrics for a company.

http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-para...

TLDR of my post: The vanity metric paradox - every metric, sufficiently optimized becomes a vanity metric, as it stops being the make-or-break qualifier for a company.

kolbe2• 8 years ago

I wish more people knew and understood this concept.

It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.

velp• 8 years ago

Can you explain how the recession was caused by "government's choice of credit worthiness metrics", instead of rollbacks of regulation?

kolbe2• 8 years ago

I'm not sure why you think these things are mutually exclusive. And as far as Goodhart's Law is concerned in this case, they're intertwined concepts.

The government's choice to allow securities rated by S&P/Moody's to be treated as gospel allowed/encouraged banks, insurance companies, GSE's and pension funds to take far more risk than was systemically reasonable.

RelaxSelf• 8 years ago

yep, seen several idiots whose work habits are destructive be promoted for gaming stats

seen stats make people discourage/delay income/work to avoid anomalies/spikes in stats because if there's a spike it makes it look like you are slacking the next month, or makes a manager think you should be able to constantly hit the spike

stcredzero15• 8 years ago

During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.

AcerbicZero• 8 years ago

Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.

stcredzero15• 8 years ago

That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example.

There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.

lainga• 8 years ago

I recall the Luftwaffe ace rolls for the invasion of Poland reporting that 300% of the Polish tanks fielded had been destroyed.

jancsika• 8 years ago

Why guess about documented history?

sonnyblarney13• 8 years ago

When the enemy has limited personnel and equipment - targets such as body count, vehicles destroyed etc. are very good measures of progress.

And FYI the US handily and clearly won the Vietnam war; the South Vietnamese insurgency was defeated and the North Korean Army was obliterated.

Due to political pressure at home, the US withdrew after the war, and a short while later the North Vietnamese were able to rebuild their army and waltz in and topple the fragile S. Vietnamese government. The Communists then murdered 10's of thousands of people in the streets and imprisoned 100's of thousands and of course enforced totalitarian rule which kept millions in fear and poverty.

Were the Americans to have simply left a contingent of soldiers in the South, thereby keeping the legitimate threat of violence, it's unlikely the South would have fallen and might very well look like S. Korea today.

To your point, 'body count' is not a very good number to use when you're fighting an insurgency where there are a lot of irregular forces who can come in and out of fighting, or where soldiers can be replenished (i.e. North Vietnam can recruit willy nilly on their turf).

But it's still a valid measure and though you don't hear about it these days (for PR reasons) you can absolutely be assured that all NATO forces meticulously con't how many enemy forces are destroyed, and that it is a 'target' on many missions.

Semirhage3• 8 years ago

And FYI the US handily and clearly won the Vietnam war; the South Vietnamese insurgency was defeated and the North Korean Army was obliterated.

That’s a serious, if delusional claim that requires a loooooot of support. “Handily” in particular seems like a ridiculous way to state it, even if you only look at US casualties.

mikeash3• 8 years ago

If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out.

dredmorbius3• 8 years ago

Oh Mai Lai, as George Takai might say.

(And mispronounce.)

inanutshellus• 8 years ago

And as soon as a measure becomes known, it becomes the target.

IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure."

"I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks."

"What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."

spodek• 8 years ago

And as soon as a measure becomes known, it becomes the target.

Soon after, 90% of the target is still an 'A', which becomes the new target.

On a related note of unintended consequences, the Cobra Effect: https://en.wikipedia.org/wiki/Cobra_effect

niftich• 8 years ago

Gamification increases as understanding improves.

Information is exploited.

stcredzero15• 8 years ago

[Targeting Intensifies]

coredog64• 8 years ago

Before the VW diesel cheating scandal, it was alleged that US manufacturers were taking advantage of the EPA's test cycle procedure to "tune" ECM values when the dome light was active.

More than that, CAFE standards have become the ceiling for fuel economy rather than the envisioned floor. Manufacturers game the system by classifying certain cars as SUVs, claiming extra credit for the ability to run E85, or practically giving away efficient but undesirable vehicle configurations in order to just meet the target and avoid the gas guzzler tax.

megablast• 8 years ago

Sweet. The measure is profit then.

Angostura• 8 years ago

Serious question - it may no longer be a good measure, but does that necessarily mean it is a poor target?

Assuming targets are necessary, do they not have to based on done measure?

biastoact• 8 years ago

Without dates and metrics you can’t measure progress. Like mile markers along the road (back before GPS / Smartphones). But without continuous, high trust, collaborative conversation you can’t actually make progress. That’s the gasoline powering you down the road. The goal should be Disneyland (shouldn’t it always be Disneyland?), not the next mile marker.

notEvenOnce• 8 years ago

(cough! cough! "scrum")

(cough! cough! "agile")

(cough! cough! "daily stand-ups")

jingleheimer• 8 years ago

I always felt that agile processes were a means to give non-answers to performance measurement in order to allay the consequences of that measurement and allow developers to get on with the business of making software.

How big is the task; 10 story points

How long will that take; depends on our velocity

Can it be done by this date; we're not allowed to plan that far ahead. You can put it in the 'icebox'. We might get to it eventually.

pmwhite2• 8 years ago

Yes, definitely true.

If (1) everyone is highly effective in getting tasks done, and (2) working on tasks that are reasonably well prioritized by both technical risks and business value, why exactly do you need to know more?

There are legitimate reasons that a business may need a date. But making sure people are working harder than hard by beating them over the head with a date coughed out by a gantt chart is not one of the legitimate reasons.

ozim• 8 years ago

Kind of we started optimizing for points and started to be smart about it. Because that what people above wanted. What happened we started communicating to PO more often, like: you want more points burned take this story out, close this and create an improvement.

Of course that was not childish silly sneaking and turned out we have as a team a lot better partnership with business. We deliver what is needed and can be finished.

I understand not everyone has luck to end up in such environment but it confirms there is no silver bullet. Only having right people is real silver bullet.

logicallee• 8 years ago

Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?

bennofs• 8 years ago

You could get a long life expectancy but with increasing pain as you get older. It would be a long live but not healthy. See also https://en.m.wikipedia.org/wiki/Compression_of_morbidity

scarmig2• 8 years ago

That's easy, though. If you're going for life expectancy, then there's plenty of things you can do.

1) Exclude deaths that'd hurt your life expectancy. For instance, by under counting infant mortality. E.g. does a child who dies during birth count against your life expectancy? Arguments can go both ways, but if your hospital's funding depends on having good life expectancy numbers in your community... well, there you go. IIRC this is one factor that helps explain Cuba's surprisingly high life expectancy numbers, though it doesn't account for most of it.

2) Overtreat end of life issues. Instead of focusing on quality of life and general health throughout the lifespan, put all your funding into heroic interventions that let 90 year olds live on artificial respiration for months. This is what the United States does.

3) Systematically deceive. Have a significant homeless population? Rural population that's deeply unhealthy? Make sure you put a ton of resources into measuring outcomes for your urban population with relatively good health access, and the peasants you miss are just a sign of not getting enough grant money from the WHO. Governments that put resources into getting measurements of the most marginal populations are penalized, while those that deprioritize studies of those marginal populations would have better numbers for their marketing brochures. Even though we'd expect the former to have actual better outcomes than the latter!

LeifCarrotson• 8 years ago

You can get a life expectancy of 100 by putting all your senior citizens (who are much more susceptible to infections, broken bones, etc) on IV nutrition in iron lungs in an intensive care wing. That would be awful.

Better to measure quality-adjusted life-years, in a way that accounts for the fact that it's sometimes better to take a risk of death with a possibility of full recovery over a guarantee of lost function.

I can't possibly enumerate all possible metrics and their effects, but I bet we could do pretty well at imagining faults for anything that can be constructed.

mannykannot• 8 years ago

Yes; the real problem is when a proxy is substituted for what is really wanted - for example, when lines of code written is taken for a measure of how much usefulness has been created, or the percentage graduated is taken as a measure of the effectiveness of education.

Proxies are not intrinsically bad, either, but they have to be difficult to achieve without also achieving what you really want - when this is not the case, making the proxy measurement a target motivates people to seek workarounds.

Life expectancy might be a proxy, but it is a pretty good one.

triviatise2• 8 years ago

Life expectancy is actually a really great example of the opposite. People routinely claim our health system is bad compared to europe because our average life expectancy is lower.

It turns out african americans skew our metrics because they have

1) high infant mortality 2) high murder rates of young men

The high infant mortality is irrespective of wealth and has a disproportionate impact on average life expectancy because the death of a 0 age person has a huge impact on the average. When you look at the rest of the population, our whites are on par with europe and our asians are on par with asian countries.

One way this is gamed is that some countries have different standards for reporting infant mortality.

<<The infant mortality rate is defined as the number of deaths of children under one year of age, expressed per 1 000 live births. Some of the international variation in infant mortality rates is due to variations among countries in registering practices for premature infants. The United States and Canada are two countries which register a much higher proportion of babies weighing less than 500g, with low odds of survival, resulting in higher reported infant mortality. In Europe, several countries apply a minimum gestational age of 22 weeks (or a birth weight threshold of 500g) for babies to be registered as live births. This indicator is measured in terms of deaths per 1 000 live births.>>

In the US it is gamed in reverse in the sense that the average hides the true problem is with subpopulations and not the entire US medical system as a whole. Our (gamed) poor performance in this statistic is used as a measure to support moving to a substantially different system.

scarmig2• 8 years ago

It's not like Europe doesn't have marginalized populations. Your argument is basically "if you exclude the parts of the US population that are worst treated by our healthcare system, then ours is the best!" There is something meaningful in comparing the "median" or "modal" health experience in different countries, but "we treat black people badly compared to white people in the USA so you have to only look at white statistics" is a terrible defense of the system.

Your point about infant mortality statistics being gamed is on point, though (and I pointed it out in a sibling commment).

davidmanheim2• 8 years ago

This is probably where I should mention my two longer articles that discuss this in depth; https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-... https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspec...

As well as my more recent paper categorizing the different ways it occurs; https://arxiv.org/abs/1803.04585

Symmetry• 8 years ago

That was a much better take on Goodhart's Law than the Wikipedia article. It's easy to give up and say "Measurement is impossible!" but really we have to balance different factors and measurements are more dangerous at some times than at others.

rsync• 8 years ago

Lots of content at ribbonfarm is very, very good ... highly recommended ...

https://www.ribbonfarm.com/2010/11/05/ancient-rivers-of-mone...

https://www.ribbonfarm.com/2009/03/02/fools-and-their-money-...

The Return of the Barbarian

Can Hydras eat unknown unknowns for lunch

The Premium Mediocre Life of Maya Millennial

waltherg• 8 years ago

Since you’re here, what does the following property stand for in your second paper?

s ∈ AifM(s)/gec

defen• 8 years ago

Not OP but that should probably render as: s ∈ A if M(s) >= c

So: s (system state) is a member of A (permissible region) if M(s) (proxy measure of s since we can't directly observe it) is greater than or equal to c (threshold)

eboyjr• 8 years ago

Reducing a situation to a single measurement hides the dimensionality. I think intuition accounts for more kinds of measurement albeit less accurate or describable.

bonniemuffin• 8 years ago

Thanks for this great work-- I spend a lot of time thinking about this stuff and enjoy reading other people's takes on how to deal with the impossibility of finding the One True Metric to rule them all.

To keep myself sharp looking out for the perverse incentives, I have to remind myself: don't be a slave to the metrics. We're data-informed, not data-driven. Data informs; people drive.

robertk• 8 years ago

Great stuff. Say hi to Scott!

Are you both at MIRI now?

njarboe4• 8 years ago

The breaking of this law when one uses profit as the measure of a successful company or business is why I think that a well set up market system actually works. People and companies can set up making a profit as their target and it is still a good measure of success, because it is actually what you want.

The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not. A hard thing to get right but forcing people to inform you of the ingredients of food for sale, is an example of a good regulation I think.

retbull• 8 years ago

Unless the highest profit thing is to poison a town full of people. Which has happened when profit is what you measure as success.

njarboe4• 8 years ago

Correct. "The key for a good society is regulating the places where making max profits can induce bad behavior and leaving alone where it does not." Start of the second paragraph.

jetrink• 8 years ago

The implication behind Goodhart's Law is that people are good at gaming performance measures. You can game profit measures too[1], but it's difficult and there are strong incentives not to. Another counter-example that doesn't involve fraud is Amazon, which was unprofitable for a long time.

1. https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal

dfee2• 8 years ago

“Profits” in your context being imagined not real.

The gaming was done on the accounting side (revenue recognition) of course.

nkoren3• 8 years ago

Even disregarding the potential for negative externalities etc., profit alone can still be a terrible KPI.

I worked for a large engineering consultancy that was very profit motivated. They figured out a brilliant way to maximise profit: get rid of business development! Only pay for work when clients are paying you to do it. This was GREAT for quarterly profit margins, at the expense of the project pipeline.

As the pipeline dried up, they figured out a new way to maximize profit: get rid of engineers! The customers are big, stupid organisations, and it'll take them a long time to notice. You can totally maximise quarterly profit margins until they do.

And that's how you can simultaneously be highly profitable and in a corporate death spiral.

Moral of the story: reality is a fundamentally complex place. You need multiple KPIs to get a handle on it -- and ideally those KPIs should be mutually contradictory enough to force you to engage with that complexity head-on, rather than boiling things down to game-able metrics which ultimately will defeat themselves because they are such a poor proxy for reality.

njarboe4• 8 years ago

I would say the complexity is in balancing profit over time. Here is your contradictory force. In the end that large engineering consultancy was making zero profit and was worth nothing. Whether running a business into the ground will give you more overall profit than just selling it and putting the money into government bonds is something one can consider, but in your case was the consultancy owned by the people making these short-term profit maximizing decisions? The agent/principal problem for companies has not been solved.

It is probably not a coincidence that the top 6 US companies by market cap (Apple, Amazon, Alphabet, Microsoft, Facebook, Berkshire Hathaway) are still run by their founders or were run by their founders for most of the companies existence before a recent step-down (Microsoft) or death (Apple). Most public companies aren't and they struggle to grow or even survive over time.

pmichaud• 8 years ago

I think you're not trolling, but I'm not sure.

The first thing anyone smart does when they make a metric is think about what rules or other measures have to be in place to prevent obvious gaming of the metric to the detriment of the real goal.

When this is done at scale to markets it's called "regulations."

Markets are not even a little Goodhart proof.

legulere• 8 years ago

Many companies don't optimize for profit but optimize for market share and try to be around net zero, the best example being Amazon.

dfee2• 8 years ago

Another article on the front page decries Harvard’s holistic approach to admission - with arguments that admission should be based on a single measure of academic performance (like standardized test scores, GPA, etc.).

lostcolony• 8 years ago

Yes, but holistic approaches have -also- become a target. Think of the over-achieving high schoolers doing as many extra-curriculars as possible just for their college application.

stevenwoo• 8 years ago

The most interesting approach I have seen as an alternative to current ones is the one that suggests emphasizing the student body of the school - leading parents to self integrate their K-12 schools on racial and socioeconomic factors. This would be unassailable under the current legal attacks used to fight against affirmative action.

https://ylpr.yale.edu/shifting-scope-how-taking-school-demog...

gojomo• 8 years ago

And thus for performance measures, "security through obscurity" can make sense.

triviatise2• 8 years ago

We hold people accountable for actions instead of outcomes, since outcomes (even profit) can be gamed at the overall detriment of the organization.

We talk about the outcome we want, look at the levers that will get the outcome, and then agree to the actions that we are going to take.

If the actions dont result in the forecasted outcome then the underlying assumptions are wrong. The assumptions are wrong if 1) people arent doing the tasks properly 2) the relationship between the task and the outcome are different than what we thought. If the issue is 1, we provide training. If the issue is 2, we use the revised assumption for future forecasts.

We dont use the word target anymore, we use the word forecast.

fallingfrog• 8 years ago

That's why you know your company is in trouble when shareholder becomes the customer and the stock price becomes the product..

dqpb• 8 years ago

This is one reason why the grading system in schools is absurd and incompetent.

joker3• 8 years ago

People respond more or less rationally to their incentives. Who knew?

stcredzero15• 8 years ago

The same principle applies to attributes and qualities which mark successful entrepreneurs. The same principle applies to attributes and qualities which mark good developers. The same principle applies to attributes and qualities which mark good employees. Good people. Good politicians. Activists. Good pundits. Intellectuals. Top tier school degrees.

Hacker News karma.

Shouldn't we all be looking out for the "targeting" and be trying to find the next measure?

tlarkworthy• 8 years ago

This is also demonstrated quite well in machine learning with the test set. The target is test error, but you don't let algorithms optimize that directly and instead they optimize against training error. Goodhart's law is a manifestation of over-fitting in social systems.

em3rgent0rdr• 8 years ago

Reminds me of all those computer architecture papers that optimize for IPC to the detriment of overall performance. Or the CPU frequency wars which led to ever deeper pipelines which were completely obliterated with branch mispredicts. Or the current trend of increasing core count without actually being able to do any useful work on them...

Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

Related Stories