How Good Are FiveThirtyEight Forecasts?

https://projects.fivethirtyeight.com/checking-our-work/

by jacobedawson • 7 years ago

34 48 7 years ago

projects.fivethirtyeight.com

PaulHoule• 7 years ago

This 'calibration' is a missing link for machine learning.

A prediction is much more valuable if you know how good of a prediction it is than if you don't.

paulgb7• 7 years ago

I'd caution about putting too much stock in calibration, though. It's worth noting that you can achieve perfect calibration just by giving each of n candidates 1/n probability of winning.

robto2• 7 years ago

I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time.

[0]https://en.wikipedia.org/wiki/Vermin_Supreme

empath75• 7 years ago

All of the people complaining about how Five Thirty Eight is a fraud because they "predicted" that Hillary Clinton would win should read this. They gave Trump a significant chance of winning, more than almost any other poll aggregator did. In fact, before the election, Nate made a post outlining all the reasons that Trump could win and gave him a 36% chance of doing so. They literally said that trump would win based on polls like that 36% of the time, so it shouldn't have been that shocking that he did.

https://fivethirtyeight.com/features/election-update-dont-ig...

You might say that a forecast that's so ambivalent isn't useful, but if someone told you that there's a 36% chance of rain, you'd plan your beach trip accordingly. I don't see why predicting a 36% chance of the underdog winning the presidential race is any less informative.

keymone• 7 years ago

can downvoters elaborate on why they disagree?

ghaff2• 7 years ago

It's probably a confusing the map with the territory sort of thing. FiveThirtyEight predicted Clinton would probably win and didn't. Many people are (understandably) upset she didn't win and they're lashing out at FiveThirtyEight for having given them what turned out to be false hope.

bryanlarsen• 7 years ago

538 gave Trump and the Cavaliers similar odds in 2016, something I pointed out to a bunch of people after the Cav's won but before Trump did. Golden State were substantial favorites, yet the Cav's beat them. Nobody is blaming 538 for calling GSW the favorites, few people think their probabilities were wrong. Why is it so different for the presidential election?

objektif9• 7 years ago

They obviously got 16 very wrong. However, what is even more concerning is that Nate lately became a political hack, doing the ugly work for the democratic establishment. Just look at his twitter you can clearly see it. So I would take any "data science" he does with a grain of salt.

Edit: An example of his tweets:

"It's probably worth noting that while this group, Justice Democrats, calls Biden "out-of-touch" with the "center of energy" in the Democratic Party, only 26 of the 79 candidates it endorsed last year won their primaries, and only 7 of those went on to win the general election."

Why didn’t he add the statistic that Biden already ran 2 times in the past and won 0 times.

paulgb7• 7 years ago

I have seen his Twitter feed, do you have any specific examples? Obviously he's talking a lot about Democrats right now more than Republicans because Trump is the presumed Republican nominee, but covering the race isn't "doing ugly work for the Democratic establishment."

objektif9• 7 years ago

Please explain to me what is the point of such a tweet.

pwinnski3• 7 years ago

They did not, he has not, and one should not any more than one always should.

538's forecast was easily the most accurate for November 2016, by a decent margin. The majority of critics of 538 are "sore winners" who delight in their side having won despite being given a less than 50% chance, but as Silver pointed out at the time: his chances were similar to those in Russian Roulette, and most people consider that too risky to play.

paulgb7• 7 years ago

Right. It's like saying "Ha! You were wrong! You said there was a 83% chance I wouldn't roll a six, and I did"

protomyth• 7 years ago

538's forecast was easily the most accurate for November 2016, by a decent margin.

No, Investor’s Business Daily was the most accurate.

objektif9• 7 years ago

If thats your understanding of making a prediction then he was accurate by your standards. You clearly did not watch the election night predictions. NYT live model was miles better than 538s.

https://fivethirtyeight.com/features/how-i-acted-like-a-pund...

cljs-js-eval2• 7 years ago

This gets to the core of Taleb's beef with FiveThirtyEight - what political forecasts are calibrated? The ones that were forecasted 6 months out, or the last forecast before the end of the election?

They may only be calibrated when enough of the facts are in to make predicting easier. Still useful, but not calibrated in the way the general public might think.

Symmetry2• 7 years ago

If you dive a bit more they look at each forcast at each time interval for which it is made. So they include the 12 month out one, the 11 month out one, etc when assessing calibration.

cljs-js-eval2• 7 years ago

Ah, that's very useful to know. I take my statement back then, this is better calibration than I thought.

joker3• 7 years ago

This is good in the sense that having poorly calibrated forecasts is bad, but it's not quite as strong as you might want it to be. Imagine that you're tasked with predicting the next state of a traffic light that just cycles through green, yellow and red over and over again. If at every step you forecast that there's a 33% chance of each color, you have a very well-calibrated forecaster, but it's clear that you can do better.

FiveThirtyEight probably actually is doing better, but I suspect that they're not choosing to highlight those stronger metrics because they're complicated to explain.

jsnell• 7 years ago

But isn't that exactly what the last third of the article discusses?

petraeus• 7 years ago

Can anyone explain how 538 got the last election so wrong?

Its quite obvious in hindsight their model is flawed, but Im not understanding how is it flawed.

KeenFox• 7 years ago

Did they? As far as I can recall 538 has always been clear that they deal in probabilities, not binary predictions. They gave him a small chance of winning for a reason, and it turned out chaos favored one of the low odds scenarios.

fourmajor• 7 years ago

If you're referring to either the midterms or the 2016 presidential election, they didn't get either particularly wrong.

How Good Are FiveThirtyEight Forecasts?

Related Stories