Nassim Taleb vs. Nate Silver: who is right about election forecasting?

http://quant.am/statistics/2020/10/11/taleb-silver-feud/

by probe • 6 years ago

93 169 6 years ago

handmodel• 6 years ago

This article is mischaracterizing Silver's model. The author writes:

"if the election were to happen today, what is the probability of each candidate winning?"

This is not at all what Silver's model does. Silver writes and makes clear that there is more variance in the model a few months out than a few days out. (In other words - Biden may be 90% now but four months ago if the polls were identical he would be at less than 90% since there would be more time for voter change and real world events.)

I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast.

I may say that four months from now the chance it rains in Los Angeles is 2%. However, the day before with more info I may be able to say it will rain 90% of the time. There isn't anything wrong with this, even though Taleb seems to suggest this wrong (from my read)

smallnamespace• 6 years ago

Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk.

Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.

pochamago• 6 years ago

I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.

coliveira2• 6 years ago

The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.

edmundsauto2• 6 years ago

This is simply untrue. The 538 models are calibrated against a lot of previous real-world outcomes.

repsilat• 6 years ago

Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious.

A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.

pwinnski2• 6 years ago

We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors.

Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.

smeeth3• 6 years ago

Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle.

I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.

kadoban• 6 years ago

That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called.

I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.

dragonwriter10• 6 years ago

Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle.

False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)

vincentmarle• 6 years ago

I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast.

"Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day.

[1] https://arxiv.org/pdf/1703.06351.pdf

kelnos2• 6 years ago

If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point.

But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.

Tomminn4• 6 years ago

"if the election were to happen today, what is the probability of each candidate winning?"

This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1].

For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for.

[1] https://news.ycombinator.com/item?id=24164814

ashtonkem8• 6 years ago

This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.

raverbashing2• 6 years ago

That's an interest take on it

I'd say the way of presenting results as a "sum to 100%" fails to present uncertainty in an intuitive way

moultano3• 6 years ago

It was not. You are mistaken about the history.

asdfasgasdgasdg• 6 years ago

The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events.

Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident.

As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.

Tomminn4• 6 years ago

Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high.

If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.

FandangoRanger• 6 years ago

It's impossible to believe any forecast, nobody fessed up after 2016 nor did they explain why their models are wrong or attempt to update them. You're less than 24 hours away from the permanent death of the political polling and forecasting industry in the USA.

Nate's already coping really hard on Twitter.

ponker• 6 years ago

Well pre-2016 there was the assumption that the American electorate basically is capable of self-governing. This assumption has been invalidated so while the polling industry hasn’t gone away it’s now more like the sports forecasting industry, where people are speculating on an essentially theatrical performance, than something substantively driving good governance.

jjeaff• 6 years ago

That's not true, there have been countless articles about why polling went wrong last time.

Also, polling wasn't that wrong. Due to sampling bias, several swing states were off by as much as 5-6 percentage points. But national polling bad Hillary winning by something like 4% in the popular vote.

Instead, she won the popular vote by 3%.

JumpCrisscross2• 6 years ago

several swing states were off by as much as 5-6 percentage points

Most state polls have margins of error ranging from 3 to 6 points. A 3-point MoE state poll is a really expensive poll.

gdulli• 6 years ago

Believing that occurrence of an event with a forecasted 30% chance (or 2% chance) of occurring invalidates a model is so fundamentally a misunderstanding of the topic.

geofft• 6 years ago

My DM: "You're going to try to hit this guy with your bare hands, even though you have +0 strength?"

Me: "Does a 17 hit?"

My DM: "His armor class was 16. It was supposed to be impossible for you to hit him with a d20. I have just witnessed the permanent death of the dungeon mastering industry in Faerûn. I will now cope on Twitter, goodbye."

TameAntelope• 6 years ago

Can you elaborate a bit on what you mean by, "nobody fessed up after 2016"?

From my understanding there wasn't the "big miss" that got called out in media, and to the extent that the rust belt/upper midwest had state level polling variance, it was determined to be due to the lack of accounting for education as part of polling.

I guess if it doesn't rain and the weatherperson says it should have rained, do we give up on even trying?

techwizrd• 6 years ago

I don't believe this is accurate. NYT[0], HuffPo[1], Princeton Election Consortium[2], 538[3,4], and others did analyze their model's failures and update their models and methodologies in response.

0: http://www.nytimes.com/2016/11/16/upshot/presidential-foreca...

1: http://www.huffingtonpost.com/entry/pollster-forecast-donald...

2: http://election.princeton.edu/2016/11/09/aftermath/

3: https://fivethirtyeight.com/features/why-fivethirtyeight-gav...

4: https://fivethirtyeight.com/features/how-i-acted-like-a-pund...

MR4D2• 6 years ago

FTA, “ I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?”

If that’s true, then Nate Silver just does poll analysis, not predictions. Predictions should be on the day of the election, not the day you’re reading their blog post.

In other words, Taleb is right.

throwawaygh6• 6 years ago

It's not true.

MR4D2• 6 years ago

What’s not true?

throwawaygh6• 6 years ago

the election were to happen today, what is the probability of each candidate winning?

He's explicitly not doing this.

Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?"

Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori.

The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.

It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.

coliveira2• 6 years ago

Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.

seibelj• 6 years ago

Totally agree, he comes off as a hack to me. But he’s famous and I’m not, so shrug

sillysaurusx• 6 years ago

The same is true of pg, but his discoveries seem obvious only in hindsight.

(I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)

raverbashing2• 6 years ago

Frequentists miss the obvious stuff all the time, Taleb is not smarter, it's other people who are tied in dogmatic thinking

rramach2• 6 years ago

My favorite story when I hear someone dismiss an argument saying 'that is obvious':

Lazarsfeld was writing about “The American Soldier”, a recently published study of over 600,000 servicemen, conducted by the research branch of the war department during and immediately after the second world war. To make his point, Lazarsfeld listed six findings that he claimed were representative of the report. Take number two: '“Men from rural backgrounds were usually in better spirits during their Army life than soldiers from city backgrounds.”

“Aha,” says Lazarsfeld’s imagined reader, “that makes perfect sense. Rural men in the 1940s were accustomed to harsher living standards and more physical labour than city men, so naturally they had an easier time adjusting. Why did we need such a vast and expensive study to tell me what I already knew?” Why indeed.

But Lazarsfeld then reveals the truth: all six of the “findings” were in fact the exact opposite of what the study found. It was city men, not rural men, who were happier during their army life. Of course, had the reader been told the real answers in the first place, they could just as easily have reconciled them with other things they already thought they knew: “City men are more used to working in crowded conditions and in corporations, with chains of command, strict standards of clothing, etiquette, and so on. That’s obvious!” But this is exactly the point Lazarsfeld was making. When every answer and its opposite appears equally obvious then, as he put it, “something is wrong with the entire argument of ‘obviousness'”

More here: https://www.newscientist.com/article/mg21128210-100-the-huma...

partiallypro4• 6 years ago

I don't know much about Taleb, but one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out. Then when they don't work out, instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models. His sports models are -especially- bad; no better than a coin flip in some instances. I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego, which always makes me take him with a grain of salt. When it comes to politics his former work with Obama's campaign and DailyKos also make me question if he can be truly unbiased in his predictions, but that's a separate issue.

throwawaygh6• 6 years ago

> The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.*
...he's very smug...

Right, it's just dueling personalities.

Robotbeat2• 6 years ago

Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. See: https://mashable.com/2012/11/07/nate-silver-wins/

He's not perfect (for instance, I think calling the 2016 model too certain early on is valid and I think some other people have basically equivalently-good models, but are less well known), but if confidence is smugness, then he has a right to be.

cortesoft4• 6 years ago

What is your criteria for saying "his models don't work out"? If he says "candidate A has a 90% chance of winning", and then they don't win, that doesn't mean his model was wrong. In the same vain, if the 90% candidate won, it also doesn't mean his model was correct either.

You have to look at a lot of his predictions, and see if the percentages match up.

When you say his models don't work out, are you saying you have done that analysis and found that his percentages don't match the results? If so, I would love to see that analysis.

kelnos2• 6 years ago

I frankly don't care if someone is smug or not; I want to know who builds the best models. If Silver's models are often wrong, who would you recommend we look into instead?

wutbrodo• 6 years ago

one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out.

What does it mean for a probabilistic prediction to "not work out"? If I tell you that the odds of flipping 2 heads in a row with a fair coin is only 25%, and it happens, did my model "not work out"?

instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models.

I really don't intend this in any sort of rude way, but I think you might be interpreting his response as dismissive because it's accurate...

It's difficult to say a model "doesn't work out" based on a single event, especially when it puts a significant probability on the less-likely option. For example, 538's 2016 forecast gave Trump a roughly 1 in 3 chance of winning: that's _higher_ than the odds of flipping 2 heads in a row.

Measuring the quality of a model is more complicated: one way would be applying the model to the same outcome multiple times, but this isn't possible for single-event forecasts. Another way to see how calibrated your predictions are is to aggregate multiple predictions and see how often they line up with reality: an event predicted to occur with X% probability should occur X% of the time.

Lucky for us, 538 has done exactly this analysis[1]! Naturally, it's internal, so take it with a grain of salt, but it looks like their predictions are fairly well-calibrated.

[1] https://projects.fivethirtyeight.com/checking-our-work/

dragonwriter10• 6 years ago

he's very smug about this models but they often enough don't work out.

Taking all of the Presidential forecasts together, there's too few to really generalize about that. Taking the down allot results though, the odds have been pretty close to what he's forecast.

So I'm not sure what your basis is for saying they “often enough don't work out”.

His sports models are -especially- bad

Maybe, but his political models are especially good.

I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego

He's never really come off that way to me, not that whether or not he has a big ego has any bearing on the quality of his models.

tempestn2• 6 years ago

I came here to say exactly the same thing. Silver and 538 are explicit about not doing this. Their models do include the uncertainty of future events, and do adjust closer to 50/50 given the same polls the further from the election it is. That's why Biden's odds have been ticking upwards slightly over the past couple weeks despite polls remaining pretty stable.

So in theory, the forecast should be exactly what it appears: the percentage of the time the candidate would be expected to win given the situation they are in at the time of the forecast. So if on Oct. 14 it gives Biden a 72% chance, it means that a candidate in Biden's position, with his polling numbers, and the economic conditions and everything else that is the case on Oct. 14, would go on to win from there 72% of the time.

I agree that it appears the volatility of the 2016 forecast was larger than you would expect given a correct model behaving that way. I expect that uncertainty is better expressed in the 2020 model. However it's certainly conceivable for such a model to swing correctly. IE, there's nothing impossible about a candidate having a 90% chance of winning on X date, then having circumstances change such that they only have a 50% chance at Y date. (In fact, if you ignore the discontinuity on election day, at least 10% of the time their odds would have to pass through 50% at some point, as they would go on to lose. And of course there would also be scenarios when it would dip to that level or below and yet still win.)

stopachka12• 6 years ago

I think Taleb’s point is a bit more damning — implying the software does not do what it says it would.

If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election.

This doesn’t mean that all prediction suffers from this effect, and you have to be skeptical everywhere. I don’t think that’s a fair characterization. Taleb has won by mispriced options. His main statement is that the true probability of this particular event is much closer to 50%

cyphar4• 6 years ago

That's what confidence intervals are for. You cannot condense both your expectation value and uncertainty in a single percentage (it is not correct to say that the probability of an outcome is 50% if 80% of likely outcomes result in one candidate winning). Otherwise you'll end up with every 5-week weather forecast saying there's a 50% chance of rain.

dragonwriter10• 6 years ago

If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election.

Taleb is, simply, wrong.

The difference from 100%/0% is the uncertainty. If new information confirms rather than contradicts prior information, that uncertainty goes down, if it contradicts it, the uncertainty goes up. You expect most of the time the general trend over time to be declining uncertainty, but when there is a period of continuing new information at odds with the prior information, you get a period of increasing uncertainty.

So if you had a very large number of Presidential election cycles with the same model, you'd expect most of them to generally trend toward greater certainty over time, but you'd expect a few of them to have extended periods of declining certainty.

Does Silver's Presidential model work that way? It's hard to tell. One, because there aren't a lot of cycles to look at, and because they aren't the exact same model, is it's not quite what you'd want to look at it to assess that. Well, 2012 and 2020 have had very much the general shape of growing certainty you'd expect, while 2016 didn't.

Tomminn4• 6 years ago

If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal.

This is Taleb's point, and it is solid as a rock.

moultano3• 6 years ago

But he provides no evidence that this is the case. The 2016 election was extremely volatile due to what actually happened during it, not the modelling. There were several bombshell press events all throughout the campaign that dramatically shifted polls.

throwawaygh6• 6 years ago

If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal.

That's not how poll averaging works. Or, at least, that's not how poll averaging works in the 538 model.

egonschiele• 6 years ago

Incredibly click-baity title and incendiary first paragraph were hard to get past.

fortenforge• 6 years ago

I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?

As others have pointed out, this is definitely not what Nate's model was doing. In fact, he did have a version of the model that did exactly this, the so called "Now-Cast" and as expected it was even more volatile than the real model:

https://projects.fivethirtyeight.com/2016-election-forecast/...

dragonwriter10• 6 years ago

Right, the main model intends, in fact, to be predicting the probability of various outcomes when the election is actually held, that's why there is accounting for non-poll fundamentals (economy and other things) that are observed to influence poll changes over time that phase out as time runs out for them to move the polls.

travisgriggs• 6 years ago

So... Taleb is a like purist programmer (pick your favorite paradigm) and Silver is more like a pragmatist programmer that uses a "hybrid" language model?

There's values for both types in programmerdom, is there not the same for both in the world of statisticians?

stopachka12• 6 years ago

“if I tell you an event has a 0% chance of occurring, I cannot change my mind and tell you tomorrow it now has a 50% chance of occurring. Otherwise I shouldn’t have told you it has a 0% chance in the first place”

I think this is the key to Taleb’s argument, and it’s pretty damning for Mr Siver.

If there’s high uncertainty, you can’t say whether someone’s probability to win is low. The uncertainty widens the probability.

For example, Say I told you there’s a 2% chance it will rain Monday next year in New York - This would _have_ to be a bogus prediction. So much could happen until then. I have _high_ uncertainty. There’s no way that uncertainty is accounted for with 2%

ashtonkem8• 6 years ago

Yeah, except both of them are modeling how people will vote, and people change their minds.

Saying that one should’ve held a constant chance of a candidate winning no matter what the polls do is frankly insane. This stance basically requires that you pretend that the outcome is determined and that voters opinions either don’t matter or don’t change. It’s utter nonsense, especially since we know that in 2016 undecided voters broke late for Trump. If your model doesn’t react to that, what are you even modeling?

stopachka12• 6 years ago

Taleb’s point is not that your prediction should not change.

His point is that if it does change, and frequently at that, the purported 90% chance at t(n) is bogus. That prediction needed to account for the volatility

cortesoft4• 6 years ago

The prediction should have more data to it about variance, but it isn't necessarily wrong.

We have good data about rain, I am sure we could find out what percentage of the time it rains in New York on a given day during that time of the year. There might be high variance, but that doesn't mean it is wrong to say there is a 2% chance it rains on a particular Monday.

This is how insurance works, how sports betting works, how option pricing works. Just because something has high variance doesn't mean we can't give probabilities that work out over a large number of samples.

stopachka12• 6 years ago

Let’s think about the rain analogy a bit:

Imagine someone offered you this as a bet. They’ll pay you 3 dollars if it rains Monday next year. Otherwise you pay them 90 dollars.

Now, it’s pretty intuitively clear that at a year interval, there’s way too much uncertainty to take this bet. If you accounted for it, you would approach something like the average chance of raining on any particular day in November in New York.

You are right that with enough data, you would be able to price this correctly.

Taleb’s point is that Nate is not doing that, by demonstration: since Nate’s models oscillated so much, it demonstrates that he did not account for it.

awb12• 6 years ago

Say I told you there’s a 2% chance it will rain Monday next year in New York - This would _have_ to be a bogus prediction.

Yet betting odds exist, because we can predict things far into the future accounting for variance and risk by looking at history. If it's never rained on 2% of Monday's in NYC then a 2% prediction would make sense.

From the article: > Premise 1: If you give a probability, you must be willing to wager on it

The best way to settle this is have Talib bet on Trump to win at 50% odds (or whatever he thinks is appropriate) every day up until the election and Silver to bet using his odds (currently around 10%). May the more accurate prediction prevail. Too late for this election, but there will be plenty more elections in the future.

stopachka12• 6 years ago

You are indeed right.

Look at Nate’s current prediction though: 89% that Biden will win.

I highly doubt anyone would take betting odds of 89% Biden. This is actually one of Taleb’s main critiques

ramraj07• 6 years ago

Your argument doesn't hold water at all. Today's blurb in the forecast page - "First, the forecast is now totally polls-based; that is, any advantage our forecast gave Trump for the economy or for being an incumbent is no longer factored in. Second, the uncertainty around how much the polls will change between today and Election Day is also no longer an issue." - that means Nate Silver is explicitly saying his forecast is always trying to predict what is going to happen on November 3.

Heres an alternative hypothesis that involves no fancy math just anthropology - Taleb is right that the stats don't make sense, but Nate is also not an idiot - it's just in his financial best interests to make a website that can throw out confidence inducing pseudonumbers that will make people like us visit it every few hours for months at a time every two years. Perhaps he knows perfectly well the farce he's pulling (at least at a fundamental level though he might have convinced himself otherwise as academics always do) but just plays the game to make sure he's relevant. And thus, we are all the idiots.

rramach2• 6 years ago

Nice summary of the Nassim-Nate debate. A formal analysis/discussion of election forecasting is in this paper: http://www.stat.columbia.edu/~gelman/research/published/jdm2...

Edward says that Nate's prediction should be interpreted as: “If nothing else changes between now and the election, Joe Biden has a 85% chance of winning.” (Silver’s argument) But the problem is this prediction is not testable as election only occurs once at the end. Thus, one should use Nate's early predictions as pure entertainment.

Nate's final predictions show that they are well-calibrated across all of his political predictions but it is hard to estimate how accurate his model predictions are just for presidential elections given he has predicted only a few so far (unless one assumes all elections have similar uncertainty, which is clearly not true).

baron_harkonnen• 6 years ago

I've recently come to believe that Taleb really doesn't understand probability. I know this sounds like some insane flippant remark, but I saw a video where he talked about the election forecast as a martingale bounded between 0 and 1 and then made some claims about variance with these bounds to back up his claim against Silver.

For his claim to make sense you would have to view probabilities as linear, which is not simply not how probabilities work. An event that has a 0.01 chance of happening is an order of magnitude more likely than one that has as 0.001 chance. However the different between and 0.5 and 0.501 is essentially negligible (hence probability is not linear because 0.001 does not mean the same everywhere).

I've generally given Taleb a pass as a bit of a crank that doesn't care about the philosophical interpretation of probability, but after seeing him misunderstand probability in such a major way I've started writing him off entirely.

edit: I should add I'm not particularly more convinced that Silver really understands all that much either, just that this is all two very loud people that don't know much. Neither should be taken seriously. This isn't Bradly Efron arguing with Andrew Gelman.

inimino2• 6 years ago

Based on what I've read of Taleb, if you think he doesn't understand some patently obvious facts about probability, it's going to take more than saying that you watched a video and didn't think it made sense for me to give your claim any weight.

amw-zero4• 6 years ago

There is a very clear flaw in your mathematical reasoning here, let's walk through it.

Consider your second probability comparison: the difference between a probability of 0.500 and 0.501. You have this correct, the difference is 0.001.

Now reconsider your first statement: that a probability of 0.010 is an order of magnitude more likely than a probability of 0.001. This is true, but is completely irrelevant because it is a completely different comparison. A more appropriate comparison would be to compare probabilities of 0.010 and 0.011. The difference of these is again 0.001.

You compared proverbial apples to oranges, and you cannot trust the conclusions you make from invalid comparisons. A difference in probability of 0.001 does mean the same thing no matter what the starting probability is. Probability is "linear" in this way.

Now that we know this to be true, I would encourage you to rethink your position on Nassim Taleb. He understands non-linearity quite well, in fact I would argue that his entire purpose on Earth (at least in his eyes) is to remind people about non-linearity as a concept. It comes across in all of his books and most of his public appearances.

On the matter of probability, I trust him very deeply. Whether or not I trust him is also inconsequential, because he has serious mathematical literacy and you can evaluate his propositions for yourself.

j7ake3• 6 years ago

The difference between probability 0.001 and 0 is also 0.001, but it's clearly not the same as a change in probability from 0.500 to 0.499.

I think the point is that it's the change in the log-odds that make differences comparable, not the change in the linear scale.

smeeth3• 6 years ago

Edit: as pointed out in the replies, I was wrong that the model was "explicitly" a now cast, and I would like future readers of my comment to know that without deleting things. I would still like to note that there was clearly a change in the 2020 model to increase variance as a function of time to the election when compared to the 2016 model. I think this is substantively the same.

Some of the top comments are making the same mistake, probably because they aren't aware of this twitter beef history (and I envy them), so I'm putting this in a separate comment.

These comments are all pointing out that Taleb's criticism is of something that Silver's models don't do:

the election were to happen today, what is the probability of each candidate winning?

What people seem to be forgetting, is that Silver's 2016 model explicitly did this. Taleb made his criticisms starting in 2018 (even published an options pricing paper on the subject) then Silver changed his model for the 2020 cycle so that it no longer was an "if the election were today" model.

I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here. Yes, I know Taleb is a colossal asshole people love to hate, but he's usually right about these sorts of things.

Here is a pretty fair and balanced take on it for those who would like to read more: https://towardsdatascience.com/why-you-should-care-about-the...

moultano3• 6 years ago

This isn't true. He had both a "nowcast" (if the election were held today) and a prediction of the final result. The only change in that regard between 2016 and now was to stop publishing the nowcast.

Edit: Even after your edit I don't think the claim is true. The probabilities in this election didn't change much because the campaign season was comparatively uneventful and the polling didn't change much, not because of a fundamental change in the model. 2016 had tons of bombshell events which is what caused the chaos in the prediction model.

ashtonkem8• 6 years ago

One of 538’s models did that in 2016, not all of them. They made it very clear that they did not recommend the “nowcast” because it tended to confuse people.

dragonwriter10• 6 years ago

What people seem to be forgetting, is that Silver's 2016 model explicitly did this.

Well, there was a sub model (the “nowcast”) in the 2016 (and, IIRC, earlier) forecasts that did this, but the headline model did not, it's always been a forecast of the election day results.

I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here.

The change you are describing literally did not happen. The headline model was never an “if the election were held today model”, so it cannot have changed from that to something else. Giving Taleb credit for that change is as sensible as giving Trump credit for ending the COVID-19 pandemic.

person_of_color• 6 years ago

Why don’t we use blockchain for accurate, anonymous polling? Rewards in BTC?

tmsh• 6 years ago

I think you can either:

  * make predictions with a range
  * or make predictions with a certainty coefficient (p value, etc.)

Predicting a single value without either is not a truly quantified "prediction."

It's weird now not getting that can lead to confusion. I agree with others in that those who don't communicate both values of information aren't being 100% clear.

shalmanese2• 6 years ago

I don’t understand why there’s so much attempted theory when we can just directly compare Nate’s model against a market, in this case, the presidential prediction market. What you found for 2016 was that the market both had about the same volatility over time as Nate’s model AND was more wrong about the end result, thus showing that the variances in the model were likely driven by actual uncertainty, not spurious over confidence.

aYsY4dDQ2NrcNzA• 6 years ago

I don’t trust the prediction markets.

https://freakonomics.com/2008/10/02/manipulation-in-politica...

http://web.archive.org/web/20090130093117/http://www.fivethi...

ashtonkem8• 6 years ago

And nor should you. But one can compare 538’s model against models made by other statisticians.

shalmanese2• 6 years ago

But the question that was being asked was whether there was an arbitrage opportunity available which, regardless of whether the market is manipulated or not, has an empirical answer, not merely a theoretical one.

elie222• 6 years ago

That's one case in 2008.

If volume and stakes are low then sure, they're not too reliable.

If you don't think prediction markets aren't great predictors you should make bets against that market. If you're a better predictor than the market you should make good money. In practice most people can't outguess the market. Some experts can and do make good money.

The reason markets are a good predictor because as soon as an expert sees the market make a bad prediction, they'll make a bet against the market. This in turn will move the price towards the true price. Assuming there are enough experts out there with large enough pockets (which is true for popular bets) then the market will quickly reflect the experts' view of events which is by definition very hard to beat (you have to be more expert than the experts).

thaumaturgy• 6 years ago

The disagreement between Taleb and Silver wheels on technical minutiae in statistics, but the discussions around them by lay-people seem to be about something else altogether.

The majority of 538's ad impressions come from people who start F5-mobbing the site during election time because they want to know before anyone else who's going to win. (A tiny few might be from people trying to figure out which races are worth donating towards.)

So that puts 538 into this awkward position where their audience is demanding something that 538 is unable to provide, but if they get too in-your-face in saying, "look, that's not how this works", then they stand to lose a lot of ad revenue during peak season.

That leads them to do very silly things, like bury this:

A 10 percent chance of winning, which is what our forecast gives Trump, is roughly the same as the odds that it’s raining in downtown Los Angeles. And it does rain there.

...in the same page as this:

We simulate the election 40,000 times to see who wins most often.

And that's some bullshit.

Silver and the rest of his staff over the last few months have spent a CVS receipt's worth of text on what went wrong with their predictions -- er, sorry, "models" -- in 2016, and why this year is different, and how uncertainty works, and why they're offering no guarantees really, "but please do keep coming back, and hey check out this cool new simulator thingy where you too can make guesses that are about as accurate as ours".

I almost really wouldn't care, except for one kind of big problem with 538: It is a political observer-effect in action.

Journos, hacks, and other newspeople trying to get one more page impression or soundbite before the end of the day keep referencing 538. When 538 says a candidate is doing well according to their models, it can and almost certainly is changing people's behavior. Likewise when 538 says a candidate is doing poorly. That's hugely problematic and, annoyingly, the more accurate 538 becomes, the more pronounced and dangerous this effect will be.

ashtonkem8• 6 years ago

538 gave Trump one of the highest chances of winning from a non-partisan outfit; 1/3 if I recall correctly. Far better than, say, the NYT that gave him 1% odds. I’m not sure how you can look at that and declare them to be hacks.

Traster2• 6 years ago

There's a small problem with your comment- 538 didn't predict the 2016 election wrong. They gave Trump a roughly 30% chance of winning. 30% happens. That's all. People act like 538 said "Clinton will win!" but that's absolutely not what they did.

Cytobit• 6 years ago

> We simulate the election 40,000 times to see who wins most often.
And that's some bullshit.

I assumed this was a reference to Monte Carlo simulations. What am I missing here?

dragonwriter10• 6 years ago

Taleb insists that Clinton never should’ve received a probability of winning of 90%. Even if polls were heavily in favor of Clinton at the time, he says Silver should’ve taken into account the uncertainty that polls would change over the next few months leading up to the election, or the possibility of major news breaking.

Silver's model fairly explicitly does address both the likely direction and the uncertainty in future poll movements, based on historical evidence. The 2016 model results were more volatile than you'd like to see, but it's at least plausible that's because 2016 was an unusual cycle out in the statistical tail in behavior. The 2020 forecast has behaved in a more conventional manner with mild noisy ups and downs early on, and then a smooth progression of greater certainty, the 2012 projection was similar. While this obviously is way to few to generalize from (but better than try to assess from the behavior of 2016 alone), it's consistent with the idea that the model usually behaves in the way you'd expect a forecasting model to behave, and 2016 was just a collision of unlikely events.

ceilingcorner• 6 years ago

Silver has such an army of fanboys that it’s impossible to even remotely criticize 538 without getting downvoted into oblivion. It’s a weird sort of fanatical statistical religious belief.

The fact of the matter is that election polling is not really accurate anymore. People are afraid to publicly admit their support of a certain candidate, even in a supposedly anonymous phone call. Considering that people have been doxxed online for this, I really don’t blame them for being skeptical and private.

That’s not even mentioning the social aspects of the media. Even if Trump were ahead in all the polls, do you think Silver and other journalist-statisticians would report it as such? I doubt it.

Pundits seems to consistently miss larger geopolitical trends, like Brexit, offshoring of jobs to east Asia, rise of right wing parties in Europe and India, and so on. These qualitative trends will end up being far more influential than a collection of polls.

75dvtwin• 6 years ago

I actually think both are wrong (Taleb little bit less so)

Because they are using empirical evidence (eg previous data) from 'experiments' that are effectively, unrelated to current situation.

In my personal view, of course,

Most previous elections in the last 30+ years, were between RINOs (Republican In Name Only) and Democrats. Those are pretty much two fractions of the same bribe-taking, accountability-avoiding global Cartel.

This Election is between the representative of citizens of the Republic (Trump) and the Cartel's candidate.

This really did not happen from the time Bushes-then-Clintons took over both parties and turned them into co-existing fractions of one party.

- - -

Another analogy: we do not use same methods and design constraints when building anti-lock breaks control software, as we do when we build a daily revenue reporting system.

- - -

Frame of reference is different basically. And we do not know the tensor(s) that help us to move between the coordinate systems.

- - -

So we are not going to be able to use even 2016 polls (because then folks did not realize how Corrupt the RINOs+Dems are )

- - -

My prediction is that Trump wins, GOP takes House and retains Senate.

This will also be the largest percent wise vote for a GOP president, by Latino and African American voters.

- - -

Will check back in 9 days or so, to see if I was right !

Cheers.

kodyo• 6 years ago

I lie to pollsters.

jgalt212• 6 years ago

I think Don Rumsfeld could definitely help bridge the gap between Nassim and Nate.

https://en.wikipedia.org/wiki/There_are_known_knowns

Traster2• 6 years ago

There is a lot of misunderstanding about now-casts and probability in this thread, but for me this discussion between two quite able mathematically minded poeple comes down to the exact same thing as what idiots argue with Nate Silver about. Nate Silver has a model that predicts the likely outcome of the election based on poll data, economic data, and a few others factors. Taleb saw Clinton's predicted victory peaked at 90% and says that's wrong - because the uncertainty is so high that you can't be 90% confident.

Even if Clinton were ahead at that point in the election there was a greater than 10% chance of something changing before election day. But this is just the idiots critique - You said it was 90% would happen but the 10% happened! Well, yes, that happens 10% of the time, and if you're making lots of predictions it happens often.

But let's examine the critique, was Clinton a 90% favourite to win? Well probably, I mean in order for Trump to win from that position not only did he need a decent polling error in his favour he also needed the almost unique situation of the FBI director announcing he was re-opening an investigation into Clinton. We can actually see from the polls the effect this had. That sounds like something, in combination with a polling miss, and in combination with a favourable distribution of Trump's vote by state combined to give Trump a win. It seems reasonable to me to fit that firmly under a 10% probability of happening. If that was in your list of likely scenarios at the point the model was at 90% then you are massively over-estimating how often shocks like that happen.

The 90% criticism of 538 only works if you can provide a reasonable discussion of why you think that what happened between the 90% and the result had more than a 10% chance of happening.

In fact I think 90% of Nassim's problem is that he says things like "When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS."

He's totally wrong. Nate says "Extremely Possible" because when he says X is 75% people hear "X is 100%", and if Nassim had actually paid as much attention to 538 as he should before making that criticism, then he would know that. Every single election cycle 538 have struggled against the fact that people don't understand probabilities intuitively.

Nassim Taleb vs. Nate Silver: who is right about election forecasting?

Related Stories