Nate Silver correctly called 50 out of 50 states

https://plus.google.com/114613808538621741268/posts/LfoYJdWSK71

by btilly • 14 years ago

667 269 14 years ago

plus.google.com

lbrandy• 14 years ago

Once California has its results in, I'm guessing his prediction on the popular vote will make it 51/51.

He was down by 200k a few minutes ago, now he's only down by 110k. California has about 20 million more votes to count, if Obama takes them 55/45, should be good enough to take the popular vote.

jbester• 14 years ago

You're forgetting washington DC

btilly3• 14 years ago

He predicted that Obama would get 50.8% of the popular vote.

Currently Obama stands at 50.3% of the popular vote. However the majority of the currently uncounted ballots are in Oregon and Washington. Therefore Obama's share is likely to go up a smidge.

Still getting the popular vote to within 0.5% is pretty good. Especially considering that many national polls going in were consistently calling for a Romney win there.

mhartl• 14 years ago

This remains to be seen. As of this writing, Florida and Virginia have yet to be called.

arn2• 14 years ago

Here's VA http://electionresults.virginia.gov/resultsSW.aspx?type=PRE&... Reporting at 100%. 50%/48% in favor of Obama.

ComputerGuru• 14 years ago

I've been a huge fan of Nate Silver since forever, and I must say, I really regret his "acquirement" by NY Times. When it was his own website, it was a much more user-friendly experience.

For instance, though I have an NYT account, every other page view I get redirected to a login prompt if I don't press Esc fast enough (this page has expired? WTF?). It also made him more of an independent, though he is (falsely) discounted on many counts by other parts of media, his affiliation with NYT is yet another facet for them to criticize.

lacker• 14 years ago

Try browsing the NYT incognito - it seems to avoid the login prompts.

Osmium2• 14 years ago

Disagree with this. NYT brought a much better user experience and much nicer data presentation, and leant him a feeling of legitimacy too. There's a reason the NYT has the reputation it does.

rm999• 14 years ago

I disagree. Here's an archive from post-2008 election: http://web.archive.org/web/20081106113055/http://www.fivethi...

The old page contains more information yet looks friendlier. The nytimes style is more streamlined, but it became too sanitized. The content became sanitized too; Nate Silver could be more himself when it was his blog. He gave personal accounts, he was more honest, etc.

mayneack2• 14 years ago

Edit: missed that part

thedufer• 14 years ago

I believe that's what he means by hitting escape quick enough.

w1ntermute2• 14 years ago

It's just a lease, he can leave next year (and take the 538 brand with him).

27182818284• 14 years ago

I pay for an NYTimes subscription because they provide content above and beyond the background noise of the top social sharing websites like Reddit.

I'd probably pay for 538 alone depending on the price asked.

monochromatic• 14 years ago

Why do I have to enter my google profile information to open this?

evoxed• 14 years ago

It's just a brief G+ post with two links, but if you want to see what it's talking1bout just go here instead: http://fivethirtyeight.blogs.nytimes.com/

ericdykstra• 14 years ago

Nate Silver's calculations are certainly more sophisticated that a simple time-weighted average of legitimate polling numbers, but are they more accurate?

His baseball projection system, PECOTA, was extremely complex, but barely, barely outperformed a simple 3-year weighted average with an age component (Marcel), and some years was worse. Other projection systems that didn't take player comps into account were better overall than PECOTA (CHONE being the best, before the creator was hired away by a team).

3JPLW3• 14 years ago

The difference? Sample size. In baseball, everyone has the same data. It's all the data available. There's no sampling going on; it's a projection from past performance. Which, in many ways, is way more difficult.

His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.

spicyj• 14 years ago

> to combine multiple polls to increase the effective sample size and shrink confidence intervals

How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.

_delirium2• 14 years ago

I agree with that, but that's part of why I'm not convinced of his overall model's necessity: I think he gets the vast majority of his predictive power simply by being a good poll aggregator, who lays a reasonable statistical model over the polls.

His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008).

On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.

Osmium2• 14 years ago

I think I speak for everyone when I say thank god politics is not like baseball :)

But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.

lifeisstillgood• 14 years ago

>> I think I speak for everyone when I say thank god politics is not like baseball :)

politics is not nearly as important. :-)

viviantan• 14 years ago

Revenge of the Nerd. Sweet, sweet revenge :)

bgilroy26• 14 years ago

It shows how important it is to have experience and know your domain when you build a model and how understanding your subject matter is more important than cleverness.

Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.

bitcartel• 14 years ago

2010 British Election (Number of seats)

Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57

clarky07• 14 years ago

wasn't he badly wrong in the 2010 midterms? I'm not convinced this means he cracked anything. RCP uses simple average of polls, and they got 50/51 correct with Florida being the only wrong if it goes dem (which it looks like but hasn't been called).

cma• 14 years ago

If you take all the simple weighted average with an X component and then selectively choose X after the fact, you can beat a lot of things.

kinofcain• 14 years ago

And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%.

I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate.

The predictions, as they get better, are going to asymptotically approach the actual results.

Ecio78• 14 years ago

And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%

Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...

thewarrior• 14 years ago

I've been reading Fooled by randomness , so i would ask , what proportion of his forecasting record can be attributed to just plain luck ?

waterlesscloud6• 14 years ago

It's looking like the Real Clear Politics Poll average is going to be 49/50.

It was 48/50 in 2008 and 49/50 in 2004.

lftl• 14 years ago

Wes Colley, who runs one of the college football BCS ranking models, also did an electoral model. His model is much simpler, essentially taking the median poll result in each state.

http://www.colleyrankings.com/election2012/

Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).

daniel-cussen• 14 years ago

Maybe he got better.

pbreit2• 14 years ago

Are evaluating baseball players and poll results even similar? One has a binary result, the other a spectrum. One is physically and mentally extremely difficult, the other a simple vote.

foobarqux• 14 years ago

Silver's model does seem unjustifiably complicated. For a simple model that managed to predict the electoral college see http://election.princeton.edu

hayksaakian3• 14 years ago

Let's be honest though, the number of likely outcomes was low enough that simply picking the correct one isn't too impressive.

Edit: I meant to say that the prediction itself was unspectacular, however I respect the meticulous-ness of his process.

pfisch• 14 years ago

What the hell are you talking about? If you actually placed bets by nate silver you could make quite a lot of money.

The number of outcomes is very high on a parlay bet like this.

hayksaakian3• 14 years ago

What I mean is that if you put 100 random people and a room, and explain the likely outcomes to them, its reasonable to expect that each of those outcomes is chose at least once.

There was not anything wildly unexpected that happened, and therefore his predictions being correct seems unspectacular.

fl3tch• 14 years ago

One way to look at is that there were only about 6 states that could go either way, so he picked 6/6.

achompas2• 14 years ago

I don't know what you mean by this.

Are you referring to "Obama wins" as a potential coin flip? Because Silver's prediction was more nuanced than that.

Are you talking about his predictions for the 50 states, which represented a potential (albeit naive) 2^50 combinations? Because he called some very close states, including a Florida election that he forecasted as an Obama victory by hundredths of a percent.

Or are you just trolling?

hayksaakian3• 14 years ago

The only highly uncertain states were Ohio, Virginia, and Florida. (Based on the time before results started coming in). So really there were 9 safe predictions, having picked one of them does not make the OP's story spectacular.

cjensen• 14 years ago

In this particular election, so far there have been zero surprises. In other words, an idiot who simply read the polls could have made the correct prediction simply by being unclever.

This election is not a validation of Silver's models because this particular election did not need a model.

mayneack2• 14 years ago

But he won as compared to all the pure poll averages. One presumes that weighting of polls is the biggest factor in his model now over others, so that sounds pretty good to me.

keypusher• 14 years ago

He has consistently predicted election results throughout his career based on some of the most extensive polling analysis of anyone out there. Alleging that he just made a couple lucky guesses is ridiculous.

chaz2• 14 years ago

The number of possible outcomes is not the same as the probabilities of those outcomes. If I have a weighted coin that shows heads 90% of the time, there are two possible outcomes, but I'm happy to bet you $1 heads each time we flip the coin.

One successful or unsuccessful outcome is not evidence that the model works or doesn't work, but this is how probabilities work.

tlrobinson• 14 years ago

I wonder how much money he could be (is?) making in prediction markets like Intrade.

crntaylor10• 14 years ago

It's hard to make really significant money on InTrade. I don't think you could make a living at it, for example - there's not enough liquidity.

There were some fun opportunities to arbitrage InTrade against BetFair this election, if you could be bothered with the faff of setting up an account on both of them. InTrade had Obama at 66-66% for a while, whereas BetFair had Obama at 79-80%. You could buy on InTrade and hedge on BetFair, and make a guaranteed profit (even after t-costs).

jessriedel• 14 years ago

The presidential market on InTrade was quite liquid. $300k of volume with a bid-ask spread of like 0.2%.

Do you know why there was such a persistent gap for arbitrage between InTrade and BetFair? I don't know about BetFair's vig and rules, but InTrade has only a couple of small fixed transaction costs. ($5/mo to have an account and $10 to pull your money out, I think.) Seems like it should have been eaten up.

denzil_correa4• 14 years ago

Quite impressive. Prediction in any form is difficult; more so when there are a million people involved.

moistgorilla• 14 years ago

Actually, I believe when there are a million people involved it's actually easier. As long as you have access to good data that is.

denzil_correa4• 14 years ago

    As long as you have access to good data that is.

This is very difficult in itself. Even once you have it; it is not easy. More data does not lead to better prediction. It's a fallacy which has been proven wrong time and again.

ptarjan• 14 years ago

It is difficult to make predictions, especially about the future.

-- Yogi Berra

trhtrsh2• 14 years ago

Wow: http://www.larry.denenberg.com/predictions.html

"Who first said "It is difficult to make predictions, especially about the future (or one of its many variants)?"

maaku• 14 years ago

Including Alaska and Hawai'i, with zero precincts reporting!

</sarcasm>

MartinCron• 14 years ago

If he's wrong on AK and HI, I will eat my hat, a moose, and a rainbow.

jlgreco3• 14 years ago

While you're eating a rainbow, might I suggest challenging the horizon to a race? ;)

damian2000• 14 years ago

I was looking at the Intrade prediction market a couple of days ago.. I think they got everything correct too.

arn2• 14 years ago

Presumably Silver's projection is "priced in" to Intrade.

usaar333• 14 years ago

Nope. Florida was predicted as going for Romney up until results starting coming in: http://www.intrade.com/v4/markets/contract/?contractId=74573...

nikcub2• 14 years ago

Most markets got the winner right, which isn't a surprise, but they didn't get the margin right. I found Obama to win 310-330 electoral college votes at $8 on Betfair yesterday (and I put it on, and it looks like I won).

The betting markets did believe that Obama would win, but there was a lot of money on the race being closer and not as wide as Silver was predicting. This is probably because the media narrative of the past two weeks was that the race is close.

There is a rumor that some Republican donors were shortening the odds in the margin betting in order to perpetuate the line that the race is closer. I don't know how true that is, but it would be possible to move the market with a few large bets.

the_cat_kittles3• 14 years ago

Where did you hear that rumor? I have all but concluded that on my own based on the bizarrely low prices on obama shares on intrade over the past week. It would be very interesting to hear something more than speculation.

tmister• 14 years ago

Nate's Prediction and actual results side by side http://twitter.com/cosentino/status/266042007758200832/photo...

pseut9• 14 years ago

So, Nate Silver seems to deserve the even higher profile he'll have after this election, but his model is pretty explicitly not predicting the outcomes of any individual states. They're giving estimated probabilities of the various outcomes, and the (presumptive) outcome that happened was given a probability of ~20%, which was more likely than any other particular outcome, but a far cry from "calling the states."

The big difference is that if this were to happen over and over again ("this" meaning that the most likely predicted outcome actually materializes), that would be (weak) evidence against his model -- if the model's right, he should "correctly call" the states only about 20% of the time. No more and no less.

But good for him, because he's had a lot of unjustified criticism for the last month or so, so he's due some (slightly) unjustified praise.

trhtrsh2• 14 years ago

Huh?

http://fivethirtyeight.blogs.nytimes.com/

"State-by-State Probabilities"

pseut9• 14 years ago

Do any of them equal 1? Saying that >50% probability means that the model picks the state for Obama would be fine as a classification rule, but he's trying to estimate the probabilities, not forecast who will win.

The ~20% comes from the electoral vote distribution, where you can back out the mix of states that gives 332 Electoral college votes.

furyofantares5• 14 years ago

Take Florida, for example. He predicted a 50.3% chance of an Obama win. If Romney actually wins it, he's only a little bit wrong - counting him as 0/1 on Florida in this case wouldn't be accurate. If Obama wins it, he's only a little bit right - counting him as 1/1 on Florida here wouldn't be accurate.

It's hard to quantify just how close he was. I give him props because his prediction was that Florida was very close and that turned out to be true no matter which way the state actually ends up going in the final result.

Now take Virginia, which was one of the later states to be called. He had Obama at 79.4% chance to win. But it turned out to be reasonably close -- which makes me wonder if we really had enough data to justify such a large %. It's a highly polled state, so maybe we did and we just came somewhat close to hitting that 20% of the time where all the polling was inaccurate. Or maybe he had Obama's chances too high.

Another way to judge a prediction model like this is to credit him with .794/1 for Virginia, rather that 1/1. For Florida he gets .503 if Obama wins, and .497 if Romney wins. That doesn't capture it perfectly, either, but at least it doesn't give him 0/1 in Florida if Romney wins even though he successfully predicted that the state was super close.

danielweber4• 14 years ago

If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of them should have gone to Obama.

Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.

programminggeek2• 14 years ago

http://isnatesilverawitch.com/

keithvan• 14 years ago

Ahem, he would be a warlock.

programminggeek2• 14 years ago

He should have the right to self-identify as a witch if he wants. You have no right to take that away from him.

PakG1• 14 years ago

Or witch doctor. And he'd be in good company.

http://www.businessinsider.com/a-kenyan-witch-doctor-predict...

ebiester• 14 years ago

No. Males are witches as well, Warlock translates to oath breaker, and is not considered with the pagan community (at least the parts with which I have interacted) as a male term for witch.

celoyd• 14 years ago

Proprietor of that site here.

Nope, “witch” can apply to men just as well as women and always has. In fact, you’ll find several dictionaries citing Malory’s “Le Morte d’Arthur” (1485) as an early usage, where Merlin is referred to by some as a “wytche”.

It’s also the standard academic term across gender. If memory serves, it’s used this way in the groundbreaking study “Witchcraft, Oracles and Magic Among the Azande” by E. E. Pritchard, which set the standard for the anthropology of witchcraft.

It’s also used for a man in the television show “Futurama”, for a more contemporary colloquial reference.

nollidge4• 14 years ago

Or wizard, if we're in the Harry Potter universe.

paulsutter• 14 years ago

Nate Silver seems like a pretty smart guy, wouldn't it be better if he spent his time doing something more productive? Predicting the outcome of an election may have practical applications in gambling or for a hedge fund, and yes he gets mad publicity and attention from women sure, but aren't there more useful ways to apply statistics?

EDIT: Haldean, your point is excellent and cancels out mine absolutely. Good thinking, you are entirely right. I would add that his attention and publicity itself should increase the credibility of rational analysis in the news, and that alone would be a great accomplishment.

fourstar• 14 years ago

Psh I think it's awesome and he's making statistics hawt right now. Gotta find your niche in something to make a name for yourself, anyway.

hooande• 14 years ago

To his credit, he was wise enough to publish a book at just the time when people want to hear from him most. The downside of being the king of political prediction is that you're only popular once every four years. But you're really really popular, and he seems to be aware of and capitalizing on that.

adestefan• 14 years ago

It's really every two years since he was one of the first that showed Republicans picking up big gains in the US House and governorships in 2010.

haldean• 14 years ago

Instead of thinking of him as a statistician who works for a newspaper, think of him as a statistically-minded journalist. I wish _more_ people did what he did: apply scientific reasoning to national and global issues.

yen223• 14 years ago

He began his career predicting baseball scores, so I guess that's more useful?

gruseom3• 14 years ago

He has utterly discredited the horse race scam of big media political coverage. That is a significant contribution, because such coverage has been so influential in managing public opinion and how people think about politics.

He did it first and more dramatically in the 2008 primaries, but this time feels more like the watershed.

It's not all Nate's doing, of course; this is a long term trend and many people are working on it.

patrickk• 14 years ago

The idea that this was "a close race", "too close to call", was a narrative pushed by the media to make it more artificially exciting to drive viewers.

Pundits hate him for it because it shows how content-free, and ideological/ opinion-driven their arguments are rather than sound judgement based on available data.

klenwell• 14 years ago

Imagine, too, a scenario where the results deviated significantly from his predictions. In the past, the public, guided by the media, might have been satisfied shrugging it off as "god's will" or "people are just unpredictable" or even "pshaw, more conspiracy theories!" Now I like to believe there will be more pressure to investigate and rationally explain the source of that deviation. Are dead people really voting in Chicago? Were voting machines hacked in Cuyahoga County? I feel like Silver's counter-narrative -- and more critically, his methods -- have been an important contribution to a fair and open democratic process.

pfortuny3• 14 years ago

You've got to read the answers above concerning the UK 2008 election... Just for the record about 'horse race' 'discredited' and 'big media'...

This is beginning to look to me like 'big stats'.

waterlesscloud6• 14 years ago

Now here's a bet I'm willing to take.

The horse race coverage of the next political race will be just as strong as it was this time. This isn't the triumph of data over narrative, it's just data showing it can be more predictive.

Narrative still has more mass popularity, and it always will.

nollidge4• 14 years ago

> attention from women

Ha, well, that's got to be annoying for him: he's gay.

calinet6• 14 years ago

There is room in this world for every corner of productivity. He's doing some very good work; other people are doing very good work in exactly the other fields you're talking about. One is not inherently better than the other, since it takes many different outcomes and works to create this thing we call society.

In fact, you could say that the maximal way to be productive might be to be diversely productive, since it produces the widest array of new ideas and conclusions, and tests every nook and cranny of possibility. Surely we have enough people in the world and in the country to provide for that diversity.

mamatta• 14 years ago

quick image reference on Nate's predictions versus actual results: http://cl.ly/image/0744432U0S0L

quangv• 14 years ago

I'd love for him to go to past elections and see how well our political voting system works...

the_cat_kittles3• 14 years ago

You might like the economic sub discipline of "Social Choice Theory"

ww520• 14 years ago

Hard cold numbers won again.

The venom directed at Nate Silver before the election was astonishing. I guess the pundits had sensed their careers of predication were numbered and fought back hard, but Nate was proved right again. Kudos.

rndmize• 14 years ago

Perhaps this will lead to a greater interest in statistics and data in general politics and less in pundit opinions. Haha, who am I kidding...

mkr-hn3• 14 years ago

There will always be a market for uninformed jabbering. Pundits wouldn't worry if they looked at their own record.

DanBC• 14 years ago

Is there a "pundit prediction tracker" anywhere? Somewhere that tracks the predictions made by talking heads and shows how often they're right or wrong?

> uninformed jabbering.

And for informed jabbering. It's frustrating that analysis is being replaced with opinion, especially when it's clear that opinion driven prediction is garbage.

a_bonobo• 14 years ago

Doesn't that destroy the allegations of voting fraud in the US? (Not a US-citizen here)

If there was widespread fraud with the voting machines as alleged elsewhere, the voting outcome should have diverged widely from the predictions. The predictions are so close to the actual outcome, and because from what I understand the predictions are mostly based on polling, there couldn't have been much fraud, or am I wrong?

Flippant edit: Assuming of course that Nate Silver wasn't in on the fraud and didn't adjust his predictions accordingly.

jlgreco3• 14 years ago

Alternatively, it just suggests that people who engage in voter fraud tend to suck at it.

mseebach2• 14 years ago

Not on it's own, no. At most, is shows that fraud wasn't widespread enough (or equally widespread on both sides) to skew the election away from the will of the people.

Also, fraud is much more effective in smaller races - of which there are lot, many of them too small to have as good poll coverage as the presidential one - so if there is fraud, those races are probably where to look for it.

espeed• 14 years ago

In 2004 there was evidence of voting fraud in Ohio -- the exit polls weren't matching the results (http://whatreallyhappened.com/WRHARTICLES/2004votefraud_ohio...).

Bush won Ohio, which helped him win the election by the narrowest of margins. Ohio used Diebold electronic voting machines, and Diebold's CEO Walden W. O'Dell was a long-time Bush loyalist. Later it was confirmed that Diebold voting machines could be hacked remotely (http://news.ycombinator.com/item?id=3045086).

So yes, being able to create an independent, accurate model will add another layer of verification and will hopefully help keep voting fraud in check.

dkrich2• 14 years ago

I've got to say that I've been a cynical admirer of Silver since the last election.

Why cynical? Because I began to notice throughout the Republican nomination process that his observations varied wildly on which Republican candidate presented great values on intrade. I think over time he gave a nod to every candidate, save Michelle Bachmann, Tim Pawlenty, and John Huntsmann. From the outset, Romney was the favorite to win, and not surprisingly, he won. But throughout the nomination process where much less was known, many of his projections proved to be wrong. So despite all of the contrarian opinion on which candidates may pull it out against popular opinion (or perception) these faded away until Romney was the clear favorite, Silver predicted him to win, and eventually he carried the nomination.

Once in the Presidential election, Obama was the favorite. Incumbent Presidents rarely lose, and Mitt Romney was not exactly a compelling candidate.

All that said, he's clearly a brilliant dude, and I give him a lot of respect for the accuracy of his predictions. I just haven't found them to be hugely useful for any practical purposes. I guess people tend to be enamored by the things they don't understand (statistical analysis), and I don't think there's anything wrong with that.

cdf• 14 years ago

Nate Silver claims to be a statistician, not a clairvoyant.

While there is a sense of inevitability of Romney's nomination, I think it is obvious that the Republican voters agree with you that he is not a compelling candidate and were desperately looking for alternatives, and as each candidate wilted in the spotlight, they kept moving on to the next, and polling numbers reflected that, and Nate Silver was merely reporting that shift.

dkrich2• 14 years ago

So is that at all at odds with what I said?

I'm saying that I haven't found his predictions to be useful because they have not correctly predicted anything that I myself didn't predict with the same information without any statistical analysis. Sure, he predicted it with 3% greater accuracy. So what?

Sadly what makes most elections predictable is how shitty the candidates are on one side.

To be fair I don't think that a statistical analysis will ever be useful in an election unless the election is absolutely a dead heat. Most elections for the past several years (with the obvious exception of 2000) have been fairly predictable from the outset.

the_cat_kittles3• 14 years ago

There were a couple notable projections that he was considerably off on, particularly some Santorum primary numbers late in the game. But, even someone with a projected 98% chance of winning should lose 2% of the time. In the aggregate, Silver's projections are excellent. I would like to see something like a Brier score for his projections, and compare it to various polling organizations, and other analytical pundits (do any notable ones exist?) to see who has done the best. On a related note, it would be nice to have a place where people could publicly lock in projections, and develop a record. Maybe that would help put the Joe Scarboroughs of the world out of work.

gwern2• 14 years ago

If it helps, http://marginoferror.org/2012/11/08/aggregating-the-aggregat... reports a RMSE for just his state-level margin of victory.

But as part of a project I was aggregating predictions as well, and for his state wins I got 0.02833, for his Senate-level predictions I got a Brier of 0.04484545, and for his Presidency win & Senate wins & state wins, I got a Brier of 0.03710118.

Hope this helps!

zecho• 14 years ago

>I just haven't found them to be hugely useful for any practical purposes.

His models are dry, boring even. But they're based in some semblance of data. When compared to the vast majority of the pundit class, and political journalism in general, his model is a breath of fresh air if you want to understand the state of the campaigns beyond what some pundit heard from "a source" or some journalist asked of a random campaign worker.

zabuni• 14 years ago

He makes mention of that in his book. He uses the example of online poker players, which he used to be. He doesn't consider himself great at that either, just better than the hordes of rank idiots that played during the online poker bubble before the feds shut him down. One they were chased away by the various lawsuits, he lost a great deal more of games.

He also says that effort/optimization falls under the Pareto principle, where the last 20% of your optimizations is about 80% of your work, so it's a lot harder to get from being pretty good to world class. But any model that takes into account prior data will get you a hell of a lot of the way there.

digikata• 14 years ago

In the primaries there are more candidates and fewer polls so you get sparser measurements of the state of a given candidate's appeal. So maybe it's natural for the predictions to have more variation at that stage.

josephlord3• 14 years ago

I'm not from or in the US but can I ask a slightly off-topic question?

Is Romney really the best the Republican's can offer? It seemed from here (UK) that he was not a compelling candidate from the start but most the others in the nomination process were massively flawed in some way or other and/or their political views were just too far from the mainstream to really fly.

Romney might still have won if Obama had made a mess of something in the last days before the election as a couple of percent swing would have changed everything but it just seems like the Republicans should be able to do better.

Two questions:

1) Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)

2) If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?

andrewcooke2• 14 years ago

[not american] my impression is that it is the latter - that it is hard to appeal consistently to both the party (to become the chosen candidate) and to the nation. hence the accusations of flip-flopping and the arguments from "reasonable republicans" that romney would actually have been quite good even though he had to say various odd things.

i wonder if they will attempt to change how candidates are selected.

[above i have tried to be impartial; a more personal take is that the republican party has become accustomed to lying to its supporters and is now trapped with the results. to have to rely on people trusting that their candidate is "bluffing" does not seem healthy.]

---

more on topic: the results reporting really disappointed me. the take-away from silver's work is not that this is very complex. it's that the electoral college system plus most people voting the same way over time means that only a few results are uncertain. so focussing on those places allows you to predict the election. similarly, reporting all the other (known) data is irrelevant. it's not news, and it's not exciting.

yet on election night we saw every result being treated equally. there was no acknowledgement of the reality. it looked exactly like an entertainment show.

was there any presentation of the results, anywhere, that ignored most of the data and focussed only on the swing states? that spent the rest of the time analyzing what had happened in the light of the clearly expected, accurately predicted result?

hindsightbias• 14 years ago

1) No. Just look at the first RNC primary debate videos. These are the Republicans rocket scientists.

2) Right pundits are already arguing that Romney wasn't extreme enough. This is not a condition they can get out of. All future candidates will have to meet socialcon and Koch requirements before even thinking about swing or blue states.

richardwhiuk• 14 years ago

Better candidates may have decided to wait out this time - battling against a incumbent is always harder.

nl2• 14 years ago

Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)

Yes. In this election John Huntsman would have had a good chance of winning the election by carrying the moderates and picking up a few Democrat voters (although he probably would never have got enough Republican voters). In many ways Rick Perry was a better candidate than Romney (in terms of who could win the election), but he screwed up in the debate and I think the US is a little wary of another Texas Republican.

If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?

Sort of. To win the primary you need to carry either the Christian conservatives or the business wing of the Republican party. In this election the Christian conservatives vote was split many ways which left Romney.

Romney's big problem was that he didn't appeal to Republicans enough for them to vote for him. He was sort of a Republican version of John Kerry - someone they thought moderates could vote for (which was kind of, somewhat true - he wasn't 100% insane anyway), but he didn't really appeal to his own party that much.

nollidge4• 14 years ago

As a liberal American, my thesis is that the Republican party is extremely fractured between fiscal conservatives (who are largely centrist or even liberal on social issues) and the straight-up social conservatives. Which means that finding a single individual who can coherently represent all of them is basically impossible.

So enter Romney, who seemed willing to be mercurial - literally adopting different positions in front of different audiences and scoffing whenever anybody pointed it out. But that cynical strategy was necessary to have any viability whatsoever in such a diverse party.

pdog• 14 years ago

I thought Romney was an excellent candidate, but his campaign was rather hamstrung by the Republican party.

stcredzero6• 14 years ago

> 1) Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)

Lots. Governor of New Jersey Chris Christie was often mentioned. The candidates who stepped in the ring for the Republican party were second stringers who saw their chance. The A Players were either not available or too smart to step in the ring. (The economy was wrecked under the Republican watch and Obama was in the process of fixing it.)

greedo2• 14 years ago

The GOP fielded a horrible panel of candidates; all were very flawed and Romney was the most likely out of any of them to win against the President. They also hamstrung each other during the primary campaign in ways too numerous to list. This gave the Democratic Party a huge advantage since the President didn't have to undergo much scrutiny during this period. Incumbency is really hard to overcome.

1. There are numerous candidates who might prove compelling in 2016. I don't think Christie belongs in that group since he's being viewed as a pariah after the tropical storm. Rubio would satisfy the more conservative elements plus potentially bring in the Latino vote.

2. I don't think the GOP has huge structural issues, but I'll grant that the Democratic Party is much more unified.

The GOP also has the nickname as the stupid party. Things like Todd Akin's comments (and Mourdock's) tainted the entire party. Had they repudiated both immediately and stuck with it even in the face of losing those two Senate seats (which they lost anyways) they would have done better in attracting women voters. Instead, the comments reinforced the Democratic meme that the GOP hated women.

chc2• 14 years ago

In the last election, the Republicans ran John McCain†, who is actually pretty much dead center in the context of US politics. It seems to me they ran Romney this time because 1) Romney has pretty strong economic credentials and Obama hasn't been outstanding in that department, and 2) going centrist didn't help last time. Also, like others said, the best candidates are probably waiting until next time, because Obama hasn't messed up badly enough for his supporters to turn on him en masse, so this year's candidate was likely to be a sacrificial lamb.

† McCain's running mate, Sarah Palin, was nutjob who seemed to be there purely to appeal to the demographics he didn't, but the man himself has historically been pretty even-handed.

epaga• 14 years ago

Even with an incredibly exact model, to actually get ALL 50 right really does require a lot of luck in addition to skill. Even if he's 98% accurate for each state (which seems amazingly high to me), that's still only a 1 in 3 chance to get all 50 right.

shalmanese• 14 years ago

He really only got 7/7 right. The other 43 states were not really in contention.

crntaylor10• 14 years ago

It's clearly great work by Nate and the rest of the 538 team. However, as has been pointed out, it's not that hard to build a model that gets close to predicting everything correctly.

I built a model in a couple of hours on Sunday afternoon, which simply takes all the most recent polling data, takes an average, does a quick fudge to adjust for the number of polls, and then runs 10,000 simulations to get a probability for each state. The source is on Github:

    https://github.com/chris-taylor/USElection

and the predictions are in this gist:

    https://gist.github.com/4012793

The result? My model gets 50/51 correct if Florida eventually goes DEM (which looks likely) or 51/51 correct if Florida goes REP.

Edit: full disclosure - with all data up to 6 Nov 20120 it predicts Colorado to be a toss-up, and I manually broke the tie in favour of Democrats, based on earlier models favouring them in that state.

chaz2• 14 years ago

Nice work.

The success, though, isn't his specific model. It's that Silver was able to market the idea that using statistical models is better than a table full of talking heads at predicting an outcome. And that a "dead heat" doesn't mean it's a coin flip. In hindsight, it seems almost ludicrous that it hasn't gained greater traction before.

He deserves a ton of credit for championing analytics into pop/political culture, but I'm sure we'll see many more models and a lot more statisticians during the next election, some of which will be better than Silver's models (if he's still doing them). I'm all for people asking to see the data instead of the media sound bites.

To keep this HN relevant: it was the marketing that drove his startup success. The product (the model) isn't quite perfect and an excellent substitute product was created by someone else in very short order (you). But he put together a great blog, grew out his brand, and eventually saw hockey stick growth.

crntaylor10• 14 years ago

Yep, I agree with this. I really built the model for fun, just to see how close it was possible to get with a few hours work.

A lot of forecasting problems have a structure like this one did:

- There is a lot of publically available data, and simple statistical models based on that data give pretty good predictions with very little effort.

- If you're an expert with a lot of time on your hands, you can put a lot of effort into improving the forecasts by 1-2%.

- In the end, it comes down to the effective business use of your model. Nate Silver had an excellent brand around his model.

In fact, I'd wager that he knows his model is more complex than it needs to be, but part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".

mathattack• 14 years ago

You hit it on the head. It wasn't the idea itself, it's his marketing of it. This is how Levitt broke out of the pack with Freakonomics. It wasn't just being novel or counterintuitive, it was getting up on the roof an shouting about it.

MattSayar• 14 years ago

So basically, Nate Silver is to politics what Billy Beane is to baseball?

marcamillion• 14 years ago

I agree to an extent. His product was clear much better than the existing alternatives - but he also made a brilliant move by choosing the NYT as his exclusive distribution outlet.

It's more than just marketing. It's largely about distribution.

MichaelApproved• 14 years ago

Great. Now you can go on https://intrade.com/ , make some bets and clean up.

crntaylor10• 14 years ago

I already did.

stcredzero6• 14 years ago

Putting just over $2000 on Obama several days ago would've gotten you $800.

ssharp• 14 years ago

Silver's model isn't particularly advanced on election day. The further you get from election day, the more advanced his forecast is, accounting for various factors and ebbs and flows in polling data -- convention bounces, economic news, etc.

A big part of Silver's success has not been his math, but rather his excellent writing. He offers easy to consume analysis of the polls and makes a boring (to the masses) subject interesting. He can be portrayed as a stats geek when presented to the public, but really he's a great communicator who also knows his data.

shardling2• 14 years ago

A lot of the work silver doesn't isn't just aggregating the polls. He tries to take into account the probability that the polls are wrong.

According to a couple of his blog posts, going just by the most recent polls (and their margins of error) Obama had a 100% chance of winning -- but in the past, polls have sometimes been systematically misleading.

hullo• 14 years ago

the hard work that silver did wasn't in being able to predict the election 2 days in advance, any middle schooler with a calculator could have done that this time around. the work that he did was in being able to forecast the results months in advance, adjusting for a broader variety of inputs, including historical research.

crntaylor10• 14 years ago

That's fair, but I'd point out two things:

1. Even though any middle schooler with a calculator could have predicted the election 2 days ago, it seems that not many of them did - at least, if you judge based on the odds for a Democrat win you could get on InTrade and BetFair 2 days ago (65% and 75% respectively).

2. One neat thing about my model is that, in principle you can run it in "historical" mode and see what its predictions would have been at any given point in the past. So I can see what probability of a DEM win I would have assigned 2 months ago, and see if that's significantly different from the forecast from a more complex model.

rhizome5• 14 years ago

It's one thing to hold the bottle, quite another to aim the funnel from 10ft away.

pbreit2• 14 years ago

Was he really forecasting the result weeks/months ago or just showing what the result would be if he vote happened on that day, according to polling data?

qdpb• 14 years ago

Yes

But what people are saying is not "Nate Silver's predictions 1 month before are spot on". They are saying "his predictions ON THE DAY OF ELECTION were spot on".

Which is not that hard to do. Does not mean Nate Silver is a hack. Means people are not measuring his success appropriately.

#NateSilver2016

joekrill• 14 years ago

Yes it's quite easy to write off the simplicity after the fact. Kudos.

Tyrannosaurs• 14 years ago

Is there an element in this of the fact that if you looked at with any degree of sophistication it wasn't actually as close as some made out?

Pundits and the media gain from the narrative that it was on a knife edge because it keeps people coming back for more, but in reality Romney never really picked up, let alone sustained, the sort of position in the swing states that would have led to a win. It was never comfortable for Obama and Romney wasn't out of it but the idea it was neck and neck was fairly dramatically over stated.

Not undermining what Silver did but it seems to me that the same pundits who spoke out against him made him look more impressive than it was by claiming the race was closer than it was.

pseut9• 14 years ago

I don't think that his writing gets enough credit -- explaining what is happening to generate the predictions is really useful. Pretty much everyone that builds models from the state level polls seems to have made more or less the same predictions, but I've found his discussion to be the most interesting and informative (I may have missed some other good sources though).

enjo• 14 years ago

I'd be interested to see how your model works for Senate races. That seems to be the bigger challenge.

House races seem to be far to data sparse to be reliable.

crntaylor10• 14 years ago

Maybe I'll get around to adding a forecast for the senate/house as well. Although any forecasts I make would be somewhat less compelling now that the results are out...

brianchu• 14 years ago

Keep in mind, however, that we could not have expected Nate Silver's own model to predict 50/50 states. Given that Nate predicted that there was a 50.3% chance that Florida would go to Obama, the only difference between this being a story of predicting 50/50 states and this being a story of missing at least one state is a coin toss. The reasons his model turned out to be 100% accurate is sheer luck.

Kurtz79• 14 years ago

I think it's realatively easy, for a smart guy with a solid understandings of statistics to come up with a reasonable model (weight polls by sample size, come up with probability distribution for each state, run x Monte Carlo simulations and take the average of the result...).

But a thing is coming up with a good model, another is to defend it publicly putting your reputation on the line, while it would have been a much safer bet simply to say 'it's too close to call', as many so-called experts did.

Full credit to Nate, he made math cool for a lot of people.

brownbat• 14 years ago

Doesn't that mean he undercalibrated some of his predictions?

If you say 50 events should happen with a 60% probability, and they all happen, shouldn't you have upped your confidence?

That said, since most his naysayers were saying he was OVERconfident at the time, kudos to him for proving them wrong.

jmduke• 14 years ago

Such extrapolations are really tough -- keep in mind, for each state we only have results with a sample size of 1.

monk_the_dog• 14 years ago

If the predictions are independent, then I agree with you. I can think of reason why they would not be independent (trying to account for non-random polling samples, for example).

3JPLW3• 14 years ago

They aren't independent. The developer of the 512 paths to the White House lamented this in getting conditional probabilities into the path choices. The simulations take into account the national popular vote polls and similar-state demographics (if one state goes one way, it's more likely that a similar one will, too).

Look at his EV histogram; 332 EVs had the highest probability of occurrence at nearly 20%. While there are other ways to get to 332, I'd imagine most of that percentage is from a map like last night. That's clearly not the partial probability of each state multiplied together.

k2enemy• 14 years ago

I wonder if having such good predictions is ultimately harmful to the democratic process? If I were only voting on the presidential election, I would not have bothered going to the polls because my state was solidly on one side and the chance of my vote being pivotal was basically zero. But some of our local elections were less certain, so I went for those.

As the prediction models get better, get applied to local elections, and get more publicity, I wonder what it will do to people's incentive to vote? I'm much more likely to go out and cast my vote when the outcome is uncertain and I think I have a chance of being the pivotal vote. But if super accurate forecasts tell me my vote won't matter, then maybe I won't bother standing in line for an hour.

pfortuny3• 14 years ago

When the incentive of the people changes, then the outcome will change as well.

Spooky232• 14 years ago

It's a little much to claim that this guy called it. The minor detail of actually conducting the polling was done by others.

Tuna-Fish• 14 years ago

The point isn't that Silver is some superhuman guessing machine. The point is that Silver explicitly did nothing particularly special over applying sound statistics over publicly available data, came out with very different predictions than the traditional media did, took a lot of flak from them for it, and turned out to be exactly correct in every meaningful way.

538 is a statement against the traditional election coverage, in effect saying that math is better than pseudo-objective talking head bullshit. From his blog:

Nevertheless, these arguments are potentially more intellectually coherent than the ones that propose that the leader in the race is “too close to call.” It isn’t. If the state polls are right, then Mr. Obama will win the Electoral College. If you can’t acknowledge that after a day when Mr. Obama leads 19 out of 20 swing-state polls, then you should abandon the pretense that your goal is to inform rather than entertain the public.

http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...

This is why the more math-literate portions of the internet have erupted with cheers for Silver.

danielweber4• 14 years ago

and turned out to be exactly correct in every meaningful way.

Silver himself wouldn't call his predictions correct.

He was not making thumbs-up v thumbs-down predictions. When he said "60% chance of Obama winning this state," it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong.

Spooky232• 14 years ago

Ok then, mea culpa. I guess I take it for granted that anyone actually pays attention to talking heads. IMO, they have been completely useless since the OJ trial.

Codhisattva• 14 years ago

Silver affirms that public opinion polling is accurate - as long as you know how to read the data.

Not to diminish Silver's work - I'm just saying the data he uses needs to be accurate in order for his models to be accurate.

elbac• 14 years ago

I'd like to see Nate run the numbers which would point to districts where there were statistical anomalies on the lookout for election funny business.

caycep• 14 years ago

what is the technique he is using? is it something like a generalized linear mixed model? (i.e. like using R's lme4 module)?

alexmr• 14 years ago

I'd like to see the comparison for -90,-60,-30,-15 days from election too.

Nate Silver correctly called 50 out of 50 states

Related Stories