How did NPR’s pre-election poll get things so so so wrong?

https://statmodeling.stat.columbia.edu/2022/11/10/how-did-nprs-poll-get-things-so-so-so-wrong-the-friends-family-coworkers-question/

by Tomte • 4 years ago

24 78 4 years ago

statmodeling.stat.columbia.edu

PaulHoule2• 4 years ago

Not sure what "there" there is here.

Republicans did not overwhelm but it looks like they will get control of the house. The senate is hanging by a thread. Since Reagan there have always been Democratic senators that vote with the Republicans. Between that and the filibuster, the Democrats would need to control something like 65 seats in the Senate to really say they control the Senate. We'll be in the divided government state which is normal for American politics.

As for "friends and family" I expected my wife to vote Democratic and my son to vote Republican, probably my coworkers to vote Democratic and many of my friends but one long-term friend of mine is the ringleader of the local Republican party. So my answer to that question would be "mu".

zdragnar• 4 years ago

There was a lot of hype around a "red wave", not only in the senate but towards election day several governorships where the Democratic governor had a lead completely vanish.

I suspect much of this boils down to pollsters consistently undercounting republican support for several election cycles and over-correcting this time around.

0x500x792• 4 years ago

I think that is the point: Polling/Outlets had them getting a massive lead, keep in mind too that this was the worst performance for midterms for the party that does not hold the presidency in a very long time. Usually the party that has the presidency takes a larger share of the house in midterms. The fact that the polling was calling for a "Red Wave" and that the "Red Wave" severely underperformed is the thing that is being investigated now.

PaulHoule2• 4 years ago

538 has been all through this. 538 expected the Senate to be close and for Republicans to win the house, which seems to be the outcome.

https://fivethirtyeight.com/features/our-best-tool-for-predi...

It's normal for the president's party to lose some ground in the midterms. Biden's approval numbers were poor which was not good for the dems... but, that's countered by Trumpism being unpopular. Republicans who were sane did pretty well, lunatics like Oz didn't. Trumpism was so toxic that it was a winning strategy for Democrats to promote the worst Republicans in their primaries.

chomp• 4 years ago

One possibility is that Republican family/friends/coworkers were more public about their political views, compared to Democratic family/friends/coworkers.

I live in the South. You learn very quickly as a Democrat to keep your mouth closed about politics. I know everyone in my life who votes R consistently, but the D's are harder to pin down.

Election polling is even harder than weather forecasting these days. At least cold fronts follow general patterns.

gorgonian• 4 years ago

I live in California and it’s the same with the parties switched.

tjs8rj• 4 years ago

Everyone said the reverse of this when the polling failed in 2016 - that Republicans were afraid to share their political affiliation for fear of being cancelled etc

theprincess5• 4 years ago

It's pretty simple. In Democratic areas you don't speak out about being Republican and in Republican areas you don't speak out about being a Democrat. I'm from the South, lived in Oregon and New York for a while, and now live in the South again so I've seen the dynamics play both ways.

Social ostracism is more vicious from Democrats but I've known of Republicans to spread outrageous rumors about Democratic voters and in places like where I grew up, if you're a man, you'll be called gay/pedo and possibly beaten (usually not jumped, but aggressively taunted with slurs until a fight ensues from what I've seen IRL) unless you're well liked for some other reason like that you're well connected, have money, or are really masculine/served in the military.

Overall I'd say it's better to be a Republican in a Democratic area than the opposite since the looming threat of physical violence is worse than people just thinking you're a bad person.

The main reason why Republicans in Democratic areas would have to worry would be if they work in a creative industry where being known as a bigot of some kind would cause people to not like to collaborate with you but even then you're relatively safe as long is you're only bigoted against one of the less protected minority groups like trans people (probably why trans people are the culture war topic du jour).

taeric2• 4 years ago

Not everyone said the reverse. And this is too close to "whataboutism" that is typically not in good faith. When you have some legitimately bad leaders on one side, it is hard to take it serious that folks are afraid "of being cancelled" when they are not afraid of the very real harm being threatened by their political affiliation of choice.

Overtonwindow• 4 years ago

When I lived in Redwood City, CA I learned to keep my mouth shut that I was a conservative. If it had gotten out then my job would’ve been under threat, and maybe my own personal safety.

taolegal• 4 years ago

wonder why this is getting downvoted, cause I know people would think differently about me if I said I was conservative, they're all about diversity until you don't agree with them

helloSirMan2• 4 years ago

Seconded. I lived in Texas and learned to keep the peace and blow with the wind. Now, I'm in PNW, I'm similarly unsure how to proceed here. I'd fail any purity test administered.

travisgriggs• 4 years ago

Fellow PNW’er here. Welcome!

I think it’s actually pretty easy. Look at any county voting map. Prevailing party vote tracks population density without fail. American politics has turned into “The country mouse and the city mouse”.

Besides that, there’s a general sub vibe (e.g. Dog whistle effect) that usually signals the predominant opinion in an area.

chasd00• 4 years ago

No one wants to talk politics in the real world. Now on the internet it's a very different matter.

jjulius• 4 years ago

I'd be a bit more careful to paint such a large region with such a broad brush, there's nuance within. I grew up in the PNW, but lived in Atlanta and East Cobb in my 20's from 2010-2016. I never minded being the person in the room with an opposing viewpoint (and, importantly, never felt like I couldn't/shouldn't be that person), and know plenty of people who vote D consistently.

taeric2• 4 years ago

Agreed on being careful, but I also have to ack that a broad brush works way better than you seem to be implying.

Yes, there is nuance. But, also yes, GA has a runoff happening with a candidate that is bonkers. I am not at all clear I see any nuance helping explain that one.

notadev• 4 years ago

I’m in the inverse position, a conservative who lives in one of the most progressive counties in the country, and I work in Tech. I’m basically politically agnostic as far as anyone is concerned.

andrew_• 4 years ago

I live in the South. It's nothing like that where I live because I'm in a large city in the South. The Rs are as loud and as proud as the Ds. And then there are a ton of people who are tired of the tribalism and politic ruining friendships and just choose not to care or talk about it. There isn't any fear about it in my corner of the South - we choose those we associate with wisely.

tablespoon5• 4 years ago

I live in the South. You learn very quickly as a Democrat to keep your mouth closed about politics. I know everyone in my life who votes R consistently, but the D's are harder to pin down.

That's polarization for you. In other places, it's the exact opposite.

helloSirMan2• 4 years ago

At this point I'm not sure how good polling can be carried out, in terms of data collection methods:

Phone behavior is generationally defined, I'm not sure if the average gen Z'er talks via the phone function of...a phone. It would be much more reasonable to expect an app based communication.

Ask the captain of the SS Boaty McBoatface about polling via internet. It's all polls in an echo chamber and big targets for those that would wish to introduce bias.

rob742• 4 years ago

Boaty McBoatface doesn't have a captain, it's a robotic autonomous underwater vehicle (https://en.wikipedia.org/wiki/Boaty_McBoatface), one of three carried by the RSS Sir David Attenborough, that was initially supposed to get that name, but then... didn't.

amir734jj5• 4 years ago

I would say they underestimated the impact of federal abortion ban had on women. I live in Wisconsin and seen it first hand. Women who are struggling financially wanting to vote Republican but they don't and they vote democrat because they have young girls at home and worried about their future and having options. It was a mistake on Republicans part to ban abortion at the federal level. Personally, I'm against abortion and I find it immoral but I also think government shouldn't interfere with women's healthcare.

Mdndndns• 4 years ago

Republicans never banned abortion at the federal level.

pc862• 4 years ago

Abortion is not banned at the federal level.

amir734jj5• 4 years ago

You are playing with semantics. Please look into Dobbs decision.

Source: https://en.m.wikipedia.org/wiki/Dobbs_v._Jackson_Women%27s_H....

nozzlegear5• 4 years ago

This is technically correct, but don't forget that Senator Graham had attempted to introduce a federal abortion ban immediately after the Dobbs decision. The threat of a federal abortion ban being implemented by Republicans was no doubt on the minds of many women who voted against them in this election.

piva00• 4 years ago

I'm totally against abortion and I find it immoral

I'd recommend in the topic of abortion to not use absolutes, in this case you are also stating you are against abortions in case of rape, when a fetus is missing vital organs (brain, heart, etc.) and so on...

amir734jj5• 4 years ago

You are right. Wrong choice of words. Thank you.

CoastalCoder• 4 years ago

Excellent point regarding the need to make finer distinctions on the topic.

Also worth mentioning that depending on one's reasons / world view, those distinctions may or may not be significant.

devwastaken2• 4 years ago

There was not a federal abortion ban. Point to me a law that says abortions are federally illegal.

What was done is a current supreme court reversed a precedent that was created by a older supreme court. Roe v Wade. This precedent should never have been created in the first place, it was a violation of powers. If we want to force states into making it legal then that must be done at a legislative level.

Imagine if the current supreme court just said "nope abortions are illegal because constitution says murder is wrong somewhere and the founders didn't like abortion.". It has just as much merit. Therefore we must ensure they don't have the power. Regardless of whether we consider the result right or wrong.

blamazon• 4 years ago

And if abortion is first on the list of women's reproductive rights to take away, what's next? Affordable access to reliable birth control is currently enjoyed by many voters in this country.

amir734jj5• 4 years ago

Agreed. It's sad that we are even talking about this and it's even a possibility.

nozzlegear5• 4 years ago

The people replying to you arguing on semantics that abortion is not banned at the federal level are technically correct and conveniently omitting the fact that Senator Graham had attempted to introduce a federal abortion ban after Roe v Wade was overturned. The threat of a federal abortion ban was real, and was certainly on the mind of all the people who voted against the Republican party this election.

amir734jj5• 4 years ago

Yes. Thank you. I'm not very up to date with politics and what I wrote above is just my observation as a voter.

gfdsgfsdg2• 4 years ago

He may have introduced an abortion ban but the bill never passed and Republicans in Congress never voted for it. The poster above you said Republicans banned abortion at the federal level which is 100% wrong.

PathOfEclipse3• 4 years ago

So much deception and lies.

Graham introduced a 15-week limit on abortions, not an outright ban: https://www.dailywire.com/news/sen-lindsey-graham-introduces...

15 weeks is similar to the limit in France and most European countries, which lean far more left than the U.S: https://en.wikipedia.org/wiki/Abortion_in_France

Also: https://en.wikipedia.org/wiki/Abortion_in_Europe

France only allows abortions up to 14 weeks unless doctors certify the mother's life is a risk. Graham's bill similarly "offers exceptions for rape, incest, and the protection of the life of a mother during pregnancy." And The bill also "prohibits the prosecution of women who seek abortions."

The point is: framing the proposed law as a "ban" is disingenuous because no one thinks abortion is banned in Europe, and yet this law is completely in line with European abortion policy. But dishonest framing is what the democrat party is all about, unfortunately.

The reality is that abortion gets more obviously bad for both humans involved the later the abortion occurs.

https://www.liveaction.org/news/researchers-abortionists-abo...

"That study found that a woman’s risk of death from abortion increased by 38% for each additional week of gestation. Compared to women who underwent abortions at or before eight weeks (two months), women who underwent abortions in the second trimester were found to be “significantly more likely to die of abortion-related causes.” The results showed that “[u]p to 87% of deaths in women who chose to terminate their pregnancies after 8 weeks of gestation may have been avoidable if these women had accessed abortion services before 8 weeks of gestation.”"

And from the baby's perspective:

https://www.operationrescue.org/about-abortion/late-term-abo...

"a lethal dose of the heart medication Digoxin is injected into the baby’s heart or into the amniotic fluid directly through the woman’s abdomen. (Potassium chloride is infrequently substituted for Digoxin.) Digoxin gives the baby a fatal heart attack."

Not exactly a peaceful or painless way to die! However, the alternative method, dismemberment, is even worse!

https://www.frc.org/dismembermentabortion

If you want to abort, you should do it quickly, both to cut down on your own risks and to limit the pain and suffering of the human whose life you are ending. I would not like to experience death by heart attack or dismemberment either, if I can avoid it. And laws that protect the health of its citizens are clearly in scope. A late-term ban protects the health of adult women. That's one of the reasons these restrictions exist in other parts of the world.

rob742• 4 years ago

Overturning Roe v. Wade certainly played a part, but I think it was, more generally, moderate Republican and Republican-leaning voters saying that they're not happy with the course the Republican party is on: turning back the clocks by 50 years (not only on abortion - one justice also suggested revisiting other related decisions, e.g. the legality of contraception, same-sex relations and marriage: https://www.theguardian.com/world/2022/jun/24/clarence-thoma...), picking candidates by their loyalty to Trump rather than their competence and so on.

shagie2• 4 years ago

They didn't get it wrong - they were trying to understand if it was a better (or useful) model by carrying out an experiment by adding it to the other polling questions.

It turned out that the model developed from that question was a worse predictor, but that isn't wrong any more than carrying out an experiment that didn't match the expected results is wrong.

That they are coming back and looking at it again is good.

Note also that the people who are asking these questions (good questions, just asked wrong) come from an economic rather than polling background and getting the right wording for a question is very hard.

mcphage• 4 years ago

It turned out that the model developed from that question was a worse predictor, but that isn't wrong any more than carrying out an experiment that didn't match the expected results is wrong.

I don't know if that's entirely true. From the article:

Oddly enough, NPR just reported these numbers with a straight face, without saying anything about how off they seemed to be.

Which is to say, they performed an experiment. That experiment failed—which, as you say, is a totally acceptable outcome of an experiment. But when reporting their results, they don't seem to have indicated that (a) what they were reporting on was an experiment, or that (b) the results didn't seem promising, or in line with expectations. Instead, the article implied that they reported the results as if they were meaningful predictions.

shagie2• 4 years ago

Give the transcript a read - https://www.npr.org/transcripts/1134434712

---

FOUNTAIN: ... So, Barbara, we have a question for you.

CARVALHO: Uh-oh.

FOUNTAIN: Would you be willing to let us test one of these methods in one of your polls?

CARVALHO: Oh, sure.

FOUNTAIN: Really?

CARVALHO: Well, if they're - I mean, if they're appropriate, objective, you know, questions, that would be great.

GUO: So here's the exciting news. We worked with Barbara to draft two questions - one, where we ask people to predict the election and another asking them who the folks in their social circles are voting for.

FOUNTAIN: She said she would slip them into a poll Marist is doing before the midterms, run some analyses and let us know how useful they might be for the future of polling.

...

CARVALHO: When we look at who people think that their social contacts are going to vote for, we actually found among registered voters a plus-10 Republican...

GUO: That's a big difference.

CARVALHO: ...And a plus-14 among those who say that they're definitely going to vote. When we look at who they think is most likely to win the election in their district, we see among registered voters a plus-12 Republican.

FOUNTAIN: Wow.

CARVALHO: ...And a definite vote of a plus-16.

FOUNTAIN: The wisdom-of-the-crowds type questions are heavily favoring the Republicans, much more than the standard question, though Barbara is careful to mention that all of this is just preliminary, and the real test is when she puts all this data together and does some analysis. And besides, it's a little hard to figure out which questions are working better on a national poll. But...

GUO: If the election goes more toward the Republicans, is that a little bit more evidence for maybe the usefulness of these questions?

CARVALHO: Absolutely. And so what we will take a look at - and mind you, we're not ending the experiment here.

ggm2• 4 years ago

What might be wrong is 2nd order effects of other people building belief in the question.

Was it made clear in advance it was experimental?

xboxnolifes• 4 years ago

Why would that be important?

treis• 4 years ago

This just seems to be a mismatch of expectations. The Republicans won the election in that they are going to have more seats than they did before. People are saying they "lost" because they didn't win as much as people expected. To use a sports analogy, they won the game but didn't cover the spread.

zaphod12• 4 years ago

Yes absolutely. In 2010 - Obama's first midterm - republicans gained like 64 seats in the house. It was a massacre for anything with a D next to their name on the ballot, expect in the most solidly blue locales. This was not like that at all. More of a regression to the mean in most places.

tablespoon5• 4 years ago

This just seems to be a mismatch of expectations. The Republicans won the election in that they are going to have more seats than they did before. People are saying they "lost" because they didn't win as much as people expected. To use a sports analogy, they won the game but didn't cover the spread.

They under-performed very, very badly. It's like eking out a narrow win because (somehow) the sun was always in your opponents eyes and the wind was always at your back. Sure you got a win, but a win of that magnitude means you suck and will probably lose badly in the future.

cudgy5• 4 years ago

They might still get control of the Senate and the House, which is like winning the Super Bowl with a last minute field goal. They’ll take it if it happens and the teams (political talking points) will all be different in 2 years anyway.

jeff-davis• 4 years ago

Is accurate polling a good thing?

It's not clear to me that democracy is helped by knowing what everyone else thinks.

cudgy5• 4 years ago

It does not help. It basically serves several purposes:

1) forms a talking point for political speculators in media to get views and clicks

2) used by political strategists to figure out what lies they can tell voters

3) used by political pundits to scare voters into action and lull voters into inaction. “Oh no! The Republican/Democratic red/blue wave is coming!” This discourages voters of the perceived victorious party from voting and encourages voters from the perceived losing party to vote.

awinder• 4 years ago

“nobody — nobody — was predicting that the Republicans would win by 12 or 16 percentage points.”

An excess 14% thinking republicans will win doesn’t predict a 14% margin, it predicts 14% excess probability that republicans win, by some amount.

Even that I’d be skeptical over though. I think this question is basically measuring which media narrative has become predominant & by how much.

How did NPR’s pre-election poll get things so so so wrong?

Related Stories