Yes Prime Minister, questionnaire design matters

https://www.ipsos.com/en-uk/yes-prime-minister-questionnaire-design-matters

by admp • 2 years ago

72 30 2 years ago

tgv• 2 years ago

I'm afraid they wasted their sample. It's a large company, so it's probably filler for them, but still. If they had asked the same question at the end, the effect would have been clearer (assuming no big differences in audience, because that really happens). Now you can ascribe the effect to e.g. people misreading the "oppose" question.

milliams2• 2 years ago

I had a similar thought. They could otherwise have asked the two end questions from a cohort without the lead-up to get the baseline.

fweimer• 2 years ago

The questions were flipped in the original TV programme.

garaetjjte• 2 years ago

I think it is not representative anyway because of how famous the sketch was.

dr_kiszonka• 2 years ago

If, like me, you can't read the questions in the results table, here is a legible version: https://datawrapper.dwcdn.net/jZ5sT/full.png

changoplatanero• 2 years ago

Wait there is barely any difference in the response to the last question in both surveys!

milliams2• 2 years ago

This is a problem with the way the data is presented and the questions asked, but there's a 10 point swing between "not opposed" and "in favour", and likewise between "not in favour" and "opposed".

londons_explore• 2 years ago

There is a 15% difference, and that takes the overall results past the critical midway position that matters in a referendum.

kwertyoowiyop• 2 years ago

This show and its predecessor “Yes Minister” are excellent. And as we see, just as relevant today as ever.

mrtksn• 2 years ago

I wonder if people in politics or civil service feel the same about this show as the people in tech feel about Silicon Valley series.

These series are also scary because they somehow managed to remain relevant. Maybe the way the world works hasn't changed that much after all.

LaundroMat• 2 years ago

Us humans make the (human) world work and we haven't changed a lot in a very long time indeed.

tomatocracy• 2 years ago

If you like the series then this article[0] which was published in the LRB early in its run is well worth a read - it gives you a good idea of how the series came about and why it feels very closely observed. Reportedly many politicians at the time, including Margaret Thatcher, felt it to be very closely observed.

But the really clever thing about the writing is that you never know which party Jim Hacker is a member of and it is equally believable regardless of which one - it's so remarkable that it's a series about politics which manages to be really quite apolitical in that sense.

0. https://www.lrb.co.uk/the-paper/v02/n10/antony-jay/informed-...

arp2424• 2 years ago

And/or people have started behaving according to the standards set in the show because they assume that everyone is that cynical. This is kind of what worries be about these extremely cynical satires and depictions.

On the other hand, it's not like politics is better in the US which had The West Wing, which is very aspirational and not cynical at all so idk...

throwup2382• 2 years ago

Based on anecdata from people in the UK and Australia, the Australian show Utopia [1] is the most realistic take on the civil service while The Thick of It is the most realistic take on party machinations.

Some civil servants find Utopia too painful to watch because it hits too close to home. The show has made several very prescient predictions [2].

[1] https://en.m.wikipedia.org/wiki/Utopia_(Australian_TV_series...

[2] https://www.news.com.au/sport/sports-life/abc-comedy-series-...

matt3210• 2 years ago

This is a feature not a bug in questionnaire campaigns.

arp2424• 2 years ago

I don't agree with the conclusion; 3% difference is basically within the error margins for these types of things. I typically assume a ~5% error margin, which is nice round number I just pulled out of my ass, but it seems roughly accurate.

cool_dude85• 2 years ago

Re-read the question. The first one is 45-favor, 38-oppose, the second one is 48-oppose, 34-favor.

IshKebab2• 2 years ago

No it isn't. The first one is 45-favour, 38 don't favour. That's a totally different thing.

This proves the idea that you can get a result that points the way you want it to. But it doesn't quite prove that the leading questions have any effect.

To do that they would need to ask 2 more sets of questions with the "oppose"/"support" questions inverted. Otherwise the effect could just be due to the different wording of the final question.

quatrefoil• 2 years ago

You're misreading the answers. In sample A, 45% are in favor of reintroducing National Service and (38% oppose it). In sample B, the phrasing is reversed, meaning that 34% are in favor (and 48% oppose it). So, the difference is 11 percentage points.

IshKebab2• 2 years ago

(38% oppose it)

No, you're misreading too - see my other comment.

arp2424• 2 years ago

Eh right, do'h >_<

But in that case you need to add a control for both questions without the prior four questions. How do you know the difference in phrasing doesn't matter?

loudandskittish• 2 years ago

"First of all, to be clear, we would never ask questions on such a topic in this way."

I love how they worded that. Having once worked for Ipsos, they DEFINITELY do it, but the sentence is correct, I don't recall any instances of it being done "on such a topic."

toyg• 2 years ago

The art of pollsters is to produce results reflecting the hopes of whoever commissioned them, but with just enough even-handedness to look not-obviously-prepackaged.

xyzelement• 2 years ago

I had a survey construction class during my MBA. This is mainly obvious stuff, but indeed should be studied and practiced because people screw this up all the time. Assuming your goal is to find out what people think (vs, have the survey tell you what you want it to tell you) - you have to realize that most people plop down their at-the-moment emotional state as the survey response, and that state is most recently influenced by the survey itself.

Eg, if I asked you "should Israel dismantle the Hamas terrorist organization responsible for the 10/7/2023 murder of 1500 people" - most people would say "yes" because the words "dismantle the terrorist organization" resonate with most people.

If I asked them "do you support a ceasefire in Gaza" - most people would also say yes, because the word ceasefire would resonate with them.

Obviously these would be contradictory results. What this survey wouldn't tell you is how many people actually think about this topic and which way they lean.

A better designed survey would be something like this:

Question 1: How aware are you of the current conflict in Gaza (1-5) Question 2: In your own words, how would you describe the conflict? Question 3: If 1 is "Ceasefire at all costs" and 10 is "eliminate terrorists at all costs", where would you place your opinion?

Having these 3 datapoints would enable you to filter by those who actually have an opinion (or, find out what percentage has an opinion), use question 2 to validate that and/or gain new insights and use 3 to identify how strongly people lean one way or another.

Doxin• 2 years ago

Do note that your two example questions aren't quite as diametrically opposed as you make it out to be. You can be in favor of both and still be consistent.

How you would dismantle Hamas while having a cease fire in place is left as an exercise to the reader.

PoignardAzur• 2 years ago

Not only that, but "do people consider these options to be mutually incompatible" is itself interesting information.

I wouldn't expect the average survey to handle it too well, though.

Yes Prime Minister, questionnaire design matters

Related Stories