Nate Silver Finds Unusual Patterns in a Polling Firm's Data. Was it fabricated?

http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html

by gluejar • 17 years ago

98 25 17 years ago

Particularly interesting is the further analysis done in the comments where similar statistical methods are used to analyse other polling agencies' results. Nearly every single one seems to line up to look almost exactly like the Quinnipiac results, further damning Strategic Vision. But what interests me more is the question of exactly why this distribution comes up; it's not Benford's Law, since probabilities aren't a power-law distribution, yet it seems consistent across multiple sources.

callmeed2• 17 years ago

With newspapers suffering so much, it's refreshing to see a blogger taking the time to perform investigations like this. We need more of this.

cwan7• 17 years ago

My sense has been newspapers have been suffering so much because they don't take the time to perform investigations like this. This is hardly the only example with the Acorn videos quite recently for instance. We do definitely need more of this.

callmeed2• 17 years ago

I was thinking along the lines of their decline in ad/classified revenue-which is used to fund their investigative journalism. Are you saying the quality of old media investigations has declined independently?

philwelch2• 17 years ago

The trouble with the Acorn videos is that they're a partisan attempt to discredit the opposition. While the videos themselves seem legit, I'd rather not leave exposes of political groups up to their political enemies. Political enemies only want to discredit their opponents--they're not interested in truth, and for every Acorn video there's a Swift Boat smear campaign with no basis in fact.

eli2• 17 years ago

Let's be fair; that's not really why papers are hurting financially. It was a sudden and drastic drop in advertisers, not people switching to blogs. Newspapers used to make lots of money on ads from national retailers, auto dealers, financial companies, and real estate. Those are four of the worst possible groups to have as your clients.

jonknee• 17 years ago

A true statistics hacker. I love reading his analysis--he was the go to source during the election for anything poll related.

isamuel• 17 years ago

This is pretty damning stuff. In a well-functioning media environment, it would be sufficient to seriously damage the credibility of the pollster involved (Strategic Vision), or at least shift the burden to them to explain their behavior.

Instead, we already see clues about what will happen: the pollster is threatening a lawsuit, which will be a sure loser but nonetheless drain resources and time from its target (either Nate himself or fivethirtyeight). The polls from the offending pollster will, I'm completely certain, be reported with utter credulity in the 2010 election cycle and beyond. The fact that the books are cooked will not even be mentioned in media reports, and will ultimately matter not at all.

This is because we simply do not have a well-functioning media environment.

cwan7• 17 years ago

I think their lack of disclosure has already severely damaged them.

Ultimately, in this case it would seem the damage will be pretty brutal unless Nate Silver was wrong. Either they don't sue and don't disclose and the result is no one trusts them enough to pay for their polling (after all, what's the point in paying for a poll if it's going to come under question afterwards?) or they'll sue while not releasing the support for their polls/data on their merits - and that also opens them up to discovery which could be even more damaging.

The best case scenario at this point is if they can publicly prove Nate Silver is wrong. And based on their policies in the past of not releasing the additional supporting data, I suspect this won't happen.

smhinsey• 17 years ago

It does nothing to help their credibility that they are basically riding the coat tails of another firm, called Strategic Vision Inc. (the firm he is challenging is Strategic Vision, LLC), which is something Nate mentions in one of his posts on the matter. That right there ought to serve as a pretty big negative indicator of credibility, but it doesn't seem to matter much to the media, if they even do enough research to notice that they aren't the "real" Strategic Vision.

danger• 17 years ago

Here's a bit more on the subject: http://news.ycombinator.com/item?id=846514

kevinpet• 17 years ago

It seems to me that if there were one data source I would expect to not be consistent with Benford's law it would be polling data. Benford's law is usually seen in numbers that follow some skewed distribution. Polling data groups around maybe 45% depending on the number undecided. I don't know what I would expect, but polling extremely close races, or polling close to the election, I would expect to see lots of high digits (6 7 8 9). I wouldn't expect many 41 or 31 (those questions are uninteresting) so the 1s come from mostly 51s.

He may be right, but I'm going to need a little more convincing if someone wants to assume that polling data would exhibit Benford's law.

Nate Silver Finds Unusual Patterns in a Polling Firm's Data. Was it fabricated?

Related Stories