I disagree with Turing and Kahneman regarding statistical evidence (2014)

https://statmodeling.stat.columbia.edu/2014/09/03/disagree-alan-turing-daniel-kahneman-regarding-strength-statistical-evidence/

by acqbu • 4 years ago

129 50 4 years ago

statmodeling.stat.columbia.edu

woliveirajr• 4 years ago

@dang: the original title should be "I disagree with Alan Turing and Daniel Kahneman regarding the strength of statistical evidence", the way it was edited it became... clickbait ?

function_seven• 4 years ago

Probably done to get it under 80 characters.

Maybe

    I Disagree with Turing & Kahneman Regarding the Strength of Statistical Evidence

will work? It's exactly 80.

pvg• 4 years ago

Email this stuff in if you want it to reach a dang.

kqr2• 4 years ago

The conclusion is a bit at odds with the rest of the piece:

And that’s interesting. When stupid people make a mistake, that’s no big deal. But when brilliant people make a mistake, it’s worth noting.

Maybe not. In the words of the author,

maybe these are real effects being discovered, but you should at least consider the possibility that you’re chasing noise. When a striking result appears in the dataset, it’s possible that this result does not represent an enduring truth or even a pattern in the general population but rather is just an artifact of a particular small and noisy dataset.

pessimizer• 4 years ago

Clearly brilliant people saying that something that later turns out to be noise is statistically undeniable signal is worth noting. I'm not sure what you're seeing here.

kqr2• 4 years ago

I don't think it is worth noting. Brilliant people make mistakes too, just not at the same rate as the rest of us. That one or even a few make the a specific mistake means nothing -- it's statistical fluke, even if it seems noteworthy in isolation.

ev7• 4 years ago

Kahneman admitted that some of the studies his book cited were underpowered here:

https://replicationindex.com/2017/02/02/reconstruction-of-a-...

arolihas• 4 years ago

Reminds me a lot of Beware the Man of One Study by Scott Alexander

https://www.lesswrong.com/s/BQBqPowfxjvoee8jw/p/ythFNoiAotjv...

mannykannot• 4 years ago

Perhaps one should also beware of the study by one man. I have a vague recollection of hearing that, at the time Turing was expressing his opinion of telepathy, there seemed to be experimental evidence for it, but it turned out that all the evidence came from one person, using unsound methods (including, IIRC, counting "one before" or "one after" guesses as successes if the results had higher than the expected frequency of one of those outcomes.)

Tentatively, that person may be Joseph Banks Rhine.

googlryas• 4 years ago

Yes - I had to read into that part of Turing, since I had never previously come across his views on ESP, but it seems like his error was not that he believed the statistics to be more powerful than they were - but rather he was too credulous in believing there weren't charlatans intentionally fudging the stats in order to help their point. Basically - the stats were powerful, but they were just fake.

ChrisArchitect• 4 years ago

Discussion from 8 years ago: https://news.ycombinator.com/item?id=8263243

randcraw• 4 years ago

It’s good to have an open mind.

Is it? That's the very basis for error that begat this article. In fact, Turing's and Kahneman's minds were too open. They didn't express sufficient reservation by demanding more rigorous tests of those claims.

Perhaps a better maxim would be, "It's good to have a mind that's open just enough to entertain the impossible."

clint• 4 years ago

Its funny, I considered Turing's state of mind to be not open enough to the possibility that he might be wrong.

n4r92• 4 years ago

My thoughts exactly. "This claim is true and you mustn't disagree" is quite closed-minded.

arto• 4 years ago

"By all means let's be open-minded, but not so open-minded that our brains drop out." -- Richard Dawkins

eyelidlessness• 4 years ago

An appropriately simplistic quote from someone who habitually and sometimes famously demonstrates being stubbornly narrow minded. But then, how else would he not know that brains don’t fall out of—nor are placed in—minds.

bloak• 4 years ago

That saying, or ones like it, are also attributed to Carl Sagan, G. K. Chesterton, James Oberg, Arthur Hays Sulzberger, Virginia Gildersleeve, Bertrand Russell, Max Radin, ...

This article mentions Walter M. Kotschnig as the author of the earliest known instance of the saying, but notes that even Kotschnig seemed to be quoting something that was already known at the time:

https://www.skeptic.com/insight/open-mind-brains-fall-out-ma...

tpoacher2• 4 years ago

I prefer Terry Pratchett's quote:

"The problem with having an open mind is that people will insist on coming along and trying to put things in it."

mr_toad• 4 years ago

In addition to statistical evidence people should really examine the plausibility of the mechanism in question. Most accounts of ESP have no description of how it might actually work. The theories are typically dualistic, vitalistic and even plain magical.

xyzzyz• 4 years ago

What kind of “mechanism of action” you are looking for? I mean, physicists provide no “mechanism of action” for the forces they stipulate to exist. There is no “mechanism of action” for electromagnetism, it just does what it does. Similarly, ESP force might not have “mechanism of action”, it just could do what it does.

What you are doing here is that you are betraying two huge hidden assumptions that you implicitly make. First is that you are most likely expecting to obtain “mechanism action” in some kind of mathematical terms, which implicitly assumes ability to use mathematics to explain the observed universe. Second is that you are looking for “mechanism of action” for ESP in terms of known physical forces and theories. Here, you implicitly assume that known physics is all there is.

Now, I do not believe in ESP either, my point is only that you are demanding rigor from ESP you don’t from electromagnetism.

sebastialonso• 4 years ago

imo having an open mind means rather exclusively to allow the possibility to be moved by scientific evidence. Not just believe for the sake of it or because of some kind of bias If we don't keep an open mind, in this sense, we'd still believe the Sun rotates around the Earth.

glacials• 4 years ago

Hard not to chuckle at:

People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psvchological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions.

(emphasis mine)

Not only does the statement say that “people” make this mistake then go on to cite a questionnaire only of professional psychologists, but the questionnaire is the prototypical non-random random sample the very psychologists taking it are allegedly proving misinformation about. Questionnaires select for people with the time, inclination, and attention to start and then finish a questionnaire.

Infamously, experiments held at universities bias towards undergrads. Experiments that reward participation bias towards people motivated by the reward. Experiments that don’t reward participation bias towards people good-natured enough to contribute just for science.

Randomness is hard.

pixl97• 4 years ago

Using undergrads is how you get W.E.I.R.D. science

autosharp• 4 years ago

Certainly very ironic. But is this line of argument incorrect?

E.g. if I said "I don't believe in the Holy Book because in verse 7 it says that one cannot trust anything written in any books". Isn't that an analogous reasoning?

I think these examples/arguments are ultimately about exposing a liar-paradox statement, and when you can show such a statement you have proven that something isn't right.

raverbashing• 4 years ago

Randomness is hard.

Yes. And might I say everything has a statistical bias. Picking people for an experiment is not like throwing dice

Your sample will always be biased, and it would be crazy to try and get a random sample from "the world". (and even then you have temporal bias - you can't run your experiments on people from the XIX century)

jstx12• 4 years ago

Kahneman's full quote is just embarrassing:

The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you.

So so arrogant, and he ended up being wrong about it too. It's very hard to take him seriously after reading this.

klysm• 4 years ago

How do you end up with such a fundamentally flawed epistemology?

FabHK• 4 years ago

I'd say that it wasn't the epistemology that was flawed, but the studies. (Though, ok, arguably it was a flaw in the epistemology that it didn't allow for flaws in the studies.) But the more interesting question is how the initial studies could have been so flawed. At any rate, those results were not replicated, and science marches on.

neuronexmachina• 4 years ago

It's interesting that his "proposal to deal with questions about priming effects" was just a year later (2012): https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/s...

My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction.
I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is selfdefeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field.

His more recent (2022) retrospective on the episode is also pretty interesting: https://www.edge.org/adversarial-collaboration-daniel-kahnem...

austinjp• 4 years ago

I think you're somewhat misinterpreting his point. He's not saying "I'm correct whether you like it or not". Instead, he's saying "we are all prone to cognitive errors, whether you like it or not".

Admittedly, the experiments he conducted and cited were found to be flawed. But I think his point, as I understand it, still stands.

tacoooooooo• 4 years ago

it almost seems even more relevant given the outcome of his experiments...

bobthechef2• 4 years ago

He's not saying "I'm correct whether you like it or not". Instead, he's saying "we are all prone to cognitive errors, whether you like it or not".

Restated: "I'm correct, about us all being prone to cognitive errors, where you like it or not."

goatlover• 4 years ago

It sounds like he's saying his students were wrong to express disbelief over the counter intuitive findings of a study because the statistics were such that it left no option but to accept the results. Which is arrogant and wrong. Studies can be faulty or fail to replicate, statistics can be misapplied or fudged, future studies also contradict prior findings, etc.

Blind faith in the results of a study is not good science. If a study is counter intuitive, one should question it.

tgv• 4 years ago

The point doesn't stand, at least, not for psychology papers: even if the data have been collected correctly, the method was flawless, and the statistics have been done properly, the conclusions (as written in the paper under "Conclusions" heading) are always beyond the scope of the experiment, often far beyond it. I've never seen a psych paper that limits itself to just the experiment. They always present the outcome as proof for some kind of broader theory (which usually has been proven wrong in the same style by somebody else).

jbullock35• 4 years ago

Kahneman's full quote is just embarrassing

Yes. It is. But as another poster has noted, he largely recanted in 2017 [1]. And this is to his credit. Few scholars -- and fewer prominent ones -- have been willing to reverse course in as public a fashion as this.

[1] https://replicationindex.com/2017/02/02/reconstruction-of-a-...

mrow84• 4 years ago

“The argument is inescapable: Studies that are underpowered for the detection of plausible effects must occasionally return non-significant results even when the research hypothesis is true – the absence of these results is evidence that something is amiss in the published record. ”

This is a great insight.

jstx12• 4 years ago

I am still attached to every study that I cited, and have not unbelieved them

So he admits that the studies don't replicate but he still believes in their conclusions? Somehow that makes him look worse, not better, in my eyes but maybe I'm too critical.

diffeomorphism• 4 years ago

so, so arrogant.

Yeah, your reaction is. The whole point was to be humble and accept that you might be wrong. In particular, if well-done research contradicts your prior belief, then you "have no choice" but to consider that you might have been wrong.

It turns out that the "well-done" was not the case here, but I find it hard to think your post was in good faith.

tome• 4 years ago

But Kahneman didn't say "no choice but to consider" he said "no choice but to _accept_" which is a much stronger statement.

pfisherman• 4 years ago

How could Turing have thought this? I don’t know much about Turing but it does seem, when reading old-time literature, that belief in the supernatural was pretty common back then, lots of mention of ghosts etc.

I think that today we overestimate how much we know, and discount the existence of “supernatural” phenomena that we don’t fit into our model of the universe.

Karma (or something like it) is a thing that probably exists; but we do not have the necessary concepts or tools to qualitatively or quantitatively describe how it works.

bobthechef2• 4 years ago

I'm not sure what you mean by karma in this case, but I wanted to note that the word "supernatural" is very often misused. Once you start prodding people about what they mean by "supernatural", you quickly discover that they don't know what "natural" or "physical/material" mean, much less what "supernatural" means, and the whole conversation descends into grumbling.

Here's one definition of "supernatural order" [0].

[0] https://www.newadvent.org/cathen/14336b.htm

CyberDildonics• 4 years ago

The only reason to believe that karma exists is that we have an emotional desire for it to exist.

kingludite• 4 years ago

I doubt we can get places if the reader of this comment ws to produce thoughts about what I think about what the author thinks psychologists think.

I'm not comfortable with the idea the unmatched creative engineering powers of mother nature did not discover something as simple as radio.

We could power down the electrical grid and do the experiment? But I can just call everyone, what use is telepathy?

The cat is in front of the house when someone arrives (which is never at the same time sometimes skipping a day) but the rest of the day he is nowhere near on the camera footage. For him there is a point, it means food!

Sheldrake would love it.

amai• 4 years ago

Kahneman and his book „Thinking fast and slow“ are totaly overrated. In fact „despite Kahneman's previous contributions to the field of decision making, most of the book's ideas are based on 'scientific literature with shaky foundations'. A general lack of replication in the empirical studies cited in the book was given as a justification.“ Unfortunately you will find that statement only at the end of https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow. It should be stated at the beginning. So many wrong assumptions especially in economics, but also management theory are based on this book. But nothing of is book can be replicated: https://replicationindex.com/2020/12/30/a-meta-scientific-pe... Unfortunately far to few people know about this.

tpoacher2• 4 years ago

"Whenever there is a simple error that most laymen fall for, there is always a slightly more sophisticated version of the same problem that experts fall for."

--Amos Tversky

I disagree with Turing and Kahneman regarding statistical evidence (2014)

Related Stories