Psychology Journal Bans Significance Testing

http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/

by tokenadult • 11 years ago

152 82 11 years ago

phren0logy• 11 years ago

There was a question yesterday about Evidence Based Medicine vs Science Based Medicine. The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help.

For example, EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice. The issue is that, as this article points out, even with a statistically significant result for such research, there's no plausible mechanism by which a rabbit's foot makes you lucky. Therefore the RCT is just one part of the whole picture, and subject to a specific type of manipulation.

srunni• 11 years ago

Something related to keep in mind when it comes to p-values is the false discovery rate:

If you run lots of phase 2 trials with different drug candidates where only a minority (lets say 10%) actually work, then with standard trial statistics (80% power and 5% false positive rate) you will get 4.5% false positive and 8% true positives – so less than 2 out of every 3 positive trial results were real. A much lower success rate than the 5% error rate commonly assumed.

http://www.forbes.com/sites/davidgrainger/2015/01/29/why-too...

This recent (open access) review in R. Soc. Open Sci. the Forbes article was largely inspired by is worth a read too: http://rsos.royalsocietypublishing.org/content/1/3/140216

Alex39173• 11 years ago

The SBM criticism of EBM is the over-reliance on Randomized Controlled Trials that meet p=0.05, without looking at the prior probability that a treatment would help.

The strength of significance testing is that it purposely doesn't try to tell you how likely something is to be true, only how likely the data you got was the result of chance assuming the treatment is no better than placebo. You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it.

My concern with bayesian approaches is that, like with frequentist approaches, the truth is still fundamentally unknowable, only now you're encouraged to put a number on that and pretend that it's science. While bayesian approaches totally make sense in trying to determine a patient's likelihood of having some disease when there is already data available for the prevalence in a population and the sensitivity and specificity of the tests, using bayesian logic to weight clinical trials strikes me as being highly dubious.

It would be one thing if SBM actually developed a framework to give a weight to each methodological feature of a trial, but so far I haven't seen much work to build a functioning system. Though if you're really honest about all the ways that you can have positive results without something actually being true, it seems like almost no amount of research will ever have a significant effect on the prior.

mdbco4• 11 years ago

You're absolutely correct that Bayesian approaches are not magical and do not suddenly supply you with vastly more information than frequentist approaches (particularly when you have a really poor prior, in which case the Bayesian approach will be similarly poor). Bayesian statistics is certainly very popular right now, but it should not be looked at as some sort of panacea for all statistical problems.

However, I would say that Bayesian approaches do have a big advantage in terms of helping with the interpretation problems that plague frequentist significance testing. Namely, as the OP article points out, Bayesian approaches reformulate the testing question in a way that is more intuitive, i.e. "what is the probability of the hypothesis given both the prior probability and the new data?". So yes, Bayesian methods surely do not fix everything, but since interpretation of statistics is such a major concern, they can be quite beneficial.

yummyfajitas8• 11 years ago

You're still taking the prior probability into account when trying to figure out the truth, you're just not putting a number on it.

This is a danger sign - you are doing the same things the Bayesians do, just informally, less explicitly, and probably incorrectly.

The fact is that to make a good decision, eventually you need to compute a single number. This is an elementary fact of topology:

https://www.chrisstucchio.com/blog/2014/topology_of_decision...

That number will be based on some unproveable assumptions. That's a fact of Godel's incompleteness theorem, if nothing else. So given this, why is it "dubious" to make those assumptions explicit and obvious?

WhitneyLand2• 11 years ago

But how could a randomized clinical trial make this mistake? What experimental design could make the rabbit's foot look good?

"EMB would say that if you have an RCT that shows that a lucky rabbit's foot works, then you have reasonable evidence to put that into practice."

sunir• 11 years ago

You have heard "19 times out of 20" described in the news? That is the 0.05 restated for laypeople. 1 time out of 20 you will get a false positive, in this case that the rabbit's foot worked.

analog31• 11 years ago

I wonder if it just boils down to this: Exclusive reliance on any single tool by an entire field for a long enough time period will eventually lead to a proliferation of bad results.

rcthompson3• 11 years ago

More like exclusive reliance on a single tool will lead to a requirement for everybody to use it, even if they don't understand how to use it properly, which leads to both unintentional misuse (by experimenters who don't understand it) and the inability to catch intentional misuse (because adjudicators don't understand it either).

po• 11 years ago

The audio is a bit quiet but the 'dance of the p values' video [1] linked to by the article is absolutely fantastic and well worth watching.

[1] https://www.youtube.com/watch?v=ez4DgdurRPg

MengerSponge• 11 years ago

Good! This is a big step in improving everybody's work. If you aren't up on this research, here are the two articles that you need to read.

Why Most Published Research Findings are False: http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

Revised Standards for Statistical Evidence: http://www.pnas.org/content/110/48/19313.short

mdbco4• 11 years ago

The article is certainly correct that p-values and confidence intervals (or confidence sets, in multi-dimensional contexts) are widely misunderstood, not just in psychology or other social sciences, but in the hard sciences as well. The problem is even worse when you look outside of academia at common practices in more applied settings.

As suggested, a good approach is to take p-values not as conclusive or decisive, but rather as a tool that must be supplemented by other statistics. In particular, the article emphasizes Bayesian methods, which can certainly provide additional information, but this approach can also be rather limited when priors are not well-defined or are entirely unknown, which is unfortunately often the case in many problem domains.

One potential question is how to determine the nature of the distinction mentioned in the conclusion between "preliminary research" and "confirmatory research", particularly in cases where statistics provide the primary evidence, as in, e.g. psychology. Further studies in the same vein as the preliminary research can certainly provide additional supporting statistical evidence, but this doesn't escape the problem that all of the evidence is probabilistic in nature. The key issue here is that since statistical approaches can only give probabilistic evidence that a hypothesis is correct, then they strictly cannot tell you what is certainly true, so even confirmatory research is quite open to falsification. So we wouldn't want the label of "confirmatory research" to somehow suggest to the public the idea that it is certainly correct.

esfandia• 11 years ago

Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors? When someone tests positive for a cancer test, the priors are known (probability of cancer in the general population is usually very low, and the false positive rate of the test may be relatively high), and so usually that first test is merely indication that further tests are needed. So when you don't know the prior and you observe a low p-value on something, isn't that just "preliminary research" that needs to be further confirmed with other methods or at least the same test but using other data?

mdbco4• 11 years ago

Is the cautious approach then to treat a p-value in the absence of priors on the same level as a p-value in presence of unfavorable priors?

In the presence of a poor prior the Bayesian probability would be biased in some way, so frequentists would say that the p-value in the absence of priors is actually superior in this case. Bayesians would reply that if they thought the prior might be poor then they would simply consider multiple different priors, but it's not clear how this would improve things much over the frequentist approach that simply assumes that the prior is unknown.

So when you don't know the prior and you observe a low p-value on something, isn't that just "preliminary research" that needs to be further confirmed with other methods or at least the same test but using other data?

Yes, when you observe a p-value with low significance it should definitely indicate to you that more testing is necessary, either by using different testing methods, gathering new samples, or even just increasing the original sample size if that's possible. What I was trying to suggest in my last paragraph was that this should be the case even when we have highly significant p-values, because even significant p-values are not decisive. So even when we have "confirmatory research" that is highly statistically significant, we should still do all of the things that we would do when we have a p-value with low significance. It is sometimes the case that this subsequent research will overturn even very highly statistically significant results (though often this is unfortunately because mistakes in the original statistical methodology are uncovered).

Kenji• 11 years ago

The problem is this: You need a solid background in statistics and some mathematics to perform these tests properly. Most apparently, people from that field lack these skills. If someone can't hit a nail on the head with a hammer, blaming and banning the use of hammers won't solve a thing.

manicdee• 11 years ago

Taking hammers away will solve the problem of holes in things which aren't supposed to have them, including thumbs, walls, people's heads, and candy jars.

zmjones• 11 years ago

there are just other tools that will get abused. bayes factors for example

viraptor2• 11 years ago

Depends what are you trying to solve - people's understanding of p-values, or many false results? If you care about getting true results and there are easier ways to get there than p-values, why would it be a bad solution?

Similar thing to what happens in software security. Sure, you could tech people how to use memory management properly - and then watch them fail again and again. Or you could just provide an alternative solution like rust which changes the issue completely.

VieElm• 11 years ago

I kind of which that the widely read publications, like newspapers, that have turned out science articles based on all these now pretty much discredited research articles would issue retractions or at least clarification for each of the articles that has been challenged in this regard.

CountBayesie• 11 years ago

The type of analysis being banned is often called a frequentist analysis

I find that there is a trend of associating "bad statistics" with "Frequentists Statistics" which isn't really fair. If you found a statistician trained only in Frequentist methods and asked their opinion on experiment design in psychological research they would likely be just as appalled as any Bayesian.

I'm a big fan of Bayesian methods, but the solutions of "we'll solve the problem of misunderstanding p-values by removing them!" is still a problem of misunderstanding p-values! The misunderstanding is the issue, not the p-value.

colechristensen• 11 years ago

I think what this journal doing is probably a good thing, but only as the lesser of several evils. The truth is something more like... it is easier for soft sciences to abuse frequentist statistics than bayesian. Both have merit, it cannot be argued, but it is simply easier to produce meaningless conclusions with frequentist statistics done wrong.

This situation is so bad that it merits banning frequentist for this journal and I think that's reasonable. This doesn't mean that every journal in every field should, but perhaps it will be a useful temporary measure to improve quality.

stdbrouw6• 11 years ago

The problem is that p-values are begging to be misunderstood, and in fact you cannot use them as a decisionmaking procedure without "misinterpreting" them – after all, you're deciding whether to accept the hypothesis P(HA|D) based on 1-P(D|H0) on the grounds that, while they're not the same, they're proportional. (In that sense the p-value is like the poor man's likelihood ratio.) There's nothing wrong with p-values as a concept, but there's everything wrong with p-values in hypothesis testing. The misunderstanding is baked in.

im3w1l• 11 years ago

You can update your posterior based on the p-values yourself though. "Well those eggheads may have disproved X, but X is just common sense, so I'm gonna keep believing it anyway. U-until I see more studies confirming the finding I mean."

rpglover64• 11 years ago

I think the problem is not "if you find a frequentist (as opposed to bayesian) statistician", but "if you find a frequentist (as opposed to bayesian) e.g. biologist".

Non-statisticians have been trained using bad, frequentist methods, and one way of forcing them to retrain is by forcing them to learn new statistical tools to get published.

rcthompson3• 11 years ago

I spent about a hour explaining p-values to a fellow graduate-level researcher a few weeks ago. I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it. In undergrad, hypothesis testing was more or less taught as "do this inscrutable calculation and if the result is 0.05 or less, you win". The point is, in my experience, a lot of people really don't get p-values, even graduate-level and post-graduate scientists.

viraptor2• 11 years ago

I agree. I'm in the pool of people who don't really understand p-values even after taking 2 statistics courses. The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it. It may be a wrong number and you'll never know. This is completely different to many other practical applications of math where you can verify your result in various ways, or validate your answer against the initial assumptions, or test your program against lots of inputs.

mdbco4• 11 years ago

It's actually entirely possible to do some "checking of your answer" for p-values, as well. As you mentioned, for practical math, you can often validate your answer against the initial assumptions. This is true for statistical testing too, as it typically relies on many theoretical assumptions. So what you can do in practice is propose a different set of assumptions, perform the hypothesis test in a manner that follows those new assumptions, and see if you obtain a similar result. Typically, for any given hypothesis that you want to test, there are several possible methods for performing that test, so you can redo your test many times. This is one type of robustness checking, which includes many other things as well (e.g. running your test over subsamples or resamples of the data, checking for sensitivity to outliers, etc). Good statisticians generally like to do lots and lots of robustness checking.

yummyfajitas8• 11 years ago

The main issue I've noticed is that compared to many other calculations, you just have no idea if your result is correct. You can make wrong assumptions, wrong calculation, apply wrong methods, but in the end you get a number... and that's it.

The same is true in Bayesian statistics, and even simple formal reasoning with no statistics in sight. If you make wrong assumptions, you'll get the wrong result.

The only thing you can expect statistics to do is help you change your opinion about the relative merits of opposing theories. If both your opposing theories are wrong, you will still be equally wrong.

The true flaw with frequentist statistics is that it goes out of it's way to hide this fact from you. In contrast, Bayesian stats forces you to explicitly choose a prior, enumerate your assumptions, and accept that your conclusion is based on these things.

NamTaf2• 11 years ago

Let me replay it to you and see if I understand it, because I don't know if I do:

You assume a null hypothesis that (usually) represents the status quo of no influence between the theory and the data. You then collect data. The p value then describes the probability of that data aligning with / being as a result of the null hypothesis. In other words, a p <= 0.05 says that you have <5% chance that the data came from the theory stated in the null hypothesis - that is, you have a 95% confidence that you can reject the null hypothesis in favour of your new theory.

Is that correct? I may have minced terms there because my stats training is woefully inadequate, but I think I adequately conveyed the concept?

Tloewald2• 11 years ago

No there's a 5% chance the null hypothesis is correct. the probability your theory is correct is unknowable.

yummyfajitas8• 11 years ago

Not quite. The 5% represents the chance that if the null hypothesis is true, you would draw data at least as extreme as the data you just saw in a repeated experiment.

Computing the probability that the data came from the theory stated in the null hypothesis would require a (Baysian) prior.

Also, Tloewald's reply is completely and inexorably wrong. Tloewald seems to want a Bayesian answer, which frequentist statistics can't give you.

SapphireSun• 11 years ago

Just to add to what the others say, rejecting the null hypothesis is also not evidence that your alternative hypothesis is correct.

Often the hypothesis testing framework is stated something like:

    H0: µ = 0 (null hypothesis)  
    Ha: µ ≠ 0 (alternative hypothesis)

When you reject H0, it means that you can be somewhat confident that there was was some kind of distortion in your data that moved the mean away from (in this case) 0.

You can create a number of theories that purport to explain this mechanistically, but you'll often need particular setups like a randomized controlled trial that can eliminate alternative explanations. When you've eliminated all the competing reasonable hypotheses, there can only be one. If you can use that hypothesis to make non-obvious predictions, that's further proof that it's right as well.

Hypothesis testing is there to tell you when to take an effect seriously, it doesn't tell you whether your explanation is right outside of very carefully constructed circumstances (i.e. where if you see a particular effect, only one theory can explain it).

bartkappenburg• 11 years ago

We were forced not to think in probabilities during my econometrics education if we talked about confidence intervals.

Say you have a confidence interval of 95% or higher (p value <= 5%), then the right thing to say is:

95% of the variance within the data is explained by the model (you've come up with).

rcthompson3• 11 years ago

Others have already explained it well enough, but I'll just add a few important concepts. First, if you remember nothing else, remember this: the p-value has absolutely nothing to do with the alternative hypothesis. In the context of p-values, there is no alternative hypothesis; it doesn't exist. The p-value only says anything about the relationship of your data to the null hypothesis. The second thing to keep in mind is that any null hypothesis likely has lots of assumptions built in to it, and if any of those is violated, the associated p-value is invalid. So you should always be aware of what those assumptions are, and try to verify that your data satisfied then if possible. (Almost no one does this, of course.) For example, the standard t-test assumes, among other things, that your data is normally distributed and that all groups have equal variance. If either of these assumptions is violated, the test is invalid.

Lastly, if you're in a situation where you're performing a large number of tests, always correct for multiple testing (i.e. false discovery rate or some similar method). Also, you can take advantage of your large data set to construct a negative control (e.g. by shuffling samples, the exact method will vary) and verify that your chosen statistical test gives a flat distribution of p-values, which is the expected result when all of the null hypotheses are true. If you have an excess of small p-values in the null data set, this indicates your test is producing false positives and is not reliable (presumably because one or more assumptions have been violated).

Al-Khwarizmi2• 11 years ago

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph:

"the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interval. Rather, it means merely that if an infinite number of samples were taken and confidence intervals computed, 95% of the confidence intervals would capture the population parameter"

And I say, "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?"

Where was the catch? I remember there was one (which is enough for practical purposes because it means I won't say in a paper that the probability of the parameter being in the interval is 5%) but I feel dumb for not being able to see at first glance something that is supposed to be basic statistics...

Al-Khwarizmi2• 11 years ago

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time, right?

If I have a variable that is always positive, then I could have a weird procedure to generate confidence intervals that gives me the interval [-inf,0] 5% of the time and the interval [0,inf] 95% of the time.

This would meet the definition of confidence intervals perfectly, and yet when I get [-inf,0] the real probability of the parameter being in the interval is 0%, and when I get [0,inf], it's 100%.

I wonder how large this discrepancy may be in practice (as this is obviously a made-up extreme case).

stdbrouw6• 11 years ago

"The probability of this parameter being in the interval" is what's called a credible interval, and it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely.

In general, frequentists just don't like to talk about the properties of this particular sample, only about long-term frequencies – hence the name. Why? Because they object to the idea of probability as a degree of belief rather than as an objective measure, and given that attitude the statement that "there's a 95% probability the parameter is in this interval" doesn't make any sense: either it's in the interval or it isn't.

logicallee2• 11 years ago

I pretty much just kept rephrasing the definition in slightly different ways until the person finally got it

well don't keep us hanging... (what was it?)

cafebeen• 11 years ago

Here's the key passage:

<quote> BASP will require strong descriptive statistics, including effect sizes. We also encourage the presentation of frequency or distributional data when this is feasible. Finally, we encourage the use of larger sample sizes than is typical in much psychology research, because as the sample size increases, descriptive statistics become increasingly stable and sampling error is less of a problem. </quote>

In other words: report the effect size, plot the data, and increase the N.

torgoguys• 11 years ago

Obligatory xkcd: http://xkcd.com/882/

jwilliams• 11 years ago

I think this Wikipedia section summed it up really well for me: http://en.wikipedia.org/wiki/P-value#Alternating_coin_flips

Extreme example, but explains how the context can shift the findings significantly.

kriro• 11 years ago

This is very interesting for me. I recently switched from AI to Human Computer Interaction which is more cross discipline, specifically with a strong influence from psychology. I'll gladly admit that I don't have the best understanding of NHST but I think I understand it well enough.

Interestingly enough there was a post on HN somewhat recently about Bayesian alternatives (BEST). The paper that was linked was: ftp://ftp.sunet.se/pub/lang/CRAN/web/packages/BEST/vignettes/BEST.pdf

And the recommended book I settled on was by the author of that paper (Doing Bayesian Data Analysis)

I feel like I'm "ahead of the curve" thanks to HN :D

stdbrouw6• 11 years ago

Banning p-values makes sense to me, as they force you to declare an effect as either significant or not significant, rather than looking at the preponderance of the evidence and building knowledge over multiple experiments. It also leads us to focus on statistical uncertainty at the expense of all of the other kinds of uncertainty researchers are faced with: do I have the theory to back this up, am I actually measuring what I am trying to measure, is this coherent with other findings in the field? I do think the editors might be right when they say banning p-values will make the quality of the research go up, not down.

But if you read the original editorial, up at http://www.tandfonline.com/doi/pdf/10.1080/01973533.2015.101..., you can see that they also reject confidence intervals and Bayesian reasoning with uniform priors (which is really the same thing) without providing any guidance at all on better procedures. I fear that will just lead readers to try and guess the reliability of the data themselves, or worse, interpret the sample statistics as numbers without any associated uncertainty.

So they're doing away with poor statistical procedures, but at what cost? It's like that old joke: we've found a 100% reliable cure for cancer – bombing the planet until everybody's dead.

nabla9• 11 years ago

Methodological flexibility in statistical modeling has gone trough the roof. New tools and estimation methods make Bayesian methods easy to use: Jags and Stan, Hamiltonian Monte Carlo, the variational Bayesian approximations, expectation propagation and even probabilistic programming (note: Bayesian in this context does not mean subjectivity. It's the ability to quantify uncertainty in the results and increased flexibility in the modeling).

New methods and tools are faster to use and give better results, but they require more statistical knowledge. If the scientist applying these methods or peer review can't understand the advances, it's all for nothing. Many sciences are methodologically very conservative to the extent that it holds the science back.

How do you increase the statistical knowledge of the field so much that peer review and researchers can be expected to understand and use the new methods if they can't be trusted with p-value?

Booktrope4• 11 years ago

If the stats aren't based on samples of real-world data, rather than very rigorously designed experimental comparisons, I think there's another reason to be wary of p-values. [Note, in what follows I probably use some terminology wrong, because I'm not a statistician, but I do think the point is important and I don't see much written about it.] In the real world data is not a bunch of independent events, but events (or data points) that are interconnected in ways that are difficult to quantify. I once heard a presentation by an expert in AB testing at a leading tech company who became wary of his results and brought his concerns about non-independence of events that were being tested to corporate statisticians. (He was concerned that, even though the AB testing procedure supposedly randomized the test, interconnectedness within website traffic was not being accounted for.) By his account, the statisticians agreed it was a problem but recommended that he assume that variations due to non-independence would more or less balance each other out. He wasn't satisfied with this and said that he went ahead and did some more measurements, then ran out all the binomial expansions rather than relying on approximations. When he did this more detailed work, he found out that with at least some web-based AB tests, where conventional statistical formulas showed a p-value of .05, he was getting a confidence level of more like 30%. (I don't think you could call his measurement a p-value because he wasn't using the formulas normally used to compute p-values, but I think what he was saying was more or less that the p-value from the forumalae was .05 when it should have been .30 from a more rigorous look)

As I'm not a statistician by trade I don't keep up on the literature very well, but interconnectedness of data does seem to me to be a very important issue. I'm wondering if anyone can point me to some helpful reading to understand this side of the issue better. In particular, is there any approach to AB testing that can reliably address the issue of data interconnectedness in the kind of situation described above?

stdbrouw6• 11 years ago

Could you expand on what you mean by "interconnectedness within website traffic"?

If one person visiting the site has no influence on other people visiting the site, then measurements of their behavior will be independent. If Facebook tests a different interface on half of their users and the changed behavior of those users indirectly has some impact on the behavior of the control group, then your measurements would have some level of dependence – I can imagine that this could happen but it's not clear to me that this would be a common scenario. The same would happen if you measure the behavior of the same person more than once – but in this case there's many procedures for working with paired or autocorrelated data.

Booktrope4• 11 years ago

The presenter I mentioned did not go into details about what interconnectedness he found, but I think it's quite obvious that people visiting a site do have influence on other visitors, which is at least a part of the underlying issue. On the simplest level, most web sites have share buttons to make it as easy as possible for visitors to influence other traffic. Or, other examples, a trending tweet can massively influence patterns of usage of a website or Facebook page (or many tweets with small reach can cause many small influences); or an RSS feed might influence patterns of tweeting or posting elsewhere. I think there are myriad other interconnections within web traffic. We do a great deal of work to drive traffic that is based on the premise that different visitors to websites are mostly interconnected. It's these factors that give me pause when I think about statistical measures that are premised on an assumption that we're measuring independent events.

chrisseaton• 11 years ago

I never understood where 0.05 came from. It seems like an arbitrary, magic number. Aren't those bad in science? Shouldn't we have some reason for every number we use? Why 0.05 instead of 0.049?

nzp• 11 years ago

It is somewhat arbitrary. 0.05 corresponds to roughly 2, and 0.01 to roughly 3 standard deviations under the normal distribution. In high energy physics, for example, in the past a result with 3 sigma significance was considered “a discovery”, but it turned out there were a lot of false signals, so it was upped to 5 sigma (~ 3 x 10^{-7} percent chance of getting the effect assuming there is no effect in reality), while 3 sigma (or 4) result is considered “evidence of effect“. Unfortunately in softer sciences, considering their relatively generally poor state of research standards, we have no idea if 0.05 or 0.01 levels really are good standards of “discovery”.

return0• 11 years ago

i think this is also important on a conceptual level. There's just too much literature in life sciences nowadays which, instead of formulating bold hypotheses with clearly distinct effects, is happy to report miniscule effects that just crossed the arbitrary significance barrier.

MollyR2• 11 years ago

I might be missing the forest from the tree's.

The paper talks about how it seems researches are "hacking" (their word) p-values.If the researchers lack the ethics using one form of statistics, what is really stopping them from misusing Bayesian Analysis?

Sidenote: While we are talking about bayesian stuff. I recently ran into the sleeping beauty problem ( http://en.wikipedia.org/wiki/Sleeping_Beauty_problem ) and it showed how different interpretations of an experiment can lead to people believing in two very different answers in an almost religious way. Some people argued this thought experiment shows a clear flaw in bayesian thinking. I'm not sure either way, I'm still thinking about the puzzle.

Gravityloss• 11 years ago

That's a cool problem. It took a while to understand the angle.

I'd say I'm a halfer. I think the extreme sleeping beauty problem highlights how if you're woken up, it's not any more probable it's a head or a tails awakening, since, for a reason I can't explain, the million tails awakenings "don't accumulate".

In a sense, it's like the opposite of the Monty Hall problem, since here the sleeping beauty receives no information whatsoever during the experiment.

MollyR2• 11 years ago

I'd say I'm a halfer too, but a lot of physicists/philosophers say 1/3, and it really bothers me that I can't seem to sync up with world leaders.

source : http://rfcwalters.blogspot.com/2014/08/the-sleeping-beauty-p...

logicallee2• 11 years ago

the main reason to ban significance testing in all fields is because of logicallee's getcher scientific results agency.

Our prices are:

$10,000 random study, no results guaranteed. Not recommended! Highly likely to be damaging.

$20,000 basic study, p=0.5 - study inconclusive but implies it's at least not "more likely" that the damaging/negative result is correct. (Assuming uniform priors.) No scientific value.

$100,000 weak FUD. Suggests that the damaging/negative results (assuming bayesian reasoning or uniform priors) may be incorrect, at a suggestive p<0.10 level. Not conclusive and invites further studies which can strengthen damaging/negative result! Not recommended unless further studies are unlikely. Consists of ten studies, one of which is published with remaining buried. Unscientific.

$200,000 basic refutation. Refutes the damaging/negative result at a statistically significant p<0.05. Likely to be referenced and accepted. However, due to significance level, links to logicallee's getcher scientific results agency should still be avoided! (Invites skepticism.) Scientific. May take up to six months.

$500,000 Silver refutation. Highly significant refutation at p<0.02. Study can be extremely rigorous and sponsorship can be public. The results should be referred to and referenced as widely as possible. Unpublished results (the other 49 studies in the series) to remain unpublished and unreferenced. Due to the number of parallel studies to be involved, may take up to one year for these results. Highly scientific.

$1,000,000 Gold refutation. Highly significant refutation at p<0.01, with extreme amounts of data to be published. Should become the gold standard for data in the field, and the most significant effect published. Full refutation of damaging/negative result, with full scientific rigor. Should be widely promoted. May take up to 24 months to produce data. Gold standard of science.

ACADEMIC BONUS: prices are free for tenured professors, thousands of whom can do all the studies they want and only publish if they see some significant effect.

EDIT:

in other words, http://xkcd.com/882/

Psychology Journal Bans Significance Testing

Related Stories