P-values are inconsistent

http://www.johndcook.com/blog/2010/03/03/p-values-are-inconsistent/

by ekiru • 16 years ago

3 4 16 years ago

roundsquare2• 16 years ago

How do you do a statistical test against "mu is between a and b?"

If H0 (null hypothesis) is mu = a (and you know sigma = 1 and its normal) then you can pretty easily calculate the p value from an observation x by taking z = x - u and checking that against the normal distribution.

However, how do you do that is H0 is a < mu < b? Do you assume that mu is uniformly distributed between a and b? If so, then H' does not imply H. E.g. if H is true then pr(mu > 0) is 0.5, but if H' is true then pr(mu > 0) = 0.52/1.34 < 0.5.

Or, am I wrong about how to do the hypothesis test?

sesqu2• 16 years ago

You use the cumulative distribution function, which is defined as P(X≤x). More specifically, denoting the cdf of the standard normal distribution by 𝛷, P(X̅-a𝜎<µ≤X̅+b𝜎) = P(a<Z≤b) = P(Z≤b)-P(Z≤a) = 𝛷(b)-𝛷(a) for large samples.

Edit: Err, maybe I got that wrong. Doing the above calculation would favor the broader hypothesis (p=0.047 vs. p=0.043).

roundsquare2• 16 years ago

Well, if we leave the details aside, you do something where you take a probability distribution between a and b right? If thats the case, I think my original point stands, neither hypothesis really implied the other one because they say different things about the probability that x is in certain ranges.

I'm sure there are other problems with using the p value as a measure of certainty (though I don't know them) but this specific criticism seems silly.

P-values are inconsistent

Related Stories