There are many difficulties with using p-values and the 5% cut-off level (alpha) to analyze results: this paper homes in on one of them. The point is so obvious that I think it deserves a more straightforward numerical example working from first principles. Lets assume there are 1000 relationships being probed and that samples are large so we can take the power to be 100% - i.e. if there is a relationship the study will definitely find it. Assume further that 1% of these relationships are true. Then, of the 1000 relationships about 10 will be true (and so found by the studies) and a further 50 (5% of 1000 or strictly 990) or so will be false positives. This means that 10 out the sixty significant results are genuine - i..e 10/60 or 17% of the reported significant results will be true. This is clearly likely to be less with lower power levels and bias.

This is, however, so obvious that I find it difficult to believe that researchers are not aware of this. The problem is the uncritical use of p-values. There is, of course, a very large literature on these problems.

There are two obvious solutions, hinted at, but not made explicit in the article. First, avoid the uncritical use of p-values. Second, use Bayes theorem. The Bayesian approach to statistics takes account of prior probabilities explicitly which is what is wanted here.

Michael Wood
Portsmouth Business School
michael-wood@myport.ac.uk