Reader Comments

Post a new comment on this article

Do the results support the null?

Posted by Dienes on 07 May 2013 at 14:32 GMT

The authors use Bayes factors to show the data as a whole support the null hypothesis. They state "The Cauchy prior makes weaker assumptions about the likely effect sizes under the experimental hypothesis and is therefore preferred, although we report results based on both." This in fact puts the argument the wrong way round - to convince a reluctant reader that the null is supported, where there is room for reasonably doubt, one should use assumptions that would bias results in the opponent's favour - still obtaining support in favour of the null is then all the more convincing.
In their comment the authors report an average d of 0.034. This corresponds to an r of 0.017, which can be made normal with a Fisher's z transform, leaving the estimate still at 0.017, with a standard error of 0.057 (given degrees of freedom of 311, as reported by the authors in their comment). Now the crucial bit - what does the theory of social priming predict? That is, what range of effect sizes are plausible on the theory? The bigger the range/the more heavy tailed this distribution, the easier it is to get evidence for the null. So let's use a normal rather than a Cauchy. And let's make the SD of this normal as tight as is barely reasonable. As the authors report, the previous effect sizes were large, and none ever as small as 0.5. A d of 0.5 corresponds to an r (and Fisher's z) of 0.24. So if we use a normal with an SD of 0.24, we have made it as hard as reasonable to get support for the null. In fact we shall center the normal on 0, to make it as hard as possible to distinguish the theories. Now let's chop off all effects in the negative direction, so the prediction is of an effect in the direction numerically obtained, making it even harder to support the null. Using the Bayes factor calculator here: http://www.lifesci.sussex... this produces a Bayes factor of 0.30 - where values less than 1/3 are strong evidence for the null,on the way round this calculator works. In sum, the studies included in this average rather compellingly support the null hypothesis, and they cannot, as a set, be criticized on the basis of sensitivity, i.e. haring too few subjects. Thus the discussion can reasonably move on to the question of under what precise conditions the effect does exist and what moderates its strength.

No competing interests declared.