Fig 1.
The relationship between sample size, effect size, and statistical power in theory and practice.
(Left) In an ideal world, sample size and effect size independently contribute to statistical power; (Right) In an actual world, sample size and effect size are often entangled. Increased sample sizes tend to couple with decreased effect sizes, resulting in less statistical power.
Table 1.
Replication projects and their measures of replicability.
Fig 2.
Comparisons of predictors of replicability.
Coefficient estimates from separate logistic regressions predicting replication success from the sample size and effect size of the original study, controlling for study design. Error bars represent 95% confidence intervals.
Fig 3.
Distribution of bootstrap simulations for replicability predictions by sample and effect sizes.
The bootstrap distributions of partial R2 for replicability, with dashed vertical lines representing the median partial R2 values: 0.66% for sample size and 2.88% for effect size.
Fig 4.
Revealing the trade-off: Larger sample sizes associated with smaller effect sizes.
In the actual world, larger samples studies investigated smaller effects (rs = -0.49, p < 0.001). Five extreme values with N > 1000 and/or Cohen’s d > 4 were not shown in this graph but included in analyses. Excluding them yielded a similar pattern.
Fig 5.
Perceived importance of study criteria for replicability among psychological scientists.
Psychological scientists rated “large sample size” as the most important criterion in predicting the replicability of a study. Responses were asked “in judging a research study, to what extent do you think each of the following factors would predict its replicability?” and provided on a 5-point scale ranging from “not at all” to “an extreme amount.” Error bars represent ±1 SE.
Table 2.
Comparison of manuscripts in ideal vs. actual world scenarios based on power analysis.
Manuscripts in the ’Ideal World’ column represent expected values based on the power calculated with G*Power, while the ’Actual World’ column reflects observed data from our meta-analysis. Highlighted rows indicate the most highly powered research papers.