Figure 1.
Distribution of P-values for 10,000 two sample t tests for Normal data.
Means were compared between two groups of 10 observations from a Standard Normal distribution.
Figure 2.
Distribution of P-values for 10,000 two sample t tests for Lognormal data.
Means were compared between two groups of 10 observations from a Lognormal distribution.
Figure 3.
P-values from four realisations of 10,000 correlated t tests for Normal data.
Means were compared between two groups of 10 observations from a Standard Normal distribution where each test used variables with correlation 0.5 with the other variables.
Figure 4.
Distribution of P-values for chi-squared tests and Fisher’s exact tests for two by two tables.
Chi-squared and Fisher’s exact test, both two-sided and one-sided, were calculated for the comparison of two samples of size 10 with a binary outcome variable with probability 0.5 of being 0 and 0.5 of being 1.
Figure 5.
P-values for chi-squared and Fisher’s exact tests for large samples with fixed size groups.
Chi-squared and Fisher’s exact test, both two-sided and one-sided, were calculated for10,000 comparisons of two samples of size 1,000 with a binary outcome variable with probability 0.5 of being 0 and 0.5 of being 1.
Figure 6.
P-values for chi-squared and Fisher’s exact tests for large groups of varying size.
Chi-squared and Fisher’s exact test, both two-sided and one-sided, were calculated for10,000 comparisons in samples of size 2,000, both group and outcome having probability 0.5 of being 0 and 0.5 of being 1.