Fig 1.
Nonblind studies had higher effect sizes than their paired blind studies, on average (n = 83 pairs).
The thin lines show individual pairs, while the thick line shows the average effect size for each study type. The size of the dots is inversely proportional to the variance of the effect size, such that larger dots indicate more precise estimates. For clarity, two unusually large effect sizes are off the scale (dotted lines: g = 18.0 and 9.1).
Table 1.
Effect of each parameter on z scores, as estimated by model averaging of the top four models in the set (those with an Akaike weight > 0.05).
Note that the predictor and response variables were rescaled to have a mean of 0 and standard deviation of 0.5 prior to running the models [37]. The FoR category × Blindness interaction was not among the top models, indicating it had no detectable effect on z score. The effects of each FoR category are shown relative to the reference level “Agricultural and veterinary sciences.” The Importance column gives the summed Akaike weights of all the models containing the focal predictor; high importance shows that a parameter was present in many of the best models, and was thus a good predictor of z scores relative to the other measured variables.
Fig 2.
Density plots showing the distribution of z scores taken from putatively experimental blind and nonblind papers.
The dotted line shows z = 1.96 (z scores above this line are “significant” at α = 0.05), and the numbers give the sample size (number of papers) and the percentage of papers that were blind for this dataset. The bottom-right figure shows the median z score (and the interquartile range) in each FoR category for blind and nonblind papers.
Table 2.
Results of a generalized linear model with the proportion of significant p-values in each paper as the response variable and binomial errors.
Note that the predictor and response variables were rescaled to have a mean of 0 and standard deviation of 0.5 prior to running the model [37]. The effects of each FoR category are shown relative to the reference level “Agricultural and Veterinary Sciences.” The 95% CIs for each parameter were estimated as ±1.96 * SE.
Fig 3.
Density plots showing the distribution of the proportion of significant p-values per paper (i.e., the number of p-values <0.05, divided by the total number of p-values) in putatively experimental blind and nonblind papers.
The numbers give the sample size (number of papers) and the percentage of papers that were blind for this dataset (note the higher sample size relative to Fig 2). The bottom-right figure shows the median proportion of significant p-value papers (and the interquartile range) in each FoR category for blind and nonblind papers.