Table 1.
Mean consensus report (CR) scores (±standard deviation, SD) across evaluation panels for all proposals and for proposals with disagreements among raters in their Individual Evaluation Report (IER) score and between Consensus Report (CR) and average Individual Evaluation Report (AVIER) scores.
Table 2.
Associations and differences between consensus reports (CR) and average individual evaluation report (AVIER) scores and inter-rater agreement (average deviation index, AD index) for all proposals, and for different actions and panels*.
Fig 1.
Distribution of average deviation (AD) indices for all proposals.
Positively skewed distribution shows that the majority of proposals had AD index below 10 points (20,988 out of 24,897 proposals or 84.3%).
Table 3.
Distribution of proposals with disagreements among raters in their Individual Evaluation Report (IER) score and between Consensus Report (CR) and average Individual Evaluation Report (AVIER) scores across evaluation panels*
Fig 2.
Association between raters' agreement (AD index—lower score means greater agreement) and the average Individual Evaluation Report (AVIER) (A) or consensus report (CR) (B).
Line indicates the regression line. Circles—individual proposals.
Table 4.
Pearson’s inter-correlations of IER criteria of different raters*.