Table 1.
Steps in calculation of ROC AUC.
Figure 1.
Difference in diagnostic performance of two tests showing readers from a multi-reader study.
Change in diagnostic performance of CT colonography for the detection of polyps; difference with computer assisted detection (CAD) minus without CAD. Results from individual readers. A. Comparison of increase in the number of patients with a correct diagnosis with change in ROC AUC. The five readers selected for illustrative purposes as examples for the rest of the article are labelled from 1 to 5. B. Arrows indicate values of sensitivity and specificity for each reader, the arrow bases showing unassisted read values and the arrow head the CAD assisted read values for the same reader. C. Difference in ROC AUC using two methods for fitting ROC curves. ROC AUC could not be calculated for reader 4 using LabMRMC method.
Figure 2.
Distribution of confidence scores for patients with and without polyps.
Each histogram shows the distribution of confidence score values using CAD CT colonography for an individual reader separately for patients with (brown) and without polyps (yellow) based on the reference standard. Five readers are shown in plots labelled 1 to 5.
Figure 3.
Distribution of published confidence scores.
Published confidence scores assigned by individual readers (Reader 1 and Reader 2) reported for patients without disease (DN) and patients with disease (DP) from two studies. In Hussain et al. [17], classification of MR imaging of cirrohotic liver used categories: 1 definitely benign; 2 probably benign; 3 possibly malignant; 4 definitely malignant. In Westphalen et al. [18], classification of MR imaging of peripheral zone tissue from patients with prostate cancer used categories: 1 likely benign; 2 possibly benign; 3 equivocal or indeterminate; 4 possibly malignant; 5 likely malignant.
Figure 4.
Different curve fitting methods.
ROC plots each for an individual reader using CT colonography without CAD. Green dots indicate real data points underlying curve fitting. ROC curve are shown extrapolated from these data using DBM MRMC (red dotted line) and PROPROC software (blue solid line). Five readers are shown in plots labelled 1 to 5.
Table 2.
ROC AUC using different methods and different ROC curves.
Figure 5.
Impact of few false positives.
ROC curve for reader 4 using CT colonography with CAD. The data from the original read (orange curve) includes two patients where false positive polyps have been indicated with confidence scores of 40 and 50. Perturbing these two patient scores to values of 70 (yellow curve) and 20 and 70 (brown curve) demonstrate changes in ROC curves. Sensitivity and specificity are expressed as percentages.