Definitions

Reliability tests are statistical tests that decide whether two evaluation methods agree when evaluating the same data. Confidence intervals provide a range around the sample mean where the true population mean is likely to fall [13,14]. Narrow confidence intervals indicate less uncertainty and can be used to assess whether repeated evaluations are consistently close. The lesser known tolerance intervals define a range that is expected to contain a specific proportion of the population with a given level of confidence [14]. They help determine whether most evaluations will fall within an acceptable range. In CMR, a tolerance range was initially designed to cover 95% of intrareader differences [15]. This tolerance range was then used to assert equivalence when a new evaluation method’s confidence interval was covered by it. Over time, these tolerance ranges were generalized for use in quality assurance analyses within CMR and applied to evaluate the acceptability of AI-based QP evaluations [4,5,16–18]. However, they focus on the acceptability of the bias, without addressing the variance. This seems suboptimal because QP evaluation concerns individual patients, which may deviate from the bias – these deviations should be known and limited.

Visualizations of statistical tests

In order to communicate reliability tests and dataset characteristics more effectively, statistical plots were developed. Correlation plots [19], box plots [20], and Bland-Altman plots [21] are common visualization methods to communicate the results of statistical tests. However, these plots have limitations that may poorly communicate important evaluation method differences (Fig 1). Correlation plots illustrate correlational relationships between evaluation pairs, but present bias poorly [21,22]. Box plots present the median, and scales to many dataset comparisons, however, they gloss over evaluation pairs and may conceal trends and skewedness. Bland-Altman plots present bias and evaluation pairs, which reveals trends. However, while these conventional plots are descriptive they do not present whether two evaluation methods agree (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Classical Statistical Plots.

Illustrative examples for correlation, box and Bland-Altman plots with capabilities and limitations listed on the right. Legend: PCC: Pearson’s correlation coefficient, stddev: standard deviation.

https://doi.org/10.1371/journal.pone.0323371.g001

The aim of this paper is to establish a statistical plot that communicates the reliability of quantitative parameters in CMR by visualizing clear criteria for agreement in addition to descriptive statistical properties. To this end, first, a statistical test is designed to decide evaluation agreement based on bias and variance tolerance ranges. Second, a novel statistical plot is established that combines descriptive statistical properties with agreement criteria, and third, the plot’s utility is demonstrated for AI assessment.

Extended verity plot

1. Presents a two-part figure with both acceptability decisions (for bias and variance) on the left, and elements pertaining to the statistical trends on the right
2. Draws the zero difference line
3. On the left the reliability decision is presented as the reliability plot (3a, 3b, 3c):
   1. a) Plots the bias confidence interval and color-codes the bias’s acceptability
   2. b) Plots the variance tolerance interval and color-codes the variance’s acceptability
   3. c) Integrates the two tolerance ranges (±BTR, Bias ± VTR) as guiding figure elements to indicate reliability decisions
   4. d) Illustrates plausibility of normal distribution with a density plot
4. On the right:
   1. a) Plots the average difference of the evaluation method in comparison to the reference evaluation (bias)
   2. b) Plots the 95%-tolerance interval limits of V% of covered differences (bias ± V%-tolerance interval) of the repeated evaluation
   3. c) Plots the paired data points as averages (or reference evaluation values) on the x-axis and differences on the y-axis
   4. i) This visualizes potential trends and skewedness

Condensed verity plot

1. Draws the line for equality of evaluation (zero difference)
2. Presents multiple dataset comparisons simultaneously
   1. A density plot supports trend and skewedness detection
   2. The paired data points are plotted without x-values, collapsing vertically to present only the differences
3. A reliability decision plot is plotted as in the first figure

1. 3) Testing the Statistical Plots

The Verity plots are used to evaluate an AI method for cardiac image segmentation of short-axis cine data as well as an AI method for parametric mapping reference point detection and segmentation. To this end, tolerance ranges are created, several AI methods are trained, the Verity plots are assessed in diverse settings and compared to classical statistical plots. Finally, customization options are presented for the plots.

1. 3.1) Creation of Tolerance Ranges

For 30 CMR studies, volumes and function as well as T1 and T2 parametric values were evaluated twice by one expert reader, and once by a second expert. The volume and function QPs were the left and right ventricular (LV, RV) end-systolic, end-diastolic and stroke volumes (ESV, EDV, SV), ejection fraction (EF), LV myocardial mass (LVM) and ED papillary muscle mass (PM). The parametric mapping values were the midventricular average value (T1 PRE Midv Avg, T2 Midv Avg) as well as six segmental values based on dividing the midventricular slice into segments with a reference point according to the American Heart Association model (T1 PRE AHA_(n), T2 AHA_(n) for n in 7–12). For each QP the intra- and interobserver differences were calculated. For each parameter ±1.96 standard deviations of intraobserver and interobserver differences defined the BTR and the VTR.

1. 3.2) Assessing Visualization Capabilities of Verity Plots

In order to assess the capabilities of Verity plots, we trained multiple AI methods, used Verity plots to evaluate the acceptability of their QPs, and compared the visualizations to traditional alternatives.

Training of artificial intelligence methods

For short-axis cine, we trained a U-Net [23] on 150 cases. The 150 cases were annotated by an expert in ES and ED with LV endocardial, myocardial and papillary muscle contours, as well as RV endocardial contours. We created four AI methods by varying the rasterizations for the different cardiac structures during training that increased the LV myocardium and papillary muscles, while decreasing the LV and RV endocardial structures. For parametric mapping methods, we trained three U-Nets on 224 cases for a cascaded annotation AI, in which a first U-Net detects the LV, which is enlarged for subpixel-accuracy and annotated by two further U-Nets [16]. The first contours the myocardium, and the second detects the reference point, at which the RV connects to the LV myocardium. During evaluation several security rims are used for the AI methods (−5%, 0%, 5%, 10%) to provide multiple QP results. For an in-depth description of the employed U-Net architecture and training procedure see the supplemental material (S1_File.pdf).

Testing and demonstrating verity plots

To test the utility of Verity plots, the short-axis cine AI methods were evaluated on 318 cases; the parametric mapping AI methods were evaluated on 59 cases from the same dataset. Verity plots were presented for a selected subset of QPs; all plots for all QPs are available in the appendix. Density plots within the Verity plots were used to evaluate whether a normal distribution may be assumed by visually assessing symmetry, a single peak, and quickly decreasing tails. The visualization capabilities of Verity plots were compared to traditional alternative plots: correlation plots, box plots and Bland-Altman plots. The results of this comparison are summarized in a table.

1. 3.3) Customization Options

While Verity plots were designed to be extensive in their presentation, they may be reduced for simple comparison environments when it is desirable to limit visual attention to certain statistical tests or properties of the distribution. To this end, Verity plots are customizable, including options for horizontal presentation and condensing the plots by merging or removing visual elements.

More specifically, Verity plots allow for turning certain elements on and off (in- or excluding: reliability tests, density plots), in- or excluding percentiles in the density plot, moving the legend locations, and merging the density and test elements. The condensed Verity plots allow for vertical and horizontal presentation, in- or excluding reliability decisions, density plots and strip plots, and percentiles within the density plot. Both plots allow for choosing the colours of the density plot faces and points.

Ethics declarations and data access

All numerical datasets required to reproduce the results are available in the supplemental material. The use of patient data for this study was approved by the local ethics committee of the Charité Universitätsmedizin Berlin as retrospective study (study ID: EA 1253 21). The MRI datasets cannot be shared due to institutional rules but may be available on special request. For research purposes, datasets were accessed from the 07.04.2024–01.12.2024. Datasets were anonymized such that individuals could not be identified.

Utility of density function for normal distribution assessment

Initial evaluation of the short-axis cine cases revealed poor AI generalization to scanners different from those in the training dataset: the AI method failed to segment cases obtained from external scanners. Verity plots display these failures as systematic outliers that violate the normal distribution (Fig 4, top). The AI method was reassessed on the internal cases of the dataset, from which 35 externally scanned cases were removed (Fig 4, bottom).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Utility of Density Function for Outlier Detection in Verity Plots.

The top two figures depict three AI-expert comparisons in condensed Verity plots (left), and of the first AI to the expert in the extended Verity plot (right). Density and point plots reveal systematic outliers (a, b). In the right plot the linear trend of the outliers indicate a systematic failure (b). The systematic outliers break the normal distribution and invalidate the test. These segmentation failures were removed to produce the two plots below. The data are normally distributed and the first two AI methods have acceptable biases and variances. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval.

https://doi.org/10.1371/journal.pone.0323371.g004

Plot comparison: Verity, correlation, box and Bland-Altman plots

Verity Plots were used to evaluate all QPs listed in Table 1 (for all AI methods described in Methods); the illustrations for all AI methods are available in the appendix (S1_Fig, S2_Fig, S3_Fig, S4_Fig). Three illustrative QP comparisons (T2 midventricular average, RV EDV, RV EF) are shown in Fig 5 for extended Verity, correlation, box, and Bland-Altman plots.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Plot Comparisons: Verity, Correlation, Box and Bland-Altman Plots.

Three quantitative parameters: T1 Midventricular Average, RV EDV, and RV EF (rows 1–3) illustrate differences in expressiveness between Verity, correlation, box and Bland-Altman plots (columns 1–4). Verity plots depict the acceptability of bias and variance clearly, with negative trends visibly accessible in RV EDV and RV EF. Correlation plots show correlations and a zero difference line. PCCs are visible, biases and trends are difficult to see. Box plots offer the median, the spread (50% of differences) as the box’s extent, and 98% of differences as whiskers. The data spread is visible, but trends and outliers are concealed. Bland-Altman plots show the bias, and 1.96*standard deviations as agreement limits. They show the bias and variance of data, as well as trends and outliers. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval, PCC: Pearson’s correlation coefficient, Midv.: midventricular, RV: right ventricle, EF: ejection fraction, EDV: end-diastolic volume.

https://doi.org/10.1371/journal.pone.0323371.g005

The extended Verity plots supported distinctions between acceptability, problematic outliers and assessing the relevance of trends. Correlation plots (column 2 in Fig 5) presented correlations clearly, the variability and the negative trends were difficult to see for RV EDV and RV EF, and biases could not be estimated well. Box plots (column 3 in Fig 5) communicated median differences and the position of the bulk of data in relation to the zero differences line. They gave no visual indications to expected parameter ranges since the y-axis exclusively communicated differences. The trends in RV EDV and RV EF were not visible in the box plots. Bland-Altman plots (column 4 in Fig 5) communicated mean differences, and the lines of agreement visualized the limits to the bulk of data well. Trends were discernible for RV EDV and RV EF. None of the traditional plots communicated whether the differences were acceptable. The analysis is systematically summarized in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of Reliability Visualization Methods.

https://doi.org/10.1371/journal.pone.0323371.t002

Customization options

Verity plots can be customized to simplify or augment information for users. In Fig 6 customizations are applied on randomly generated data. Plotted horizontally (Fig 6A) they increase the focus on how method-method differences distribute around zero. For small sample sizes the density plot may be omitted for clarity (Fig 6B). Legend positioning was changed between A and B. The third Verity plot (Fig 6C) merges the density plot with the reliability decision plot. Face and point colours were also changed. In Fig 6D the reliability plots were merged with the density plots, for which percentiles were removed. The density plot can be replaced by a box plot (Fig 6E), which may be useful when a normal distribution is not present, but a quantitative description remains useful. The final subfigure (Fig 6F) removed the reliability plot while keeping the bias confidence rhombus and variance intervals. It further switches colours between the density plot and the points and moves the density plot to the left.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Customization Options of Verity Plots.

Condensed Verity plots can be plotted horizontally (A) to focus attention on the bias and variability around the vertical zero difference line. In (B) the density plots were removed to focus on the statistical test; the spread of differences is presented with the plotted points. In (C) the Verity plot’s density plot was merged with the reliability plot, colours were changed to red. In (D) the points were removed, and the density plot was merged with the reliability test. In (E) the density plots were replaced with box plots, and the reliability tests were removed. In (F) the density plot was moved entirely to the left, and colour coding was chosen separately. Legend: BTR: bias tolerance range, VTR: variance tolerance range, TI: tolerance interval, AI: artificial intelligence.

https://doi.org/10.1371/journal.pone.0323371.g006

Tolerance ranges

The BTR and VTR reflect limits of reproducibility in clinical research that capture key confounding influences through intra- and interreader evaluations. Intrareader differences, by definition, compare an expert to themselves, while interreader evaluations highlight differences between two experts. Intra- and interobserver analyses thus isolate reader-specific contributions to annotation and QP differences. The persistence of variations in intrareader analyses suggests that some of these differences arise from the inherent challenges of image interpretation. Interobserver analyses may introduce additional confounders, such as differing interpretation approaches, further contributing to larger variability.

The VTR is consistently one to two times larger than the BTR in Table 1. Due to the nature of intra- and interreader differences this is reasonable, as the confounding factors that cause the variability in the BTR are also present in the VTR, while the VTR additionally contains reader-specific idiosyncrasies. However, the QPs differ in how strongly they are affected by other readers. For volume QPs the VTRs are about twice as large, stroke volumes are close to 1.5, and ejection fractions are 1.1 times larger than BTRs. This makes sense since consistent contouring differences between readers impact end-systolic and diastolic volumes equally and cancel out in the function QPs. Interestingly, mapping parameters were quite resistant to interreader confounders (factor 1.25–1.4). Possibly because myocardial contour thickness differences do not add up, and may be compensated for by the 5% safety-rim.

Statistical plots for quantitative parameter assessments

The classical correlation, box and Bland-Altman plots were used as plot-benchmarks since they provide timeless statistical visualization tools. Correlation plots are nearly 200 years old [19,24,25], and, while box and Bland-Altman plots are more recent (1977 and 1986), they are omnipresent in all quantitative fields [20,21]. Bland’s and Altman’s goal was to provide a method that accounted for bias in assessing agreement, rather than relying solely on correlation as a sufficient measure to define agreement [22]. However, while they highlight differences, they lack the means to convey whether the level of difference between methods is acceptable. Verity plots build on the Bland-Altman approach by incorporating the difference plot but replacing the 95% limits of agreement (LoA) with tolerance intervals. This adaptation is necessary for compatibility with the reliability tests, as comparing the LoA – which is the 95% confidence interval for differences – with the VTR would be inappropriate since the VTR itself represents the 95% confidence interval of another reader’s differences.

Modifications to the Bland-Altman plot have been proposed, such as enhancing the LoA with confidence intervals [26–28], which show the likely upper and lower bounds of the LoA. However, these changes are often not well-justified because Bland-Altman plots are not designed to assess the precision of the LoA itself. Instead, they aim to use the LoA to ensure that a sufficiently large percentage of differences fall within the limits. For this reason, confidence intervals around the LoA do not add meaningful information, they rather complicate the plot and add a misleading indicator. In contrast, tolerance intervals are the correct statistical choice to replace the LoA lines: they provide a better limit for the variance while accounting for uncertainty.

In recent decades modern plots such as violin plots [29], raincloud plots [30], bean plots [31], and swarm plots [32] have emerged as potent visualization tools in exploratory data analysis. Violin plots are two-sided kernel-density plots (as are the density plot in Verity plots) merged with the quantile information of a box plot [29,33,34]. Bean plots are similar to violin plots but include the individual data points (“beans”) for visualization of small datasets [31]. Condensed Verity plots provide a similar unification of information. Raincloud plots are similar to Verity plots in the regard that they merge density with strip plots. However, they opt for integrated box plots rather than the reliability plots making raincloud plots nonparametric and descriptive. Swarm plots display individual data points next to each other to avoid overlapping [32]. For medium sized datasets this allows for a visual estimation of the density. However, none of these plots integration the decision criteria that were central to the design of Verity plots.

AI development supported by Verity plots

The central issue of developing reliable AI for automatic QP generation in CMR is the vast space of hyperparameters. This includes numerous tuneable parameters that define an AI method, such as data augmentation strategies, neural network architecture choices, as well as the integration of geometrical assumptions into the AI pipeline. For instance, cardiac structure segmentation networks may be based on diverse architectures, such as the Fully Convolutional Network [11,35], the U-Net [23], or several variations (e.g., Attention U-Net [36], Trans-U-Net [37], etc.). These architectures may further vary layer types, channel depths, convolution sizes, normalization methods, and activation functions. Similarly extensive lists of hyperparameters apply to data augmentations and geometrical assumptions.

Previously, AI optimization for segmentation tasks has focused on manually designing architectures and fine-tuning hyperparameters [38]. When automated, hyperparameter optimization often targeted a small set of parameters using optimization-libraries [39,40]. More recently, however, the field has shifted towards training populations of AI models that explore the hyperparameter space, using intermittent evaluations to identify promising configurations [41,42]. Condensed Verity plots, with their integrated reliability tests, provide a visually intuitive way to assess the performance of AI populations as they optimize and refine their hyperparameters.

Limitations

Verity plots were exclusively tested on quantitative parameters in CMR and could be difficult to extend to or apply in other quantitative areas. Verity plots, like all visualization methods, rely on human interpretation, which may introduce variability in assessment across different users. Future studies should evaluate their usability across a broader range of clinical and technical settings. As a parametric test, the agreement test communicates whether two QP assessment methods agree, assuming that their differences are normally distributed and a sample size larger than 30 is provided. Non-parametric confidence and tolerance intervals for a non-parametric agreement test are possible but would require larger sample sizes, which are often infeasible for QPs based on medical imaging.