Table 1.
Results of prediction models.
Fig 1.
Algorithm performance comparisons for predicting 30-day mortality.
Model area under the curve (AUC) comparisons for 30-day mortality. Abbreviations: Logistic regression (LR), supported vector machine (SVM), random forest (RF), multilayer perceptron (MLP), extreme gradient boosted tree (XGB).
Fig 2.
Test data algorithm performance comparisons for predicting 30-day mortality.
The left panel displays the full area under the curve (AUC) for logistic regression (LR) algorithm vs extreme gradient boosted (XGB) algorithm. The middle panel displays the true positive rate vs the alert rate (e.g. 1% alert rate would capture the 1% at highest risk for 30-day mortality) at selected alert rates (1%, 2%, 5%, 10%, and 20%), along with details about the sensitivity and positive predictive value (precision) at those alert rates for the LR and XGB models. The right panel of displays calibration curves along with the Brier score loss–an indication of the overall calibration of the LR and XGB models.
Fig 3.
The Extreme gradient boosted tree model calibration for predicting 30-day mortality.
The left panel shows that the extreme gradient boosted (XGB) tree model produces a wide range of risk estimates for the test data set individuals, from 0.0 to 1.0. The right panel is a calibration curve and shows that each risk score bin accurately represents the true average mortality rate of those individuals in that bin. In other words, the right panel shows that the predicted mortality curve (green line) follows the actual mortality curve (dashed line) closely.
Fig 4.
Example output from clinician-facing software risk tool.
Descriptors on risk context and contributors correspond to the numbers in Fig 4 as follows: (1) Clinically-determined risk criteria were defined to determine when to display message boxes with below average risk (green), above average risk (yellow), and relatively elevated risk (red) messages. (2) Patient’s individual risk of 30-day mortality (as estimated by the XGB model) is presented as relative to the average risk of 30-day mortality among all patients undergoing the same procedure (e.g. aortic surgery). (3) Patient’s individual risk of unplanned admission is presented as relative to the average risk of unplanned admission among all patients undergoing the same procedure (e.g. aortic surgery). (4) Summarizes individualized risk of the most likely adverse events; presented as relative to the average risk for a given adverse event among all patients undergoing the same procedure (e.g. aortic surgery). (5) Factors (e.g. preexisting comorbid diseases, or previous procedures) contributing to risk are scaled and sized (as estimated by logistic regression) to reflect relative contribution to overall risk.