Table 1.
Population characteristics of children’s national patient sample.
Fig 1.
Area under the receiver operating characteristic (AUROC) curves (a), area under the precision recall (AUPRC) curves (b), and calibration plots (c) for the multi-institutional Dynamic Criticality Index models applied to the single-site test dataset (n = 26,401). Outcomes were ICU or routine care for the four future time periods of >6–12 hours, >12–18 hours, >18–24 hours, and >24–30 hours. The 95% CIs are included in panels a and b. a. AUROC for four future time periods. The area under the receiver operating characteristic curves for classifying care as non-ICU or ICU for the respective future time periods are presented in each panel. b. AUPRC for four future time periods. The area under the precision-recall curves and 95% CIs are included in each panel. The areas were computed with integral approximations, and the CIs were computed using a logit method [21]. c. Calibration plots for four future time periods. The y-axis shows the expected proportion of ICU care areas for the time periods based on the risk intervals, and the x-axis shows the observed proportion of ICU care areas in the time periods. A linear regression is reported in each panel, with their respective R2, and the fitted mean is represented with the solid line. The line of identity is the dashed line. The risk intervals were composed by requiring a minimum of 200 data points per risk interval, resulting in 389, 375, 365, and 356 risk intervals in the four future time period models, respectively. The circles indicate the observed and the expected proportions of ICU 6-hour time periods over ascending Criticality Index intervals. 0.0%, 0.0%, 0.3%, and 0.0% of the risk intervals within each plot have a Cohen’s h value <0.2 indicating large effect sizes for differences between observed and expected proportions.
Table 2.
Performance metrics of the Dynamic Criticality Index models developed from the multi-institutional database applied to the single-site test dataset (A) and the single-site Dynamic Criticality Index models applied to the single-site test dataset (B).
Fig 2.
Area under the receiver operating characteristic (AUROC) curves (a), area under the precision recall (AUPRC) curves (b), and calibration plots (c) for the single-site Dynamic Criticality Index models applied to the single-site test dataset (n = 3,018). Outcomes were ICU or routine care for the four future time periods of >6–12 hours, >12–18 hours, >18–24 hours, and >24–30 hours. The 95% CIs are included in panels a and b. a. AUROC for four future time periods. The area under the receiver operating characteristic curves for classifying care as non-ICU or ICU for the respective future time periods are presented in each panel. b. AUPRC for four future time periods. The area under the precision-recall curves and 95% CIs are included in each panel. The areas were computed with integral approximations, and the CIs were computed using a logit method [21]. c. Calibration plots for four future time periods. The y-axis shows the expected proportion of ICU care areas for the time periods based on the risk intervals, and the x-axis shows the observed proportion of ICU care areas in the time periods. A linear regression is reported in each panel, with their respective R2, and the fitted mean is represented with the solid line. The line of identity is the dashed line. The risk intervals were composed by requiring a minimum of 200 data points per risk interval, resulting in 287, 272, 250, and 242 risk intervals in the four future time period models, respectively. Within each interval, we computed the average expected risk of ICU admission and the observed risk of ICU admission. The circles indicate the observed and the expected proportions of ICU 6-hour time periods over ascending Criticality Index intervals. 96.9%, 97.1%, 96.0%, and 95.0% of the risk intervals within each plot have a Cohen’s h value <0.2, indicating there are small effect size differences between observed and expected proportions.
Fig 3.
Variable importance for the single-site and multi-institutional Dynamic Criticality Index models.
Percentages of the 30 most frequently important covariates across all models in the single-site (light grey) and multi-institutional (dark grey) models are presented as determined by the LIME methodology. The red and black lines represent the 95% confidence interval. Red confidence intervals indicate that the single-site and multi-institutional confidence intervals do not overlap and the pairs of features have a Cohen’s h >0.2, indicating that the proportion of use has a practical difference. The data reported was computed using the cases where the predicted risk is between 0.245 and 0.255.
Table 3.
Single-site Dynamic Criticality Index performance metrics for sensitivities of 0.85, 0.90, 0.95, and 0.99 representing four potential decision thresholds for the four future time periods.
Fig 4.
Percent accuracy of predictions for patients in the test sample who transferred from routine care to ICU care (n = 124) (a), and from ICU to routine care (n = 478) (b) for the single-site Models of the Criticality Index-Dynamic. The denominator varies by model because patients were required to receive care in a location for a minimum of 6 hours prior to transfer and have ≥1 new data element for a new prediction to be made.