Patient cohort, variables used, and overall performance

We included 2,868,808 PCIs in the NCDR CathPCI registry; 2,314,446 (80.7%) prior to 2014 for model training and 554,362 (19.3%) after 2014 for validation and model interpretation. The mean (SD) age of patients was 64.6 (12.0) years and 68.3% were male (Table 1). Overall, there were 118,327 (4.1%) major bleeding events: 98,167 (4.2%) major bleeding events in the training set and 20,160 (3.6%) in the validation set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient characteristics.

https://doi.org/10.1371/journal.pdig.0000906.t001

Models were trained in a stratified, five-fold cross validation. Model performance metrics at each stage are provided in Table 2, and each stage is described further below. The model area under the receiver operating characteristic curves (AUROCs) increase from 0.812 to 0.845, and model area under the precision recall curves (AUPRCs) increase from 0.203 to 0.242, respectively. These results included decision tree models that use all available data for risk stratification. S1 Table lists all the variables used in our XGBoost-trained model. We maintained a number of extracted features persistent across prior models by Rao et al.³ and Mortazavi et al. ⁴, in order to demonstrate that all the available variables, when considered in a staged setting, allow further clarity towards the dynamic nature of Risk of Major Bleeding. We also sought to evaluate if the multicollinearity of these variables impacted the modeling, as a separate experiment, in order to clearly identify performance gains as a result of variables versus performance gains as a result of the dynamic staging of models. By removing all the created, binary indicator features, leaving only their continuous or categorical variables from which they were extracted, we actually saw a slight decrease in model performance, with AUROCs ranging from 0.808 to 0.843 and AUPRCs from 0.198 to 0.237. Because of the way the gradient boosted decision trees model each weak learner across the participants in the training dataset, it makes sense that the non-linear risk is better modeled with a wider array of features, some continuous, some categorical, and some extracted binary daughter variables. Further detail of this additional experiment is described in the S1 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of model performances for bleeding prediction. We compare model performance by area under the receiver operating characteristic curve (AUROC), the area under the precision recall curve (AUPRC), the Brier Skill Score, and the Reliability and Resolution of the Brier Decomposition. Brier Skill Score and Resolution are higher for better performance, reliability should be lower for better performance.

https://doi.org/10.1371/journal.pdig.0000906.t002

The respective receiver operating characteristic plots and precision-recall curve plots for each stage of the model are presented in S1 and S2 Figs, respectively (and S3 and S4 Figs for the multicollinearity analysis). For each of these figures, a single fold from the five-fold cross-validation was selected for plotting, with the confidence intervals in Table 2 demonstrating confidence in an accurate illustration.

The staging of models is designed to be aligned with key clinical decision-making points. However, by the definition of major bleeding, it is possible that these are confounded with periprocedural bleeding rather than bleeding that can take place up until 72 hours after the procedure. As a result, the final two models can only be used as a decision-making model if major bleeding has not yet occurred. Finally, discussed further in the S2 Text (and in S2 Table), we found that the gains in performance and risk stratification were consistent across Femoral and Radial subgroups.

Interpreting SHAP plots

One approach for interpreting models is SHapley Additive exPlanations (SHAP) [11]. SHAP feature importance plots for the Presentation Model and the Closure Model are in Fig 1 (with intermediate models shown in S5–S8 Figs and SHAP plots with the colinear variables removed in S9–S14 Figs). In each plot, the variables are sorted in order of decreasing importance. The color of a variable relates to the value of that variable, while location along the x-axis represents how much that variable contributes to the risk of bleeding. Features most strongly driving predictions of high bleeding risk appear on the right with high SHAP values, while features most predictive of low bleeding risk appear at the left. The specific plots, per stage, are described further below. Note that the plots do not show the interactions across the depth of the tree but present a broader view of overall risk per feature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. SHAP Tree explainer for (a) Model 1: Presentation and (b) Model 6: Closures.

Variables are sorted in order of decreasing importance. Variable color relates to the value of that variable, while location along the x-axis represents how much that variable contributes to the risk of bleeding. Features most strongly driving predictions of high bleeding risk appear on the right, while features most predictive of low bleeding risk appear at the left. The higher a variable is, the greater overall importance that variable exhibits for the model. Red values indicate high values for that variable (if continuous or ordinal) or “true” (if binary), while blue values indicate the opposite. Points to the right of the axis (positive SHAP values) indicate that a feature of that value increases model estimate of bleeding risk, while points to the left of the axis (negative SHAP values) indicate that a feature of that value decreases the model estimate of bleeding risk. For instance, on the top row of Fig 2, a preprocedural hemoglobin greater than 13 is associated with an increased risk of bleeding, while value less than 13 is associated with a decreased risk. The wide range of SHAP values for this variable shows that while the direction of association is constant, the degree to which this feature impacts risk is not constant.

https://doi.org/10.1371/journal.pdig.0000906.g001

Stage 1: Clinical presentation (Model 1)

The initial model uses information available to a clinician at the time that a patient presents and predicted bleeding risk with an AUROC of 0.812 and AUPRC of 0.203. The Brier skill score of this model is 0.088, representing the degree that this model improves over a naïve model (higher is better). The Brier reliability is 2.6E-4, representing distance to true probabilities (lower is better), while the resolution is 3.3E-3, representing forecast distances to the mean rate (higher is better). The SHAP plot (Fig 1A) shows that this model is most strongly driven by pre-procedural hemoglobin and coronary artery disease symptoms at presentation. The inclusion of both continuous variables and dichotomized show the importance of different values of this variable across decision trees in the final model. Variable nonlinearity is also demonstrated here. Looking at the top two rows of the figure, hemoglobin > 13 is associated with an increased risk, but the continuous hemoglobin variable shows that very low hemoglobin is more strongly associated with bleeding, while a relatively higher hemoglobin is associated with a lower risk of bleeding, but a wider array of increases in risk across the intermediate values just above the threshold make the dichotomous variable represent increased risk rather than more varied risk at different thresholds.

Intermediate decision points and variables

A complete discussion of the model performance, variable importance, and patient reclassification of each intermediate stage are available in the S3 Text. In brief, the model AUROCs improve to 0.817, 0.825, 0.832, and 0.844 with variable importance changing as additional data becomes available.

Decision 3: Closure method (Model 6)

The final decision point is closure. This decision had minimal effect on overall prediction, with AUROC remaining at 0.845 and AUPRC improving slightly to 0.242. The Brier skill score improved slightly to 0.119, the Brier reliability improved to 1.0E-4, and the Brier resolution improved to 4.3E-3. Fig 1B shows a SHAP explanatory plot for this model. Manual compression is the closure method most strongly predictive of increased bleeding risk (12^th most informative variable).

Reclassification from Model 1 to Model 6

Total reclassification from the initial model to the final model is shown in Table 3. Among 123,712 patients classified as low risk by the initial model, 14,441 (11.7%) were reclassified as moderate risk, while 723 (0.6%) patients were reclassified as high risk. Among the 14,441 patients re-classified as moderate risk, 1.4% experienced bleeding events. Among those 723 patients reclassified to high risk, 12.5% experienced bleeding events. Among 270,485 patients classified as moderate risk by the initial model, 82,418 (30.5%) were reclassified to low risk by the final model, while 16,577 (6.1%) were reclassified to high risk by the final model. Among the 82,418 patients reclassified to low risk 0.5% experienced bleeding events, while among the 16,577 patients reclassified to high risk 7.0% experienced bleeding events. Finally, among 160,165 patients classified as high risk by the initial model, there were 40 (<0.1%) patients reclassified to low risk, and 43,265 (27.0%) patients reclassified to moderate risk. The 40 patients reclassified to low risk experienced no bleeds (bleeding rate of 0%), while 2.5% of the 43,265 patients reclassified to moderate risk had a bleeding event. Intermediate reclassification results are available in the S3 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Shift table across models. Top value in each cell is number of patients classified into that risk bin by the two respective models. The bottom value in each cell indicates the actual bleeding rate of all patients within that cell.

https://doi.org/10.1371/journal.pdig.0000906.t003

Limitations and future directions

There are several key limitations and opportunities for future research in our study. First, real-time data, which would be necessary for proactive implementation of dynamic models, can be challenging to acquire. The NCDR relies upon manual chart abstraction for many variables, such as past medical history variables and as a result, data we have at our disposal is older, and at risk for data drift issues with respect to changes in clinical practice. Our best efforts to adjust for this was the temporal split. The implementation of such a system at scale within an electronic health record environment, where data may be available within near real-time capacity, requires additional investigation of natural language processing techniques, models that handle advanced time-series data, and appropriate evaluation of user interface design for reducing clinician burden when interacting with such a model. This, additionally, would require an evaluation of the quality of the features being fed into the model to ensure those risk factors identified in this work are easily available and readily of high quality without the rigorous review the registry data undergoes.

The second is the timing of the available variables. The registry abstracts some variables as pre- and intra- procedure, so some confounding may exist from reactions to bleeds during the procedure. This nature of the variables could prevent accounting for some intra-procedural variables, such as if there was initially acute stent closure after PCI, which could require additional steps in the procedure. Additionally, the lack of timing prevents us from demonstrating model degradation as a function of time. It can only be inferred through the use of additional variables here. That said, we still find same estimation improvements and risk changes through the addition of medication, which we believe supports our conclusions for needing a dynamic model.

Finally, the definition of the bleeding outcome includes multiple items in a composite form, which includes a hemoglobin cutoff. This suggests that patients with a low baseline hemoglobin were more likely to experience this outcome, because a small drop would put them below the threshold as opposed to a substantial drop being needed for other patients.

Conclusion

We have developed a model that provides dynamically updated bleeding risk assessments by incorporating information available at different stages of patient care among patients undergoing PCI. These methods demonstrate evolution in variable importance as clinical decisions are made through course of care. For the risk of in-hospital bleeding, variable importance and bleeding risk changes as variables are included from cardiac catheterization and PCI. Accounting for the time-varying nature of data and capturing the association between treatment decisions and changes in risk provide up-to-date information that may guide individualized care throughout a hospitalization.

Study cohort

This study included all index PCIs in the National Cardiovascular Data Registry (NCDR) CathPCI registry version 4.4 from July 2009 through April 2015, which contains patient data from over 1500 sites across the United States [5,14]. To examine the improvement of dynamic data over static risk prediction models, we used the same cohort as Mortazavi et al. [4]. Therefore, we excluded patients during readmissions, who died in the hospital, who had missing data regarding if they had any bleeding events, or who underwent coronary artery bypass grafting (CABG) during the index admission [3,4].

The primary outcome was any in-hospital bleeding event within 72 hours after the start of the PCI procedure. Bleeding was defined using the definition employed in prior NCDR risk models as a hemoglobin drop ≥3 g/dL, whole blood or packed red blood cell transfusion, or intervention/surgery at the bleeding site to reverse/stop or correct bleeding. We further excluded patients with multiple, unknown, or brachial access sites to evaluate the treatment decision point of radial versus femoral access. We additionally excluded patients with multiple closure methods. This final exclusion was added because multiple closure methods may be associated with multiple access or may represent bleeding (e.g., a femoral closure device did not work and so manual pressure was held), which would bias the model to predicting risk of bleeding after the bleed has occurred.

Variables of interest

This study considered all data available from the CathPCI Registry prior to patient discharge [5]. This included all data used by the existing models [3,4], as well as additional variables as described below. The current full existing NCDR bleeding risk model [3] uses 31 variables: 23 patient characteristics at the time of presentation and 8 characteristics related to coronary anatomy and lesion characterization (see S4 Text). A more recent work added additional variables from the registry related to those 31 variables, finding improved predictive performance (4). Our study includes further additional data. The additional data considered in this study consists of additional laboratory data, past medical history, additional coronary anatomy (including percent stenosis), stent type, and closure method categories (manual compression, sealant, mechanical, suture, patch, staple, other, or none).

We recognize that the variables have multicollinearity, particularly the derived indicator features based upon the continuous feature counterparts. As this work sought to demonstrate changes in risk over time, we did not remove any variables used in the prior two studies, as we sought to only change the time and addition of new variables at each stage in evaluation. As a result, we acknowledge some co-linearity of variables as a potential limitation in performance and in interpretation. As a result, the S1 Text also discusses the same conducted analysis with a reduced feature set, where those particular indicator features have been removed, and we find that performance is comparable, though slightly stronger when including all possible predictors for the model to explore the decision space with.

Staged model analysis

We sorted all available CathPCI data into key decision stages of a PCI episode of care. First, we defined three decision points that affect bleeding risk: 1) choice of access site (radial versus femoral); 2) choice of medications (including those administered 24 hours pre-procedure and intra-procedure); and 3) choice of closure device indicator variables. While the choice of closure for radial access would be expected to be none, there are closure devices used with Radial Access. Without an ability to do a chart review, we take this data as is (and discuss further in the S4 Text).

Using these three key decision points, we evaluated variables available at three stages: 1) variables available at patient presentation prior to PCI; 2) variables available after diagnostic coronary angiography; and 3) variables related to the PCI procedure. Combining these three decision points and the information available at each of them, six models were designed (Fig 2). The first included only variables available at presentation to the cardiac catheterization lab. Each subsequent model added either a decision node or information that could inform the next decision. The final model included all variables and clinical decisions through the PCI procedure, evaluating remaining bleeding risk for post-procedural care. The final details of the variables and which stage they are provided to are provided in the S4 Text. Knowing that radial and femoral access choice lead to different findings, we also conducted a subgroup analysis in which we evaluated model performance on each subgroup, discussed further in the S2 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Model hierarchy.

Each model integrated information of all features from prior models, as well as an added set of features. Passage through an episode of care proceeds from the top to the bottom. Case study risks over time at each stage are shown here.

https://doi.org/10.1371/journal.pdig.0000906.g002

Data preparation

The NCDR utilizes a high-quality review and adjudication process, ensuring minimal missing data across variables [15]. Additionally, several steps in data preparation were necessary prior to model development, including data cleaning to determine the true rate of missing, including for daughter variables only collected if the indicator variable suggests it should have. Details are provided in the S4 Text.

Once data was cleaned, the missing values were imputed using multiple multivariate feature imputation. Each missing feature was modeled using Bayesian ridge regressors trained in a round-robin fashion. This imputation was performed using the Iterative Imputer package in scikit-learn 0.24.1 [16], based on the multivariate imputation by chained equations (mice) package for R [17]. Following imputation, binary and ordinal variables were set to the nearest allowed value. Multiple imputations were produced by sampling from the regressor models multiple times; each discrete sampling was a new overall sample from the model. This sampling was used to produce five folds of imputations. We used a re-imputation technique for handling the longitudinal nature of the data, producing multiple imputed datasets at each stage through an episode of care [18]. The imputation models were trained on the training dataset prior to cross validation [19]. The test sets were multiply imputed, but the regressors used for this imputation were trained only on training data to avoid data leakage. A positive case was one in which a major bleed occurred, a negative case when no major bleed occurred.

Training, testing, and evaluating

The cohort was divided into temporal subsets: an initial 80% subset (July 1, 2009 through December 31, 2013) for model training and a later 20% subset (January 1, 2014 through December 31, 2014) for model validation [20,21]. This temporal split allows for the fairest possible evaluation of the model through testing it on the most distinct subset. This approach protects against biasing the result of the overall model by including more recent data, as in other recent NCDR models [22]. While an external validation set would be preferable, this method of separating data allows for the most realistic testing of the model (testing models trained on retrospective data on newly-seen patients). Patient characteristics for both the entire dataset as well as the training and validation splits are shown in Table 1.

As discussed above, five imputation folds were created for each stage. The model used was XGBoost, a gradient descent boosted decision tree model [23], as this model has demonstrated improved performance over linear classifiers [4]; because prior work demonstrated the improvement in model performance of XGBoost over linear classifiers (logistic regression), this work focuses only on the improvement in the XGBoost model over the stages of analysis [4]. We conducted an internal five-fold cross-validation to tune the hyperparameters of the model for each stage and each fold. The hyperparameters tuned were maximum depth of each tree (2, 4, 6, or 8), number of tree estimators (100, 500, 1000, or 5000), and learning rate for the model (0.1, 0.15, 0.2, or 0.3). This internal cross-validation was performed using a halving grid search as implemented by the scikit-learn package HalvingGridSearchCV [16]. Optimal performance was found in 27 of the 30 experiments (6 stages * 5 folds) to be given by 1000 estimators with a max depth of 2 and a learning rate of 0.1.

The five imputation folds allow for training and validating the model multiple times, providing both estimates of overall model performance and uncertainty [20]. From this, we calculated the area under the receiver operating characteristics curve (AUROC) for evaluating model discrimination (c-statistic). The AUROC, alone, may be insufficient to properly describe model performance, particularly in the presence of imbalanced event rates, therefore we supplement this measure with additional measures of predictive performance [24]. To better understand positive predictive value across the full range of risk stratification, we also calculated the area under the precision recall curve (AUPRC). Briefly, the precision-recall curve calculates the tradeoff between precision (positive predictive value) and recall (sensitivity) across the full range of thresholds [25]. This model calibration also allows calculation of the Brier Skill Score, which provides an assessment of model calibration on an easy to interpret scale of 0–100% for calibration fit, as well as the Brier Decomposition (Reliability and Resolution), which provides measures of over and under estimation of the calibration curve.

Variable importance

Beyond model performance, model interpretation is a key factor in clinical utility [20]. One approach for interpreting models is SHapley Additive exPlanations (SHAP) [11]. SHAP attributes an importance value to each feature of a given set, therefore allowing for an ordering of features from greatest to least impact on model output. SHAP values were generated to provide visual understanding about the impact of factors driving changes in accuracy of the risk prediction model and decisions through the stages outlined in Fig 1.

To further understand dynamic risk predictions following a decision, shift tables were generated. For these, patients were classified into categories of low risk (<1%), moderate risk (1%-4%), or high risk (>4%) of bleeding, based upon the training set overall event rate of 4.2% and empirically divided to approximately balance patients between categories across all models [22]. These shift tables are useful for visualizing changing risks of bleeding before and after a decision or for comparing the performance of the initial and final models. Shift tables were selected over confusion matrices because a confusion matrix would require setting specific thresholds for a positive or negative prediction; shift tables are a more sophisticated approach to demonstrating confusion matrix results by demonstrating changes in predicted probability without pre-selecting a specific decision threshold.

Case studies

To further understand dynamic risk and changes in variable performance, we then cherry picked two example cases that had changes in risk and explain their journey through the episode of care and the results of which are described in the S5 Text.

All analyses were conducted in Python version 3.8.6 or R version 4.0.3. Data analysis was performed using scikit-learn 0.24.1 [16] and XGBoost 1.3.3 [23] for gradient descent boosting. SHAP explanations were generated and visualized with SHAP 0.38.1 [11]. Model calibrations generated in R with mgcv 1.8-33 [26] and calibration variances with sandwich 3.0-0 [27]. Source code is available online at: https://github.com/stmilab/DynamicBleeding/. Ethics oversight for data and analyses were provided and approved by the Yale Human Research Protection Program (IRB #0607001639). This study was conducted on a retrospective, de-identified registry data, therefore, no consent was needed or obtained for this work.