Readmissions and Death after ICU Discharge: Development and Validation of Two Predictive Models

Introduction Early discharge from the ICU is desirable because it shortens time in the ICU and reduces care costs, but can also increase the likelihood of ICU readmission and post-discharge unanticipated death if patients are discharged before they are stable. We postulated that, using eICU® Research Institute (eRI) data from >400 ICUs, we could develop robust models predictive of post-discharge death and readmission that may be incorporated into future clinical information systems (CIS) to assist ICU discharge planning. Methods Retrospective, multi-center, exploratory cohort study of ICU survivors within the eRI database between 1/1/2007 and 3/31/2011. Exclusion criteria: DNR or care limitations at ICU discharge and discharge to location external to hospital. Patients were randomized (2∶1) to development and validation cohorts. Multivariable logistic regression was performed on a broad range of variables including: patient demographics, ICU admission diagnosis, admission severity of illness, laboratory values and physiologic variables present during the last 24 hours of the ICU stay. Multiple imputation was used to address missing data. The primary outcomes were the area under the receiver operator characteristic curves (auROC) in the validation cohorts for the models predicting readmission and death within 48 hours of ICU discharge. Results 469,976 and 234,987 patients representing 219 hospitals were in the development and validation cohorts. Early ICU readmission and death was experienced by 2.54% and 0.92% of all patients, respectively. The relationship between predictors and outcomes (death vs readmission) differed, justifying the need for separate models. The models for early readmission and death produced auROCs of 0.71 and 0.92, respectively. Both models calibrated well across risk groups. Conclusions Our models for death and readmission after ICU discharge showed good to excellent discrimination and good calibration. Although prospective validation is warranted, we speculate that these models may have value in assisting clinicians with ICU discharge planning.


Introduction
Prolonged duration of stay in the intensive care unit (ICU) is costly, is stressful for patients and families, reduces the number of beds available for other patients, and can increase risk for iatrogenic and nosocomial complications. [1] ICU daily care costs are 2-3 fold higher than costs on general medical -surgical wards, reflecting both higher staffing ratios and greater resource consumption. [2] Strategies to decrease ICU length of stay (LOS) can improve patient throughput and increase the number of patients that can be cared for in capacity-constrained ICUs. Patients also benefit from shorter exposure to the disruptive ICU environment, and may have less sleep disruption caused by intensive monitoring and frequent audible alarms. [3,4].
Early discharge from the ICU is not without risk. If patients requiring high intensity care are discharged before they can be safely cared for in a lower acuity care environment, they are at risk for both complications and delayed recognition of clinical deterioration. The former can result in the need for unplanned ICU readmission; the latter can result in patient death. Patients readmitted to ICUs have higher risk-adjusted mortality and lengths of stay. [5][6][7] The actual increases in mortality and LOS may be modified by contextual factors such as bed occupancy rates and patient inflow volumes. [8,9] In addition, ICU readmission also places stress on patients and families.
Determining who is ready for ICU discharge is a daily challenge for ICU leaders, especially in units with high occupancy rates. Traditionally these decisions are made by attending physicians, in collaboration with other members of the ICU care team. [10] Due to the highly subjective nature of these decisions, there is considerable variability in determining discharge readiness. [11] There are few data on why patients deteriorate after ICU discharge, and differentiating problems present at the time of discharge from those that originate after discharge oftentimes is not possible. In the absence of this information it is generally assumed that the shorter the time between discharge and readmission or death, the more likely the patient was not 'ready' to be discharged from the ICU. As a result, 48 hours has historically been considered the primary timeframe for evaluating the quality of ICU discharges. [12].
Several studies have evaluated post-discharge patients and identified variables that predict these complications. [6][7][13][14][15][16][17][18][19][20][21][22] Previously identified predictors of death or readmission include duration of ICU LOS, Glasgow Coma Scale (GCS) score at the time of ICU discharge, mean arterial blood pressure and ICU admission source. [15] Some investigators have attempted to create decision support tools to assist in discharge readiness assessment. Zimmerman and co-workers utilized the probability of next day risk for life support as a proxy means for determining discharge readiness. [21] They reported that the greatest risk factors were the current day's therapy and the Acute Physiology and Chronic Health Evaluation III (APACHEH) score (Cerner Corp, Kansas City, MO). The SWIFT score, which focused on patients with readmission or death within 1 week of ICU discharge, demonstrated moderate discrimination, although significantly higher than the day of discharge APACHE III score. [13] The present study leverages a large and rich database of over 1,500,000 critically ill patients cared for in several hundred United States ICUs. [23,24] The objective of this study was to attempt to develop robust predictive models, when embedded in electronic clinical information systems (as the ICU Discharge Readiness Score), might have value as a decision support tool to assist ICU leaders in making discharge decisions. We hypothesized that the very large sample size and the inclusion of several hundred different ICUs would provide sufficient power and heterogeneity to enable the creation of generalizable predictive algorithms.

Methods
This was a retrospective, multi-center, exploratory cohort study utilizing ICU patients in the eICUH Research Institute (eRI) database with a complete hospitalization between January 1, 2007 and March 31, 2011. Detailed descriptions of the eRI database are provided elsewhere. [23,24] Although specific data on hospital demographics were not included in the analytic dataset, the eRI database represents geographically dispersed hospitals, with approximately 50% being teaching hospitals, 34% with over 500 beds and 12% with less than 100 beds. [11,13] This study was exempt from IRB oversight as there were no patient interventions due to the retrospective design and the security schema for the eRI database was analyzed and re-identification risk was certified (45 Code of Federal Regulations 164.514(b)(1) and 45 C.F.R. 164.514(b)(1)(i); HIPAA Certification #80503C) as meeting safe harbor standards by Privacert, Inc. (Pittsburgh, PA). All patients discharged from participating ICUs were included in the analysis unless any of the exclusion criteria were met. Patients were randomized in a 2:1 fashion to development and validation cohorts. Patients with the following conditions were excluded from the analysis: ICU LOS of less than 4 hours; age ,16 years; expired in the ICU; discharge location of transfer to another ICU or locations external to the hospital; and the presence of a ''do not resuscitate'' (DNR) or ''comfort measures only'' (CMO) order at ICU discharge. Due to the retrospective design, all discharges from the ICU were made at the discretion of the attending physician. The primary study objective was to develop two predictive models; one predicting death and the other predicting readmission within 48 hours of ICU discharge.
Differences in baseline characteristics between the development and validation cohorts were assessed with Pearson Chi-square for categorical data, student t-test for normally distributed continuous variables and Wilcoxon-rank-sum for non-normally distributed continuous variables. Using the development cohort, associations between predictor variables and the primary outcomes (death or readmission) were evaluated using multivariable logistic regression. Continuous variables were assessed for non-linear relationships with the primary outcome using locally weighted scatterplot smoothing (LOWESS). Non-linear relationships were handled via introduction of spline terms (knots) or categorizing continuous variables. Spline terms were introduced to create intervals of existing linear relationships which changed slopes at knots designated by visual inspection of the locally weighted scatterplot smoothing.
59 different variables were evaluated for inclusion in the predictive models for post-discharge death and readmission within 48 hours of ICU discharge based upon clinician assessment of possible relevance. Variables included: patient demographics, ICU admission diagnosis, admission severity of illness determined by the APACHE score, intensive care interventions, complications occurring during the ICU stay, and laboratory and physiologic variables from the last 24 hours of the ICU stay. A complete list of variables included is described in Table S1. In order to reduce the number of diagnoses used in the model the 407 unique APACHE admission diagnoses were consolidated into 26 diagnosis groups (Table S2). Diagnoses were first ranked by prevalence and then grouped according to pathophysiology, with all rare diagnoses unrelated to newly created diagnosis groups categorized together as ''Other''. The number of patients in the development cohort with original data available is presented for each predictor. To reduce the potential for introducing bias due to missing data patterns, multiple imputation was used for all predictors included in the final model unless specified in Table S1. [25][26][27] Multivariable regression was used to create five imputations using chained equations (ICE) via the ''mi impute chained'' command in Stata 12 (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP).
A combination of methods was used to identify the initial set of possible predictors of death or readmission within 48 hours of ICU discharge. These included prior literature, clinical knowledge and forward and backward step-wise multivariable logistic regression. [5,13,18,22] Variables were included in the step-wise regressions if the difference in log likelihood between the null versus extended models produced a p-value ,0.05 using the log-likelihood ratio test for readmission and a p-value ,0.01 for death. A more conservative threshold was used for the risk of death model due to the greater number of variables significantly associated with the outcome. All variables identified by these means were included in the initial development models. These were reduced to more parsimonious models by examining the difference in area under the receiver operating characteristic curve (auROC) between the null and extended models. As the ultimate goal is to develop predictive models that can be embedded in electronic clinical decision support tools, inclusion in the final models was based on balancing model performance against availability of data in the clinical information system. Therefore, some variables which did not tangibly improve model performance were excluded even if a significant association with the endpoint existed. As opposed to epidemiology studies seeking to quantify the relationship between specific variables and outcomes, the focus of predictive modeling research is on model accuracy. Therefore, collinearity between variables was allowed (e.g. use of both average heart rate and highest heart rate) because it improved performance of the predictive models. However, allowing multiple related variables makes it difficult to clinically interpret the adjusted odds ratios. The unadjusted odds ratio (OR) is presented for each of the covariates included in the final models.
The primary analytic measure used to assess model discrimination was the auROC for the development and validation cohorts. The Hosmer-Lemeshow goodness-of-fit test along with visual inspection of calibration curves were used to assess calibration across deciles of risk. The median and range for the discrimination and calibration across the five imputed datasets was reported. Performance of the models also were assessed in different ICU patient types (e.g., medical, surgical) and across hospital size and teaching status by comparing actual to predicted event rates within groups. A secondary validation step was performed to simulate expected real-time performance as a clinical decision support tool in the ICU. Fitted values (predicted probabilities) were determined using patients' clinical data at 24 hour intervals prior to ICU discharge. The median and interquartile range (IQR) of the fitted values were calculated for up to 4 days prior to ICU discharge for 3 patient groups: a) patients discharged without an event of interest within 48 hours; b) patients discharged with an event within the subsequent 48 hours; and c) patients who did not survive the ICU stay. A linear regression (with a robust variance estimator clustered by patient) of the fitted values across the last   Table 1. Although the median LOS appears low for this cohort of ICU survivors, the mean and standard deviation for the ICU and hospital LOS were 3.0 (63.6) days and 4.3 (65.0 days) respectively.
Of the 59 variables initially analyzed in the development set, 26 and 23 were retained in the final models for death and readmission, respectively. Tables 2 and 3 show the unadjusted ORs of variables used in each model. Eight variables present on admission to the ICU were retained in at least one of the final models; admission diagnosis (including whether related to elective or emergent surgery), admission source, unit type, ICU visit number, age, and BMI. The remaining predictors came from data obtained during the last 24 hours of the ICU stay. Numerous continuous predictors had non-linear associations with the study outcomes, and were handled with spline terms or by categorizing in the final model. For example, for average heart rate over the last 24 hours, the odds of death decreased by 6% for each increase in beat per minute (bpm) up to 60 bpm, but increased by 5% for each bpm above 60. The relationship between the independent variables and the two separate outcomes were not necessarily consistent. In general, the relationships between independent variables and outcome were stronger for predicting death than readmission. In some cases there were clinically different relationships, as observed with average diastolic blood pressure over the prior 24 hours, where the odds of death increased by 8% for each mmHg over 100 mmHg while readmission risk was unchanged above 82 mmHg.
Across the multiply imputed datasets, the final readmission model produced a median auROC of 0.71 (range: 0.7058-0.7061) in the development set (N = 469,976) and 0.71(range: 0.7060-0.7068) in the validation set (N = 234,987). Figure 2 displays the median ROC curve for the validation cohort. Figure 3 displays the final model predicting death within 48 hours of ICU discharge,      [28], the calibration across deciles of risk for both models are presented in Figures 4 and 5 to provide clinical perspective to differences in actual to predicted rates. The actual to predicted rates of death and readmission across categories of hospital bed count and teaching status are presented in Table 4. Actual to predicted rates for different ICU types are presented in Table 5.
The median sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) at various predicted probabilities of early death and readmission after ICU discharge are presented in Table 6. Due to the relatively low prevalence of each outcome, both models tend to have high negative predictive values and low positive predictive values. 17% of ICU discharges went to a step-down unit compared to 83% to a general ward. Using a robust variance estimator clustered by ICU, the unadjusted OR for death was 41% higher (p,0.03) if discharged to a step-down unit, but this association was reduced once adjusted for the predicted log-odds of death (OR = 1.31; p = 0.02). The unadjusted odds of readmission were 35% lower in a step-down unit and remained 36% lower after adjusting for the predicted log odds of readmission (p,0.001 for both).
In order to simulate performance in a real-time environment, fitted values were generated for patients between one and four days prior to ICU discharge in the validation cohort. Table 7  shows the median and IQR for the predicted risk of death and readmission in patients who did not have a complication within 48 hours of ICU discharge, had a complication within 48 hours after discharge and those who did not survive their ICU stay. The average change in predicted risk of death across the last four ICU days is graphically represented in Figure 6.

Discussion
The rate of readmission and death within 48 hours of ICU discharge was approximately 2.5% and 0.9%, respectively, in this heterogeneous critically ill population from 219 hospitals and 402 ICUs. Predictive algorithms for death and readmission within 48 hours of ICU discharge developed from this large dataset had excellent and moderate discrimination, respectively, and both calibrated well across risk strata. Despite efforts to limit the number of variables in the predictive models, optimal performance required more than 20 variables, a number that significantly exceeds human processing capacity but relatively simple for a computer to analyze. We intend for these models to be calculated automatically on a continuous basis for patients all patients eligible for consideration for ICU discharge. Although predictions would only be generated for patients with complete data available, the use of multiply imputed data suggests the discrimination of the    models are robust to the missing data patterns observed in this cohort. We speculate that post discharge risk estimates might be helpful in discharge decision making given the time devoted to this activity and the adverse consequences of delaying discharge or sending patients out prematurely.
Prior studies of post discharge adverse events have generally focused on readmission as the endpoint of interest or have used a combined endpoint of readmission or death. [13,17,18] This appeared to be a rational approach because it seemed plausible that both unexpected readmission and death were due to post discharge clinical deterioration. Moreover, these smaller studies lacked sufficient numbers of post discharge deaths to analyze the two endpoints separately. The auROC in these previous studies ranged between 0.62 and 0.74, and calibration was generally poor. [13,17,18] It should be noted that the validation arm for the SWIFT score was a cohort of patients from a different institution, which is inherently more difficult to achieve good calibration than during internal validation such as ours. [13] In our analysis, the auROC for the readmission model was similar to those described in prior studies, although calibration was excellent, especially considering it is rare to observe a positive goodness of fit test with large sample size. [28] Moreover, the model performed well across different types of ICUs and across hospitals of different sizes and teaching status which we hypothesize is a function of the broad population from which it was generated. The moderate discrimination of our model suggests that a significant proportion of factors contributing to ICU readmission are not clinically observable at the time of ICU discharge. These may include the quality of care provided at the lower acuity (receiving) unit and provider variability in assessing readmission need. There also appears to be a lower likelihood of readmission when the lower acuity unit is a step-down unit compared to a general ward. We attempted to minimize the influence of care provided after discharge by limiting the outcome to within 48 hours of ICU discharge, but despite this design, latent variables appear to affect readmission risk. Surprisingly, this limitation was not a major factor when predicting death shortly after ICU discharge.
With 469,976 patients and over 4,389 deaths in the development cohort, we were able to model death as a distinct outcome; the differences between the two models were striking. The risk of death model had extremely high discrimination reflected in an auROC of 0.92. Although this seems implausibly high, close examination of the predictors included in the model indicated very strong relationships with death after discharge ( Table 2). Although the explanation for superior discrimination is unknown, we speculate that the endpoint of death is less likely to be influenced by subjective decisions driven by social and political factors. In support of this hypothesis, the model predicting death requires fewer static patient characteristics such as, admission diagnosis, type of ICU and admission source than the readmission model. It also is likely that ICU readmission criteria vary among the 219 hospitals included in this cohort. We also noted that many  physiologic variables, when significantly abnormal, were associated with a high risk of death, but a low risk of readmission. For example, the risk of death increased linearly for white blood cell counts above 9,000 cells/mL, but the risk of readmission declined above 40,000 cells/mL (Tables 2 and 3).
Although the models were developed on data available on the day of discharge, the secondary validation we performed using patient data from one, two and three days prior to ICU discharge suggest that the model might provide clinical utility. The model for death discriminated between patients who were stable enough for ICU discharge from those who were not. Despite excluding all patients who died in the ICU during model development, the predicted risk of death in these patients was dramatically higher than those who were discharged without complication, even up to four days prior to discharge (Table 7 and Figure 6).
Ideally these models can be incorporated into an electronic clinical decision support tool for use in ICU discharge planning, with a goal of improving patient safety and increasing ICU throughput. Zimmerman and Kramer have previously suggested that patients admitted to the ICU with a low predicted risk of active treatments, referred to as ''low-risk monitor'' patients, may not necessarily require an ICU admission. [29] Using the eRI database from 2008, Lilly et al. reported that approximately 40% of ICU admissions had low day 1 mortality risk and received no major therapies. [24] The trends observed in Table 7 appear to support this notion that a substantial proportion of ICU days are attributable to patients not requiring intensive care treatments. In translating these into CDS tools, we believe that due to the differences in performance, clinicians should view the risk of readmission as complementary information to the more accurate predictions generated from the model predicting death. We also believe there is value in defining multiple categories of risk, rather than trying to use a single threshold to determine whether a patient is ready for discharge. For example, if a threshold for 'low risk' of death is defined as a prediction under 0.1%, greater value can be derived from the high NPV. With this threshold, 32% of the validation cohort is defined as 'low risk' at the time of discharge with a false negative rate of 0.046%. For readmission, if  The ICU Discharge Readiness Score 'low risk' is defined as below 1% predicted probability of readmission, 19% of the validation cohort was below this threshold and the false negative rate was only 0.5% ( Table 6). The simulation study also identified many patients with low risk of death and readmission who stayed in ICU for additional days. 21% and 18% of all patients discharged would have been classified as 'low risk' of death and readmission, respectively 24 hours prior to their actual ICU discharge. Prospective validation is clearly warranted, but there appears to be an opportunity to use this tool to support clinical programs aimed at reducing unnecessary ICU days. As Zimmerman and Kramer point out, ''Improved resource use and reduced costs might be achieved by strategies to provide care for these patients on floors or intermediate care units.'' [29] At the other end of the risk spectrum, our data indicate that more than 2% of discharges either died or required readmission. If 'high risk' were defined as greater than 5% predicted risk of death, 2.4% of those discharged would have been identified as 'high risk' at the time of discharge, with 22% of these dying in the subsequent 48 hours. We speculate that some of these patients were not recognized as being at high risk for death after discharge and providing this information may help clinicians decide to prolong the ICU stay in these patients. Although we could not confirm this, it seems plausible the unexpected deaths represented sudden cardiorespiratory arrest whereas a more gradual deterioration may often be required for a readmission to occur. With an auROC of 0.92, these deaths were predictable and potentially preventable. Given this is a very small proportion of ICU discharges, considering a prolonged ICU stay to mitigate risks may be a reasonable choice.
This study has several important limitations. Calculation of the ICU Discharge Readiness Score is relatively complex and cannot be performed manually, although it can be easily programmed to provide electronic clinical decision support. As a retrospective study, the accuracy and completeness of data is limited by the quality of documentation in the clinical information system, which likely is less accurate than data collected in a tightly controlled clinical trial. This approach was utilized to create a very large sample size with many sentinel events (death and readmission) and to include data from many different institutions. Several strategies were employed to minimize artifacts. Vital signs in the eRI database are archived as 5-minute medians (from the 1-minute averages received from bedside monitors via interfaces). Many of the variables used 24 hour averages to minimize the impact of outlier data. Patients with documented care limitations were excluded from analysis because death in some of these patients was seen as their expected outcome, and readmission would have been inappropriate. It is possible that some patients with care limitations lacked appropriate documentation and were included in the analysis.
In general, data completeness was reasonable, especially considering missing data tends to be more common on the day of ICU discharge when patients are more stable than earlier in the ICU stay. Most variables were available during the final 24 hours before ICU discharge with the exception of GCS scores. The high proportion of missing GCS values on the day of discharge may be due to variability in the use of GCS scores in some ICU populations and the tendency of eICU Programs to use the remote management software for population management, rather than focusing on comprehensive documentation for the medical record. Some participating health systems have not implemented an interface to import nursing flowsheet data from the primary electronic medical record, which results in absence of GCS values. As shown by other investigators (e.g., SWIFT) and confirmed in this analysis, GCS is an important predictor of death and readmission after ICU discharge.(12) Introduced by Rubin, the method of choice for reducing bias introduced by missing data is through multiple imputation. [25][26][27] Development and validation on multiply imputed data provides confidence that our models are robust to variations in real-world documentation practices. Because our ultimate goal is to have accurate predictive models that can be used in large numbers of patients, rather than establishing causality, decisions regarding which data elements are used in the final models must balance data availability, data reliability and model performance considerations. High degrees of collinearity (e.g., use of both average heart rate and highest heart rate) were allowed because the goal was not to determine the independent effect of factors such as, average heart rate, but to provide the best estimate of the risk of post-discharge death and readmission. As a result, it is not possible to clinically interpret the adjusted odds ratios for many of the variables (data not shown). Also, the list of variables predictive of readmission and death presented is not exhaustive since variables were only retained if they resulted in improved performance of the models. Lastly, these models should be viewed as tools to support clinical workflow rather than replace clinical judgment. We believe any successful program for improving ICU discharge planning will require establishing standardized processes to reduce the variability with which these decisions are generally made today.

Conclusion
Death and readmission within 48 hours of ICU discharge are clinically distinct outcomes, requiring independent predictive models. The predictive models for death and readmission calibrate well across deciles of risk and exhibit excellent and moderate discrimination respectively. The model predicting the risk of death accurately discriminated between patients who would and would not experience a complication as early as four days prior to ICU discharge. We speculate that these predictive models may improve ICU discharge planning if incorporated into a clinical decision support application that can provide actionable information to ICU clinicians.