Patients

This retrospective analysis evaluated data from hospitalized adult patients from three hospitals—Yale New Haven Hospital (YNHH), St. Raphael’s Hospital (SRH), and Bridgeport Hospital (BH)—admitted from 12/31/2012 to 2/09/2016 who had at least two inpatient creatinine values who were considered for inclusion. YNHH is a large tertiary care center set in an urban setting, SRH is a community teaching hospital, and BH is a private acute care hospital in an urban setting. All three hospitals used the same inpatient EHR (Epic, Verona, Wisconsin). Patients who were missing discharge times and those with an admission or prior ICD-9 or ICD-10 code consistent with end-stage kidney disease were excluded. Patients with an admission creatinine greater than or equal to 4 mg/dl were also excluded (Fig 1). Ethics approval was obtained from the institutional review board and the study operated under a waiver of informed consent from the Yale Human Research Protections Program.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow diagram of the patient cohort with distribution of data among training and validation data sets.

https://doi.org/10.1371/journal.pmed.1002861.g001

Study reporting

This study utilized the Strengthening the Reporting of Observation Studies in Epidemiology (STROBE) guidelines, as per the S1 Checklist, for reporting.

Variable ascertainment

Demographics, vital signs, and laboratory data were obtained directly from the EHR, and all time-variable data (e.g., creatinine) was assigned a timestamp for further analyses. Our data set included the following variables, which were extracted electronically from the EHR: age, sex, race (measured as black or non-black), whether the admission was a surgical admission, patient history of congestive heart failure (CHF), hypertension, liver disease, on-admission Elixhauser score, laboratory findings including last bicarbonate, blood urea nitrogen (BUN), chloride, baseline creatinine, last creatinine, change in creatinine over the last 48 hours, hemoglobin, potassium, sodium, white blood cell count, platelet count, requirement during hospitalization for Bilevel Positive Airway Pressure (BiPAP), contrast studies, invasive ventilation, ICU admission, cardiac catheterization, red blood cell (RBC) transfusion, administration of angiotensin converting enzyme inhibitors (ACEs) or angiotensin II receptor blockers (ARBs), antibiotics, chemotherapy, diuretics, narcotics, nonsteroidal anti-inflammatory drugs (NSAIDs), vasopressors, proton pump inhibitors, and statins. Each patient would have a data row for each time point that a measurement (e.g., a vital sign) took place. Variables were carried forward through time until a new measurement for that variable took place. A visualization of this data set construction for a theoretical patient is provided in S1 Fig.

Clinical definitions

AKI was defined per the Kidney Disease: Improving Global Outcomes (KDIGO) creatinine criteria [20]. Due to the relative sparsity of urine output data on the general hospital wards, urine output criteria for AKI were not considered. Thus, our definition of AKI was an increase in serum creatinine by 0.3 mg/dL in 48 hours or an increase in serum creatinine of 1.5 times baseline, which was defined as the lowest measured creatinine over the preceding 7 days. Also, given that the data set only contained inpatient data, community-acquired AKI was not studied in this model. The primary goal was to predict “AKI within 24 hours,” i.e., any set of measurements within 24 hours of AKI onset were considered “positive,” while any measurements prior to 24 hours of AKI onset (and all time points among the patients who did not develop AKI) were considered “negative.” AKI was only defined in relation to inpatient creatinine values; a patient’s “baseline” creatinine was not imputed from previous hospitalizations or outpatient lab studies. Time points before the first creatinine measurement would have “missing” values and AKI could not develop until a patient would have at least two measured creatinine values. Time points occurring after AKI onset were not included in modeling.

Statistical methods

We used descriptive statistics to characterize patients across the three hospitals and by AKI status. To compare continuously and across the study hospitals, we used Kruskall–Wallis and chi-square testing, respectively. We assessed the univariable differences in patient characteristics at time points prior to AKI and not prior to AKI by univariable logistic regression with clustering at the patient level to account for the nonindependence of within-patient observations.

For our AKI prediction model, two-thirds of data from YNHH were used for training and one-third for internal validation. External validation was performed in the two other study hospitals (BH and SRH). We utilized a discrete-time logistic regression approach for prediction, with a new prediction being generated each time a covariate value was updated in the EHR. We again accounted for nonindependence of within-patient observations by clustering at the patient level. Candidate covariates were selected based on prior research and additionally considered variables measured in >90% of hospitalized patients (S1 Table) [21, 22]. Model covariates were divided into four classes: demographics, which included all time-invariant covariates; medications; and laboratory values and procedures. Given the uniquely strong performance of the rate of change of creatinine, we additionally modeled this feature independently. We assessed model performance using the AUC, again clustering at the patient level. The model was trained and evaluated on the primary outcome, a time-varying binary variable representing “AKI onset within 24 hours of this time point.” We measured covariate significance as the absolute value of the covariate’s Wald z-score in the full multivariable model.

While the primary outcome was AKI within 24 hours, we also evaluated model performance on the other kidney-relevant outcomes, including sustained AKI within 24 hours (defined as at least two consecutive creatinine values consistent with an AKI diagnosis), inpatient dialysis, and inpatient death.

We assumed time-varying variables were static after measurement, e.g., if the potassium level was measured as 5.1 meq/L, it was considered to be 5.1 meq/L at all future time points unless remeasured. Missing data was limited to laboratory values and represented less than 10% of all observations. Missing data were not imputed; it was decided to not generate predictions for time points when relevant data was unavailable, as this is what would occur in real-time should models be built into the medical record. In large data sets, case-wise deletion of incomplete records may be less biased than various imputation methods [23, 24]. Stata v. 15 (StataCorp., College Station, Texas) was used for all statistical tests.

Ethics statement

Ethics approval was obtained from the institutional review board and the study operated under a waiver of informed consent from the Yale Human Research Protections Program.

Strengths

Our study has several strengths. First, our model is generalizable to a variety of inpatient care settings, as it was developed on a large cohort from a tertiary care center (YNHH) and further validated at a community teaching (SRH) and nonteaching (BH) institution and maintained good predictive performance across the three diverse care settings. Second, our model is generalizable to a variety of inpatients, as it was developed on data from patients in both hospital floor, surgical, and ICU settings. Lastly, our laboratory-based model is simple and holds promise for ready implementation into the EHR; this model is currently live and being evaluated prospectively in one of the study hospitals.

Limitations

Our study should be interpreted in the light of several limitations. First, the predictability of the model yielded similar AUCs to prior models with respect to predicting at least stage I AKI; thus, we do not claim to have developed a model that is far superior to other models [10, 16, 17]. We also recognize that a recent study found AUCs of 0.9 for AKI prediction; we note, however, that this model predicted stage II AKI (or greater) rather than stage I (or greater), and our goal was to predict all hospital AKI irrespective of severity [16]. We hypothesize preventing stage I may be more clinically beneficial for reversing disease rather than a stage II prediction, at which point it may be too late to change disease course. Additionally, this and other models contain a high quantity of variables, some calculated, which may be difficult to operationalize across diverse health systems and EHR platforms. Second, our model is descriptive in nature; while it can reveal factors associated with imminent AKI, it is unable to imply causality or provide explanations for unexpected findings. Third, given sparsity of data, we were unable to include urine output into the model, and we recognize that urine output has significant implications in terms of AKI incidence [30]. In addition, because we limit our definition of AKI to look back 7 days from admission for baseline creatinine, it is likely that our model misses some patients who present with AKI but do not have recent creatinine values from which the patient would meet our definition. Fourth, all hospitals included in our analysis utilized the Epic EHR system, and the model’s performance should ultimately be validated on healthcare data obtained from other EHRs. A final limitation is that our model does not suggest the best manner in which providers should react to notification of imminent AKI in a patient.

Future implications

To date, there remain limited interventions for treatment of hospitalized patients who develop AKI. The development of predictive models could aid in AKI prevention and change a patient’s course of disease.

The parsimonious model developed in this study should be further evaluated on other large hospital data sets to validate model performance. Further work should be performed to determine optimal implementation of such a model into the EHR and how best to use the predictions to affect provider behaviors while minimizing intrusiveness of such interventions and maintaining sensitivity to increasing concerns of “alert fatigue” [31].

The laboratory model presented in this paper can be implemented in practice with different cut-points for prediction to either optimize sensitivity or specificity of a model. For example, from our AUROC curves of the laboratory-only model, prediction could be generated with approximately 80% sensitivity and 50% specificity. With a prevalence of AKI of approximately 15% (as seen in our data set), such a prediction would carry a positive predictive value (PPV) of 22%. This cut-point may be well-suited for low-level interventions (such as additional monitoring or avoidance of nephrotoxins). Alternatively, a cut point could be generated with approximately 87.5% specificity but only 37.5% sensitivity. At the same prevalence, this prediction would be expected to have a PPV of 35%. With this more specific prediction, one might consider providing higher-level interventions, such as provision of intravenous fluids or usage of pharmacy consultants. Further work needs to study such model implementations and see how they best fit within workflow and optimize patient care.

In addition, after the deployment of such a model on the EHR, specific interventions could be randomized and tested among patients to evaluate novel therapies for prevention of AKI.

Conclusion

Our study developed an original model for 24-hour prediction of AKI in hospitalized patients with good predictability in the domains of AKI prediction as well as sustained AKI, requirement for renal replacement therapy, and mortality. Previously, integration of real-time predictive analytics for AKI have been limited, in part due to implementation difficulties and in part due to large model complexity [32–34]. Compared to prior studies, our study has shown that a parsimonious model using only laboratory values maintains good performance and lends itself to ready implementation in EHR. We anticipate that broad implementation of this algorithm would change the treatment paradigm of AKI from reactive to proactive, which will afford new opportunities for the evaluation, diagnosis, and treatment of the syndrome.