Improved predictive models for acute kidney injury with IDEA: Intraoperative Data Embedded Analytics

Background Acute kidney injury (AKI) is a common complication after surgery that is associated with increased morbidity and mortality. The majority of existing perioperative AKI risk prediction models are limited in their generalizability and do not fully utilize intraoperative physiological time-series data. Thus, there is a need for intelligent, accurate, and robust systems to leverage new information as it becomes available to predict the risk of developing postoperative AKI. Methods A retrospective single-center cohort of 2,911 adults who underwent surgery at the University of Florida Health between 2000 and 2010 was utilized for this study. Machine learning and statistical analysis techniques were used to develop perioperative models to predict the risk of developing AKI during the first three days after surgery, first seven days after surgery, and overall (after surgery during the index hospitalization). The improvement in risk prediction was examined by incorporating intraoperative physiological time-series variables. Our proposed model enriched a preoperative model that produced a probabilistic AKI risk score by integrating intraoperative statistical features through a machine learning stacking approach inside a random forest classifier. Model performance was evaluated using the area under the receiver operating characteristic curve (AUC), accuracy, and Net Reclassification Improvement (NRI). Results The predictive performance of the proposed model is better than the preoperative data only model. The proposed model had an AUC of 0.86 (accuracy of 0.78) for the seven-day AKI outcome, while the preoperative model had an AUC of 0.84 (accuracy of 0.76). Furthermore, by integrating intraoperative features, the algorithm was able to reclassify 40% of the false negative patients from the preoperative model. The NRI for each outcome was AKI at three days (8%), seven days (7%), and overall (4%). Conclusions Postoperative AKI prediction was improved with high sensitivity and specificity through a machine learning approach that dynamically incorporated intraoperative data.

Introduction Acute kidney injury (AKI) is one of the most common, yet underdiagnosed, postoperative complications with lasting consequences [1,2]. It is associated with an increase in mortality, short-and long-term morbidity, chronic kidney disease, and cardiovascular disease [3][4][5][6][7]. An episode of postoperative AKI imposes an average hospital cost increase of $9000, even after adjusting for all other complications [8,9]. The implementation of existing clinical guidelines for prevention and treatment of AKI is often hindered by the inability to accurately and timely assess the risk for AKI while accounting for the dynamic nature of the pathophysiological events during surgery.
With the advancement of digitalization in clinical medicine and the widespread availability of electronic health records (EHR), a number of predictive models have been developed to estimate the risk for AKI in different clinical settings, including after surgery [10,11]. A majority of the existing AKI risk models are limited to preoperative factors [12], applicable only to a specific surgery type [13,14]. Some of the available online prognostic calculators [15] are designed for intensive care unit (ICU) patients only, and do not take any surgical features into account. While preoperative models for AKI mainly rely on patients' pre-existing health conditions and general risks associated with the type of surgical procedure, there is a wealth of intraoperative data reflecting the acute physiological responses to the stresses of surgery that is being ignored. Although recent studies have demonstrated an association between AKI and low intraoperative hemoglobin and hypotension [7,16,17], there is still a lack of comprehensive postoperative AKI prediction models that dynamically integrate physiologic intraoperative data with preoperative information.
Thus, there is a need for intelligent, accurate, and robust systems that are able to leverage temporal information related to patients' physiological changes during surgery. We have recently developed and validated a machine learning algorithm, MySurgeryRisk, which predicts preoperative risk for major postoperative complications, including AKI, using EHR data [18]. The aim of this study was to develop and validate a dynamic machine-learning algorithm that readjusts the preoperative risk for AKI, using physiological time series and other data collected during surgery, to provide a personalized risk panel for AKI with both preoperative and immediate postoperative risk assessments.

Materials and methods
This study was approved by the University of Florida Institutional Review Board and Privacy Office as an exempt study with a waiver of informed consent. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) recommendations were followed under the Type 2a analysis category (random split sample development and validation) (S1 Table) [19].

Data source
The University of Florida Integrated Data Repository was used as an honest broker to assemble a single center longitudinal perioperative cohort for all patients admitted to the University of Florida Health for longer than 24 hours following any type of operative procedure between January 1, 2000 and November 30, 2010 by integrating electronic health records with other clinical, administrative, and public databases as previously described [12]. The resulting dataset included detailed information on patient demographics, diagnoses, procedures, outcomes, comprehensive hospital charges, hospital characteristics, insurance status, laboratory, pharmacy, and blood bank data as well as detailed intraoperative physiologic and monitoring data for the cohort.

Participants
We identified patients 18 years of age or older that were admitted to the hospital for longer than 24 hours following any type of inpatient operative procedure. If patients underwent multiple surgeries, only the first surgery was used in our analysis. Patients with end stage renal disease prior to admission (n = 1,935) and patients with missing serum creatinine values during hospitalization (n = 6,636) were excluded from our analysis. From the remaining cohort there were 2,911 patients who had complete intraoperative data for all vital signs, laboratory values, and medications.

Outcomes
The main outcome was the development of postoperative AKI within the first seven days after surgery. The secondary analysis modeled a) the risk for the development of AKI within the first three days after surgery, and b) the risk of developing postoperative AKI at any point during the hospitalization for the index surgery. AKI was defined using the consensus Kidney Disease: Improving Global Outcomes (KDIGO) criteria as at least a 50% or 0.3 mg/dl increase in serum creatinine relative to the reference creatinine [20]. Reference creatinine was determined based on the availability of measured creatinine prior to admission. The minimum serum creatinine value was used if results were available within seven days of the index hospitalization. If not available, the median serum creatinine value obtained within 8-365 days prior to admission was used. For patients without a prior creatinine value within the year prior to admission and no history of chronic kidney disease, an estimated reference serum creatinine was used [21][22][23]. The estimated reference serum creatinine was calculated by solving the abbreviated "Modification of Diet in Renal Disease" equation for creatinine, assuming a glomerular filtration rate of 75 ml/minute/1.73 m 2 . After the first seven days of the index hospitalization, the minimum serum creatinine from the preceding seven days was used as the reference creatinine [12,24]. Patients with chronic kidney disease and end stage renal disease prior to admission were identified using the validated combination of The International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes [25,26]. Exact dates were used to calculate the duration of mechanical ventilation and intensive care unit stay. A set of previously described criteria was used to annotate the remaining clinical outcomes [8].

Predictor features
The preoperative risk assessment used demographic, socio-economic, administrative, clinical, pharmacy, and laboratory data available prior to surgery to derive 285 preoperative predictor features from 69 preoperative variables (S2 Table). Preoperative comorbidities were derived using up to 50 ICD-9-CM codes as binary variables and the Charlson Comorbidity Index [7,18,27,28]. A reference estimated glomerular filtration rate (eGFR) was calculated for all patients using the reference serum creatinine, sex, race, and age [29]. The immediate postoperative risk reassessment used the following intraoperative variables: five physiologic time series (mean arterial blood pressure (MAP), systolic blood pressure, diastolic blood pressure, minimum alveolar concentration (MAC) of inhaled anesthetics, and heart rate (HR)), 21 repeated laboratory measures, and other discrete variables (intraoperative medications, duration of the operation, anesthesia type, etc.) (S2 Table).

Sample size
Two thousand nine hundred eleven (2,911) patients were included in the cohort. The data was randomly split into 70% for training and 30% for validation. The algorithm was trained on the development cohort while results were reported from the validation cohort. By using 30% of the cohort for validation (n = 873), the overall sample size allows for a maximum width of the 95% confidence interval for the area under the receiver operating characteristic curve (AUC) of 0.08, when prevalence of AKI is between 30% and 40%.

Predictive analytics workflow
The proposed IDEA (Intraoperative Data Embedded Analytics) algorithm is conceptualized as a dynamic model that readjusts the preoperative risk for AKI using physiological time series and other data collected during surgery. The resulting adjusted postoperative risk is assessed immediately at the end of surgery. This flow simulates the clinical task faced by physicians involved in perioperative care where patients' preoperative information is subsequently enriched by the influx of new data from the operating room. The final output produces IDEA, a personalized risk panel for AKI after surgery (Fig 1) with both preoperative and immediate postoperative risk assessments. The IDEA algorithm consists of two main layers, preoperative and intraoperative, (Fig 2) each containing two cores, data transformer and data analytics. Phase I, all available health care data from the electronic health record and other public datasets is fed into the preoperative model. The preoperative model calculates a risk score for postoperative acute kidney injury (AKI) and shares the risk score with the clinical team before the surgery. Phase II, the preoperative data is combined with intraoperative data and fed into the IDEA algorithm which calculates a risk score for postoperative acute kidney injury and shares the risk score with the clinical team after surgery. In the data transformer core, the algorithm transforms data from its native format into new complex variables that are optimized for use in predictive models (S2 Table). Data preprocessing was done using a set of automated rules to remove errors and outliers. Time series variables in the intraoperative layer were truncated to match the corresponding surgery start and stop times for each patient. Extreme values in the physiological time series data, defined as values outside the allowable range using expert opinion and medical knowledge [30], were replaced by the average of their five nearest neighbors. Outlier detection and removal was performed on laboratory results by replacing the top and bottom 1% of data using random uniform values generated from the 95%-99.5% and 0.5%-5% percentiles, respectively. Missing nominal variables were replaced with a distinct ''missing" category, whereas missing continuous variables were replaced by the median value for a given variable [18,31]. Categorical and nominal variables, with more than two levels, were further optimized by calculating conditional probabilities for a particular variable value (such as each surgeon's ID number or each zip code in the dataset) to be associated with the occurrence of the complication using a separate dataset. The probabilities were calculated as the log of the ratio of the prevalence of a particular variable value among cases with one or more complications to cases without complications [28]. Surgical procedure codes were optimized using a forest of trees approach to reduce the 4-digit primary procedure ICD-9-CM codes that correspond to the anatomical location of surgery. Each node represents a group of procedures, with roots representing the most general groups of procedures and leaf nodes representing the specific procedures [28]. This grouping method reduces the number of discrete procedure codes from 318 to 174 and improves the analysis of low frequency procedures (S1 Methods). Statistical features were derived from the physiological time series data, which included minimum, maximum, mean, short-and long-term variability [32], time spent between different value ranges (determined by the number of standard deviations away from the mean value), and the percentage of time spent in each of the previously mentioned value ranges. Short-and long-term variability were computed using the base and residual signals [32]. The base signal represents the smoothed out version of the time series computed using a convolution filter, while the residual signal represents the difference between the original signal and base signal. The long-term variability is the standard deviation of the base signal and the short-term variability is the standard deviation of the residual signal. For repeated laboratory measurements, the percentage of abnormal values (the percentage of values outside the normal ranges from the Logical Observation Identifiers Names and Codes (LOINC) data table [https://loinc.org/downloads/]), value counts (number of laboratory results obtained during the surgery), and variances were derived (S1 Methods).
In the data analytics core, the IDEA algorithm was trained to calculate patient-level preoperative and immediate postoperative risk probabilities for AKI. In the first stage, the algorithm was trained to calculate preoperative risk probability for AKI using preoperative data only. Subsequently, the intraoperative data available at the end of surgery was used as an additional input for the algorithm to recalculate the postoperative risk probability for AKI. The preoperative risk probability was calculated using a generalized additive model (GAM) with logistic link function [18,28,31]. All models were adjusted for non-linearity of covariates using nonlinear risk functions estimated with thin plate regression splines [18,28,33]. The best GAM model was picked using a five-fold cross validation technique on 70% of the randomly selected training data cohort and the preoperative prediction scores were generated as the output.
Intraoperative Data Embedded Analytics (IDEA), employing a random forest classifier, was used to enrich the preoperative risk model with intraoperative data. Two approaches were compared, one where risk probability output from the GAM model was combined with the intraoperative data and used as input for the random forest (stacked model), and the other where all preoperative data was embedded with the intraoperative data as input for the random forest (full model). All intraoperative statistical features, along with the preoperative prediction scores, underwent a univariate analysis and only statistically significant (based on the F-test statistic) features were considered for the random forest model [34]. The feature selection and other hyper parameters in scikit-learn [35] random forest classifier (i.e., number of trees, maximum features for the best split, minimum number of samples required to be at a leaf node) were tuned simultaneously using a grid search technique with five-fold cross validation on the same 70% training data cohort (S1 Fig).

Model validation
All of the models were validated using the 30% validation cohort of 873 patients (S1 Fig). The results were reported using 1000 nonparametric bootstrap replicates generated from the R boot package [36]. Using the prediction results obtained from the 1000 bootstrap cohorts, nonparametric confidence intervals for each of the performance metrics were calculated.

Model performance
Each model's discrimination was assessed using the area under receiver operating characteristic curve (AUC) and model accuracy by determining the fraction of correct classifications for each model. Stratification into high-and low-risk groups was done by calculating the optimal cut-off point based on the maximum Youden Index [37] computed during the training process for each outcome. Using the optimal thresholds for risk probabilities, a classification table was built from which sensitivity, specificity, and positive and negative predictive values were calculated for each model. Absolute risk was calculated as the percentage of cases for which acute kidney injury (AKI) occurred in low-and high-risk groups, respectively. Relative risk was calculated as the ratio of the absolute risk of AKI between high-and low-risk groups. The absolute risk was calculated for high-and low-risk groups for all three models and were compared using the R package "DTComPair" adjusting for multiple comparisons using the Bonferroni method. The Net Reclassification Improvement (NRI) index [38] was used to quantify how well the postoperative model reclassifies AKI patients compared to the preoperative model. Model calibration was tested using the Hosmer-Lemeshow statistic. Bootstrap sampling and nonparametric methods were used to obtain 95% confidence intervals for all performance measures. All analyses were performed using Python 2.7 [39], SciPy 1.0.0 [40], R 3.4 [41], and SAS 9.4 (Cary, NC) software.

Participant baseline characteristics and outcomes
Among 2,911 patients who underwent inpatient surgery requiring a hospital admission of at least 24 hours in a quaternary-care academic center, 1,339 (46%) developed postoperative AKI prior to discharge. Of the patients that developed postoperative AKI, 1,163 (87%) had AKI onset within seven days of surgery. Only 176 (13%) of the AKI patients had late AKI onset (more than seven days after surgery), while 75% of the AKI patients had AKI onset within the first 72 hours after surgery (S3 Table).
The cohort included data from 129 surgeons with an average of 149 procedures per surgeon ( Table 1). The acuity of the patient population was high, as 46% of surgeries were categorized as either non-elective or were associated with emergent or urgent hospital admission. Among the cohort, 62% of patients required ICU admission > 48 hours. The cohort had a median ICU length of stay of four days (25 th -75 th percentiles two-nine days) and a median hospital length of stay of eleven days (25 th -75 th percentiles seven-twenty days). The overall mortality was 4.8% at thirty days and 16% at one year after index admission. A wide range of comorbidities was documented on admission with cancer and diabetes mellitus being most prevalent. One fourth of the patients were from rural areas and 12% of the patients resided in neighborhoods with a household income below the poverty level [42]. The prevalence of examined complications ranged from 3% for wound complication to 62% for intensive care unit admission > 48 hours (Table 1 and S3 Table). Acute kidney injury, admission to ICU for > 48 hours, and mechanical ventilation for > 48 hours were the most common complications among all surgeries. The distribution of outcomes and preoperative clinical characteristics did not differ between training and validation cohorts.
The severity and duration of intraoperative hypotension was directly correlated with the risk of developing postoperative AKI (Fig 4). Patients with prolonged periods of blood pressure within the low range of normal values, yet above the traditional threshold for hypotensive treatment (60-65 mmHg), still carried a higher risk for AKI (Fig 4A). Similarly, persistently elevated heart rate was associated with the risk of AKI (Fig 4D).

Other intraoperative variables and risk of acute kidney injury
The following intraoperative arterial blood gas panel measurements were found to be significantly higher in patients that developed AKI and were correlated with increased risk of developing AKI: partial pressure of carbon dioxide, percentage of carboxyhemoglobin, percentage of methemoglobin, and variance in bicarbonate. Additionally, arterial oxygen content was found to be significantly lower in AKI patients (Table 2 and S3 Table). Among other intraoperative laboratory tests, white blood cell count, red cell distribution width, and lactic acid were significantly higher in patients who developed AKI, while platelet count, red blood cell count, hemoglobin, and hematocrit were significantly lower in AKI patients (Table 2 and S3  Table). The correlation between important features and AKI risk probability was not linear (Fig 5). The total amount of blood products administered during surgery was significantly associated with progressive increases in AKI risk probability (Fig 5G). Patients who developed AKI were more likely to receive diuretics (18% vs 7%) and vasopressors (74% vs 62%) during surgery and were administered less intravenous fluid intraoperatively (Table 2 and S3 Table). The duration of surgery was significantly higher in patients with AKI and was highly correlated with the risk for AKI (Fig 5H). Interestingly, patients with AKI were more likely to have their operation performed between 7 pm and 7 am. Similar trends were observed for the secondary outcomes: AKI with the onset in the first three days of surgery and postoperative AKI occurring at any time during the hospitalization for the index surgery (S3 Table).

Risk score stratification and model performance
The IDEA algorithm calculated the risk for developing AKI (ranging from 0 to 1) at two distinct time points: once preoperatively using only preoperative data and once immediately after surgery after enriching the preoperative data with physiological responses to surgery and intraoperative events. The algorithm automatically determined the optimal threshold for stratifying patients into low-and high-risk groups (Fig 2 and S2 Fig). The predictive performance for all three models in the validation dataset was very good (Fig 6)    three postoperative days and AKI onset before discharge) were similar and demonstrated the same trends as the primary outcome postoperative models, with AUC values ranging between 0.82 and 0.88 (S4 Table). Most significantly, according to the feature importance scores from the trained postoperative random forest models (S5 Table), only three of the top ten important features were preoperative variables (zip code, chronic kidney disease, and attending surgeon) while all other important features were derived from the following intraoperative variables: lactic acid, total blood products administered during surgery, red cell distribution width, and diastolic blood pressure (S5 Table).

Reclassification of risk groups with intraoperative data
The observed absolute risk (the percentage of patients in a risk group that developed AKI within the first seven days after surgery) was distinctly different between the low-and highrisk groups for the preoperative model and both postoperative models ( Table 3). The relative risk (the ratio of the absolute risk between the high-and low-risk groups) was significantly higher for the postoperative stacked and full models, 4.6 (95% CI 3.7-5.7) and 5.1 (95% CI 4.1-6.4) respectively, compared to the preoperative model, 3.4 (95%CI 2.8-4.0).
To assess the incremental improvement of classifying patients into the correct risk groups by the addition of intraoperative data, the net reclassification improvement (NRI) was calculated for both the stacked and full postoperative models. The calculated NRI (net percentage of correctly reclassified cases after addition of the intraoperative data) showed a statistically significant improvement for both the stacked and the full postoperative models with 7% (95% CI (3%, 12%), p < 0.005) and 11% (95% CI (5%, 16%), p < 0.0005) of cases correctly reclassified, respectively (Fig 6). The NRI for patients that developed postoperative AKI within seven days after surgery was approximately 12% (95% CI (8%, 15%), p <0.0001) for both postoperative models, while the NRI for patients that did not develop postoperative AKI was -4% (95% CI (-4%, -7%), p<0.001) and -2% (95% CI (-5%, 1%), p>0.5) for the stacked and full models, respectively. Both postoperative models were able to reclassify approximately 40% of the false negative patients from the preoperative model correctly into the high-risk group (Fig 7). This demonstrates that intraoperative data can be used as a significant predictor to correctly assess Intraoperative data embedded analytics for acute kidney injury  patients' increased risk for postoperative AKI within seven days of surgery (compared to preoperative risk). However, the addition of intraoperative data is unlikely to accurately lower patients' risk for the development of postoperative AKI within seven days of surgery. The secondary outcomes, postoperative AKI onset within three days of surgery and postoperative AKI onset before discharge, also showed statistically significant NRIs comparing the postoperative models to the preoperative model. However, for the outcome predicting development of postoperative AKI within three days of surgery, both postoperative models were more effective  Table). Furthermore, the patients in the low preoperative risk group for postoperative AKI who developed AKI within seven days of surgery that were correctly reclassified into the high-risk group by the postoperative models, had lower intraoperative mean arterial blood pressure values and higher intraoperative blood product volumes compared to those who remained in the low-risk group (Fig 8). Conversely, the patients in the high preoperative risk group for postoperative AKI who did not develop AKI within seven days of surgery that were correctly reclassified into the low-risk group by the postoperative models, had higher intraoperative mean blood pressure values and lower intraoperative blood product volumes compared to those who remained in the high-risk group.

Discussion
Using a large single-center cohort of surgical patients, we developed and validated a dynamic machine-learning algorithm that readjusts the preoperative risk for postoperative AKI using physiological time series data and other data collected during surgery to provide a personalized risk panel for acute kidney injury with both preoperative and immediate postoperative risk assessments. This work expands on our previously validated MySurgeryRisk algorithm which predicts preoperative risk for major postoperative complications, including AKI [18], to leverage temporal enrichment of the preoperative model with the new information related to patients' changes in physiological status during surgery. The advantages of the algorithm include a) prediction entirely based on routinely available preoperative and intraoperative data, b) universal applicability to any surgical context, c) exportability to other EHR systems, and d) the ability to handle any data type in EHR (including time series and sparse data). Most importantly, the dynamic reassessment of the risk for postoperative AKI using temporal enrichment with intraoperative data allows for more precise reclassification of AKI risk based on how patients' clinical trajectory progresses. While preoperative models mainly asses risk based on patients' pre-existing health conditions and general risks associated with the type of planned procedures, the addition of intraoperative time series data reflects acute physiological responses to the stresses of surgery, which provides a better risk assessment for postoperative AKI to a physician who would want to use the model. For example, a patient without significant comorbidities who undergoes a moderate risk surgery would have low-risk for postoperative AKI based on a preoperative model. If that patient was to develop one or more a Absolute risk was calculated as the percentage of cases for which acute kidney injury occurred in the low-and high-risk groups, respectively. b Relative risk was calculated as the ratio of the absolute risk of the occurrence of acute kidney injury between the high-and low-risk groups.
c Patients were classified as low-risk if their prediction score was less than or equal to cutoff and high-risk otherwise.  recognition that certain types of admission medications, or even the timing of the operation itself, are risk factors for AKI confirms some recent observations [44][45][46]. Furthermore, the Association between intraoperative variables (mean arterial pressure and administered blood products) and postoperative risk group reclassification. The first column represents patients that developed postoperative acute kidney injury (AKI), yet were classified as low-risk by the preoperative model. The second column represents patients that did not develop postoperative AKI, but were classified as high-risk by the preoperative model. The boxes represent 95% confidence intervals for the given variables. (A) Patients correctly classified as having high-risk for AKI by the postoperative stacked model tended to have lower mean arterial pressure. (B) Patients correctly classified as having low-risk for AKI by the postoperative stacked model tended to have higher mean arterial pressure. (C) Patients correctly classified as having high-risk for AKI by the postoperative stacked model tended to have greater volumes of administered intraoperative blood products. (D) Patients correctly classified as having low-risk for AKI by the postoperative stacked model tended to have lesser volumes of administered intraoperative blood products.
https://doi.org/10.1371/journal.pone.0214904.g008 dynamic changes in intraoperative blood pressure and heart rate were among the most important features contributing to the risk of developing postoperative AKI in our model. This indicates that the duration and magnitude of intraoperative hypotension as well as concomitant changes in heart rate significantly increase the risk for postoperative AKI. Intraoperative hypotension is a well-recognized risk factor for multiple postoperative complications [47][48][49], with biological plausibility that is well accepted among anesthesiologists and surgeons, yet no consensus exists regarding the optimal blood pressure target to support the perfusion of critical organs during surgery [50,51]. The red cell distribution width, platelet count, and the duration of operation were among the most important intraoperative predictors for postoperative AKI, and while some of them have been previously reported as risk factors for AKI in other patient populations, this is the first report for surgical patients [52][53][54].
Most of the existing risk models for acute kidney injury are focused on the general hospital population [10,11] or are designed for critically ill patients [15]. Many of the previous risk models for surgical AKI were limited to either a specific type of surgery or the use of preoperative risk factors for statistical modeling (mainly logistic regression) with reported AUCs between 0.77 and 0.84 [1,2,55,56]. The risk prediction models that have incorporated intraoperative variables are scarce and mainly focused on patients undergoing cardiac surgery with reported AUCs ranging between 0.72 and 0.81 [13,14,57]. The fact that these models did not fully utilize the available time-varying physiological data during the surgery, or machine learning approach, may have contributed to their lower performance compared to our model. This study has some limitations. Firstly, only the first surgery was used for patients that underwent multiple surgeries for building our proposed predictive models. Secondly, the algorithm was only provided with the data and outcomes in the training dataset, without explicit definitions of features. Thirdly, because the algorithm ''learned" the features that were most predictive for the risk of developing postoperative AKI implicitly, it is possible that the algorithm is using features previously unknown to or ignored by physicians. Fourthly, the expansion of input features to include operative notes may increase the accuracy, but will require more elaborate computational approaches. Lastly, the algorithm has been trained to capture practice patterns for individual providers in the referral population of a large academic medical center in North Central Florida. Further training and validation of the algorithm is necessary in a dataset with different population characteristics and practice patterns.

Conclusions
In a large single-center cohort of surgical patients, our proposed Intraoperative Data Embedded Analytics (IDEA) algorithm employed a machine learning approach based on a random forest classifier to improve patients' postoperative acute kidney injury (AKI) risk prediction with high sensitivity and specificity by utilizing intraoperative data. The IDEA algorithm was able to correctly reclassify approximately 40% of patients who were considered low-risk for postoperative AKI by preoperative model to high-risk. This illustrates the importance of intraoperative data in AKI risk stratification. Given the association between AKI and increased morbidity, mortality, and cost, it is important for clinicians to have dynamic AKI risk prediction algorithms capable of adjusting AKI risk as new information becomes available. Further research can address other post-surgical complications as well as validation of the proposed algorithm on external datasets.
Supporting information S1 Methods. (DOCX) S1  Fig. Model training flow diagram. The cohort of size 2,911 with 231 features was randomly split into training (70%) and testing (30%) cohorts. A random forest classifier was used to train the AKI prediction model (we used 5-fold cross validation for hyperparameter tuning and feature selection) for all three outcomes separately and performance was tested using the testing (validation) cohort. (TIF) The first row represents the reclassification of (A) patients that developed postoperative acute kidney injury (AKI) within three days of surgery and (B) patients that did not develop postoperative AKI within three days of surgery for the three day secondary outcome. The second row represents the reclassification of (C) patients that developed postoperative AKI before discharge and (D) patients that did not develop postoperative AKI before discharge for the overall secondary outcome. (TIF)