Comparison of C-statistics among models.

In this study, the models were denoted by their algorithm, dataset type (baseline or time-series dataset), and number of variables (all or v+number). For example, a Random Forest (RF) model based on the baseline dataset including x variables was denoted RF_base_vx. Fig 1A shows the C-statistics of the models used for predicting the primary outcome (ESKD or death) over 1 year. The C-statistics of RF_time_all and RF_time_v8 were higher than 0.85 (p<0.05). The C-statistics for the prediction of ESKD over 1 year showed high accuracy for most of the models (Fig 1B). In particular, the C-statistics of 5 models were more than 0.9 for the prediction over 1 year (p<0.05). Moreover, these models exhibited high C-statistics for the prediction of death over 1 year (Fig 1C). Similarly, the models showed high prediction accuracies for the primary outcome over 2 and 3 years (S1 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Model performances for prediction in outcomes over 1 year at model selection stage.

Figures show the C-statistics for the prediction of outcomes with 95% CIs of the models. A: Primary outcome. B: ESKD. C: Death. Abbreviations: ESKD, end-stage kidney disease; eGFR, estimated glomerular filtration rate; UPCR, urinary protein-to-creatinine ratio; KFRE, kidney failure risk equation; RF, Random Forest; GB, Gradient Boosting Decision Tree; XG, eXtreme Gradient Boosting; LRM, logistic regression model; CI, confidence interval.

https://doi.org/10.1371/journal.pdig.0000188.g001

Comparison of model performance in subclasses.

The C-statistics of the primary outcome were also compared among models for subclasses (S2 Fig). In each subclass, the models showed high C-statistics. S1 Table summarizes the applicability of the models to patients with various conditions. Most of the models with all the variables provided high C-statistics. In particular, RF_base_all, RF_time_all, and RF_time_v8 showed high performances. Considering these results and the calculation speed of the server, RF_time_all and RF_time_v8 were selected as representative models for the development of a Web-based risk-prediction system.

The relationship between the risk of the primary outcome and its predicted probabilities was analyzed (Fig 2). The hazard ratios (HRs) of RF_time_all and RF_time_v8 were associated with high probabilities (p<0.0001). For the models to be informative and show whether a patient was at a high risk, the cutoff levels of the predicted probability were 8.9% for RF_time_all and 8.2% for RF_time_v8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Relationships between risk of primary outcome and its predicted probabilities at model selection stage.

The figure shows the relationships between the risk of the primary outcome and its predicted probabilities. Histograms of the probabilities and the log hazards of the primary outcome are shown in the upper and lower panels, respectively. A: RF_time_all. B: RF_time_v8. Abbreviation: RF, Random Forest.

https://doi.org/10.1371/journal.pdig.0000188.g002

Risk of outcome events and high-risk patients.

Among the patients, 5.5% were classified into the high-risk groups on the basis of the cutoff levels of RF_time_all and RF_time_v8, and they showed different characteristics from the low-risk group (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline characteristics of risk groups classified on the basis of the results of machine-learning models.

https://doi.org/10.1371/journal.pdig.0000188.t001

Survival probabilities of the high-risk groups classified on the basis of RF_time_all and RF_time_v8 were lower than those of the low-risk groups (Table 2). The HRs of the primary outcome of the high-risk groups classified on the basis of RF_time_all and RF_time_v8 were 104.9 (95% CI 70.81, 155.3) and 90.9 (95% CI 62.29, 132.7), respectively. Likewise, the HRs of ESKD were 103.3 (95% CI 70.0, 152.4) and 89.6 (95% CI 61.6, 130.3), and the HRs of death were 14.85 (95% CI 9.75, 22.62) and 14.36 (95% CI 9.43, 21.86) for RF_time_all and RF_time_v8, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Outcome events of risk groups classified on the basis of machine-learning models.

https://doi.org/10.1371/journal.pdig.0000188.t002

Study population.

The EMR data as a dataset were extracted from the server in Kawasaki Medical School Hospital. The dataset was preprocessed for analysis because it was composed of all data of all patients in a column. The dataset was divided by variables, and the separated data were merged for each patient and medical examination day in a table for analysis. The dataset contained time-series data of patients (time-series dataset n = 83,703) (S3 Fig).

The exclusion criteria were as follows: patients younger than 20 years, patients on any type of dialysis, and patients who had malignancies. The patients with malignancies often show a shorter survival than others and tend to die before they reach ESKD. Their disease courses are different from the typical course of CKD. If their data, which are not related to CKD, had been included among the data for machine learning, they may affect the performance of machine-learning models and variable selection. Therefore, patients with malignancies were excluded in this study. Moreover, patients who had missing baseline data were excluded from the dataset. Thus, the time-series data obtained on every medical examination day from 3,714 patients were used in the development of machine-learning models (time-series dataset n = 66,981). The median (IQR) number of repeatedly measured times per patient was 16 (7, 29). The first-visit data of a patient were treated as baseline data (baseline dataset n = 3,714).

In this study, two types of machine-learning model were developed using baseline data or time-series data. To develop and select the models, the data were further assigned into the development dataset (including 80% of the patients, baseline dataset n = 2,967, time-series dataset n = 31,763) and a selection dataset (baseline dataset n = 747) on the basis of patient ID number.

The development dataset contained baseline data and time-series data, each of which was further divided into two datasets on the basis of patient ID number. About 70% of the development dataset was used to train the models (baseline dataset n = 2,076, time-series dataset n = 22,234), and the rest was used to test the models (baseline dataset n = 891, time-series dataset n = 9,529). As a result, four datasets were prepared: the baseline dataset to train the models (baseline dataset n = 2,076), the time-series dataset to train the models (time-series dataset n = 22,234), the baseline dataset to test the models (baseline dataset n = 891), and the time-series dataset to test the models (time-series dataset n = 9,529). The two training datasets included the same patients as well as the two testing datasets.

On the other hand, the selection dataset contained only baseline data (baseline dataset n = 747). Baseline characteristics and outcome events are shown in S5 Table and S6 Table, respectively.

Variables.

The variables were CKD-related factors and medications in the EMR data [4,25–28]. They were selected for global use on the basis of their being common measurements used at outpatient clinics and ease of data input by users at implementation. The variables with more than 20% of their time-series data missing were not included in the analysis, because these variables were considered not to be usually measured at clinical settings.

The following variables were used in the analysis as follows (S7 Table): age; gender; DM; hypertension; history of CVD; eGFR; serum albumin, sodium, potassium, calcium, phosphorus, low-density lipoprotein, and uric acid levels; white blood cell count; hemoglobin level; urinary protein-to-creatinine ratio (UPCR); and use of renin-angiotensin-aldosterone system inhibitors, phosphorus absorbents, vitamin D, statins, uric-acid-lowering medicines, and erythropoietin-stimulating agents. The primary outcome was ESKD or death. If no outcomes were observed within the follow-up period, the observation data were treated as censored data. The onset of ESKD was defined as the initiation of renal replacement therapy. None of the patients in this study had received a kidney transplantation.

eGFR was calculated using the equation for the Japanese population on the basis of the serum creatinine level measured using the enzymatic method [29]. The baseline data were defined as the first measurements within 30 days of the initial eGFR. KFRE was calculated using eGFR, age, gender, and natural logarithm (urine albumin-to-creatinine ratio, UACR) [14]. Negative values of KFRE were treated as zero. Because the Japanese national health insurance covers UACR measurement only for DM patients, UACR was estimated from the UPCR using Weaver et al.’s estimation formulae [30]. The last observation data carried forward were input in place of all missing time-series data.

Machine-learning models.

The scikit-learn and numpy packages of Python were mainly used in model development. RF, Gradient Boosting Decision Tree (GB), and eXtreme Gradient Boosting (XG) models were developed using RandomForestClassifier, GradientBoostingClassifier, and XGBClassifier in scikit-learn, respectively. Their parameters were determined using a grid search algorithm, GridSearchCV, to search for the hyperparameters for the models, which were the number of estimators (trees), the maximum percentage of features used in each tree, maximum depth, and class weight. The parameters of GridSearchCV were the maximum number of trees of 1000, depth of 10, leaf of 10, and sample split of 10. The parameters of the selected models were as follows. RF_time_all: number of trees = 1000, maximum depth = 10, minimum samples leaf = 5, minimum samples split = 5; RF_time_v8: number of trees = 1000, max depth = 10, minimum samples leaf = 5, minimum samples split = 5. In the development of the models with selected variables, the variables were ranked by their permutation importance in each model using numpy and then selected. pickle was used to save and load these trained models.

Model development.

We developed risk-prediction models to classify the primary outcome over 1 year using the RF, GB, and XG models, which were trained to predict the primary outcome. Each type of machine-learning model included all 22 variables or selected variables. Variables were selected by their permutation importance in each model (S4 Fig). The numbers of variables selected were 7 for RF_base, 8 for RF_time, 13 and 6 for GB_base, 7 for GB_time, 17 and 7 for XG_base, and 10 and 6 for XG_time.

A multivariate LR model was developed using the hierarchical backward elimination procedure. The LR model with 11 selected variables was developed (S8 Table).

Statistical analyses.

Baseline characteristics and outcomes were compared between the development and selection datasets using the chi-square test, t-test, and Mann-Whitney U test, as appropriate. After the proportional hazards assumption was confirmed using a double logarithmic plot, multivariate Cox proportional hazards models were used to compare the risks of outcomes between groups. The results are presented here as HRs with 95% CIs. In the analysis of the competing risks of ESKD and death, we used Fine and Gray competing risk regression models. Statistical significance was defined as a two-sided p<0.05. All statistical analyses were carried out using SAS version 9.4 (SAS Institute, Cary, NC, USA), Python 3.8.2 (Python Software Foundation, DE, USA), and R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).

Risks of outcomes.

A well-developed model is one for which the predicted probabilities strongly correlate with the risks of the outcomes. To evaluate the relationship between the probability predicted using the models and the risk of the primary outcome, univariate Cox proportional hazards models including the spline terms of the predicted probability were developed using the model selection dataset.

Moreover, for users to easily understand the severity of risks in a patient, cutoff levels of the predicted probabilities were determined for the categorization of patients. A cutoff level of each model was determined on the basis of the Youden index on the receiver operating characteristic curve of the prediction of the primary outcome over 1 year using the development dataset [31]. Then, the patients were categorized into high- and low-risk groups.

Study population.

The machine-learning models were validated using a dataset of a cohort study different from the development dataset. The CKD cohort study was conducted for three years. CKD patients who regularly visited the outpatient clinic of Kawasaki Medical School were registered (n = 67,957) (S5 Fig). The exclusion criteria were the same as those for the dataset used for the model development. Moreover, the patients who had no eGFR data were also excluded. A total of 26,906 patients were included in the analysis and followed up during the study period. The data were extracted from the cohort dataset derived from an EMR server in Kawasaki Medical School Hospital. The data at the first visit of patients during the study period were treated as baseline data. The dataset contained baseline data only.

Variables.

The names and definitions of the variables were the same as those for the dataset used for the model development (S7 Table). Multiple imputation was conducted to account for missing data in analyses. The primary outcome was defined as ESKD or death for the follow-up period. The data of patients who did not have either of these outcomes were treated as censored data. The probability of the primary outcome was estimated using the machine-learning models and used as a risk index.

Statistical analysis.

The C-statistics and 95% CIs for the accuracies of the prediction of the outcomes, such as the primary outcome, ESKD, or death, of the estimated probability were evaluated using the 1000-times bootstrap method. The C-statistics of the models for the outcomes over 1 year were statistically compared with that of KFRE using the bootstrap method.

For the probabilities to be used in clinical settings, not only incident risk but also survival analysis considering a long follow-up period was required. Thus, univariate Cox proportional hazards models including the spline terms of the probability were used to evaluate the relationship between the probability and risk of the primary outcome.

Then, the patients were divided into two groups on the basis of the cutoff levels determined at Step (2). The hazard of the primary outcome in the high-risk group was compared with that in the low-risk group. In the analysis of the competing risks of ESKD and death, Fine and Gray competing risk regression models were examined.