Recalibration and Validation of the Charlson Comorbidity Index in Korean Incident Hemodialysis Patients

Background Weights assigned to comorbidities to predict mortality may vary based on the type of index disease and advances in the management of comorbidities. We aimed to develop a modified Charlson comorbidity index (CCI) in incident hemodialysis patients (mCCI-IHD), thereby improving risk stratification for mortality. Methods Data on 24,738 Koreans who received their first hemodialysis treatment between 2005 and 2008 were obtained from the Korean Health Insurance dataset. The mCCI-IHD score were calculated by summing up the weights which were assigned to individual comorbidities according to their relative prognostic significance determined by multivariate Cox proportional hazards model. The modified index was validated in an independent nationwide prospective cohort (n=1,100). Results The Cox proportional hazards model revealed that all comorbidities in the CCI except ulcers significantly predicted mortality. Thus, the mCCI-IHD included 14 comorbidities with re-assigned severity weights. In the validation cohort, both the CCI and the mCCI-IHD were correlated with mortality. However, the mCCI-IHD showed modest but significant increases in c statistics compared with the CCI at 6 months and 1 year. The analyses using continuous net reclassification improvement revealed that the mCCI-IHD improved net mortality risk reclassification by 24.6% (95% CI, 2.5-46.7; P=0.03), 26.2% (95% CI, 1.0-51.4; P=0.04) and 42.8% (95% CI, 4.9-80.8; P=0.03) with respect to the CCI at 6 months and 1 and 2 years, respectively. Conclusions The mCCI-IHD facilitates better risk stratification for mortality in incident hemodialysis patients compared with the CCI, suggesting that it may be a preferred index for use in clinical practice and the statistical analysis of epidemiological studies.


Introduction
Although the prognosis of dialysis patients has improved significantly in recent years, mortality rates for these patients remain several times higher than those for individuals in the general population [1]. Comorbidities in dialysis patients, which are highly prevalent and characterized by simultaneous coexistence, are often associated with substantially high mortality rates and socio-economic burden [1][2][3][4][5][6]. Thus, the prognostic impacts of comorbidities may influence diverse decisions during clinical practice and should be adjusted as strong confounding factors in statistical analyses. Consequently, comorbidities should be carefully assessed and summarized by an index that is intuitive to physicians for clinical and research uses. More importantly, such an index should reflect current advances regarding the management and outcomes of comorbidities and should be updated accordingly.
Among the current indices, the Charlson comorbidity index (CCI) which was developed in 1987, is the most popular and simple. It has been successfully validated in diverse chronic disease populations, including dialysis patients [7][8][9][10]. Nevertheless, there are several reasons to recalibrate and subsequently validate the index in dialysis patients to obtain better predictions regarding their mortality. Because the assigned weights of comorbidities in the CCI were generated from a general medical inpatient population, some comorbid conditions might have different significance in predicting mortality among dialysis patients [11,12]. In addition, the management of dialysis patients and their comorbidities has advanced considerably, and the contributions of comorbidities to mortality rates might have changed as well [13]. Although some previous studies proposed modification of the CCI in dialysis patients, these modified indices were developed based on relatively small numbers of dialysis patients and were not externally validated through appropriate performance analyses [14,15].
In the present study, we aimed to modify the CCI by recalibrating the weights of comorbidities in the CCI using a Korean national population-based registry database, to validate the index in a separate nationwide prospective end-stage renal disease (ESRD) cohort, and to compare its performance with the CCI.

Data source
Development cohort. Deidentified data on all Koreans aged 18 years who received hemodialysis (HD) as their first modality of renal replacement therapy between January 2005 and December 2008 were obtained from the Korean Health Insurance database (n = 28,032). Data on all Korean HD patients could be reviewed because Koreans are obligated to register with the national insurance system, which consists of health insurance and medical aid. Subjects who died (n = 2,751), changed their modality of renal replacement (n = 229), or underwent kidney transplantation (n = 314) within 3 months after starting HD were excluded. Therefore, a total of 24,738 patients were included in the analysis to develop a modified CCI for incident HD patients (mCCI-IHD).
Validation cohort. All patients aged 18 years who started HD as their first modality of renal replacement therapy between August 2008 and December 2012 in 31 centers of the Clinical Research Center (CRC) for ESRD and remained on HD for more than 3 months were initially screened for the validation cohort (n = 1,165). The CRC for ESRD is a nationwide multicenter prospective cohort of patients with ESRD in Korea (NCT00931970). Patients who died within 3 months of the commencement of hemodialysis (n = 65) were excluded from the analyses. A total of 1,100 incident HD patients were included to externally validate the mCCI-IHD.

Data collection
Comorbidities were defined based on the International Classification of Diseases, 10 th Revision (ICD-10) (S1 Table) [16]. For the development study, information on comorbidities that constitute the CCI, including myocardial infarction (MI), congestive heart failure (CHF), peripheral vascular disease (PVD), cerebrovascular disease, dementia, chronic pulmonary disease, connective tissue disease, ulcer disease, mild liver disease, diabetes, hemiplegia, diabetes with end-organ damage, any tumor (including leukemia and lymphoma), moderate to severe liver disease and metastatic solid tumors were derived by extracting ICD-10 codes leading up to the initiation of dialysis therapy [17]. The comorbidity data of the validation cohort were ascertained by clinicians of CRC for ESRD through history taking and review of electronic health record. Acquired immunodeficiency syndrome was not included because no patient was so diagnosed in either cohort. Mortality data were obtained from the Statistics Korea and National Health Insurance Claims databases. In the validation cohort, demographic characteristics included body mass index (BMI), health security system, primary cause of ESRD, and timing of referral to nephrology; early referral was defined by HD initiation occurring more than 12 months after the first encounter with a nephrologist. The laboratory data included pre-dialysis fasting blood samples taken 1 month after the initial HD and included the following: hemoglobin, albumin, calcium, and phosphorus.

Statistical analyses
All statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC) or R version 3.0.3 (R Development Core Team, Austria). The data were summarized according to means±SDs or medians±IQRs for continuous variables and proportions for categorical variables. The multivariate Cox proportional hazards model was used to develop new comorbidity weights. We computed adjusted HRs and β coefficients while accounting for age, sex, and all 15 comorbidities simultaneously. We excluded an unnecessary item of severe renal disease given its presence in all of the study subjects. Prognostic weights were derived by dividing each β coefficient by the lowest one with statistical significance and rounding to the nearest integer [18]. Following this procedure, a comorbidity score was calculated for each subject by summing up the weights. The prognostic powers of the CCI and mCCI-IHD were compared by estimating Kaplan-Meier curves and multivariate Cox regression models. To compare the capacity for discrimination between the indices, a c statistic was calculated using a receiver-operating characteristic curve [19]. Additionally, the continuous net reclassification improvement (cNRI) was calculated to determine the extent to which the mCCI-IHD improved mortality predictions

Development of the new comorbidity index (mCCI-IHD) for the prediction of mortality
Baseline characteristics of the development cohort are listed in Table 1. The mean age at initiation of HD therapy was 57.9 years and 59.5% of patients were men. The median follow-up duration was 47.7 months. A total of 76.3% of the subjects had one or more comorbidities. Among the 15 comorbidities, diabetes (with and without end-organ damage) was most prevalent (49.6%), followed by chronic pulmonary disease (16.3%), CHF (14.1%), and cerebrovascular disease (13.0%). The survival rates of the subjects were 92.4%, 87.5%, 78.9%, and 72.3% at 6 months and 1, 2, and 3 years of follow-up, respectively. Table 2 lists the adjusted HRs, β coefficients and weights for each comorbidity. All comorbidities except ulcers significantly predicted mortality. The relative weights of comorbidities were calculated by their ratios with the lowest β coefficient of chronic pulmonary disease. As expected, metastatic solid tumors were the strongest predictor of mortality among comorbidities; accordingly, the weight assigned to this condition was the highest. Moderate to severe liver disease, any tumor (including leukemia and lymphoma), diabetes with end-organ damage, dementia, and cerebrovascular disease followed consecutively.
The summed scores of the original and recalibrated weights were applied to each patient in the development cohort. In the Cox proportional hazards model adjusted for age and sex, the HRs of the CCI and mCCI-IHD scores for mortality were 1.42 (95% CI, 1.39-1.45; P<0.0001) and 1.72 (95% CI, 1.66-1.78; P<0.0001), respectively. Based on the CCI and mCCI-IHD scores, the subjects were further collapsed into 4 risk groups according to the following criteria: no comorbidity, 50 th percentile, 50 th -90 th percentile, and >90 th percentile. To determine the cutoff values of the comorbidity scores for each risk group, we examined the distributions of the scores using a histogram (Fig 1). The cutoff values of the CCI scores corresponding to the 50 th and 90 th percentiles were 4 and 6, whereas those of the mCCI-IHD scores corresponding to the same percentiles were 4 and 9. Fig 2 shows the Kaplan-Meier curves for the CCI and mCCI-IHD differentiated by the 4 risk groups in the development cohort. The appearance of survival curves across increasing comorbidity scores was similar for both indices, with anticipated poorer survival for higher scores (P<0.001). Application and validation of the mCCI-IHD Table 1 lists the characteristic of the validation cohort (n = 1,100). The majority of subjects were men, with a mean age of 60.7 years. A total of 143 mortalities (13.0%) were observed during the median follow-up of 22.0 months. Along with the development cohort, diabetes was the most common comorbidity (57.3%), followed by CHF (13.6%), cerebrovascular disease (13.1%), and PVD (10.4%). The mCCI-IHD and CCI scores were then calculated for patients in the validation cohort. Fig 3 shows the survival curves obtained using the Kaplan-Meier method and multivariate Cox regression analysis (adjusted for age, sex, health security system, timing of referral to nephrologists, primary cause of ESRD, BMI, hemoglobin, albumin, calcium, and phosphorus) in the validation cohort, which were differentiated by the 4 risk groups of the CCI and mCCI-IHD. The risk groups based on each index provided some discrimination for survival, indicating that increasing comorbidity index scores were associated with lower cumulative survival. However, multivariate-adjusted survival curves for the mCCI-IHD risk groups showed a highly significant linear gradient and were quite well separated, whereas the model using the CCI seemed less able to distinguish among the three lowest-risk groups (Fig 3). As detailed in Table 3, increased c statistics and cNRI for the mCCI-IHD supported better performance characteristics of the mCCI-IHD compared to the CCI. The mCCI-IHD exhibited modest but significant increases in c statistics compared with the CCI at 6 months and 1 year but not at 2 years in multivariate analyses adjusted for age, sex, health security system, timing of referral to nephrologist, primary cause of ESRD, BMI, hemoglobin, albumin, calcium, and phosphorus. The analyses using cNRI revealed that the mCCI-IHD improved net mortality risk reclassification by 24.6% (95% CI, 2.5-46.7; P = 0.03), 26.2% (95% CI, 1.0-51.4; P = 0.04) and 42.8% (95% CI, 4.9-80.8; P = 0.03) relative to the CCI at 6 months and 1 and 2 years, respectively, after adjustment for the above-mentioned confounders. This finding indicates that the mCCI-IHD may provide better risk stratification for mortality in incident HD patients than the CCI. Additionally, the mCCI-IHD also improved the predictability for mortality compared with the age-adjusted CCI (cNRI, 25.6%; P = 0.04 and cNRI, 45.7%; P = 0.02 at 1 and 2 years, respectively) and the mCCI (cNRI, 48.0%; P = 0.02 at 2 years) after adjusting the confounding factors.

Discussion
In the present study, we modified the CCI using population-based registry data that included nearly all Korean patients who began HD between 2005 and 2008 to develop a comorbidity index that provides better risk stratification in incident HD patients. In addition, we validated the usefulness of the modified index in an independent prospective cohort by comparing its performance with the original index. To our knowledge, this is the first study to recalibrate the CCI using a large sample of incident HD patients and to validate the index using an independent nationwide prospective observational cohort of ESRD patients.
Despite highly prevalent comorbidities and their influence on mortality rates in ESRD patients, it has not yet been determined which comorbidities significantly affect mortality and what weight each comorbidity carries when predicting mortality. Therefore, many previous studies have examined the influence of comorbidity on survival in patients with ESRD [23][24][25], and several comorbidity instruments such as the CCI [14,26], the index of co-existent disease . CCI, no comorbidity (n = 5,866, score 2); 50 th percentile (n = 10,501, scores 3-4); 50 th -90 th percentile (n = 6,663, scores 5-6); and >90 th percentile (n = 1,708, score 7). mCCI-IHD, no comorbidity (n = 6,651, score 0); 50 th percentile (n = 9,740, scores 1-4); 50 th -90 th percentile (n = 7,072, scores 5-9); and >90 th percentile (n = 1,275, score 10). CCI, Charlson comorbidity index; mCCI-IHD, modified Charlson comorbidity index for incident hemodialysis patients. (ICED) [27], the Davies index [14], and the Khan index [4,14] have been validated in dialysis patients. Among the indices, the ICED has been shown to be a valid method of assessing comorbidity in dialysis patients and appears superior to the others based on its discriminatory ability [23,28,29]. However, this index consists of more than 100 variables, which makes it largely inapplicable for clinical practice or statistical analysis. The Davies index, which consists of 7 comorbidities, is a simple and clinically applicable instrument [30]. However, all the  D). CCI, score 2 (n = 236, 21.5%); scores 3-4 (n = 537, 48.8%); scores 5-6 (n = 234, 21.3%); and score 7 (n = 93, 8.5%). mCCI-IHD, score 0 (n = 247, 22.5%); scores 1-4 (n = 480, 43.6%); scores 5-9 (n = 290, 26.4%); and score 10 (n = 83, 7.5%). a Adjusted for age, sex, health security system, timing of referral to nephrologist, primary cause of end-stage renal disease, body mass index, hemoglobin, albumin, calcium, and phosphorus. CCI, Charlson comorbidity index; mCCI-IHD, modified Charlson comorbidity index for incident hemodialysis patients. comorbidities in this index are assigned equal weights based on ill-defined criteria, which means there might be a lack of consideration for the differential impacts of the comorbidities on mortality. The Khan index also has limitations due to its reliance on weak evidence for selecting comorbidities as predictors of mortality and its use of equal weights for each comorbidity [4]. In contrast, the CCI is the most widely used and validated because of its convenience and relatively well-defined weights for comorbidities. Nevertheless, the CCI remains highly disputed as an index to be applied directly to current ESRD populations because the weights for certain comorbidities in the CCI may be overestimated when considering the contemporary management of comorbidities. Thus, the applicability of the CCI to current clinical practice or statistical analysis may be tenuous. Indeed, the adjusted HRs of comorbidities in the present study, especially in the case of cardiovascular comorbidities, which are the main causes of death in this population, were lower than those in the original study of Charlson et al. [26]  . Similar patterns were found when the results presented here were compared with HRs in a study from the early 2000s, which modified the CCI based on a relatively small number of ESRD patients (n = 237) [15]. Conversely, a . Given that cardiovascular comorbidities are the most common cause of death in this population [1,31], this finding suggests that the mCCI-IHD properly reflects the differential significance of cardiovascular comorbidities in this population, thereby providing superior performance in mortality risk stratification.
The results of performance analyses demonstrated that the mCCI-IHD exhibited significant but modest improvements in c statistics at 6 months and 1 year compared with the CCI, exhibiting increases of only 0.03-0.04. In this regard, it has become increasingly recognized that increases in c statistics of a test model with a large effect size may be small when compared to a strong baseline model such as the CCI [32,33]. Thus, even though it is a popular metric through which to capture discrimination, c statistics may not be useful for comparing the performance of two risk prediction models [34][35][36]. In contrast, cNRI has been proposed to assess the degree of improvement in discrimination of risk prediction between the models and is not strongly affected by the effect size of the baseline model [32,33]. Furthermore, this metric can capture the degree of improvement in discrimination after accounting for correlation with other variables in the risk prediction model [32]. Therefore, we additionally calculated the cNRI to quantify the magnitude of correct reclassification, and the mCCI-IHD showed significant positive values compared with the CCI, age-adjusted CCI, and mCCI after adjustment for socio-economic, clinical and laboratory parameters related to the survival of HD patients [1,[37][38][39][40]. These findings indicate that the mCCI-IHD may provide better risk stratification for mortality in incident HD patients compared with the original and previously modified CCI.
Our study had limitations. First, our results from the development cohort were derived from the administrative database of the national insurance system. Although the administrative data have advantages, such as conservation of time and resources and consistency in diagnosis due to the uniform use of the ICD-10, there may be inappropriate diagnoses due to the drawbacks of the coding system and the physicians' preference toward diagnostic codes with higher reimbursements [41]. In addition, this dataset did not contain parameters such as biochemical measures and socioeconomic status. Therefore, we could not adjust these variables in the analysis for recalibrating the weights of comorbidities. Second, because there was no patient with acquired immunodeficiency syndrome in either cohort, this condition was not taken into consideration. Third, we excluded those subjects who died within 3 months after starting HD to rule out patients with critical illness or terminal conditions with renal failure. However, this could bias the assigned severity weights of the comorbidities because of a higher mortality in early dialysis period. Fourth, the mCCI-IHD was derived from a single ethnic group of Korean HD patients. Therefore, generalizability of the index can be limited. Nonetheless, a major strength of the present study is that patient mortality was determined using a large nationwide, population-based data set. Furthermore, the modified index was validated using an independent database. However, additional validation using disease-specific cohorts and other national data may be necessary.
In conclusion, we propose a new scoring system that provides better risk stratification for mortality in incident HD patients compared with the CCI after adjusting for socio-economic, clinical and laboratory parameters. Therefore, it may be a preferred index for use in clinical practice and the statistical analysis of epidemiological studies.