Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study

Introduction Having developed a clinical prediction rule (CPR) for bacteremia among hemodialysis (HD) outpatients (BAC-HD score), we performed external validation. Materials & methods Data were collected on maintenance HD patients at two Japanese tertiary-care hospitals from January 2013 to December 2015. We enrolled 429 consecutive patients (aged ≥ 18 y) on maintenance HD who had had two sets of blood cultures drawn on admission to assess for bacteremia. We validated the predictive ability of the CPR using two validation cohorts. Index tests were the BAC-HD score and a CPR developed by Shapiro et al. The outcome was bacteremia, based on the results of the admission blood cultures. For added value, we also measured changes in the area under the receiver operating characteristic curve (AUC) using logistic regression and Net Reclassification Improvement (NRI), in which each CPR was added to the basic model. Results In Validation cohort 1 (360 subjects), compared to a Model 1 (Basic Model) AUC of 0.69 (95% confidence interval [95% CI]: 0.59–0.80), the AUC of Model 2 (Basic model + BAC-HD score) and Model 3 (Basic model + Shapiro’s score) increased to 0.8 (95% CI: 0.71–0.88) and 0.73 (95% CI: 0.63–0.83), respectively. In validation cohort 2 (96 subjects), compared to a Model 1 AUC of 0.81 (95% CI: 0.68–0.94), the AUCs of Model 2 and Model 3 increased to 0.83 (95% CI: 0.72–0.95) and 0.85 (95% CI: 0.76–0.94), respectively. NRIs on addition of the BAC-HD score and Shapiro’s score were 0.3 and 0.06 in Validation cohort 1, and 0.27 and 0.13, respectively, in Validation cohort 2. Conclusion Either the BAC-HD score or Shapiro’s score may improve the ability to diagnose bacteremia in HD patients. Reclassification was better with the BAC-HD score.


Introduction
Bacteremia is a disease with a high mortality rate [1][2][3][4]. Early diagnosis and treatment are keys to improving prognosis. However, due to the variety of clinical presentations of bacteremia, it is not always first on the list of possible diagnoses. In this context, several clinical predictive models (CPRs) of bacteremia in the general population have been developed [5][6][7][8]. Among these models, those developed by Shapiro et al. (Shapiro's model) [9] have been widely validated and internationally recognized [10][11][12].
Patients on hemodialysis (HD) are known to have a higher morbidity and mortality rate from bacteremia compared to the general population. Previous cohort studies have shown that the incidence of bacteremia in patients on maintenance HD is 10. 40-18.98 per 100 personyears [13][14][15][16], which is much higher than the incidence in the general population of 0.22 per 100 person-years [1]. The annual mortality due to sepsis, a severe complication of bacteremia, in HD patients is 50-100 times higher than that of the general population [17,18].
The most frequent cause of bacteremia in the general population is urinary tract infections caused by Escherichia coli [19][20][21][22], whereas the most frequent cause in HD is Staphylococcus aureus [23][24][25]. Since many blood test findings in HD patients are often affected by dialysis, the accuracy of items included in existing CPRs developed in the general population, such as blood counts and serum creatinine levels, may be greatly affected. [12]. In order to address the peculiarities of bacteremia among patients with HD, we developed a CPR specific to bacteremia in HD patients, the BAC-HD score [26].
Since the external validities of Shapiro's model [12] and the BAC-HD score in patients on maintenance HD have not been verified, we assessed the diagnostic accuracy of these two prediction models for bacteremia.

Materials and methods
The present study was approved by the ethics committees of Iizuka Hospital (17167-436), Okinawa Prefectural Chubu Hospital (H28-51) and Saku General Hospital (R201701-01). The study was conducted in accordance with the ethical standards of the Declaration of Helsinki. Since all patient information analyzed in this study was retrospective, participants' written informed consent was not required by the ethics committee. All data were fully anonymized before authors accessed them. We accessed the medical records to obtain data at Okinawa Prefectural Chubu Hospital from February 9th to February 13th, 2017, and at Saku General Hospital from February 24th to 26th, 2017. The study results are reported in accordance with the Standards for Reporting Diagnostic Accuracy (STARD) statement [27].

Study design and participants
We conducted a cross-sectional study of maintenance HD patients at two tertiary-care teaching hospitals.
Data were collected from medical records from January 2013 to December 2015 in each facility. We enrolled consecutive participants on maintenance HD who were aged � 18 y with two sets of blood cultures drawn at admission because of suspicion of bacteremia. Exclusion criteria were participants who met any of the following items: 1) inpatients transferred from another hospital, 2) patients with a vintage of dialysis < 2 months, 3) patients also receiving peritoneal dialysis, and 4) patients receiving HD less than once a week.

Index tests
Two clinical prediction rules (CPR) for bacteremia were adopted as index tests.

Reference standard
The reference standard was bacteremia, as per the results of the admission blood cultures. Contamination was defined as: one of the two sets of culture bottles was positive, or cases where certain species of bacteria known to be contaminants, such as diphtheroids, Bacillus spp., Propionibacterium spp., Micrococci, Corynebacterium spp., and coagulase-negative Staphylococci were detected. Finally, an external consensus panel of infectious disease physicians with > 10 y clinical experience and Japanese Board of Infectious Disease certification who were blinded to the present study design determined whether a culture was contaminated or not based on the above definitions and their clinical expertise.

Statistical analysis
Validation cohorts. Since there are no standard criteria for obtaining blood cultures, it was suspected that the selection of subjects may have differed depending on the facility. Therefore, two validation cohorts were set based on the logic that it is desirable to verify the validity at multiple facilities. Validation cohort 1 (360 subjects) and validation cohort 2 (96 subjects) were defined as patients at Okinawa Prefectural Chubu Hospital and Saku General Hospital, respectively.
Descriptive statistics. We analyzed each item with the two CPRs and used proven bacteremia as a reference standard as well as other clinical information, including sex, blood pressure, respiratory rate, hemodialysis vintage, and presence of diabetes mellitus. Continuous and categorical variables are presented as the median (interquartile range: IQR) and number (percentage), respectively (Table 1).
Basic model. The basic model to assess the value of reclassification of CPRs was conducted using a logistic regression model with explanatory variables of sex (0: female, 1: male), age (y), mean arterial pressure (mmHg), heart rate, body temperature (˚C), presence of diabetes mellitus, HD vintage (months), and white blood cell count (× 10 9 /L). These items were selected by clinicians as those typically used when evaluating patients for bacteremia in their daily medical practice. In validation cohort 2, it was clear at the planning stage that the respiratory rate was often missing, so the respiratory rate was excluded from the basic model.
Added value of CPRs. The discriminatory abilities of the basic model (Model 1), the model that added BAC-HD score (Model 2), or the model that added Shapiro's score (Model 3) to the basic model were assessed by calculating the area under the receiver operating characteristic curve (AUC) using a logistic regression model. Calibration of each model was performed based on the slope and intercept of the calibration plot [28,29].
The number of patients correctly reclassified by adding the CPR to the basic model is shown using Net Reclassification Improvement (NRI) ( Table 2). The prediction probabilities in the three models were stratified based on the tertile of the prediction probabilities of the basic model, as low (< 0.08), intermediate (0.08-0.2), or high (> 0.2).
Assessment of performance. To evaluate potential cut-off scores, we computed the sensitivity, specificity, likelihood ratio, positive predictive value, and negative predictive value for the CPRs. For brevity, only the values in validation cohort 1 are summarized. Handling of missing values. All missing values were addressed using multiple imputations by chained equation treated as missing at random; ten imputed datasets were created. Three logistic regression models were conducted on each of 10 datasets and combined with Rubin's rule.
All statistical analyses were performed using Stata version 15.0 (Stata Corp., College Station, TX, USA).

Study participants
As the final analytic cohort, there were 360 participants and 96 participants in validation cohort 1 and validation cohort 2, respectively, as shown in     increased the number of patients predicted to be low-probability from 142 to 215, of whom 208 were not bacteremic (NPV 96.7%). The number of patients predicted to have a high probability of bacteremia was increased from 40 to 45, of whom 19 were bacteremic (PPV 42.2%). Adding Shapiro's score (Model 3) increased the number of patients predicted to be low-probability from 142 to 177, of whom 169 were not bacteremic (NPV 95.4%). The number of patients predicted to be high probability was reduced from 40 to 36, of whom 14 were  Table 2 and the NRIs for addition of BAC-HD score and Shapiro's score were 0.27 and 0.13, respectively.

Assessment of performance
The sensitivities, specificities, likelihood ratios and predictive values for possible cut-off scores in CPRs in validation cohort 1 are shown in Table 3.

Discussion
This is the first study to validate and compare the external validity of the prediction rules for bacteremia in HD patients. BAC-HD showed an excellent added value, suggesting that it can  be a useful tool for improving the diagnostic ability of bacteremia in dialysis patients. In addition, BAC-HD is a simple CPR consisting of five items, and has the advantage of being highly versatile in clinical settings. On the other hand, although Shapiro's score has many components, its added value was inferior to the BAC-HD score. The reason for this was considered to be the effects of three characteristics of patients on maintenance HD. The first is the difference in the bacteremia etiology. Bacteremia in the general population is often due to gram-negative rods (GNR) [19][20][21][22], usually due to urinary tract infections (UTI) [1,30], while patients on HD often have gram-positive cocci (GPC) bacteremia due to cutaneous infections [23][24][25]. In our experience, UTI and BSI often have different clinical presentations. Therefore, it is considered that Shapiro's score, a CPR developed in a population with a high rate of UTI, could not predict bacteremia in maintenance HD patients with a high rate of cutaneous infections. This is consistent with our previous studies showing that the systemic inflammatory response syndrome (SIRS) criteria [31] and the quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA) score [32,33] were not useful in predicting bacteremia in HD patients. Second, the clinical information of patients on maintenance HD such as body weight, vital signs [34], electrolytes, blood urea nitrogen, and creatinine vary greatly between dialysis and non-dialysis days. Since bacteremia is assumed to develop regardless of the timing of dialysis, the predictive ability of these values may be impaired. Third, patients with end-stage renal failure are often immunocompromised and may have different clinical presentations [35].
This study has two strengths. First, since we verified the added value in two validation cohorts, the robustness of the results is likely increased. Second, the basic model was used to show the added value of CPRs. Although some external validation studies only showed discrimination and calibration of the CPR itself, this is not enough to evaluate the degree of improvement in predictive ability [36].
This study also has some limitations. First, since only Japanese patients with HD were included, its validity in other ethnic groups is unknown. On the other hand, AV-fistula (AVF) is used as vascular access in more than 93% [37] of Japanese on HD, satisfying the 65% goal of the fistula first initiative [38]. Since AVF is associated with a lower risk of infection compared to other vascular access methods, especially central venous catheters [39], it seems significant that added value was shown in a population with a high proportion of AVFs, which is the desired result. Second, since this study analyzed retrospective medical data, there are risks of bias caused by missing values or lower measurement accuracy. However, we performed multiple imputations for missing data to minimize such a bias [40]. Prospective studies are needed for better verification. Third, since bands, which were one of the components of Shapiro's score, were not evaluated, the score may have been underestimated. Since a number of facilities cannot always measure the proportion of bands, such as facilities included in a previous validation study of Shapiro's score [12], this is considered acceptable in terms of versatility. Since there are studies (including this present study) that have modified the items included in Shapiro's score, future validation of these is awaited [41]. Fourth, since there were no standard criteria for when to obtain blood cultures, the possibility that there was some degree of arbitrariness in the decision to draw blood cultures cannot be denied. However, since we used data before the development of the BAC-HD score, it is unlikely that the items in the BAC-HD score influenced the decision on whether or not to draw blood cultures. Furthermore, it is possible that some of the patients whose blood cultures were not drawn included cases of bacteremia. Fifth, it is unclear whether blood cultures were collected from a central venous catheter (CVC) that was the site of vascular access. However, since it is unlikely that both sets of blood cultures were collected from the CVC, and the number of CVCs was small, we believe that the effect of this was not significant.

Conclusions
We verified the added value of the BAC-HD score and Shapiro's score to the usual criteria for predicting bacteremia in HD patients. We suggest that either the BAC-HD score or Shapiro's score may improve the accuracy of predicting bacteremia in patients on HD. Reclassification was better with the BAC-HD score.
Improving the diagnostic ability is expected to contribute to the early initiation of appropriate treatment and improve the prognosis of bacteremia.
Supporting information S1