Derivation and Validation of a Scoring System to Identify Patients with Bacteremia and Hematological Malignancies at Higher Risk for Mortality

Background The aim of this study was to develop and validate a reliable clinical prediction rule that could be employed to identify patients at higher likelihood of mortality among those with hematological malignancies (HMs) and bacterial bloodstream infections (BBSIs). Methods and Findings We conducted a retrospective cohort study in nine Italian hematological units. The derivation cohort consisted of adult patients with BBSI and HMs admitted to the Catholic University Hospital (Rome) between January 2002 and December 2008. Survivors and nonsurvivors were compared to identify predictors of 30-day mortality. The validation cohort consisted of patients hospitalized with BBSI and HMs who were admitted in 8 other Italian hematological units between January 2009 and December 2010. The inclusion and exclusion criteria were identical for both cohorts, with type and stage of HMs used as matching criteria. In the derivation set (247 episodes), the multivariate analysis yielded the following significant mortality-related risk factors acute renal failure (Odds Ratio [OR] 6.44, Confidential Interval [CI], 2.36–17.57, P<0.001); severe neutropenia (absolute neutrophil count <100/mm3) (OR 4.38, CI, 2.04–9.43, P<0.001); nosocomial infection (OR, 3.73, CI, 1.36–10.22, P = 0.01); age ≥65 years (OR, 3.42, CI, 1.49–7.80, P = 0.003); and Charlson Comorbidity Index ≥4 (OR, 3.01, CI 1.36–6.65, P = 0.006). The variables unable to be evaluated at that time (for example, prolonged neutropenia) were not included in the final logistic model. The equal-weight risk score model, which assigned 1 point to each risk factor, yielded good-excellent discrimination in both cohorts, with areas under the receiver operating curve of 0.83 versus 0.93 (derivation versus validation) and good calibration (Hosmer-Lemshow P = 0.16 versus 0.75). Conclusions The risk index accurately identifies patients with HMs and BBSIs at high risk for mortality; a better initial predictive approach may yield better therapeutic decisions for these patients, with an eventual reduction in mortality.


Introduction
Intensified treatment protocols with chemotherapy and/or hematological stem cell transplantation (HSCT) result in greater chances of curing patients with hematological malignancies (HMs). However, these potentially life-saving treatments increase the risk of infectious complications. Bloodstream infections (BSIs) are among the most common and severe complications observed in patients with HMs, particularly if they are neutropenic, with a prevalence ranging from 11 to 38% [1][2][3][4][5][6]. In addition, the onset of BSIs within 5 days of stem cell infusion has been reported in approximately 35% of patients who underwent HSCT [7].
Several studies have evaluated the epidemiological and clinical characteristics of bacterial BSI (BBSIs) in patients with HMs [5,8,9,11]. However, some important uncertainties remain, and to the best of our knowledge, no scoring system has yet been developed that predicts the risk of mortality in patients with HMs and concurrent BBSI.
The aim of the present study, conducted in 9 large Italian hospitals, was to develop and validate a reliable, easy-to-use, clinical prediction rule that could be employed to identify patients with higher likelihood of mortality among those with HMs and BBSI.

Ethics Statement
The institutional review board (Comitato Etico, Università Cattolica del Sacro Cuore) approved the study, and informed consent was waived because of the retrospecive observational nature of the study.

Setting and Study Design
To identify risk factors for mortality in patients aged $18 years with HMs and BBSI, we conducted a cohort study in nine Italian hematological units. The derivation cohort consisted of patients with BBSI and HMs admitted to the Catholic University Hospital, located in Rome, between January 2002 and December 2008. Recurrent episodes of BBSI for the same patient were excluded from the study. The primary outcome measured was all-cause mortality 30 days after BBSI onset. The survivor and nonsurvivor subgroups were compared to identify predictors of 30-day mortality.
The validation cohort consisted of individuals hospitalized with BBSI and HMs who were admitted to 8 other Italian hematological units between January 2009 and December 2010. The inclusion and exclusion criteria were identical to those used for the derivation cohort, and patients included in the validation cohort were matched with those in the derivation set cohort according to type and stage of HMs.

Definitions and Variables Analyzed
Data collected from hospital charts and the laboratory database included patient demographics, disease and disease stage at time of BBSI, type of HSCT (autologous or allogenic), medical history, clinical/laboratory findings, treatment, and outcome of infection.
The following terms were defined before the data analysis. A BBSI was defined as an infection manifested by (I) the presence in at least 1 blood culture of bacteria other than skin contaminants (i.e., diphtheroids, Bacillus spp., Propionibacterium spp., CoNS, micrococci) or (II) the presence of any bacterial species in at least 2 consecutive blood cultures in a patient with a systemic inflammatory response syndrome [15].
The date of the 1st positive blood culture (index culture) was regarded as the date of BBSI onset.
Infections were classified as polymicrobial if 2 or more different genera were recovered from specimens drawn during the first 48 h of infection, regardless of whether the isolates came from the same or different blood culture sets.
The BBSI was classified as nosocomial if the index blood culture had been drawn more than 48 h after admission to our hospital [16]. When the index culture had been drawn within the first 48 h of hospitalization, the infection was classified as healthcare-associated or community-acquired as defined by Friedman et al. [17].
Neutropenia was defined as an absolute neutrophil count (ANC) of ,500 cells/mm 3 . Neutropenia was considered prolonged if the duration was $10 days and severe if the ANC was ,100 cells/ mm 3 .
Acute renal failure was defined as a serum creatinine value .2 mg/dL in patients with previous normal renal function or an increase of .50% of the baseline creatinine level in patients with preexisting renal dysfunction.
The impact of comorbidities was determined by the Charlson Comorbidity Index [18].

Statistical Analysis
Continuous variables were compared by Student's t test for normally distributed variables and by the Mann-Whitney U test for non-normally distributed variables. Categorical variables were evaluated with the x2 or two-tailed Fisher's exact test. Odds ratios (ORs) and 95% confidence intervals (CIs) were calculated to evaluate the strength of any association that emerged. Values are expressed as the means 6 standard deviations (SD) (continuous variables) or as percentages of the group from which they were derived (categorical variables). Two-tailed tests were used to determine statistical significance; a P value of ,0.05 was considered significant.
Variables associated with mortality in the univariate analysis (P #0.10) were included in a logistic regression model, and a backward stepwise approach was used to identify independent predictors of mortality; in addition, to develop a scoring system that could be applicable at the onset of bacteremia, the variables unable to be evaluated at that time (for example, prolonged neutropenia) were not included in the final logistic model. Variables were retained in the final model if the P value was # 0.05. The final regression model was transformed into a pointbased rule. An equal-weight risk score model, which assigned 1 point to each risk factor, was assessed; in addition, an unequalweight risk score model, with weighted scores assigned to each variable obtained by dividing each regression coefficient by half of the smallest coefficient and rounding to the nearest integer, was also performed [19].
The discriminatory power of the prediction rule in the derivation group was expressed as the area under the receiveroperating characteristic curve (AUROC). An AUROC of 0.5 indicates no discriminative ability, and perfect discrimination (i.e., a test with 100% sensitivity and 100% specificity) is reflected by an AUROC of 1. An AUROC exceeding 0.8 is usually indicative of good to excellent prediction; those in the 0.7-0.8 and 0.6-0.7 ranges reflect moderate and low predictive power, respectively. The sensitivity and specificity of the prediction rule -each with 95% CIs -were calculated at different cut-off values. Positive and negative predictive values (PPV and NPV, respectively) were obtained with standard methods.
All statistical analyses were performed using the Intercooled Stata program, version 11, for Windows (Stata Corporation, College Station, Texas, USA).

Results
Four hundred and ninety-four patients with HMs and BSI were included in the study. Figure 1 indicates the distributions of patients (both from the derivation and validation set) according to the type and stage of HMs (matching criteria).

Derivation Cohort
Two hundred and fifty-one patients with HMs and BBSI met the inclusion criteria for the derivation cohort. Four were excluded because of missing data; thus, a total of 247 cases were included in the analysis. Table 1 summarizes the main clinical and demographic characteristics of patients included in the derivation cohort.
In the logistic regression analysis, the five variables found to be independently associated with 30

Validation Cohort
Three hundred and forty-eight patients with HMs and BBSI were observed in the 8 hospitals involved in the validation study, and 247 were selected by matching criteria with patients from the derivation set; these were included in the validation cohort. Their baseline characteristics are summarized in Table 1. Compared to the validation cohort, the derivation cohort contained a higher percentage of patients with chronic viral hepatitis (P,0.001), who had received corticosteroids (P = 0.02), and had indwelling urinary catheter (P,0.001); on the contrary, compared to the derivation cohort patients, patients included in the validation set had higher rates of diabetes mellitus (P,0.001), neutropenia (P,0.001), indwelling central venous catheter (CVC) (P,0.001), and total parenteral nutrition (P,0.001). In terms of the etiological agents causing BBSI, monomicrobial cases were caused more frequently by Gram-positive bacteria in the derivation set (P,0.001) and by Gram-negative bacteria in the validation set (P,0.001).

Construction and Validation of the Predictive Scoring System
Derivation set. A scoring system that could be used to predict mortality was developed based on the independent risk factors that were identified in the multivariate analysis. An equalweight risk score model was assessed first, assigning 1 point to each risk factor. The distributions of scores according to outcome and of variables for different score points are reported in Tables 3 and 4, respectively.
The AUROC for these data was 0.83 (95% CI, 0.78-0.89), indicating that the model is an excellent predictor of mortality ( Figure 2). The results of Hosmer-Lemshow chi-squared testing (P = 0.17) were indicative of good calibration. The prediction rules derived using this scoring system are listed in Table 5 for different thresholds, with the associated sensitivities, specificities, positive and negative predictive values, and overall accuracy. Using a cutoff score of 3 points to discriminate between high-risk and low-risk patients, the scoring system was found to have a sensitivity of 58%, a specificity of 90%, a positive predictive value of 61%, a negative predictive value of 89%, and an overall accuracy of 83%. Patients with scores of $3 points had an OR for mortality of 5.12 (95% CI 5.75-27.85, P,0.001).
The unequal-weight model, which assigned a different weight to each risk factor based on the multivariable logistic regression coefficient, had the same AUROC values and similar calibration for both the cohorts; because the equal-weight risk score model was easier to apply than the unequal model, we chose to report on the first model only.
Validation set. The prediction rules derived from the scoring system in the validation set are listed in Table 5 with the prognostic performance parameters for the main cut-offs. The ORs for mortality were even higher than those observed in the derivation cohort: 78.46 (95% CI 23.50-293.09, P,0.001) for scores .3. As shown Figure 2, when the prediction rule was applied in the validation cohort, the model once again exhibited excellent predictive power (AUROC 0.95; 95% CI, 0.89-1.00) and good calibration (Hosmer-Lemshow P = 0.75).
Application of the model in the combined cohort. When we combined the two cohorts (n = 494), the predictive effects of the model were similar to those observed in the derivation and validation sets. The ORs for mortality associated with a score of $3 was 24.18 (95% CI 12.94-45.33, P,0.001). The 3 cut-off displayed a sensitivity, specificity, PPV, NPV, and an overall accuracy of 67%, 92%, 63%, 93%, and 88%, respectively. In the combined cohort, the prediction rule had an AUROC of 0.86 (95% CI, 0.83-0.89) (Figure 2).

Discussion
A score that can be used to predict the likelihood of mortality is useful in the evaluation of patients with severe infections because it offers criteria to use when choosing a clinical management strategy. In high-risk populations, such as those admitted to intensive care units (ICUs), several scores are already widely used to predict clinical outcomes [20][21][22]. However, published studies incorporating analyses that establish a risk score for mortality in patients with HMs are scarce and mostly based on adult or pediatric patients in the setting of ICU admissions or on patients who developed febrile neutropenia [23][24][25][26].
To the best of our knowledge, there have been no publications to date reporting the analysis of mortality-based scores in adult non-ICU patients affected by HMs with concurrent BBSI. We have developed and validated an easy-to-use risk stratification tool that is based on five variables that were found to be independently associated with mortality in a population of 494 patients.
Our study was conducted in nine hematological centers that regularly admit high numbers of patients with HMs. The possible confounding effects of different types of HMs and/or various stages of treatment were minimized by matching patients in the derivation and validation cohorts according to these parameters.
The multivariate model identified five factors associated with a higher mortality for patients with BBSI and HMs. These include Table 3. Distribution of scores in the derivation and validation sets. Our score is simple to calculate and is constructed from variables that are readily available at the time of admission and can be generated from the demographic characteristics, elements of the patient history, and routine clinical findings. This score provided good discrimination of mortality risk in both the derivation and validation sets, with AUROCs indicative of an excellent predictive power. Furthermore, the fact that the patients included in the validation cohort came from 8 different hospitals and were hospitalized during different time periods increases the likelihood that our findings can be generalized to a broad range of patients with HMs.
The overall mortality rate was lower (12.1%) in the validation cohort than in the derivation cohort (20.6%); this finding is important considering the better global performance of this score in the validation cohort. In addition, the prevalence of Gramnegative agents, which are usually associated with a worse outcome, is significantly higher in the validation cohort than in the derivation cohort; this result reinforces the predictive value of the score independent of the etiological agents responsible for the BBSI.
When a threshold of $3 was used, the specificity of prediction was over 90% in the derivation set and 94% in the validation set.  Although sensitivity was relatively low at 58% in the derivation set, it reaches 83% in the validation set, and the high specificity of the prediction could improve the targeting of patients with a higher mortality risk. Decisions related to the initiation of more intensive care treatments are challenging, especially when they concern patients with cancer. While the decision to admit patients with cancer to ICUs is difficult to make and should be based on the prognosis for each patient, the application of close patient monitoring and lifesupport measures should be implemented in all patients with BBSI and a high risk of mortality (i.e., $3 points with our score).
In the updated clinical practice guideline for the use of antimicrobial therapy in neutropenic patients with cancer developed by the Infectious Diseases Society of America (IDSA), two different risk classifications have been proposed for identifying high-risk patients: the first is based on expert opinion and identifies high-risk patients as those with anticipated prolonged ($7 days duration) and profound neutropenia (ANC ,100 cells/mm 3 following cytotoxic chemotherapy) and/or significant medical comorbidities; the second classification risk proposed is the Multinational Association for Supportive Care in Cancer (MASCC) scoring system, with a cut-off #21 to identify highrisk patients [27]. The MASCC scoring system, which had been validated only for patients with febrile neutropenia and cancer, has been subsequently applied in a cohort of bacteremic patients with cancer, confirming an approximate correlation between score and risk of complications and death; however, no stratifications have been made between patients with HMs and solid cancers (who could have different characteristics, for example, regarding the role of neutropenia in the development and outcome of BBSIs) [28]. In addition, as highlighted in the updated IDSA guidelines [27], a fundamental difficulty with the MASCC system is the lack of a clear standardized definition of one of its major criteria, i.e., the ''burden of febrile neutropenia'' and symptoms associated with that burden, which might complicate the uniform application of the MASCC tool.
Our scoring system has been developed specifically for hospitalized patients with HMs and concurrent BBSI, and it can be applied soon after BBSI onset. For this reason, it could be used for risk assessment in the setting of patients with HMs and BBSI, as well as in association with the classifications proposed in the IDSA guidelines for the management of patients with febrile neutropenia and cancer.
Finally, inappropriate antimicrobial therapy during the empirical phase of treatment has been well demonstrated as the main risk factor for mortality in non-hematological patients with BBSI [29,30], and the same finding has also been reported in patients with HMs [12,13]. The use of the present scoring system could provide useful information for prescribing a more broad spectrum empirical therapy according to individual (i.e., previous colonization and/or infections) and local bacterial epidemiology in patients with scores $3, until microbiological data become available.
It is important to note that the application of the scoring system in empirical treatment decision-making processes needs further validation to quantify its value as a risk assessment tool compared with the clinical judgment of hospitalists, which is likely to be variable from one setting to another.
In conclusion, we stress that BBSI in patients with HMs is a frequently observed clinical condition that requires prompt recognition and treatment with adequate antibacterial therapy, with consideration of the increasing number of multi-drug resistant etiological agents. Our results have demonstrated that patients with BBSI at higher levels of mortality risk can be reliably identified by the application of a simple clinical prediction rule based on five easy-to-define variables that are readily available at the time of BBSI onset. This type of risk stratification could be an important strategy for improving clinical decision-making in highrisk patients, such as patients with HMs.