Introduction

Older adults are at high risks of developing serious complications from influenza due to the immune changes with increasing age compared to younger population [1]. Approximately 70%–85% of influenza-related deaths and 50%–70% influenza-related hospitalizations occurred in older population [1]. When older adults are infected with influenza, 67% of them become housebound temporarily and 25% become bedbound temporarily [2]. Therefore, influenza contributed a significant burden to the mortality and morbidity in older population [2].

Mortality and morbidity in older adults with influenza are high, thus, predicting adverse outcome and early intervention are important. However, clinical manifestations in older adults with influenza are usually atypical and complicated with an underlying illness [3]. A Canadian study recruiting patients aged ≥60 years in the emergency department (ED) revealed that only 31% had temperature of ≥37.8°C and cough, with or without sore throat, which are the criteria for influenza-like illness defined by the United States Centers for Disease Control and Prevention [3]. In 2018, Chung et al. proposed a Geriatric Influenza Death (GID) score, intending to develop a useful clinical decision rule (CDR) to help in decision making for this population [4]. The GID score consists of five predictors as following: severe coma (Glasgow coma scale score of ≤8, 2 points), histories of cancer or coronary artery disease (1 point for each history), elevated C-reactive protein (CRP) levels (>10 mg/dl, 1 point), and bandemia (>10% band cells, 1 point) [4]. The GID score ranges from 0 to 6 [4]. When an older adult is diagnosed with influenza in the ED, the GID score can be calculated easily, and 30-day mortality can be predicted [4]. In a previous study including three risk groups, including low risk group (≤1 point, 1.1% mortality), moderate risk group (2 points, 16.7% mortality), and high risk group (≥3 points, 40% mortality), the area under the curve (AUC) and the Hosmer-Lemeshow goodness of fit tests were 0.861 and 0.578 for the GID score [4]. While the previous study showed that he GID score had good performance in predicting mortality in older patients with influenza in the ED, its validation study is still unavailable. Therefore, we conducted this multicenter study to externally validate the GID score.

Methods

Results

A total of 5,508 patients were identified in this study (Table 1). The mortality rate was 2.2% (121/5508). The mean age (standard deviation) was 76.6 (7.4) years and percentage of males was 49.3%. Patients with mortality predominated in the moderate elderly (41.3%), followed by the old elderly (38.8%), and then the young elderly (19.8%). The average mortality rate was highest in the old elderly (5.2%, 47/902), followed by the moderate elderly (2.2%, 50/2234) and the young elderly (1.0%, 24/2372). Compared to the survival group, the mortality group had lower GCS, SBP, and body temperature but higher heart rate and respiratory rate. Patients with mortality had more underlying comorbidities, including hypertension, COPD, CVA, malignancy, CHF, dementia, and bedridden than patients who survived. As to laboratory data, patients with mortality had higher WBC, bandemia, platelet, serum creatinine, and hs-CRP but lower hemoglobin than those who survived. The average stay in hospital was 7.2 ± 7.5 days. The GID score was higher in patients with mortality than those who survived (1.7 ± 1.1 vs. 0.8 ± 0.8, p < 0.01). The missing data and given values are listed in the S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of all older patients with influenza between 2009 and 2018 in ED.

https://doi.org/10.1371/journal.pone.0283475.t001

Compared to patients with GID score of 0, ORs of mortality in patients with GID score of 1, 2, and ≥3 were 3.08 (95% confidence interval [CI]: 1.66–5.71), 6.69 (95% CI: 3.52–12.71), and 23.68 (95% CI: 11.95–46.93), respectively (Table 2). The AUC was 0.722 (95% CI: 0.677–0.766) (S1 Fig), and the Hosmer-Lemeshow goodness of fit test was 1.000.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Logistic regression analysis, AUC, and Hosmer–Lemeshow goodness of fit test for validating the accuracy of GID score in older patients with influenza in ED.

https://doi.org/10.1371/journal.pone.0283475.t002

In the performance analysis, the NPV of the GID score across three score groups were excellent (0.994 with GID score ≥1, 0.987 with GID score ≥2, and 0.982 with GID score ≥3). For predicting mortality, the GID score cutoff of 1 had the best sensitivity (0.893) and NPV (0.994), and the cutoff of 3 had the best specificity (0.968) (Table 3). Mortality rates associated with GID score of ≥3, GID score of 2, GID score of 1, and GID score of 0 were 13.3%, 4.2%, 1.9%, and 0.6%, respectively (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Mortality rates in three groups of GID score (0 vs. 1 vs. 2 vs. ≥3).

GID, geriatric influenza death.

https://doi.org/10.1371/journal.pone.0283475.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance of GID score in predicting mortality in older patients with influenza in ED.

https://doi.org/10.1371/journal.pone.0283475.t003

Discussion

This external validation study showed that GID score had an acceptable discrimination and a good fit for predicting mortality for older patients with influenza in the ED. The mortality rate increased with the GID score. The GID score performed best in NPV. GID score of ≥1 had the best sensitivity and NPV for predicting mortality, whereas GID score of ≥3 had the best specificity.

Compared to the original study that developed the GID score, the mortality rate in this validation study was lower (2.2% vs. 4.9%). Possible reasons are different population and measurement of mortality. The original GID score was developed in an ED of a medical center in northern Taiwan [4], whereas the validation study was conducted in three hospitals in southern Taiwan. In addition, the validation study recruited 5,508 patients, which is a far larger sample size than in the original study (409 patients). We adopted in-hospital mortality in the validation study because of the difficulty in following 5,508 patients after discharge, which is also different from the original study. The “in-hospital mortality” should be lower than the “30-day mortality” due to the shorter follow-up period. In the literature, the mortality in older adults with influenza has a great variation, ranging from 0.009% to 14.3% in the nursing home for the elderly [6] due to different race, medical care system, and time period.

The AUC in the validation study is 0.722, lower than the 0.861 AUC in the original study [4]. Other studies have revealed that CDRs always perform better in the dataset which they derived than their applications either in internal or external validations [6, 7]. Lower performance in the validation study may be due to differences in patient samples and prevalence of the disease, over-fitting, unsatisfactory model derivation, absence of important predictors, and differences in interpretation and measurement of predictors [8, 9]. Even if a CDR is well developed, it is not necessarily be generalizable to new populations [8]. Thus, external validation, such as this study, is an essential process to assess the performance of a CDR [8].

Aims of external validation includes taking the original model and its predictors and regression coefficients, measuring the predictor and outcome variables in a new population, applying the original CDR to these data to predict the outcome of interest, and comparing the predictive performance of the CDR by analyzing outcomes [8, 10]. The evaluation of a CDR performance can be done by discrimination, calibration, and measurement to quantify clinical usefulness such as decision curve analysis [8, 11]. A CDR can be revised according to results if it performs poorly in the external validation [8]. Despite external validation as an essential process for clinical use of CDR, few CDRs have been externally validated [8, 12–14]. In a systematic review of 101 CDRs showed that 76% of them had no such validation, 17% had narrow validation, 8% had broad validation, and none had impact analysis [14].

The GID score performs well for the NPV across different scores in this study. In addition to the well-developed GID score, another explanation may be due to low mortality rate in the study population, which also explains the low PPV (0.031 with the GID score cut-off of 1, 0.058 with the cut-off of 2, and 0.132 with the cut-off of 3) in this study. The GID score cut-off of 1 had a sensitivity of 0.893, which suggests that the cut-off “1” may be used for identifying patients with a high mortality risk. The GID score cut-off of 3 had a specificity of 0.968, which suggests that the cut-off “3” may be used for identifying patients with a low mortality risk. The cut-off point depends on the aim of clinicians.

In the development and evaluation of CDR, three main stages, including derivation, validation, and impact analysis were found [8, 15]. Suggested requirements for designing external validation include a prospective multicenter design, a minimum of 100 outcome events, and a framework of generalizability to enhance the interpretation of findings [8, 16]. Suggested requirements for types of external validation are conducting temporal, geographical, and domain validation studies and meta-analysis using a published framework to summarize the overall performance of the CDR [8, 17]. After the validation, refinement of a CDR, including model updating or adjustment are suggested [8]. Final aim of a CDR is to improve the quality of care, thus, the impact of a validated CDR on patient outcomes and clinician behavior is suggested to be examined [8, 18]. Evaluation of cost-effectiveness and long-term implementation and dissemination are also advised after the impact analysis [8].

The major strength of this study is that it is the first external validation study for GID score. The study had a large-sample size and used multicenter design. There are some limitations as the follows. First, there were different epidemics within the time range that could have biased the results. The major circulating viruses were influenza A (H1N1) in 2009, 2010, 2015, and 2018; influenza A (H3N2) in 2012, 2013, 2014, and 2016; and influenza B in 2011 and 2017 [19]. Second, the definition of disease in the previous paper was presence of fever and identification of flu in nasal swabs whereas in this case it is influenza diagnosis or prescription of common anti-flu drugs. The measurement of outcome was in-hospital mortality in this study, which is different from 30-day mortality in the original study for developing GID score. The difference of disease definition and outcome measurement between two studies may contributed to a lower accuracy in this study. However, the result of this study for predicting in-hospital mortality could provide an important reference for validating GID score for different outcome. Because we had no data about the percentage of death within 30 days after discharge, further study about this issue is needed. Before the widely use of the GID score, more validation studies, effects of epidemics of different viruses, refinement, and preparation for impact analysis may be warranted.

Conclusions

This multicenter study assessed the external validity of the GID score. The discrimination is acceptable, and the fit is good. The GID score has the best performance for NPV. The GID score cut-off of 1 is suggested to be used for identifying patients with a high mortality risk, and the cut-off 3 is suggested to be used for identifying patients with a low mortality risk. Further studies, including more validation studies, refinement, and preparation for impact analysis are suggested.

Supporting information

Acknowledgments

We thank Ms. Chia-Jung Chen and Ms. Tzu-Lan Liu for their assistance in data collection and management and Ms. Yu-Shan Ma for the assistance with statistics. We thank Enago for the English revision.