Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

External Validation of a Case-Mix Adjustment Model for the Standardized Reporting of 30-Day Stroke Mortality Rates in China

  • Ping Yu,

    Current address: Department of Neurology, The Second Hospital, Hebei Medical University, Xinhua District, Shi Jiazhuang, Hebei Province, China

    Affiliation Department of Neurology, The Second Hospital, Hebei Medical University, Shijiazhuang, China

  • Yuesong Pan,

    Affiliations Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China, Beijing Key Laboratory of Translational Medicine for Cerebrovascular Disease, Beijing, China

  • Yongjun Wang,

    Current address: Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Dongcheng District, Beijing, China

    Affiliations Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China, China National Clinical Research Center for Neurological Diseases, Beijing, China, Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Xianwei Wang,

    Affiliation Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Liping Liu,

    Affiliations China National Clinical Research Center for Neurological Diseases, Beijing, China, Neuro-intensive Care Unit, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Ruijun Ji,

    Affiliations China National Clinical Research Center for Neurological Diseases, Beijing, China, Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Xia Meng,

    Affiliation Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Jing Jing,

    Affiliation Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

  • Xu Tong,

    Affiliation Department of Neurology, Tangshan Gongren Hospital, Hebei Medical University, Tangshan, Hebei, China

  • Li Guo ,

    guoli6hb@163.com (LG); yilong528@gmail.com (YW)

    Affiliation Department of Neurology, The Second Hospital, Hebei Medical University, Shijiazhuang, China

  • Yilong Wang

    guoli6hb@163.com (LG); yilong528@gmail.com (YW)

    Affiliations Tiantan Clinical Trial and Research Center for Stroke, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China, China National Clinical Research Center for Neurological Diseases, Beijing, China, Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China

External Validation of a Case-Mix Adjustment Model for the Standardized Reporting of 30-Day Stroke Mortality Rates in China

  • Ping Yu, 
  • Yuesong Pan, 
  • Yongjun Wang, 
  • Xianwei Wang, 
  • Liping Liu, 
  • Ruijun Ji, 
  • Xia Meng, 
  • Jing Jing, 
  • Xu Tong, 
  • Li Guo
PLOS
x

Abstract

Background and Purpose

A case-mix adjustment model has been developed and externally validated, demonstrating promise. However, the model has not been thoroughly tested among populations in China. In our study, we evaluated the performance of the model in Chinese patients with acute stroke.

Methods

The case-mix adjustment model A includes items on age, presence of atrial fibrillation on admission, National Institutes of Health Stroke Severity Scale (NIHSS) score on admission, and stroke type. Model B is similar to Model A but includes only the consciousness component of the NIHSS score. Both model A and B were evaluated to predict 30-day mortality rates in 13,948 patients with acute stroke from the China National Stroke Registry. The discrimination of the models was quantified by c-statistic. Calibration was assessed using Pearson’s correlation coefficient.

Results

The c-statistic of model A in our external validation cohort was 0.80 (95% confidence interval, 0.79–0.82), and the c-statistic of model B was 0.82 (95% confidence interval, 0.81–0.84). Excellent calibration was reported in the two models with Pearson’s correlation coefficient (0.892 for model A, p<0.001; 0.927 for model B, p = 0.008).

Conclusions

The case-mix adjustment model could be used to effectively predict 30-day mortality rates in Chinese patients with acute stroke.

Introduction

Predicting 30-day stroke mortality rates in patients with acute stroke plays an important role in evaluating the prognosis of stroke patients [1,2]. Most previously published models have limited utility in clinical practice, as a result of being validated either in ischemic stroke [3,4] only or in hemorrhagic stroke [5,6] only, rather than in both. Although some models have been validated to accurately predict the mortality after ischemic stroke in Asian populations [7,8], they were not generalizable to both ischemic stroke and hemorrhagic stroke. As a result, these models cannot be used to compare the quality of care among different stroke treatment centers.

Case-mix adjustment models have been developed to meet the need of reporting case mix-adjusted mortality outcomes for stroke treatment services. However, most of these models included multiple variables, including the Oxfordshire community stroke project (OCSP) classification [9,10], plasma glucose levels [11], or those difficult to obtain, such as the GCS verbal score [12]. These limitations may explain why none of these case-mix adjustment models have been incorporated into routine clinical practice.

Recently, Benjamin D. Bray et al developed and validated a model to predict the risk of death for patients with acute stroke within 30 days after admission [13]. This model was derived from a large, prospective national registry of unselected cases of acute stroke in hospitals in England and Wales. The model was relatively simple and easy to implement. Two final models were eventually developed. Model A included age (<60, 60–69, 70–79, 80–89, and ≥90 years), presence of atrial fibrillation on admission, National Institutes of Health Stroke Severity Scale (NIHSS) score on admission, and stroke type (ischemic versus primary intracerebral hemorrhage). Model B was similar to Model A, but included only the consciousness component of the NIHSS score (NIHSS _1A) instead of all of the components comprising the NIHSS score [13]. However, there were few data on whether the model was suitable to use for Asian patients with acute stroke. Research has consistently shown that the incidence of stroke has been surging dramatically in low-to middle income countries. The average onset age for first stroke among individuals in China is approximately 10 years younger than among individuals in Western countries, and individuals in China have a higher percentage of hemorrhagic stroke[14]. Therefore, heterogeneity exists among stroke patients in Asian and Western countries. The aim of our study was to evaluate the accuracy of both model A and model B using data from the China National Stroke Registry (CNSR).

Methods

External validation cohort information

The CNSR is a prospective, nationwide hospital-based stroke registry created between September 2007 and August 2008. Eligible patients were enrolled if they were ≥18 years old and had ischemic stroke, transient ischemic attack, intracerebral hemorrhage, or subarachnoid hemorrhage within 14 days of the index event from 132 hospitals across China. The stroke events were confirmed by brain CT or MRI within 14 days after the onset of symptoms. The design, rationale and baseline characteristics of the CNSR have been reported elsewhere [15]. The registry was approved by the central Institutional Review Board at Beijing Tiantan Hospital. Informed written consent was obtained from all patients or their designated relatives. Authors had no access to identifying participant information during or after data collection. Our present study included only patients with acute ischemic stroke or primary intracerebral hemorrhagic stroke.

Risk factors definition

The baseline severity of neurological impairment was evaluated by the National Institute of Health Stroke Scale (NIHSS) within 24 h after admission. These data were collected via an interview conducted by trained research coordinators. Other data, including patient demographics (e.g., gender, age) and vascular risk factors, were extracted from the medical records. Baseline vascular risk factors included history of hypertension (history of hypertension or antihypertensive drug use), stroke or TIA (defined as being confirmed in a medical chart), dyslipidemia (history of dyslipidemia or lipid-lowering drug use), diabetes mellitus (history of diabetes mellitus or hypoglycemic drug use), atrial fibrillation (history of atrial fibrillation confirmed by at least one electrocardiogram or presence of this type of arrhythmia during hospitalization) and history of coronary heart disease, current or previous smoking, etc.

Outcome measures

The main outcome of interest was all-cause mortality rate within 30 days after admission, which was confirmed via telephone by trained research personnel at Beijing Tiantan Hospital.

Statistical analyses

Age and NIHSS scores were reported as medians (interquartile range); all the other values were percentages. Significance testing of age and NIHSS was performed via t-test for continuous variables and the others were by χ 2 test for categorical variables. Discrimination of the case-mix adjustment model in our study was assessed by receiver operating curve analysis and estimation of the area under the receiver operator curve (c-statistic) and by calibration by plot of observed versus predicted mortality with 10 deciles of predicted risk.

The case-mix adjustment model on which our analysis was based has been published elsewhere [13]. The final two models (Model A and Model B) produced from Bray were both used directly and validated in our study. Comparison between the observed and predicted mortality was assessed by Pearson’s correlation coefficient. Given the large sample size used in our study, we did not use Hosmer and Lemeshow goodness-of-fit test, which is known to be sensitive to sample size, to assess calibration. All analyses above were conducted using SAS (version 9.4; SAS Institute, Cary, NC).

Results

Among the 22,216 patients enrolled in the CNSR, there were 12,415 patients with ischemic stroke and 3,255 patients with primary intracerebral hemorrhagic stroke who consented to participate in follow-up. After excluding 1,722 patients without NIHSS_1A information, a total of 13,948 patients with complete 30-day mortality information were included in our study. Comparing the demographics of CNSR patients with stroke with those in the Sentinel Stroke National Audit Program (Table 1), CNSR patients were younger (median, 67 versus 77 years of age), more likely to be women (68.0% versus 50.2%), were more likely to have had hemorrhagic stroke (20.7% versus 10.5%), and had a lower mortality rate at 30 days (10.9% versus 13.2%). There were no clinically significant differences in baseline characteristics between patients included and those excluded; however, there was a slightly higher proportion of history of stroke or TIA and dyslipidemia among the patients included (Table 2).

thumbnail
Table 1. Characteristics of the model derivation and the external validation cohorts.

https://doi.org/10.1371/journal.pone.0166069.t001

thumbnail
Table 2. Characteristics between patients with NIHSS and those without NIHSS in CNSR.

https://doi.org/10.1371/journal.pone.0166069.t002

Based on the case-mix adjustment model A with full NIHSS score, the c-statistic of our external validation cohort was 0.80 (95% confidence interval, 0.79–0.82). For the case-mix adjustment model B, which included only the NIHSS_1A rather than the full NIHSS, the c-statistic of our external validation cohort was 0.82 (95% confidence interval, 0.81–0.84) (Fig 1). Excellent calibration was reported in the plot of observed versus predicted mortality rates in model A (Pearson correlation coefficient 0.892; P = 0.0005) and model B (Pearson correlation coefficient 0.927; P = 0.008) (Fig 2).

thumbnail
Fig 1. ROC curve analysis of the case-mix adjustment model predicting 30-day stroke mortality rates in the CNSR data set.

Receiver operating characteristic (ROC) curve analysis of the case-mix adjustment stroke model A(A)and model B (B) in the China National Stroke Registry (CNSR) data set.

https://doi.org/10.1371/journal.pone.0166069.g001

thumbnail
Fig 2. Observed vs predicted 30-day mortality in the external validation sample from the CNSR data set.

Observed vs predicted 30-day mortality after admission of case-mix stroke for model A(A) and model B(B), according to 10 deciles of predicted risk in the external validation sample from the China National Stroke Registry (CNSR) data set. Overall, observed and expected mortality rates were highly correlated (Pearson’s correlation coefficient 0.892 in model A and 0.927 in model B), which indicates good calibration.

https://doi.org/10.1371/journal.pone.0166069.g002

Discussion

Our study showed that the two final simple models developed by Benjamin D. Bray et al could be used to predict the 30-day mortality rates in unselected type of stroke patients in the Chinese population. Model B, which included only the consciousness component of the NIHSS score, showed excellent performance, similar to that of Model A, suggesting the importance of the consciousness component of the physical examination. Thus, 30-day mortality might be evaluated based on Model B in stroke patients without complete NIHSS information, which is important in non-stroke centers.

Many studies have shown age to be a potent predictor of mortality after stroke [4,5,11,1618], including both ischemic and hemorrhagic stroke. Atrial fibrillation is an important marker for cardioembolic stroke, which is associated with poor outcomes in thrombolytic therapy and with high mortality [19], probably due to the large infarction volume, a high risk of hemorrhage transformation, and lack of collateral blood supply. NIHSS score at admission, as a stroke severity marker, is a well-established and important predictor of mortality after stroke [7,20,21]. Although level of consciousness is only a component of the NIHSS score, it can forecast mortality after stroke both in the early phase and the long-term phase [22,23]. It can reflect the severity of stroke, possibly because severe loss of consciousness is often due to brain stem, large artery embolic or hemorrhagic stroke.

Many other case-mix adjustment models predicting mortality after stroke require complex assessments (Table 3), which limits their feasibility in routine stroke care [12,24]. The case-mix adjustment model in our study contains only items that could be easily extracted from the in-hospital medical records in China, and it is simple. Thus, it may be helpful for governments to monitor the stroke care services in different hospitals and for health insurers to analyze the cost-effectiveness of stroke care.

thumbnail
Table 3. Selected case-mix adjustment models predicting mortality after stroke and their variables.

https://doi.org/10.1371/journal.pone.0166069.t003

There were limitations to this study. First, the centers participating in the CNSR might not be representative of the varying types of Chinese hospitals because the study recruited more centers in urban areas than in rural areas. Second, there might be selection bias due to the difference in history of stroke or dyslipidemia between the patients included and those excluded. Third, patients with undetermined stroke type were not included in the model derivation and validation; however, CT scanning is very common in China and is available 24 hours a day, 7 days a week in 99.2% of the hospitals, according to the China National Stroke Registry.

Conclusions

In our study, we found that the case-mix adjustment model could be used to accurately predict 30-day mortality in Chinese patients with acute stroke.

Acknowledgments

We thank all participating hospitals, colleagues, nurses, and imaging and laboratory technicians for their contributions to the CNSR.

Author Contributions

  1. Conceptualization: PY LG Yilong W.
  2. Formal analysis: YP.
  3. Funding acquisition: Yongjun W.
  4. Investigation: PY RJ.
  5. Methodology: Yilong W. LL.
  6. Project administration: XM.
  7. Resources: XM.
  8. Software: YP.
  9. Supervision: Yilong W. LG.
  10. Validation: YP.
  11. Writing – original draft: PY JJ.
  12. Writing – review & editing: RJ XT XW.

References

  1. 1. Carandang R, Seshadri S, Beiser A, Kelly-Hayes M, Kase CS, Kannel WB, et al. (2006) Trends in incidence, lifetime risk, severity, and 30-day mortality of stroke over the past 50 years. JAMA 296: 2939–2946. pmid:17190894
  2. 2. Feigin VL, Lawes CM, Bennett DA, Barker-Collo SL, Parag V (2009) Worldwide stroke incidence and early case fatality reported in 56 population-based studies: a systematic review. Lancet Neurol 8: 355–369. pmid:19233729
  3. 3. Saposnik G, Kapral MK, Liu Y, Hall R, O'Donnell M, Raptis S, et al. (2011) IScore: a risk score to predict death early after hospitalization for an acute ischemic stroke. Circulation 123: 739–749. pmid:21300951
  4. 4. Papavasileiou V, Milionis H, Michel P, Makaritsis K, Vemmou A, Koroboki E, et al. (2013) ASTRAL score predicts 5-year dependence and mortality in acute ischemic stroke. Stroke 44: 1616–1620. pmid:23559264
  5. 5. Hemphill JC 3rd, Bonovich DC, Besmertis L, Manley GT, Johnston SC (2001) The ICH score: a simple, reliable grading scale for intracerebral hemorrhage. Stroke 32: 891–897. pmid:11283388
  6. 6. Tuhrim S, Dambrosia JM, Price TR, Mohr JP, Wolf PA, Hier DB, et al. (1991) Intracerebral hemorrhage: external validation and extension of a model for prediction of 30-day survival. Ann Neurol 29: 658–663. pmid:1842899
  7. 7. Zhang N, Liu G, Zhang G, Fang J, Wang Y, Zhang X, et al. (2012) A risk score based on Get With the Guidelines-Stroke program data works in patients with acute ischemic stroke in China. Stroke 43: 3108–3109. pmid:23042659
  8. 8. Zhang N, Liu G, Zhang G, Fang J, Wang Y, Zhang X, et al. (2013) External validation of the iScore for predicting ischemic stroke mortality in patients in China. Stroke 44: 1924–1929. pmid:23652267
  9. 9. Myint PK, Clark AB, Kwok CS, Davis J, Durairaj R, Dixit A, et al. (2014) The SOAR (Stroke subtype, Oxford Community Stroke Project classification, Age, prestroke modified Rankin) score strongly predicts early outcomes in acute stroke. Int J Stroke 9: 278–283. pmid:23834262
  10. 10. Abdul-Rahim AH, Quinn TJ, Alder S, Clark AB, Musgrave SD, Langhorne P, et al. (2016) Derivation and Validation of a Novel Prognostic Scale (Modified-Stroke Subtype, Oxfordshire Community Stroke Project Classification, Age, and Prestroke Modified Rankin) to Predict Early Mortality in Acute Stroke. Stroke 47: 74–79. pmid:26578661
  11. 11. Wang Y, Lim LL, Levi C, Heller RF, Fischer J (2001) A prognostic index for 30-day mortality after stroke. J Clin Epidemiol 54: 766–773. pmid:11470384
  12. 12. Counsell C, Dennis M, McDowall M, Warlow C (2002) Predicting outcome after acute and subacute stroke: development and validation of new prognostic models. Stroke 33: 1041–1047.
  13. 13. Bray BD, Campbell J, Cloud GC, Hoffman A, James M, Tyrrell PJ, et al. (2014) Derivation and external validation of a case mix model for the standardized reporting of 30-day stroke mortality rates. Stroke 45: 3374–3380.
  14. 14. Liu L, Wang D, Wong KS, Wang Y (2011) Stroke and stroke care in China: huge burden, significant workload, and a national priority. Stroke 42: 3651–3654.
  15. 15. Wang Y, Cui L, Ji X, Dong Q, Zeng J, Wang Y, et al. (2011) The China National Stroke Registry for patients with acute cerebrovascular events: design, rationale, and baseline patient characteristics. Int J Stroke 6: 355–361.
  16. 16. Wang Y, Lim LL, Heller RF, Fisher J, Levi CR (2003) A prediction model of 1-year mortality for acute ischemic stroke patients. Arch Phys Med Rehabil 84: 1006–1011.
  17. 17. Weimar C, Ziegler A, Konig IR, Diener HC (2002) Predicting functional outcome and survival after acute ischemic stroke. J Neurol 249: 888–895.
  18. 18. Weimar C, Konig IR, Kraywinkel K, Ziegler A, Diener HC, German Stroke Study Collaboration. (2004) Age and National Institutes of Health Stroke Scale Score within 6 hours after onset are accurate predictors of outcome after cerebral ischemia: development and external validation of prognostic models. Stroke 35: 158–162.
  19. 19. Wang XG, Zhang LQ, Liao XL, Pan YS, Shi YZ, Wang CJ, et al. (2015) Unfavorable Outcome of Thrombolysis in Chinese Patients with Cardioembolic Stroke: a Prospective Cohort Study. CNS Neurosci Ther 21: 657–661.
  20. 20. Smith EE, Shobha N, Dai D, Olson DM, Reeves MJ, Saver JL, et al. (2013) A risk score for in-hospital death in patients admitted with ischemic or hemorrhagic stroke. J Am Heart Assoc 2: e005207.
  21. 21. Cheung RT, Zou LY (2003) Use of the original, modified, or new intracerebral hemorrhage score to predict mortality and morbidity after intracerebral hemorrhage. Stroke 34: 1717–1722.
  22. 22. Henon H, Godefroy O, Leys D, Mounier-Vehier F, Lucas C, Rondepierre P, et al. (1995) Early predictors of death and disability after acute cerebral ischemic event. Stroke 26: 392–398.
  23. 23. Sheikh K, Brennan PJ, Meade TW, Smith DS, Goldenberg E (1983) Predictors of mortality and disability in stroke. J Epidemiol Community Health 37: 70–74.
  24. 24. Teale EA, Forster A, Munyombwe T, Young JB (2012) A systematic review of case-mix adjustment models for stroke. Clin Rehabil 26: 771–786.
  25. 25. Fullerton KJ, Mackenzie G, Stout RW (1988) Prognostic indices in stroke. Q J Med 66: 147–162.
  26. 26. Allen CM (1984) Predicting the outcome of acute stroke: a prognostic score. J Neurol Neurosurg Psychiatry 47: 475–480.
  27. 27. Gompertz P, Pound P, Ebrahim S (1994) Predicting stroke outcome: Guy's prognostic score in practice. J Neurol Neurosurg Psychiatry 57: 932–935.
  28. 28. Frithz G, Werner I (1976) Studies on cerebrovascular strokes. II. Clinical findings and short-term prognosis in a stroke material. Acta Med Scand 199: 133–140.