We have previously developed and validated a prognostic model to predict the risk for unfavorable outcome in Dutch adults with bacterial meningitis. The aim of the current study was to validate this model in adults with bacterial meningitis from two developing countries, Vietnam and Malawi. Demographic and clinical characteristics of Vietnamese (n = 426), Malawian patients (n = 465) differed substantially from those of Dutch patients (n = 696). The Dutch model underestimated the risk of poor outcome in both Malawi and Vietnam. The discrimination of the original model (c-statistic [c] 0.84; 95% confidence interval 0.81 to 0.86) fell considerably when re-estimated in the Vietnam cohort (c = 0.70) or in the Malawian cohort (c = 0.68). Our validation study shows that new prognostic models have to be developed for these countries in a sufficiently large series of unselected patients.
Citation: Schut ES, Brouwer MC, Scarborough M, Mai NTH, Thwaites GE, Farrar JJ, et al. (2012) Validation of a Dutch Risk Score Predicting Poor Outcome in Adults with Bacterial Meningitis in Vietnam and Malawi. PLoS ONE 7(3): e34311. https://doi.org/10.1371/journal.pone.0034311
Editor: Katharina Kranzer, London School of Hygiene and Tropical Medicine, United Kingdom
Received: October 18, 2011; Accepted: February 25, 2012; Published: March 28, 2012
Copyright: © 2012 Schut et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Bacterial meningitis remains an important cause of morbidity and mortality worldwide, even though effective antibiotic therapy is available. – The most common causes are Streptococcus pneumoniae and Neisseria meningitidis, accounting for 85% of cases in adults. – Fatality rates in patients with meningitis caused by these microorganisms are significant, with rates of 10% and 26% in high-income countries and 15% and 50% in low-income countries. , ,  Of surviving patient up to 50% have neurologic deficits, hearing loss,  and cognitive deficits. , 
We previously developed and validated a risk score to predict the risk for unfavorable outcome, using two cohorts of Dutch adults with bacterial meningitis. , ,  We first derived a score for the risk for an poor outcome by performing logistic regression analyses of data from a prospective cohort study (n = 696).  A key set of independent prognostic variables was selected from 22 potential predictors. A nomogram based on these key variables was constructed to facilitate use in clinical practice. To validate this nomogram, we used data from our randomized controlled trial on adjunctive dexamethasone therapy in adults with bacterial meningitis (n = 301).  In the analysis, 6 of 22 variables that are routinely available within 1 hour after admission were robust enough for inclusion in the final risk score: age, heart rate, Glasgow Coma Scale score, cranial nerve palsies, a cerebrospinal fluid leukocyte count less than 1,000 cells/mm3, and gram-positive cocci in cerebrospinal fluid Gram's stain. The concordance index for the risk score was 0.84 (95% confidence interval, 0.80 to 0.87) in the original cohort and 0.81 (95% confidence interval, 0.74 to 0.87) in the external validation cohort.
This simple bedside risk score is based on six variables that are routinely collected within an hour of admission and helps physicians to reliably stratify bacterial meningitis patients at initial presentation with respect to the risk for an unfavorable outcome.  Risk assessment can be important for physicians because of decisions about the level of care (ward or high-care facility) but may be even more important for informing the patient and his or her relatives. The use of our model may also facilitate the interpretation of future clinical studies in adults with bacterial meningitis because the risk for adverse outcome or the effect of therapeutic interventions is likely to vary in different patient groups.
However, meningitis populations differ substantially depending on the geographic area; the utility of the risk score in other parts of the world is unknown. , ,  The aim of the current study was to assess whether this risk score has a similar predictive value in adult meningitis populations from Vietnam and Malawi as previously found in the Netherlands.
The Dutch risk model was developed from a series of 696 episodes of adult culture-proven community-acquired bacterial meningitis from a nationwide prospective cohort study. , ,  The following six variables were associated with a higher risk for a poor outcome at discharge: higher age, increased heart rate, lower Glasgow Coma Scale score, presence of cranial nerve palsies, a cerebrospinal fluid (CSF) leukocyte count less than 1,000 cells/mm3, and Gram-positive cocci in CSF Gram's stain. Poor outcome was defined as a score on the Glasgow Outcome Scale (GOS) of 4 or lower assessed at discharge. The GOS is a well-validated 5-point scale ranging from 1 (dead) to 5 (good recovery). Further details can be found in two previous publications. , 
The Vietnam cohort was a randomized controlled trial (RCT) of adjunctive dexamethasone therapy for adults in Vietnam.  Patients older than 14 years with suspected bacterial meningitis were eligible to enter the study. Inclusion criteria were clinical evidence of meningitis (defined as nuchal rigidity, with elevations in the white-cell count and protein concentration in the CSF) and at least one of the following: bacteria detected in CSF by Gram's or acridine orange stain; a positive CSF latex agglutination test; pathogenic bacteria cultured from either the blood or the CSF; or a clinical history of less than 7 days of illness, with a cloudy CSF, a white-cell count with more than 60% neutrophils, and a ratio of CSF to blood glucose that was less than 50%. The score on the modified Rankin Scale (mRS) was assessed one month after randomization; unfavorable outcome was defined as a score of 1–5, or death.  Scores on the GOS were not assessed in this study. Data were complete for all six risk score predictors in 408 of 435 Vietnamese patients (94%). Dexamethasone did not improve the outcome overall in this study, although patients with microbiologically proven disease appeared to benefit. We included patients from both treatment arms.
The Malawi cohort was a RCT of adjunctive dexamethasone therapy for adults in Malawi with bacterial meningitis.  Eligible patients were 16 years or older and had suspected meningitis in combination and either cloudy CSF, bacteria in CSF on Gram's staining or a CSF leukocyte count of more than 100 cells/mm3 with >50% neutrophils. The score on the GOS was assessed 40 days after randomization; unfavorable outcome was a GOS score of 1–4. Data were complete for all six risk score predictors in 349 of 465 Malawian patients (75%); data concerning cranial nerve palsies were missing in 99 patients (21%). Dexamethasone did not improve the outcome in this study. We included patients from both treatment arms.
Validation was performed by examining calibration, discrimination, and reassessing the logistic regression coefficients of individual predictors of the Dutch model in the Vietnamese and Malawian dataset. The original coefficients of the Dutch logistic regression model were applied to the patients in the validation datasets to calculate the risk score and predicted risk for each patient. The discrimination of the nomogram model in the validation cohorts was quantified by calculating the area under the Receiver operating characteristic (ROC) curve based on a model that only contained the risk score based on the Dutch coefficients. The area under the ROC is also known as the concordance or c-statistic. Bootstrap sampling (no. of samples 1999) was used to obtain 95% confidence intervals of the c-statistic by taking the 2.5th and 97th percentile value of the bootstrap samples. Calibration or goodness of fit of the original prognostic model was assessed by examining the differences in observed versus predicted probabilities of unfavorable outcome across the range of predicted risk in both the Malawi and Vietnam cohorts. If predictions from the nomogram were consistently too high or too low, we examined whether a change in intercept would improve the calibration. We re-estimated the coefficients of the individual predictors of the nomogram model in the validation cohorts and determined whether these estimates in the validation cohort differed significantly from the original estimates in the Dutch cohort using the independent two-sample Z-test. All analyses were performed using SAS® software version 9.2 (SAS Institute, Cary, North Carolina, USA).
Unfavorable outcome was defined as a score on the GOS of 4 or lower assessed at discharge in The Netherlands and Malawi. In the Vietnam dataset unfavorable outcome was defined as a score on the mRS of 1–5, or death one month after randomization. In the Vietnam dataset additional analyses were performed for mRS scores 0–2 vs. 3–5 or death.
The original studies were approved by the research ethics committees of the University of Malawi College of Medicine and the Liverpool School of Tropical Medicine (Malawi study), and the ethics committee of the Hospital for Tropical Diseases (Vietnam study). Informed written consent was given.
There were many differences in demographic and clinical characteristics between the Dutch cohort (n = 696) and the Vietnamese (n = 426) and Malawian (n = 465) cohorts (Table 1). Patients in Vietnam (median age, 43 years) and Malawi (median age, 33 years) were younger as compared with Dutch patients (median age, 50 years). Antibiotics were administered prior to admission in 55% of Vietnamese patients, and in 25% and 9% of Malawian and Dutch patients respectively (p<0.0001). Patients in Vietnam were predominately male (84%) and those in Malawi were almost uniformly HIV-infected (90%). CSF cultures were negative in 49% and 34% patients from Vietnam and Malawi, as compared <1% in the Dutch cohort. The most common causative bacteria were Streptococcus pneumoniae in The Netherlands (51%) and Malawi (55%), and Streptococcus suis in Vietnam (32%). The percentage of patients with unfavorable outcome varied between cohorts (Netherlands, 34%; Vietnam, 49%; Malawi, 65%).
The Dutch model (original c-statistic = 0.84, 95% CI 0.81–0.86)  was less discriminatory in Vietnamese patients (c = 0.70, 95% CI 0.65–0.75) and Malawian patients (c = 0.68, 95% CI 0.63–0.73). The ROC curves are shown in Figure 1. Calibration showed that the Dutch model underestimated the risk of poor outcome both in Malawi and Vietnam (Table 2). Changing the cut-of for unfavorable outcome in Vietnam (score on mRS 0–2 vs. 3–5 or death) did not improve performance of the risk score. The most striking differences between predicted and observed outcome were found in the lower risk groups. Formal goodness of fit tests clearly confirmed the poor calibration: Vietnam, X2 = 15.04, degrees of freedom = 4, p = 0.005; Malawi, X2 = 29.75, degrees of freedom = 4, p<0.0001). When analyses were restricted to patients with CSF-culture positive bacterial meningitis, the discriminative power of the models in Vietnam (n = 219; c = 0.68, 95% CI 0.60–0.74) and Malawian patients (n = 300; c = 0.73, 95% CI 0.67–0.79) remained low. Analyses for patients treated with dexamethasone and placebo showed similar results, with wider confidence intervals.
Several coefficients from the original six predictors re-estimated in the Malawian and Vietnamese cohorts differed markedly and statistically significantly from the original value in the Dutch cohort (Table 3). In Vietnam, low CSF WBC count was associated with much lower risk of unfavorable outcome as compared with that in the Netherlands (odds ratio [OR] 1.39, 95%CI 0.85–2.28 vs. 3.92, 95%CI 2.51–6.13; p = 0.002). Also the impact of type of bacteria seen in the Gram's stain was significantly different (p<0.001) in Vietnam, in particular the presence of other bacteria was associated with a much higher risk of poor outcome than in the Dutch population (Table 3).
In Malawi, coefficients of tachycardia (OR 0.99, 95%CI 0.53–1.87 vs. 2.97 95%CI 1.67–5.28; p = 0.01), score on the Glasgow Coma Scale (OR 0.79, 95%CI 0.73–0.85 vs. 0.87 95%CI 0.82–0.93; p = 0.05), and bacteria seen in Gram's stain results (p<0.001) differed significantly with those in the Dutch population. In Malawi, no bacteria seen with Gram's stain was associated with the largest increase in risk for a poor outcome, followed by having a negative Gram's stain. In the Dutch population, having Gram positive cocci was associated with the largest increase in risk (see table 3).
We also examined whether inclusion of other characteristics would enhance discriminatory power of the nomogram. In Vietnam, inclusion of prior antibiotic treatment would enhance discriminatory power (OR 1.90, 95% CI 1.18–3.05, p = 0.008). In Malawi, inclusion of blood hemoglobulin concentration (OR 0.84, 95% CI 0.78–0.92, p = 0.0001) and prior antibiotic treatment (OR 0.37, 95% CI 0.22–0.61, p = 0.0001) would enhance discriminatory power.
Our study shows that the Dutch meningitis risk score cannot be used to predict unfavorable outcome in middle- and low-income countries such as Vietnam and Malawi. The discriminatory ability of the Dutch model was substantial reduced in Vietnam and Malawi, indicating that there is more overlap in risk scores between patients with and without an unfavorable outcome. Furthermore, the predicted probabilities of a poor outcome were too low.
Several factors could have contributed to the poorer performance of the Dutch model in these countries. Patients from Malawi were almost uniformly HIV-infected. The high rate of HIV infection has been associated with worse prognosis and is not included in the risk score.  Second, the absence of bacteria in a CSF Gram's stain, a factor included in the prognostic score associated with improved prognosis in the Netherlands, might have the opposite effect in Malawi, as it may indicate partially treated pneumococcal meningitis or tuberculous meningitis, which are associated with a much poorer prognosis. ,  Other factors such as suboptimal supportive health care, limited access to health care, and differences in standards of care, i.e., nursing care, fluid management, are also expected to limit the discriminatory power of our model in the Malawian population. Inclusion of blood hemoglobulin concentration, a surrogate marker of malnutrition, and prior antibiotic treatment which was associated with improved prognosis,  could enhance discriminatory power of a multivariate prognostic model in Malawi.
In Vietnam, the majority of patients with Gram positive cocci seen on the CSF Gram stain had meningitis due to S. suis, the commonest cause of acute bacterial meningitis in southeast Asia.  Reported mortality rates are lower (range 3–18%) than in patients with other types of bacterial meningitis. – In contrast with Malawi, prior antibiotic use was associated with unfavorable outcome in Vietnam.
A low CSF WBC count was associated with poor outcome. Clinical studies have shown that lower CSF WBC counts on admission in patients with bacterial meningitis are associated with sepsis and systemic compromise and adverse outcomes later in disease course. , , ,  Animal studies in a pneumococcal meningitis model showed that lower CSF WBC counts early in disease course were associated with high bacterial load, which correlates with intracranial complications and poor outcome.  These experiments also showed that later in disease course, higher CSF WBC counts correlated with high bacterial loads and were associated with poor outcome.
Our study has limitations. Our validation study was based on patients participating in a RCT where they had to meet specific inclusion criteria suggesting that more typical meningitis patients may have been included, and that patients at each end of the disease spectrum (early and very advanced disease) were less represented. This implies that the model is less suited to predict outcome in these patients.
Our results show that the Dutch prognostic model cannot be used accurately to predict outcome in Malawi and Vietnam. In addition, the efficacy of adjunctive dexamethasone varied between populations. The European trial showed a beneficial effect of dexamethasone,  while the Malawi trial showed no effect,  and the Vietnam trial only showed benefit in proven bacterial meningitis cases.  An individual patient data meta-analysis showed no effect when these populations were combined.  A follow-up observational study in the Netherlands showed that the introduction of adjunctive dexamethasone improved prognosis from bacterial meningitis, an effect similar to that found in the European RCT.  Meningitis populations in high, middle and low income countries appear to be different and prognostic models can not be used interchangeably between these populations. Further studies of prognostic models and underlying pathogenesis must therefore be performed in each population in order to help design rational treatment strategies.
Analyzed the data: ESS JBR DvdB. Contributed reagents/materials/analysis tools: MS NTHM GET JF DvdB. Wrote the paper: ESS MCB MS NTHM GET JF JBR DvdB.
- 1. van de Beek D, de Gans J, Tunkel AR, Wijdicks EF (2006) Community-acquired bacterial meningitis in adults. N Engl J Med 354: 44–53.D. van de BeekJ. de GansAR TunkelEF Wijdicks2006Community-acquired bacterial meningitis in adults.N Engl J Med3544453
- 2. Brouwer MC, Tunkel AR, van de Beek D (2010) Epidemiology, diagnosis, and antimicrobial treatment of acute bacterial meningitis. Clin Microbiol Rev 23: 467–492.MC BrouwerAR TunkelD. van de Beek2010Epidemiology, diagnosis, and antimicrobial treatment of acute bacterial meningitis.Clin Microbiol Rev23467492
- 3. van de Beek D, de Gans J, Spanjaard L, Weisfelt M, Reitsma JB, et al. (2004) Clinical features and prognostic factors in adults with bacterial meningitis. N Engl J Med 351: 1849–1859.D. van de BeekJ. de GansL. SpanjaardM. WeisfeltJB Reitsma2004Clinical features and prognostic factors in adults with bacterial meningitis.N Engl J Med35118491859
- 4. Scarborough M, Thwaites GE (2008) The diagnosis and management of acute bacterial meningitis in resource-poor settings. Lancet Neurol 7: 637–648.M. ScarboroughGE Thwaites2008The diagnosis and management of acute bacterial meningitis in resource-poor settings.Lancet Neurol7637648
- 5. Heckenberg SG, Brouwer MC, van der Ende A, Hensen EF, van de Beek D (2011) Hearing loss in adults surviving pneumococcal meningitis is associated with otitis and pneumococcal serotype. Clin Microbiol Infect Epub. SG HeckenbergMC BrouwerA. van der EndeEF HensenD. van de Beek2011Hearing loss in adults surviving pneumococcal meningitis is associated with otitis and pneumococcal serotype.Clin Microbiol Infect Epub
- 6. van de Beek D, Schmand B, de Gans J, Weisfelt M, Vaessen H, et al. (2002) Cognitive impairment in adults with good recovery after bacterial meningitis. J Infect Dis 186: 1047–1052.D. van de BeekB. SchmandJ. de GansM. WeisfeltH. Vaessen2002Cognitive impairment in adults with good recovery after bacterial meningitis.J Infect Dis18610471052
- 7. Hoogman M, van de Beek D, Weisfelt M, de Gans J, Schmand B (2007) Cognitive outcome in adults after bacterial meningitis. J Neurol Neurosurg Psychiatry 78: 1092–1096.M. HoogmanD. van de BeekM. WeisfeltJ. de GansB. Schmand2007Cognitive outcome in adults after bacterial meningitis.J Neurol Neurosurg Psychiatry7810921096
- 8. Weisfelt M, van de Beek D, Spanjaard L, Reitsma JB, de Gans J (2008) A risk score for unfavorable outcome in adults with bacterial meningitis. Ann Neurol 63: 90–97.M. WeisfeltD. van de BeekL. SpanjaardJB ReitsmaJ. de Gans2008A risk score for unfavorable outcome in adults with bacterial meningitis.Ann Neurol639097
- 9. de Gans J, van de Beek D (2002) Dexamethasone in adults with bacterial meningitis. N Engl J Med 347: 1549–1556.J. de GansD. van de Beek2002Dexamethasone in adults with bacterial meningitis.N Engl J Med34715491556
- 10. Nguyen TH, Tran TH, Thwaites G, Ly VC, Dinh XS, et al. (2007) Dexamethasone in Vietnamese adolescents and adults with bacterial meningitis. N Engl J Med 357: 2431–2440.TH NguyenTH TranG. ThwaitesVC LyXS Dinh2007Dexamethasone in Vietnamese adolescents and adults with bacterial meningitis.N Engl J Med35724312440
- 11. Scarborough M, Gordon SB, Whitty CJ, French N, Njalale Y, et al. (2007) Corticosteroids for bacterial meningitis in adults in sub-Saharan Africa. N Engl J Med 357: 2441–2450.M. ScarboroughSB GordonCJ WhittyN. FrenchY. Njalale2007Corticosteroids for bacterial meningitis in adults in sub-Saharan Africa.N Engl J Med35724412450
- 12. Torok ME, Chau TT, Mai PP, Phong ND, Dung NT, et al. (2008) Clinical and microbiological features of HIV-associated tuberculous meningitis in Vietnamese adults. PLoS One 3: e1772.ME TorokTT ChauPP MaiND PhongNT Dung2008Clinical and microbiological features of HIV-associated tuberculous meningitis in Vietnamese adults.PLoS One3e1772
- 13. Ganiem AR, Parwati I, Wisaksana R, van der Zanden A, van de Beek D, et al. (2009) The effect of HIV infection on adult meningitis in Indonesia: a prospective cohort study. AIDS 23: 2309–2316.AR GaniemI. ParwatiR. WisaksanaA. van der ZandenD. van de Beek2009The effect of HIV infection on adult meningitis in Indonesia: a prospective cohort study.AIDS2323092316
- 14. Mai NT, Hoa NT, Nga TV, Linh le D, Chau TT, et al. (2008) Streptococcus suis meningitis in adults in Vietnam. Clin Infect Dis 46: 659–667.NT MaiNT HoaTV NgaD. Linh leTT Chau2008Streptococcus suis meningitis in adults in Vietnam.Clin Infect Dis46659667
- 15. Wertheim HF, Nguyen HN, Taylor W, Lien TT, Ngo HT, et al. (2009) Streptococcus suis, an important cause of adult bacterial meningitis in northern Vietnam. PLoS One 4: e5973.HF WertheimHN NguyenW. TaylorTT LienHT Ngo2009Streptococcus suis, an important cause of adult bacterial meningitis in northern Vietnam.PLoS One4e5973
- 16. Lun ZR, Wang QP, Chen XG, Li AX, Zhu XQ (2007) Streptococcus suis: an emerging zoonotic pathogen. Lancet Infect Dis 7: 201–209.ZR LunQP WangXG ChenAX LiXQ Zhu2007Streptococcus suis: an emerging zoonotic pathogen.Lancet Infect Dis7201209
- 17. Weisfelt M, van de Beek D, Spanjaard L, Reitsma JB, de Gans J (2006) Attenuated cerebrospinal fluid leukocyte count and sepsis in adults with pneumococcal meningitis: a prospective cohort study. BMC Infect Dis 6: 149.M. WeisfeltD. van de BeekL. SpanjaardJB ReitsmaJ. de Gans2006Attenuated cerebrospinal fluid leukocyte count and sepsis in adults with pneumococcal meningitis: a prospective cohort study.BMC Infect Dis6149
- 18. Weisfelt M, van de Beek D, Spanjaard L, Reitsma JB, de Gans J (2006) Clinical features, complications, and outcome in adults with pneumococcal meningitis: a prospective case series. Lancet Neurol 5: 123–129.M. WeisfeltD. van de BeekL. SpanjaardJB ReitsmaJ. de Gans2006Clinical features, complications, and outcome in adults with pneumococcal meningitis: a prospective case series.Lancet Neurol5123129
- 19. Giampaolo C, Scheld M, Boyd J, Savory J, Sande M, et al. (1981) Leukocyte and bacterial interrelationships in experimental meningitis. Ann Neurol 9: 328–333.C. GiampaoloM. ScheldJ. BoydJ. SavoryM. Sande1981Leukocyte and bacterial interrelationships in experimental meningitis.Ann Neurol9328333
- 20. van de Beek D, Farrar JJ, de Gans J, Mai NT, Molyneux EM, et al. (2010) Adjunctive dexamethasone in bacterial meningitis: a meta-analysis of individual patient data. Lancet Neurol 9: 254–63.D. van de BeekJJ FarrarJ. de GansNT MaiEM Molyneux2010Adjunctive dexamethasone in bacterial meningitis: a meta-analysis of individual patient data.Lancet Neurol925463
- 21. Brouwer MC, Heckenberg SG, de Gans J, Spanjaard L, Reitsma JB, et al. (2010) Nationwide implementation of adjunctive dexamethasone therapy for pneumococcal meningitis. Neurology 75: 1533–9.MC BrouwerSG HeckenbergJ. de GansL. SpanjaardJB Reitsma2010Nationwide implementation of adjunctive dexamethasone therapy for pneumococcal meningitis.Neurology7515339