Several ecologic studies have suggested that the bacillus Calmette-Guérin (BCG) vaccine may be protective against SARS-CoV-2 infection including a highly-cited published pre-print by Miller et al., finding that middle/high- and high-income countries that never had a universal BCG policy experienced higher COVID-19 burden compared to countries that currently have universal BCG vaccination policies. We provide a case study of the limitations of ecologic analyses by evaluating whether these early ecologic findings persisted as the pandemic progressed.
Similar to Miller et al., we employed Wilcoxon Rank Sum Tests to compare population medians in COVID-19 mortality, incidence, and mortality-to-incidence ratio between countries with universal BCG policies compared to those that never had such policies. We then computed Pearson’s r correlations to evaluate the association between year of BCG vaccination policy implementation and COVID-19 outcomes. We repeated these analyses for every month in 2020 subsequent to Miller et al.’s March 2020 analysis.
We found that the differences in COVID-19 burden associated with BCG vaccination policies in March 2020 generally diminished in magnitude and usually lost statistical significance as the pandemic progressed. While six of nine analyses were statistically significant in March, only two were significant by the end of 2020.
Citation: Ledesma JR, Lurie P, Yorlets RR, Daly G, Chrysanthopoulou S, Lurie MN (2022) Spurious early ecological association suggesting BCG vaccination effectiveness for COVID-19. PLoS ONE 17(9): e0274900. https://doi.org/10.1371/journal.pone.0274900
Editor: Wenping Gong, The 8th Medical Center of PLA General Hospital, CHINA
Received: November 21, 2021; Accepted: August 31, 2022; Published: September 20, 2022
Copyright: © 2022 Ledesma et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying the results presented in the study are available from the COVID-19 Data Repository by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (https://github.com/CSSEGISandData/COVID-19).
Funding: ML and PL receive funding from the Sydney E. Frank Foundation. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Since the identification of SARS-CoV-2 in December 2019 in Wuhan, China, COVID-19 has spread to humans in every country in the world, resulting in over 250 million infections and five million deaths as of this writing . Throughout the pandemic, scientists have considered the possibility of repurposing existing preventive and therapeutic agents, including the century-old  bacillus Calmette-Guérin (BCG) vaccine. Currently, 157 countries mandate universal BCG vaccination, and 19 countries (e.g., the United States, Italy, Belgium) only vaccinate high-risk groups . In 2020, laboratory analyses found that BCG vaccination induces specific immunity against the envelope protein of SARS CoV-2 .
However, to date, most of the evidence for BCG protection against COVID-19 has come from a series of ecologic studies, including a highly-cited pre-print by Miller and colleagues in March 2020 . In that paper, Miller et al. found that middle/high- and high-income countries that never had a universal BCG policy experienced higher COVID-19 incidence and mortality compared to countries that currently have universal BCG vaccination policies. In addition, this analysis suggested that the older the universal BCG policy (and thus presumably the larger the proportion of the population vaccinated), the lower the national COVID-19 mortality rate. Since this early work, there have been several observational studies and a meta-analysis of four ecologic studies that yielded a statistically significant pooled negative correlation between the percentage of the population vaccinated with BCG and COVID-19 mortality at the country level (random effect pooled r: -0.48 [95% confidence interval: -0.61 to -0.35]) .
In April 2020, the WHO announced that, in the absence of positive results from RCTs (the gold standard for evaluating vaccine efficacy), there was insufficient evidence of a causal relationship between the BCG vaccine and COVID-19 infection . As of March 2021, there were more than two dozen clinical trials underway across the globe to evaluate whether BCG protects against COVID-19. Completed assessments to date include a case-control study  that did not demonstrate effectiveness and an unpublished RCT  conducted in elderly patients in Greece claiming effectiveness that is limited by small sample size and high loss to follow-up. In the interim, we aimed to evaluate whether the early ecologic findings that BCG appeared to protect against COVID-19 persisted as the pandemic progressed.
To assess the robustness of the Miller et al. findings, we replicated their analysis using identical statistical methods for each month in 2020 subsequent to their March 2020 analysis. Those authors examined the relationship between BCG and COVID-19 morbidity and mortality from January 1, 2020 –March 21, 2020 using COVID-19 data from Google News. Because Google News does not provide historical data by month, we utilised data on COVID-19 incidence and mortality from the COVID-19 Data Repository at Johns Hopkins University. This database compiles data from several reliable sources, including the World Health Organization, several country-specific Centers for Disease Control and Prevention and ministries of health, and provides historical data by month .
A comparison between the Miller et al. March 2020 dataset and the March 2020 dataset generated by the Johns Hopkins Repository showed these two sources yield similar results, with the Hopkins Repository providing on average 1.07 [range: 0.00 to 2.45] and 1.09 [range: 0.50 to 2.33] times higher mortality and incidence counts, respectively, for each country. In addition, replicating the Miller et al. analyses with the Hopkins Repository data for March provided similar statistical results at the p = 0.05 level.
Like Miller et al., we evaluated three outcomes: per capita COVID-19 mortality (number of deaths from COVID-19 per 100,000 population), incidence (number of COVID-19 cases per 100,000 population), and mortality-to-incidence ratio (the number of deaths divided by the number of cases). Like Miller et al., we extracted data on country-specific BCG vaccination policies from the BCG World Atlas , restricting our analyses to countries that 1) have a population of over one million people, and 2) are classified as middle/high- or high-income per World Bank categories . Wilcoxon Rank Sum Tests were then used to compare population medians between countries with universal BCG policies and those that never had such policies. Pearson’s r correlations were also computed to evaluate the association between year of BCG vaccination policy implementation and COVID-19 outcomes. Statistical tests were considered significant if p<0.05 (2-sided).
Relationship between policies and outcomes
Miller et al. examined 55 middle/high- and high-income countries that had universal BCG polices in place and five that had never had a universal BCG vaccination policy. In March 2020, those countries with universal vaccination policies had statistically significantly lower COVID-19 mortality, incidence, and mortality-to-incidence ratios than those that never had such BCG policies using Wilcoxon Rank Sum Tests. When we repeated their analyses for subsequent months, we initially found that COVID-19 mortality rates in countries with no history of BCG policies continued to significantly exceed those in counties with current BCG policies, with the magnitude of that difference essentially stable after May 2020 (Fig 1). In contrast, differences between these groups with respect to COVID-19 incidence were not significant from May to October, although they became statistically significant again in November and December. For COVID-19 mortality-to-incidence ratios, statistically significant differences were present through July, but not thereafter.
Relationship between duration of policies and outcomes
Miller et al. also examined the correlation between the number of years of implementation of the BCG policy (for the 45 countries for which data on year of policy implementation were available) and the same three COVID-19 outcomes. Twenty-eight countries had current universal BCG policies, while 17 previously had them.
For those with current BCG policies (Fig 2A), a statistically significant positive association was present for mortality rate in March 2020 but not thereafter; Pearson correlation coefficients for incidence rate and mortality-to-incidence ratio were not statistically significant at any period but the correlation coefficients were lower in all months than in March 2020 (and sometimes negative suggesting that more recent BCG policy implementation was associated with lower burden of COVID-19). For countries with previous universal BCG vaccination policies (Fig 2B), statistically significant findings for mortality and incidence in March 2020 were no longer apparent by April and May 2020, respectively, and remained non-significant throughout the study period. The correlation coefficient for mortality-to-incidence ratio was never statistically significant. Except for a small increase in the correlation coefficient for incidence in April 2020 among countries that no longer have universal BCG policies, all correlation coefficients were lower in every month after March 2020 for all three outcomes.
Our analysis demonstrates that the differences in COVID-19 burden associated with BCG vaccination policies that Miller et al. observed in March 2020 generally diminished in magnitude and usually lost statistical significance as the pandemic progressed. While six of nine analyses were statistically significant in March, only two were significant by the end of 2020, including none of the six correlations between year of BCG policy implementation and COVID-19 burden, three of which were originally considered statistically significant.
Our case study illustrates that the timing of the initial analysis was an important factor in the apparent detection of associations, as we found that the indices measured by Miller et al. varied substantially and typically diminished in magnitude over time. This finding was also observed in a more limited confirmatory analysis, in which Lindestam Arlehamn et al. repeated the analysis of a different ecologic study  and found that certain associations apparent in March 2020 were no longer present in August 2020 .
Ecologic studies have provided valuable initial understandings of the effects of various exposures on population-level disease outcomes. Innovative ecologic studies of lactation on breast cancer , salt intake and blood pressure , and male circumcision on HIV transmission , have provided important insights into causal hypotheses at the individual-level. Other advantages of these studies includes the ability to efficiently examine trends over time, similar to the analysis here where the BCG-COVID relationships were reassessed as the pandemic progressed. Further, ecologic studies are a critical part of the methodological toolbox for evaluating the impact of policy changes and interventions by examining pre and post population-level outcomes.
However, the limitations of ecologic studies have also been well-described and many are magnified here. First, the ecologic fallacy describes the hazards of using population-level data to make inferences at the individual level . Second, neither dichotomising countries by national BCG vaccination policies nor the number of years policies were in effect captures the substantial variability in actual vaccination coverage [3, 17]. Third, particularly in the early phases of the pandemic, there was wide variation in testing rates, case definitions, and case reporting. Recent modeling estimates indicate that regions with generally high BCG vaccination rates have estimated COVID-19 deaths rates 10 to 14 times greater than reported estimates while regions with low BCG vaccination rates have estimated mortality rates 1.4 to 2 times greater than reported estimates , a bias that would tend to produce a spurious finding of vaccine efficacy . Fourth, countries have, to varying extents, implemented a range of interventions to limit the spread of SARS-CoV-2 and ecologic studies to date have not been able to fully adjust for this variation. Fifth, previous studies have yet to standardise for different distributions of effect modifiers (e.g. age, sex, risk factors). Finally, analyses that are snapshots in time do not account for the dynamic nature of the pandemic as countries have reached peaks in COVID-19 burden at different times and have typically experienced a series of sub-epidemics within their borders.
Subsequent ecologic studies of the relationship between BCG vaccination and COVID-19 attempted to overcome these biases. For example, several studies attempted to account for potential exposure misclassification by using vaccine coverage rather than policy as the exposure of interest [11, 20–28]. Others used multivariable frameworks to account for confounding [11, 22, 23, 25, 27–35], but important confounding variables such as health care access, health care quality, and health consciousness were typically not addressed. Moreover, potential differential outcome misclassification bias, coexistence of COVID-19 mitigation policies, and the dynamically changing nature of the pandemic remain and have yet to be fully accounted for in these analyses.
Although ecologic relationships may point the way for additional research, our findings suggest that significant caution is in order in their interpretation. We agree with the WHO’s recommendation that, in the context of COVID-19, the BCG vaccine should only be used in RCTs. While we await those data, the present study underscores the need for caution in evaluating COVID-19 ecologic analyses, and highlights the limitations of this study design more broadly.
- 1. Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. The Lancet Infectious Diseases 2020; 20: 533–534. pmid:32087114
- 2. Luca S, Mihaescu T. History of BCG Vaccine. Maedica 2013; 8: 53–8. pmid:24023600
- 3. Zwerling A, Behr MA, Verma A, et al. The BCG World Atlas: A Database of Global BCG Vaccination Policies and Practices. PLoS Medicine 2011; 8: e1001012. pmid:21445325
- 4. Nuovo G, Tili E, Suster D, et al. Strong homology between SARS-CoV-2 envelope protein and a Mycobacterium sp. antigen allows rapid diagnosis of Mycobacterial infections and may provide specific anti-SARS-CoV-2 immunity via the BCG vaccine. Annals of Diagnostic Pathology; 48. Epub ahead of print October 2020. pmid:32805515
- 5. Miller A, Reandelar MJ, Fasciglione K, et al. Correlation between universal BCG vaccination policy and reduced mortality for COVID-19: an epidemiological study. medRxiv 2020; 2020.03.24.20042937.
- 6. Khera D, Chugh A, Khasbage S, et al. BCG vaccination provides protection against COVID 19: A systematic review and meta-analysis. Research Square. Epub ahead of print 2020.
- 7. The World Health Organization. Bacille Calmetter-Guerin (BCG) vaccination and COVID-19, https://apps.who.int/iris/bitstream/handle/10665/331745/WHO-2019-nCoV-Sci_Brief-BCGvaccination-2020.1-eng.pdf (2020, accessed January 3, 2021).
- 8. Bates MN, Herron TJ, Lwi SJ, et al. BCG vaccination at birth and COVID-19: a case-control study among U.S. military Veterans. Human Vaccines & Immunotherapeutics 2021; 1–8. pmid:34643480
- 9. Tsilika M, Taks E, Dolianitis K, et al. Activate-2: a double-blind randomized trial of BCG vaccination against COVID-19 in individuals at risk. medRxiv. Epub ahead of print 2021. pmid:35865520
- 10. The World Bank. World Bank Country and Lending Group. The World Bank, https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups (2020).
- 11. Escobar LE, Molina-Cruz A, Barillas-Mury C. BCG vaccine protection from severe coronavirus disease 2019 (COVID-19). Proc Natl Acad Sci U S A 2020; 117: 17720–17726. pmid:32647056
- 12. Lindestam Arlehamn CS, Sette A, Peters B. Lack of evidence for BCG vaccine protection from severe COVID-19. Proceedings of the National Academy of Sciences 2020; 117: 25203–25204. pmid:32994350
- 13. MacMahon B, Lin TM, Lowe CR, et al. Lactation and cancer of the breast. A summary of an international study. Bull World Health Organ 1970; 42: 185–94. pmid:5310137
- 14. Stamler J. The INTERSALT Study: background, methods, findings, and implications. The American Journal of Clinical Nutrition 1997; 65: 626S–642S. pmid:9022559
- 15. Wamai RG, Morris BJ, Bailis SA, et al. Male circumcision for HIV prevention: current evidence and implementation in sub-Saharan Africa. J Int AIDS Soc 2011; 14: 49–49. pmid:22014096
- 16. Piantadosi S, Byar DP, Green SB. The ecological fallacy. American Journal of Epidemiology 1988; 127: 893–904. pmid:3282433
- 17. Gopalaswamy R, Ganesan N, Velmurugan K, et al. The Strange Case of BCG and COVID-19: The Verdict Is Still up in the Air. Vaccines (Basel) 2020; 8: 612. pmid:33081331
- 18. Wang H, Paulson KR, Pease SA, et al. Estimating excess mortality due to the COVID-19 pandemic: a systematic analysis of COVID-19-related mortality, 2020–21. The Lancet 2022; 399: 1513–1536.
- 19. Brooks DR, Getz KD, Brennan AT, et al. The Impact of Joint Misclassification of Exposures and Outcomes on the Results of Epidemiologic Research. Current Epidemiology Reports 2018; 5: 166–174.
- 20. Mariita RM, Musila JM. A study on the relationship between Bacillus CalmetteGurin (BCG) vaccination and Covid-19 prevalence: Do other confounders warrant investigation? Journal of Public Health and Epidemiology 2020; 12: 142–150.
- 21. Mohamed Hussein AAR, Salem MR, Salman S, et al. Correlation between COVID-19 case fatality rate and percentage of BCG vaccination: is it true the vaccine is protective? The Egyptian Journal of Bronchology 2020; 14: 25.
- 22. Ogimi C, Qu P, Boeckh M, et al. Association between live childhood vaccines and COVID-19 outcomes: a national-level analysis. Epidemiology and Infection 2021; 149: e75. pmid:33722335
- 23. Li WX. Worldwide inverse correlation between Bacille Calmette–Guérin (BCG) immunization and COVID-19 mortality. Infection 2021; 49: 463–473. pmid:33495884
- 24. Lerm M. On the relationship between BCG coverage and national COVID-19 outcome: could ‘heterologous’ herd immunity explain why some countries are better off? Journal of Internal Medicine 2020; 288: 682–688. pmid:33107999
- 25. Urashima M, Otani K, Hasegawa Y, et al. BCG Vaccination and Mortality of COVID-19 across 173 Countries: An Ecological Study. International Journal of Environmental Research and Public Health 2020; 17: 5589. pmid:32756371
- 26. Kinoshita M, Tanaka M. Impact of Routine Infant BCG Vaccination on COVID-19. Journal of Infection 2020; 81: 625–633. pmid:32795481
- 27. Wickramasinghe D, Wickramasinghe N, Kamburugamuwa SA, et al. Correlation between immunity from BCG and the morbidity and mortality of COVID-19. Tropical Diseases, Travel Medicine and Vaccines 2020; 6: 17. pmid:32868985
- 28. Klinger D, Blass I, Rappoport N, et al. Significantly Improved COVID-19 Outcomes in Countries with Higher BCG Vaccination Coverage: A Multivariable Analysis. Vaccines (Basel) 2020; 8: 378. pmid:32664505
- 29. Berg MK, Yu Q, Salvador CE, et al. Mandated Bacillus Calmette-Guérin (BCG) vaccination predicts flattened curves for the spread of COVID-19. Science Advances; 6. Epub ahead of print August 7, 2020. pmid:32923613
- 30. Brooks NA, Puri A, Garg S, et al. The association of Coronavirus Disease-19 mortality and prior bacille Calmette-Guerin vaccination: a robust ecological analysis using unsupervised machine learning. Scientific Reports 2021; 11: 774. pmid:33436946
- 31. Ebina-Shibuya R, Horita N, Namkoong H, et al. Current national policies for infant universal bacille Calmette-Guérin vaccination were associated with lower mortality from coronavirus disease 2019. Clinical and Experimental Vaccine Research 2020; 9: 179. pmid:32864376
- 32. Ricco’ M, Ranzieri S. BCG vaccination and COVID-19: Was flattening the curve just an illusion? Infectious Diseases Now 2021; 51: 495–497. pmid:33585830
- 33. Hensel J, McAndrews KM, McGrail DJ, et al. Protection against SARS-CoV-2 by BCG vaccination is not supported by epidemiological analyses. Scientific Reports 2020; 10: 18377. pmid:33110184
- 34. Gong W, Wu X. Is the tuberculosis vaccine BCG an alternative weapon for developing countries to defeat COVID-19? Indian Journal of Tuberculosis 2021; 68: 401–404. pmid:34099209
- 35. Joy M, Malavika B, Asirvatham ES, et al. Is BCG associated with reduced incidence of COVID-19? A meta-regression of global data from 160 countries. Clinical Epidemiology and Global Health 2021; 9: 202–203. pmid:33163696