Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Impact of vaccine herd-protection effects in cost-effectiveness analyses of childhood vaccinations. A quantitative comparative analysis

  • Marisa Holubar ,

    Contributed equally to this work with: Marisa Holubar, Maria Christina Stavroulakis

    Affiliation Department of Medicine, Division of Infectious Diseases and Geographic Medicine, Stanford University School of Medicine, Stanford, California, United States of America

  • Maria Christina Stavroulakis ,

    Contributed equally to this work with: Marisa Holubar, Maria Christina Stavroulakis

    Affiliation Department of Pediatrics, Icahn School of Medicine at Mount Sinai/ Elmhurst Hospital Center, New York, New York, United States of America

  • Yvonne Maldonado,

    Affiliation Department of Pediatrics, Division of Pediatric Infectious Diseases and Department of Health Research and Policy, Senior Associate Dean for Faculty Development and Diversity, Stanford University School of Medicine, Stanford, California, United States of America

  • John P. A. Ioannidis,

    Affiliations Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, United States of America, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, United States of America

  • Despina Contopoulos-Ioannidis

    dcontop@stanford.edu

    Affiliations Department of Pediatrics, Division of Pediatric Infectious Diseases and Department of Health Research and Policy, Senior Associate Dean for Faculty Development and Diversity, Stanford University School of Medicine, Stanford, California, United States of America, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, United States of America

Impact of vaccine herd-protection effects in cost-effectiveness analyses of childhood vaccinations. A quantitative comparative analysis

  • Marisa Holubar, 
  • Maria Christina Stavroulakis, 
  • Yvonne Maldonado, 
  • John P. A. Ioannidis, 
  • Despina Contopoulos-Ioannidis
PLOS
x

Abstract

Background

Inclusion of vaccine herd-protection effects in cost-effectiveness analyses (CEAs) can impact the CEAs-conclusions. However, empirical epidemiologic data on the size of herd-protection effects from original studies are limited.

Methods

We performed a quantitative comparative analysis of the impact of herd-protection effects in CEAs for four childhood vaccinations (pneumococcal, meningococcal, rotavirus and influenza). We considered CEAs reporting incremental-cost-effectiveness-ratios (ICERs) (per quality-adjusted-life-years [QALY] gained; per life-years [LY] gained or per disability-adjusted-life-years [DALY] avoided), both with and without herd protection, while keeping all other model parameters stable. We calculated the size of the ICER-differences without vs with-herd-protection and estimated how often inclusion of herd-protection led to crossing of the cost-effectiveness threshold (of an assumed societal-willingness-to-pay) of $50,000 for more-developed countries or X3GDP/capita (WHO-threshold) for less-developed countries.

Results

We identified 35 CEA studies (20 pneumococcal, 4 meningococcal, 8 rotavirus and 3 influenza vaccines) with 99 ICER-analyses (55 per-QALY, 27 per-LY and 17 per-DALY). The median ICER-absolute differences per QALY, LY and DALY (without minus with herd-protection) were $15,620 (IQR: $877 to $48,376); $54,871 (IQR: $787 to $115,026) and $49 (IQR: $15 to $1,636) respectively. When the target-vaccination strategy was not cost-saving without herd-protection, inclusion of herd-protection always resulted in more favorable results. In CEAs that had ICERs above the cost-effectiveness threshold without herd-protection, inclusion of herd-protection led to crossing of that threshold in 45% of the cases. This impacted only CEAs for more developed countries, as all but one CEAs for less developed countries had ICERs below the WHO-cost-effectiveness threshold even without herd-protection. In several analyses, recommendation for the adoption of the target vaccination strategy depended on the inclusion of the herd protection effect.

Conclusions

Inclusion of herd-protection effects in CEAs had a substantial impact in the estimated ICERs and made target-vaccination strategies more attractive options in almost half of the cases where ICERs were above the societal-willingness to pay threshold without herd-protection. More empirical epidemiologic data are needed to determine the size of herd-protection effects across diverse settings and also the size of negative vaccine effects, e.g. from serotype substitution.

Introduction

Cost effectiveness analysis (CEA) studies [1] have been increasingly used worldwide and in the US in particular [2,3] for the development of national immunization strategies. CEA conclusions can be affected by different methodological choices, modeling choices and populations targeted. Baseline vaccine efficacy assumptions can be influential [412]. Of particular interest is the potential impact of indirect vaccine effect assumptions, and specifically vaccine herd-protection (positive effect) and serotype substitution (negative effects) in the community. Herd-protection is the reduction of the disease in non-vaccinated susceptible individuals from widespread humoral immunity and/or decreased carriage (e.g. nasopharyngeal carriage) in vaccinated individuals in the community, that lead to decreased likelihood of non-vaccinated individuals having contact with infected/infectious individuals [13]. Although this phenomenon is widely described, empirical epidemiologic data on the size of indirect vaccine effects are limited. Vaccine CEAs that include indirect vaccine effects in their analyses either use modeling or extrapolate data from studies conducted in other countries, which may have different disease epidemiology.

Vaccine herd-protection has been reported for several childhood vaccinations including pneumococcal (e.g. PCV7, PCV10 and PCV13) [1318] meningococcal, [11,1921] rotavirus [22,23] and influenza vaccines. [24,25] We evaluated the overall impact of including herd-protection assumptions in CEAs for these four childhood vaccinations. We addressed the following questions: How often does the inclusion of herd-protection change the conclusions of CEAs and give favorable results for the target vaccination strategy? How large is the impact of including herd-protection in CEAs? How often does the inclusion of herd-protection drive estimates below the willingness-to pay cost-effectiveness-thresholds? Is there a consistent pattern of herd-protection impact across these four vaccines? And finally, is the impact of herd-protection larger in CEAs for more-developed countries and when industry is involved?

Methods

Inclusion and exclusion criteria

In this evaluation we considered economic analyses characterized by their authors as CEAs, cost-utility analyses or cost-benefit analyses. We will use the term CEA for consistency unless stated otherwise. We included analyses published in English that targeted childhood vaccination strategies for pneumococcal, meningococcal, rotavirus or influenza disease in infants, children or adolescents. Analyses of strategies vaccinating only adults were excluded. We further considered those studies that reported incremental cost effectiveness ratios (ICER) between the target vaccination strategy and the comparator vaccination strategy (or no vaccination) both with and without herd-protection. CEAs that included herd-protection in their base-case scenario (or primary model) were considered eligible if they also performed separate sensitivity/subgroup/sub-model/scenario analyses without herd-protection. Analyses that reported only cost-effectiveness ratios for single vaccination strategies, and not incremental cost-effectiveness ratios between compared vaccination strategies, were excluded. Data for composite indirect vaccine effects including both herd-protection (positive effect) and serotype substitution (negative effect) or only serotype substitution were not included in our primary analyses (data were very limited to allow for meaningful separate analyses).

Our primary ICER metric was the ICER per quality adjusted life-years gained (QALYs), which was the ratio of the incremental cost divided by the QALYs gained (ICER per-QALYs) by the target vaccination strategy versus the comparator vaccination strategy (or no vaccination). We also considered as secondary metrics the ICER per life-years gained (ICER per-LYs) and ICER per disability adjusted life-years avoided (ICER per-DALYs).

Search strategy

We searched PubMed and the Tufts CEA registry [26] (last search was January 1, 2014; search-strategy in Text A in S1 File). For the Tufts CEA Registry searches we entered the type of vaccine (i.e. influenza vaccine) into the basic search function. We also screened the reference list of prior systematic reviews of CEA for the four targeted childhood vaccinations. Reviews, commentaries, editorials, letters, abstracts from meetings and articles published in non-English language were excluded. Articles were screened at Title/Abstract level and potentially eligible articles were reviewed in full text.

Data extraction

From each eligible study we extracted the following information: first author; journal; year; compared interventions (target vaccination strategy versus comparator vaccination strategy or no vaccination [for the characterization of vaccination strategies as target or comparator we used the authors’ definitions and if this was unclear, we considered as target the most recently approved vaccine]); perspective for the cost-analysis (societal or health care system); model (static vs dynamic model [dynamic models are able to produce empiric results influenced by herd-protection indirect-vaccine-effects while static models rely on assumptions for herd-protection]); target population (cohort model vs population model); vaccination coverage rates; monetary unit used (currency and year); industry involvement; size of assumed/modeled herd-protection effect (and reference(s) cited thereof); quantitative ICERs with and without herd-protection per-QALYs gained, per-LYs gained, and per-DALYs avoided (between the compared vaccination strategies; [negative ICERs indicated cost-saving with the target vaccination strategy vs. the comparator]) and the authors’ interpretation of the CEAs findings (the target vaccination strategy was recommended, not recommended, or statement was unclear). Pertinent quantitative data presented only in figures were also extracted using the WebPlot Digitizer software [27]. Estimates in foreign currencies were converted to US dollars for the same year using the OANDA’s currency calculator tool [28]. All values were subsequently inflated to 2016 US dollars to allow comparability of results [29].

When CEAs studies reported analyses for different pairs of compared vaccination strategies, perspectives (e.g. societal or healthcare) and/or for different countries, we considered these as separate analyses.

Scenarios considered

For our “without herd-protection” analyses we used the base-case scenario if it was clearly defined as without herd-protection. If the base-case scenario was not clearly defined, we selected the scenario that was without herd-protection and had the least number of additional assumptions for other parameters (e.g. for discount rates, vaccination coverage rates, waning vaccination immunity, etc.). We considered studies using either static models (provided that sensitivity analyses with herd-protection were also considered in addition to analyses without herd-protection) or dynamic models. If the study used a dynamic model and the base-case scenario already included herd-protection, we selected sensitivity/subgroup/sub-model/or scenario analyses that were “without herd-protection” and had the same assumptions for all other parameters as the base-case scenario.

For the analyses “with herd-protection” (if multiple scenarios were reported), we always selected the scenario closest, in terms of additional parameter assumptions, to the base-case scenario without herd-protection. In 3 pneumococcal-conjugate vaccine [3032] and 1 rotavirus vaccine study [33] where different herd-protection assumptions were considered, we decided a priori to keep the analyses for a herd-protection rate closest to 15%. Moreover, we considered only analyses that included herd-protection for both of the compared vaccination strategies.

Two reviewers (MCS, MH) independently extracted data and discrepancies were further evaluated by a third reviewer (DCI) and solved by consensus.

Statistical analysis

For each compared vaccination strategies we calculated the absolute differences of ICERs per-QALYs gained, of ICERs per-LY gained and of ICERs per-DALYs avoided, “without herd-protection” minus “with herd-protection.” A difference with a positive value indicates that the ICER with herd-protection was more favorable than the ICER without herd-protection. We evaluated the pattern of impact of herd-protection in CEAs across different ICER-metrics and across difference vaccines (median and inter-quartile ranges of ICERs’ differences) and compared ICER-differences across metrics and across vaccines by the non-parametric Kruskal-Wallis test.

In cases where the comparator-strategy was no vaccination we also calculated how often the inclusion of herd-protection led to crossing the cost-effectiveness threshold, from an ICER above that threshold without herd-protection to an ICER below that threshold with herd-protection [34]. For more-developed countries a threshold of $50,000 (or ~£30,000 respectively), representing the assumed societal willingness-to-pay for a QALY (or LY) gained, is often used [3538]; while for less-developed countries the WHO-cost-effectiveness-threshold of X3GDP/capita (gross domestic product per capita) is often used (Text B in S1 File) [39]. In exploratory analyses, we also evaluated whether the number of ICER-analyses crossing the cost-effectiveness threshold (without vs with herd-protection) differed according to country setting (more-developed versus less-developed countries, as defined in Figure A in S1 File); industry involvement; perspective (societal versus health care) and model (dynamic vs static).

We also captured how often the authors concluded that the target vaccination strategy would be considered cost-effective only if herd-protection was also taken into account.

Results

Characteristics of included CEAs

We screened 469 articles and identified 35 eligible studies [3033; 4070] (20 for pneumococcal conjugate vaccines; 4 for meningococcal vaccines (3 conjugate vaccines and 1 polysaccharide vaccines); 8 for rotavirus and 3 for influenza-virus vaccines) with a total of 81 separate analyses for different compared vaccination strategies, countries, and/or perspectives (37 pneumococcal, 13 meningococcal, 22 rotavirus and 9 influenza vaccination strategies analyses); and a total of 99 ICER-analyses (n = 55 per-QALYs [26 studies]; n = 27 per-LYs [15 studies] and 17 per-DALYs [5 studies] (Fig 1, Table 1 and Table A in S1 File). Each study could have reported more than one of these ICERs.

Industry was involved in 69% (24/35) of the CEA studies (and in 64% [63/99] of ICER-outcome-analyses respectively) (Table 1 and Table A in S1 File); the healthcare perspective was analyzed in 70% (69/99) of ICER-outcome-analyses and static models were used in 83% (29/35) of the CEA studies (and 73% [72/99] of ICER-outcome analyses respectively). Additional characteristics of the included studies and ICER-analyses, including the models, assumed vaccination coverage rates and herd-protection assumptions, are described in detail in Table 1 and Tables A- C in S1 File.

In 16% (16/99) of ICER-analyses the target vaccination strategy was already cost saving even without herd-protection (Table 2). In 35% (6/17) of ICERs per-DALYs analyses the estimates without herd-protection were <$150.

thumbnail
Table 2. Differences in ICER/QALY, ICER/LY and ICER/DALY with vs. without Herd protection*.

https://doi.org/10.1371/journal.pone.0172414.t002

Incremental effects with herd-protection (n = 99 ICER-analyses)

The median absolute ICER-differences without vs with herd-protection were $15,620 (IQR: $877 to $48,376; range $-35,835 to $422,085) for ICERs per-QALYs; $54,871 (IQR: $787 to $115,026; range $-12,719 to $246,657) for ICERs per-LYs; and $49 (IQR: $15 to $1,636; range $5 to $13,581) respectively for ICERs per-DALYs (Table 2, Table E in S1 File and Figure B-2 in S1 File).

The bar-plot of the differences across all 4 vaccines of ICERs per-QALYs, with vs without herd-protection, are shown in Fig 2 and Figure C in S1 File; and of the differences of ICERs per-LYs and ICERs per-DALYs in Figure D in S1 File.

thumbnail
Fig 2. Barplot of ICER per-QALYs without vs. with herd-protection across all four childhood vaccines.

X-axis: ICERs per-QALY with herd protection (values inflated to 2016 US dollars, [29]); Grey bars: ICERs per-QALY without Herd Protection; Black bars: ICERs per-QALY with herd-protection. Abbreviations: P = pneumococcal vaccines, M = meningococcal vaccines, R = rotavirus vaccines, F = influenza vaccines.

https://doi.org/10.1371/journal.pone.0172414.g002

Across all vaccines, inclusion of herd-protection gave more favorable results in 89% (88/99) of ICER-analyses (85% of ICER per-QALYs; 89% of ICER per-LYs and in 100% of ICER per-DALYs analyses) (Table D in S1 File). In the remaining 11/99 ICER-analyses the target-vaccination strategies were already cost saving without herd-protection.

When the target vaccination strategies were not already cost saving without herd-protection (n = 83/99), ICERs were always more favorable with inclusion of herd-protection (Table 2 and Table D in S1 File). Among the 16 ICER-analyses that were already cost saving without herd-protection (12 ICER per-QALYs and 4 ICER per-LYs analyses), inclusion of herd-protection gave additionally more favorable cost-saving results in 31% (5/16) of those.

Crossing of cost-effectiveness thresholds

Overall, among 79 ICER-analyses that compared target vaccine strategies vs no vaccination, 48% (38/79) had ICERs that without herd-protection were above the cost-effectiveness threshold of $50,000 for more developed countries and X3GDP/capita for less developed countries (37 ICER-analyses for more-developed countries and 1 ICER-analysis for a less-developed country) (Table D in S1 File). (The WHO-thresholds of X3GDP/capita reported in individual studies for less developed countries are shown in Text B in S1 File).

In 45% (17/38) of those ICERs that were above the cost-effectiveness threshold without herd-protection, the ICERs decreased below that threshold with inclusion of herd-protection (9 ICER per-QALYs and 8 ICER per-LYs analyses) (Table D in S1 File).

This phenomenon was observed only in analyses for more-developed countries. In all but one of the 29 ICER-analyses for less-developed countries, the target vaccination strategy was already below the cost-effectiveness threshold of X3GDP/capita even without inclusion of herd-protection. The ICER per-LY analysis for pneumococcal vaccine PCV7 for Brazil [55] was above the X3GDP/capita cost-effectiveness threshold and remained slightly above that threshold with herd-protection (the ICER per-DALY analysis for the same study was below the X3GDP/capita threshold without herd-protection).

Subgroup analyses according to industry involvement, perspective (healthcare vs societal) and model used (dynamic vs static)

There was no difference in the number of ICER-outcome-analyses that crossed the cost-effectiveness threshold of $50,000 for more-developed countries and X3GDP/capita for less-developed countries (without vs with herd-protection) according to industry involvement, perspective analyzed or model used (among those that were above that threshold without herd-protection and were comparing a target vaccine against no vaccine). These thresholds were crossed in 50% (8/16) of those analyses with industry-involvement vs 41% (9/22) without industry-involvement (p = 0.58); in 44% (7/16) of those analyses with the societal perspective vs 45% (10/22) with the healthcare-perspective vs (p = 0.92) and in 54% (13/24) of those analyses with static-models vs 29% (4/14) with dynamic models (p = 0.13) (Table D in S1 File).

Comparative analyses across ICER-metrics and vaccines

The scatterplot of the three ICER-metrics, with vs without herd-protection, is shown in the in Fig 3. The ICER-differences varied significantly according to the metric used (p = 0.0006 by Kruskal-Wallis, for ICER-differences per-QALYs vs per-LYs vs per-DALYs) (Figure B-2 in S1 File). Moreover, in 59% (10/17) of ICER per-DALYs analyses the differences were <$100 [33, 48] (Table 2). These studies were in low-income countries and according to the World Bank, for low-income countries (with GDP/capita ≤ $735), interventions with ICERs per-DALY ≤$150 are considered attractive [71] (Table 2).

thumbnail
Fig 3. Scatterplot of ICERs (per-QALY gained, per-LY gained and per-DALY averted) with vs. without herd-protection across all four vaccines.

Y-axis: ICERs without herd protection and X-axis: ICERs with herd-protection (values inflated to 2016 US dollars, [29]); Dashed lines in the horizontal and vertical axis correspond to $50,000 threshold without and with herd-protection respectively. (ICERs in the left upper quadrant indicate cases where ICERs were > $50,000 without herd-protection and crossed that threshold with Herd Protection).

https://doi.org/10.1371/journal.pone.0172414.g003

The differences in ICERs per-QALYs (without vs with herd-protection) across all four vaccines are depicted in the box-plot in Figure B-2 in S1 File and at the bar-plots in Fig 2 and Figure C in S1 File. (p = 0.0002 by Kruskal-Wallis). The respective differences for ICERs per-LYs and per DALYs across vaccines were not statistically significant (p = 0.79 and p = 0.37 respectively by Kruskal-Wallis) (Figure D-1 in S1 File and Figure E-1 in S1 File).

Authors’ conclusions

The final authors’ conclusions appear in Table F in S1 File. The authors clearly recommended at least one target vaccination strategy in 69% (24/35) of studies and in another six studies considered that they could have been cost-effective under certain assumptions (herd-protection was considered among the key assumptions that would have changed the conclusions in 4 studies) (Table D in S1 File). Finally, 5 studies did not recommend the target vaccination strategies. Furthermore, the target vaccination strategy was clearly recommended in 83% (20/24) of industry-funded studies vs. 55% (6/11) of non-industry-funded studies (p = 0.07).

Discussion

In this quantitative comparative analysis of the incremental-cost-effectiveness-ratios for four childhood vaccinations with vs without vaccine herd-protection, we showed that inclusion of herd-protection effects had a substantial impact in the estimated ICERs. In cases where the ICERs were above the cost-effectiveness threshold (of assumed societal-willingness-to-pay of $50,000 for more-developed countries or X3GDP/capita for less-developed countries) without herd-protection, inclusion of herd-protection led to crossing of that threshold in 45% of cases, making the target vaccination-strategy more attractive option. This impacted only CEAs for more-developed countries, as all but one of CEAs for less-developed countries had ICERs below the WHO-cost-effectiveness threshold even without herd-protection. We were not able to draw robust conclusions for true differences among those crossing the above thresholds with herd-protection, according to vaccine type, industry involvement status, perspective of analysis and CEA-model used due to the small numbers of analyses within individual subgroups. Among analyses that were not already cost saving without herd-protection, inclusion of herd-protection always gave more favorable results. Moreover, in a third of analyses that were already cost saving, the inclusion of herd-protection gave additionally more favorable cost-saving results.

The ICER-differences varied significantly according to the metric used, with the largest differences seen with the ICERs per-QALYs and per-LYs. The ICER-differences per-DALYs were small; however, even without herd-protection, the ICERs per-DALYs estimates were significantly smaller than ICERs per-QALYs or ICERs per-LYs estimates. In several studies, recommendation for the adoption of the target vaccination strategy depended on the inclusion of herd-protection effects in the calculations. Moreover, the ICER per-QALY differences varied according to vaccine, with the largest differences seen with pneumococcal and meningococcal vaccines. A possible explanation for that could be that the herd-protection effects considered for the pneumococcal and meningococcal vaccines were overall larger than those for the rotavirus and influenza vaccines; however the herd-protection assumptions considered varied significantly across studies even within the same vaccine group as shown in Table B in S1 File. Although no significant differences were detected across vaccines in the ICER-differences per LYs and per-DALYs, the data were more limited.

Empirical epidemiologic data on the size of vaccine-herd-protection effects in different countries were limited and most studies extrapolated herd-protection assumptions from other countries, used fixed herd-protection assumptions or applied modeling [72]. A prior systematic review by van de Vooren et al. [73] showed that among 10 European pneumococcal-conjugate vaccine CEA studies, only one study based herd-protection assumptions on national data, while most of the remaining studies used information for herd-protection and serotype substitution based on an American study. Although the approach of using assumptions rather than actual epidemiologic data for herd-protection might be appropriate for economic evaluations in settings where a vaccine is still being considered (2/10 studies), the majority of these studies were done in countries where the vaccines were already recommended [73]. Extrapolation of herd-protection effects from different countries should be cautiously done as differences in the dominant circulating strains, transmissibility of strains and other social factors (e.g. social mixing situations) as well as differences in the vaccination dosing schedules and vaccination coverage rates may impact the herd-protection effects [19]. Loo et al. [17] showed that indirect vaccine effects from pneumococcal-conjugate vaccines vary widely according to dosing schedule and endpoint studied (e.g. vaccine-serotype associated invasive pneumococcal disease, nasopharyngeal pneumococcal carriage, and pneumonia). This suggests that in order to be most useful, future CEA methodology must account for this complicated epidemiology.

It has been previously shown that most published economic-analyses (not limited to vaccines) reported favorable ICERs for the experimental interventions [74] and industry-sponsored economic-analyses in particular were more likely to report favorable ratios compared to non-industry sponsored CEA studies [7476] In our sample of evaluated vaccine-CEAs the non-industry funded CEAs studies were less than a third of the total number of studies, to allow for detection of true between-group differences.

Our quantitative comparative analysis differs from prior systematic reviews of CEA studies for childhood vaccinations, as these prior reviews were mainly qualitative descriptive reviews. A detailed discussion of the differences between our analysis and these prior reviews is included in Text C in S1 file. In brief, in our quantitative comparative analysis with vs without herd protection: a) we applied very strict criteria for the ICER-analyses to be compared to optimize comparability of results (considering only ICER-analyses where all other parameters/assumptions, except for herd-protection, were the same); b) we calculated the size of the ICER-differences with vs without herd-protection across diverse childhood vaccines and for different ICER-metrics (per-QALYs, per-LYs and per-DALYs) to increase our power to detect true differences; c) we inflated all monetary-values to 2016 US dollars to increase comparability of results across studies and d) we explored factors that could explain the observed differences in the impact of herd-protection across studies, such as country setting, industry involvement status, CEA-perspective, vaccine type and CEA-model used.

Some study limitations should be acknowledged. We analyzed only the positive indirect vaccine effects from herd-protection. However, available data were very limited to allow for the performance of meaningful separate analyses for the impact of negative indirect vaccine effects, such as serotype substitution (either alone or in combination with herd-protection), in the ICER estimates. Only five pneumococcal vaccine CEA-studies included data with and without additional vaccine indirect effects. (We discussed those in detail in Text D in S1 File). We only analyzed economic analyses published in English; pertinent studies published in other languages [77, 78] might have been missed. We used benchmark cost-effectiveness-thresholds to assess the impact of including herd-protection in vaccine CEAs for more-developed countries. These thresholds are arbitrary [34, 79] but nevertheless are widely used to characterize interventions as cost-effective and worth adopting. Moreover, for less-developed low-income countries we applied the widely used WHO cost-effectiveness-thresholds of X3GDP/capita [39].

Overall, there is a need for continued surveillance and collection of robust empirical epidemiologic data on herd-protection positive vaccine effects and negative indirect vaccine effects e.g. from serotype substitution across diverse populations, countries, for different vaccination dosing schedules and vaccination coverage rates. Moreover, further methodological research is needed for the identification of the most efficient methods for incorporating herd-protection effects in economic analyses [19] There is progress towards that direction as international guidelines for the standardization of economic evaluations for vaccines have been recently developed by the European Vaccine Economics Community [80]. Dynamic models should be preferably used in those analyses, instead of static models, as they include the interaction between individuals and therefore account for indirect vaccine-effects [80]. The routine inclusion in vaccine economic analyses of the negative vaccine indirect effects, such as serotype substitution, in addition to the positive herd-protection effects, is necessary. Especially in the case of pneumococcal vaccination there is a need for more sophisticated models that count for carriage of different serotypes and not only for infection or illness. New approaches for the assessment of vaccine herd-protection, such as cluster-randomized trials that can assess vaccine-direct effects, herd-protection effects and negative indirect effects, e.g. from serotype replacement, even before the introduction of vaccines into public health programs, should be considered [81]. Moreover, pre-licensure assessment of vaccine herd-protection should not be used as a replacement for post-licensure assessments, as only post-licensure studies in diverse populations can provide an accurate estimate of vaccine’s herd-protection effects [8184].

Acknowledgments

Dr Contopoulos-Ioannidis had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Author Contributions

  1. Conceptualization: DCI JPAI YM.
  2. Data curation: MH MCS DCI.
  3. Formal analysis: MH MCS DCI.
  4. Investigation: MH MCS DCI JPAI YM.
  5. Methodology: MH MCS DCI JPAI YM.
  6. Project administration: MH MCS DCI.
  7. Software: DCI MCS.
  8. Supervision: DCI.
  9. Validation: MH MCS DCI.
  10. Visualization: MH MCS DCI JPAI YM.
  11. Writing – original draft: DCI MH MCS.
  12. Writing – review & editing: MH MCS DCI JPAI YM.

References

  1. 1. Owens DK, Qaseem A, Chou R, Shekelle P. High-value, cost-conscious health care: concepts for clinicians to evaluate the benefits, harms, and costs of medical interventions. Ann Intern Med. 2011;154(3):174–180. pmid:21282697
  2. 2. Kim JJ. The role of cost-effectiveness in U.S. vaccination policy. The New England Journal of Medicine. 2011;365(19):1760–1761. pmid:22010866
  3. 3. Advisory Committee on Immunization Practices (ACIP): Guidance for Health Economic Studies (http://www.cdc.gov/vaccines/acip/committee/guidance/economic-studies.html Accessed 1/2014.
  4. 4. Bilcke J, Beutels P. Reviewing the cost effectiveness of rotavirus vaccination: the importance of uncertainty in the choice of data sources. PharmacoEconomics. 2009;27(4):281–297. pmid:19485425
  5. 5. Boonacker CW, Broos PH, Sanders EA, Schilder AG, Rovers MM. Cost effectiveness of pneumococcal conjugate vaccination against acute otitis media in children: a review. PharmacoEconomics. 2011;(3):199–211 pmid:21250759
  6. 6. Beutels P, Thiry N, Van Damme P. Convincing or confusing? Economic evaluations of childhood pneumococcal conjugate vaccination—a review (2002–2006). Vaccine. 2007;25(8):1355–1367. pmid:17208339
  7. 7. Ogilvie I, Khoury AE, Cui Y, Dasbach E, Grabenstein JD, Goetghebeur M. Cost-effectiveness of pneumococcal polysaccharide vaccination in adults: a systematic review of conclusions and assumptions. Vaccine. 2009;27(36):4891–4904. pmid:19520205
  8. 8. Bilcke J, Van Damme P, Beutels P. Cost-effectiveness of rotavirus vaccination: exploring caregiver(s) and "no medical care'' disease impact in Belgium. Medical Decision Making: an international journal of the Society for Medical Decision Making. 2009;29(1):33–50
  9. 9. Goossens LM, Standaert B, Hartwig N, Hovels AM, Al MJ. The cost-utility of rotavirus vaccination with Rotarix (RIX4414) in the Netherlands (Structured abstract). Vaccine. 2008;1118–1127. pmid:18215445
  10. 10. Postma MJ, Jit M, Rozenbaum MH, Standaert B, Tu HA, Hutubessy RC. Comparative review of three cost-effectiveness models for rotavirus vaccines in national immunization programs; a generic approach applied to various regions in the world. BMC Medicine. 2011;9:84. pmid:21740545
  11. 11. Getsios D, Caro I, El-Hadi W, Caro JJ. Assessing the economics of vaccination for Neisseria meningitidis in industrialized nations: a review and recommendations for further research. Int J Technol Assess Health Care. 2004;20(3):280–288. pmid:15446757
  12. 12. Manski CF. Vaccination with partial knowledge of external effectiveness. PNAS. 2010;107(9):3953–3960. pmid:20160102
  13. 13. Isaacman DJ, Strutton DR, Kalpas EA, Horowicz-Mehler N, Stern LS, Casciano R, et al. The impact of indirect (herd) protection on the cost-effectiveness of pneumococcal conjugate vaccine. Clinical Therapeutics. 2008;30(2):341–357. pmid:18343273
  14. 14. van Hoek AJ, Sheppard CL, Andrews NJ, Waight PA, Slack MP, Harrison TG, et al. Pneumococcal carriage in children and adults two years after introduction of the thirteen valent pneumococcal conjugate vaccine in England. Vaccine. 2014;32(34):4349–4355. pmid:24657717
  15. 15. Hammitt LL, Akech DO, Morpeth SC, Karani A, Kihuna N, Nyongesa S, et al. Population effect of 10-valent pneumococcal conjugate vaccine on nasopharyngeal carriage of Streptococcus pneumoniae and non-typeable Haemophilus influenzae in Kilifi, Kenya: findings from cross-sectional carriage studies. Lancet Glob Health. 2014;2(7):e397–405. pmid:25103393
  16. 16. von Gottberg A, de Gouveia L, Tempia S, Quan V, Meiring S von Mollendorf C, et al. Effects of vaccination on invasive pneumococcal disease in South Africa. The New England Journal of Medicine. 2014;371(20):1889–1899. pmid:25386897
  17. 17. Loo JD, Conklin L, Fleming-Dutra KE, Knoll MD, Park DE, Kirk J, et al. Systematic review of the indirect effect of pneumococcal conjugate vaccine dosing schedules on pneumococcal disease and colonization. Pediatr Infect Dis J. 2014;33 Suppl 2:S161–171.
  18. 18. Loughlin AM, Hsu K, Silverio AL, Marchant CD, Pelton SI. Direct and indirect effects of PCV13 on nasopharyngeal carriage of PCV13 unique pneumococcal serotypes in Massachusetts' children. Pediatr Infect Dis J. 2014;33(5):504–510. pmid:24670957
  19. 19. Kauf TL. Methodological concerns with economic evaluations of meningococcal vaccines. PharmacoEconomics. 2010; 28(6):449–461. pmid:20465314
  20. 20. Ramsay ME, Andrews NJ, Trotter CL, Kaczmarski EB, Miller E. Herd immunity from meningococcal serogroup C conjugate vaccination in England: database analysis. BMJ. 2003;326(7385):365–366. pmid:12586669
  21. 21. Cohn AC, MacNeil JR, Harrison LH, Hatcher C, Theodore J, Schmidt M, et al. Changes in Neisseria meningitidis disease epidemiology in the United States, 1998–2007: implications for prevention of meningococcal disease. Clin Infect Dis. 2010;50(2):184–191. pmid:20001736
  22. 22. Payne DC, Staat MA, Edwards KM, Szilagyi PG, Weinberg GA, Hall CB, et al. Direct and indirect effects of rotavirus vaccination upon childhood hospitalizations in 3 US Counties, 2006–2009. Clin Infect Dis. 2011;53(3):245–253. pmid:21705316
  23. 23. Lopman BA, Curns AT, Yen C, Parashar UD. Infant rotavirus vaccination may provide indirect protection to older children and adults in the United States. J Infect Dis. 2011;204(7):980–986. pmid:21878425
  24. 24. Loeb M, Russell ML, Moss L, Fonseca K, Fox J, Earn DJ, et al. Effect of influenza vaccination of children on infection rates in Hutterite communities: a randomized trial. JAMA. 2010;303(10):943–950. pmid:20215608
  25. 25. Piedra PA, Gaglani MJ, Kozinetz CA, Herschler G, Riggs M, Griffith M, et al. Herd immunity in adults against influenza-related illnesses with use of the trivalent-live attenuated influenza vaccine (CAIV-T) in children. Vaccine. 2005;23(13):1540–1548. pmid:15694506
  26. 26. Cost Effectiveness Analyses (CEA) Registry. The Center for the Evaluation of Value and Risk in Health and the Institute for Clinical Research and Health Policy Studies. Tufts Medical Center. https://research.tufts-nemc.org/cear4/. (Accessed 1/2014)
  27. 27. WebPlot Digitizer; http://arohatgi.info/WebPlotDigitizer/app/ (Accessed 1/2014).
  28. 28. OANDA. Historical exchange rates (http://www.oanda.com/currency/historical-rates/). Accessed 1/2014)
  29. 29. Bureau of Labor Statistics. United States Department of Labor. CPI (Consumer Price Index) inflation calculator; http://www.bls.gov/data/inflation_calculator.htm (accessed 3/2016)
  30. 30. Díez-Domingo J, Ridao-López M, Gutiérrez-Gimeno MV, Puig-Barberá J, Lluch-Rodrigo JA, Pastor-Villalba E. Pharmacoeconomic assessment of implementing a universal PCV-13 vaccination programme in the Valencian public health system (Spain). Vaccine. 2011 Dec 6;29(52):9640–8 pmid:22027484
  31. 31. Tyo KR, Rosen MM, Zeng W, Yap M, Pwee KH, Ang LW, et al. Cost-effectiveness of conjugate pneumococcal vaccination in Singapore: comparing estimates for 7-valent, 10-valent, and 13-valent vaccines. Vaccine. 2011 Sep 2;29(38):6686–94. pmid:21745516
  32. 32. Urueña A, Pippo T, Betelu MS, Virgilio F, Giglio N, Gentile A, et al. Cost-effectiveness analysis of the 10- and 13-valent pneumococcal conjugate vaccines in Argentina. Vaccine. 2011 Jul 12;29(31):4963–72. pmid:21621575
  33. 33. Atherly DE, Lewis KD, Tate J, Parashar UD, Rheingans RD. Projected health and economic impact of rotavirus vaccination in GAVI-eligible countries: 2011–2030. Vaccine. 2012 Apr 27;30 Suppl 1:A7–14.
  34. 34. Evans C, Tavakoli M, Crawford B. Use of quality adjusted life years and life years gained as benchmarks in economic evaluations: a critical appraisal. Health Care Manag Sci. 2004;7(1):43–49. pmid:14977093
  35. 35. Grosse SD. Assessing cost-effectiveness in healthcare: history of the $50,000 per QALY threshold. Expert Review of Pharmacoeconomics & Outcomes research. 2008;8(2):165–178.
  36. 36. National Institute for Health and Clinical Excellence. Updated guide to the methods of technology appraisal—June 2008. NICE.
  37. 37. Neumann PJ, Cohen JT, Weinstein MC. Updating cost-effectiveness—the curious resilience of the $50,000-per-QALY threshold. The New England Journal of Medicine. 2014;371(9):796–797. pmid:25162885
  38. 38. Cost effectiveness and strategic planning (WHO-CHOICE): Threshold values for intervention cost-effectiveness by Region. http://www.who.int/choice/costs/CER_levels/en/, August 14, 2015.
  39. 39. Garber AM, Phelps CE. Economic foundations of cost-effectiveness analysis. Journal of Health Economics. 1997;16(1):1–31. pmid:10167341
  40. 40. Bergman A, Hjelmgren J, Ortqvist A, Wisløff T, Kristiansen IS, Hogberg LD, et al. Cost-effectiveness analysis of a universal vaccination programme with the 7-valent pneumococcal conjugate vaccine (PCV-7) in Sweden. Scand J Infect Dis. 2008;40(9):721–9 pmid:18712627
  41. 41. Blank PR, Szucs TD. Cost-effectiveness of 13-valent pneumococcal conjugate vaccine in Switzerland. Vaccine. 2012 Jun 13;30(28):4267–75. pmid:22521287
  42. 42. Chuck AW, Jacobs P, Tyrrell G, Kellner JD. Pharmacoeconomic evaluation of 10- and 13-valent pneumococcal conjugate vaccines. Vaccine. 2010 Jul 26;28(33):5485–90. pmid:20554066
  43. 43. Earnshaw SR, McDade CL, Zanotti G, Farkouh RA, Strutton D. Cost-effectiveness of 2 + 1 dosing of 13-valent and 10-valent pneumococcal conjugate vaccines in Canada. BMC Infect Dis. 2012 Apr 24;12:101. pmid:22530841
  44. 44. Giglio ND, Cane AD, Micone P, Gentile A. Cost-effectiveness of the CRM-based 7-valent pneumococcal conjugated vaccine (PCV7) in Argentina. Vaccine. 2010 Mar 8;28(11):2302–10. pmid:20064478
  45. 45. Gomez JA, Tirado JC, Navarro Rojas AA, Castrejon Alba MM, Topachevskyi O. Cost-effectiveness and cost utility analysis of three pneumococcal conjugate vaccines in children of Peru. BMC Public Health. 2013 Oct 30;13:1025. pmid:24171921
  46. 46. Hoshi SL, Kondo M, Okubo I. Economic evaluation of vaccination programme of 7-valent pneumococcal conjugate vaccine to the birth cohort in Japan. Vaccine. 2012 May 9;30(22):3320–8 pmid:22386745
  47. 47. Hubben GA, Bos JM, Glynn DM, van der Ende A, van Alphen L, Postma MJ. Enhanced decision support for policy makers using a web interface to health-economic models—illustrated with a cost-effectiveness analysis of nation-wide infant vaccination with the 7-valent pneumococcal conjugate vaccine in the Netherlands. Vaccine. 2007 May 4;25(18):3669–78. pmid:17360082
  48. 48. Kim SY, Lee G, Goldie SJ. Economic evaluation of pneumococcal conjugate vaccination in The Gambia. BMC Infect Dis. 2010 Sep 3;10:260. pmid:20815900
  49. 49. Martí SG, Colantonio L, Bardach A, Galante J, Lopez A, Caporale J, et al. A cost-effectiveness analysis of a 10-valent pneumococcal conjugate vaccine in children in six Latin American countries. Cost Eff Resour Alloc. 2013 Aug 30;11(1):21. pmid:24004943
  50. 50. McIntosh ED, Conway P, Willingham J, Hollingsworth R, Lloyd A. Pneumococcal pneumonia in the UK—how herd immunity affects the cost-effectiveness of 7-valent pneumococcal conjugate vaccine (PCV). Vaccine. 2005 Feb 25;23(14):1739–45. pmid:15705480
  51. 51. Melegaro A, Edmunds WJ. Cost-effectiveness analysis of pneumococcal conjugate vaccination in England and Wales. Vaccine. 2004 Oct 22;22(31–32):4203–14. pmid:15474710
  52. 52. Newall AT, Creighton P, Philp DJ, Wood JG, MacIntyre CR. The potential cost-effectiveness of infant pneumococcal vaccines in Australia. Vaccine. 2011 Oct 19;29(45):8077–85 (Newall (a)) pmid:21864617
  53. 53. Ray GT, Whitney CG, Fireman BH, Ciuryla V, Black SB. Cost-effectiveness of pneumococcal conjugate vaccine: evidence from the first 5 years of use in the United States incorporating herd effects. Pediatr Infect Dis J. 2006 Jun;25(6):494–501. pmid:16732146
  54. 54. Rubin JL, McGarry LJ, Strutton DR, Klugman KP, Pelton SI, Gilmore KE, et al. Public health and economic impact of the 13-valent pneumococcal conjugate vaccine (PCV13) in the United States. Vaccine. 2010 Nov 10;28(48):7634–43. pmid:20883739
  55. 55. Vespa G, Constenla DO, Pepe C, Safadi MA, Berezin E, de Morales JC, et al. Estimating the cost-effectiveness of pneumococcal conjugate vaccination in Brazil. Rev Panam Salud Publica. 2009 Dec;26(6):518–28. pmid:20107706
  56. 56. Wisløff T, Abrahamsen TG, Bergsaker MA, Løvoll Ø, Møller P, Pedersen MK, et al. Cost effectiveness of adding 7-valent pneumococcal conjugate (PCV-7) vaccine to the Norwegian childhood vaccination program. Vaccine. 2006 Jul 17;24(29–30):5690–9. pmid:16735083
  57. 57. Christensen H1, Hickman M, Edmunds WJ, Trotter CL. Introducing vaccination against serogroup B meningococcal disease: an economic and mathematical modelling study of potential impact. Vaccine. 2013 May 28;31(23):2638–46. pmid:23566946
  58. 58. De Wals P, Erickson L. Economic analysis of the 1992–1993 mass immunization campaign against serogroup C meningococcal disease in Quebec. Vaccine. 2002 Jun 21;20(21–22):2840–4. pmid:12102036
  59. 59. Hepkema H, Pouwels KB, van der Ende A, Westra TA, Postma MJ. Meningococcal serogroup A, C, W₁₃₅ and Y conjugated vaccine: a cost-effectiveness analysis in the Netherlands. PLoS One. 2013 May 31;8(5):e65036. pmid:23741448
  60. 60. Trotter CL, Edmunds WJ. Reassessing the cost-effectiveness of meningococcal serogroup C conjugate (MCC) vaccines using a transmission dynamic model. Med Decis Making. 2006 Jan-Feb;26(1):38–47 pmid:16495199
  61. 61. Atkins KE, Shim E, Carroll S, Quilici S, Galvani AP. The cost-effectiveness of pentavalent rotavirus vaccination in England and Wales. Vaccine. 2012 Nov 6;30(48):6766–76 pmid:23000223
  62. 62. Bakir M, Standaert B, Turel O, Bilge ZE, Postma M. Estimating and comparing the clinical and economic impact of paediatric rotavirus vaccination in Turkey using a simple versus an advanced model. Vaccine. 2013 Jan 30;31(6):979–86 pmid:23219433
  63. 63. Bruijning-Verhagen P, Mangen MJ, Felderhof M, Hartwig NG, van Houten M, Winkel L, et al. Targeted rotavirus vaccination of high-risk infants; a low cost and highly cost-effective alternative to universal vaccination. BMC Med. 2013 Apr 26;11:112. pmid:23622110
  64. 64. Jit M, Bilcke J, Mangen MJ, Salo H, Melliez H, Edmunds WJ, et al. The cost-effectiveness of rotavirus vaccination: Comparative analyses for five European countries and transferability in Europe. Vaccine. 2009 Oct 19;27(44):6121–8. pmid:19715781
  65. 65. Newall AT, Dehollain JP, Creighton P, Beutels P, Wood JG. Understanding the cost-effectiveness of influenza vaccination in children: methodological choices and seasonal variability. Pharmacoeconomics. 2013 Aug;31(8):693–702. (Newall (b)) pmid:23645539
  66. 66. Mangen MJ, van Duynhoven YT, Vennema H, van Pelt W, Havelaar AH, de Melker HE. Is it cost-effective to introduce rotavirus vaccination in the Dutch national immunization program? Vaccine. 2010 Mar 19;28(14):2624–35. pmid:20109593
  67. 67. Rozenbaum MH, Mangen MJ, Giaquinto C, Wilschut JC, Hal E, Postma MJ. Consensus Group on Dutch Rotavirus Vaccination (CoRoVa-Group). Cost-effectiveness of rotavirus vaccination in the Netherlands; the results of a consensus model. BMC Public Health. 2011 Jun 10;11:462. pmid:21663620
  68. 68. Tu HA, Rozenbaum MH, de Boer PT, Noort AC, Postma MJ. An update of "Cost-effectiveness of rotavirus vaccination in the Netherlands: the results of a Consensus Rotavirus Vaccine model". BMC Infect Dis. 2013 Jan 30;13:54 pmid:23363553
  69. 69. Clements KM, Chancellor J, Nichol K, Delong K, Thompson D. Cost-effectiveness of a recommendation of universal mass vaccination for seasonal influenza in the United States. Value Health. 2011 Sep-Oct;14(6):800–11. pmid:21914499
  70. 70. Pitman RJ, Nagy LD, Sculpher MJ. Cost-effectiveness of childhood influenza vaccination in England and Wales: Results from a dynamic transmission model. Vaccine. 2013 Jan 30;31(6):927–42. pmid:23246550
  71. 71. Laxminarayan R, Chow J, Shahid-Salles SA. Intervention Cost-Effectiveness: Overview of Main Messages. In: Jamison DT, Breman JG, Measham AR, et al., editors. Disease Control Priorities in Developing Countries. 2nd edition. Washington (DC): World Bank; 2006. Chapter 2. Available from: http://www.ncbi.nlm.nih.gov/books/NBK11784/ (Last accessed 1/2016)
  72. 72. Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB, Greenberg D, et al. Bias in published cost effectiveness studies: systematic review. BMJ. 2006;332(7543):699–703. pmid:16495332
  73. 73. van de Vooren K, Duranti S, Curto A, Garattini L. Cost effectiveness of the new pneumococcal vaccines: a systematic review of European studies. PharmacoEconomics.2014;32(1):29–45. pmid:24288207
  74. 74. Baker CB, Johnsrud MT, Crismon ML, Rosenheck RA, Woods SW. Quantitative analysis of sponsorship bias in economic studies of antidepressants. Br J Psychiatry.2003;183:498–506. pmid:14645020
  75. 75. Miners AH, Garau M, Fidan D, Fischer AJ. Comparing estimates of cost effectiveness submitted to the National Institute for Clinical Excellence (NICE) by different organisations: retrospective study. BMJ.2005;330(7482):65. pmid:15601681
  76. 76. Ray GT. Pneumococcal conjugate vaccine: review of cost-effectiveness studies in Australia, North America and Europe. Expert Review of Pharmacoeconomics & Outcomes research. 2008;8(4):373–393.
  77. 77. Souza CPR, Ribeiro JGL, Moraes JC, Berezin E, Canaveira Monteiro RSD, Presa J. Cost effectiveness analysis of 13-valent pneumococcal conjugate vaccine in prevention of pneumococcal disease within the SUS scenario. Economic evaluation of pneumococcal vaccine. J Bras Econ Saude. 2009; 1(1): 3–10
  78. 78. Walker DG, Hutubessy R, Beutels P. WHO Guide for standardisation of economic evaluations of immunization programmes. Vaccine. 2010;28(11):2356–2359. pmid:19567247
  79. 79. Shiroiwa T, Sung YK, Fukuda T, Lang HC, Bae SC, Tsutani K. International survey on willingness-to-pay (WTP) for one additional QALY gained: what is the threshold of cost effectiveness? Health Econ. 2010; 19(4):422–37. pmid:19382128
  80. 80. Ultsch B, Damm O, Beutels P, Bilcke J, Brüggenjürgen B, Gerber-Grote A, et al. Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a Eu.ropean Vaccine Economics Community. Pharmacoeconomics. 2015. [Epub ahead of print].
  81. 81. Clements J, Shin S, Ali M. New approaches to the assessment of vaccine herd protection in clinical trials. Lancet Infect. Dis 2011; 11: 482–87 pmid:21616458
  82. 82. Miller E, Andrews NJ, Waight PA, Slack M PE, George RC. Herd immunity and serotype replacement 4 years after seven-valent pneumococcal conjugate vaccination in England and Wales: an observational cohort study. Lancet Infect Dis 2011; 11: 760–68 pmid:21621466
  83. 83. Waight PA, Andrews NJ, Ladhani SN, Sheppard CL, Slack M PE, Miller E. Effect of the 13-valent pneumococcal conjugate vaccine on invasive pneumococcal disease in England and Wales 4 years after its introduction: an observational cohort study. Lancet Infect. Dis 2015; 15: 535–43 pmid:25801458
  84. 84. Moore MR, Link-Gelles R, Schaffner W, Lynfield R, Lexau C, Bennett NM, et al. Effect of use of 13-valent pneumococcal conjugate vaccine in children on invasive pneumococcal disease in children and adults in the USA: analysis of multisite, population-based surveillance. Lancet Infect. Dis. 2015; 15: 301–09. pmid:25656600