Vaccines to prevent COVID-19: A living systematic review with Trial Sequential Analysis and network meta-analysis of randomized clinical trials

Background COVID-19 is rapidly spreading causing extensive burdens across the world. Effective vaccines to prevent COVID-19 are urgently needed. Methods and findings Our objective was to assess the effectiveness and safety of COVID-19 vaccines through analyses of all currently available randomized clinical trials. We searched the databases CENTRAL, MEDLINE, Embase, and other sources from inception to June 17, 2021 for randomized clinical trials assessing vaccines for COVID-19. At least two independent reviewers screened studies, extracted data, and assessed risks of bias. We conducted meta-analyses, network meta-analyses, and Trial Sequential Analyses (TSA). Our primary outcomes included all-cause mortality, vaccine efficacy, and serious adverse events. We assessed the certainty of evidence with GRADE. We identified 46 trials; 35 trials randomizing 219 864 participants could be included in our analyses. Our meta-analyses showed that mRNA vaccines (efficacy, 95% [95% confidence interval (CI), 92% to 97%]; 71 514 participants; 3 trials; moderate certainty); inactivated vaccines (efficacy, 61% [95% CI, 52% to 68%]; 48 029 participants; 3 trials; moderate certainty); protein subunit vaccines (efficacy, 77% [95% CI, −5% to 95%]; 17 737 participants; 2 trials; low certainty); and viral vector vaccines (efficacy 68% [95% CI, 61% to 74%]; 71 401 participants; 5 trials; low certainty) prevented COVID-19. Viral vector vaccines decreased mortality (risk ratio, 0.25 [95% CI 0.09 to 0.67]; 67 563 participants; 3 trials, low certainty), but comparable data on inactivated, mRNA, and protein subunit vaccines were imprecise. None of the vaccines showed evidence of a difference on serious adverse events, but observational evidence suggested rare serious adverse events. All the vaccines increased the risk of non-serious adverse events. Conclusions The evidence suggests that all the included vaccines are effective in preventing COVID-19. The mRNA vaccines seem most effective in preventing COVID-19, but viral vector vaccines seem most effective in reducing mortality. Further trials and longer follow-up are necessary to provide better insight into the safety profile of these vaccines.


Introduction
Analysis (PRISMA) statement [9,10]. This review was carried out following recommendations outlined in the Cochrane Handbook of Systematic Reviews of Interventions [11]. More details on our methods can be found in our published protocol [8,12].

Search strategy and selection criteria
An experienced information specialist performed weekly literature searches. For details regarding databases, searches, and screening methods see our published protocol [8] or S1 File.
We searched for and included randomized clinical trials, irrespective of publication status, year, and language. We included trials with any participant irrespective of prior exposure, age, sex, comorbidities, immune status, and risk group [8].
We included any vaccine aiming to prevent COVID-19 irrespective of dose and duration of administration. We included randomized clinical trials with any control group, i.e. vaccine versus placebo, 'active placebo', no intervention, another vaccine aiming at preventing COVID-19, or any other 'active' comparator [8].

Data analysis
Two authors independently screened and extracted data. Our primary outcomes were allcause mortality; vaccine efficacy defined by either preventing COVID-19 symptoms plus positive polymerase chain rection (PCR) test, preventing severe COVID-19 symptoms plus positive PCR test, or preventing positive PCR test only; and serious adverse events [10]. Our secondary outcomes were health-related quality of life and adverse events not considered serious [8]. We used the trial results reported at maximum follow-up for all outcomes. We used intention-to-treat data if provided by the trialists [8]. Several exploratory outcomes were predefined in our protocol and will be presented in a subsequent publication [8].
Risk of bias assessment. Our risk of bias assessment was based on the Cochrane risk of bias tool version 2 (RoB 2) [13].
Association measures. We calculated risk ratios (RRs) with 95% confidence interval (CI). In meta-analysis, we also calculated the Trial Sequential Analysis-adjusted CIs. We used forest plots to illustrate summary effect sizes of the comparative effectiveness among interventions.
Statistical synthesis. We undertook our analysis according to the Cochrane Handbook of Systematic Reviews of Interventions [11], Keus and colleagues [14], and our eight-step assessment [15]. We used Stata version 16.1 to analyse data using the metan command for metaanalysis and network suite of commands for network meta-analysis [16][17][18]. We assessed our intervention effects with both a random-effects (RE) meta-analysis (DerSimonian and Laird) [19] and a fixed-effect (FE) meta-analysis (Mantel-Haenszel) [20] for each preventive comparison. We reported the more conservative point estimate of the two [8,15]. We assessed three primary outcomes and two secondary outcomes. We, therefore, considered a p value of 0.0167 or less as the threshold for statistical significance to adjust for multiplicity [8,15].
We performed Trial Sequential Analysis on all outcomes, to calculate the diversity-adjusted required information size (DARIS) and the cumulative Z-curve's breach of relevant trial sequential monitoring boundaries [21][22][23].
We performed a network meta-analysis as described in S1 File and our protocol [8].
We assessed heterogeneity through visual inspection of forest plots, using the I 2 statistic [11,24,25], and by estimating the between-study variance using the DerSimonian and Laird method [26,27]. We investigated heterogeneity through subgroup analyses.
Additional analyses. We performed subgroup analysis assessing different types of vaccines; trials at low compared with at high risk of bias; and trials without against with vested interests. To assess the potential impact of missing data, we performed sensitivity analyses using 'best-worst' and 'worst-best' analysis [8,15].
Summary of findings and assessment of certainty. We created summary of findings tables including each of the prespecified outcomes (all-cause mortality, vaccine efficacy, serious adverse events, health-related quality of life, and non-serious adverse events) (Tables 1-4). We used the five Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) considerations (bias risk of the trials, consistency of effect, imprecision, indirectness, and publication bias) and CINeMA to assess confidence in the meta-analysis and network meta-analysis findings, respectively [28][29][30][31]. We assessed imprecision using Trial Sequential Analysis [8,32,33]. , of which 35 trials [34, 36-42, 44-50, 53-59, 61-65] randomizing 219 864 participants provided data for our predefined meta-analyses. See PRISMA flowchart (Fig 1) for details regarding the literature search and the selection of trials.

All-cause mortality
Inactivated vaccines. Six trials assessing inactivated vaccines reported on all-cause mortality. Meta-analysis (FE) showed that these vaccines versus controls may result in a large reduction of all-cause mortality, but the confidence interval was compatible with no effect (risk ratio (RR), 0.48 [95% CI 0.12 to 1.97]; p = 0.31; I 2 = 0.0%; 53 399 participants; very low certainty; Fig 2). mRNA vaccines. Five trials assessing mRNA vaccines reported on all-cause mortality. Meta-analysis (RE) showed that these vaccines versus placebo may reduce all-cause mortality, but the confidence interval was compatible with no effect (RR, 0.63 [95% CI 0.21 to 1.84]; p = 0.39; I 2 = 0%; 75 926 participants; low certainty; Fig 2).
Protein-subunit vaccines. Four trials assessing protein-subunit vaccines reported on allcause mortality. Meta-analysis (FE) showed that these vaccines versus controls may result in a reduction of all-cause mortality, but the confidence interval was compatible with no effect (RR, 0.46 [95% CI 0.09 to 2.36]; p = 0.35; I 2 = 0.0%; 15 634 participants; very low certainty; Fig  2). Viral vector vaccines. Three trials assessing viral vector vaccines reported on all-cause mortality. Meta-analysis (FE) showed that these vaccines versus controls may result in a large reduction of all-cause mortality (RR, 0.25 [95% CI 0.09 to 0.67]; p = 0.01; I 2 = 0.0%; 67 563 participants; low certainty; Fig 2).
Trial sequential analysis and sensitivity analysis on mortality. Trial Sequential Analyses showed that we did not have enough information to confirm that any of the vaccines reduced mortality with 20% or more. The sensitivity analyses showed that incomplete outcome data bias had the potential to influence the results of all the vaccines (S1 File).
Network meta-analysis on mortality. Network meta-analysis on mortality included 7 trials randomizing 168 701 participants comparing placebo, inactivated, mRNA, protein-subunit, and viral vector vaccines. There was no evidence of violation of the transitivity assumption for age and sex ( Table 5). The network plot showed a star network without closed loops (S1 File).
All the individual vaccines except NVX-CoV2373-Novavax and Gam-COVID-Vac-Sputnik-V may decrease mortality, but their confidence intervals were compatible with no effect: Based on the surface under the cumulative ranking curves, the Ad26.COV2.S-Janssen vaccine had the greatest likelihood of being the most effective vaccine in reducing mortality (P- � The basis for the assumed risk (e.g., the median control group risk across studies) is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the comparison group and the relative effect of the intervention (and its 95% CI). CI: Confidence interval. Pc: Proportion in control group with outcome. RR: Risk ratio. DARIS: Diversity-adjusted required information size. NA: Not applicable. NR: Not reported. GRADE Working Group grades of evidence.
High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate.
Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate.

Very low quality:
We are very uncertain about the estimate.
Analyses of all vaccines. When we analyzed all eighteen vaccines that reported on allcause mortality, meta-analysis (RE) suggested that the vaccines versus placebo result in a large reduction of all-cause mortality (RR, 0.41 [95% CI 0.22 to 0.77]; p = 0.0049; I 2 = 0.0%; 212 482 participants; low certainty; S1 File). TSA, however, showed that we did not have enough data to confirm that the vaccines reduced mortality with 20% or more (S1 File).
Protein-subunit vaccines. Two trials assessing protein-subunit vaccines reported vaccine efficacy on symptomatic COVID-19 participants. Meta-analysis (RE) showed that protein-subunit vaccines versus placebo may result in a large reduction of symptomatic COVID-19 participants, but the confidence interval was compatible with no effect (efficacy, 77% [95% CI −5%

PLOS ONE
Vaccines to prevent COVID-19 to 95%]; p = 0.06; I 2 = 92.6%; 17 737 participants; low certainty; Fig 4). Visual inspection of the forest plot and I 2 suggested substantial evidence of heterogeneity. Viral vector vaccines. Five trials assessed viral vector reported vaccine efficacy on symptomatic COVID-19 participants. Meta-analysis (RE) showed that viral vector vaccines versus placebo likely result in a large reduction of symptomatic COVID-19 participants (efficacy,  4). Visual inspection of the forest plot, I 2 , and estimation of between-study variance (τ 2 = 0.38) indicated substantial heterogeneity. Subgroup analyses of the different vaccines (favoring the Gam-COVID-Vac-Sputnik-V vaccine) and regarding vested interests (favoring without vested interests) showed evidence of differences (p<0.05) (S1 File).
Network meta-analysis on prevention of symptomatic COVID-19 participants with positive PCR test. Network meta-analysis on vaccine efficacy included eleven trials randomizing 205 916 participants comparing placebo/control with inactivated, mRNA, protein-subunit, or viral vector vaccines. There was no evidence of violation of the transitivity assumption for age and sex (Table 5).
Prevention of severe COVID-19 symptoms with positive PCR test or prevention of positive PCR test only. Vaccine efficacies on prevention of asymptomatic and severe disease are presented in the supplementary material (S1 File).
Serious adverse events. Inactivated vaccines. Seven trials assessing inactivated vaccines reported on serious adverse events. Meta-analysis (RE) showed that inactivated vaccines versus placebo may decrease serious adverse events slightly, but the confidence interval was also compatible with no effect (RR, 0.84 [95% CI 0.68 to 1.06]; p = 0.15; I 2 = 0.0%; 53 839 participants; low certainty; S1 File).
mRNA vaccines. Five trials assessing mRNA vaccines reported on serious adverse events. Meta-analysis (RE) showed that mRNA vaccines versus placebo may increase serious adverse events slightly, but the confidence interval was also compatible with no effect (RR, 1.10 [95% CI 0.91 to 1.33]; p = 0.31; I 2 = 0.0%; 75 926 participants; low certainty; S1 File).
Viral vector vaccines. Four trials assessing viral vector reported on serious adverse events. Meta-analysis (FE) showed that viral vector vaccines versus control may decrease serious adverse events, but the confidence interval was also compatible with no effect (RR, 0.82 [95% CI 0.64 to 1.05]; p = 0.12; I 2 = 0.0%; 68 640 participants; low certainty; S1 File).
Health-related quality of life. None of the included trials assessed health-related quality of life. Non-serious adverse events. Inactivated vaccines. Eleven trials assessing inactivated vaccines reported on adverse events considered non-serious. Meta-analysis (RE) showed that inactivated vaccines versus placebo have little or no effect on adverse events not considered serious (RR, 1 Viral vector vaccines. Three trials assessing viral vector vaccines reported on adverse events considered non-serious. Meta-analysis (RE) showed that viral vector vaccines versus control may increase of adverse events not considered serious, but the confidence interval was also compatible with no effect (RR, 1.21 [95% CI 1.00 to 1.85]; p = 0.05; I 2 = 80.9%; 8 909 participants; very low certainty; S1 File).
Our pairwise meta-analysis showed that the viral vector vaccines decreased mortality. Network meta-analysis suggested that the viral vector vaccine Ad26.COV2.S-Janssen was most effective in reducing mortality. All COVID-19 vaccines were more likely to reduce mortality than placebo or control, but longer follow up and more participants are needed to confirm this with a higher degree of certainty.
All vaccines, except inactivated vaccines, increased the risk of having a non-serious adverse event. We found no difference on serious adverse events, for any of the included vaccines (Ad26.COV2.S-Janssen, BNT162b2-Corminarty, CoronaVac, Gam-COVID-Vac-Sputnik-V, mRNA-1273-Spikevax, NVX-CoV2373-Novavax, and Vero Cell), but our Trial Sequential Analyses indicated that we did not have enough data to reject any differences.
Our results should been seen together with observational evidence suggesting rare serious adverse events associated with the vaccines [68]. The results of our review is supported by the first nationwide observational studies showing high vaccine efficacy of BNT162b2-Corminarty and of BNT162b2-Corminarty and ChAdOx1 nCoV-19-Vaxzevria against COVID-19 hospital admissions [69,70].
We included 35 trials randomizing a total of 219 864 participants that contributed to our analyses. Of these trials, ten were at overall low risk of bias, seventeen at overall some concerns, and eight trials at overall high risk of bias. The certainty of evidence according to GRADE ranged from very low to moderate for the pairwise meta-analyses and the CINeMA were very low for all our network meta-analyses.
Most trials compared vaccines versus placebo. A few trials compared the vaccine versus a control vaccine not active towards SARS-CoV2 (MenACWY) [37,40,45]. Only trials assessing inactivated, mRNA, protein-subunit, and recombinant viral vector vaccines contributed to the assessment of our primary outcomes.
We found very similar results for efficacy of vaccine prevention for the two mRNA-based vaccines. This is not surprising as the construction differs only in details of the formulation of nanoparticles used for protection of the mRNA. In contrast, marked heterogeneity was observed regarding the three adenovector-based vaccines. Again, this is perhaps not surprising as the adenoviral backbones are different: ChAdOx1 nCoV-19, Ad26, or Ad5. Preclinical studies in mice and non-human primates have pointed to significant variation in the capacity of different adenoviral backbones to induce adaptive immunity [71][72][73]. Another cause could be the fact that the vaccines were not compared head-to-head in the same populations under similar conditions.
The emergence of SARS-CoV-2 variants of concern like in the UK (alpha/B.  [74][75][76][77]. This should be taken into consideration when trying to compare vaccine efficacies derived from studies carried out at different time-points and/or locations. The efficacy of existing vaccines against emerging variants needs to be closely monitored and the adaptability of the vaccines to potential immunity evading mutations needs to be assessed. The trials in our systematic review did not include pregnant participants. Pregnant women constitute a vulnerable group for COVID-19 [78,79]. Systematic reviews of observational studies showed that compared to pregnant women without COVID-19, pregnant women with this infection may require more frequent admissions to intensive care units (odds ratio (OR),  [78,79]. For these reasons, many countries like the USA [80][81][82] and Israel [83] are presently assessing vaccine effectiveness in pregnant women. The World Health Organization (WHO) recommended mRNA vaccines only to pregnant women at highest risk after consultation with their physician [84], and trials are ongoing [85,86].
Our review has several strengths. Our methodology was described in detail in a protocol published before the literature searches were initiated [8]. We systematically assessed the risks of systematic errors through bias risk assessments, we conducted Trial Sequential Analyses to control random errors and guide our GRADE assessments of the domain 'imprecision', and we adjusted our thresholds for statistical significance to control the risks of random errors [15]. Our review is to our knowledge the first review to perform network meta-analysis comparing the different COVID-19 vaccines. It is also more updated than previous systematic reviews [7,87,88]. Our living systematic review will be continually updated to incorporate relevant new evidence as it becomes available [8,89,90].
Our review also has limitations. Most of the larger phase III trials presented only interim analyses. The follow-up ranged from only 35 to 92 days. A longer follow-up would most likely yield more events and lead to more robust conclusions. Moreover, the trials did not report vaccine efficacy similarly. They used different cut-offs, follow-up durations, and definitions of vaccine efficacy. Especially the efficacy of preventing severe COVID-19 was not equally reported for the different vaccines. Furthermore, most trials only reported efficacy in the group of participants followed more than two weeks after the last vaccination and only reported adverse events for a confined time. Accordingly, we may look at biased results. Moreover, reporting of adverse events was variably and insufficient in several trials.
The trials were conducted in different countries. The participants of the different trials could therefore have been exposed to different variants of SARS-CoV2. We were unable to fully extract comparable data from the ChAdOx1 nCoV-19-Vaxzevria trials due to the lack relevant data from the individual trials, as the trials reported their results in a meta-analysis [37,45,52,91].

Conclusion
Our systematic review shows that the inactivated vaccines, mRNA vaccines, protein-subunit vaccines, and viral vector vaccines are effective in preventing infection with SARS-CoV2. Current evidence shows that mRNA vaccines seem most effective in preventing COVID-19, but viral vector vaccines seem most effective in reducing mortality. The inactivated vaccine (Coro-naVac), mRNA vaccines (mRNA-1273-Spikevax and BNT162b2-Corminarty), protein-subunit vaccine (NVX-CoV2373-Novavax), and viral vector vaccines (Gam-COVID-Vac-Sputnik-V, ChAdOx1 nCoV-19-Vaxzevria, and Ad26.COV2.S-Janssen) do not seem to increase the risk of serious adverse events according to published randomized trials. Most of the vaccines increase the risk of adverse events not considered serious. Our results may help guide authorities when deciding what vaccines to incorporate into their vaccine programs weighing our results against the potential rare events seen in observational studies.
Supporting information S1 File.