Ethics statement

This work was approved under the UNSW Sydney Human Research Ethics Committee (approval HC200242). Individual participant informed consent was not required for this study as it was an analysis of previously published data. There was no direct interaction with human participants, and no new data was collected from human participants.

Vaccine protection against progression from SARS-CoV-2 infection to severe COVID-19

In order to find evidence of vaccine protection against disease progression we searched PubMed for all studies published in English from 01 Dec 2020 to 23 Jan 2025 which assessed outcomes in a population who were already infected with SARS-CoV-2 and where rates of progression to severe COVID-19 were compared between vaccinated and unvaccinated people. We excluded studies that did not compare outcomes in vaccinated versus unvaccinated populations (e.g., studies were excluded if they only compared boosted individuals to those with a primary course, or compared one vaccine to another), and studies which recruited from emergency department presentations, since it is likely that individuals presenting at emergency departments may have already been progressing to more severe symptoms (see Supplementary materials in S1 Appendix for full search strategy).

For each of the clinical studies, we extracted data on administered vaccine, number of doses received, infecting SARS-CoV-2 variant, rates of prior infection, the clinical outcome used to define progression to severe outcomes, and the statistical risk measure (e.g. aOR, aHR, etc.) used to compare the risk of progression between the vaccinated and unvaccinated groups. Where the effect of vaccination on disease progression was disaggregated by the vaccine received, this data was extracted. Where the effect of vaccination was not disaggregated by vaccine in a paper, we firstly treated different mRNA vaccines (i.e., BNT162b2 and mRNA-1273) as equivalent, and whole inactivated vaccines (i.e., CoronaVac and BBIBP-CorV) as equivalent, then we recorded the most common vaccine type received by participants (or excluded the study if the most common vaccine type was received by less than 50% of the vaccinated population). Each group of vaccinated individuals included in the study was categorised as having received a particular vaccine regimen which was defined as a combination of vaccine type and the number of doses received by the individuals. Where vaccine protection was reported against single SARS-CoV-2 variants, this data was extracted. If protection against a single variant was not available, combined variants were used, noting that any studies that combined vaccine effect across pre-Omicron and Omicron variants were excluded (according to our exclusion criteria). Further details are provided in the Supplementary materials in S1 Appendix.

Neutralising antibodies

To obtain data on neutralising antibody titres after vaccination for a variety of vaccine regimens we used the Stanford University Coronavirus antiviral and resistance database [12]. This database contains post-vaccination in vitro neutralisation titres from studies that were comparing neutralisation between variants using a variety of assays. We searched the database for neutralisation titres in people with no prior infection against the ancestral strain, or variants present in the clinical studies (i.e., Alpha, Beta, Delta, Omicron BA.1, Omicron BA.2, Omicron BA.4, and Omicron BA.5), and for each of the vaccines present in the clinical studies (see Supplementary materials in S1 Appendix), resulting in data from 301 underlying immunological studies (2,156 separate titre estimates). We performed a meta-analysis of the neutralising antibody data from this database using a linear mixed-effects model (using the lmer function from the lme4 package in R) [13], with random intercepts for study (see Supplementary materials in S1 Appendix). This produced global estimates of the neutralisation titre after vaccination with different vaccine regimens, against different variants and at different times after vaccination (for example, we obtained a global estimate of the neutralisation titre to Omicron BA.1 one month after 3 doses of BNT162b2).

Linking neutralising antibody levels and disease progression data

The neutralising antibody titres obtained from the meta-analysis above were then matched to the clinical data on vaccine protection from progression to severe COVID-19. Matching was performed based on: vaccine regimen, SARS-CoV-2 variant, and time since last dose where available. Where the clinical data that was reported could be matched to more than one category of neutralisation data, we matched it to the geometric mean of the relevant neutralisation titre categories (e.g., if the clinical data reported outcomes for a population who were infected with either Omicron BA.1 or Omicron BA.2, then the geometric mean of the GMTs for the relevant variants was used). The full details of matching of vaccine effectiveness studies to results of the neutralisation meta-analysis are described in the Supplementary materials. Once matching was completed, we analysed the relationship between vaccine-induced neutralising antibody titre and protection from progression to severe disease (see Statistical methods for details).

We have previously analysed the association between vaccine-induced neutralising antibody titre and protection from mild and severe COVID-19 [3]. Here we have used these relationships to model protection against progression from mild to severe infection as a validation step. Similarly, we have previously analysed studies of passive antibody administration as a treatment for COVID-19, early after symptom onset, and found a relationship between the administered dose (neutralising antibody titres) of passively administered antibodies and protection against progression to severe outcomes [14]. To allow comparison with these prior works, we normalised the estimated neutralising antibody titres used in our study (from the Stanford database) [12] to a scale of “fold-convalescent” also using data obtained from the Stanford database [12]. We extracted (from the Stanford database) the aggregate geometric mean neutralisation titre (against ancestral SARS-CoV-2) in convalescent individuals at both 1 month and 2−6 months post infection with ancestral SARS-CoV-2. The geometric mean of these two time periods (indicated with a dashed line in Fig E in S1 Appendix) was then used to normalise other neutralisation titre estimate to a fold-convalescent scale (i.e., by dividing a GMT by this convalescent titre). This fold-convalescent scaling approach was chosen in order to best align with the scaling approach we employed previously in Khoury and colleagues [3].

Statistical methods

Protection from progression was reported in the primary clinical studies using a variety of statistical risk measures (e.g., relative risk (RR), odds ratio, hazard ratio), which all converge when events (progression to severe outcomes in this study) are rare [15]. These estimates of protection were aggregated using an inverse variance weighted multilevel linear mixed-effects model (using the rma.mv function with maximum likelihood estimation from the metafor package in R) [16]. Vaccine protection was expressed as the natural log-transformed RR, and variance in these protection estimates was calculated from the 95% confidence intervals published in the underlying studies. For the random-effects structure, we used random intercepts at the study level (Level 3) and at the effect-size-within-study level (nested at Level 2), reflecting the nonindependence of multiple effect sizes per study. This model was compared to the simpler model, with between-study clustering only, using likelihood ratio tests (with restricted maximum likelihood fitting, see Supplementary materials in S1 Appendix). More complex random effects structures were not explored. Candidate covariates were identified by clinical relevance and data availability. We evaluated all possible combinations of the candidate covariates (see Supplementary materials in S1 Appendix), excluding interactions between them, and compared the models using AICc (Akaike information criterion with correction for small sample sizes) from the MuMIn package in R [17].

The matched neutralisation titres were then used in a univariate meta-regression by including log₁₀(GMT) as the only fixed-effect covariate in a model with the same multi-level random effects structure described previously. To test the robustness of this relationship we investigated whether this relationship was sensitive to the inclusion of additional covariates in multivariate analyses. Statistical significance of covariates was calculated using likelihood ratio tests. Ninety-five% Confidence intervals for all meta-regression model outputs were calculated as 1.96 times the standard error, and the 95% confidence regions on regression lines were calculated using the predict() function in R.

The reported estimates of vaccine protection against progression, from the underlying clinical studies, were visualised using Forest plots. Funnel plots were used to assess potential publication bias.

All correlations were assessed using a Spearman correlation. All statistical analyses were performed in R version 4.2.1, with a threshold for statistical significance of 0.05.

Vaccine protection against progression from SARS-CoV-2 infection to severe COVID-19

We first aimed to quantify vaccine protection against progression to severe COVID-19. Thus, we searched PubMed for studies that contained data on the risk of progressing from SARS-CoV-2 infection to severe COVID-19, comparing vaccinated versus unvaccinated individuals so that we could estimate the protection afforded by vaccination (see Supplementary materials in S1 Appendix for full search strategy). Our search identified 2,280 unique records of which 25 [18–42] met our inclusion criteria (Supplementary materials, Tables A-B, Fig A in S1 Appendix). We extracted data on administered vaccines (including type, number of doses, and timing of last dose where available), SARS-CoV-2 variant (typed at infection or reported to be the dominant circulating variant at the time of the study), and reported results relating to vaccine protection against disease progression. This was done using the most disaggregated data (based on vaccine, number of doses, variant and time since last dose) available from each study, leading to a total of 137 estimates of vaccine protection against progression to severe outcomes being extracted from the 25 studies, with individual studies contributing anywhere from 1 [18–25] to 40 [36] estimates of vaccine protection. The studies reported a variety of different clinical endpoints, with some studies reporting on a single endpoint, while others reported results separately for multiple clinical endpoints. The most commonly available endpoint was hospital admission (20/25 studies, 130/137 observations), so our main analysis used this endpoint where it was available and used ICU admission in the remaining five studies (7/137 observations). The proportion of individuals in each study who had a history of prior infection was 0% in 5/25 studies, <10% in 7/25 studies, 10%–20% in 1/25 studies and was not reported in 12/25 studies (Table B in S1 Appendix). Importantly, our analysis always focussed on protection in vaccinated individuals (who may have received 1 dose of Ad26.COV2.S, or 2, 3 or 4 doses of all other vaccines) compared to unvaccinated individuals in the same study. The vaccine protection from progression reported in all underlying studies has been summarised in Forest plots (Figs B-C in S1 Appendix). Across all included data (pooling different vaccines, variants, number of doses, time since last dose, etc) the observed vaccine protection from progression to severe COVID-19 was 73% (RR 0.27, 95% CI [0.21,0.33], p < 0.001) (Table G in S1 Appendix).

Assessing predictors of vaccine protection against progression

We expected vaccine protection against progression may differ by factors such as vaccine regimen, infecting variant, and days post-vaccination. Therefore, to account for these potential factors, as well as other potential confounders, we considered all candidate covariates (see Supplementary materials in S1 Appendix) and compared models containing all possible combinations of these using AICc. The model with the lowest AICc included vaccine regimen, variant, time since last dose and age group (<40 years or 40−79 years) as covariates (Forest plot Fig 2, Tables H-I in S1 Appendix). We observed from this multivariate model that vaccine protection against progression was higher after a third or fourth dose (when compared to unvaccinated individuals) than it was after a primary course only (e.g., 2 dose BNT162b2 versus unvaccinated (RR 0.33, 95% CI [0.22,0.49]), 3 dose BNT162b2 versus unvaccinated (RR 0.18, 95% CI [0.11,0.28])). Protection against progression was also higher against pre-Omicron variants than Omicron variants (RR 0.59, 95% CI [0.51,0.68]), and higher in younger cohorts (RR 0.66, 95% CI [0.48,0.91]) for mean/median age <40 yrs compared to 40−79 years (Fig 2, Table I and Fig D in S1 Appendix). Together, this provides evidence that vaccines are effective at preventing COVID-19 progressing to severe COVID-19 and there is some evidence that this differs by age, SARS-CoV-2 variant, time since last dose and vaccine regimen.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Predictors of vaccine protection against progression to severe outcomes.

Points and lines denote estimates and their 95% confidence intervals for the adjusted relative risk (aRR) of the covariates in the model of vaccine protection. The dotted line is a relative risk of 1 (i.e., equivalent to risk in the reference group). Values lower than 1 (to the left of the dotted line) occur where risk in the stated group is lower than the reference group, while those greater than 1 have higher risk than the reference group. For clarity we also report the corresponding adjusted Vaccine effectiveness (aVE) for each vaccine regimen. aVE is calculated as 100 x (1 – aRR) and is reported here against the Omicron variant in the 40–79 years age group as at day of most recent dose (Table I, Fig D in S1 Appendix). N/A, Not applicable.

https://doi.org/10.1371/journal.pmed.1005230.g002

Relationship between neutralisation titres and protection against progression

We next aimed to determine if vaccine protection against progression to severe COVID-19 is associated with neutralising antibody titres after vaccination. Since neutralising antibody titres were not measured in the clinical studies themselves, we linked each estimate of vaccine protection from progression in the clinical studies to an estimate of the geometric mean neutralising antibody titre in a matched population of vaccinated individuals [12] (see Table 1, Tables C–F and Fig E in S1 Appendix). Matching was based on vaccine regimen (vaccine and number of doses), variant, and time since last dose. This then allowed us to test whether neutralising antibody titres were associated with vaccine protection from progression to severe disease.

We found a significant correlation between matched neutralisation titres and protection from disease progression (Spearman r = 0.41, p < 0.001, Fig 3). Furthermore, in a univariate meta-regression analysis, neutralisation titres were found to be a significant predictor of vaccine protection against progression (RR 0.57 per 10-fold increase in GMT, 95% CI [0.51,0.64], p < 0.001), (Fig 3, Table J in S1 Appendix).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Relationship between (geometric mean) neutralisation titre and vaccine protection against progression to severe outcomes.

Here we show on the x axis, the geometric mean neutralisation titre (GMT) that was matched to each reported result of protection (vaccine effectiveness) against progression to severe diseases (shown on y axis). Protection against progression is calculated as 100 x (1 – aRR), where aRR is adjusted relative risk. Shapes represent vaccination levels (circles for primary course, triangles for groups with 3 or more doses). Colours show vaccine type, and larger point sizes indicate protection estimates with narrower confidence intervals. The black line is the fitted model from our univariate meta-regression model of neutralisation titre and protection (i.e., no additional covariates for variants or ages). The grey shaded region shows the 95% CI.

https://doi.org/10.1371/journal.pmed.1005230.g003

To test the robustness of this association, we considered whether the association remained after adjusting for additional covariates, including vaccine regimen, variant, time since last dose and age group. Neutralising antibodies remained significantly associated with vaccine protection regardless of whether any one of these additional covariates was added to the model (Table K in S1 Appendix). Interestingly, we also noted that vaccine regimen and time since last dose were no longer significant covariates once neutralising antibody titres were included in the model (LRT p = 0.075 and p = 0.92, respectively, Table K in S1 Appendix). This suggests that differences in vaccine protection between regimens (and time since last dose) were largely explained by neutralising antibody titres in this meta-analysis. Age group (<40 versus ≥40 yrs) and variant (Omicron versus pre-Omicron) remained significant covariates along with neutralising antibody titres (LRT, p = 0.016 and p = 0.011 respectively, Table K and Fig F in S1 Appendix). Even so, in these models, as well as in a model containing all three predictors (age group, variant, and neutralising antibody titre) neutralising antibodies remained significantly associated with protection against progression (Supplementary materials and Table L in S1 Appendix). Together, suggesting that the association between neutralising antibodies titres and protection against progression observed here is robust to the inclusion of potential covariates.

To test the robustness of the association between neutralising antibody titres and protection against progression, we also performed a series of additional sensitivity analyses (see supplementary results, Tables M-N and Figs G-I in S1 Appendix). These included repeating the analysis with COVID-19 related death as an outcome (Table M in S1 Appendix), and the influence of individual studies on the results (Figs G-H in S1 Appendix). We noted that four studies had rates of progression >10% and we conducted a sensitivity analysis excluding these four studies which showed consistent results (see supplementary results, Tables B, M in S1 Appendix). In order to assess whether the inclusion of studies with either unknown or high (>10%) rates of prior infection was impacting our results, we ran a sensitivity analysis excluding these studies, and found no changes to our conclusions (Table M in S1 Appendix). In all cases, neutralising antibody titres remained a significant predictor of protection from progression. Visual inspection of funnel plots showed some asymmetry (Fig I in S1 Appendix), suggesting the possibility of publication bias; however, it is likely that the asymmetry is at least partly due to study heterogeneity given the high between-study variability (Level 3 I² = 71.8%).

Together, our analyses suggest that neutralising antibodies are associated with protection from progression to severe outcomes, and that this relationship is robust despite study differences and after adjustment for potential confounders.

An external validation of the relationship between neutralising antibody titres and protection from progression

To assess whether the relationship between neutralisation and vaccine protection against progression found here is consistent with prior work, we compared it with the relationship predicted from an analysis of randomised controlled trials of COVID-19 vaccines (in March 2021) [3,43]. This relationship reported by Khoury and colleagues also used a meta-analysis and was not focussed on the outcome of progression to severe outcomes and had limited power due to the low event counts for severe outcomes. However, the study quantified the relationship between neutralisation titres and protection against both symptomatic and severe COVID-19 (Fig 1A and 1B respectively). In that study, at any given neutralisation titre, the protection against severe outcomes was higher than the protection against symptomatic infection. For any neutralisation titre, the (multiplicative) difference between these curves represents the model’s predicted vaccine protection against progression to severe disease (Fig 1C, Supplementary materials in S1 Appendix). After converting the neutralisation titres in the above meta-analysis to a fold-convalescent titre to make them comparable with Khoury and colleagues, we find that the relationship identified here between neutralisation titres and protection from progression to severe disease lies within the confidence interval of the predicted relationship derived in March 2021, based on comparing vaccine protection from mild versus severe COVID-19 from the vaccine RCTs (Fig 4A). This concordance highlights that two independent studies, using different data, from different underlying study types and different methods arrive at very similar relationships between neutralisation titres and protection from progression to severe COVID-19, providing some support for the estimated association.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison with previously determined relationships between antibody titre and protection.

(A) The red line shows the predicted relationship between vaccine protection against progression to severe outcomes and neutralisation titre (which was derived from the symptomatic and severe relationships predicted from an analysis of RCTs (randomised controlled trials) of COVID-19 vaccines in 2021 [3], see Supplementary materials in S1 Appendix). The black line shows the relationship between vaccine protection and neutralisation titres derived here (same as Fig 3, with the scale on the x-axis converted to fold-convalescent – see supplementary material). Shaded regions are 95% CIs. (B) Comparison of protection against progression to severe disease seen between vaccination and therapeutic antibody administration. The green line shows the previously determined relationship between neutralisation titres and protection from progression in those with COVID-19 who received therapeutically administered antibodies [14], while the black line shows the meta-regression model developed here (same as Panel A). The yellow shaded region shows the “protection gap” between the two associations, for the common range of neutralising antibodies. Notice that for any given level of neutralising antibody titres, protection against progression to severe disease is higher from vaccination than passively administered antibodies alone. Protection against progression is calculated as 100 x (1 – RR), where RR is relative risk.

https://doi.org/10.1371/journal.pmed.1005230.g004

Considering neutralising antibodies as mechanistic in protection against progression to severe outcomes

In the above, we observe an association between neutralising antibody titres and vaccine protection against progression to severe COVID-19. However, it remains to be determined whether neutralising antibodies are mechanistic in preventing progression to severe outcomes or merely a surrogate biomarker of overall immune response magnitude [44]. Studies on the therapeutic use of passively administered antibodies (both monoclonal antibodies and convalescent plasma) to prevent progression to severe COVID-19 [14] provide an opportunity to directly assess the mechanistic role of antibodies in preventing progression.

Previously, in a meta-analysis of passive antibody treatments, we have shown a dose response relationship between the administered dose of passive antibodies (normalised by in vitro neutralisation titre/IC50) and their effectiveness at preventing progression to severe outcomes [14]. We compared the protection against progression from vaccination (estimated above) and that observed after the therapeutic use of passive antibodies at similar neutralising antibody titres (see Supplementary materials in S1 Appendix) and found that protection against progression to severe disease is indeed lower in passive antibody administration than it is for vaccination for any given neutralising antibody titre (Fig 4B). This suggests there is a “protection gap” between the protection that can be achieved with passively administered antibodies during early infection and vaccine protection for a given vaccine-induced neutralising antibody titre. We quantified this protection gap as an approximately 63% lower risk (RR = 0.37, 95% CI [0.24, 0.55]) of progression to severe outcomes with active vaccination (average across neutralising antibody titres, see Supplementary materials in S1 Appendix), compared to passive antibodies. This quantity represents the average additional protection observed after active vaccination compared to after passive antibody administration for the range of neutralising antibody titres common to both passive antibody treatments and vaccination.

There are several fundamental differences between vaccination and passive antibody therapy (and especially monoclonal antibody therapy) that could explain this protection gap. Firstly, in vaccinated individuals, antibodies will be present prior to infection, while for those receiving passive antibody treatment, antibodies are generally administered 2–5 days after symptom onset [14]. Secondly, vaccination induces a more diverse immune response, including polyclonal antibodies, multiple antibody isotypes, T-cells and other cellular immune responses (all or some of which may provide additional protection not seen when administering passive antibody treatments). Finally, active vaccination produces memory responses that are capable of recall. Thus, the antibody titres measured prior to infection may not be reflective of the level of antibodies present during infection. These differences lead us to expect lower effectiveness of passive antibody treatments compared to vaccination.

A number of previous studies have shown that administering therapeutic antibodies earlier after COVID-19 symptom onset yields higher protection from progression to severe outcomes [14,45,46]. We previously estimated the relationship between time-of-administration of passive antibody (post-symptom onset) and protection from severe outcomes (See Fig S11 and Table S6 from Stadler et al. [14]). In that study, we estimated that if antibody administration had occurred on the day of symptom onset, then the risk of progression would be approximately 52% lower (RR 0.48, 95% CI [0.28, 0.82]) compared to the actual time of administration in the studies analysed (which occurred an average of 4.2 days after symptom onset). The level of increased protection between active vaccination and passive antibody administration (63% lower risk of progression in vaccination than passive antibody treatment) is very similar to the increased protection expected if antibodies were given at symptom onset. This suggests that most of the protection gap between passive antibody therapy and active vaccination (~77%, Fig J in S1 Appendix) might be explained by passive antibodies only being administered several days after symptom onset rather than being present since the time of infection (as is the case in active vaccination).