Genetic distance as an early warning signal during the original vaccine formulation period

To evaluate whether SARS-CoV-2 genetic distance can serve as an early warning signal for declining VE at population level, we first investigated both longitudinal genomic and epidemiological data separately. We analyzed 7,722 genomes from cases that we sequenced during the original vaccine formulation period (i.e., block 1) and examined weekly changes across three key aspects (Fig 2).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Relationship between spike gene amino acid distance to original vaccine formulation and time-varying vaccine effectiveness for case sequences included in the model calibration, April 2021 to September 2022.

Panels (A) and (B) display individual dots, where each dot represents a single sequenced case included in the model calibration. The y-axis indicates the number of amino acid substitutions in the spike gene from the case sequences relative to the original formulation of mRNA COVID-19 vaccines. (A) Comparison of sequences between unvaccinated (yellow) and recently vaccinated individuals (red; 14-90 days since the most recent dose). The red line represents the 3-week rolling average of (%) recently vaccinated individuals among sequenced infections over time; gaps in this line represent intervals with zero observations for this category. (B) Sequences colored by variants of concern assigned to each case sequence (Alpha, black; Delta, blue; Omicron BA.1, red; BA.2, orange; BA.4, purple; BA.5, magenta; XBB.1.5, yellow). Vertical dispersion of dots of the same color reflects within-variant genetic heterogeneity, while the presence of multiple colors in a single week indicates variant co-circulation. (C) Week-level vaccine effectiveness (VE) estimates with 95% credible interval from the time-varying VE model for the period 14-90 days since the most recent dose. Vertical grey boxes denote distinct emergence periods, while the dotted line indicates the introduction of the updated bivalent vaccine formulation.

https://doi.org/10.1371/journal.pcbi.1014329.g002

First, we investigated shifts in the proportion of vaccinated individuals among sequenced infections over time (Fig 2A). We defined vaccine breakthrough cases as individuals who had received at least one vaccine dose 14 or more days prior to a laboratory-confirmed SARS-CoV-2 infection. From mid-January to mid-February 2022, we observed a rapid rise in breakthrough cases among recently vaccinated individuals (i.e., 14–90 days post-last dose) during the Omicron BA.1 period, comprising roughly 20% of all sequences collected weekly (Fig 2A).

Second, we analyzed the weekly average spike gene amino acid distance between the sequenced cases and the vaccine formulation sequence (Fig 2B). During the original vaccine formulation period, we showed that the weekly average genetic distance between case sequences and the vaccine sequence ranged from 10 to 34 amino acid substitutions in the spike gene. During the Delta emergence period, the amino acid distance showed little change compared to earlier variants. However, during the Omicron BA.1 emergence period, we observed that average weekly amino acid distance increased from 10.9 in the first week of December 2021 to 33.3 in the first week of January 2022. The amino acid distance remained relatively stable during the BA.2 and BA.4/BA.5 periods. Within-group variations were also evident both between variants and temporally throughout their respective periods of circulation (S2 Fig). Our analysis of the full spike gene captured a broader range of variations (Fig 2), offering a more comprehensive view than receptor binding domain alone (S3A and S3B Fig). We also tracked temporal changes in nucleotide differences relative to the NCBI reference sequence (S3C Fig). Overall, we demonstrated high correlation between the spike gene amino acid differences used in our main analysis and two alternative measures: restricting the analysis to amino acid substitutions in the receptor binding domain alone (S3B Fig) or extending the analysis to more granular, nucleotide changes within the spike region (S3D Fig).

Third, we examined temporal changes in population-level VE (Fig 2C). Time-varying VE estimates for the period 14–90 days since the most recent dose gradually declined from 80% (95% credible interval (CrI): 74%, 85%) in April 2021 to around 60% spanning the 6-month period before BA.1 emergence. Following the emergence of BA.1, VE dropped sharply from 58% (95% CrI: 47%, 68%) in early December 2021 to 31% (95% CrI: 13%, 46%) by late January 2022. These low VE estimates persisted during the BA.2 and BA.4/BA.5 periods and could not be fully explained by bias from the depletion of susceptibles (S4 Fig). We also showed that after bivalent boosters were introduced in mid-September 2022, which reduced the spike amino acid distance between circulating variants and the vaccine formulation sequence, VE rebounded. In early October 2022, within three weeks of the bivalent booster introduction, mean VE increased by 12 percentage points to 40% (95% CrI: 6.22%, 63.19%), compared to 28% (95% CrI: -9.98%, 54.58%) in late August 2022.

Quantifying the relationship between amino acid distance and vaccine effectiveness

Based on the observed inverse relationship between the number of amino acid substitutions and VE over time (Fig 2), we hypothesized that population-level amino acid distance in the spike gene can serve as a proxy for vaccine protection. We therefore quantified their direct relationship using our meta-regression model and found that a mean distance of 10 amino acids in the spike gene corresponds to a 15.4% (95% CrI: -2.00%, 34.6%) decrease in predicted VE.

To validate the model’s ability to predict VE using weekly amino acid distance data, we generated one-month-out predictions and compared them to observed VE values (Fig 3). We found that the model closely followed the observed VE trends across all prediction months (mean absolute error (MAE) = 3.33) and successfully captured changes surrounding the bivalent vaccine update. All VE values during the eight-month prediction window—four months before and after the vaccine update—fell within the 95% prediction intervals (Fig 3B). We also showed that this model outperformed a reference model that had the same autoregressive structure but excluded amino acid distance information (Fig 3C); the baseline AR model yielded a higher MAE (MAE = 5.86) and a higher continuous ranked probability score (CRPS) throughout most of prediction months (S5 Fig). Additionally, the baseline model produced markedly wider prediction intervals, reflecting increased uncertainty in its prediction. These findings highlight the added value of incorporating genomic information in enhancing the model's accuracy in predicting future VE compared to a model relying solely on historical VE data.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison of model stages and validation of one-month-ahead vaccine effectiveness (VE) predictions.

(A) Comparison of first-stage time-varying VE estimates and corresponding 95% credible intervals (CrI) (i.e., Fig 2C) in green and “observed” VE estimates from the amino-acid-distance meta-regression model in purple. Vertical grey boxes denote variant emergence periods, and the dotted line indicates the bivalent vaccine introduction. (B - C) Validation of the observed VE (purple ribbons carried over from panel A) to (B) one-month-ahead predicted VE from the amino-acid-distance meta-regression model (red diamonds), which uses week-level amino acid distance as input, and (C) the baseline meta-regression model (orange diamonds), which does not incorporate amino acid distance information. Colored ribbons represent 95% CrI for observed (purple) and predicted (red and orange) values.

https://doi.org/10.1371/journal.pcbi.1014329.g003

Estimating changes in vaccine effectiveness for the XBB.1.5 vaccine formulation: simulating real-world scenarios without vaccination data

After calibrating and validating our model (Fig 3), we then sought to translate these findings into a real-world framework to inform changes in predicted vaccine protection as new variants emerged. To evaluate how population-level amino acid distance from the sequence of different COVID-19 vaccine formulations changed across multiple vaccine updates, we extended our model across the different formulation periods. Throughout vaccine update blocks, we utilized only measured weekly amino acid distance as the input to predict the percent reduction in VE; we present VE estimates based on a 3-week rolling average of amino acid distance to enhance visualization.

During the second vaccine (block 2), the increase in minimum amino acid distance to either formulation was more nuanced than in block 1. Block 2 was characterized by the prolonged co-existence of BA.5 and XBB strains, which led to a gradual rather than step-wise increase in population-level mean amino acid distance (Fig 4A). For instance, with the emergence of XBB strains, the mean distance rose from 12.2 in late-December 2022 to 18.2 in late-January 2023 and 23.7 by mid-September when the 2023/24 XBB vaccine (i.e., third vaccine) became available, marking the first annual COVID-19 vaccine update. This increase in amino acid distance predicted a 34.3% (95% CrI: -2.4%, 70.4%) reduction in VE against the second vaccine (2022/23 vaccine strains) by mid-September 2023.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Amino acid differences in spike gene between case strains and the updated formulation of mRNA COVID-19 vaccines.

(A) Weekly minimum amino acid substitutions in the spike gene between case strains and either of the two strains included in the bivalent mRNA COVID-19 vaccine formulation (May 2022 – September 2023). (B) Amino acid differences relative to the XBB.1.5. formulation of mRNA COVID-19 vaccines (May 2023 – July 2024). In both panels, each dot represents a single sequenced case and sequences are colored by different variants of concern prevalent during this period. The grey dotted line represents the introduction of the vaccine update. The grey shaded area represents sequences that were collected prior to updated vaccine availability. The red line represents the weekly mean predicted percent reduction in VE based on the 3-week rolling average of amino acid distance from the vaccine strain.

https://doi.org/10.1371/journal.pcbi.1014329.g004

For the third vaccine update (block 3), we observed a more than two-fold increase in mean amino acid distance, mirroring the pattern seen during the original vaccine formulation period (Fig 4B). Specifically, the mean amino acid distance rose from 12.3 in mid-December 2023 to 30.2 in mid-January 2024, corresponding to the transition from XBB to JN.1 dominance. Based on this increase in amino acid distance, we predicted a 43.4% (95% CrI: -5.7%, 90.1%) reduction in VE by mid-January 2024. Our findings also suggest a further 9.8% decline in VE against circulating variants by June 2024, coinciding with the emergence of KP lineages, reaching an estimated 53.2% (95% CrI: 7.0%, 110.4%) reduction compared to VE against the XBB.1.5 vaccine strain.

To assess the generalizability of our findings, we evaluated temporal changes in amino acid distance across the Northeast United States using all publicly available sequences from the NCBI database. Although the precise timing of the abrupt increase in mean amino acid distance varied by state, the overall evolutionary trajectory of circulating strains relative to the vaccine formulation closely mirrored our observations in Connecticut (S6 Fig). This approach demonstrates how the model can support real-time surveillance by quantifying expected declines in vaccine protection as new variants emerge.

Ethics statement

The study protocols for pathogen genomic sequencing of remnant diagnostic samples (IRB Protocol ID: 2000033281) and impact of SARS-CoV-2 genetic variants on immune escape and vaccine breakthroughs (IRB Protocol ID: 2000031374) was reviewed and approved by the Yale University institutional review board.

Data sources and genetic sequences

We used whole genome sequencing (WGS) data collected from the Yale New Haven Hospital (YNHH) system in Connecticut (CT), US between the week of 4^th April 2021 to the week of 21^st July 2024 through the Yale SARS-CoV-2 Genomic Surveillance Initiative. Sample collection, RT-qPCR, and sequencing protocols have been described in detail elsewhere [43,44]. Metadata and vaccination records were obtained from the YNHH system and Center for Outcomes Research and Evaluation and matched to WGS data using unique sample identifiers. We used standardized ZIP-code-level CT income data as a potential covariate, with raw data obtained from the US Census Bureau [45].

We highlighted emergence periods for better visualization and understanding of variant timelines. For uniformity, we applied the same emergence period definitions previously established in Chen et al.[44] In short, these periods represent short intervals during which two variants co-circulated, defined as the time from when a variant accounted for 5% of Global Initiative on Sharing All Influenza Data (GISAID)-reported cases in CT to when it reached its maximum frequency in the population [44].

Epidemiological data

Our study focused on CT residents to ensure consistency in surveillance and vaccination policies. Cases were defined as individuals who tested positive for SARS-CoV-2 infection via PCR and who had samples sequenced at the WGS level. To maintain a consistent definition of VE, only samples taken from outpatient settings were included. We excluded samples for whom WGS information was not available or did not meet the required genome sequence coverage (i.e., genome coverage >95%). Other exclusion criteria included duplicate samples from an individual (to exclude chronic infections) and samples from non-CT residents. Given the vaccine rollout and eligibility timelines differed across age groups, we only included adult patients (18 + years) at the time of sample collection. Cases were aggregated at the week level to reduce sparsity, and subsequent analyses were conducted at the week level.

To assess VE over the first 103 weeks, corresponding to the first two years of the data collection (April 4, 2021 to March 19, 2023), we obtained individual-level metadata from test-negative controls. Controls were selected from samples taken from individuals in outpatient settings who met all criteria for the YNHH system testing initiative, but who tested negative for SARS-CoV-2 infection via PCR. If an individual had both negative and positive test results, we only included the positive test as a case. Any subsequent negative test from the same individual within two weeks of a positive test was excluded to avoid potential misclassification.

A patient’s vaccination status was determined at the time of sample collection, and time since last vaccination was calculated as the difference between the sample collection date and date of last vaccine receipt. Primary analyses were conducted based on time since last vaccination to account for waning immunity, and uniform criteria were applied regardless of the vaccine manufacturer or number of doses received. Time since last vaccination was categorized as: < 14 days, 14–90 days, 91–180 days, ≥181days. The total number of doses received more than 14 days prior to sample collection was also counted for each individual and included as a covariate in our model. For visualization, we plotted 3-week rolling averages to show the proportion of recently vaccinated individuals among both the cases with collected sequences and test-negative controls over time. This allows us to observe trends in vaccine coverage in both groups.

To ensure that our controls were representative of the population, vaccine coverage among controls was compared against available population vaccine coverage data in CT (S1 Fig). We obtained data on the CT vaccination trends from the CDC [46].

Vaccine updates and corresponding sequences

Comirnaty (Pfizer-BioNTech) or Spikevax (Moderna Tx.) were the most widely available and used COVID-19 vaccines in the US. Updates to these two mRNA-based vaccines mostly followed a similar timeline throughout the study period with three updates as of October 2024 [47].

Vaccine blocks were defined based on FDA approval dates, though the exact cut-off dates were selected arbitrarily for analysis purposes. The FDA authorized the use of the bivalent (original formula + BA.4 / BA.5) vaccine as a booster dose on August 31, 2022 [48]; we assigned September 15, 2022 as the start of bivalent vaccine availability, regardless of the vaccine manufacturer [49]. The monovalent XBB.1.5 formulation was approved on September 11, 2023 [50]; we assigned September 15, 2023 as the beginning of the third block.

Genetic distance model

We defined genetic distance as the number of amino acid substitutions between the case and mRNA vaccine formulation spike gene sequences. We aligned sequences using nextclade [51], extract the spike gene region from aligned WGS, we used bedtools [52] to trim the specified genomic coordinates at the nucleotide level. We then translated the nucleotide sequences to amino acid. We measured genetic distance using Hamming distance, which calculates the number of mismatched positions between aligned sequences [53]. Hamming distance was selected as it has been shown to correlate well with antigenic distances and has been effectively used in influenza vaccine research [53–55]. For nucleotide-level comparisons, we calculated the distance between each case sequence and the NCBI reference genome (MN908947), the basis for the original vaccine formulation.

For each vaccine block, we compared case sequences to the corresponding vaccine sequence available during that time period. For example, sequences from block 1 were compared against the original mRNA vaccine sequence, while sequences from block 3 were compared against the 2023/2024 XBB.1.5 formulation. For block 2, we used the minimum amino acid differences between the case strain and either of the two strains in the bivalent vaccine.

Time-varying vaccine effectiveness (VE) model

We assessed time-varying VE using a hierarchical Bayesian time-series method, building on the widely used test-negative design that compares the prevalence of vaccination among SARS-CoV-2 cases and test-negative controls. The binary outcome variable for an individual i at time t (is equal to one if the individual is a case and zero if they are a control. We used logistic regression to model the probability of being a case or control for each individual and time point () as a function of vaccination status and other covariates. Additionally, we extended the conventional test-negative design approach by incorporating correlated random effects to allow the association between vaccination status and case/control probability to smoothly change across time. Specifically, the model is given as:

(1)

where is the time-varying intercept parameter that accounts for changes in baseline risk of SARS-CoV-2 infection for each week t; is an indicator variable for the vaccination category (0 (reference): unvaccinated, 1: < 14 days, 2: 14 – 90 days, 3: 91 days – 6 months, 4: > 6 months) of individual i in week t; is the time-varying regression parameter that describes the association between vaccination category j and the probability of being a case; and is a vector of covariates for individual i for which the corresponding effect estimates do not change across time. The vector of other covariates includes age group, standardized income at the zip-code level, gender, number of vaccine doses received before sample collection, and a “routine flag” (binary variable) to indicate individuals who are thought to be testing routinely (defined as 1 for individuals with more than 50 testing records over the 103 weeks included in our dataset and 0 otherwise).

The time-varying parameters, and, were modeled as a function of a “global” fixed effect parameter and independent (across parameter) autoregressive correlated random effects, such that:

The autoregressive structure specified for the parameters allows both intercept and coefficients to evolve dynamically while borrowing strength across nearby time points. We allow the data to determine the strength of temporal correlation by estimating the parameters. To complete the model structure, we assigned prior distributions to the remaining model parameters. Specifically, we assigned independent log-gamma(3, 2) priors to the precision parameters of the random effects. For the fixed effects, including and , we used weakly informative Normal(0, ) priors. Additionally, we selected independent Uniform(-1, 1) distributions for the autoregressive parameters ().

We used R-INLA (integrated nested Laplace approximations) (www.r-inla.org) [56,57] to fit the model. R-INLA is a computationally efficient approximation to a widely used Markov chain Monte Carlo (MCMC) approach, and provides marginal posterior inference for the included model parameters [58]. We generated 10,000 samples from an approximated marginal posterior distribution for each parameter in parallel using pbmcapply and parallel R packages. Along with posterior means of , the quantile based, equal tailed 95% credible intervals (CrI) of were constructed for each vaccination category. For better visualization, both the posterior mean and CrI for 1- exp were also calculated and plotted, representing VE (%) at each week.

Quantifying the relationship between amino acid distance and vaccine effectiveness

We used a random effects meta-regression model fitted in the Bayesian setting to quantify the relationship between amino acid distance and VE over time, focusing on the 14–90 days post-vaccination category (j = 2) to represent short-term vaccine-induced immunity. We modeled the outcome variable, the observed log OR of infection at time t () from the time-varying VE model as following a normal distribution centered around the latent true log OR (), with variance defined by the squared posterior standard deviations (). Then, the true but unobserved log OR is modeled as a function of amino acid distance and random effects. Specifically, the full model is given as:

where is the key regression parameter that quantifies the association between amino acid distance and the relative risk of infection; is the baseline intercept that accounts for variations in baseline risk across all time points; captures time-varying random effects with an autoregressive structure similar to the first-stage model; and is an independent error term that follows a zero-mean normal distribution. To complete the model structure, we assigned weakly informative priors to the remaining parameters. Specifically, we assigned Normal(0, ) priors for the fixed effects, including and . For the random effects (), we assigned a Uniform(-1, 1) prior to the autoregressive parameter (), consistent with our first-stage model, and a Gamma(0.01, 0.01) prior to the precision parameter governing the variability.

We used MCMC methods to fit the model and sampled from the joint posterior distribution of all model parameters using the rjags package [59] in R. We initialized four MCMC chains with an adaptive phase of 500,000 iterations, followed by 100,000 iterations after convergence. Convergence was assessed via Geweke diagnostics, which yielded Z-scores of -0.648 for and 0.675 for , indicating no obvious signs of non-convergence. Posterior inference was based on posterior means and a 95% CrI for key parameters including .