Study overview

To establish the relative benefit of incorporating interaction between age groups in short-term epidemiological forecasts, we implemented four age-stratified semi-mechanistic models, which each estimate a time-varying Next Generation Matrix (NGM). This matrix is inferred as the interaction matrix between age groups under the assumption that all infections in each age group are informed by the sum of past infections in all age groups weighted by the distribution of time between infections, the generation interval distribution and the NGM. Two of the models included interaction between age groups: The first of which was informed by contact data from the CoMix study (regularly collected during the period of study). To evaluate the benefit of regularly collected contact data over interactions informed by previously collected contact data, we implemented a second instance of the same model using data collected in the POLYMOD survey. This survey conducted in 2008 was designed to measure rates of contact in the population for parameterisation of mathematical models of infectious transmission. The POLYMOD data represent contact during a non-pandemic period and in our study represented an interaction assumption that may be made if the regularly collected CoMix data were not available. Gimma et. al. [19] compared CoMix survey rounds and POLYMOD data in detail, showing that the mean contact rates varied substantially in all age groups between CoMix and POLYMOD, but also between the rounds of CoMix. In particular, the relative rate of contact in children varied between rounds. For example, compared to POLYMOD, there was a much greater reduction in childrens’ contacts than adults’ during periods that schools were closed, however as schools re-opened the childrens’ contacts almost returned to parity with POLYMOD, whereas adults’ contacts remained reduced. Hence the CoMix data added information on both time-varying mean contacts and variation in the interaction between age-groups.

In the third model the interaction between age groups was not informed by contact data at all, but estimated entirely from historical epidemiological data. We do not anticipate the inferred contact matrix to necessarily reflect the true contact rates due to the aforementioned challenges with identifiability. Rather, we wish to test if an over-fit contact matrix from this model performs better than the model based on the collected data. We compared these models with a fourth model which allowed no interaction between age groups.

We applied this to reported cases, as a commonly available quantity for forecasting epidemic dynamics [5]. This, however, incurs a secondary challenge due to potential variability in reporting of cases by age and over the course of an epidemic, which may serve to complicate our interpretation of the application of contact data to forecasts. Hence, to isolate the impact of incorporating contact data we chose to additionally apply the models to estimated infection incidence from a repeated cross-sectional household survey of infections.

We made forecasts of weekly reported cases using the data from the UKHSA Covid-19 dashboard. For convenience we used the full case time-series aggregated to weekly incidence and truncated at different forecast dates, rather than the data available on each forecast date. Although this does not give a full picture of the real-time applicability and performance of the model, it avoids complications in delays in gathering case reports which require additional treatment prior to application of a forecasting model such as truncation of the most recent data or now-casting [28]. Secondly, we applied the models to estimates of weekly incidence of infection estimated [26] from national infection prevalence data, again with the full final data set truncated at each forecast date rather than snapshots available at the time. To further isolate the role of contact data in the forecasts of infections, we used weekly age-stratified estimates of antibody prevalence to inform age-specific susceptibility.

Data

We accessed daily, age-stratified, case data from the UK COVID-19 dashboard [29] on 11th May 2022. We aggregated this data to weekly incidence by taking the sum of the previous 7 days, aligned such that the weekly data is reported on the proposed forecast dates, to forecast weekly case counts in future weeks. The case reports were grouped in seven decade groups between zero and 69 years with a single group for over 70 year olds (0–9, 10–19, …, 60–69, 70+). Dates in the case data corresponded to the day on which the infected individual was swabbed.

We accessed aggregates of SARS-CoV-2 infection prevalence and antibody prevalence collected as part of the COVID-19 infection survey (CIS) through the CIS Website [27] on 18th March 2022. Here infection prevalence is defined as PCR test positivity prevalence under the protocol of the CIS. We used data covering the period between August 2020 and January 2022 to estimate weekly infection incidence and antibody prevalence for seven age groups (2–10, 11–15, 16–24, 25–34, 35–49, 49–69 and 70+). In addition to the CIS data, we used vaccination data published by the National Health Service and accessed via the UK coronavirus dashboard [29] on the same date (Fig A in S1 Text).

We generated SARS-CoV-2 infection incidence and antibody prevalence time-series for the period between August 2020 and January 2022 using an approach described elsewhere [26,30] (Fig 1), henceforth described as the inc2prev model. To establish a weekly time-series of infections we took the sum of incident infections in each week on a sample-by-sample basis and calculated the credible intervals from the resultant sum, this constitutes the distribution of weekly infections according to our modelled estimates. To establish a weekly time series of antibodies we took the antibody prevalence distribution on the last day of each week and calculated credible intervals from the full posterior sample. We accounted for the uncertainty in the estimated antibody prevalence and infection incidence by including them as latent parameters in our model with the prior at each day informed by the estimated distributions above (see the Transmission Model section of this paper for more details)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Estimated incidence of A) infection, B) antibody prevalence from the ONS Community Infection Survey (CIS) and C) case reports from the UKHSA COVID-19 Dashboard, each shown by age group.

https://doi.org/10.1371/journal.pcbi.1011453.g001

We combined these data with social contact data collected as part of the CoMix social contact survey (CoMix) [19,31], a multinational, weekly, cross-sectional survey of social contacts. We used published weekly contact matrices from the UK arm of CoMix, generated under the framework described previously [20].

Transmission model

We extended the concept of the Next Generation Matrix to include transmission interval distributions (the generation interval for infections, and the interval between a positive test in infector and infectee for cases). Here, the number of incident cases or infections I(t) at time t was given by the sum of the products of the next generation matrix N and the age-stratified vector of cases or infections on dates between t-s_max and t-1, weighted by the transmission interval distribution w(s). (1) where s_max is a fixed upper-limit of the transmission interval distribution, set to 4 weeks and w(s) is assumed to follow a discretised log-normal distribution with time since the primary event (infection or positive test of the infector): (2) where F_LNorm is the cumulative distribution function of the log-normal distribution with parameters w_μ and w_σ. Under the social contact hypothesis [17], the next generation matrix is calculated by multiplying the contact matrix, C(t) quantifying the mean number of contacts between age groups, with vectors of age-specific susceptibility, , and infectiousness, , where each element, s_a and i_a give the specific susceptibility and infectiousness of age group, a [13].

(3)

We assumed that age specific infectiousness, is inherent and unrelated to time varying factors associated with the epidemic. We assumed that age-specific susceptibility included two components: (4)

The first (s_ab,a) is the age-specific acquired immunity to infection in age group a, this is informed by weekly antibody prevalence estimates from inc2prev. We used a leaky definition of antibody effectiveness in line with the definition used in the estimation of the infection and antibody timeseries: (5)

Where Φ is the effectiveness of antibodies in preventing infections in an exposed member of the population and A_a(t) is the antibody prevalence in age-group a at time t. The second component () is due to an age-correlated variation in inherent susceptibility to infection and unrelated to time-varying factors associated with the epidemic. Both and were assumed to remain constant in time, such that all of the variation in the next generation matrix by time is governed by changes in contacts and estimated antibody derived immunity. Both and were fit as random effects in a hierarchical framework (Table 1). To account for uncertainty in estimates of antibody prevalence, A_a(t) was also estimated in our model, the prior was based on the mean and standard deviation of the previously estimated antibody prevalence using inc2prev (A_μ,a(t), A_σ,a(t)).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Model parameters and priors.

https://doi.org/10.1371/journal.pcbi.1011453.t001

Parameter estimation and forecasting

To allow variation in parameter values over the course of the study period, we parameterised the model with the estimated antibody prevalence and contact matrices and fitted it to 8 weeks of weekly estimated infection or reported case data prior to the forecast date. We fitted the model using Hamiltonian Monte Carlo, implemented in the Stan probabilistic programming language [32] we assessed the performance of the model fits by monitoring convergence metrics and transition divergences.

We fitted to the mean infection time series under the likelihood: (6)

Where is the vector of the overall uncertainty in the modelled infections for each age group., Each age-group element (e.g. σ’_I,a, for age group a) combines the time-varying, age specific inherent uncertainty in the NGM model, σ_I,a, and the standard deviation of the infection estimates, I_σ,a, (7)

σ_I,a is constructed at each time point from the estimated coefficient of variation CV_I,a and infection incidence such that: (8)

Where I_μ,a(t) is the mean of the infections estimated using inc2prev in age group a. This ensures the uncertainty scales with the magnitude of the infection incidence estimates. We fit to the case time series c(t) under the likelihood: (9)

Where σ_C is the modelled uncertainty in cases and is constructed from the estimated coefficient of variation CV_C at each time point for each age-group (a) such that: (10) which ensures the uncertainty scales with the magnitude of the reported case incidence. To incorporate the contact data in the CoMix based model we jointly fit the contact matrices under the likelihood: (11)

Where, C_ab(t) is the rate of contact between age groups a and b at time t, C_μ,ab is the mean rate of contact recorded between age groups a and b, and (12) and C_σ,ab is the observed standard deviation of the measured contact rate and σ_cm is the uncertainty in the posterior contact rates.

Each of the models estimated a NGM which varied over the 8 weeks of prior data only by changes in the estimated contact matrices and antibody inferred immunity, whilst the inherent susceptibility and infectiousness vectors were assumed constant for the whole modelled period. However, as each forecast date was modelled independently, all parameters were able to vary between forecasts.

The priors we employed are given in Table 1. Antibody prevalence priors were set to the distribution of the estimate provided by the model used to estimate incidence [30] and relative susceptibility and infectiousness vector elements were set such that the Secondary Attack Rate (SAR) was roughly half that of estimates of Household SAR in literature [33]—which aimed to account for reduced risk of transmission to known contacts outside the household. The prior for the log-mean (w_μ) and log-standard-deviation (w_σ) of the transmission interval had a mean and standard deviation of 5 days to reflect the broad distribution or transmission intervals recorded in literature [34–36], these were converted to the appropriate log-parameters for the log-normal framework in Eq 2, and their prior was set to be normally distributed with a standard deviation of 20% of the mean.

We used posterior distributions of the parameters (Table 1) to project infections and cases forwards up to four weeks after each forecast date. We note that contact data directly relevant to the dates forecasted would not be known on the forecast date, so we used the contact data corresponding to the week of the forecast date itself, assuming that these also reflected contacts in the following week. For the case forecasts we offset the contact data by 7 days to account for delay between infection and specimen date and used the generation interval as a proxy of the test-to-test distribution [37], which is consistent with a 5 day incubation period and a 2 day report delay [38].

Model evaluation

We evaluated the performance of the NGM models (CoMix-data, POLYMOD-data, No-contact-data and No-interaction) across 29 forecast dates between October 2020 and December 2021. We chose this period as there was major disruption to the CoMix survey during July 2020 and following changes in the survey in June 2020. We excluded dates after December 2021 due to the complication of the emergence of the Omicron variant, which has been shown to evade immunity to a greater extent than earlier variants [39]. This would complicate our interpretation of antibody prevalence as a mix of Omicron-specific and previously acquired antibodies persist in the population.

We evaluated the forecasts against the reported number of cases or mean estimated number of infections in the week forecasted, for case and infection forecasts respectively. We evaluated the forecasts based on Continuous Ranked Probability Score (CRPS) and a measure of bias (see appendix for definitions) each implemented using the scoringutils R package [40]. The CRPS measures the ‘distance’ of the predictive distribution to the observed data-generating distribution, hence a lower score indicates more accurate predictions and therefore a higher performing model. To summarise the performance of the entire model, we calculated the overall CRPS as the mean CRPS across all age-groups. The bias measures the tendency for a model to over (positive value) or under (negative value) predict the incidence in its projections, hence a bias of zero is optimal.

We also assessed the models calibration by evaluating the central interval coverage, henceforth referred to as coverage, of each model’s forecast (Proportion of incidence points which fell in the ranges projected by the forecast model’s posterior distribution of future cases).

To provide a comparator as a lower bound of performance, we also evaluated two baseline models. These baselines were intended to represent naive assumptions, which may be applied without the use of a model. The first baseline assumed no change in incidence from the day the forecast was made. The second calculated the change in incidence between the forecast date and each week within the four week forecast horizon, the rate of change is projected as an exponential extrapolation based on the previous two weeks of data. Both baselines were modelled without uncertainty and, consequently, the CRPS reduced to the mean absolute error. To provide a clear comparison of performance with and without interaction between age groups, we provide all CRPS scores relative to the score of the no-interaction model (rCRPS).

As well as the overall performance of each forecasting model, we also evaluated the forecasts by grouping forecasts made by each model in two ways. Firstly, we aggregated the forecasts by age group—showing the relative performance of the models when forecasting incidence in particular age categories. Secondly, to evaluate how performance changed over the course of the analysis period, we scored the forecasts separately for seven key periods of the pandemic (Table 2). For this we used the periods used in Gimma, et. al. [19] which overlapped with our analysis, with the addition of two periods that were not covered by the previous CoMix work. Due to the small number of weeks covered by ‘Christmas’ and ‘Lockdown 3 schools open’ we combined these with ‘Lockdown 3’ and ‘Lockdown 3 Easing’ respectively. The periods we analysed are defined by the non-pharmaceutical interventions in effect over the course of the UK epidemic, however, to support interpretation of the forecast performance, we also characterised them based on the trajectory of the epidemic within each period as detailed in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Pandemic period names and dates.

https://doi.org/10.1371/journal.pcbi.1011453.t002

Age-specific transmission parameters

Finally, to compare the implicit assumptions within the models we applied, we assessed how the values of the relative susceptibility and infectiousness parameters s and i varied over the pandemic. To provide an interpretable quantification of these parameters, we used the age-specific values estimated in the model to calculate ratios of susceptibility and infectiousness of younger adults, and older adults relative to that of children. Due to the different age-stratification in the data available, the broader age-bands here varied between case forecast models and infection forecast models: Children were defined as up to 15 for infections and up to 19 for cases, younger adults were defined as 16–49 for infections and 20–49 for cases and older adults were defined as over 50 in both instances.

Forecasts

We made forecasts with a horizon of one, two, three and four weeks at fortnightly intervals (Figs 2 and B—G in S1 Text) between 30th October 2020 and 26th November 2021 (29 forecast dates). Visually the forecasts deviate more from the true data at longer forecast horizons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

A) and B) Infections and cases, respectively, forecast using the CoMix-data based next generation model, the no-contact-data, no-interaction and POLYMOD-data data based model, for each age group (top to bottom) and forecast horizon (left to right). projected infections from each model (coloured bars) and black points show infection estimates and reported cases in plots A and B respectively. The estimates being forecast on each axis are shown as solid points; those not being forecast are shown as rings. C) and D) show the continuous ranked probability score relative to the score of the “no interaction” model for each forecast date, calculated from the Infection and Case forecasts respectively.

https://doi.org/10.1371/journal.pcbi.1011453.g002

Model evaluation

To assess the relative performance of each of the models for different forecast horizons, we calculated evaluation metrics separately for each forecast horizon across all forecast dates (Fig 3 A and Table A in S1 Text). For an alternative approach using multivariate evaluation across age groups and time horizons, see Held et al. (2017) [25].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Continuous ranked probability score relative to the score of the no-interaction model.

A and B show overall rCRPS (A) and Bias (B) of each model when applied to case (left) and infection (right) data for each forecast horizon. B. and C. show the CRPS relative to the no-interaction model against forecast horizon disaggregated by age for infection and case data respectively. The colour of the points shows the corresponding model.

https://doi.org/10.1371/journal.pcbi.1011453.g003

When forecasting case reports from the UK Covid-19 dashboard [29], the non-interaction model performed better than any of the models which allowed interaction. The next best model was the model with no contact data which performed similarly to the no-interaction model, particularly at longer time horizons with rCRPS of 1.02, 0.99, 1.01 and 1.01 (relative to the no-interaction model) respectively at horizons of one to four weeks in ascending order. The two models that incorporated contact data both performed poorly when considering the CRPS relative to the no-interaction model, with the POLYMOD model performing the worst. However, the relative performance of both models improved at longer time horizons, with the CoMix model performing similarly to the other NGM models at four week horizons (rCRPS of 1.53 and 3.78 at one week horizon reducing to 1.01 and 1.53 at four week horizon for the CoMix and POLYMOD data models respectively). Both CoMix and POLYMOD forecasts had a substantial positive bias, showing that, on average, they over-predicted cases with bias between 0.15 and 0.25. The other models tended to under predict cases by a smaller margin (between 0 and 0.18). The exponential baseline performed worse than the no-interaction model at all forecast horizons (rCRPS 1.35, 1.25, 1.11 and 1.09 at one to four weeks in ascending order). The fixed value baseline initially performed second to worst for one week forecasts (rCRPS = 1.76) but improved as the horizon increased, eventually becoming the best performing forecast at four week horizons (rCRPS = 0.74).

When forecasting infection incidence estimated from UK prevalence survey data [27], the no-interaction model performed second only to the no-contact-data model (rCRPS = 0.89) for horizons of one week, followed closely by the CoMix-data model (rCRPS = 1.05). The POLYMOD-data model performed worst when forecasting one week horizons with a rCRPS of 1.21. However, at two week horizons the no-interaction model became the worst performing model overall—which remained true for three and four week forecast horizons. In these cases the CoMix-model performed best of all the models including the baseline models with rCRPS of 0.68, 0.64 and 0.57 (relative to the no-interaction model) for two, three and four week horizons respectively. The second best performing NGM model at two and three week horizons was the no-contact-data model, rCRPS of 0.82, 0.85 respectively. At four week horizons the POLYMOD-data model was second best performing NGM model with a rCRPS of 0.76. The baselines both did worse than all but the POLYMOD-data model when forecasting at a one week horizon, however the performance of the fixed value baseline improved relative to all of the NGM models at longer forecast horizons and produced the second best performing forecasts overall for forecast horizons of three and four weeks (after the CoMix-data model) with rCRPS of 0.79 and 0.68 respectively.

We compared the relative forecast performance scoring predictions in each age group separately (Figs 3, B, and C in S1 Text and Table B in S1 Text). When forecasting infection incidence, we found that the CoMix model and no-contact-data model forecast infections better than the no-interaction model in middle-aged adults and older adults (35+ years old) for all forecast horizons. The models also performed best at forecast horizons of two weeks or more in young children (2–10 years old) and older adults. In contrast, the no-interaction model performed much more similarly to the interaction models for forecasts within younger adults (16–34 years old). The same was also true for older age groups (60+) in the case forecasts but not for children, middle aged adults or children. For infections, the performance of all the models improved in all age categories relative to the exponential extrapolation baseline as forecast horizon increased, the fixed value baseline improved relative to the no-interaction model in all age categories but provided poorer forecasts than the CoMix model in all age-categories and time horizons. For case forecasts however the fixed value baseline improved relative to all of the models as horizon increased, providing the best forecasts at four week horizons in age groups between 0 and 59 years.

We divided the analysis into seven periods (Fig 4 and Table C in S1 Text), within each of which national restrictions on social activity remained broadly consistent. For consistency we used the same periods as those presented in Gimma et. al. [19], which presents the key findings of the CoMix study. The relative improvement in performance for the CoMix-data model was most consistent when forecasting infections during the Christmas and Lockdown 3 period, which was the only period where the CoMix-data model performed the best overall at all forecast horizons. When forecasting cases, the CoMix-data model also performed relatively well during this period at forecast horizons of two or more weeks, but performed comparably to the no-interaction model at one week forecast horizon. The Comix model’s infection forecasts outperformed all other NGM models in the two periods following this (Lockdown 3 easing and Opening up) for forecast horizons of two weeks or more, where only the fixed value baseline model improved on its score. Similarly for the Lockdown 2 easing period, the CoMix-data model performed better than the no-interaction model at all forecast horizons, but the no-contact-data model performed better for one and two week forecast horizons. The improved performance of the CoMix model was not wholly reflected in the case forecasts. In particular, the CoMix model performed more poorly than the no-interaction model at all time horizons during the Lockdown 3 easing period. However the CoMix model performed better during the Lockdown 2 period, than the other NGM models for case forecasts, whereas for infection forecasts the CoMix model performed comparably to the no-interaction model during this period.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Continuous ranked probability score relative to the score of the no-interaction model against increasing forecast horizon (one to four weeks).

Panels left to right show each pandemic period, Panels top to bottom show forecasts of cases and infections.

https://doi.org/10.1371/journal.pcbi.1011453.g004

Forecast calibration

Of the four NGM models, the CoMix data-based model was best calibrated for case forecasts at one and two week horizons and at all horizons for infection forecasts. This is evidenced by closer agreement between the proportion of true values in each central range of the predictive distribution (50% and 90%) with the value of the range (Fig 5A and 5B). Calibration typically became poorer at longer forecast horizons, with more true values falling outside the specified ranges than would be expected. We also note that none of the forecasts were particularly well calibrated when considered overall forecast dates. The best performing forecast, infection forecasts made by the CoMix model at a one week horizon, saw fewer than 75% of true values fall within the 90% confidence range of the associated projections and fewer than 40% within the 75% confidence range. Separating the forecasts by period of the pandemic (Figs H and I in S1 Text) revealed that the CoMix model was best calibrated for ‘Christmas and Lockdown 3’ and ‘Lockdown 3’ periods, for both the case and infection forecasts. In particular the CoMix model’s forecast of infections was very well calibrated during the ‘Christmas and Lockdown 3’ period, with more than 80% of true values falling within the 90% confidence range of the forecast at all horizons. The other models were also relatively well calibrated during these periods. Overall the other periods were much more poorly calibrated. In particular, the “Lockdown 2” and “Lockdown 2 easing” periods were very poorly calibrated across all models with no true values falling within the 90% confidence range for the no-contact-data and no-interaction model forecasts during the ‘Lockdown 2’ period. The baseline models are not presented as they do not include confidence ranges—hence the forecast coverage is zero for all forecasts by definition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Calibration of the forecasts made by each of the next-generation-matrix-based models.

A) The proportion of observed mean incidence of infection estimates (obtained using inc2prev) and B) the proportion of observed case numbers that fall within the 50% and 90% central interval of the relevant forecasts of the four models. C) and D) the percentage of observed mean incidence of infection estimates (obtained using inc2prev) falling below each quantile of the forecasts at one and four week horizons respectively. E) and F) the percentage of observed case counts falling below each quantile of the forecasts at one and four week horizons respectively.

https://doi.org/10.1371/journal.pcbi.1011453.g005

Age-specific susceptibility and infectiousness

We extracted posterior estimates of time-varying age-specific infectiousness (i) and susceptibility (s) to assess the biological plausibility of these parameters (Fig 6). For the CoMix model the susceptibility of younger adults (16–49 years old for infections and 20–49 for cases) and older adults (50+ years old) was higher relative to children (under 16 for infections and under 20 for cases). In the early part of 2021, this began to shift such that first susceptibility reduced relative to children in the older adults and then in younger adults. Ultimately by the end of the period evaluated, children had higher susceptibility relative to all adults. A similar pattern was present in all models that allowed interaction between age groups. Infectiousness broadly remained equal between age groups, with the exception of a small number of outlying values within which there is no clear trend.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6.

Susceptibility (A) and Infectiousness (B) and of younger adults (16–50 for infections and 20–50 for cases) and that of elderly adults (50+) each relative to children (under 16 for infections and under 20 for cases) by forecast date (hue). Each model is shown in panels left to right.

https://doi.org/10.1371/journal.pcbi.1011453.g006