Ethics statement

UKHSA have an exemption under regulation 3 of Sect. 251 of the National Health Service Act (2006) to allow identifiable patient information to be processed to diagnose, control, prevent, or recognise trends in, communicable diseases and other risks to public health.

Forecast targets

Our forecasting targets are key metrics for winter hospital pressures, identified by working with healthcare operational managers. In the Winter 2024–25 season, UKHSA forecast new test-positive hospitalisations for COVID-19, Influenza, respiratory syncytial virus (RSV), and norovirus test positive cases. The basis of this paper is primarily to present and analyse forecasts of COVID-19 and Influenza hospital admissions with the most comprehensive operational ensembles. Both COVID-19 & Influenza operational ensembles contained more than three models, whereas the RSV operational ensemble contained only three models, and norovirus a single model. For this reason, we present and evaluate the RSV and norovirus forecasts in S1–S10 Figs. We exclude these pathogens from the retrospective sub-ensemble analysis due to having insufficient operational models developed this season.

Consistent and comprehensive data collection across hospitals in England is performed by the National Health Service (NHS). In secondary care settings, if an individual presents severe disease symptoms, a diagnostic test will be performed with results reported to UKHSA via the Second-Generation Surveillance System (SGSS). The data and forecasts are, for each date of a test, the number of unique inpatients with a new positive test taken within the past 24 hours. The data can be revised for up to 14 days following submission, however, in practice the extent of revisions is small shown in S11 Fig. All forecasts are produced weekly, at daily granularity, with a 14-day forecast horizon.

The forecast targets discussed in this paper were produced at multiple spatial scales. Influenza and COVID-19 admissions were forecast at the level of integrated care boards (ICBs) and then aggregated to produce regional and national forecasts. An ICB is a sub-regional NHS body responsible for the management of local health services. ICBs are each nested within one of the seven NHS regions. A map illustrating NHS regions and ICBs is presented as S12 Fig.

Our secondary forecast target approach is an epidemic trend direction assessment. We categorise an epidemic as one of “decreasing”, “stable”, or “increasing”, an ordinal classification. This is a simplification of our forecasts, but useful for many users. The trend direction for a given forecast day is estimated by first calculating the 7-day right aligned rolling average of forecast target and forecast sample value. The direction is then a percentage change between the rolling average target, and a rolling average forecast 14 days ahead (our forecast horizon). A upward change between target data and forecast value of is considered an “increase”, while a downward change of is a “decrease”. Changes less than 20% are considered “stable”. This procedure produces trend directions for each time step in the forecast horizon, up to the final forecast day. For forecast users we only show the final forecast trend direction as the most relevant overall indication of direction. This method has desirable qualities – we can use the approach across many pathogens without alteration and can deploy it at a range of spatial and population scales, which are more challenging with absolute changes.

We calculate the trend direction probability as the proportion of posterior samples that are between each threshold boundary, for example if 350 of 500 posterior sample predictions meet the “increase” definition, we have a probability of increase of 70%. The specific threshold of 20% was chosen following consultation with forecast users as a meaningful change in trend that maintained coherence with other surveillance metrics users were familiar with. Surveillance indicators within UKHSA and elsewhere are often classified using a 10% week-on-week change, which we extend to our 2-week horizon, which would be a 21% increase (). We approximate the 21% change to 20% to make the method clearer for users. We are currently considering how to extend this approach to even longer time horizons, but feedback from users will be the primary driver.

The individual models used operationally in the 2024–25 season are described in Table 1. Each GAM model is a permutation of previous work [18], with the ETS models also described in [10]. The GP growth rate, Mean GR and Median GR models were conceptually new to the English operational ensemble this year, with further details given in S2 Section. Full implementation details are available in the supporting code https://github.com/jcken95/sub-ensemble-evaluation.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Name and description of each model used in during the winter 2024-25 season with supporting description and package name.

https://doi.org/10.1371/journal.pcbi.1014644.t001

Scoring rules and scoring analysis data

As our forecasts are probabilistic in nature, we generate entire probability distributions, via quasi-posterior sampling, to provide uncertainty quantification about future values of the target. This set of samples is converted into predictive quantiles; equi-tailed 50% and 90% intervals are reported to users, as well as the median forecast value. However, for evaluation purposes we store equi-tailed intervals with the following nominal coverages: 50%, 90%, as well as predictive medians – the intervals presented to users each week. Proper scoring rules [19] are used to assess probabilistic forecasts in a way which encourages the forecaster to provide an honest assessment of uncertainty about the value to be forecast.

The appropriate scoring rule to be used depends on the nature of the quantity of interest. For our predictions of admissions, we use the per capita weighted interval score (pcWIS). This is the weighted interval score (WIS) applied on a per capita scale, that is, is replaced by in the definition of WIS, with quantiles also divided by the population catchment size [20]. Scoring a transformed forecast can be more useful than scoring a raw forecast [21]. This per capita approach allows us to compare scores across different spatial geographies on the same scale, as some have larger population catchments than others, e.g., nation versus region. The WIS is a weighted average of interval scores. If a prediction interval is used to estimate , where is the nominal coverage, is the lower bound of an interval estimate and is the corresponding upper bound, then the per capita interval score is

(1)

where is the indicator function. If we have prediction intervals as estimates of , as well as a median estimate , then the pcWIS is

(2)

with the usual choice of being , and [22]. For pcWIS we evaluate the forecast by averaging across all 14 days of the forecast horizon, as all are shown to forecast users.

The second score we use is for assessing the ordinal component, trend direction, (“increase”, “stable”, “decrease”) of our forecasts using the ranked probability score (RPS) [23]. Let be the forecast probability of category . Let the observed category be with . The ranked probability score is:

(3)

where both pcWIS and RPS are negatively oriented (lower values indicate better performance). For the trend direction scoring we only evaluate the last day of the forecast at 14 days, as the only direction shared with users.

Both pcWIS and RPS are transformed scoring rules with attractive properties which, when taken together, account for different forecast user preferences [21] The pcWIS is an absolute metric, where periods of high values will be penalised more – which is a quality attractive to hospital managers where staff and beds have limited numbers. For example, the growth rate is less useful in this setting than knowing when the quantity of admissions will go above a threshold. The use of a per-capita transformation allows for easier comparison between locations with different population catchments in this absolute scale. The RPS penalises error of the relative prediction, which is a discretised transformation of the epidemic growth rate, which can be more informative to public health officials responsible for interventions.

In all cases we evaluate the predictions against the finalised data set collected after all revisions have been received (14 days following the end of the data forecasted).

Retrospective simulation experiment

Due to operational constraints, not all the models used in the 2024–25 season were developed or in use at the start of the season, a common challenge for comprehensive evaluation both for in-house modelling efforts and in collaborative hubs where submission is optional. In Fig 1 we see that some models were introduced in the late season, or models were used at different times. Models are sometimes dropped from an ensemble because either the predictions are unrealistic, or an issue which would have been too time consuming to fix before the deadline for forecasts to be shared with users. This means that scores aggregated from predictions over the season do not provide a fair comparison of model performance, as some models are present only in easier periods to forecast. Since only one operational ensemble is used per disease per week, we cannot be certain if we used the “best” possible ensemble, or if individual constituent models were appropriate.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. (Top) UKHSA weekly 14-day ahead operational ensemble forecasts for COVID-19 and Influenza, with count of observed admissions shown as black dots.

Forecasts are a median prediction (coloured lines) and a 90% prediction interval (colour-matched bands). (Bottom) Graphical representation of which models featured in the operational ensemble over time for each admission target. For operational delivery reasons the Influenza season began a week earlier than for COVID-19.

https://doi.org/10.1371/journal.pcbi.1014644.g001

To help answer these questions we conducted a large simulation experiment. We re-ran each of the models available by the end of the season, producing 14-day forecasts every week from the start of the season until the end of the season. Then, to understand how each model might impact the operational ensemble, we produced all possible sub-ensembles containing 3 models. Computing all possible ensemble combinations would be highly computationally expensive: for a suite of models there are ensembles to be constructed at many spatial geographies. However, the method is general, and any ensemble size is theoretically manageable.

The models and data used to generate these forecasts are based on the final model configurations at the end of the season, and revised data. However, the scale of data revisions is small for this data, with an average 1.3% and 4% change for COVID-19 and Influenza respectively, shown in S11 Fig.

To further reduce computational expense, we used quantile averaging (mean quantile value or mean probability) to produce sub-ensemble forecasts; a common practice for forecasting hubs [5]. In contrast, for operational ensemble forecasts during the season, we constructed an ensemble via unweighted posterior stacking [8,10]. We then score all models and sub-ensembles via the scoringutils package [24].

Generalised Additive Models (GAMs) and averaging to understand the effect of a model in an ensemble on score

We smoothed the sub-ensemble scores using methods available in the mgcv package, and used an averaging method (Equation 1) to estimate the effect of an individual model in a sub-ensemble. This method is used to understand how a model influences the pcWIS as well as RPS. For all scores and diseases, our GAM structure is as follows:

(4)

where is the average score of sub-ensemble for a location level (i.e., nation, region, ICB) over all forecast horizons for a given prediction start date is a global smooth for the score over time, and are sub-ensemble and location level specific smooths – for each smooth we used penalised thin plate regression splines with knots placed every two weeks – and and are random effects for the sub-ensemble and location level respectively. The structure reflects our forecasting unit: location and time. We use a Gaussian error structure after log-transforming our scores to prevent negative score predictions (pcWIS and RPS are non-negative, real values). We use standard residual-based diagnostics to investigate our GAMs with results in S13–S16 Figs of the supplementary information. There are some minor to moderate issues with our GAMs which are a limitation of the analyses relying on the GAMs.

For a given forecasting unit, we define the average effect of including a model in a sub-ensemble at prediction start date and location level averaged over the forecast horizons as

(5)

Where is the mean score of the forecasts across the forecast horizon (days 1–14), is the set of all considered sub-ensembles – in our specific case this is all sub-ensembles that can be made of size 3 – and is the set of sub-ensembles (of size 3) where is a component model. This is inspired by a functional ANOVA [25] decomposition, and therefore could be extended to understand how combinations of models contribute to sub-ensemble performance. This is, for a fixed location level, the average score from sub-ensembles containing a given model, minus the average score across all sub-ensembles. This is also conceptually similar to the use of Shapley values to understand component model importance [17].

The gratia package [26] allows us to generate quasi-posterior samples of from the models fitted in mgcv; we therefore provide median estimates of the effect, as well as 90% credible intervals, by computing each component sum of Equation (5) on a sample-by-sample basis.

Pareto front

A unique aspect of our forecasting product is the two outputs: a 14-day forecast of target values, as well as the trend direction estimate. This can be viewed as a set of competing objectives, as a single operational ensemble may not strictly optimise both scores. In addition, we have model outputs at different spatial scales. We therefore have 6 competing objectives: pcWIS at national, regional and ICB geographies, and RPS at those same geographies. Optimising performance for a single metric represents an oversimplification of practical forecasting challenges and diverse customer needs.

A Pareto Front (PF) [27] is a tool commonly used when there are multiple competing objectives and no single best solution. Members of a PF can be considered as candidates for a best solution to a multi-objective optimisation problem.

Consider our collection of sub-ensembles , and suppose are possible sub-ensembles and are negatively oriented scoring rules applied to the sub-ensembles at a given forecasting unit. We say dominates if and there exists such that . We then define the PF as

We perform two separate Pareto analyses. First, we perform a simplified Pareto analysis which weights the scores to reduce the dimensionality of the problem: we work with

(6)

and we weight pcWIS to obtain pcWIS* in the same way. This weighted approach is possible when the scores are on the same scale, which is inherent for RPS, and made possible by using pcWIS rather than WIS. This gives us a two-dimensional front. The weights are chosen to represent a belief that the national forecast is twice as important as the other levels, since this forecast is used by the most senior health officials in the country. However, the other forecasts are not unimportant, since they are used by a wide range of individuals. A “full” Pareto analysis which uses all scores at all spatial geographies is performed with results available in the supplementary materials (S2 Table).

Operational forecasts

Operational ensemble forecasts, and the component models used in each forecast, are presented in Fig 1. Our operational forecasts were able to accurately capture disease dynamics, with 14-day-ahead predictions generated on a weekly cadence. The gap in each set of forecasts is due to a break over the winter holiday period when no forecasts were produced. S17 Fig shows that the nominal 90% prediction intervals often had close to 90% coverage across diseases and spatial scales. For COVID-19 the forecasts were most closely calibrated after the winter break. For Influenza, the nominal coverage was usually greater than expected, with a drop in coverage in mid-November due to an earlier than expected increase in influenza admissions. Forecasts presented to customers also included regional and ICB breakdowns. S18 Fig shows time series of trend direction estimates. Regional forecasts, and a collection of ICB forecasts are available in S19 and S20 Figs. Equivalent plots for RSV admissions and geography and age and national Norovirus cases are available in S1-S4 Figs.

Scoring analysis of operational forecasts

For the national operational ensemble forecast, we saw that for COVID-19 the pcWIS generally improved over the season (Fig 2). The slow-changing epidemic dynamics meant accurate forecasts were regularly produced [28]. For Influenza, the pcWIS increases up to the epidemic peak (late December), then decreases again. The number of hospital admissions around the peak of the epidemic was hardest to predict. Our forecast had large uncertainty around the epidemic peak. The predictive intervals were well-calibrated with the nominal coverage: 90% observed coverage statistics were close to the nominal coverage for all diseases and geographies (S3 Table). However, observed coverage at the 50% level was often higher than expected. The mean pcWIS for the operational ensemble was , which is slightly higher (worse) than the mean for all individual models: . For influenza the mean operational pcWIS was 5.62 , which is lower than the mean across all individual models: 1.34 .

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Average scores per prediction start date for COVID-19 and Influenza hospital admissions by model during the 24/25 season.

Scores for a given prediction start date are averaged over the 14-day forecast horizons, with lower values indicating better performance. On the top row for each disease is the pcWIS, and on the bottom row the RPS score. The Operational ensemble (the forecast presented to users) is highlighted in black.

https://doi.org/10.1371/journal.pcbi.1014644.g002

In terms of operational ensemble forecast trend direction, RPS was low for COVID-19, often near the theoretical lower bound of 0, with occasional spikes. This indicates the trend direction estimates were typically accurate, but the spikes indicate where trend direction estimation was relatively poor (Fig 2). Again, this is likely due to the simple, stable dynamics of the epidemic. For Influenza, the operational ensemble often performed well, obtaining small RPS values. The performance measure by RPS was consistent near the epidemic peak. As with pcWIS, the RPS values indicate the operational ensembles were, on average, better than individual models. For COVID-19, the mean RPS for the operational ensembles was 0.163, compared to 0.188 for individual models. For Influenza, the mean RPS values for the operational ensemble and individual models are 0.161 and 0.271 respectively. This indicates our ensembles are better at trend estimation than individual models, on average. Scores over time for each model and spatial scale are shown in S21 Fig. To compare the relative skill of forecasts at different population levels we refer the reader to S22 Fig. pcWIS are RPS are both scale-agnostic scoring rules, however, we see that, in general, larger population areas generally correspond to lower (better) score values. This suggests that the larger geographies are usually easier to predict.

Retrospective simulation analysis of COVID-19 forecasts

Fig 3 shows the effects of individual models on sub-ensemble performance for COVID-19 modelling. For pcWIS, we see that the GP growth rate model was detrimental to sub-ensembles; the pcWIS increased by up to 50% around December, when hospital pressures approached their peak. For RPS, the effects of models on sub-ensemble performance are smoother. We see that, again, GP growth rate is least useful around December-January. However, many other models become less useful as the season progresses. For RPS contribution we see worsening in the performance estimates for ETS, cubic regression and Gaussian process models over the study period. However, random walk and mean growth rate models show improving scores with time. The GP growth rate model is an experimental growth rate model, which can give unusual results when the growth rate is close to 0, which may explain poor performance, particularly around November to December. However, this did offer better-than-average performance towards the end of the season (see S23 Fig). Comparisons of performance over time for the operational ensembles against individual models at each spatial resolution are given in S21 Fig; the operational ensemble is rarely the best model but typically outperforms most individual models. Season average scores for operational ensembles and individual models are given in S1 Table for COVID-19 and Influenza; this shows that ensembles (either operational or matched) usually provide the best numerical or trend direction forecast of COVID-19 and Influenza. In 8 out of 12 combinations (2 diseases, 2 metrics, 3 location levels) an ensemble is the best model. Ensembles were better than the majority of individual models in the cases where the ensemble was not optimal, see S1 Table for all mean score values by model across the sesaon.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Using data generated from a retrospective evaluation of many sub-ensemble combinations the effect of including a model in an ensemble is estimated using a GAM.

Shown are the estimated effects of including a model in an ensemble (expressed as a percentage change in pcWIS or RPS) for COVID-19 forecasts from the overall mean score across all sub-ensembles at that point in time.

https://doi.org/10.1371/journal.pcbi.1014644.g003

Retrospective analysis of Influenza forecasts

Fig 4 shows the effects of individual models on sub-ensemble performance for Influenza modelling. We see that the improvement provided by models fluctuates over time. For example, we see that the cubic regression model is worst around the peak (late December – early January): cubic regression models tend to provide approximately exponential dynamics. However, ETS, random walk and Gaussian process models do well around the peak; all of these models have either “flat” predictions, or in the case of the Gaussian process, predictions which revert towards a constant mean. The median growth rate is an experimental model which uses growth rates from previous seasons to provide future forecasts. This provided reasonably good pcWIS contributions. In England, the peak in influenza admissions varies slightly from year to year [29], which may explain relatively poor performance around the peak for the median growth rate model.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Using data generated from a retrospective evaluation of many sub-ensemble combinations the effect of including a model in an ensemble is estimated using a GAM.

Shown are the estimated effects of including a model in an ensemble (expressed as a percentage change in pcWIS or RPS) for forecasts from the overall mean score across all sub-ensembles at that point in time.

https://doi.org/10.1371/journal.pcbi.1014644.g004

Pareto analyses

Weighted analysis.

Since the weighted Pareto analysis is two-dimensional, we can plot the results. Note that in the Pareto analysis, we use an average score over the entire season. This allows us to understand which sub-ensemble combination would have been best over the entire season, on average.

In Fig 5, we see that the PF for COVID-19 is constructed of three sub-ensembles. Each sub-ensemble contains a random walk model. Two contain one of {cubic regression, Gaussian process, mean growth rate}. Since the COVID-19 epidemic was near flat in the 2024–25 season, a random walk component in the ensemble is intuitive. The cubic regression, Gaussian process and mean growth rate models all behave quite differently. We also see a cluster of points far away from the PF. Every model in this sub-ensemble contains a GP growth rate model, suggesting this model is detrimental to average sub-ensemble performance. The values used for the PF analysis are available in S25 Fig. We see in the S23 Fig of retrospective forecasts that, on occasion, the GP growth rate forecasts exhibited unusual behaviour. However, the difference in score values is typically small between even the best and worst models, when averaging over the season.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. The performance of different sub-ensembles in terms of RPS* & pcWIS*.

The sub-ensembles are given abbreviations for clarity to indicate sub-ensemble membership. The scores are generated as the weighted sum of the average value at each geography level (Equation 6). Pareto front for best performing models in pcWIS* and RPS* are shown in blue.

https://doi.org/10.1371/journal.pcbi.1014644.g005

The PF for Influenza under the weighted approach is small. The two members both contain random walk and median growth rate models. Both these models had 90% prediction intervals which almost always contained the true value (S24 Fig), but which were perhaps too wide. The third model in each sub-ensemble (cubic regression or ETS) often had quite narrow prediction intervals. The forecasts for both of these models around the peak could be quite poor but were reasonable in the run up to the peak and then after the peak.

Comparison of retrospective forecasts to operational forecasts

Our operational ensemble could contain any number of models from one to the number of production models available at that time. Model inclusion in the operational ensemble is a subjective judgement, but we use a model in our weekly operational ensemble if the model predictions are epidemiologically plausible. We see in Fig 6 that, in general, our operational ensemble was of comparable performance to the retrospectively fitted ensembles. We also include a “matched ensemble”. This is an ensemble of the individual retrospective models, where the individual models used to construct the ensemble are the same as the models in the operational ensemble. The matched ensembles typically, but not strictly, outperformed the operational ensembles. This suggests a bias in favour of retrospectively fitted models and sub-ensembles over the operational ensemble as they are fit using revised data and end-of-season hyperparameters. The pcWIS values for the operational ensemble are on the same scale as the pcWIS values for the retrospective sub-ensembles (Fig 6). Over the season the average pcWIS values for COVID-19 and Influenza operational ensemble forecasts were 4.69 × 10 ^− 7 and 5.62 × 10 ^− 7 respectively. For RPS, the mean RPS scores over the season were and for COVID-19 and Influenza respectively. Our operational ensembles occasionally gave large RPS values, especially for COVID-19. These spikes in RPS correspond to weeks where the admissions were stable and we forecast a relatively low probability of a stable trend. That being said, the operational ensemble’s mostly likely trend direction at the national level matched the observed trend direction on the majority of occasions: 56% of COVID-19 trend direction forecasts and 63% of Influenza trend direction forecasts matched the observed outcome. Note that the number of correct mostly likely trend direction predictions is not a proper scoring rule, but is an easy to understandable performance metric for a variety of non-technical stakeholders. Table 2 presents results at all spatial resolutions. Our Influenza RPS values for the operational ensemble were often marginally higher than most of the other retrospective sub-ensembles. Minimum mean, overall mean, and maximum mean scores are available in S4 Table of the supplementary material.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Percentage of occasions the most likely trend direction forecast match the observed outcome by spatial resolution. Influenza forecasts were correct on the majority of occasions. COVID-19 forecasts were not as performant, but were at least as good as random chance (33%) for each geography.

https://doi.org/10.1371/journal.pcbi.1014644.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of how the operational ensemble performs relative to the retrospective 3-model sub-ensembles, the retrospective individual models and the matched ensemble.

The scores are presented over time to demonstrate the varying performance at different epidemic phases and at a national geography for pcWIS (top) and RPS (bottom). The break in the red Operational Ensemble and cyan Matched Ensemble lines are due to the forecasting break which occurs over the Christmas and New Year.

https://doi.org/10.1371/journal.pcbi.1014644.g006

Operational forecasts

Hospital admissions forecasts in the 2024–25 season were broadly accurate. The COVID-19 forecasts were especially accurate; however, the disease dynamics had limited complexity that season. Influenza forecasts were more challenging, especially when the peak was thought to be around the winter break in operations. The operational ensembles are usually better than individual operational models (Fig 2), which is consistent with existing literature [5,30].

There are numerous challenges to real time modelling including data quality issues and models failing due to bugs found in real-time. Code improvements to our modelling workflow over Summer 2024, most notably using the targets package [31], led to an easier-to-maintain collection of modelling pipelines compared to previous years [10]. Another year of experience producing forecasts under pressure led to better real-time decision making; combined with a more mature modelling suite and internal data quality monitoring tools, this resulted in all of our modelling products being delivered on time, or ahead of schedule without failure. We frequently delivered to customers at short notice due to the scheduling of critical system leader meetings.

Notable however, is the Christmas winter holiday period, where there is an agreed pause with users due to changes in surveillance reporting cadences, staff availability and senior officials’ availability. This year, this pause occurred during the peak of the Influenza wave, a period of high-pressure meaning users relied on older forecasts to make decisions. While automation of modelling can improve delivery, capacity of staff trained to assure work is also a key challenge, and forecasts must align to other surveillance data cadences they rely on. The difficulty in this break period is highlighted by the retrospective models poor performance during this time shown in Fig 6, across both pcWIS and RPS.

Effects of models on sub-ensemble scores

We have proposed a method for discriminating models in ensembles, which we tackled by developing “sub-ensembles”. Similar work includes LASOMO (leave all subsets of models out) and LOMO (leave one model out) methods to understand the importance of models in an ensemble [17]. A novel feature of our approach is to understand how a model may impact multiple different ensembles: the utilisation of GAM smooths can be used to understand the expected improvement in a scoring metric and provide uncertainty about these estimates. This approach gives a detailed understanding of which models are most useful in an ensemble during different epidemic phases, relative to other candidate models. It is possible to estimate component model importance and use this to construct a weighted ensemble, which can have performance benefits [32]; this only shows us which models are useful in relative terms rather than treating models as “good” or “bad”.

When discussing requirements with users of our forecasts there was no clear consensus on what geography or scale (relative or absolute) the forecasts were most important for. This is because the purpose of the forecast varies for the different audience members. For some users the accuracy in terms of absolute values was critical, whereas others found the general direction most important. Users of the forecasts have responsibilities across the national, regional, and local scales. These competing priorities are challenging to directly encode, which motivated our use of a Pareto front analysis. This analysis enables us to highlight the trade-off between different models directly on the front, making the choice between performance for different user types more explicit. The Pareto front gives us a much broader view on the model choice problem than our GAM-based averaging approach on a single evaluation metric alone. An alternative approach to this problem, would be to encode multivariate scoring rules, which should be explored operationally as future work [33].

The geographical weights chosen for the weighted Pareto front analysis are challenging to select and represent our view of the average utility across all users of each spatial level of forecasting. A different opinion of the relative utility of each spatial level may produce a different Pareto front of sub-ensembles. While each user group would prefer higher weighting of the geography most relevant to their work, we assume that the national forecast is used more often by local users than visa-versa to understand overall pressure as well as their own. However, due to aggregation of forecasts and different model structures, not all models perform equally well across spatial scales. Some models assume a national trend in admissions, whereas others assume each ICB is independent which would further drive the possible variation in results due to a different weighting scheme.

We selected a model combination size of 3 models for the sub-ensemble analysis to generate sufficient combinations of models to perform this analysis and have meaningful ensembles. The performance of a given sub-ensemble is a combination of 1) the accuracy & precision of each individual model and 2) the interaction between individual models when ensembled, such as whether model properties such as bias and sharpness cancel. As a result, there should be some similarity between individual models on the Pareto front if the sub-ensemble size were to change. However, the variance of scores across sub-ensembles would decrease as the sub-ensemble size increases. A larger cohort of models would be needed to explore this effect in future work.

Our approach is general and can be applied to any collection of forecasts generated at consistent intervals. The approach is sufficiently flexible to allow users to perform the analysis for different scoring rules, performance metrics or forecasting strata.

We presented a new GAM-based method for understanding how models influence ensemble performance, which can guide decisions about which models can or should be dropped from a modelling suite in favour of new modelling approaches. In particular, as a result of this analysis we have deprecated the COVID-19 Gaussian process growth rate model in favour of alternative approaches. Our Pareto analyses showed us which ensembles were useful when we have multiple scoring criteria which cannot be simultaneously optimised. This is a novel application of the Pareto front to retrospective evaluation of real-time modelling tools.

We also saw that our manually chosen ensemble often performed worse than the retrospectively fitted sub-ensembles. However, observing the published forecasts, our predictions are reasonable. This is reassuring to users of UKHSA forecasting products that the predictions generated are high-quality. However, our operational ensembles often contained more models than the retrospectively fitted sub-ensembles (Fig 1). This shows that the largest possible ensemble, such as those used in forecasting competitions and hubs [5,34,35], are not a guaranteed route to improved predictive performance; a recent study advocates for ensembles with components providing complementary information, instead of simply aiming for as large an ensemble as possible [36]. Caution needs to be exercised when opportunistically including models in an ensemble: our ensembles should be diverse and have unique predictive features. This is seen in, for example, our influenza forecasts. It is well-documented in the literature that a diverse ensemble tends to lead to better results [8]. The ETS model tends to have a stable trend and narrow predictive bands, by contrast, the Historic GR model often had a non-stable trend with wide predictive intervals. Other influenza models had different behaviours to these models (S20 Fig).

This research has shown that the complex biological systems of infections over time are difficult to forecast consistently well, with varying model performance over time – shown by the GAM based contribution analysis. Furthermore, we showed that different aspects of the system – absolute scores vs relative change are not equally easy to forecast under each model, with different combinations of models having different skill, as demonstrated by the Pareto Front analysis. We showed that models which depend on growth rate assumptions tend to add value in incline and decline phases, whereas models that assume time series trends or flat forecasts can improve performance when epidemic waves are more stochastically variable. Having the tools available to inspect different aspects of ensemble performance allows public health forecasters to evaluate more rigorously, and therefore draw better conclusions to plan future model combinations.

Limitations

Limitations of our approaches are as follows. We have only used pure-statistical approaches in these ensembles. The ensemble could be improved with the addition of semi-mechanistic approaches which include parameters such as susceptible population sizes and infection hospitalisation rates. Semi-mechanistic approaches are distinct addition to our future ensembles [37,38]. Bespoke solutions can be complex to develop, but packages such as EpiNow2 provide multiple modelling options [39]. As with any scoring-based approach, this work is retrospective, thus cannot be used in real time. However, scoring can be used mid-season to assess relative performance of models and contribution to model performance. Although there is evidence in the literature to suggest 4 or more models construct a robust ensemble [40], we chose sub-ensembles of size 3 due to the limited number of models available for combining.

The GAMs fit to scores for Figs 3 and 4 to understand how individual models contribute to sub-ensemble performance, like all models, have their limitations. Diagnostic plots for the GAMs are given in S14-S16 Figs. Future work should explore strength and limitations of the used of statistical model based ensemble analysis.

The (relatively) poor performance of our operational ensembles compared to retrospective sub-ensembles can be attributed to a few factors. Firstly, not all models were available throughout the season, with some being introduced very late in the season when we had more time to experiment with methods. The matched ensemble shows that even using the same models as in real-time, the retrospective individual models outperformed the operational ensemble. The 2024–25 season posed new challenges for the team as we employed new technologies to increase operational efficiency but simultaneously were often under pressure to deliver forecasts ahead of regular cadence to meet decision makers’ needs. This meant the team had less time to choose and experiment with different operational ensembles, as we had to provide forecasts to customers promptly. We were able to apply expert judgement to our forecasts and tune hyperparameters accordingly, and to include/drop models from our operational ensembles; for COVID-19 this led to comparable performance when comparing the matched ensemble to individual models and sub-ensembles. For Influenza, the matched ensemble often outperformed individual models and sub-ensembles (Fig 6). Furthermore, data is revised in real-time which can degrade model performance operationally when compared to models forecast using retrospectively available data.

Our future work will be to diversify our ensembles by including infection dynamic mechanism, as well as using models which use more long-term historic data to use information such as seasonality to improve modelling. This will allow us to have a rich ensemble at the start of the season and provides a suite of modelling tools for use in the event of an incident or unexpected pandemic. We will expand RSV and norovirus operational ensembles to gain the benefits which ensembles can bring to performance. This will allow us to perform similar analyses for those diseases.

We have already removed models from our suite based on this analysis and could use this analysis to further improve future operational ensembles. Our operational ensembles are currently unweighted, thus one idea is to construct ensemble weights for a future season or dynamically within season [41,42]. For example, the values could be used to calculate time-varying ensemble weights [8]; one challenge will be understanding how to use values for different scoring rules, forecasting outputs, or spatial resolutions to construct a single set of weights. As the size of our ensemble grows, we can use our Pareto analyses as a first-pass for model evaluation: models which do not feature in a Pareto front should be considered for deprecation.