Ethics statement

The study was undertaken as urgent public health action to support Australia’s COVID-19 pandemic response. The study obtained data under the National Health Security Agreement for the purposes of national communicable disease surveillance. Contractual obligations established strict data protection protocols, as agreed between the University of Melbourne and sub-contractors and the Australian Government Department of Health and Aged Care. Oversight and approval for use in supporting Australia’s pandemic response, and for publication, were provided by the data custodians (represented by the Communicable Diseases Network of Australia, CDNA). The ethics of the use of these data for these purposes, including publication, was agreed by the Department of Health and Aged Care with CDNA. Consent was not obtained, these data were anonymised at the source and were not reidentifiable. All methods were carried out in accordance with the relevant guidelines and regulations.

COVID-19 case data

For target data, we used de-identified line lists of reported cases for each Australian state and territory, extracted at each week of the study period from the National Notifiable Disease Surveillance System (NNDSS). The data included the jurisdiction (state or territory), the date of symptom onset (where available), the date when the case notification was received by the jurisdictional health department, and whether the infection was epidemiologically deemed to be acquired locally or overseas. We imputed missing symptom onset dates and estimated reporting delays using a time-varying delay distribution to characterise the duration between symptom onset and case notification [28]. Symptom onset dates were reported for 82% of cases in 2020. As Australia’s response pivoted to “reopening” the country, and daily case counts greatly increased, symptom onset reporting decreased to 48% of cases in 2021, and 38% in 2022. Nationally aggregated daily case counts are shown in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. National daily confirmed COVID-19 cases and participating teams.

Top: Australian daily confirmed COVID-19 cases for the 2020–2022 calendar years. Dominant strain(s) are identified by alternating backgrounds (white and shaded), as determined by epidemiological assessment in situation reports [13]. Bottom: The periods over which each team contributed to the ensemble forecast. Team names and modelling approaches are listed in Table 2.

https://doi.org/10.1371/journal.pcbi.1014199.g001

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. The teams that contributed to the ensemble forecasts over the study period.

https://doi.org/10.1371/journal.pcbi.1014199.t002

Forecasting teams and models

Four modelling teams contributed forecasts for inclusion in the ensemble, using different types of model (see Table 2). This included two stochastic compartmental models, developed by the Defence Science and Technology Group (DST) and the University of Melbourne (UoM), which incorporated different assumptions about effective reproduction numbers and population immunity; a branching process model developed by the University of Adelaide (UoA); and an autoregressive model developed by Monash University (MU). We began with three teams contributing to the ensemble (MU, UoA, and UoM) and the DST team began participating in August 2021. After a four-month period where all four teams contributed to the ensemble forecast (2021-11-25 to 2022-03-22, inclusive), the UoA team stopped participating due to limited staff availability. Throughout this manuscript we will refer to the models, and the forecasts that they produced, using the acronyms listed in Table 2 and used in this paragraph.

The UoM team began contributing national forecasts in April 2020 [29], using a model adapted from a long-running seasonal influenza forecasting project [30–33], while the other teams created new models in response to the emergence of local COVID-19 transmission. All models were subject to ongoing development throughout the study, and these developments are documented in our weekly situation reports [13]. When new model features were introduced, retrospective forecasts were validated against previously-reported data before the new model iteration was considered for inclusion in the ensemble.

Teams were able to use other data sources, in addition to the Australian COVID-19 case data. The MU team used global COVID-19 case data from Johns Hopkins [34] to inform their global autoregression model. The DST and UoA teams made use of Google mobility data to inform transmission estimates, the DST and UoM teams made use of behavioural survey data to inform case ascertainment [35], and the DST, UoA, and UoM teams made use of vaccination data from the Australian Immunisation Registry to inform vaccine coverage in their models. Where multiple teams used a common data source, they did not necessarily do so in an identical manner. For example, the DST and UoM teams made different assumptions about how much of the variation in case numbers reported by day-of-week was attributed to transmission versus case ascertainment.

Inclusion/exclusion criteria

If there were known, or strongly suspected, data quality issues for any jurisdiction that could not be resolved with post-processing, we did not include ensemble forecasts for those jurisdictions in the weekly situation report. Models were also excluded when the forecasts produced for that week were assessed to be inappropriately sensitive to trends in the most recent case counts, which were known to be subject to time-varying reporting delays [28]. This decision-making process was facilitated by direct communication with data custodians in the relevant jurisdictions and national public health committees, which enabled rapid awareness of data quality issues.

Each week, the modelling teams generated forecasts from the provided data extracts and shared the results for peer review by the broader analysis team (comprising all modelling teams and other analysts, which we will refer to as the “consortium” hereafter) in a private online discussion board (in a Slack workspace). Peer review findings and retrospective performance results for each model were then tabled for discussion in weekly online consortium meetings, where we collectively reached agreement on which models, and model iterations, to include in the ensemble forecast for each jurisdiction that week (see Fig 2). This approach of ongoing discussion and reaching a collective consensus has been shown to improve the performance of human predictions in domains such as geopolitics [36].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. A flowchart of the weekly peer review and inclusion/exclusion decision process.

Shown here is an example for the model iteration(s) provided by a single modelling team in a given week. The dotted box indicates activities and decisions made by the entire consortium.

https://doi.org/10.1371/journal.pcbi.1014199.g002

Models were excluded when their outputs differed dramatically from the consortium’s collective expectations and the current epidemiological assessment, or when a technical issue was identified by the modelling team or by the peer review process. Importantly, each model was evaluated against the most recent data rather than competing against the other models (e.g., via benchmarks such as skill scores), with the purpose of deciding whether the model outputs were likely to be a reliable basis for decision-making [37]. In other words, because we did not believe that any participating model truly captured the real-world system, we therefore did not use performance metrics to undertake model selection. Rather, teams were able to use these metrics to inform model development. The outcomes of these collective decisions are captured in publicly available reports [13].

Ensemble forecast construction

Model forecasts were submitted in the form of 2000 simulated trajectories of daily case counts over the 28-day forecast horizon for each of the eight jurisdictions. We generated equal-weight ensemble forecasts for each jurisdiction by sampling trajectories from each team’s forecasts for that jurisdiction.

When a team provided forecasts from multiple model iterations, we treated these forecasts as arising from a single model with a discrete hyper-parameter (i.e., having a different value for each model iteration), and averaged over the iterations when including the team’s forecasts in the ensemble. We believe that this was the most sensible approach to take, particularly when our primary interests were to evaluate how well each model performed relative to the other models, and relative to the ensemble. The modifications made to each model throughout the study period were much smaller than the methodological and structural differences between the models. Accordingly, the addition or removal of one model iteration impacted the relative weights of all model iterations from that team, but did not affect the weights allocated to other teams.

The use of equal weights for each team was strongly motivated by the risk of sub-optimal forecasts supporting a poor decision/recommendation. An ensemble consistently performs better than a single model, and there is a large body of evidence in the forecasting literature that it is very hard to out-perform an equal-weight (“simple average”) ensemble; this is known as the “forecast combination puzzle” [22,23]. This reliability of a “simple average” ensemble was more valuable for our purposes than any potential (and likely very small) gain in performance from using a more complex weighting scheme.

For a given week w, each team provided forecasts F_t(w) for M model iterations . Teams provided k = 2000 sample trajectories for each model iteration, and we accepted a subset M_t(w) of these iterations for inclusion in the ensemble. Weights were distributed equally across the N(w) teams with at least one model iteration included in the ensemble forecast:

The ensemble forecast distribution F_ens(w) was defined to be the weighted sum of model iteration forecasts :

As described above, we held weekly meetings in which we reviewed individual model forecasts and the ensemble forecast prior to including these forecasts in our weekly situation reports to government. Forecast performance was evaluated visually (using daily credible intervals) against previous weeks’ forecasts and the most recent data and quantitatively, as we explain in the following sections.

Forecast performance

We quantified forecast performance using the Continuous Ranked Probability Score (CRPS). CRPS is a generalisation of the Mean Absolute Error (MAE) to probabilistic forecasts, and evaluates the entire forecast distribution F against the observed data y:

where denotes the Heaviside step function.

It is interpreted on the natural scale of the data y, with a minimum value of zero indicating a perfect point forecast. CRPS is a proper scoring rule, which means that its expected score is minimised when the forecast distribution matches the distribution of the observations, and is the most commonly used scoring rule in the evaluation of probabilistic forecasts. When evaluated for log-transformed distributions against log-transformed observed values , it can be interpreted as a probabilistic version of the relative error [38].

We used CRPS skill scores to compare the relative performance of each team’s forecasts F_t against that of the ensemble F_ens, for the observed data y:

The maximum skill score is 1 (obtained for a perfect point forecast F_t). Positive skill scores indicate that the forecast F_t has out-performed the reference forecast (in this case, the ensemble forecast ) while negative skill scores indicate that the forecast F_t has performed worse than the reference forecast.

Forecast calibration

Forecasts were visualised as median-centred credible intervals for daily case counts in each jurisdiction. For example, the 50% credible interval for a forecast distribution F was calculated as the inter-quartile interval [0.25, 0.75], and the 95% credible interval as the inter-quartile interval [0.025, 0.975].

Credible intervals represent the plausibility that the observed data y will fall within the interval range. The coverage of a given X% credible interval is the observed proportion of the data y that fall within this credible interval. When evaluated over a range of credible intervals, this quantifies how well calibrated the forecasts F are, with respect to the observed data y.

A set of forecasts is probabilistically calibrated with respect to observed data if there is statistical consistency between these two sets, such that X% of the data y fall within the X% credible interval of the forecasts F [39].

Forecast bias

We also evaluated forecast bias B(F, y) as per Funk et al. [40]:

An unbiased forecast would assign equal probability mass to values above and below the observed data y (B(F, y) = 0), while a completely-biased model would assign all probability mass above y (B(F, y) = 1) or below y (B(F, y) = −1).

Models included in each ensemble forecast

As shown in Table 3, all models were regularly included in the ensemble forecasts. The DST and UoM teams often contributed forecasts generated from multiple model iterations, and when models were excluded from the ensemble it was most often a single model iteration from either team. The most common reason for exclusion was for a model to exhibit large and systematic biases in the forecasts produced in previous weeks (i.e., retrospective posterior predictive checks). In several instances the UoA team encountered computational issues and were unable to provide forecasts in a timely manner. Due to staff leave, some teams did not provide forecasts over the Christmas 2021 holiday period (during “Omicron BA.1”). At each week of the study period, we accepted forecasts from multiple teams (with a minimum of two teams) and always produced an ensemble forecast, since an ensemble is expected to out-perform any individual model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. A summary of models included in the ensemble forecasts. The percentage of ensemble forecasts for which each model was included, and the number of models included in each ensemble forecast (minimum, maximum, and mean), reported separately for each period in which particular strain(s) dominated. Note that models could be excluded from the ensemble for various reasons, from being assessed as inappropriately sensitive to trends in the most recent data, or otherwise producing suspect predictions, to being unavailable due to technical issues and/or delays in model development (see “Inclusion/exclusion criteria”).

https://doi.org/10.1371/journal.pcbi.1014199.t003

Summary of forecast performance

Ensemble forecasts for the months of March, April, and May 2022 are shown in Fig 3 for four of the eight Australian jurisdictions (forecasts for all jurisdictions over the entire study period are shown in Figures A–E in S1 Text). These example forecasts demonstrate key features that were exhibited by the ensemble forecasts over the entire study period (2020–2022). In brief, the ensemble was generally in good agreement with the data. Consistent with findings from COVID-19 forecasting studies in other countries [6–11,41], it was an ongoing challenge to predict the timing of change points. The forecasts exhibited a tendency to overshoot peaks, and to undershoot the onset and ending of individual waves. In particular, the forecasts tended to predict substantially larger peaks than were ultimately observed. However, as shown in the 4-week forecasts in Fig 3, the observed peaks still fell within the forecast credible intervals. The extremely broad intervals for the Victorian forecast for 15 March 2022 arise from a single model, which was assessed as highly uncertain but consistent with the data, and included in the ensemble that week.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Ensemble forecasts over the months of March, April, and May 2022.

Forecasts are shown for four Australian jurisdictions: New South Wales (NSW); Queensland (QLD); South Australia (SA); and Victoria (VIC). The left columns shows 1-week forecasts for each ensemble forecast (1–7 days ahead), the middle column shows 2-week forecasts for every second ensemble forecast (1–14 days ahead), and the right column shows 4-week forecasts for every fourth ensemble forecast (1–28 days ahead). Shaded regions illustrate the 50%, 60%, 70%, 80%, 90%, and 95% credible intervals, and black lines depict the data as reported at the end of the study period.

https://doi.org/10.1371/journal.pcbi.1014199.g003

Ensemble forecast coverage (shown in Fig 4) was reasonable across most pandemic waves. The best coverage was observed during the “Pre-Delta” and “Delta” periods where, for example, almost exactly 50% of the ensemble 50% credible intervals included the true value. During this time, Australian jurisdictions responded rapidly to local disease activity in pursuit of strong suppression [14], and while the precise impact of these interventions was challenging to predict, there was only limited local COVID-19 transmission and case counts remained low. Australia then pivoted from pursuing strong suppression to reopening and accepting substantial levels of local COVID-19 transmission. In the subsequent “Omicron BA.1” period, the models failed to anticipate the massive surge and rapid decline in daily case counts; forecasts were overly confident and exhibited a tendency to first undershoot and then overshoot the true case counts. Forecast coverage improved markedly for the “Omicron BA.2/BA.5” and “Multiple variants” periods, with a slight tendency towards being overly uncertain, and was almost as good as during the “Pre-Delta” and “Delta” periods. This was likely due to a combination of having quantitative data about trends in test-seeking behaviour [35] (to inform the mechanistic models) and the slower rates of growth and decline in cases, relative to the “Omicron BA.1” period. For periods where the individual models exhibited marked differences in forecast coverage, the ensemble forecast coverage was similar to, if not better than, the coverage of the best individual model.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Observed forecast credible interval coverage.

Observed coverage of the forecast credible intervals for each individual model, and for the ensemble, with respect to the ground truth case counts as reported after the end of the study period. Results are reported with respect to each dominating strain (refer to Fig 1). Dashed lines indicate perfect coverage. Points below the dashed lines indicate overly certain forecasts, points above the dashed lines indicate overly uncertain forecasts.

https://doi.org/10.1371/journal.pcbi.1014199.g004

As might be expected, observations that lay further into the future were harder to predict. Forecast performance was highest for short lead times, and steadily decreased as the lead time increased. This was true not only on aggregate, but also for the majority of individual observations. As shown in Table 4, for a given observation the forecast performance consistently improved from the initial prediction (when the observation was 22–28 days ahead) to the final prediction (when the observation was 1–7 days ahead).

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Ensemble forecast (log-transformed) CRPS values. These values can be interpreted as a probabilistic version of relative error. Mean values are shown for lead times of 1–7 days (“1 week”), 8–14 days (“2 weeks”), 15–21 days (“3 weeks”), and 22–28 days (“4 weeks”) in each jurisdiction, and across all jurisdictions (bottom row). The forecasts performed best at the shortest lead times (i.e., when CRPS values were smallest) and performance decreased in proportion to lead time (i.e., CRPS values increased).

https://doi.org/10.1371/journal.pcbi.1014199.t004

Forecasts exhibited substantial biases around epidemic peaks. This is illustrated in Fig 5, which shows the bias for each model and the ensemble for forecasts generated in the 4 weeks leading up to, and the 4 weeks after, each observed peak. Forecasts generated 2–4 weeks prior to observed peaks undershot the data (negative bias) more often than they overshot the data (positive bias). In contrast, forecasts generated from 1 week prior to 1 week after the observed peaks had a tendency to overshoot the data (positive bias), which highlights the challenge of identifying peaks in real-time when surveillance data are subject to ascertainment biases and reporting delays [12].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Forecast bias around peaks.

Forecast bias for each model and the ensemble, shown for forecast origins within 4 weeks either side of the largest peak (minimum 200 cases) observed in four jurisdictions (columns) for each dominant strain (rows). Positive bias indicates a tendency to overshoot the data, negative bias indicates a tendency to undershoot the data. The “Multiple variants” period is not included in the figure, because case counts were generally flat and the observed peaks occurred towards the end of the study period. See Figure F in S1 Text for forecast bias around these peaks over all eight jurisdictions.

https://doi.org/10.1371/journal.pcbi.1014199.g005

No model consistently produced the least biased forecasts for weeks preceding or following the observed peaks. When there was a wide spread in bias, the ensemble forecast was often less biased than most, if not all, of the individual model forecasts. Similar trends are evident in individual model and ensemble forecast performance (see Figures G and H in S1 Text); no model consistently produced the most accurate forecasts for weeks preceding or following the observed peaks, and the ensemble tended to mostly out-perform individual models.

While the ensemble forecasts struggled to accurately predict peak sizes and timing (evident in the measure of forecast bias, discussed above), in most weeks preceding or following an observed peak, a proportion of the forecast trajectories were positively correlated with the pre-peak and post-peak case counts (Figure I in S1 Text). This indicates that the forecasts could at least partially characterise the qualitative trends in future case counts (i.e., whether case counts would increase, decrease, or remain stable).

Relative model performance

Recall that at each week of the study period, sample trajectories of daily case counts were provided over four-week horizons for each Australian jurisdiction. For each individual daily case count prediction, we ranked the individual models and the ensemble using CRPS on log-transformed values [38]. The results for days where at least one case was reported are shown in Fig 6. Collectively, the individual models were more likely than the ensemble to be the top-ranked forecast, but no single model dominated the top ranking. The strength of the ensemble was that it was most often the best or second-best forecast. Skill scores for each model, relative to the ensemble, were consistently below zero, further reinforcing the finding that no model consistently out-performed the ensemble (Fig 7). The model rankings for each ensemble forecast (Figure J in S1 Text) also demonstrate that there were no obvious predictors of which model would perform the best for a given time period or epidemiological context.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Cumulative rankings for each individual model and for the ensemble.

Rankings are shown for days where at least one case was reported, calculated using CRPS on log-transformed values. While the ensemble was rarely the top-ranked model, it was at least 2nd-best approximately two-thirds of the time (66.7%), and was almost always in the top three (98.7%).

https://doi.org/10.1371/journal.pcbi.1014199.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Distribution of skill scores for each individual model, measured against the ensemble.

Results are shown for days where at least one case was reported (around 53% of all observations). Skill scores were calculated using CRPS on log-transformed values [38]. Positive values indicate performing better than the ensemble (with a maximum skill of 1 for a perfect point forecast), negative values indicate performing worse than the ensemble, and the vertical dashed line marks the threshold skill of zero (performance equal to the ensemble). Stars indicate mean skill scores. The MU model performed better than the other models, with a median skill score of 0.002. Mechanistic models performed well for extended periods of zero cases (median skill scores of 0.75–0.97) but were otherwise outperformed by the ensemble, with median scores of −0.07 (UoM), −0.14 (DST), and −0.19 (UoA).

https://doi.org/10.1371/journal.pcbi.1014199.g007

Pre-Delta: May to October 2020

We now highlight how the forecasts supported public health activities in each phase of the study period, beginning with the Pre-Delta phase. This first period of sustained local COVID-19 transmission began in mid-2020, where the majority of cases occurred in the state of Victoria. Ensemble forecasts for this period are presented in Figure A in S1 Text. The performance of the UoM forecasts over this second wave has already been reported [12], and we begin by comparing the ensemble forecasts to the UoM forecasts over this wave.

In the weeks prior to the peak in daily cases (3 August 2020), the upper bounds of the UoM forecasts substantially overshot the true peak, and predicted sustained exponential growth in cases. This exponential growth was tempered in the ensemble, which averaged over three models (MU, UoA, and UoM). Accordingly, the ensemble outperformed the UoM forecasts (which had a mean skill score of -0.37, see Figure K in S1 Text), even though the ensemble forecasts tended to undershoot the data. By late July, we identified that case ascertainment and reporting delays had substantially increased, and our right-truncation adjustments were under-estimating the true case counts in the most recent days prior to each forecast. In the weeks after the peak, the UoM forecasts confidently predicted a sustained decrease in cases, while the MU and UoA models exhibited broader credible intervals and tended to overshoot the data. As a result, the UoM forecasts performed better than the ensemble forecasts (mean skill score of 0.31, see Figure K in S1 Text).

Daily case incidence in Victoria steadily decreased from the August peak. In early September the Victorian government announced a gradual easing of restrictions, subject to reaching specific 14-day moving average case thresholds. Because the ensemble forecast comprised individual trajectories from each contributing model, we were able to calculate the proportion of all trajectories that satisfied a given 14-day case threshold on each date in the forecast horizon, and reported this as the daily probability of achieving each target threshold. These predictions were provided to the Victorian government throughout September and October (and are shown in Moss et al. [12]).

Delta: June to December 2021

The onset of the Delta variant in June 2021 marked the first instance of sustained local COVID-19 transmission for most Australian jurisdictions, and public health responses focused on preventing local transmission. Accordingly, daily case counts remained very low in most jurisdictions, only exceeding 25 cases per day in the Australian Capital Territory (peak of 51), New South Wales (peak of 1,495), and Victoria (peak of 1,955). Ensemble forecasts for this period are presented in Figure B in S1 Text.

Perhaps the single greatest value of the forecasts in this period was to demonstrate how rapidly case counts could increase if local transmission was left unchecked.

In May 2021, New South Wales Health set a target of achieving 80% coverage of the adult population with two vaccination doses by the end of December 2021 [42]. This was followed in August 2021 by an interim target of 70% coverage of the adult population with two doses, to encourage vaccine uptake and to begin easing restrictions for fully vaccinated people [42]. As a consequence, there was substantial vaccine roll-out over the 4-week forecast horizons from August onward. This was accompanied by rapid model development from the DST and UoM teams, with the effects of vaccination incorporated into these models in September 2021. The statistical MU model did not require explicit adjustment for vaccination effects, although vaccination was indirectly accounted for in the model’s structure and updated parameter estimates based on post-vaccine global case counts.

The forecasts were generally in very good agreement with the data in each jurisdiction. In particular, the mean CRPS values for New South Wales and Victoria were consistently lower than for the Pre-Delta wave in Victoria (see Figure G in S1 Text).

Omicron BA.1: January to March 2022

This wave coincided with a national pivot from pursuing strong suppression to reopening and management of substantial levels of local COVID-19 transmission. Ensemble forecasts for this period are presented in Figure C in S1 Text.

The forecasts struggled to predict the massive surge and subsequent decline in daily case counts, as demonstrated by decreased coverage (Fig 4), increased bias (Fig 5, Figures L and M in S1 Text), and worse coverage than other waves (Figures N and O in S1 Text).

This was due, at least in part, to mechanistic models failing to account for significant reduction in vaccine protection against Omicron, rapid changes in case ascertainment [35], and reduced mixing due to school holiday effects over December and January. Also in January, the vaccine roll-out began for children aged 5–11 years, and booster doses were recommended for at-risk individuals [43].

A primary concern in all jurisdictions was the timing of the epidemic peak, because the substantial increase in cases was impacting workforce capacity and causing disruptions to food supply chains, which in turn prompted panic buying [44]. The forecasts consistently predicted that the peaks would occur around 2 weeks later than they actually occurred (and for the mechanistic models, this was robust to adjustments such as relaxing assumptions regarding case ascertainment and immunity). Despite this inaccuracy, the forecast predictions that case activity would peak and begin to decrease in a matter of weeks remained an important and useful message for government.

Omicron BA.2/BA.5: April to August 2022

This period saw the gradual replacement of Omicron BA.1 with Omicron BA.2 and BA.5, and case counts in all jurisdictions steadily decreased after the large Omicron BA.1 peaks. Ensemble forecasts for this period are presented in Figure D in S1 Text.

The forecasts reported on 11 June, generated from daily case counts up to 31 May (inclusive), predicted downwards trends in all jurisdictions, consistent with the trends in the most recent case counts. On 16 June we received additional genomic data from New South Wales that allowed us to estimate the transmission advantage of Omicron BA.5, relative to Omicron BA.1 and BA.2. By incorporating this transmission advantage into the mechanistic models, our next forecasts predicted upwards trends in all jurisdictions despite no such trend in the reported aggregate data. Consistent with these predictions, daily case counts began to increase later in June, and all jurisdictions experienced a peak in July (see S4 Fig in S1 Text).

By predicting this inflection point before it occurred, and by doing so based on appropriate interpretation of relevant local data, these forecasts helped convince public health stakeholders that the current downwards trend in cases would not be sustained.

Multiple variants: September to December 2022

Following the peak and decline of Omicron BA.5, Australia experienced circulation of numerous COVID-19 variants, with no single variant becoming dominant. Data from international contexts provided evidence that many of these variants had growth advantages over Omicron BA.1, BA.2, and BA5, although it was unclear how these estimates might translate to Australia’s immune landscape. Ensemble forecasts for this period are presented in Figure E in S1 Text.

Relative to the preceding Omicron BA.2/BA.5 period, the forecasts exhibited similarly good coverage (Fig 4) and performance (see Figure P in S1 Text). When case counts began to increase in November, the forecasts were confident that the epidemic peaks would be similar in size, or smaller than, the Omicron BA.5 peaks. This was a reassuring message that would prove to be accurate, and brings us to the end of our study period.

Communication of outputs

Over the 3.5 years of work summarised here, an ongoing concern was to ensure that model predictions and other quantitative outputs were reported in such a way that our stakeholders (spanning state, territory, and national governments and public health committees) would be able to interpret them appropriately and accurately act on and communicate their implications. While we did not have the capacity to undertake formal assessments of how the 152 weekly situation reports that we produced over this study period were interpreted, as described by McCaw and Plank [45] we did have open communication channels with our stakeholders and were able to explore and refine our reporting and visualisations in an iterative manner.

Fig 8 shows several examples of visualisations included in these reports. For example, our primary form of communicating the forecasts was daily credible intervals (see, e.g., Fig 3). However, this form can obscure the predicted size and timing of an epidemic peak, and so we explored the use of density plots to extract these features from individual forecast trajectories (Fig 8A). Another simple, yet surprisingly useful, modification to the daily credible interval figures was to overlay several randomly-selected forecast trajectories (Fig 8B), which helped to avoid the credible interval contours being misinterpreted as case count trajectories. Finally, plotting past forecasts against the most recent data (Fig 8C) was a simple way of communicating recent forecast performance that provided stakeholders with a qualitative basis for deciding how much trust to place in the current forecasts.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Example figures from weekly situation reports.

A: Density plots showing predicted peak timing and peak case incidence, from the report produced on 24 March 2022. B: Ensemble forecasts of new daily local cases, showing credible intervals and sample trajectories, from the report produced on 25 November 2022. C: Retrospective evaluation of ensemble forecasts from the previous week against new data (red points), from the report produced on 25 November 2022.

https://doi.org/10.1371/journal.pcbi.1014199.g008

Principal findings

Similar to findings reported in COVID-19 forecasting studies in other countries [6–11,41], the use of an ensemble to combine forecasts from multiple models improved forecast performance and reliability (Figs 4–6). Our ensemble forecasts provided information that supported public health responses throughout the study period, from the early periods of preventing local transmission, to the later periods of sustained local transmission and concerns about the burden placed on healthcare systems. Notable outcomes during the study period include:

• Pre-Delta: we used the forecasts to predict when 14-day average case targets would be achieved, which would trigger the gradual easing of mobility restrictions in the state of Victoria.
• Delta: this was the first instance of sustained local transmission for most jurisdictions, and the forecasts were useful for illustrating how rapidly case counts could increase if transmission was not curtailed.
• Omicron BA.1: the peak occurred 1–2 weeks earlier than the forecasts predicted, but forecast predictions that activity would peak and then decline in a matter of weeks was reassuring in the face of workplace absenteeism that disrupted supply chains.
• Omicron BA.2/BA5: the forecasts were able to correctly predict an increase in cases while the current case counts were decreasing, and this was a particularly useful message for health protection committees.
• Multiple variants: the forecasts accurately predicted that the peaks in late 2022 would not be larger than the Omicron BA.5 peaks.

Study strengths

The ensemble approach allowed us to readily incorporate forecasts contributed by teams across Australia, and provided a framework for rapid evaluation of new model iterations for potential inclusion in the ensemble. The weekly performance evaluations and model inclusion/exclusion decisions relied on measures of forecast error (using CRPS) under the assumption that existing policies would persist for the entire forecast horizon. However, as we have described above, the ensemble forecasts influenced policy decisions throughout the study period, and some of these decisions directly impacted local COVID-19 transmission and/or case ascertainment. Accordingly, when policy decisions were understood to have influenced the case data reported over a forecast horizon, we factored these effects into our analyses. In particular, where the difference between forecasts and reported data were consistent with the likely effects of the policy decisions, we considered this a successful outcome [46].

We deliberately chose to use equal-weight ensemble forecasts and, as we explain in the methods section, each model was evaluated against the most recent data rather than competing against the other models for inclusion in the ensemble [37]. As we have discussed in the results section, a model’s ranking in recent weeks was not a reliable predictor of that model’s ranking in future weeks. The strongest finding we have regarding the rankings is that the ensemble was either the best, or second-best, more often than any individual model. Both findings are consistent with ensemble COVID-19 forecast evaluations in other countries [9,10].

Finally, we note here that the ensemble forecasts presented here also served as an input for COVID-19 hospital bed occupancy forecasts (December 2021 to December 2023) that further supported Australia’s public health responses [19].

Study limitations

While pursuing the prevention of local transmission, Australian jurisdictions maintained high testing levels and the proportion of infected persons that were identified as cases was likely to be both very high, and to remain relatively constant. The ensemble forecasts performed very well during this period (the “Pre-Delta” and “Delta” phases).

When Australia transitioned to re-opening, case ascertainment was substantially reduced, and was challenging to estimate in near-real-time [35]. By February 2022, more than 94% of people over the age of 16 were fully vaccinated [47], and while the mechanistic models were subject to numerous adaptations and refinements to account for vaccination coverage, booster vaccinations, and waning immunity, throughout 2022 it became more challenging to estimate population susceptibility to the emerging variants of concern. This was due to an increasingly complex population immune landscape, resulting from heterogeneous levels of vaccination and immunising exposure, and challenges in estimating the immunogenicity of emerging variants.

However, despite these challenges, the inclusion of both mechanistic and statistical models in the ensemble substantially improved the forecast performance and reliability across the entire study period. The only period where the forecast did not capture the case data was the rapid emergence and decline of Omicron BA.1, which occurred as Australia pivoted from pursuing strong suppression to reopening, accepting substantial levels of local transmission, and rapid antigen tests were approved for self-testing and reporting [48].

Two of the three teams that participated during the “Omicron BA.1” period (DST and UoM) used mechanistic models that explicitly incorporated case ascertainment and immunity dynamics. These teams explored adjustments to case ascertainment and immunity assumptions, on the grounds that this might result in smaller and earlier predicted peaks, closer to the observed peaks. However, both teams reported that when these adjusted models were fit to the reported (pre-peak) case data, estimates for other parameters also changed in response to these adjustments, and this resulted in similar forecasts to the original models (i.e., peaks were larger and later than observed).