Fig 1.
Weekly incident reported COVID-19 cases per 100K population, nationally (in black) and per state/territory/DC (in gray), over time in panel A. Panel B shows a subset of COVIDhub-4_week_ensemble forecasts (in green) over time, with the median predictions represented as lines and points and the 95% prediction intervals in bands. Reported incident cases (counts per week) are shown in gray. In both plots, the black, dashed vertical line shows the date that public communication of the case forecasts was paused.
Fig 2.
Percent of weeks with complete submissions for all sets of team forecasts, scaled, pairwise relative Weighted Interval Score (rWIS; see Methods for description), observed 95% prediction interval coverage, by geographical scale of submitted forecasts.
Teams are sorted by increasing state/territory/DC rWIS values.
Fig 3.
Expected and observed coverage rates for central 50%, 80% and 95% prediction intervals aggregated over time and horizon for national forecasts (panel A), state/territory/DC forecasts (panel B), and the largest county forecasts (panel C). The dashed line represents optimal expected coverage. Team forecasts that had closer to nominal coverage than the COVIDhub-4_week_ensemble model at all three coverage levels are labeled on the right side of the plots.
Fig 4.
Scaled, pairwise relative Weighted Interval Score (rWIS) (see Methods for description) by spatial scale for sets of team forecasts that submitted forecasts for the US nation, states/territories/DC, and all US counties.
WIS is averaged across all horizons. The COVIDhub-baseline model has, by definition, a rWIS of 1 (horizontal dashed line). Teams are ordered by increasing state/territory/DC rWIS with the most accurate model on the left. Points for each team are staggered horizontally to show overlapping WIS values.
Fig 5.
Scaled, pairwise relative Weighted Interval Score (rWIS; see Methods for description) by location for national and state/territory/DC forecasts, averaged across all horizons through the entire analysis period.
National estimates are displayed first, followed by jurisdictions in alphabetical order. Team forecasts are ordered by increasing average state/territory/DC rWIS.
Fig 6.
Forecast accuracy over time, aggregated by geographic units, forecast horizon, and prediction date.
Panels A-C show average Weighted Interval Score (WIS); panels D-F show 95% prediction interval coverage. The black, dashed vertical line in all panels shows the date that public communication of the case forecasts was paused. The black, dashed horizontal line in panels D-F shows nominal 95% prediction interval coverage. National level forecasts are presented in A and D, state/territory/DC forecasts in B and E and large county forecasts in C and F.
Fig 7.
Estimated marginal mean Weighted Interval Score (WIS) and 95% confidence intervals for mean cases from team-specific GEE models for all 51 jurisdictions (Panel A). The 95% confidence intervals for the COVIDhub-baseline model are shown in dashed red vertical lines. Panel B presents each team’s estimated marginal mean WIS per phase, scaled to the COVIDhub-baseline model’s estimated marginal mean WIS for all epidemic phases. Teams with higher estimated marginal mean WIS values (i.e., greater than 1.0) are presented in shades of orange while teams with lower estimated marginal mean WIS (i.e., less than 1.0) are shown in shades of green. Forecasts for a team in a particular phase are marked with an asterisk (*) if the 80% confidence interval of the expected WIS outcome (normalized and on the log scale) was estimated by a model to be lower than the average expected WIS of the COVIDhub-baseline model across all phases. Panel C shows each team’s mean 95% prediction interval coverage in each epidemic phase.
Fig 8.
Percent of forecasts with predicted increasing trajectory per epidemic phase (Panel A), predicted stable or uncertain trajectory per epidemic phase (Panel B), and predicted decreasing trajectory per epidemic phase (Panel C). In each plot, epidemic phase labels are in bold when they correspond with the predicted direction of the forecast.