Empirical Bayes.

The Empirical Bayes (EB) model we draw upon is a framework developed by Brooks et al for predicting epidemics by relying on slightly modified versions of past epidemics to form possibilities for the current season [9, 10]. With this approach, we first modelled previous flu seasons using non-parametric smoothed piecewise quadratic curves generated with the cv.trendfilter method of the genlasso R package [11]. This gave us a set of smoothed trajectories {f^s} for each flu season s in the training data. Next, we drew at random with equal probability one trajectory from {f^s} and applied a series of transformations to this curve, resulting in a curve . The transformations shift the epidemic curve’s peak height, peak location, and pacing toward the peak, respectively. For these transformations, we randomly sampled a peak height, peak location, and pacing parameter, where candidate peak heights and locations came from the smoothed trajectories f^s and the pacing parameters came from a uniform distribution U[0.75, 1.25]. We then assigned a likelihood weight to the transformed curve based on how closely the curve approximated observed ILI up to the current week of the season. We lastly injected noise to the transformed curve, where the noise terms were drawn from a normal distribution with σ_s being a noise level derived from the trajectory f^s.

The EB prediction of ILI visits ℓ weeks from week t, , was the weighted median of N = 10⁵ random samples of transformed trajectories, with the likelihood weights w described above. A prediction interval was also constructed from percentiles of the N weighted samples. For the weighted median point prediction, we ranked the forecast ILI at period t+ ℓ from the N samples from smallest to largest, and computed: (2)

Autoregressive integrated moving average.

The Autoregressive integrated moving average (ARIMA) model estimates future periods’ ILI positive visit counts as a function of previous observations and forecast errors. An ARIMA(p, d, q) model is specified by three parameters: an autoregressive order term p, a degree of differencing d for making the time series stationary, and, the order of the moving average q. We used seasonal and trend decomposition using locally estimated scatterplot smoothing (STL) to remove seasonality from the raw weekly ILI counts before fitting an ARIMA(2,0,1) model. The analysis was implemented via the stlm method of the forecast R package [12]. Determination of the number of time lags p = 2, degree of differencing d = 0, and order of moving average q = 1 was based on optimization of the Akaike Information Criterion (AIC) [13].

Upon seasonally adjusting the ILI data with STL, the ARIMA(2,0,1) prediction of ILI positive visits ℓ weeks from week t, , is given by the forecasting equation: (3) where are the estimated model parameters and S_{t+ ℓ} is the seasonal component which was computed using STL and removed before fitting the model. The prediction interval is given by where is an estimate of the standard error of the ℓ weeks ahead prediction distribution and c comes from the interval coverage probability assuming a normal distribution of forecast errors. Note that the prediction intervals are estimated from the seasonally-adjusted data but may be too narrow as they ignore uncertainty associated with the STL estimation of the seasonal components.

Quantile regression forest.

Random forests are collections of bagged regression trees [14]. Upon sampling a random set of predictors, each tree generates one prediction for the next period’s count of ILI positive visits. A random forest forecast consists of the average of the predictions of all trees.

We implemented an extension to random forests, called quantile regression forest (QRF), using the quantregForest R package [15]. QRF generalizes the usual random forest prediction by estimating conditional quantiles, useful when constructing prediction intervals. The predictors included in the QRF model were: lags of weekly ILI counts; current epi-week (for epi-week definition, see [16]); lags of weekly minimum and maximum temperature; lags of weekly flu immunization rate; and city population. Other predictors were considered but were excluded by recursive feature elimination using the rfe method of the R caret package [17]. These included weekly counts of diagnosed cases of the most common strain types, which are influenza A H1N1 and H3, influenza B, rhino/enterovirus, and respiratory syncytial virus. Information on patient age, sex, and location within city were also excluded since they rendered the model computationally expensive and yielded no performance gains. To illustrate the impact of these additional covariates on performance, S1 Fig shows the effect of adding patient sex, age, sex and age together, location within city, or strain types on the performance of the “baseline” QRF. The RMSE performance metric in this S1 and S2 Figs is described in the analysis section of the paper.

Our implemented QRF used n = 2000 decision trees, sampling m = 4 of the available predictors each time. The parameter m was chosen using the conventional heuristic , where P represents the number of predictors. The QRF prediction of ILI positive visits ℓ weeks from week t, , is given by: (4) where each T_b is one of the n decision trees resulting from a bootstrapped sample of the training data with m randomly selected predictors and x_t are the values of the model predictors at week t. The prediction interval of the QRF stems from conditional quantiles computed by the quantregForest method in R. [18].

Linear regression.

We fitted a standard linear regression (LR) model with the following predictor variables: current week’s ILI positive visit count, weekly-averaged minimum and maximum temperature, city population, immunization rate, year trend, epi-week fixed effects, slope of the ILI curve at current period, and a categorical variable counting upward movements in weekly ILI positive visits over the preceding three weeks. The ILI slope and count of upward movements helped on detecting sharp increases in recent ILI positive visits and improved the prediction near the peak. The model’s prediction of ILI positive visits ℓ weeks from week t is given by: (5) where X_t is a row vector of the values for the predictors at week t, is a column vector with the estimated marginal effect of each predictor on the ℓ weeks ahead count of ILI positive visits and is the intercept estimate. The prediction interval is where is the standard error of the prediction given observed values X_t and c once again comes from the amount of coverage for the prediction interval under the assumption that errors are normally distributed.

Stacked ensemble.

The stacked ensemble (SE) computes a weighted-average prediction based on the predictions of M contributing models {EB, QRF, ARIMA, LR} described above: (6)

We used stacking, a data-driven approach, to find a set of weights w_m for combining the predictions of individual models in a manner that minimized prediction error in a held-out data set. Specifically, we found the weights: (7) where H is the size of the hold-out data and is the prediction at period t of model m trained without data from the flu season encompassing period t. In practice, the algorithm for deriving the SE weights is as follows:

1. Train each individual model to the dataset holding out all weeks comprising an flu season;
2. Obtain fitted values for the weeks in the hold-out data;
3. Repeat steps 1 and 2 using every other flu season as hold-out data;
4. Compute the weights using Eq 7 with the H weeks of predictions obtained in steps above.

The rationale for ensembling is its potential to reduce prediction error by reducing prediction variance and, in some cases, bias [19]. The benefits of ensembling are typically smaller as the predictions of individual models become more positively correlated. Furthermore, as shown by Claeskens et al [20], estimation of averaging weights introduces additional randomness to the ensembled prediction. As such, we also show results for a “naïve” ensemble using equal weights on each individual model, i.e. the case where for each of the M models contributing to the ensembled prediction.

Deriving the sampling distribution of is nontrivial as ensembling mixes the distributions of each contributing model and some individual models are non-parametric to begin with. The literature on model averaged prediction intervals is scarce, and even in simpler contexts with parametric models the constructed intervals perform poorly in terms of coverage rate on validation exercises [21]. We report a weighted average of quantiles of the distribution of each individual model prediction as the ensembles’ own distributions. In our analysis, we show the coverage rate of this back-of-the-envelope ensemble prediction interval, as well as the coverage rate of intervals constructed from each individual model.

Mean absolute error & root mean square error.

Our forecasting targets were the ℓ ∈ {1, 2, 3, 4} weeks ahead ILI positive visit counts. We summarized model performance for each target using standard measures of prediction error: MAE and RMSE. These are given by: (8) (9) where y_{t+ ℓ} is the realized weekly ILI positive count at period t + ℓ and is model m’s generated ℓ weeks ahead prediction made at period t. MAE and RMSE express an average prediction error over the T − ℓ point forecasts performed in the sample. Both metrics are increasing with the average error, meaning the models with best accuracy in the test set are the ones with the lowest MAE and RMSE. Squaring of the error implies the RMSE imposes a greater penalty for larger deviations between predicted and observed values relative to the MAE. As such, a decision maker assigning higher importance to prediction accuracy near the flu season peak may prefer a model with lower RMSE.

Mean absolute percentage error.

For the sake of interpretability, we also present each model’s MAPE, which expresses the average percentage deviation between forecast and realized outcomes. The MAPE formula is given by: (10)

MAPE is known to produce infinite or undefined values when the denominator of any summation term in Eq 10 approaches zero [22]. This disadvantage did not apply to our analysis since we had a minimum of 52 weekly ILI visits to the ED during the sample period.

Log score.

The last performance metric in our analysis is a variant of the log score measure used to rank participants on the United States Centers for Disease Control (US CDC) flu forecasting challenge [23]. The log score of a forecast measures how much probability our model assigns to an “acceptable” prediction range. While MAE, RMSE, and MAPE relate to a model’s prediction accuracy, log scoring assesses the confidence that the model’s probabilistic forecast falls within a tolerance level.

We defined a prediction as acceptable if it fell within ± 25 visits of the realized weekly ILI positive weekly count. Since we compare a mix of parametric and non-parametric models, we approximate the probability assigned to the acceptable range by the count of centiles of the forecast inside the range. That is, the log score of a ℓ weeks ahead prediction from model m at week t is computed as follows: (11) where is the c centile of the (probabilistic) forecast generated by model m. In the event the observed ILI visits fell below the first centile or above the 99th centile, we assigned the value −5 to the log score, which is slightly lower than log(0.01) ≈ − 4.6.