Data sets.

Influenza-like illness rates.

CDC defines ILI as fever (temperature of 37.8° or greater) and cough and/or sore throat without a known cause besides influenza. The prevalence of ILI is monitored through several surveillance efforts including the Outpatient Influenza-like-Illness Surveillance Network (ILINet) which collects weekly state level ILI rates from over 2, 000 healthcare providers from all states. The state level ILI rates are weighted by population size to report the wILI at different geographic levels [51]. Our models use weekly wILI rates for the flu seasons 2004/05 to 2018/19 inclusive, obtained from gis.cdc.gov/grasp/fluview/fluportaldashboard.html. Note that this data is not final, i.e. it can be revised by the CDC. To ensure reproducibility of our results, a copy of all the ILI data used can be found at our Github repository github.com/M-Morris-95/Forecasting-Influenza-Using-Neural-Networks-with-Uncertainty. A week in the CDC data represents a 7-day period that starts on a Sunday and ends on a Saturday. We assume the weekly ILI rate is representative of Wednesday (middle day) and use cubic interpolation (interpolate.interp1d from Python’s SciPy library) to generate daily ILI rates. This not only increases the number of samples (7-fold), but also provides an aligned time series with the daily temporal resolution of the Web search activity data. The deployment of a cubic as opposed to a linear interpolation to generate daily ILI rates resulted in slightly better forecasting accuracy on the test sets. We hypothesise that this is because of the increased level of smoothness (see S5 Fig), but also note that we have not fully assessed this data manipulation choice. For training the Dante forecasting model, we have also obtained regional wILI rates for the 53 US states / locations and the 10 US Health and Human Services regions. These were downloaded from the CDC for the same period above and are also available on our Github repository.

Search query frequency time series.

Search query frequencies for the US are obtained from the Google Health Trends API similarly to other studies [36, 52]. A frequency represents the fraction of searches for a certain term or set of terms divided by the total amount of searches (for any term) for a day and a certain location. We initially downloaded the daily search frequencies of a predetermined pool of 20, 856 unique health-related search queries for the period from March 2004 to May 2019 inclusive for the US. Query frequencies are smoothed using a 7-day moving average, and min-max normalisation is applied to each query’s time series during training (i.e. without using any future data). For a given test season, for each query, q, we compute the correlation score R_q with the ILI rate over the five seasons preceding the test season. We also compute a semantic similarity score S_q that measures each query’s similarity to a predefined flu concept as described in Lampos et al. (2017) [36]. Both scores are then normalised between 0 and 1 and a composite score for each query is calculated. Only the m queries with the highest U_q are used, where m is a hyperparameter (see “Hyperparameter optimisation”).

Feedforward Neural Network (FF).

The FF model has two hidden feedforward neural layers with a ReLU (max(0, x)) activation function, and a Bayesian layer (S12 Fig). The input to the network is a window of τ + 1 days of ILI rates and m search query frequencies. There is an ILI rate collection delay of δ days, in that at day t₀ we know (CDC has published) the ILI rate of day t₀ − δ. The delay is assumed to be δ = 14 days throughout our experiments. Thus, at day t₀, the input to the network consists of ILI rates, to , and search query frequencies, through . We ignore the temporal structure of the data and use an (m + 1) × (τ + 1) vector as the input to the neural network. The output of the network is an estimate of the ILI rate and corresponding data uncertainty γ days ahead.

Simple Recurrent Neural Network (SRNN).

This is a recurrent neural network which observes a time series of ILI rates and search frequencies (S13 Fig). The input to the network is the same as for FF, but without flattening into a vector. We feed the (m + 1) × (τ + 1) input matrix into a Gated Recurrent Unit (GRU) layer one day at a time. The final output of the GRU is passed to a dense layer with a distribution over its weights.

Iterative Recurrent Neural Network (IRNN).

This is a recurrent neural network which makes forecasts of the ILI rate and search frequencies one day at a time. It bases forecasts on its own previous forecasts. IRNN comprises a recurrent GRU layer and a feedforward Bayesian layer as shown in Fig 4. We have also described how model training works with pseudocode in the Supporting Information (S14 Fig). Given its special structure, IRNN does not incorporate future (for a period of 7 days after the target forecast) search query frequencies when γ = 7. Hence, for both γ = 7 and 14, the only minor difference may be due to the more recent past ILI rate inputs. As a result, the difference in performance between γ = 7 and 14 is expected to be minor given that search query frequencies are always the more recent information source (as opposed to past ILI rates). This is also empirically confirmed by our experiments (see Table 1). A caveat of the current formulation of IRNN is that the model is agnostic of the actual forecast horizon and hence its uncertainty might be underestimated for larger forecasting horizons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Diagram of the IRNN architecture where for the recurrent layers (RNN) we have used a Gated Recurrent Unit.

An ILI rate, F ∈ [0, 1], and m search query frequencies, , beginning from time point (day) t₀ − τ are fed into the network a day at a time. τ denotes the window size of past observations that we consider (τ + 1 = 56 days). The reporting delay of the ILI rates means that when an ILI rates are available up to day t₀, search query frequencies are available up to day t₀ + δ, where δ = 14 days in our experiments. Dashed arrow lines denote that the model is called for multiple time-steps (where a time step is a day). For days t₀ − τ to t₀, IRNN enters a warm-up phase where it sets the hidden states in the RNN layer without making any predictions. For days t₀ to t₀ + δ, we can observe search query frequencies, but we cannot observe ILI rates. At this stage, IRNN performs nowcasting with respect to input Q. During nowcasting the estimated ILI rate is combined with the true search frequencies Q_t use as the input for the next time step. The query search frequency estimates which are not used (as they are known to us) are shown by a faded box. For days t₀ + δ + 1 to t₀ + γ, where γ denotes the forecasting horizon, IRNN conducts pure forecasting as neither search query frequencies nor ILI rates are known for that period. Forecasted values for both of them are used as inputs for subsequent time steps. The full sequence of both predicted ILI rates and search query frequencies is used in the training loss.

https://doi.org/10.1371/journal.pcbi.1011392.g004

Data uncertainty.

Data or aleatoric uncertainty is caused by noisy observations. It can be further divided into homoscedastic (constant for all inputs) and heteroscedastic (dependent on the input) uncertainty [53]. We estimate heteroscedastic uncertainty in the output layer of the neural network by approximating the parameters of a Gaussian distribution [54] (1) where a₁ and a₂ are the outputs of the neural network. Softplus is defined by (2) where s > 0 is a scaling factor, and c = ln (e^x − 1) is an offset which makes the output equal to 1 when a₂ = 0. All hyperparameters (insofar s and c) are jointly optimised using Bayesian optimisation (see “Hyperparameter optimisation”). The network is trained by minimising NLL given by (3) where y = {y₁, …, y_T} is a series of T flu rates (ground truth), is a series of T forecasts, and is a series of associated standard deviations (data uncertainty). The first component of Eq 3 contains a residual term equivalent to the mean squared error and an uncertainty normalisation term. The second component prevents the model from predicting an infinitely large . Minimizing the NLL allows us to train an NN despite not having ground truth estimates of the data uncertainty.

Model uncertainty.

Model or epistemic uncertainty is inherent in the parameters of the model. It is caused by having insufficient data to exactly set the model’s parameters. To estimate model uncertainty, we deploy a BNN trained using variational inference [55]. BNNs have a distribution over their parameters. For our purposes, we restrict the Bayesian component to the last layer of a network, while preceding layers use deterministic weights and biases. We do this as we found it to be more stable and faster compared to training using a distribution over all the parameters of the model [56]. We use a Gaussian prior with a diagonal covariance matrix , where σ_p ∈ [0.0001, 0.1] is a hyperparameter. The form of the distributions is an implementation choice. We specify the posterior distribution to the same form (Gaussian with diagonal covariance) as the prior, such that we have a conjugate prior [57]. The mean μ_w and standard deviation σ_w of each weight in the posterior are learned. θ holds all the layer’s parameters that are updated during training; the means in the posterior are half of its parameters and the softplus of the standard deviations constitutes the other half. Instances of the model are created by sampling the weights in the Bayesian layer. For each instantiation of the model, a forecast and associated data uncertainty are determined. The variation in the forecast across the model instances provides an estimate of the model uncertainty.

Combining data and model uncertainty.

Each time that we sample from the posterior distribution of the model’s parameters (which in Experiments will be denoted by q_θ(Φ)) we create a model instance, k, that forecasts a mean and standard deviation . After drawing K samples, the predictions are combined to create a single distribution. The mean is the mean of the K forecasts, i.e. (4) and the variance is given by [31] (5) In Eq 5, the first two sum terms define the variance of the means, i.e. the model uncertainty. The third term is the mean of the variances, i.e. the data uncertainty.

Training a BNN.

Bayesian inference is used to learn the posterior distribution p(Φ|D) over the model parameters Φ, where D is the training data containing inputs and targets. In theory, the posterior can be derived by using Bayes rule, i.e. (6) where p(Φ) is a prior distribution which expresses prior belief about the parameter distribution before training data is observed. However, we cannot obtain an exact estimate of the posterior due to the integral p(D) = ∫_Φ p(D, Φ′)dΦ′, which is unavailable in closed form and requires exponential time to compute [55]. We instead use variational inference, replacing Eq 6 with an optimisation task. The posterior p(Φ|D) is constrained to a family of distributions [55], and instead of finding the exact posterior, a variational distribution q_θ(Φ) is used. θ is used to describe the distribution [58] e.g., the mean and variance for a Gaussian. The goal of training becomes to learn θ by minimising the Kullback-Leibler divergence (D_KL) to the true posterior p(Φ|D) as in (7) The KL divergence represents the average number of bits required to encode a sample from p(Φ|D) using q_θ(Φ). To compute the KL divergence, p(D) is required. However, we can instead maximise the evidence-lower-bound (ELBO) given by (8) which is equivalent to Eq 7 up to a constant and is tractable. The first component of Eq 8 is computed using Eq 3, and measures how well the model fits the training data. The second component of Eq 8 is the KL divergence between the posterior and prior distribution. The KL divergence term behaves similarly to a regulariser and encourages the model to choose a simple q_θ(Φ). When the ELBO is computed using minibatch gradient descent, the gradients are averaged over each minibatch. Therefore, to compute the ELBO the KL divergence term is weighted by the total number of minibatches during training [59]. We also use an additional KL annealing term ξ, which limits the regularisation effect of the KL term. For the FF and SRNN models this is equal to 1/M where M is the number of minibatches multiplied by KL_w [60]. For the IRNN model, M is the number of minibatches multiplied by γ = 28 and KL_w. This is because the model makes 28 outputs, and each requires calling the dense Bayesian layer. The constant, KL_w, is chosen by hyperparameter optimisation. Thus, Eq 8 becomes (9) When training the FF and SRNN models, each training step takes an [(m + 1) × (τ + 1)]-dimensional input (where m denotes the number of search queries and τ + 1 denotes the window of days, from t₀ and back, for which query frequencies and ILI rates are used) and produces a forecast estimate containing both a mean and standard deviation for the ILI rate for time (day) t₀ + γ. The parameters Φ are updated by minimising Eq 9. During each training step one sample is taken from q_θ(Φ) and used to compute the ELBO. We use backpropagation to compute gradients and update the parameters in both the Bayesian and non-Bayesian layers. The model is retrained for each time horizon γ, where γ = 7, 14, 21 or 28 days, and for each test period.

The output of the IRNN is a sequence of ILI rates and search frequencies. Although we have search data from t₀ to t₀ + δ, we use the full sequence of estimated query frequencies when backpropagating the ELBO (Eq 9) through time. When evaluating the model’s performance we are only concerned with the model’s ILI rate forecasts. The Bayesian layer is called once for each iterative prediction.

Hyperparameter optimisation.

We use Bayesian hyperparameter optimisation [61] with 5-fold cross validation where each fold is a 365-day period covering a full flu season (see S3 Table and S3 and S4 Figs). We tune the hyperparameters once before the first test period, and keep the same hyperparameters for all subsequent test seasons. For the FF and SRNN the hyperparameters are re-tuned for each of the four forecast horizons. For the IRNN the hyperparameters are tuned once, considering all four forecasting horizons (the average NLL is computed across them). The hyperparameters are the following: the size of the hidden NN layers ∈ [25, 125], the number of queries m ∈ [20, 150], the weighting of the KL divergence term in the ELBO loss KL_w ∈ [0.0001, 1.0], the scaling factor of the output’s standard deviation s ∈ [1.0, 100], the prior standard deviation σ_p ∈ [0.0001, 0.1], the number of epochs ∈ [10, 100], and the learning rate ∈ [0.0001, 0.01] for training the NNs. After the hyperparameters are tuned we re-train the model using the full training set for the number of epochs chosen. The derived model is then used for forecasting on the test set. Note that hyperparameters are not re-tuned for comparing with Dante (when Web search activity data that are more recent that the last observed ILI rate are removed), which may have disadvantaged our NN models.

Inference.

When making an estimate with a BNN based on inputs X, and with training data D, the goal is to compute an output for the entire distribution over Φ: (10) In practice is estimated using Monte Carlo sampling from p(Φ|D) [58]. At prediction time, the posterior distribution over the weights is sampled K times, each giving a output . The K estimates are combined using Eq 5 which makes an estimate for the combined model and data uncertainty. K is chosen by sampling until the final estimate of stabilises. Initially we sample 10 times and produce an estimate using Eq 5. We then run the model a further 10 times and produce a new estimate using the 20 samples. We repeat this process until increasing K by 10 does not change the estimated mean by more than 0.1%. Despite averaging over K instances of the model, we observed some instability in training the models.

To resolve this each model was trained 10 times with different initialisation seeds, i.e. the seed controlling the initial parameter values of the NN. The mean of the 10 estimates of forecasts and associated variances are our final forecast and variance. We considered alternate methods of combining estimates, such as Eq 5, and averaging the probability density functions. Ultimately we found that averaging the means and variances gave the best final forecasts. Thus, the total number of samples for making a forecast is equal to , where i ∈ {1, …, 10} denotes a different seed, and d_i ≥ 20 is the number of samples required for this seed to converge.

To make an estimate with the SRNN and FF models, the inputs are passed through the model’s layers up to the Bayesian layer. The weights in the Bayesian layer are then sampled K times, and the estimates from the K samples are combined with Eq 5 as discussed in the previous paragraph. Making an estimate with IRNN has three distinct phases: warm-up, nowcasting, and forecasting (S14 Fig). During the warm-up phase the model observes ILI rates and m search queries from t₀ − τ to t₀. This sets the hidden states of the GRU layer based on all ILI rates and search frequencies from the same days. The output of the GRU is fed into the Bayesian layer (denoted by FC_BNN in Fig 4), which estimates the input for the next time step. The Bayesian layer estimates model and data uncertainty, and has 2 × (m + 1) units. The first half of the units estimate the means of the query frequencies and ILI rate, the second half of the units estimate the corresponding standard deviations. The estimated ILI rate is a distribution which cannot be directly interpreted by a NN layer. Therefore, a sample from this distribution is combined with the true search query frequencies and fed back into the GRU layer. This is repeated from t₀ to t₀ + δ (nowcasting phase). After time t₀ + δ, no more search query frequencies are available. The estimated search query frequencies and ILI rates from each time step are fed back into the model to make subsequent forecasts. The process of making daily estimates can be repeated indefinitely, so γ, the forecasting horizon, could increased arbitrarily.

Evaluation.

We evaluate the performance for forecasting horizons γ = 7, 14, 21 and 28 days ahead. We choose weekly test dates starting from week 45 and lasting for 25 weeks. We use the 2015/16, 2016/17, 2017/18 and 2018/19 flu seasons to evaluate our model. We did not consider running experiments on data from 2019/20 or 2020/21 as flu prevalence has significantly declined, and ILI rate estimates from CDC became less reliable due to the COVID-19 pandemic. We train models for the period 05/06/2004 until the Wednesday of the 33rd week of the year in which the test flu season starts (around mid-August). We test the models on the period from the Sunday of week 44 until the Saturday of week 23 in the following year. Exact training and test periods are provided in the Supporting Information (S2 Table and S2 Fig). To compare our NN models to Dante we evaluate the model scores on the same test weeks as specified in Reich et al. (2019) [15]. When comparing the best performing NNs to Dante, the training set included all seasons except the test season, i.e. it also included data after the test season (models NN and NN_b in Table 2). We did not re-tune hyperparameters to account for training on future seasons. As discussed earlier, we do not consider training on data after the test period to be appropriate, but it allows the most direct comparison to the training setup used by Dante. We also report the performance of our best performing NNs when trained using only data prior to the test season (model NN_a in Table 2).