Data

To obtain estimates of near-term increases in SARS-CoV-2 cases, a likelihood-based model underlying the EpiEstim package in R and developed in Cori et al. [8] and Thompson et al. [9] was deployed. using software provided by Imperial College London [10] Two methods were employed: 1) the R_t Only method, a forecast based on the reproduction number and associated serial interval that predicts the future R_t that is then extrapolated to estimate the number of future cases; 2) a LSTM machine learning model (ML+R_t) that utilizes the reproduction number from the R_t Only method as an input, but also utilizes total cases and population, among other inputs, to predict the total number of cases for a given period of time.

The data in this study is based on daily reports of all daily COVID-19 polymerase chain reaction (PCR) and antigen testing results conducted in WV since March 2020 directly from the WV Department of Health and Human Resources (WVDHHR). Noteworthy is that all SARS-COV-2 testing data are required to be reported to WVDHHR. Information for each unique patient is collected and contains test procurement date, test result date, patient zip code, patient county of residence, testing site name, county where the test is obtained, and test result. As patients who test positive may be tested multiple times, only the first positive tests on a patient is considered. When applying this filter, data obtained from all testing sites is used (i.e., hospital, clinic, pharmacy, drive-through, mobile van). The number of daily cases for each county is calculated by adding the lab confirmed cases and clinical confirmed cases after filtering out repeated tests or COVID-19 diagnoses. This daily incidence data on first diagnosed infection is the basis for calculation of R_t.

R_t Only method: Producing short term predictions

Our R_t Only method relies on the methodology used in the EpiEstim package and the underlying modeling approach of Cori et al. [8] and Thompson et al. [9]. This approach relates the daily incidence (number of new cases) to past cases through an instantaneous reproduction number R_t which characterizes the daily dynamics of transmission reflects a multitude of factors relating to individual and group behavior in the community of interest.

As a brief review, daily infections within a community occur as independent random events drawn from a Poisson distribution. The probability that exactly k cases occur is , and the rate parameter Γ coincides with the average daily incidence, ⟨k⟩ = Γ. In the instantaneous R_t framework, the expected incidence on day t is a product of two quantities, the infection potential and the reproduction number, Γ_t = Λ_tR_t. The infection potential Λ_t summarizes the record of past cases in the community and the typical variation of the infectiousness of an individual over time.

The infection potential Λ_t is determined by the incidence I_t−s on prior days s = 1,2,… and the serial interval distribution w_s.

The serial interval distribution w_s reflects the time course of infectiousness of one infected individual at s = 1,2,… days from the primary infection. It encapsulates the relative increase and decrease of infectiousness of an individual, assuming all other conditions in the community remain unchanged. In practice, the serial interval is typically obtained as the normalized () distribution of time intervals between known infector-infected pairs. Based on studies done by Gostic et al. [11] and Challen et al. [12], a serial distribution extending over 100 days was used. The infection potential can be understood as the sum of the expected number of infections on day t, due to past cases in the community, under ideal “steady state” conditions, such that over time, each primary case causes exactly one secondary case.

The time varying reproduction number, R_t, captures conditions of transmission that are external to the infected individuals and reflect community behavior. In this framework, R_t is a random variable with a Gamma distribution . The parameters a_t, b_t are determined for each day through Bayesian (maximum a posteriori probability) estimation. The parameters of interest are estimated using incidence data up to and including the current day, I₁, I₂,⋯I_t as follows:

This estimated R_t distribution applies to the most recent τ days, but it requires the values of I_t′ for t′≤t going back to t′ = t−s_max where s_max is the length of the serial interval distribution. For the serial interval w_s a discretized gamma distribution was used with mean and standard deviation of t_s = 7.0 ± 4 days, provided in the software similar to Cori [8].

For the serial interval w_s a gamma distribution was used with mean and standard deviation of τ_s = 6.99±4.02 days, as given by Flaxman et al. [13]. Following Cori and Thompson’s method, a prior distribution was used with mean and standard deviation equal to 5 (a_prior = 1, b_prior = 5) [8, 9].

The semi-deterministic model for future incidence, based on Cori’s method regards the daily distributions of R_t (values of a_t, b_t) as inputs that summarize the current conditions for disease transmission within the community of interest. The serial interval distribution w_s, which is fixed with regard to time, is also an input. Thus, on day t the distribution of R_t is known and applies to this day (assessed using the most recent τ days, similar to a trailing moving average).

Next day prediction.

Assuming time series of past daily incidences {I_u}_{u = 0,1,..t} ending on day t is observed, the number of infections on the next day t+1 follows a Poisson distribution, with parameter Γ_t+1 = Λ_t+1R_t+1, where R_t+1 is also a random variable. Assuming the parameters a,b of f(R_t+1|a,b) are known, the probability of exactly k new infections on day t+1 is:

The expected number of new infections coincides with the infection potential multiplied by the expected R.

For the purpose of predicting a likelihood range for the daily incidence, CDF of R_t+1 is defined as:

A [5% - 95%] credibility interval is obtained for the daily incidence using the inverse CDF for Rand multiplying by the corresponding infection potential. This provides a smaller variance than the discrete distribution P(k) but is a more practical indication of the incidence rate.

Extrapolation over multiple days.

To go beyond the “next” day, the one-day prediction is iterated, using predicted values to expand the incidence data. One can reasonably extrapolate the current distribution of R_t to t+1 and any number of days in the future. For the short term (7 day) predictions discussed here, the value of the most recent available R_t remains the same over the prediction interval, .

The estimated incidence for day t+1 requires the infection potential on that day Λ_t+1, which is computed based on incidence up to the preceding day t.

Predictions for day t+2 and beyond can be obtained using the predictions for preceding days for the incidence and iteratively applying the approach for any number of k days into the future.

The estimate of the credibility intervals are similar to the one-day case, using only the corresponding range for the reproduction number R_t, and not compounding with uncertainty for each estimated incidence or with the additional uncertainty due to the Poisson distribution of the daily (integer) incidence. While this provides a narrower range, the credible interval serves as a relative measure of the uncertainty affecting the prediction.

Correction for imported cases.

Not accounting for imported SARS-CoV-2 cases into a county will lead to over estimation of R_t. In practice, imported cases are not able to be directly identified, so an adjustment must be made to identify them. Assuming the daily incidence I_t can be separated into imported and community-spread parts:

Then, imported cases are an additional input to the model. Imported cases are included in the infection potential because they contribute to new local infections, but are not included in the number of new cases when estimating the reproduction number:

Turning to predictions, the reproduction number and infection potential computed in the standard framework can only predict the local cases:

By definition, imported cases cannot be predicted in the R_t model; however, events when the observed number of new cases vastly exceeds the expectation from local transmission can be identified. This hindsight is used to improve the estimate of the reproduction number as follows.

The likely number of imported cases on a given day is estimated by comparing the actual incidence to the Bayesian credible interval for new local cases estimated from the previous days. This estimated past incidence is then incorporated in a corrected estimate for R_t.

In an initial pass the a_t, b_t parameters are computed for time point t based on the incidence time series {I_τ}_{τ = 0,1,⋯t−1}. The one-day predicted incidence on day t is computed as described above, using the infection potential Λ_t and the distribution of . The value corresponding to the upper θ = 95% credible interval is used as a cutoff and identify the part of the incidence that exceeds the cutoff with imported cases.

The estimated local incidence is used to provide revised estimates for the reproduction number as described above (also consistent with Cori and Thompson’s approach). Finally, the resulting R_t parameters are used for the most recent day and the full incidence to provide revised estimates for days t+1, t+2,…t+k.

ML+R_t method: Using long short-term memory network to forecast outbreaks

As previously mentioned, the LSTM method implemented in this project is meant to build on the widely used R_t Only approach described in the previous section. The novelty of this LSTM approach is that it provides for the input of epidemiological modeling while taking advantage of cutting-edge machine learning techniques. The combination of the two allows the LSTM model to incorporate the epidemiological principles used to produce the R_t estimate while adding additional information such as temporal and demographic information that can be leveraged with traditional machine learning models. Further, the calculation of R_t using the Rt Only method uses independent data sets for each county in turn creating a unique model for each county that does not consider the impact of possible relationships between counties. By contrast, the ML+R_t approach uses global trends across counties. By training on all the data, the model is not only able to take advantage of global trends, but by including spatial information, the model is also able to show how these trends impact specific counties.

Daily county-specific R_t, summary statistic information on the estimated R_t such as standard deviation, confidence intervals, and the probability of R_t >1 are also provided. Rt values computed using both 7 and 14 day intervals are included. All these factors along with temporal information such as daily information on whether it is a weekend or not, holiday or not, days passed from last major holiday, days to the next major holiday were utilized as inputs for our model.

As mentioned previously, due to the length of time it takes to receive a test result (lag time), the deflated effect on the positive cases when considering test procurement date must be considered. An average lag of 3 days for results to achieve close to actual levels was observed. To mitigate the effect of the testing lag day t, t-1, t-2 are imputed with the actual SARS-CoV-2 cases for days t-3, t-4, t-5 respectively.

A LSTM recurrent neural network [14] was implemented in Python with an Adam optimizer, as our model of interest for this analysis, permitting consideration of all available county-specific input information for the past 7 days with a prediction of the number of positive cases for the county as an output. Other advantages of the LSTM approach are the ability to exploit autocorrelation between time points and the utilization of dropout layers to remove redundant information.

In general, the LSTM models are more complex versions of recursive neural networks (RNNs). The multi-layer LSTM method deployed here follows the framework described in Fig 1 where the input layer is defined by a matrix combining the number of positive cases for county c at time point t, Y_c,t, and all inputs for county c at time point t. The inputs then move their way through the network (i.e., through the LSTM layer and dense layer) to obtain an output. The output is defined as, , the predicted daily number of cases for county c at time point t+7. LSTM can be viewed as a network where information between time points is shared. Each LSTM cell, diagramed in Fig 1, shares two pieces of information with other LSTM cells; the current state of the cell, C_t, and output of the cell, h_t, is calculated with the following formulas given input data, x_t:

Where, w are the weight variables (traditionally thought of like regression coefficients), and b are the bias variables (traditionally thought of as intercept terms). Activation functions, σ and σ′ are non-linear transformation functions such as, sigmoid and hyperbolic tangent. A feature of each cell is input, output, and forget gates. These gates are what give the LSTM, the memory property which allows it to account and adjust for auto correlation. Define:

The above are gates that define the memory of the LSTM cell and are distinct linear combinations of inputs and outputs from the previous LSTM cell with specific activations functions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The LSTM framework deployed for the proposed ML+R_t method on right, and structure of each LSTM cell on left.

https://doi.org/10.1371/journal.pone.0259538.g001

In addition, the importance of the inputs is not guaranteed (including R_t and associated summary statistics), thus dropout layers were added to allow for the identification of important inputs. The drop out layers filtered out inputs that would be considered insignificant in order to detect the important signals coming from the input data and also protect against overfitting.

Once predictions for a given week were determined, the summary statistics of the results were produced. Summary statistics included: 1) predicted number of positive cases by county, 2) predicted percent change in cases per 100,000 persons by county compared to the previous week, 3) predicted increase in number of cases compared to the previous week, and 4) predicted number of cases relative to the population size.

Metrics and evaluation

To evaluate performance of the two methods, the predicted values for new SARS-CoV-2 cases were benchmarked against the actual number of positive cases recorded for each week from January 1, 2021 through April 30, 2021. As a main goal of these new case forecasts was to target areas for diagnostic testing, each week’s prediction is used as a recommendation. These recommendations were ranked on many several metrics but most predominately on the percentage increase in cases over the previous week. To evaluate the recommendations, the total discounted cumulative gain (DCG) of each method is measured [15]. DCG is a commonly used metric in page ranking calculations and is suitable here as the information shared was used similar to page ranking calculations. As a reminder the goal of this analysis is to recommend counties of increased incidences for intervention (i.e., increased SARS-CoV-2 testing), not to predict the actual number of incidence. DCG provides a metric for comparison of differing recommendation methods, which is how both the ML+R_t and Rt Only are being used. Unlike most metrics used in machine learning such as squared error or absolute error, larger DCG values indicate better performance.

To better study performance of the ML+R_t and R_t Only methods, two separate DCG metrics are used to consider the cost of poor recommendations. The first is on the ability to identify the top counties of increase regardless of the level of increase, while the second metric considers the size of the increase (percentage) in the comparison.

To define the first metric let and y_c,t represent the number of predicted cases and actual cases over a 7-day period for the cth county at time point t respectively. To keep from biasing the evaluation towards rural areas with a low incidence, only consider those with y_c,t+1>10 are considered. Define S_t to be the set of indices, the largest 10 values of for a given time point. The Binary Discounted Cumulative Gain (Binary DCG) of a set of rankings at time point t is defined as: where I(i∈S_t) is an indicator of a correct identification of a top 10 ranking in the actual percentage increases, and q is the number of rankings used in the calculation. For example, if q = 10, then BDCG_t would only evaluate the top 10 rankings, in our setting this would be the top 10 counties, returned by a method. One may view B-DCG as a weighted identifier to measure the quality of the rankings for purposes of identifying case increases (or spikes) of the top q recommendations.

As the closeness of the predicted number of cases to the actual case number, i.e., the “quality” of the prediction, a second metric was used to consider the quality of the prediction rather than just considering a binary outcome. To accomplish this, Spike DCG is defined as:

Spike DCG considers the relative size of the spike for the top q recommendations. While Binary DCG investigates the ability of a method to correctly identify the top 10 counties, Spike DCG places value on the recommendations that are produced by identification of larger spikes. This comparison is of great importance as targeted interventions may only have finite resources to deploy so understanding the level of trust and impact expected by the two methods is of importance.

As both the R_t Only and ML+R_t methods are used to recommend county level locations for testing, it is important to investigate the quality of the top recommendations, disregarding the order and quality of the ranked predictions. This evaluation gives a sense of the quality of the recommendations produced by the methods, relative to others.

Finally, as this study is being deployed in a state with many rural areas, differences in methods between rural and non-rural areas were also analyzed. This study uses the 2013 Rural-Urban Continuum Codes (RUCC) to define rurality [16], which define a rural area as a non-metro area with population under 20,000 and is not adjacent to an urban metro area. To asses quality of the predictions provided by each method, we examined correlations between predicted and actual 7-day positive case totals. The quality of Binary DCG and Spike DCG in both rural and non-rural areas is assessed by investigating the performance of ML+R_t and R_t Only methods among lower population communities with less access to large healthcare systems. Both R_t Only and ML+R_t methods were deployed each week from January 1, 2021 through April 30,2021 using all available training data beginning in April 2002 for each of the 55 counties in the state of WV, and resulting county recommendations were retained for comparison against the actual number of cases. The code for fitting and deploying the models is publicly available [17].

Assessing rural vs non-rural results

Critically important is analysis on the performance of the two forecasting strategies in rural compared with more urban counties in WV. Correlations between predicted 7-day positive case totals and actual 7-day positive case totals are higher for non-rural counties than rural counties for both methods (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. A comparison correlation of 7-day positive case totals and 7-day actual case, and both DCG metrics (total) for the ML+R_t methods implemented when viewed by rural and non-rural counties.

https://doi.org/10.1371/journal.pone.0259538.t002

For rural areas, the two methods perform similarly with the ML+R_t method slightly outperforming R_t Only in regard to Spike DCG (Fig 6). For non-rural areas, ML+R_t outperforms Rt Only for both DCG metrics (Table 2). The R_t Only Method performs well when identifying counties in the top 10, but ML+R_t method identifies larger spikes in the top 10 recommendations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. A comparison of Spike DCG for both rural and non-rural counties.

https://doi.org/10.1371/journal.pone.0259538.g006

A secondary analysis shows that the ML+R_t method recommends for enhanced SARS-CoV-2 testing more non-rural counties than rural counties in the top 10 rankings during January and February when compared to the R_t Only method. The opposite occurs during the March and April time period during which the R_t Only method recommends more non-rural counties in the top 10 compared to the ML+R_t methods. When coupled with decreasing number of tests, leading to lower daily incidence this alleviates any concern of bias of the ML method on rural counties.

Limitations

This study addressed the West Virginia SARS-CoV-2 epidemic from January–April 2021. At that time, only one case of the Delta variant had been detected, therefore, our models do not address prediction of new SARS-CoV-2 incidence when Delta is the prevalent variant. As the Delta variant has unique epidemiologic characteristics compared to earlier SARS-CoV-2 variants such as a shortened serial interval which influences calculation of R_t, models must be adjusted as new more virulent strains of SARS-CoV-2 appear in the population [7]. This study also looks at West Virginia specifically, though the techniques could be applied broadly. As the data in this study is specifically from WVDHHR, it is unable to be compared directly to publicly available sources from other states, limiting the implications of this specific model.