https://orcid.org/0000-0002-4681-091X
Figures
Abstract
Forecasting infectious disease outbreaks is hard. Forecasting emerging infectious diseases with limited historical data is even harder. In this paper, we investigate ways to improve emerging infectious disease forecasting when little pathogen-specific training data are available. Specifically, we explore two sources of information that may be available near the start of an emerging disease outbreak: synthetic data and genetic information. For this investigation, we conducted an experiment where we trained deep learning models on different combinations of real and synthetic data, both with and without genetic information, to explore how these models compare when forecasting COVID-19 cases for US states. All models are developed with an eye towards forecasting the next pandemic. We find that models trained with synthetic data have better forecast accuracy than models trained on real data alone, and models that use genetic variants have better forecast accuracy compared to those that do not. All models outperformed a baseline persistence model, a benchmark that proved challenging for many real-time COVID-19 case forecasting models, and multiple models outperformed the COVIDHub-4_week_ensemble. This paper demonstrates the value of these underutilized sources of information and provides a blueprint for forecasting future pandemics.
Author summary
Forecasting emerging infectious diseases is difficult because, by definition, little or no historical data are available for the pathogen of interest. This lack of training data is a major barrier to using flexible machine learning models in emerging disease forecasting settings. In this paper, we investigate two sources of information likely to be available near the start of a future outbreak: synthetic outbreak data and viral genetic information. Using COVID-19 as a test case, we train transformer-based models on historical respiratory disease data, synthetic outbreaks, and SARS-CoV-2 variant information to forecast weekly cases in U.S. states. Models trained with synthetic data outperform those trained on historical data alone, and models trained on both real and synthetic data perform best overall. We also find that using variant-attributable cases improves forecasts relative to forecasting total cases directly. We show these models were competitive with the best real-time COVID-19 case forecasting models. Together, these results show that synthetic data and genomic surveillance are practical, underused tools for epidemic forecasting. They offer a path toward scalable forecasting systems that can be deployed early in the next pandemic, when reliable forecasts are most needed and hardest to produce.
Citation: Osthus D, Murph AC, Goldberg EE, Beesley LJ, Fischer WM, Parikh N, et al. (2026) Leveraging synthetic and genetic data to improve epidemic forecasting. PLoS Comput Biol 22(8): e1014630. https://doi.org/10.1371/journal.pcbi.1014630
Editor: Benjamin M. Althouse, University of Washington, UNITED STATES OF AMERICA
Received: March 18, 2026; Accepted: July 24, 2026; Published: August 27, 2026
Copyright: © 2026 Osthus et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data and code used to perform the analysis presented in this paper can be found here: https://github.com/lanl/precog/tree/main/synthetic_and_genetic_forecasting.
Funding: This work supported by the Laboratory Directed Research and Development program (https://www.lanl.gov/science-engineering/science-programs/ldrd) of Los Alamos National Laboratory (20240066DR to DO, AM, EG, LB, WF, NP, LC). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Over the past decades, infectious disease forecasting has transformed from an academic curiosity into a critical tool for public health preparedness and response. This growth has been driven by advances in modeling [2–4] and data availability [5], and by the recognition that timely forecasts can guide resource allocation, inform policy, reduce the cost burden associated with emerging health threats, and improve public health outcomes [6]. Yet forecasting remains inherently difficult: challenges include noisy and incomplete data [7], lags in reporting [8], reflexive human behavior [9], uneven policy decisions [10], and pathogen evolution [11,12]. These obstacles are particularly pronounced during rapidly evolving outbreaks, where even the best models struggle to keep pace [1]. The COVID-19 pandemic brought these struggles into acute focus [13]. On one hand, it marked an unprecedented mobilization of forecasting talent and infrastructure [14,15]; on the other, it revealed gaps—particularly in integrating real-time signals, quantifying uncertainty, and anticipating the emergence of new viral dynamics.
Among the success stories of the COVID-19 response was the rapid and widespread adoption of genomic surveillance [16]. SARS-CoV-2 genomes were sequenced and shared globally at unprecedented scale and speed, offering a near real-time continuously updated feed of the virus’s evolution [17,18]. With relatively low technical barriers and declining sequencing costs, genomic surveillance is poised to remain a cornerstone in future outbreaks. Crucially, it enabled the early identification of new variants, which often preceded observable surges in cases [19]. This makes variant tracking a potential leading indicator—a rare and powerful feature in infectious disease forecasting. Looking ahead, this stream of high-resolution, biologically meaningful data offers a promising direction for improving forecast model accuracy.
In general, forecasting systems perform best when three conditions are met: (1) the system’s underlying mechanisms are well-characterized, (2) there is sufficient historical data to train models, and (3) future trends don’t deviate too significantly from past ones [20]. Emerging infectious diseases typically violate the first two conditions, and sometimes all three. While mechanistic models—such as compartmental models [21], agent-based models [22], or phylodynamic models [23]—can capture transmission and/or evolutionary dynamics, they are often difficult to fit to incomplete and noisy real-time data, and have often been outperformed by their more flexible statistical or machine learning model counterparts in real-time forecasting exercises [24]. Flexible statistical and machine learning models are therefore promising tools for operational forecasting, but their flexibility can also be a weakness when the outbreak of interest differs substantially from the data used to develop or train them.
In this paper, we seek to develop forecasting models that are accurate, scalable, and easily deployed in a pandemic setting where little to no historical data are available for the pathogen of interest. We consider transformer-based deep learning models for forecasting because, although often costly to train, they are cheap to deploy, can maintain strong performance on new data without frequent retraining; they allow scaling to many data subsets (e.g., geographies), and offer high performance ceilings.
This modeling framework sits within a broader literature on deep learning for epidemic analysis and forecasting, including multilayer perceptrons, convolutional neural networks, and recurrent neural networks such as long short-term memory networks [25–27]. Relevant work also includes graph neural network approaches for spatio-temporal epidemic forecasting, which model geographic regions as connected nodes with coupled disease dynamics [28,29], as well as deep-learning-based phylodynamic methods that use genetic sequences or phylogenetic trees directly for epidemiological parameter inference, model selection, and characterization of transmission dynamics [30,31]. In contrast to the latter, our use of genetic data is intentionally more aggregated: we use variant labels and variant-attributable case counts as surveillance summaries rather than modeling genome sequences or phylogenetic trees directly.
For the forecasting problem considered here, the central challenge is training data. To learn, these models require large amounts of training data, yet for an emerging pathogen, by definition, pathogen-specific datasets are limited. In this setting, essentially two types of training data are available at or near outbreak onset: historical outbreak data from other pathogens and synthetic data. Neither data source will perfectly represent the pathogen of concern, yet both are available at outbreak onset and so can be used to train deep learning models before substantial pathogen-specific surveillance data have accumulated.
Historical outbreak datasets are finite, restricted to outbreaks that have occurred and were measured, and thus represent only a portion of the space of possible outbreaks. Recent work has demonstrated success in incorporating data from different pathogens and surveillance streams to improve forecasting [4,32]; this strategy is enabled by the public dissemination of public health data, e.g., [33]. Thus, while historical data do not represent the emerging pathogen (the forecast target), they do represent ostensibly relevant data, including data reporting vagaries, useful for model training. This mirrors the logic of pretraining and transfer learning in neural networks, where models trained on related source domains can learn reusable representations that improve performance or reduce data requirements in a target domain [34,35].
Synthetic datasets address many of the shortcomings of historical data. They are infinite in number (in principle): their generation is constrained primarily by compute resources. Furthermore, they can represent a diversity of outbreaks limited only by the choice of simulation parameters; measurement noise and biases can be added separately to mimic realistic surveillance systems. Researchers have recently shown success of models trained exclusively on synthetic outbreak data [36–38]. This being said, the realism of these outbreaks and the potential gains of using synthetic data in modeling are restricted by the fidelity of the simulator and by the measurement error processes.
Synthetic data must conform to real or prospective data as it is (or will be) measured. Many infectious disease models based on first principles (e.g., compartmental models or agent-based models) can straightforwardly generate time series of case-counts, hospitalizations and deaths. To make use of viral genetic variant information, as we do here, a synthetic infectious disease simulator must model outbreaks at the variant level, and aggregate variant data to produce “observed case-count” time series. In this paper we make use of one such simulator, MutAntiGen [39] (discussed in detail in Section 4). This simulator produces both time series of total observed cases (referred to as total cases, or TCs) as well as time series of each constituent variant (referred to as variant-attributable cases, or VACs).
Given this framing, we seek in this work to answer the following five research questions:
- Q1: Does training with real data or synthetic data produce better forecast performance?
- Q2: Does joint training with real and synthetic data improve COVID-19 case forecasts relative to training with either source individually?
- Q3: Two questions comparing synthetic TC and VAC training data:
- Q3a: Does training with synthetic VAC data improve COVID-19 forecasts relative to training with synthetic TC data?
- Q3b: Do models with matched training data and input data outperform models with mismatched training data and input data?
- Q4: Does including SARS-CoV-2 variant information improve COVID-19 case forecasts?
- Q5: How do these forecasts compare to real-time COVID-19 case forecasts?
To answer these five questions, we fit and compare eight forecasting models that differ in both the data used to train the models and the inputs to the models, described in Table 1. While eight models are defined in Table 1, they correspond to four different fitted deep learning models trained on different data sets (real, synthetic TCs, synthetic VACs, or real plus synthetic), each applied to two different model input types (TCs or VACs).
For concreteness, models M(r,t) and M(r,v) use the same fitted deep learning model (the model trained only with real training data), but M(r,t) predicts total cases directly and M(r,v) forecasts each variant-attributable case directly and sums up the individual forecasts; additional forecasting details are provided in Section 4.3.2.
The models defined in Table 1 allow for a systematic evaluation of how synthetic data and genetic information can improve forecast accuracy by comparing forecast performance of pairs of models. Specifically,
- (Q1) If models trained with synthetic data outperform models trained with real data, we would expect M(st,t) > M(r,t) and M(sv,v) > M(r,v), where M(A) > M(B) means M(A) outperformed M(B).
- (Q2) If joint training with real and synthetic data improves COVID-19 case forecasts relative to training with either source individually, we would expect M(a,t)> [M(r,t), M(st,t)], and M(a,v)> [M(r,v), M(sv,v)].
- (Q3a) If training with synthetic VACs improves COVID-19 case forecasts relative to training with synthetic TCs, we would expect M(sv,v) > M(st,t).
- (Q3b) If models with matched training data and input data outperform models with mismatched training data and input data, we would expect M(st,t) > M(sv,t) and M(sv,v) > M(st,v).
- (Q4) If including SARS-CoV-2 variant information improves COVID-19 case forecasts, we would expect M(r,v) > M(r,t), M(st,v) > M(st,t), M(sv,v) > M(sv,t), and M(a,v) > M(a,t).
- Q5 will be evaluated with external models later.
We next present the results of the exercise, including direct answers to all research questions stated above. We then discuss implications, limitations, and future directions of work. Details on the study design, data sources, synthetic data generation, forecasting models, and evaluation metrics are provided in Section 4.
2 Results
We evaluated weekly 1–4 week ahead forecasts of COVID-19 cases for U.S. states and Puerto Rico from June 2020 through December 2022. The study used COVID-19 as a retrospective test case for pandemic forecasting models trained without COVID-19 data, allowing us to evaluate whether historical respiratory disease data, synthetic outbreak data, and SARS-CoV-2 variant information could improve forecasts under emerging-outbreak constraints. Full study-design details are provided in Section 4.
We first provide high-level findings from our exercise in Section 2.1. Then in Section 2.2 we directly answer the research questions stated in Section 1. Evaluation metrics are defined in Section 4.
2.1 Overview of results
Fig 1 shows selected forecasts for New Mexico for all models. As expected, forecast interval widths increase with increasing horizon h. The forecasts for models trained on only real data (M(r,t) and M(r,v)) have noticeably low quantile estimates at the 0.025 level. The real data used for training are fairly noisy, which may have led to this forecast behavior. The other notable trend is the forecasts made at the peak of the Omicron (BA.1) wave in early 2022. The models that take TCs as input, M(.,t) (left column of Fig 1), produced flat to mildly dropping forecasts at the peak in early 2022, while the models that take VACs as inputs, M(.,v) (right column of Fig 1), correctly produced a steep drop in their forecasts. Fig 1 illustrates that there are systematic differences in the forecast models that break down along training data and model input lines.
Black line: total cases time series. Colored points: median forecast cases. Ribbons mark the 50%, 80%, and 95% forecast intervals. Note: y-axis is on a square root scale to better see the low-case-count forecasts.
Fig 2 shows rMAE results. Overall, we see all models had better MAE than the baseline, the worst being M(sv,t) with 0.928 rMAE and the best being M(a,v) with 0.777 rMAE. Uncertainty intervals are 95% confidence intervals, obtained via bootstrapping (see S1 Text for details). Each model had rMAE less than or equal to 1 for all forecast horizons. Fig 2 Bottom shows a running relative MAE where the rMAE on a given date corresponds to the evaluation of all forecasts available between June 1st, 2020 through the x-axis date. It is clear the results depend on the evaluation period, as models perform differently throughout different phases of the pandemic. That said, for all evaluation end times after January 2021, all models had an rMAE less than 1 (except for a momentary blip for a few models around January 2022).
Error bars were constructed via bootstrapping. (Top right) Relative MAE by forecast horizon. (Bottom) Running relative MAE. The running relative MAE for each date is the relative MAE if the evaluation period ran from June 1st, 2020 through the x-axis date.
Fig 3 shows results for WIS relative to a persistence model. Overall, all models had rWIS less than 1 ranging from 0.769 (M(r,t)) to 0.63 (M(a,v)). Each model also had rWIS less than 1 (in fact, less than or equal to 0.80) for all considered forecast horizons. The running relative WIS is less than 1 for all models and all evaluation end dates between June 2020 through December 2022 providing strong evidence that all models demonstrated better forecast performance as measured by WIS compared to M(0).
Error bars were constructed via bootstrapping. (Top right) Relative WIS by forecast horizon, and (Bottom) running relative WIS. The running relative WIS for each date is the relative WIS if the evaluation period ran from June 1st, 2020 through the x-axis date.
Fig 4 shows the empirical coverage for all models, overall and by horizon. All models exhibit undercoverage (i.e., observations fell outside their predictive intervals more often than expected), a common occurrence in COVID-19 forecasting [1]. All models showed empirical coverage similar to that of a persistence model for 50% forecast intervals, but empirical coverages closer to nominal than the persistence model for 95% forecast intervals. When we examine empirical coverage by forecast horizon (Fig 4 Bottom), we generally see that all non-persistence models have near nominal coverage at horizon 1, but that empirical coverage drifts below nominal coverage as horizon increases to 4 weeks ahead.
(Bottom) Empirical coverage by horizon. The dashed horizontal line indicates the nominal coverage. Undercoverage exists for all models and horizons.
2.2 Answering the research questions
We now directly answer our research questions laid out in Section 1.
2.2.1 Q1: Does training with real data or synthetic data produce better forecast performance?.
There is fairly strong evidence that training with synthetic data alone produces better forecast performance than training with real data alone. That evidence is presented in Fig 5, which shows the paired differences of rMAE and rWIS across 5,000 bootstrapped samples. The model comparisons were M(r,t) vs M(st,t) and M(r,v) vs M(sv,v). We see that in over 75% of bootstrapped samples, model M(st,t) outperformed M(r,t) in both rMAE and rWIS (i.e., the lower edge of the box of the box plot is at or above 0), while M(sv,v) outperformed M(r,v) in nearly 100% of bootstrapped samples.
Triangles indicate the 2.5 and 97.5 percentiles. Presented results show Model A - Model B (on the x-axis it is displayed as M(A) vs M(B)). For instance, M(r,t) vs M(st,t) in the rMAE panel presents the results of M(r,t)’s rMAE - M(st,t)’s rMAE, across all bootstrap samples. Positive numbers indicate Model B performed better than Model A. Fairly strong evidence is shown that models trained with synthetic data alone performed better than models trained with real data alone.
2.2.2 Q2: Does joint training with real and synthetic data improve COVID-19 case forecasts relative to training with either source individually?.
Yes, there is strong evidence that training with real and synthetic data is better than training with either source individually. See Fig 6. In all comparisons, at least 75% of bootstrapped samples resulted in model M(a,.) having better performance (rMAE or rWIS) than the analogous model trained with either real data alone or synthetic data alone. In all comparisons presented in Fig 6, there is no evidence that training with real and synthetic data is harmful relative to training with either source individually, making this joint training the safe choice (that is, using all available training data yielded better results than a subset).
Triangles indicate the 2.5 and 97.5 percentiles. Presented results show Model A - Model B (on the x-axis it is displayed as M(A) vs M(B)). For instance, M(r,t) vs M(a,t) in the rMAE panel presents the results of M(r,t)’s rMAE - M(a,t)’s rMAE, across all bootstrap samples. Positive numbers indicate Model B performed better than Model A. Clear evidence is shown that models trained with real and synthetic data (M(a,t) and M(a,v)), on balance, performed better than models trained with only-real or only-synthetic data.
Before moving on to question Q3, it’s worth taking a moment to contemplate why training on synthetic data alone outperformed training on real data alone (Q1) and why training on both sources outperformed either one individually (Q2). A simple hypothesis is related to the amount of training data available for each model: there are more synthetic training data (9 million examples) than there are real training data (
2 million examples), and there are (by definition) more real plus synthetic training data (
20 million examples) than there is of either source individually. Full training-data summaries are provided in Table 4. Under this hypothesis, more training data equates to better model performance. That is, maybe when comparing M(i,.) to M(j,.) for two different training data types i and j, the performance can be predicted simply by knowing the amount of training data available.
To investigate this hypothesis, we retrained the models on training data sizes of {2.5k, 25k, 250k, 1M, 2.5M, 5M, 10M} by taking subsets of the available training data of each type for all training data sizes less than or equal to all the available data. So, for clarity, M(r,.) is trained on training data sets of sizes 2.5k, 25k, 250k, 1M, as well as 2.1M, the full set of available real training examples. Similarly for models M(st,.), M(sv,.) and M(a,.). For each model/training data size, we calculate rMAE and rWIS averaged across all states, forecast dates, and forecast horizons. If our hypothesis is correct and the amount of training data is the driving force behind forecast improvement, we would expect to see similar performance for all models when trained on the same amount of training data, holding the input data type (TC or VAC) constant. If, however, for the same amount of training data we see some models consistently outperform other models (e.g., if M(a,.) > M(r,.)), that would be evidence against our hypothesis and would suggest the amount of training data does not solely explain why some models outperform others. Results from this exercise are shown in Fig 7.
Model performance improves as training data size increases until about one million training examples are used. “All” means the results when all available training examples for each model are used. Results presented are based on training runs with 25k model weight updates.
Fig 7 shows rMAE and rWIS for increasing amounts of training examples. We see that as the amount of training data increases from 2.5k examples to 1M examples, the performance of each model generally improves. In this regime, there is evidence that more training data correlates with better model performance. After models are trained with 1M training examples, however, forecast performance appears to level off. More importantly, there do appear to be systematic differences in model performance, even when holding the amount of training data constant. This can more clearly be seen in Fig 8.
Training data sizes are shown on the x-axis. “All” means the results when all available training examples for each model are used. Results are presented as Model A - Model B, and positive values mean that Model B performed better than Model A. Results only shown for 25k or more training examples.
Fig 8 shows the 95% confidence intervals for paired differences in rMAE and rWIS across 5,000 bootstrapped samples. While not universal, there is clear evidence shown in Fig 8 that the model trained with real and synthetic training data outperforms the models trained with only real or only synthetic training data even after holding the number of training data examples constant (2nd, 3rd, 5th, and 6th columns of Fig 8). Furthermore, the models trained only on synthetic data outperform those trained only on real data, even when the number of training examples is held constant (first and fourth columns of Fig 8). Fig 8 presents evidence against our hypothesis that the amount of training data is the explanation for M(a,.) outperforming all other models (Q2) and M(st,t)/M(sv,v) outperforming M(r,t)/M(r,v) (Q1). That is, something more than training data quantity is needed to explain the findings in Q1 and Q2.
Our other hypothesis for why training on real and synthetic data outperforms either source individually and why training with synthetic data only outperforms training with real data only centers around covariate shift [40]. Covariate shift in supervised machine learning (ML) refers to the change in the distribution of the input examples to a ML model between the training data (i.e., non-COVID-19, real respiratory data or synthetic data) and the testing data (i.e., COVID-19 data). Most supervised ML models assume that the training and testing data inputs (i.e., in this work, the last 20 observations of a time series) come from a common distribution. When there is a distributional mismatch, predictive performance can suffer. It could be the case that COVID-19 data are better represented by synthetic data than historical non-COVID-19, respiratory data (i.e., it could be synthetic data and COVID-19 data appear more alike than non-COVID-19, real respiratory data).
To investigate this possibility, we fit a gradient boosted model to perform binary classification trained on 150k non-COVID-19, real respiratory training examples and 150k synthetic TC training examples. The inputs to this model were the last 20 observations of a time series (mean centered and standard deviation scaled) and the labels “Synthetic TC” or “Real” were the output. No COVID-19 data were used to train this classifier. We then ran three different hold-out data sets through the classifier: 1) 8,000 instances of non-COVID-19, real respiratory data, 2) 8,000 instances of synthetic TC data, and 3) all 6,885 instances of COVID-19 data. The non-COVID-19, real respiratory data and synthetic TC data sets were used to evaluate the classifiers capabilities. The COVID-19 data were used to see if COVID-19 data better resemble non-COVID-19, real respiratory data or synthetic TC data. If the COVID-19 data are classified as “Synthetic TC” data, that would constitute evidence that COVID-19 data come from a distribution more similar to synthetic data than non-COVID-19, real respiratory data. Results are shown in Fig 9.
8,000 hold-out examples from the non-COVID-19 real respiratory and synthetic total cases, respectively are being summarized along with 6,885 COVID-19 examples. COVID-19 data are overwhelmingly classified as synthetic data rather than non-COVID-19, real respiratory data, while the classifier correctly classifies synthetic data as synthetic data and non-COVID-19, real respiratory data as non-COVID-19, real respiratory data. (bottom) UMAP arrangement of hold-out data, colored by the predicted probability synthetic. We can see the non-COVID-19, real respiratory data occupies a different part of UMAP space that the COVID-19 data, while the COVID-19 data and synthetic data have more overlap in UMAP space, shedding light on why the classifier classifies COVID-19 data as synthetic.
Overwhelmingly, COVID-19 data were classified as synthetic data rather than non-COVID-19, real respiratory data. This is seen in the top of Fig 9. Over 90% of COVID-19 instances were assigned a probability over 0.5 that the instance was synthetic TC data. The non-COVID-19, real respiratory data and the synthetic TC data were, with high probability, correctly classified as well. Over 90% of synthetic TC instances from the hold-out set were correctly assigned a probability over 0.5 of being synthetic TC, while just under 80% of the non-COVID-19, real respiratory data were assigned a probability over 0.5 of being non-COVID-19, real respiratory data. These results indicate that the trained classifier can discriminate between real and synthetic TC data.
The bottom of Fig 9 provides a UMAP [41] (Uniform Manifold Approximation and Projection) plot — a lower dimensional representation of the 20-dimensional inputs. UMAP finds a low-dimensional embedding of high-dimensional data that preserves local neighborhood relationships. Points that are close together in the 20-dimensional space are close together in the 2-dimensional plot in Fig 9 (and points that are far away remain far away). This UMAP plot helps explain why the classifier was able to discriminate between real and synthetic data and explain why COVID-19 data was overwhelmingly classified as synthetic data. The non-COVID-19, real respiratory data largely occupies the center of the UMAP plot. The synthetic data largely covers the whole space, but is particularly concentrated around the outer ring. The COVID-19 data also largely occupies the outer ring and, notably, does not occupy the center of the UMAP shape. Thus, there is evidence that the COVID-19 data inputs better match the distribution of inputs produced by synthetic data. Or, said another way, the covariate shift between non-COVID-19, real respiratory data (training) and COVID-19 data (testing) is much more pronounced than between synthetic data (training) and COVID-19 data (testing). Because the real and synthetic data, however, occupy different regions of input space, combining them results in improved input space coverage, providing insight into why M(a,t) and M(a,v) outperformed their modeling counterparts.
2.2.3 Q3a: Does training with synthetic VAC data improve COVID-19 forecasts relative to training with synthetic TC data?.
Yes, training with synthetic VAC data resulted in better forecasts than training with synthetic TC data. The evidence for this is shown in the M(st,t) vs M(sv,v) comparison in Fig 10. We see the 95% confidence intervals do not cover 0, indicating a statistically significant improvement in forecast performance for M(sv,v) relative to M(st,t) as measured by rMAE and rWIS.
Triangles indicate the 2.5 and 97.5 percentiles. Presented results show Model A - Model B (on the x-axis it is displayed as M(A) vs M(B)). For instance, M(st,t) vs M(sv,v) in the rMAE panel presents the results of M(st,t)’s rMAE - M(sv,v)’s rMAE, across all bootstrap samples. Positive numbers indicate Model B performed better than Model A.
2.2.4 Q3b: Do models with matched training data and input data outperform models with mismatched training data and input data?.
Yes, there is clear evidence that models with matched training data and input data outperform models with mismatched training data and input data. This can be seen in the M(sv,t) vs M(st,t) and M(st,v) vs M(sv,v) comparisons in Fig 10. In each comparison, the model with matched training and input data (M(st,t) and M(sv,v)) outperformed their counterpart with the same input data type but different training data type in at least 75% of bootstrapped samples. This makes intuitive sense. If the model is being trained on one data type but is then being tested on a different input type, there is the potential for both covariate shift [40] (when the distribution of inputs differs between training and testing) and concept shift [42] (when the relationship between inputs and outcomes differs between training and testing).
2.2.5 Q4: Does including SARS-CoV-2 variant information improve COVID-19 case forecasts?.
Yes, there is clear evidence that forecasting variant-attributable cases directly and summing to recover a total cases forecast improves forecasts relative to forecasting total cases directly. See Fig 11 for results. To answer this question, we compared all pairs of M(.,t) vs M(.,v). These comparisons hold the training data constant, and only differ in what data are passed in as inputs (total cases versus variant-attributable cases). The 95% confidence intervals fall above 0 for all comparisons for both metrics, indicating the models that take VACs as inputs and sum to recover total cases forecasts outperform the models that take TCs as inputs directly. Our finding that forecasting variant-attributable cases and summing produces better forecasts than forecasting total cases mirrors findings from influenza forecasting [43,44]. We conjecture that VACs have simpler and more predictable dynamics, making them easier to forecast, than the total cases time series which is a mixture of multiple co-circulating variants.
Triangles indicate the 2.5 and 97.5 percentiles. Presented results show Model A - Model B (on the x-axis it is displayed as M(A) vs M(B)). For instance, M(r,t) vs M(r,v) in the rMAE panel presents the results of M(r,t)’s rMAE - M(r,v)’s rMAE, across all bootstrap samples. Positive numbers indicate Model B performed better than Model A. Clear evidence is presented that models M(.,v) outperformed models M(.,t).
Fig 12 investigates during what phases of the COVID-19 pandemic models M(.,v) outperform models M(.,t). We partition each total cases time series into four phases: impending rise, rise, impending fall, and fall. Those phases are illustrated in the top of Fig 12. We then compute the difference in rMAE and rWIS by phase for each pair of models. We see clear evidence that the M(.,v) models outperform the M(.,t) models during the impending fall and fall phases of the pandemic. VAC and TC trained models perform more similarly to one another during the impending rise and rise phases, with TC models tending to outperform VAC models when performance is measured by MAE. The degree to which the TC models outperform the VAC models during the rising phase(s), however, is small relative to the degree the VAC models outperform the TC models during the impending fall and fall phases.
(bottom) The difference in rMAE and rWIS between models, broken down by phase for all states (not just Alabama). Points are average differences and error bars are 95% confidence intervals based on 5,000 bootstrap samples. Values greater than 0 mean the VAC models (M(.,v)) outperformed their corresponding TC model (M(.,t)). We see that the VAC models meaningfully outperform their TC counterparts during the impending fall and fall phases of the outbreak and perform more similarly to the TC models during the impending rise and rise phases.
2.2.6 Q5: How do these forecasts compare to real-time COVID-19 case forecasts?.
To answer this question, we refer to the results presented in [1]. Evaluation in [1] included relative WIS and coverage of 1–4 week ahead forecasts from July 28th, 2020 through December 21st, 2021. This comparison should be interpreted with important context. The COVIDHub forecasts were generated in real time, using data available at the time forecasts were submitted, whereas our forecasts were generated retrospectively using finalized COVID-19 case and sequence data. Although none of our models were trained on COVID-19 observations, the synthetic-data generator and observation model were developed after the COVID-19 pandemic and may have been informed, at least indirectly, by knowledge of COVID-19 dynamics. Thus, this comparison is not a strict real-time head-to-head evaluation, but rather a benchmark against a strong real-time ensemble over a common evaluation period.
See Fig 13 for WIS and coverage results for all models restricted to July, 2020 through December, 2021. All eight models outperformed a persistence (baseline) model (a feat only 7 out of 22 models accomplished in real-time). In this retrospective comparison, four models had lower rWIS than the COVIDHub-4_week_ensemble model, which had a relative WIS of 0.81 [1]. These models were the models trained on synthetic data only where the training data type matched the input data type (M(st,t) and M(sv,v)) as well as the two models trained with all the available training data (M(a,t) and M(a,v)). The models that did not outperform the COVIDHub-4_week_ensemble model were the models trained exclusively on real training data (M(r,t) and M(r,v)) and the models trained exclusively on synthetic data with a mismatch between training data type and model input type (M(sv,t) and M(st,v)). For additional context, even the worst performing model considered in this paper, M(r,t), had an rWIS that would rank fifth among the real-time forecasting models summarized in [1], again, recognizing the retrospective nature of our evaluation. Furthermore, the COVIDHub-4_week_ensemble had a 95% coverage of 0.8. All eight models had an empirical coverage between 0.84 and 0.88, closer to nominal than did the COVIDHub-4_week_ensemble. Additionally, although the VAC data input models (M(.,v)) could not have been forecast in real-time due to variant reporting delays, the TC data input models (M(.,t)) could have been forecast in real-time. It is also worth a reminder that none of the eight models in this paper used any COVID-19 data for training; they only used data (real and/or synthetic) that would have been available on or before January 1st, 2020.
Lower is better. The COVIDHub-4_week_ensemble had a relative WIS of 0.81 [1] (the horizontal dashed line). Models with rWIS below the horizontal, dashed line outperformed the COVIDHub-4_week_ensemble as measured by rWIS. (right) Empirical coverages for a 95% nominal prediction interval (solid line). The COVIDHub-4_week_ensemble’s 95% prediction intervals had an empirical coverage of 0.8 (dashed line). All models had empirical coverages closer to nominal than that.
3 Discussion
In this paper, we sought to answer two high-level questions: (1) Can synthetic data be used to train deep learning models that achieve state-of-the-art infectious disease forecasting performance? and (2) Can genetic information be used to improve infectious disease forecasting models? We conducted an exercise to answer these questions with an eye towards the next pandemic by limiting ourselves to only using training data available prior to January 1st, 2020 for forecasting COVID-19 cases. We affirmatively answered both questions. Synthetic data was a valuable training data source, as models trained on only synthetic data provided impressive forecasting performance. Real historical data for non-COVID-19 pathogens also proved to be a valuable training data source, though less valuable than synthetic on its own (Q1). Combining real and synthetic data was found to be a prudent path forward, as models trained with all available training data outperformed models trained on real or synthetic data alone (Q2). Furthermore, forecasting variant-attributable cases directly and summing of forecasts produces demonstrably better forecasts than forecasting total cases directly (Q4). Model M(a,v), the model that made use of all available training data and used variant information, was the best performing model.
The deep learning models trained and used in this exercise required training once, but required no retraining. While deep learning models can take several hours to train (details in Table 4), they take seconds to predict with and can scale to a large number of forecast settings. While we trained our models once, we could have either retrained every week as new data became available, or, more commonly, performed fine-tuning on our pre-trained model with the new COVID-19 data as it became available (e.g., [45]). Thus scaling a forecasting model from, say, US states to all US counties would be straightforward. This is in contrast to a spatial or hierarchical model that borrows information across locations, where retraining may be required every time new data are made available and scaling up from tens to hundreds or thousands of locations is not trivial (e.g., [3]). Again, these deep learning models require adequate amounts of training data to be effective, training data that are not available in an emerging disease setting. Historical outbreak data of related disease and/or synthetic data proved to be useful training data. This paper adds to the growing body of evidence that synthetic data is a valuable and under-tapped resource for infectious disease forecasting [36–38], as it provides the necessary ingredient to unlock the potential of deep learning models.
There are many opportunities to push this deep learning plus synthetic data idea further. Going back to the input/output deep learning framework, many future forecasting opportunities for improvement amount to augmenting the inputs. Forecasting geographic locations jointly rather than one at a time is an exciting path forward (recent real-time influenza forecasting success has employed a hierarchical modeling approach that borrows information across locations [3,46]). This amounts to developing training examples where, say, recent observations from all US states are the input and the next H time steps for each US state is the output, allowing the deep learning model to learn (potentially) complex inter-state relationships. To learn these relationships, however, will require suitable training data. MutAntiGen has multi-location capabilities thus could be a suitable synthetic data generator. Another application is jointly forecasting cases, hospitalizations, and deaths (as was done in [36]). Rather than forecasting variant-attributable cases, one could consider computing population genetic summaries of the viral population as time series and providing those as inputs paired with total cases. This approach would require a synthetic data simulator with appropriate evolutionary biology fidelity. In general, for this deep learning plus synthetic data idea to scale will require simulators sophisticated enough to make training data with sufficient realism, which will likely require continued simulator development.
While synthetic data is a promising resource to use, it is not a panacea. How to generate it requires thoughtful contemplation of the problem at hand. For instance, for much of the COVID-19 pandemic, data was collected and disseminated in the US at daily resolution, introducing day-of-week effects. Such effects are learnable and exploitable, but for a deep learning model to incorporate them, it would need to be presented with training data reflecting those patterns. We did not create any synthetic data having day-of-week effects, so we anticipate our approach might fail if applied to daily data. That said, to forecast at the daily scale, we could add a day-of-week routine to the observation model described in S1 Text. Similarly, to use this approach to forecast seasonal diseases, we would want to both ensure we are synthetically generating seasonal outbreaks as well to increase the context window (up to, say, C = 52 or 104), capturing at least one entire period of the outbreak.
Finally, while we conducted this retrospective forecasting study with an eye towards real-time forecasting, we acknowledge that reporting delays exist with both epidemiological and genetic data. Our results represent best-case scenarios with respect to reporting delays and more work is needed to bridge the real-time reporting delay gap.
4 Materials and methods
This section provides details on the study design, data sources, synthetic data generation process, forecasting model, and evaluation metrics used in the results above.
4.1 COVID-19 study details and data overview
4.1.1 Scope of study.
The study details are presented in Table 2. COVID-19 cases were selected as the target because they serve as a leading indicator of more severe outcomes like hospitalizations and deaths and were empirically difficult to forecast [1]. The time range and cadence correspond to data availability and public health relevance. U.S. states (plus Puerto Rico) were selected as a geographically relevant unit for public health. There is a large volume of SARS-CoV-2 genomes for US states between June 2020 and December 2022, allowing us to test the value of genetic information. Furthermore, major COVID-19 data collection and dissemination resources ramped down operation in March of 2023 [47].
4.1.2 Real COVID-19 case data.
Data on COVID-19 cases between December, 2019 and March, 2023 were obtained from the Johns Hopkins University Center for Systems Science and Engineering (CSSE) GitHub (https://github.com/CSSEGISandData/COVID-19_Unified-Dataset) [48]. This database includes COVID-19 case data compiled from a variety of sources; our analysis prioritized case counts for each location and date as reported from large, curated, cross-national databases when available. Delays in real-time reporting of COVID-19 cases were ignored in this analysis, and the data reported as of September 2023 (the date the data were pulled) for a given date were treated as known as of that date. As examples, we show the weekly total COVID-19 case counts for Alabama and California over the study period (Fig 14).
(a) Weekly total cases (TCs). (b) Proportion of sampled viral genomes assigned to each variant. (c) Variant-attributable cases (VACs), computed as TCs times the proportion of genomes assigned to each variant. VACs summed over all variants equal the TCs. Note the square-root scale on the y-axis for better visibility of low-count VACs.
4.1.3 Real COVID-19 genetic data.
The COVID-19 pandemic generated an unprecedented global collection of viral genome sequences, shared through repositories and data-sharing platforms including GISAID [17], GenBank [49], the European Nucleotide Archive [50], and, more recently, Pathoplexus [51]. Of these sources, GISAID provided the most sequence records, the broadest global representation, and the shortest times between sample collection and data release [18]. In this paper, we made use of approximately 4.5 million GISAID records for samples collected in the United States between June 1st, 2020 and December 31st, 2022. Rather than using raw genomic sequences, in this work we group sequence variants by Pango lineage designation [52] as assigned in the GISAID metadata as of September 2023; each genome is assigned to one of many discrete variant categories, which we aggregate into coherent encompassing supergroups. Aggregating these labels across time and location yields variant proportion time series, such as those shown in Fig 14B. We combine variant proportion time series with total cases time series to compute variant attributable case (VAC) time series (e.g., Fig 14C), where variant-attributable cases for each time point equal total cases times variant proportion.
We note that there is a meaningful delay between when a viral sample is collected and when its genome and metadata become publicly available. That lag — driven by lab turnaround, quality control, metadata completion, and curation — varies across geography and time [53]. This paper, like many retrospective studies, neglects this delay, but operational forecasting would need to model it. As such, the results in this paper for the models that use VACs as their input time series should be viewed in their appropriate context.
4.2 Training data
4.2.1 Real, non-COVID-19 respiratory disease data.
As early as late 2019, the outbreak later attributed to SARS-CoV-2 was described clinically as an acute respiratory illness (pneumonia) [54]. While little to no COVID-19 data would have been available on January 1st, 2020, we could have known that COVID-19 produced symptoms consistent with respiratory diseases. For this reason, we use non-COVID-19, real respiratory data available prior to January 1st, 2020 for training in this paper. All data and code used in this paper are available at https://github.com/lanl/precog/tree/main/synthetic_and_genetic_forecasting, originally derived from data found at https://github.com/lanl/precog/tree/main/infectious_timeseries_repo. Non-COVID-19 respiratory diseases include influenza, pneumonia, mumps, RSV, tuberculosis, and diphtheria (among others). In this paper, we will often use the shorthand “real data” or “real training data” to mean “non-COVID-19, real respiratory disease data.” For example, the “Training Data = Real” in Table 1 means “non-COVID-19, real respiratory disease data.”
As can be seen in Table 3, about 2,000 non-COVID-19, real respiratory time series are available for training. Those time series have an average length of about 1000 observations, and a median length of about 550 observations. A selection of the non-COVID-19, real respiratory time series are shown in Fig 15.
Over 2,000 time series are available for training, amounting to over 2 million observations.
4.2.2 Synthetic data.
In addition to the real data, we generated synthetic data to represent a wide range of possible disease behaviors. The intent of these synthetic data is to discover behaviors that are within scope of possible disease dynamics, yet not explicitly represented in the available real data observations. We generate synthetic disease data via the MutAntiGen agent-based model (ABM) [39,55,56] because it can generate viral variant turnover dynamics.
MutAntiGen. In agent-based modeling, a large-scale ecological system is simulated as a collection of autonomous decision-making entities called agents [57]. In the MutAntiGen ABM, originally developed [39] as an extension of an antecedent ABM [55], “agents” are categorized as either infected or non-infected individuals in a population susceptible to disease spread. MutAntiGen explicitly models the joint behavior of an evolving pathogen and dynamic susceptibility/resistance of the host population, allowing for non-seasonal case waves. Further details on MutAntiGen are available in S1 Text.
Some example runs of MutAntiGen are shown in Fig 16. In addition to cases over time, the simulator reports viral samples from a subset of infections. This yields time series of cases attributable to antigenic types, as shown in the lower panels of Fig 16. Viral sampling typically is proportional to number of cases, but this yields very few samples when cases are low before a new wave begins, which is exactly when data are critical for a forecasting model. We therefore modified the MutAntiGen code to sample with greater intensity when cases are low.
MutAntiGen outputs both the total number of cases (top row, TC) and the time series of cases attributed to each variant (bottom row, VAC; each line and color represents a different variant). For each time point, the sum of all variant-attributable cases (bottom row) equals the total cases (top row).
As one might imagine, an ABM intended to represent a massively complicated environmental system includes many parameters to be set by the user prior to running a simulation. These parameters greatly affect the outcomes of the simulation, but it is still difficult to predict the results of the simulation due to the innate stochasticity of ABMs [58]. The required MutAntiGen parameters and their biological interpretations, as well as our computational modifications to the original code that allow for better sampling at scale, are discussed in S1 Text.
Design to produce MutAntiGen runs Our goal in generating synthetic data was to produce a rich and diverse set of training data for a forecasting model, emphasizing effective representation of both antigenic mutation (an individual property) and turnover in dominant antigenic type (a property of populations). Since the aim is to forecast emergent diseases, we are unlikely to know ahead of time what parameter choices will best reflect an impending outbreak; therefore these simulations must cast as wide a net as is practicable, including a broad range of model parameters so that future outbreaks would be more likely to fall within that net (i.e., be represented in the sample space).
To select the parameter values we run MutAntiGen at, we draw a Latin hypercube sample (LHS) [59]. Our default MutAntiGen parameter values and ranges were informed by the literature and calibrated to represent global influenza H3N2 phylodynamics. We expanded a subset of the parameters that control the evolutionary, epidemiological, and immunological dynamics to yield more general simulations. For parameters with established empirical estimates, we set plausible bounds based on data from representative RNA viruses including influenza A, influenza B, Measles, Nipah, Dengue, Zika, and Hepatitis C.
By nature of LHS’s independent sampling, many combinations of unrealistic or unknown viruses could be generated, with parameter combinations that do not respect known biological constraints (e.g., error catastrophe, mutation scaling rates [60,61]) or outbreak conditions (R0 > 1). However, we allowed such combinations under the assumption that biologically nonviable regimes would result in simulation failure or additional data that the model would be able to determine to be irrelevant for the prediction task. Previous work has shown that the risk of negative transfer, where model performance degrades with the addition of new data, is low as long as at least some of the training data resemble the disease being forecasted [62]. The specifics of parameter bounds used in these simulations, our rationales for selecting them, and a full list of MutAntiGen parameters held constant are available in S1 Text.
Observation model for synthetic data As can be seen in Fig 16, the output of MutAntiGen can have unrealistically low noise. Real data, in contrast, are noisy, biased, and often include outliers (compare Fig 15 to Fig 16). That is, real data can be thought of as imperfect versions of idealized epidemiological data. In an effort to make the synthetic outputs of MutAntiGen more realistic, we passed each MutAntiGen output through an observation model 20 times, resulting in different imperfect versions of each MutAntiGen time series. At a high level, the observation model applies three possible transformations. First, it rescales the time axis by linearly interpolating each simulated outbreak to a shorter time series, allowing the same simulated dynamics to occur over faster timescales. Second, for half of the observation-model realizations, it applies multiplicative noise to the case counts. Third, each realization has a probability of receiving both high and low outliers, representing reporting artifacts such as data dumps or missed reports. This observation model increases both the amount and the diversity of the training data. Fig 17 shows different realizations of the observation model for a single MutAntiGen output. After sending all MutAntiGen runs through the observation model process, we generated approximately 36,000 total time series for training for both the total cases and the variant-attributable cases (see Table 3 for specifics). More details of the observation model can be found in S1 Text.
Realizations were generated by subjecting the “clean” MutAntiGen output to either scaling (random-magnitude compression of the x-axis) and (possible) addition of outliers (top row), or to scaling plus addition of noise and (possibly) outliers (bottom row).
4.3 Forecasting model
4.3.1 Training.
We frame our probabilistic forecasting problem as a conditional quantile regression problem. Let
be the conditional distribution for , the number of new infections reported at time t + h for
and
given the last C newly reported infections
. In this paper, we set C = 20 and H = 4. The context length C = 20 was chosen as a practical compromise rather than through a formal optimization study. Shorter context windows allow the model to be used earlier in an outbreak, since at least C observations are needed before a forecast can be generated. Longer context windows provide more recent history from which the model can infer outbreak dynamics, variant turnover, and changes in trajectory. We chose 20 weeks to provide several months of recent context while still allowing forecasts relatively early in an emerging-outbreak setting.
We define the quantile function as follows:
for . That is, the conditional quantile function
— doubly indexed by the quantile level
and the step ahead h — is the inverse of the conditional cumulative distribution function evaluated at the quantile level
,
. We approximate the inverse of the conditional cumulative distribution function
with a set of quantile functions
evaluated over a grid of quantile levels
where
and .
constitutes a dense grid of quantile levels, denser than we use for evaluation (see Section 4.4 for details). This dense grid, however, allows us to better approximate the tails of the forecast distributions which will be needed in the forecasting of models M(r,v), M(st,v), M(sv,v), and M(a,v) (see Section 4.3.2 for details). The quantile function provides a point forecast and forecast intervals. For instance, the point forecast is the evaluated quantile function when
(the median). The 95% forecast interval lower and upper bounds are the quantile functions evaluated for quantile levels
and
, respectively.
Given our problem statement, our next task is to estimate . To do that, we turn to deep learning [63]. Deep learning models are flexible function approximators. Provided an adequate amount of training data, deep learning models can learn continuous functions to high degrees of precision [64]. Using the training data described in Section 4.2, we train a 2-layer transformer model [65] that takes the last 20 observations of a time series as input and predicts the quantile levels in
for
. Pinball loss is used to perform the quantile regression. Training and deep learning model details can be found in S1 Text.
The results of the model fitting are four different trained deep learning models, differing only in the training data used to learn the model parameters: real data only (i.e., all non-COVID-19, real respiratory data detailed in Section 4.2.1), synthetic total cases, synthetic variant-attributable cases, and all training data (i.e., non-COVID-19, real respiratory data, synthetic total cases data, and synthetic variant-attributable cases). As is detailed in Table 4, with C = 20, we are able to generate between 2 and 21 million input/output pairs of training data where is the input and
is the output. To be clear, if there is a training time series of length T = 100, that will produce
input/output training data examples (e.g.,
,
, ...,
).
Each fitted model was trained for a fixed run length of approximately 25 million presented training examples. This value was chosen as a practical, conservative training length rather than through a formal tuning exercise. We used the same run length for all training data sources so that differences across models would not be driven by different numbers of weight-update opportunities. During training, validation pinball loss was monitored, and the final forecasting model was selected as the exponential-moving-average checkpoint with the lowest validation loss.
Table 4 presents summaries of model training. Each model was trained on between 2 and 21 million unique training examples. Each model was presented with approximately 25 million training examples during training. Thus, each training example was viewed by the model between 1 and 12 times (example views), depending on the number of available training examples. Deep learning models run the risk of memorizing the training data and thus overfitting if they are presented the same training examples repeatedly. The large numbers of training examples shown in Table 4 help protect against overfitting. The training time is a function of the number of training examples presented to the model (25 million), not the number of available training examples. This is why the training time does not scale with the number of training examples. The time to train the model (almost 12 hours) would be considered expensive (possibly prohibitive) if retraining was required every week when new data become available for real-time forecasting. The forecasting approach considered here falls under the “expensive to train but cheap to deploy” paradigm. While it takes on the order of half a day to train these models, they only need to be trained once. Forecasting with these trained models is measured on the scale of seconds (or less), making them appealing in operational settings. It is worth noting that the training time could be shortened by making use of graphical processing units (GPUs).
4.3.2 Forecasting.
Forecasting differs depending on the model input (recall Table 1). The models that take total cases as the input are straightforward to forecast. The fitted transformers produce forecasts for all quantile levels in . We only use seven of those quantile levels,
, allowing us to produce a point forecast (the median) and three prediction intervals: 50%, 80% and 95%. These quantile predictions allow us to evaluate forecasts with respect to multiple popular metrics (see Section 4.4 for details).
Forecasting the models where the model input are the variant-attributable cases (VACs) requires one more step. For a given state, forecast date (t), and forecast horizon (h), we forecast each VAC at the dense grid of 27 quantile levels in . Using those 27 quantiles as an estimate of the cumulative distribution function (CDF) for each VAC, we draw a realization from each VAC’s CDFs by inverting the CDF [66] and linearly interpolating between quantile estimates (this is why
has quantile estimates so far out in the tails). Given a draw from each VAC’s forecast distribution, we sum those draws constituting a single draw from the total cases forecast distribution. We repeat this sampling process N = 100,000 times. Then, we compute the same seven quantile levels {0.025, 0.1, 0.25, 0.5, 0.75, 0.9, 0.975} as the sample quantiles of the N draws from the total cases forecast distribution, derived from each individual VAC forecast distribution.
Notice that pairs of models in Table 1 use the same fitted transformer model to produce forecasts. For example, models M(r,t) and M(r,v) each use the same fitted transformer model, trained with only non-COVID-19, real respiratory data, but will yield different forecasts because the inputs to the model are different (recall Fig 14A & 14C).
4.4 Evaluation metrics
We evaluated forecasts using mean absolute error (MAE), weighted interval score (WIS), and empirical coverage.
MAE is defined as
where N is the number of forecasts, is the observation, and
is the point forecast. MAE is greater than or equal to 0 and is negatively oriented (smaller is better). The point forecast
for this work is the median forecast from the quantile regression. Most of the evaluation results are presented relative to a persistence model — a model whose forecast
for any
(i.e., M(0)). As such, we also define relative MAE (rMAE) as follows:
where is any model i listed in Table 1. rMAE is greater than or equal to 0 and negatively oriented. rMAE(
) < 1 indicates that model
has a better MAE than a persistence model.
WIS is a way to evaluate forecasts in an interval format. Intuitively, WIS penalizes two things: interval widths (the wider the interval width, the larger the penalty) and observations that fall outside the forecast interval (the further the observation falls outside the forecast interval, the larger the penalty). As such, WIS encourages forecasts to be sharp and well-calibrated [67]. We consider K = 3 central forecast intervals: 50%, 80% and 95% (corresponding to
, respectively). Following [68], WIS is defined as
where w0 = 0.5, ,
(the median forecast), y is the observations, and
is the interval score corresponding to
, defined as
where and
are the
and
quantiles and I() is an indicator function equal to 1 if the argument is true and 0 otherwise. WIS is greater than or equal to zero and is negatively oriented. Similar to MAE, relative WIS (rWIS) is defined as follows:
where is model i. rWIS(
) < 1 means model
has a better WIS than a persistence model. The persistence model (
) does not intrinsically produce forecast intervals. We constructed prediction intervals for the persistence model following the COVIDHub-baseline procedure described in the Supplemental Materials of [69], with the predictive median at each horizon set equal to the most recent observed weekly case count,
. To represent uncertainty around this median, we used the past observed changes in each state’s time series separately for each forecast horizon. Specifically, for a h-step-ahead forecast made at time t, we collected the historical h-step differences,
, and their negatives,
, for all available past times
. We then formed a sampling distribution for h-step-ahead changes using a piecewise linear approximation to the empirical cumulative distribution function of these values. For each horizon h, we drew 100,000 samples from this horizon-specific distribution of h-step changes and added each sampled change to the last observed value
, yielding a Monte Carlo approximation to the h-step-ahead predictive distribution. The resulting simulated incidence values were truncated at zero to prevent negative forecasts. Forecast quantiles were computed from this truncated predictive distribution, with the median forced to equal
. For more details, see the section titled “Methods for creating baseline forecast.” in the Supplemental Materials of [69].
Empirical coverage for a % forecast interval is the proportion of forecast intervals that contain the observation y. Empirical coverage is defined as
where and
are the lower and upper quantile estimates corresponding to the
forecast interval. Probabilistically well-calibrated forecasts are those whose empirical coverages are nearly equal to their nominal coverages (i.e.,
) for all
levels.
Supporting information
S1 Fig. MutAntiGen stochasticity.
Ten MutAntiGen outputs simulated from identical input parameters.
https://doi.org/10.1371/journal.pcbi.1014630.s001
(EPS)
S2 Fig. Bootstrap comparison.
Boxplots for 5,000 bootstrap samples for rMAE and rWIS from the blocked bootstrap procedure (block), taking into account state and forecast date groupings, and the vanilla bootstrap procedure (iid), which does not address groupings. Wider uncertainties are observed when taking account of the grouping variables. The white, dotted horizontal line is the point estimate for rMAE and rWIS.
https://doi.org/10.1371/journal.pcbi.1014630.s002
(EPS)
S1 Table. Initial parameter ranges of the Latin Hypercube Sample.
https://doi.org/10.1371/journal.pcbi.1014630.s003
(XLSX)
S1 Text. Supplemental Material.
Additional details on the MutAntiGen simulator, computational modifications, observation model, model training, and bootstrap procedures.
https://doi.org/10.1371/journal.pcbi.1014630.s004
(PDF)
Acknowledgments
We gratefully acknowledge all data contributors, i.e., the authors and their originating laboratories responsible for obtaining the specimens, and their submitting laboratories for generating the genetic sequence and metadata and sharing via the GISAID Initiative, on which some of this research is based. Research presented in this article was supported by the Laboratory Directed Research and Development program of Los Alamos National Laboratory under project number 20240066DR. Los Alamos National Laboratory is operated by Triad National Security, LLC, for the National Nuclear Security Administration of the U.S. Department of Energy under Contract No. 89233218CNA000001. The authors acknowledge the use of ChatGPT for text editing and assistance with selected coding tasks. The authors retain full responsibility for the manuscript’s content. This article has been approved for unlimited release under LA-UR-26–22310.
References
- 1. Lopez VK, Cramer EY, Pagano R, Drake JM, O’Dea EB, Adee M, et al. Challenges of COVID-19 Case Forecasting in the US, 2020–2021. PLoS Comput Biol. 2024;20(5):e1011200.
- 2. Brooks LC, Farrow DC, Hyun S, Tibshirani RJ, Rosenfeld R. Flexible Modeling of Epidemics with an Empirical Bayes Framework. PLoS Comput Biol. 2015;11(8):e1004382. pmid:26317693
- 3. Osthus D, Moran KR. Multiscale influenza forecasting. Nat Commun. 2021;12(1):2991. pmid:34016992
- 4. Ray EL, Wang Y, Wolfinger RD, Reich NG. Flusion: Integrating multiple data sources for accurate influenza predictions. Epidemics. 2025;50:100810. pmid:39818098
- 5. Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis. 2020;20(5):533–4. pmid:32087114
- 6.
Kolman, Shannon. Disease Forecasting Tools Can Support Policymaking During Epidemics. 2023. Accessed: 2025-07-13. Available from: https://www.ncsl.org/health/disease-forecasting-tools-can-support-policymaking-during-epidemics
- 7. Shadbolt N, Brett A, Chen M, Marion G, McKendrick IJ, Panovska-Griffiths J, et al. The challenges of data in future pandemics. Epidemics. 2022;40:100612. pmid:35930904
- 8. Beesley LJ, Osthus D, Del Valle SY. Addressing delayed case reporting in infectious disease forecast modeling. PLoS Comput Biol. 2022;18(6):e1010115. pmid:35658007
- 9. Lyu H, Imtiaz A, Zhao Y, Luo J. Human behavior in the time of COVID-19: Learning from big data. Front Big Data. 2023;6:1099182. pmid:37091459
- 10. Kapitsinis N. The underlying factors of the COVID‐19 spatially uneven spread. Initial evidence from regions in nine EU countries. Reg Sci Policy Pract. 2020;12(6):1027–46.
- 11. Korber B, Fischer WM, Gnanakaran S, Yoon H, Theiler J, Abfalterer W, et al. Tracking Changes in SARS-CoV-2 Spike: Evidence that D614G Increases Infectivity of the COVID-19 Virus. Cell. 2020;182(4):812-827.e19. pmid:32697968
- 12. Beesley LJ, Moran KR, Wagh K, Castro LA, Theiler J, Yoon H, et al. SARS-CoV-2 variant transition dynamics are associated with vaccination rates, number of co-circulating variants, and convalescent immunity. EBioMedicine. 2023;91:104534. pmid:37004335
- 13. Ioannidis JPA, Cripps S, Tanner MA. Forecasting for COVID-19 has failed. Int J Forecast. 2022;38(2):423–38. pmid:32863495
- 14. Cramer EY, Huang Y, Wang Y, Ray EL, Cornell M, Bracher J, et al. The United States COVID-19 Forecast Hub dataset. Sci Data. 2022;9(1):462. pmid:35915104
- 15. Sherratt K, Gruson H, Grah R, Johnson H, Niehus R, Prasse B, et al. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations. Elife. 2023;12:e81916. pmid:37083521
- 16. Ling-Hu T, Rios-Guzman E, Lorenzo-Redondo R, Ozer EA, Hultquist JF. Challenges and Opportunities for Global Genomic Surveillance Strategies in the COVID-19 Era. Viruses. 2022;14(11):2532. pmid:36423141
- 17. Shu Y, McCauley J. GISAID: Global initiative on sharing all influenza data - from vision to reality. Euro Surveill. 2017;22(13):30494. pmid:28382917
- 18. Korber B, Fischer W, Theiler J. Real-time monitoring of SARS-CoV-2 evolution during the COVID-19 pandemic. Cell Host Microbe. 2025;33(11):1802–6. pmid:41232510
- 19. Du H, Dong E, Badr HS, Petrone ME, Grubaugh ND, Gardner LM. Incorporating variant frequencies data into short-term forecasting for COVID-19 cases and deaths in the USA: a deep learning approach. EBioMedicine. 2023;89:104482. pmid:36821889
- 20.
Hyndman RJ, Athanasopoulos G. Forecasting: Principles and Practice. OTexts; 2018.
- 21.
Brauer F. Compartmental models in epidemiology. Mathematical Epidemiology. 2008. p. 19–79.
- 22. Tracy M, Cerdá M, Keyes KM. Agent-Based Modeling in Public Health: Current Applications and Future Directions. Annu Rev Public Health. 2018;39:77–94. pmid:29328870
- 23. Featherstone LA, Zhang JM, Vaughan TG, Duchene S. Epidemiological inference from pathogen genomes: A review of phylodynamic models and applications. Virus Evol. 2022;8(1):veac045. pmid:35775026
- 24. McGowan CJ, Biggerstaff M, Johansson M, Apfeldorf KM, Ben-Nun M, Brooks L. Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016. Sci Rep. 2019;9(1):683.
- 25. Kamalov F, Rajab K, Cherukuri AK, Elnagar A, Safaraliev M. Deep learning for Covid-19 forecasting: State-of-the-art review. Neurocomputing (Amst). 2022;511:142–54. pmid:36097509
- 26.
Adhikari B, Xu X, Ramakrishnan N, Prakash BA. EpiDeep: Exploiting Embeddings for Epidemic Forecasting. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’19. New York, NY, USA: Association for Computing Machinery. 2019. p. 577–86.
- 27. Chimmula VKR, Zhang L. Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos Solitons Fractals. 2020;135:109864. pmid:32390691
- 28.
Xie F, Zhang Z, Li L, Zhou B, Tan Y. EpiGNN: Exploring Spatial Transmission with Graph Neural Network for Regional Epidemic Forecasting. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part VI. Berlin, Heidelberg: Springer-Verlag; 2022. p. 469–85.
- 29. Wang L, Adiga A, Chen J, Sadilek A, Venkatramanan S, Marathe M. CausalGNN: Causal-Based Graph Neural Networks for Spatio-Temporal Epidemic Forecasting. AAAI. 2022;36(11):12191–9.
- 30. Voznica J, Zhukova A, Boskova V, Saulnier E, Lemoine F, Moslonka-Lefebvre M, et al. Deep learning from phylogenies to uncover the epidemiological dynamics of outbreaks. Nat Commun. 2022;13(1):3896. pmid:35794110
- 31. Perez MF, Gascuel O. PhyloCNN: Improving Tree Representation and Neural Network Architecture for Deep Learning from Trees in Phylodynamics and Diversification Studies. Syst Biol. 2025;75(4):776–95. pmid:41273351
- 32. Roster K, Connaughton C, Rodrigues FA. Forecasting new diseases in low-data settings using transfer learning. Chaos Solitons Fractals. 2022;161:112306. pmid:35765601
- 33.
University of Pittsburgh. Project Tycho. 2026. Accessed 2026-02-03. https://www.tycho.pitt.edu/data/
- 34. Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? Adv Neural Inform Process Syst. 2014;27.
- 35. Pan SJ, Yang Q. A Survey on Transfer Learning. IEEE Trans Knowl Data Eng. 2009;22(10):1345–59.
- 36.
Dudley C, Magdaleno R, Harding C, Sharma A, Martin E, Eisenberg M. Mantis: A simulation-grounded foundation model for disease forecasting. arXiv preprint arXiv:250812260. 2025.
- 37. Murph AC, Gibson GC, Amona EB, Beesley LJ, Castro LA, Del Valle SY, et al. Synthetic method of analogues for emerging infectious disease forecasting. PLoS Comput Biol. 2025;21(6):e1013203. pmid:40549797
- 38. Murph AC, Beesley LJ, Gibson GC, Castro LA, Del Valle SY, Osthus D. A disease-agnostic approach to ensemble learning for infectious disease forecasting. Nat Commun. 2026;17(1):4255. pmid:41862508
- 39. Koelle K, Rasmussen DA. The effects of a deleterious mutation load on patterns of influenza A/H3N2’s antigenic evolution in humans. eLife. 2015;4:e07361.
- 40.
Nair NG, Satpathy P, Christopher J. Covariate Shift: A Review and Analysis on Classifiers. In: 2019 Global Conference for Advancement in Technology (GCAT). IEEE; 2019. p. 1–6.
- 41.
McInnes L, Healy J, Melville J. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:180203426. 2018.
- 42. Iwashita AS, Papa JP. An Overview on Concept Drift Learning. IEEE Access. 2018;7:1532–47.
- 43. Kandula S, Yang W, Shaman J. Type- and Subtype-Specific Influenza Forecast. Am J Epidemiol. 2017;185(5):395–402. pmid:28174833
- 44. Turtle J, Riley P, Ben-Nun M, Riley S. Accurate influenza forecasts using type-specific incidence data for small geographic units. PLoS Comput Biol. 2021;17(7):e1009230. pmid:34324487
- 45. Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. LoRA: Low-rank adaptation of large language models. ICLR. 2022;1(2):3.
- 46.
Case B, Salcedo MV, Fox SJ. An accurate hierarchical model to forecast diverse seasonal infectious diseases. medRxiv. 2025;2025–3.
- 47.
Johns Hopkins University. Johns Hopkins COVID-19 data hub ends after three years. 2023. Accessed 2026-02-04. https://hub.jhu.edu/2023/03/10/coronavirus-resource-center-data-hub-ends/
- 48. Badr HS, Zaitchik BF, Kerr GH, Nguyen N-LH, Chen Y-T, Hinson P, et al. Unified real-time environmental-epidemiological data for multiscale modeling of the COVID-19 pandemic. Sci Data. 2023;10(1):367. pmid:37286690
- 49. Benson DA, Cavanaugh M, Clark K, Karsch-Mizrachi I, Lipman DJ, Ostell J. Nucleic Acids Research. 2012;41(D1):D36–42.
- 50. Leinonen R, Akhtar R, Birney E, Bower L, Cerdeno-Tárraga A, Cheng Y. Nucleic Acids Res. 2010;39(suppl_1):D28–31.
- 51. Dalla Vecchia E. Pathoplexus: towards fair and transparent sequence sharing. Lancet Microbe. 2024;5(12):100995. pmid:39284333
- 52. O’Toole Á, Scher E, Underwood A, Jackson B, Hill V, McCrone JT, et al. Assignment of epidemiological lineages in an emerging pandemic using the pangolin tool. Virus Evol. 2021;7(2):veab064. pmid:34527285
- 53. Brito AF, Semenova E, Dudas G, Hassler GW, Kalinich CC, Kraemer MUG, et al. Global disparities in SARS-CoV-2 genomic surveillance. Nat Commun. 2022;13(1):7003. pmid:36385137
- 54.
World Health Organization. Pneumonia of unknown cause – China; 2020. Accessed: 2026-03-04. https://www.who.int/emergencies/disease-outbreak-news/item/2020-DON229
- 55. Bedford T, Rambaut A, Pascual M. Canalization of the evolutionary trajectory of the human influenza virus. BMC Biol. 2012;10:38. pmid:22546494
- 56. Castro LA, Bedford T, Ancel Meyers L. Early prediction of antigenic transitions for influenza A/H3N2. PLOS Computat Biol. 2020;16(2):1–23.
- 57. Bonabeau E. Agent-based modeling: methods and techniques for simulating human systems. Proc Natl Acad Sci U S A. 2002;99(Suppl 3):7280–7. pmid:12011407
- 58.
Gilbert N. Agent-based models. Sage Publications; 2019.
- 59. McKay MD, Beckman RJ, Conover WJ. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics. 1979;21(2):239.
- 60. Drake JW, Holland JJ. Mutation rates among RNA viruses. Proc Natl Acad Sci U S A. 1999;96(24):13910–3. pmid:10570172
- 61. Holmes EC. The Evolutionary Genetics of Emerging Viruses [Journal Article]. Annu Rev Ecol Evol Syst. 2009;40(Volume 40, 2009):353–72. Available from:
- 62.
Beesley LJ, Murph AC, Osthus D, Castro LA. Transfer learning using 66 diseases for disease forecasting applications. 2026. https://arxiv.org/abs/2605.27269
- 63. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44. pmid:26017442
- 64. Hornik K, Stinchcombe M, White H. Multilayer feedforward networks are universal approximators. Neural Netw. 1989;2(5):359–66.
- 65. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN. Attention is all you need. Adv Neural Inform Process Syst. 2017;30.
- 66.
Devroye L. Chapter 4 Nonuniform Random Variate Generation. Handbooks in Operations Research and Management Science. Elsevier; 2006. p. 83–121.
- 67. Gneiting T, Balabdaoui F, Raftery AE. Probabilistic Forecasts, Calibration and Sharpness. J R Stat Soc B: Stat Methodol. 2007;69(2):243–68.
- 68. Bracher J, Ray EL, Gneiting T, Reich NG. Evaluating epidemic forecasts in an interval format. PLoS Comput Biol. 2021;17(2):e1008618. pmid:33577550
- 69. Cramer EY, Ray EL, Lopez VK, Bracher J, Brennen A, Castro Rivadeneira AJ, et al. Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States. Proc Natl Acad Sci U S A. 2022;119(15):e2113561119. pmid:35394862