Forecasts of influenza activity in human populations could help guide key preparedness tasks. We conducted a scoping review to characterize these methodological approaches and identify research gaps. Adapting the PRISMA methodology for systematic reviews, we searched PubMed, CINAHL, Project Euclid, and Cochrane Database of Systematic Reviews for publications in English since January 1, 2000 using the terms “influenza AND (forecast* OR predict*)”, excluding studies that did not validate forecasts against independent data or incorporate influenza-related surveillance data from the season or pandemic for which the forecasts were applied. We included 35 publications describing population-based (N = 27), medical facility-based (N = 4), and regional or global pandemic spread (N = 4) forecasts. They included areas of North America (N = 15), Europe (N = 14), and/or Asia-Pacific region (N = 4), or had global scope (N = 3). Forecasting models were statistical (N = 18) or epidemiological (N = 17). Five studies used data assimilation methods to update forecasts with new surveillance data. Models used virological (N = 14), syndromic (N = 13), meteorological (N = 6), internet search query (N = 4), and/or other surveillance data as inputs. Forecasting outcomes and validation metrics varied widely. Two studies compared distinct modeling approaches using common data, 2 assessed model calibration, and 1 systematically incorporated expert input. Of the 17 studies using epidemiological models, 8 included sensitivity analysis. This review suggests need for use of good practices in influenza forecasting (e.g., sensitivity analysis); direct comparisons of diverse approaches; assessment of model calibration; integration of subjective expert input; operational research in pilot, real-world applications; and improved mutual understanding among modelers and public health officials.
Citation: Chretien J-P, George D, Shaman J, Chitale RA, McKenzie FE (2014) Influenza Forecasting in Human Populations: A Scoping Review. PLoS ONE 9(4): e94130. https://doi.org/10.1371/journal.pone.0094130
Editor: Nicholas G. Reich, University of Massachusetts, United States of America
Received: December 18, 2013; Accepted: March 12, 2014; Published: April 8, 2014
Copyright: © 2014 Chretien et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Funding for JS was provided by US NIH grant GM100467 and the NIH Models of Infectious Disease Agent Study program through cooperative agreement 1U54GM088558, as well as NIEHS Center grant ES009089 and the RAPIDD program of the Science and Technology Directorate, US Department of Homeland Security, and Fogarty International Center, National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Seasonal influenza epidemics caused by influenza A and B viruses occur annually during the winter in temperate regions, resulting in around 3–5 million cases of severe illness and 250,000–500,000 deaths worldwide each year . In contrast to seasonal influenza, novel influenza A strains capable of sustained person-to-person transmission arise occasionally. These novel strains may evade existing antibody immunity and give rise to pandemic outbreaks. For example, the 1918 pandemic caused around 20–40 million deaths , while pandemics in 1957 and 1968 involved many infections but fewer deaths than in the 1918 pandemic. A 2009 pandemic strain, influenza A(H1N1)pdm09, continues to circulate as a seasonal virus.
Accurate forecasts of influenza activity based on predictive models could facilitate key preparedness actions, such as public health surveillance, development and use of medical countermeasures (e.g., vaccine and antiviral drugs), communication strategies, deployment of Strategic National Stockpile assets in anticipation of surge demands (e.g., ventilators), and hospital resource management (e.g., for staf and beds). Early in a potential pandemic, forecasts of international spread could help guide public health actions globally.
Previous reviews have assessed influenza modeling (e.g., –), but to our knowledge only one focused specifically on the use of models to forecast influenza activity, as opposed to other important applications of influenza modeling (such as improving understanding of the epidemiological dynamics or evaluating control strategies). This recent review, by Nsoesie et al. , identified 16 studies that aimed to forecast influenza outbreaks at local, regional, national, or global level. To more systematically characterize influenza forecasting methods and applications, and identify promising approaches and research gaps, we conducted a scoping review of the peer-reviewed influenza forecasting literature. We assess differences in methodological approaches and provide recommendations for future influenza forecasting models.
Materials and Methods
We adapted the PRISMA methodology  for our scoping review. In contrast to a systematic review, which focuses on a well-defined research question and may include a narrow range of study designs, a scoping review addresses broader topics and may include various study designs . We included studies that described methods to forecast future influenza activity in human populations using dynamic influenza-related surveillance data, and that tested the forecasting approach against independent data (real or simulated). We defined “dynamic” data as data collected during an epidemic or pandemic to make predictions about its subsequent course. We excluded studies that predicted current influenza activity not observed at the time of prediction because of reporting delays (sometimes called “nowcasting”).
We searched PubMed, CINAHL, Project Euclid, and Cochrane Database of Systematic Reviews for publications in English since January 1, 2000 using the terms “influenza AND (forecast* OR predict*)” in any field, and analyzed abstracts of returned publications to identify candidates for full-text review (i.e., studies for which inclusion criteria were met or for which it was not possible to determine whether inclusion criteria were met). We also manually searched the reference lists of included papers, our bibliographies, and the International Journal of Forecasting, and considered recommendations of colleagues.
We abstracted the data as follows. One of us conducted the literature search on December 4, 2013 and determined which publications to include based on the abstract or full text. For included publications, the reviewer recorded the geographic setting, data timeframe, whether the focus was seasonal or pandemic influenza, details of the input data and analytical methods, and reported forecast accuracy. Another reviewer independently abstracted the data from selected publications. The two reviewers resolved discrepancies through consensus.
We included 35 publications in the review – (Figure 1). Twenty six (74%) of the studies were published in 2009 or later, with more than one-third published in 2012 or 2013. The studies fell into 3 categories based on the epidemiological application: population-based seasonal influenza forecasting (N = 27 publications), medical facility-based forecasting of patient counts for seasonal or pandemic influenza (N = 4), and regional or global spread forecasting for pandemic influenza (N = 4) (Table 1). Most studies included areas of North America (N = 15) or Europe (N = 14 publications), while a few included areas in the Asia-Pacific region (N = 4) or had global scope (N = 3) (Table 1).
Twenty-eight studies employed temporal forecasting without a spatial component, while 7 made forecasts in time and space (Table 1). The studies used diverse forecasting methods, with 18 using statistical approaches without models for epidemiological processes and 17 employing epidemiological models (Table 1).
Among the statistical approaches, methods included time series models (N = 9 publications), generalized linear models (N = 4), Bayesian networks (N = 2), classification methods (N = 2), survival analysis (N = 1), and a prediction market (N = 1) (Table 1). The mechanistic approaches included compartmental models, which model transitions across various sub-populations (susceptible-infectious-removed [SIR] models and variants) (N = 12 publications); and agent-based models (ABMs), which model exposure, infection, transmission and behaviors for each individual in the population (N = 5) (Table 1).
Several studies coupled mechanistic models with methods to update parameter estimates and forecasts as new influenza-related surveillance data becomes available. Nsoesie et al. ,  developed a simulation-optimization algorithm for their ABM, which iteratively proposes estimates of key epidemiological parameters, uses those estimates to simulate the future course of the epidemic, and compares observed surveillance data to forecasts to revise the parameter estimates. Ong et al.  and Shaman et al. ,  used data assimilation techniques to incorporate influenza-related surveillance data into their compartmental models and update parameter estimates and forecasts.
The studies used dynamic virological (N = 14), syndromic influenza-like illness (ILI; N = 13) and other influenza-related surveillance data to forecast influenza activity (Table 2). Birrell et al.  included serological data to model pre-existing immunity, as well as virological and syndromic data. Four studies included internet search query data (Google Flu Trends) , , , . Six studies considered meteorological data , , , –, with 3 including the meteorological predictors in the final forecasting model , , .
While most studies reported various modeling outcomes, such as ILI time series, the specific outcomes used in model validation varied. Among the 27 population-based forecasting studies, 16 used weekly predictions of weekly incidence 1 or more weeks into the future in the validation (Table 3). Nine studies predicted the timing of the epidemic peak or incidence at the peak; all performed validation using at least some forecasts made at least 4 weeks before the actual peak –, –, , . The facility-based forecasting studies used 1-step-ahead – or n-step-ahead  predictions of visit counts over step sizes of 1 day  to 1 month . The regional or global pandemic spread forecasting studies used early data from the 2009 influenza A(H1N1)pdm09 pandemic to predict outcomes at national level across countries, including pandemic arrival, and peak incidence and time of peak.
The studies used various metrics for validation of forecasts against independent data, with mean (or median) absolute error and mean absolute percent error the most common metrics for forecasts of incidence (i.e., daily, weekly, or monthly incidence; peak incidence; or cumulative incidence) (Table 4; studies forecasting peak week or epidemic duration reported the time difference between predicted and observed values in the validation). Among all studies, only 2 reported accuracy as a function of estimated forecast variance , .
Comparing the accuracy of the forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely. While many studies compared models with different sets of predictors, only 2 compared distinct modeling approaches. Shaman et al.  compared their susceptible-infectious-recovered-susceptible (SIRS) compartmental model, coupled to an ensemble-adjusted Kalman filter (SIRS-EAKF), to various resampling approaches using previous influenza seasons. The SIRS-EAKF model was considerably more accurate in predicting ILI peak week for the 2012–2013 season across 108 US cities. Merler et al.  compared the performance of an ABM and a simpler compartmental model in predicting the course of the 2009 influenza A(H1N1)pdm09 pandemic in Europe, and found the simpler model failed to predict pandemic dynamics and attack rate accurately across countries.
Among the 17 studies that used epidemiological models, 8 provided results of sensitivity analysis for clinical, epidemiological, demographic, and other parameters , , , , , –; however, Nsoese et al. published a sensitivity analysis separately  for the ABM used in publications included in this review , , .
This review shows accelerating publication of influenza forecasting methods in recent years. We identified diverse modeling applications to forecast influenza and ILI activity in human populations, including various purely statistical approaches and methods based on mechanistic (i.e., epidemiological) modeling. Most models predicted influenza activity in a specific population, while several others predicted presentations at medical facilities or regional or global pandemic spread. Several models incorporated additional data besides clinical or laboratory-based surveillance data to generate forecasts, including internet search queries and meteorological data. The outcomes predicted and metrics used in validation varied. Most studies using mechanistic models did not present a sensitivity analysis for key epidemiological assumptions.
The review provides an overview and assessment of influenza forecasting, describing current approaches and highlighting research needs for this promising new domain of public health preparedness. Since the focus was on the use of models to forecast influenza activity, we included only studies that validated models against independent data, a crucial part of predictive model development since using the same data for model fitting and testing inflates estimates of predictive skill . This approach complements the review of Nsoesie et al. , which did not apply this restriction and provided a more detailed consideration of the outcomes predicted and advantages and disadvantages of the modeling methods employed.
The study has some limitations. We cannot exclude the possibility we failed to identify relevant studies, though we used broad search terms and searched multiple databases, and we would not have identified newer studies described only in conference proceedings, or unpublished studies. Papers correctly (based on our criteria) excluded may yet prove useful for influenza forecasting, and further review of these may suggest new methodologies for generating influenza predictions. The review also cannot serve as a definitive guide to forecasting approaches with greater predictive skill, since settings and methodologies varied widely and only 2 studies ,  compared distinct modeling approaches. We approached the review as a scoping, rather than a systematic, review because of this diversity. Also, the purpose was not to offer detailed critiques of modeling methodologies. Such an assessment would be useful, but we believe that a more broadly-based review of forecasting applications provides necessary context for this and other more focused assessments.
The results suggest several areas of practice and research to advance influenza forecasting in human populations (Figure 2). First, developers of influenza forecasting models and technologies should adhere to good practices in development, implementation, application, and description of epidemiological models. One possible guide , developed for veterinary epidemiology but applicable to studies in human populations, provides several recommendations that could facilitate comparison and implementation of influenza forecasting models and technologies. These include use of sensitivity analysis to assess dependence of the model to all chosen parameter values and assumptions, and provision of the computer code implementing the model (in the publication or on request).
However, in light of the methods identified in our review, some modification to these and related guidelines may be appropriate for influenza forecasting. For example, some approaches (e.g., , , , , ) optimize parameter values iteratively, as part of the forecasting algorithm; model developers do not explicitly assign parameter values or distributions. We encourage developers and users of epidemiological forecasting models to develop common, recommended practices for the field.
Second, there is a need for comparisons of diverse forecasting models using common input data and validation approaches and metrics. While some of the general advantages and disadvantages of various ILI forecasting approaches have been identified , direct comparisons would yield insight into methods that perform better than others under particular circumstances. Such initiatives are underway at the Centers for Disease Control and Prevention , Intelligence Advanced Research Projects Activity, and Department of Defense, and could help guide future efforts. Head-to-head comparisons of automated detection algorithms to identify disease outbreaks in syndromic surveillance data  also could be a useful example for comparing forecasting methods.
Third, assessments of forecasting methods should demonstrate how the accuracy of the method varies and should quantify this variability for use in real-time prediction. That is, it is not sufficient merely to predict an event; the likelihood of that prediction should also be ascribed. This quantification of likelihood, or expected accuracy, mirrors practices used in numerical weather prediction–e.g., a forecast of an 80% chance of rain tomorrow is a highly calibrated prediction of the likelihood of an event. We believe this aspect of model performance – calibration – will be a key consideration for practitioners who might use a forecasting model in an operational setting. Reporting the range or confidence intervals associated with predicted outcomes is essential in validation studies, but this alone does not help a user determine how much certainty a specific forecast warrants.
Fourth, future operational forecasting efforts should develop explicit approaches that incorporate additional expertise and analysis from scientists and public health officials. (The only documented systematic elicitation of expert judgment, for any type of modeling approach, in our review was the prediction market of Polgreen et al. .) Similar methods exist in weather and climate forecast (e.g., ). For example, meteorological forecasts are typically statistically post-processed to account for inherent model biases, and new methods for this post-processing are still being developed . These combined results are then further vetted by meteorologists to monitor anomalous prediction behavior, and communicated to the public and decision makers. Infectious disease forecasting will need to explore and develop analogous frameworks for the post-processing of multiple forecast streams, the monitoring and calibration of these probabilistic forecasts, and the communication of these predictions to public health officials for decision support.
Fifth, now that diverse ILI forecasting approaches are available and some have demonstrated promising performance in validation studies, assessments of real-world applications could spur the transition of these approaches to public health practice. Pilot studies in health departments, medical facilities, or other settings could assess forecasting applications not only for predictive skill, but for user acceptance, contributions to public health decision-making, and other outcomes at the user-model interface. Evaluations should compare various modeling approaches on these key characteristics, to identify approaches useful (not just accurate) in real-world settings. For example, forecasting ILI time series or peak week could be useful for anticipating needed surge capacity of personnel and materials, but modeling methods that permit re-estimation of outcomes under various response scenarios could provide additional support to decision-makers.
Last, model developers and decision-makers must understand each other's work better. Developers are more likely to provide useful tools if they know the key decisions users will make in preparing for or responding to influenza outbreaks. They can develop and evaluate models around those specific decisions. To apply forecasting models effectively, decision-makers should become familiar with the modeling tools they might use, and understand their strengths, limitations, and key assumptions. Efforts to link modelers and public health officials through seminars, on-the-job observation, exercises, and other activities could foster this mutual understanding and improve collaboration during emergencies.
Disclaimer: The views expressed are those of the authors, and do not necessarily represent those of the US Government.
We thank Dr. Kevin Russell, Director, Armed Forces Health Surveillance Center, for supporting this project.
Conceived and designed the experiments: JPC DG JS RAC FEM. Performed the experiments: JPC DG JS RAC FEM. Analyzed the data: JPC DG JS RAC FEM. Contributed reagents/materials/analysis tools: JPC DG JS RAC FEM. Wrote the paper: JPC DG JS RAC FEM.
- 1. World Health Organization website. Influenza fact sheet. Available: http://www.who.int/mediacentre/factsheets/fs211/en/index.html. Accessed 2014 Mar 19.
- 2. World Health Organization website. Influenza virus infections in humans. Available: http://www.who.int/influenza/GIP_InfluenzaVirusInfectionsHumans_Jul13.pdf. Accessed 2014 Mar 19.
- 3. Dorjee S, Poljak Z, Revie CW, Bridgland J, McNab B, et al. (2013) A review of simulation modelling approaches used for the spread of zoonotic influenza viruses in animal and human populations. Zoonoses Public Health 60: 383–411
- 4. Lee VJ, Lye DC, Wilder-Smith A (2009) Combination strategies for pandemic influenza response - a systematic review of mathematical modeling studies. BMC Med 7: 76
- 5. Prieto DM, Das TK, Savachkin AA, Uribe A, Izurieta R, et al. (2012) A systematic review to identify areas of enhancements of pandemic simulation models for operational use at provincial and local levels. BMC Public Health 12: 251
- 6. Coburn BJ, Wagner BG, Blower S (2009) Modeling influenza epidemics and pandemics: insights into the future of swine flu (H1N1). BMC Med 7: 30
- 7. Nsoesie EO, Brownstein JS, Ramakrishnan N, Marathe MV (2013) A systematic review of studies on forecasting the dynamics of influenza outbreaks. Influenza Other Respir Viruses. doi:10.1111/irv.12226.
- 8. Moher D, Liberati A, Tetzlaff J, Altman DG (2009) PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097
- 9. Arksey H, O'Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8: 19–32.
- 10. Shaman J, Karspeck A, Yang W, Tamerius J, Lipsitch M (2013) Real-time influenza forecasts during the 2012–2013 season. Nat Commun 4..
- 11. Nsoesie E, Mararthe M, Brownstein J (2013) Forecasting peaks of seasonal influenza epidemics. PLoS Curr 5 . doi:10.1371/currents.outbreaks.bb1e879a23137022ea79a8c508b030bc.
- 12. Hyder A, Buckeridge DL, Leung B (2013) Predictive validation of an influenza spread model. PloS One 8: e65459
- 13. Nsoesie EO, Beckman RJ, Shashaani S, Nagaraj KS, Marathe MV (2013) A Simulation Optimization Approach to Epidemic Forecasting. PloS One 8: e67164
- 14. Roberts MG (2013) Epidemic models with uncertainty in the reproduction number. J Math Biol 66: 1463–1474
- 15. Held L, Paul M (2012) Modeling seasonality in space-time infectious disease surveillance data. Biom J Biom Z 54: 824–843
- 16. Shaman J, Karspeck A (2012) Forecasting seasonal outbreaks of influenza. Proc Natl Acad Sci U S A 109: 20425–20430
- 17. Socan M, Erculj V, Lajovic J (2012) Early detection of influenza like illness through medication sales. Cent Eur J Public Health 20: 156–162.
- 18. Ajelli M, Merler S, Pugliese A, Rizzo C (2011) Model predictions and evaluation of possible control strategies for the 2009 A/H1N1v influenza pandemic in Italy. Epidemiol Infect 139: 68–79
- 19. Birrell PJ, Ketsetzis G, Gay NJ, Cooper BS, Presanis AM, et al. (2011) Bayesian modeling to unmask and predict influenza A/H1N1pdm dynamics in London. Proc Natl Acad Sci U S A 108: 18238–18243
- 20. Goldstein E, Cobey S, Takahashi S, Miller JC, Lipsitch M (2011) Predicting the epidemic sizes of influenza A/H1N1, A/H3N2, and B: a statistical method. PLoS Med 8: e1001051
- 21. Nishiura H (2011) Real-time forecasting of an epidemic using a discrete time stochastic model: a case study of pandemic influenza (H1N1-2009). Biomed Eng Online 10: 15
- 22. Nsoesie EO, Beckman R, Marathe M, Lewis B (2011) Prediction of an Epidemic Curve: A Supervised Classification Approach. Stat Commun Infect Dis 3. doi:10.2202/1948-4690.1038.
- 23. Ohkusa Y, Sugawara T, Taniguchi K, Okabe N (2011) Real-time estimation and prediction for pandemic A/H1N1(2009) in Japan. J Infect Chemother 17: 468–472
- 24. Paul M, Held L (2011) Predictive assessment of a non-linear random effects model for multivariate time series of infectious disease counts. Stat Med 30: 1118–1136
- 25. Ong JBS, Chen MI-C, Cook AR, Lee HC, Lee VJ, et al. (2010) Real-time epidemic monitoring and forecasting of H1N1-2009 using influenza-like illness from general practice and family doctor clinics in Singapore. PloS One 5: e10036
- 26. Soebiyanto RP, Adimi F, Kiang RK (2010) Modeling and predicting seasonal influenza transmission in warm regions using climatological parameters. PloS One 5: e9450
- 27. Jiang X, Wallstrom G, Cooper GF, Wagner MM (2009) Bayesian prediction of an epidemic curve. J Biomed Inform 42: 90–99
- 28. Rhodes CJ, Hollingsworth TD (2009) Variational data assimilation with epidemic models. J Theor Biol 258: 591–602
- 29. Andersson E, Kühlmann-Berenzon S, Linde A, Schiöler L, Rubinova S, et al. (2008) Predictions by early indicators of the time and height of the peaks of yearly influenza outbreaks in Sweden. Scand J Public Health 36: 475–482
- 30. Saltyte Benth J, Hofoss D (2008) Modelling and prediction of weekly incidence of influenza A specimens in England and Wales. Epidemiol Infect 136: 1658–1666
- 31. Hall IM, Gani R, Hughes HE, Leach S (2007) Real-time epidemic forecasting for pandemic influenza. Epidemiol Infect 135: 372–385
- 32. Polgreen PM, Nelson FD, Neumann GR (2007) Use of prediction markets to forecast infectious disease activity. Clin Infect Dis 44: 272–279
- 33. Sebastiani P, Mandl KD, Szolovits P, Kohane IS, Ramoni MF (2006) A Bayesian dynamic model for influenza surveillance. Stat Med 25: 1803–1816
- 34. Saramäki J, Kaski K (2005) Modelling development of epidemics with dynamic small-world networks. J Theor Biol 234: 413–421
- 35. Viboud C, Boëlle P-Y, Carrat F, Valleron A-J, Flahault A (2003) Prediction of the spread of influenza epidemics by the method of analogues. Am J Epidemiol 158: 996–1006.
- 36. Mooney JD, Holmes E, Christie P (2002) Real-time modelling of influenza outbreaks–a linear regression analysis. Euro Surveill Bull 7: 184–187.
- 37. Dugas AF, Jalalpour M, Gel Y, Levin S, Torcaso F, et al. (2013) Influenza forecasting with Google Flu Trends. PloS One 8: e56176
- 38. Spaeder MC, Stroud JR, Song X (2012) Time-series model to predict impact of H1N1 influenza on a children's hospital. Epidemiol Infect 140: 798–802
- 39. Spaeder MC, Fackler JC (2011) Time series model to predict burden of viral respiratory illness on a pediatric intensive care unit. Med Decis Mak 31: 494–499
- 40. Moriña D, Puig P, Ríos J, Vilella A, Trilla A (2011) A statistical model for hospital admissions caused by seasonal diseases. Stat Med 30: 3125–3136
- 41. Hwang GM, Mahoney PJ, James JH, Lin GC, Berro AD, et al. (2012) A model-based tool to predict the propagation of infectious disease via airports. Travel Med Infect Dis 10: 32–42
- 42. Tizzoni M, Bajardi P, Poletto C, Ramasco JJ, Balcan D, et al. (2012) Real-time numerical forecast of global epidemic spreading: case study of 2009 A/H1N1pdm. BMC Med 10: 165
- 43. Merler S, Ajelli M, Pugliese A, Ferguson NM (2011) Determinants of the spatiotemporal dynamics of the 2009 H1N1 pandemic in Europe: implications for real-time modelling. PLoS Comput Biol 7: e1002205
- 44. Hosseini P, Sokolow SH, Vandegrift KJ, Kilpatrick AM, Daszak P (2010) Predictive power of air travel and socio-economic data for early pandemic spread. PloS One 5: e12763
- 45. Nsoesie EO, Beckman RJ, Marathe MV (2012) Sensitivity analysis of an individual-based model for simulation of influenza epidemics. PloS One 7: e45414
- 46. Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction. 2nd ed. New York: Springer-Verlag. 763 p.
- 47. Woolhouse M, Fevre E, Handel I, Heller J, Parkin T, et al.. (2011) A guide to good practice for quantitative veterinary epidemiology. Available: http://www.qve-goodpracticeguide.org.uk/guide. Accessed 2014 Mar 19.
- 48. Centers for Disease Control and Prevention website. Predict the influenza season challenge. Available: http://www.cdc.gov/flu/news/predict-flu-challenge.htm. Accessed 2014 Mar 19.
- 49. Siegrist D, Pavlin J (2004) Bio-ALIRT biosurveillance detection algorithm evaluation. MMWR Morb Mortal Wkly Rep 53 Suppl: 152–158.
- 50. Palmer TN, Doblas-Reyes FJ, Hagedorn R, Alessandri A, Gualdi S, et al. (2004) Development of a European multimodel ensemble system for seasonal-to-interannual prediction (DEMETER). Bull Am Meteorol Soc 85: 853–872
- 51. McLean Sloughter J, Gneiting T, Raftery AE (2013) Probabilistic Wind Vector Forecasting Using Ensembles and Bayesian Model Averaging. Mon Weather Rev 141: 2107–2119