Skip to main content
Advertisement
  • Loading metrics

Early warning systems (EWSs) for chikungunya, dengue, malaria, yellow fever, and Zika outbreaks: What is the evidence? A scoping review

Abstract

Background

Early warning systems (EWSs) are of increasing importance in the context of outbreak-prone diseases such as chikungunya, dengue, malaria, yellow fever, and Zika. A scoping review has been undertaken for all 5 diseases to summarize existing evidence of EWS tools in terms of their structural and statistical designs, feasibility of integration and implementation into national surveillance programs, and the users’ perspective of their applications.

Methods

Data were extracted from Cochrane Database of Systematic Reviews (CDSR), Google Scholar, Latin American and Caribbean Health Sciences Literature (LILACS), PubMed, Web of Science, and WHO Library Database (WHOLIS) databases until August 2019. Included were studies reporting on (a) experiences with existing EWS, including implemented tools; and (b) the development or implementation of EWS in a particular setting. No restrictions were applied regarding year of publication, language or geographical area.

Findings

Through the first screening, 11,710 documents for dengue, 2,757 for Zika, 2,706 for chikungunya, 24,611 for malaria, and 4,963 for yellow fever were identified. After applying the selection criteria, a total of 37 studies were included in this review. Key findings were the following: (1) a large number of studies showed the quality performance of their prediction models but except for dengue outbreaks, only few presented statistical prediction validity of EWS; (2) while entomological, epidemiological, and social media alarm indicators are potentially useful for outbreak warning, almost all studies focus primarily or exclusively on meteorological indicators, which tends to limit the prediction capacity; (3) no assessment of the integration of the EWS into a routine surveillance system could be found, and only few studies addressed the users’ perspective of the tool; (4) almost all EWS tools require highly skilled users with advanced statistics; and (5) spatial prediction remains a limitation with no tool currently able to map high transmission areas at small spatial level.

Conclusions

In view of the escalating infectious diseases as global threats, gaps and challenges are significantly present within the EWS applications. While some advanced EWS showed high prediction abilities, the scarcity of tool assessments in terms of integration into existing national surveillance systems as well as of the feasibility of transforming model outputs into local vector control or action plans tends to limit in most cases the support of countries in controlling disease outbreaks.

Introduction

Epidemics of arboviral diseases transmitted by Aedes mosquitoes—such as chikungunya, dengue, yellow fever, and Zika—have emerged or reemerged over the past 5 decades, overburdening already stretched health systems. Approximately 2.5 billion people live in risk areas of Aedes-borne diseases, and, collectively, an estimated 390 million infections occur annually in about 100 countries [14]. Unfortunately, risk forecasts indicate that these epidemics will intensify and reach new geographical areas throughout the 21st century [5]. This fact is largely driven by a combination of urbanization, poor living conditions, international travel and trade, changes in mosquito distribution and abundance, climate variability, and climate change [68].

Also, malaria, an Anopheles mosquito–transmitted disease in tropical and subtropical areas, has often shown its potential for large outbreaks. This may happen in the highly endemic areas of sub-Saharan Africa but also in areas of malaria elimination in Asia and Latin America where the fading herd immunity makes people more susceptible for infections and allows local outbreaks to occur [9,10].

Vector-borne diseases can mainly be controlled through effective vector control. Even after the advent of a vaccine, as available for yellow fever, vector management will continue to be important. Only for malaria, a number of therapeutic treatment options are available [11]. Due to the high vector capacity of Aedes mosquitoes, the required level of vector control interventions to prevent transmission is usually not being achieved, and outbreaks have become increasingly frequent [11]. Data are usually provided by the routine disease surveillance systems occasionally complemented by entomological data. The information often arrives too late or is of low quality or in the wrong format [12,13]. As a result, outbreaks are usually detected too late when infections have already spread.

Forecasting disease outbreaks is highly desirable to give time to the vector control services for preparing the response. For outbreak early warning systems (EWSs), countries need standard operational procedures (SOPs) to identify consistently through alarm signals an increased outbreak risk in time and space triggering an early response. Several EWSs have been developed for our target diseases, and most of these have commonalities in their structural design, functions, and analytical approach [1417]. However, studies addressing the effectiveness of space and time prediction and how the EWS may improve coordination among the operators at national and district level are scarce. This scoping review summarizes and discusses the evidence of different EWSs, their performance, and abilities to predict outbreaks of our target diseases, providing an overview of the state of the art and recommendations for the future development of a practical outbreak prediction tool. This review will recapitulate (1) EWS prediction validity in terms of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of outbreak warning; (2) if studies addressed the operational feasibility of the EWS in terms of user-friendliness, cost, capacity of being easily implemented within countries’ national control programs, coordination, and the requirements for increasing its efficiency; and (3) the scopes of study designs for assessing the EWS effectiveness in triggering adequate response activities.

Methods

Methods used were predefined in a protocol based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [18].

Search strategy, databases, and search terms

Literature search and analysis was carried out until August 5, 2019. Data were extracted from the following databases: (1) Cochrane Infectious Diseases Group (CIDG) Specialised Register and Cochrane Central Register of Controlled Trials (CENTRAL); (2) Google Scholar; (3) Latin American and Caribbean Health Sciences Literature (LILACS); (4) PubMed; (5) Web of Science; and (6) WHO Library Database (WHOLIS). The reference lists of all included papers were screened for further relevant studies.

The inclusion criteria of articles were (1) primary research published in a peer-reviewed journal; (2) studies addressing any type of existing or developing prediction model; and (3) dealing with chikungunya, dengue, malaria, yellow fever, and Zika diseases or a combination of the diseases; presenting (a) experiences with existing EWS (stand-alone or integrated into the national surveillance systems); and (b) the development or implementation of EWS in a particular setting. EWS studies were excluded if merely investigating trends or correlations of particular alarm indicators. Studies were also excluded if they neither reported nor provided sufficient data to outline the type of mathematical model used or the temporal and spatial prediction of the tool. Further studies were excluded if they failed to demonstrate a developed or prototype of prediction model—i.e., merely presented and discussed candidate list of potential alarm indicators or studies with models constructed to elucidate transmission dynamics. No exclusion due to study design was made, except for conference abstracts, book chapters, or studies that are published by journals of a local institute.

Searches were conducted in English, assuming that most relevant studies are indexed in English, with title and abstract in English. No restrictions were applied for year of publication, geographical area, or language.

The terms “early warning,” “forecasting,” and “prediction” have been probed in a pilot search as Medical Subject Headings (MeSH) terms and as free-text terms. The terms “forecasting” and “prediction” have been discarded thereafter for the final search, as they did not show EWS in a public health sense predicting epidemics, but mostly clinical related studies predicting outcome of clinical disease.

The terms “chikungunya,” “dengue,” “malaria,” “yellow fever,” and “Zika” were used as MeSH terms and as free-text terms, depending on the function of the database. The searches were run for the combination of “early warning” AND the disease.

PubMed has been searched using MeSH terms for the diseases and free-text term for “early warning.” For Google Scholar, LILACS, and WHOLIS, free-text terms for both categories have been use. For CIDG Specialised Register and CENTRAL, the disease and “early warning” were searched as MeSH terms.

Quality assessment and assurance

Searches were performed by the team of authors and double-checked until consistent results were found. One author (LH or EG) screened all hits for their relevance. As for Google Scholar, a large number of hits were derived (>10,000 for dengue), hits were sorted by relevance, and the first 120 hits were established as a suitable number, with no additional relevant hits encountered toward the end of the search.

Two data extractors (LH and SRR for chikungunya, dengue, yellow fever, and Zika; LH and EG for malaria) independently assessed the abstracts and full text of the preselected references for potential eligibility by applying all inclusion and all exclusion criteria.

After full agreement, data were entered into a predefined Excel data extraction form by author, title, journal, publication date, and study design as well as different outcome parameters as they emerged from the studies, following the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist [19]. Extracted parameters are listed below in detail.

Risk of bias of the included studies has not been assessed, as no systematic but a scoping review was performed, but the quality was assured by strict application of inclusion and exclusion criteria, considering several quality aspects as described above.

Data extraction and analysis

The above described data extraction form was developed based on the study objectives and relevant predefined quantitative and qualitative outcome variables. Further categories were also considered including (1) studies addressing surveillance as structured form of information on disease cases, meteorological, epidemiological, and entomological information (indicator-based surveillance, IBS); and 2) studies on surveillance of unofficial or unstructured information (such as social media and community reports) or of social events (event-based surveillance, EBS). EWS using epidemiological, entomological, or climate data for outbreak prediction were termed “alarm-informed EWS.” EWS based on increased case numbers in comparison with historical data were termed “case-informed EWS.” Further characteristics of the studies taken into consideration were (a) methodological (case study, time series, and randomized control trials (RCT), etc.); (b) the models/tools that have been investigated (and their study scope); and (c) the disease(s) being investigated. Detailed information was extracted on specific outcomes as sensitivity, specificity, PPV/NPV outputs of the models, local coverage of the tool (central, district/province, or subdistrict level), integration (into national surveillance program), and the level of implementation (central, district, or subdistrict levels). Furthermore, the end user of the tool (e.g., public health officers, institutes, researchers, and others), the user-friendliness of the EWS tool, its temporal and spatial risk prediction and its integration feasibility including, resource needed to implement, prediction duration or timelag and, the duration of dataset were all captured in this review. The data extraction form has been further synthesized to evidence tables (Tables 13).

thumbnail
Table 2. Characteristics of EWARS systems under investigation.

https://doi.org/10.1371/journal.pntd.0009686.t002

thumbnail
Table 3. Main findings, conclusions, and limitations of the studies.

https://doi.org/10.1371/journal.pntd.0009686.t003

Evaluation criteria

Prediction models are typically evaluated according to their statistical and operational performance. Statistically, the sensitivity, specificity, PPV, or NPV measures are used to quantify the quality of prediction performance of the model. Based on common recommendations of appropriate cutoff for model performance, we attributed a statistical measure of >50% to high performance prediction models [20]. The spatial prediction is crucial in the context of vector control and action plan; spatial prediction alone is less useful and must be combined with the temporal model for effective public health response. Accordingly, we defined high temporal prediction by a prediction window of 1 to 12 weeks (allowing sufficient time between prediction and action taken). We defined high spatial prediction by the prediction of hotspots within geographically defined areas of relatively small spatial scale such as villages, neighborhood, or household levels in contrast to a resolution within district scale.

Results

Study characteristics

A total of 46,477 hits were obtained, with 11,710 studies focused on dengue, 2,757 on Zika, 2,706 on chikungunya, 24,611 on malaria, and 4,693 on yellow fever. After review by title and abstract as well as the application of a cutoff after 120 Google Scholar hits, sorted by relevance, 279 dengue, 22 chikungunya, 39 Zika, 89 malaria, and 13 yellow fever articles were further assessed. Based on the inclusion criteria and removing of duplicates, 151 articles were assessed as full texts, but exclusion criteria finally retained 37 articles including 28 for dengue, 2 for Zika, and 7 for malaria. Fig 1 summarizes the search process.

thumbnail
Fig 1. Screening and selection of articles for a scoping review on EWS.

CDSR, Cochrane Database of Systematic Reviews; CENTRAL, Cochrane Central Register of Controlled Trials; CIDG, Cochrane Infectious Diseases Group; EWS, early warning system; LILACS, Latin American and Caribbean Health Sciences Literature.

https://doi.org/10.1371/journal.pntd.0009686.g001

All of the 37 studies were published between 2001 and 2019, of which 33 between 2011 and 2018. Of the 28 dengue studies, 3 [12,21,22] were conducted across different countries (multicounty) and 16 in Asia (5 in China, 4 in Indonesia, 3 in Singapore, and 1 each in Cambodia, Sri Lanka, Taiwan, and Thailand). Another 8 studies were conducted in the Americas (4 in Brazil, 2 in Colombia, 1 in Cuba, and 1 in French Guiana) and 1 [23] in Europe. The 2 Zika studies were performed in the Americas [24,25].

From the 7 malaria studies that were retrieved, 1 was conducted in Colombia and 6 in Africa (2 in Ethiopia and Kenya, 1 in Tanzania, and 1 in Uganda) and 1 in the Solomon Islands in the Pacific. For details of the papers, see Fig 2 and S1 Text (country distribution of included studies).

Of the 28 studies addressing EWS for dengue outbreaks, only 4 studies presented the user perspective on implementing EWS within national programs [12,21,26,27], and 24 studies showed EWS under development (i.e., model exercise); see Table 1. Only 2 dengue studies used a prospective study design [28,29], and one applied both prospective and retrospective approaches [30]; the other 23 were retrospective analyses of surveillance data (1 of those specified as cohort study [21], 1 with a cross-sectional design [31], and 1 as a time series analysis [30]), and 2 studies were exclusively relying on mathematical simulation models [32,33]. The 2 Zika studies reported on exercises of testing EWS [24,25]. The 7 EWS malaria studies included only one existing EWS [34], while 6 other studies reported on potential EWSs or under development. The studies used a large variety of statistical methods as summarized in Table 1.

EWS features per disease

Outbreak indicators of EWSs varied considerably across the selected studies, and, hence, we stratified the findings of the relevant outcomes by the 3 diseases (no studies were retrieved for chikungunya and yellow fewer) that were ultimately retained from the search. In line with the respective case definition used in the diverse reporting systems across national surveillance programs and between diseases, the outbreak indicators were grouped as hospitalized, laboratory-confirmed (including “reported,” “confirmed,” or “imported from neighboring districts”), and suspected cases (including “probable” cases as they were often not distinguished from “suspected”) as well as case numbers resulting from mathematical modeling (no real cases but estimates based on other covariates). The alarm indicators were categorized as entomological (including an index of mosquito survival), meteorological (including the El Niño Southern Oscillation [ENSO] index and others), epidemiological (including sociodemographic information), and social media (such as “Google Trends”).

Study results

Dengue

EWS characteristics and features.

All 28 dengue studies relied on IBS, of which 24 studies were alarm informed, i.e., of outbreak predictive nature—using at least 1 meteorological, entomological or epidemiological indicator—and 4 studies [28,31,32,35] were case-informed EWS, i.e., of early outbreak detection—relying solely on previous trends of cases for outbreaks. Almost all studies demanded routine access to data as well as advanced statistical and analytical skills, but 5 studies [12,21,26,28,31] reported prediction tools that are adapted for less skilled users. The outbreak indicators used were hospitalized cases (2 studies) [12,21], laboratory-confirmed cases (23 studies), or suspected cases (3 studies) [12,28,31]. One study [30] used mathematically simulated cases. All cases were reported by the surveillance system on either weekly or monthly basis, and, occasionally, data were obtained directly from the Ministry of Health (MOH).

The dengue studies applied a spectrum of alarm indicators using meteorological (24 studies), entomological (7 studies) [12,21,26,29,32,36,37], epidemiological (5 studies) [12,21,26,36,38], and one study [36] used a cartographic indicator—these are nonclimatic indicators of altitude and aggregated census data related to urban setting.

With reference to the alarm indicators, 13 studies reported the use of data from local/regional meteorological stations, whereas 3 studies referred to international sources such as the “World Meteorological Organization” [39], the “Global Historical Climate Network” [40], and the “National Oceanic and Atmospheric Administration” [13]. Furthermore, one study [23] used data from the regional Environment and Epidemiology Network (Table 2).

Summary findings and reported limitations.

This section presents key findings extracted from EWS studies (see Table 3) following the evaluation criteria presented in the methodology and their reported limitations. Most of the studies presented sensitivity, specificity, and NPV/PPV as measurements of the validity of the corresponding models. A range between 40% and 100% for sensitivity and 12.5% to 100% for the PPV have been reported, depending primarily on the type of model and the alarm indicators used. Generally, meteorological indicators were the best performing predictors, but a combination of meteorological and other indicators outperformed the single indicator prediction—i.e., the prediction model improved when other nonclimatic alarm indicators were included to the model containing climate indicators. While several EWSs demonstrated reasonable predictive abilities, 5 models showed outstanding performance using the following alarm indicators: (1) the dynamic risk maps absolute shrinkage and selection operator (LASSO) processing multiple meteorological information (with 1 study adding on epidemiological and entomological alarm indicators) [26]; (2) autoregression integrated moving average (ARIMA) using meteorological information and Google Trends data [40]; (3) Shewhart moving average regression model (SMAR) maintaining a combination of meteorological, epidemiological, and entomological alarm indicators [12,21]; (4) seasonal autoregressive integrated moving average (SARIMA) [35]; and (5) the stochastic Bayesian maximum entropy (BME) model using a mix of meteorological and entomological alarm indicators [37]. A total of 20 EWS studies demonstrated high temporal prediction ability, but only 3 [27,28,38] showed high spatial prediction ability. Several studies reported data- or model-related limitations, with 10 studies indicating issues related to the outbreak information (such as poor or unreliable case reporting, small historical records, bias created by interventions or other confounders, and lack of geotagged weekly records). Three studies [22,41,42] referred to the lack of climate information and biased, inaccessible, or poor resolution data. Other limitations addressed the methodological approach with 3 studies [26,31,35] declaring method-related issues such as that methods (like LASSO) are not amenable to direct interpretation, requiring advanced statistical skills, or have a limited prediction robustness.

Concerning the EWS implementation, the following information has been provided: All studies described the EWS prediction coverage for national and district levels, but only 2 studies [27,38] showed predictions for the subdistrict level. Furthermore, half of the studies (14) featured the possibility of successfully implementing EWS, at national or district levels, but only 5 studies [21,26,28,36,43] showed the feasibility to be integrated into existing national surveillance programs. The users’ perception of EWS was insufficiently assessed in several studies, but 18 indicated the possibility to be used by health managers at the MOH and district levels; 5 studies [32,39,40,44,45] reported the use of EWS being limited to research institutes. The user-friendliness of EWS was reported in only 2 studies [21,31], showing a general satisfaction by users.

Zika

EWS characteristics and features.

Studies on Zika were primarily triggered by the 2015 to 2016 Zika virus (ZIKV) epidemic. Thus, only 2 studies were retrieved, which discussed the EWS for Zika outbreaks that matched our selection criteria. Both studies, of which one was an IBS-based and the second was an EBS-based study, applied new algorithms using alarm-informed EWS. Both studies described the need for routinely accessing alarm information for running the tools. One study used both confirmed and suspected cases for early outbreak warning [24], while the other study referred only to suspected cases that reflects the diagnostic and reporting complexity associated with this disease. One study employed social media information (Google Trends search) [24], and another used meteorological information as predictors [25]; however, both studies used the national surveillance programs as sources for processing their data.

Summary findings and reported limitations.

There was no validity testing presented by both studies, such as sensitivity or specificity. In one study [24], ARIMA was reported as a model to improve the prediction of Zika outbreaks when integrated with Google Trends data. The other study [25] showed the usefulness of humidity, rainfall, and maximum air temperature as alarm indicators for outbreak prediction. As concluded by Teng and colleagues, using dynamic data from Google Trends as indicators can significantly advance the prediction model mainly when more sophisticated statistical models like ARIMA are used [24]. Nevertheless, meteorological alarm indicators have generated high temporal predictions with up to 20 weeks ahead of an outbreak, but both studies failed the spatial prediction. No major limitations were reported, but one study showed a model-related limitation due to the complexity in handling temporal functions in relation to the spatial functions of the model [25].

Malaria

EWS characteristics and features.

Out of the 7 EWS malaria studies, 6 used IBS and 1 [46] used an EBS—processing at least 1 meteorological, entomological, or epidemiological indicator. All studies demanded routine access to data, and all but 2 required advanced statistical and analytical skills for practicing or integrating the tool. The outbreak indicators used were hospitalized (1 study) [47] or laboratory-confirmed cases (3 studies) [34,48,49], while 2 studies [46,50] used clinically confirmed cases (hospitalized). Data collection was done on a weekly or monthly basis, and studies demonstrated a spectrum of indicators: All used meteorological and epidemiological information, some entomological (3 studies) [47,48,51], and one study demonstrated the cartographic indicator. None of the studies assessed the user experience with EWS, but one [34] described how the involvement of stakeholders improved the support and institutionalization of the tool; 3 studies [34,46,48] stated that the tool integration into existing epidemiological and climatic data hub created a better environment for further development and use of the tool. All studies used meteorological information as predictors. Surveillance data were obtained from local, regional, or national databases, and meteorological information was obtained from local stations or satellites.

Summary findings and reported limitations.

Sensitivity and specificity were reported to be high in one study [48], mentioned to be high but not measured in 2 studies [47,49] and varied according to mosquito survival probability and temperature in a fourth study [51]. Several mathematical models were used to create predictions, with the additive and multiplicative models being the ones with high sensitivity and specificity. Several meteorological indicators were used as predictors of which temperature and rainfall were the most frequently used indicators. In general, environmental factors and climate variability correlated with malaria incidence, but no outbreak prediction model has been developed that includes different types of epidemiological, environmental, and meteorological alarm indicators. The alarm indicators have managed to generate a high temporal prediction with up to 6 months ahead of outbreak, particularly if unusual weather conditions like ENSO were concerned. Several limitations were noted: Some models were tested for specific settings (e.g., highlands in Africa), others include only remotely sensed environmental indicators—remote sensing is a useful approach in the context of climate predictions, including flood and earthquake disaster prediction, but tends to lack sufficient evidence for early warning of infectious diseases in general. Thus, when used solely in an early warning, they are viewed as a limitation in the model. Confounding factors that affect malaria risk such as land use/land cover, population mobility, local hydrology, socioeconomic factors, and public health interventions were not captured.

Discussion

When searching for high-level evidence in the area under investigation, we found 2 systematic reviews [52,53] reporting on predictive modeling tools particularly for dengue. However, we found no high-level evidence on tools applications outside their mathematical modeling or on potential alarm indicators to be used, although these were the prime focus of both reviews. The study by Racloz and colleagues has successfully summarized the benefits of combining various epidemiological tools focusing on the ability to incorporate climatic, environmental, epidemiological, and socioeconomic factors to create an EWS and has outlined optimal prediction models [52]. The second study by Louis and colleagues addressed the risk mapping–related issues and human mobility as promising alarm indicators and maintained a thorough review of their limitations [53]. Besides the structural and statistical features of the tool, our review has additionally addressed essential operational aspects (prediction quality, the implementation, integration, and user perspectives of EWS) and public health implications of EWS, independently for each retrieved disease, keeping in mind that for any outbreak prediction, a reliable surveillance system is essential. For instance, misclassifications, data arriving late, missing values, and human errors during the data entry may compromise the EWS.

Dengue

All 28 identified dengue studies maintained an IBS approach, and these demanded routine and timely access to surveillance information potentially impacting the effectiveness of early warning tools. Published studies show that consistent and frequent case reporting has stronger predictive capacity, which usually tends to be delayed by monthly reporting schedules [54]. Our results show that weekly reporting of surveillance data correlate with increased sensitivity and PPV; however, only one-third of all dengue studies presented weekly data, negatively impacting the effectiveness and potentially cost-effectiveness of the early warning process [12,21,2628,31,41,42,44].

A total of 27 dengue studies demonstrated outbreak forecasting, using multiple (9/27) and single alarm (18/27) indicators mostly meteorological ones. However, access to meteorological information on a regular time basis is challenging in several settings [55]. Three studies included regional or international data resources, which, however, demand highly skilled users and advanced digital systems for international data processing [23,33,56].

Furthermore, the mathematical model was identified to affect the predictive ability of EWS due to (1) the choice of the analytical approach employed; and (2) the range of lag times between independent variables and epidemic dengue transmission as demonstrated by Racloz and colleagues and others [52]. Among the retrieved dengue studies, almost all reported estimates on sensitivity, specificity, NPV, or PPV but their prediction performances varied substantially depending on the statistical model used and the data quality. Out of the 19 studies with reported high temporal prediction quality (Table 3), the Bayesian algorithms and generalized linear models were the most prevalent [29,31,36,39,45,57,58]. Additionally, 4 studies used LASSO [22,26,27,45], 3 with ARIMA and time series analysis [28,40,56] independently, and 2 used the Shewhart/endemic channel method [12,21]. Only 4 studies showed evidence of adaptation to less skilled users, which can be significant for public health use [12,21,26,27].

The interaction between changing climate and increasing human mobility as drivers for emerging diseases warrants novel frameworks for assessing the linkage between disease transmission, climate change, and public health intervention in order to reach effective EWS. The use of data mining techniques, such as social media or travel information, in combination with surveillance data has emerged as an alternative source of real-time high-resolution geospatial data on a large scale [59]. However, our review showed minimal evidence of studies exercising such potential alarm indicators, which limits their contribution to outbreak preparedness and response planning [59,60]. The LASSO model is one typical example of such a tool that can potentially contribute to this concept. However, LASSO or similar model concepts demand high-quality big data. As these resources are (a) typically scarce in many countries; and (b) having a tendency to complicate the interpretation of the prediction outputs [26,45], the application in data-constrained settings and by unskilled users is likely to be limited. While limited data accessibility and poor quality have been described by several studies [32,35,41,42,56], published reports highlight the benefits of combining temporal data for analysis of the temporal kinetics and spatial data for the identification of high risk areas [61]. Only 3 retrieved studies [27,28,38] demonstrated high spatial prediction abilities, all attributed to settings with advanced data and surveillance systems. Two of those showed high temporal and spatial prediction, allowing for identification of population at risk at smaller spatial units, which can significantly contribute to targeted vector control [27,38].

Zika

Since the Zika emergence in 2015, the number of PubMed references for ZIKV has risen from 181 to 516 in 2019, with a high proportion focusing on the consequences of ZIKV infection during pregnancy [62]. Only 2 studies have been identified in this review, with a focus on EWS for Zika outbreaks in the Americas. One study had explored the use of Google Trends data as a predictor [24], and the other used a set of meteorological information for generating outbreak predictions [25]. The use of “suspected cases” as outcome variables in both retrieved studies could possibly be explained by the complexity of the Zika diagnosis and the large number of mild cases [62]. For dengue, the prediction accuracy of EWS was superior when using hospitalized or laboratory-confirmed cases compared to suspected cases [12,21]. However, no measurements of sensitivity or PPV were used in both Zika studies, albeit both ARIMA and generalized linear models predicted up to 20 weeks ahead of the outbreak. Limitations of zika outbreak predictions are not discussed in the respective papers, but are likely to be similar as those shown for dengue predictions.

Malaria

Since the inception of Roll Back Malaria (RBM) in 1998, it was clear that the early detection, containment, and prevention of malaria epidemics were key elements of the Global Malaria Control Strategy and Malaria Early Warning Systems (MEWS) [63,64]. Some countries have developed epidemic risk monitoring using simple transmission risk indicators such as excess rainfall [65], but only Kenya has published the development and implementation of MEWS [34]. Since then, several studies and initiatives for EWS have been developed; however, the literature to address the tools’ scope of its predictability, implementability, and users’ perspectives are significantly scarce. The 7 studies identified in this review used different mathematical models and different combinations of indicators. Most of the mathematical models found in this review applied time series including additive and multiplicative models [34,4648], but lacked vulnerability indicators such as low immunity or drug resistance, which might be prevalent in these study settings. Meteorological indicators ranged from common indicators like rainfall and humidity to land surface temperature or vegetation indices obtained through more sophisticated satellite systems that are usually provided by projects or partnerships, of which their sustainability was never assessed. Like the case with dengue, this wide range of indicators is augmented by additional data sources sought from regional or international entities. The development of implementable and user-friendly malaria EWSs, as shown in the reviews by Githeko and colleagues and Merkord and colleagues [34,46], is a key factor for better disease preparedness and timely response activities.

Public health implications for EWS applications

The EWS tool is primarily aimed at supporting district health managers and national health planners to mitigate or prevent disease outbreaks, ideally using tools that are integrated in the national surveillance programs [66,67]. To further ensure effective functions, EWS should conceptually be perceived as an information system designed to support the decision-making of national- and local-level institutions but also enable vulnerable groups in the society to take actions to mitigate the impacts of an impending risk. As apparent from this review, users of current EWSs were mostly from the central (MOH) levels, with only few tools facilitating district-level applications. The integration of EWS into existing national surveillance program was marginal with only 5 studies [12,21,26,27,34], out of 37, demonstrating their experience of integration (Table 1). Furthermore, the majority of studies have not assessed the feasibility of implementing EWS into national programs, and a few studies have declared the need for high-skilled users and resources—2 limiting factors that are unlikely to exist at small spatial levels in settings where disease outbreaks are public health burdens. Nevertheless, there is an observed trend toward applications of more advanced statistical models with higher predictive abilities that can further advance the prediction and control of disease outbreaks. Generally, early warning and response system that are capable of demonstrating evidence of prospective predictive ability and allows technical and practical adaptations of local public health responses while augmenting communications channels between users at central and district levels are tools that are more likely to be implemented into national surveillance programs. Advancing into frameworks that can facilitate at a low-cost IT maintenance and adapted to unskilled users are features of tools that are plausibly integrable into existing national systems.

As shown in the method section, the IBS and EBS are 2 main channels of information for a functioning EWS. Almost all studies reviewed in this paper maintained an IBS type of application to support EWS, with the exception of 1 Zika- and 1 malaria-related study using the EBS approach [24,46]. The combined use of both could potentially include other sources of information, such as sources from outside the health sector, which is the prime concept of the EBS. With the majority being of IBS-based EWS, the applications of the forecasting tools tend to be less efficient and deviate from the epidemic intelligence concept—the systematic collection, analysis, and communication of any information to detect, verify, assess, and investigate events and health risks with an early warning objective, as opposed to monitoring of disease trends or burdens [68]—which ideally combines both IBS and EBS for more robust outbreak detection.

Meteorological indicators are key predictors, but they are often inaccessible on a timely basis for health services managing the EWS. Benefiting from multiple assessments of users’ perspectives while defining the tool end users, countries like Mexico and Brazil, for instance, have managed to recognize the essence of the availability of local meteorological stations and have therefore organized an improved access to meteorological information [21,29,36,58,69].

Our literature review has identified very limited information for chikungunya and yellow fever outbreaks prediction, none of them fulfilling the inclusion criteria. Studies of EWS for yellow fever outbreaks are limited probably due to the existence of vaccines for this disease. However, chikungunya has now expanded from Africa, Latin America, and Asia to the European region [70,71]. It is quite worrying that studies on EWS tools for diseases like chikungunya are scarce. Although many EWS rely on meteorological information, only 3 studies [28,29,47] performed a prospective type analysis of early warning performance, with no rigorous study could be found.

Limitations of our study

The authors decided to include EWS on malaria to broaden the scope and eventually learn mutually from applications of different fields of disease control (Aedes borne versus Anopheles borne), but also by including yellow fever (vaccine preventable) and other vector-borne disease (mainly relying on vector control interventions). By virtue of their variability in terminology and definitions as well as difficulties in synthesis, the format of a scoping review was applied as a form of review design. Nevertheless, this scoping review follows the PRISMA criteria for conducting a systematic review without performing rigorous critical appraisal of included studies due to time constraints. However, we think that by including only peer-reviewed papers focusing on implemented EWSs or those under development provides a certain guarantee for high-quality papers, which will be an added value to the existing literature. The search strategy was limited to the target diseases not including search terms as “arboviruses” or “febrile illnesses” or “priority diseases” or others, which might be potentially relevant to the review, considering the broadness of the scoping review, Furthermore, stakeholders have been occasionally involved during the 14-month duration of this scoping process—during an Andean regional meeting in Bogota and in Colombia with an expert panel in charge of EWSs. Another limitation might be that we may have missed a few relevant studies; however, by including the Google Scholar search engine, we could overcome this issue, assuming that relevant papers could be identified, which were not indexed in other databases like PubMed. Due to the language barrier, we were unable to include publications in Chinese language, but several English papers from China were included, which could adequately represent the rich experience from the Chinese context.

Conclusions

This scoping review demonstrated gaps and challenges related to the structural, statistical, and operational designs of EWS, and these varied per disease and their corresponding settings. The country surveillance system is an integral part in the overall early warning process where the lack of accessibility to timely and quality data is crucial for establishing a reasonable EWS. Nevertheless, a substantial number of studies (except for dengue) failed to demonstrate any predictive power mainly that predictions based on complicated statistical models are difficult to carry out in low- and middle-income countries. This review has furthermore revealed a significant gap in effectively evaluating the role of EWS in the disease outbreak prediction and control given that the majority of EWS assessment studies have primarily been of retrospective designs. The lack of tool assessments regarding the implementation into existing routine surveillance as well as the feasibility of translating model outputs into local vector control and action plans will unlikely support the global health agenda for controlling disease outbreaks. Likewise, the missing user perspectives in the retrieved studies signals shows that most of the EWSs remain in the academic environment, and little effort has been spent on testing their effectiveness or cost-effectiveness in reducing disease outbreaks. Collectively, findings from this review claim the need for more pragmatic and context-adapted EWS tools, which address the user perspectives and its effectiveness in predicting outbreaks in local settings and trigger response activities.

Key learning points

  • Only minimal studies have addressed the early warning system (EWS) users’ perspectives with significant lack of implementation research assessments of EWS for chikungunya, dengue, malaria, yellow fever, and Zika outbreaks.
  • While the majority of studies have focused on the development and applications of the temporal prediction of the EWS, the spatial analysis of the disease prediction is crucial for effective vector control and response but rarely discussed or assessed in the literature.
  • The EWSs should be viewed as frameworks for improving the coordination of the overall disease outbreak control and response where full stakeholder involvement and assessment are warranted.

Top five papers

  1. Chen Y, Chu CW, Chen MIC, Cook AR. The utility of LASSO-based models for real time forecasts of endemic infectious diseases: A cross country comparison. J Biomed Inform. 2018;81:16–30. doi: 10.1016/j.jbi.2018.02.014
  2. Liu-Helmersson J, Quam M, Wilder-Smith A, Stenlund H, Ebi K, Massad E, et al. Climate Change and Aedes Vectors: 21st Century Projections for Dengue Transmission in Europe. EBioMedicine. 2016;7:267–77. doi: 10.1016/j.ebiom.2016.03.046
  3. Semenza JC. Prototype Early Warning Systems for Vector-Borne Diseases in Europe. Int J Environ Res Public Health. 2015;12:6333–51. doi: 10.3390/ijerph120606333
  4. Racloz V, Ramsey R, Tong S, Hu W. Surveillance of Dengue Fever Virus: A Review of Epidemiological Models and Early Warning Systems. PLoS Negl Trop Dis. 2012;6:e1648. doi: 10.1371/journal.pntd.0001648
  5. Louis VR, Phalkey R, Horstick O, Ratanawong P, Wilder-Smith A, Tozan Y, et al. Modeling tools for dengue risk mapping—a systematic review. Int J Health Geogr. 2014;13:50. doi: 10.1186/1476-072X-13-50

Supporting information

S1 Text. Country distribution of included studies.

https://doi.org/10.1371/journal.pntd.0009686.s001

(DOCX)

References

  1. 1. Bhatt S, Gething PW, Brady OJ, Messina JP, Farlow AW, Moyes CL, et al. The global distribution and burden of dengue. Nature 2013;496: 504–507. pmid:23563266; PubMed Central PMCID: PMC3651993.
  2. 2. Paul BJ, Sadanand S. Chikungunya Infection: A Re-emerging Epidemic. Rheumatol Ther. 2018;5: 317–326. pmid:30047016; PubMed Central PMCID: PMC6251852.
  3. 3. Heymann DL, Hodgson A, Sall AA, Freedman DO, Staples JE, Althabe F, et al. Zika virus and microcephaly: why is this situation a PHEIC? Lancet 2016;387: 719–721. pmid:26876373; PubMed Central PMCID: PMC7134564.
  4. 4. Tozan Y, Ratanawong P, Sewe MO, Wilder-Smith A, Kittayapong P. Household costs of hospitalized dengue illness in semi-rural Thailand. PLoS Negl Trop Dis 2017;11: e0005961. pmid:28937986; PubMed Central PMCID: PMC2615549.
  5. 5. Liu-Helmersson J, Quam M, Wilder-Smith A, Stenlund H, Ebi K, Massad E, et al. Climate Change and Aedes Vectors: 21st Century Projections for Dengue Transmission in Europe. EBioMedicine 2016;7: 267–277. pmid:27322480; PubMed Central PMCID: PMC4909611.
  6. 6. Semenza JC, Tran A, Espinosa L, Sudre B, Domanovic D, Paz S. Climate change projections of West Nile virus infections in Europe: implications for blood safety practices. Environ Health 2016;15: S28. pmid:26961903; PubMed Central PMCID: PMC4895699.
  7. 7. Rocklöv J, Quam MB, Sudre B, German M, Kraemer MUG, Brady O, et al. Assessing Seasonal Risks for the Introduction and Mosquito-borne Spread of Zika Virus in Europe. EBioMedicine 2016;9: 250–256. pmid:27344225; PubMed Central PMCID: PMC4972550.
  8. 8. Massad E, Amaku M, Coutinho FAB, Struchiner CJ, Burattini MN, Khan K, et al. Estimating the probability of dengue virus introduction and secondary autochthonous cases in Europe. Sci Rep 2018;8: 4629. pmid:29545610; PubMed Central PMCID: PMC5854675.
  9. 9. Snow RW, Sartorius B, Kyalo D, Maina J, Amratia P, Mundia CW, et al. The prevalence of Plasmodium falciparum in sub-Saharan Africa since 1900. Nature 2017;550: 515–518. pmid:29019978; PubMed Central PMCID: PMC5660624.
  10. 10. World Health Organization. World malaria report 2019. Geneva; 2019. p. 1–232 Available from: https://www.who.int/publications-detail-redirect/9789241565721
  11. 11. World Health Organization. Dengue vaccine: WHO position paper -July 2016. Wkly Epidemiol Rec. Geneva; 2016. p. 349–364. Available from: https://www.who.int/wer/2016/wer9130.pdf?ua=1 pmid:27476189
  12. 12. Bowman LR, Tejeda GS, Coelho GE, Sulaiman LH, Gill BS, McCall PJ, et al. Alarm Variables for Dengue Outbreaks: A Multi-Centre Study in Asia and Latin America. PLoS ONE 2016;11: e0157971. pmid:27348752; PubMed Central PMCID: PMC4922573.
  13. 13. Hii YL, Zhu H, Ng N, Ng LC, Rocklöv J. Forecast of Dengue Incidence Using Temperature and Rainfall. PLoS Negl Trop Dis 2012;6: e1908. pmid:23209852; PubMed Central PMCID: PMC3510154.
  14. 14. Ellis AM, Garcia AJ, Focks DA, Morrison AC, Scott TW. Parameterization and Sensitivity Analysis of a Complex Simulation Model for Mosquito Population Dynamics, Dengue Transmission, and Their Control. Am J Trop Med Hyg 2011;85: 257–264. pmid:21813844; PubMed Central PMCID: PMC3144822.
  15. 15. Rigau-Pérez JG, Millard PS, Walker DR, Deseda CC, Casta-Vélez A. A deviation bar chart for detecting dengue outbreaks in Puerto Rico. Am J Public Health 1999;89: 374–378. pmid:10076488; PubMed Central PMCID: PMC1508601.
  16. 16. Barbazan P, Yoksan S, Gonzalez J-P. Dengue hemorrhagic fever epidemiology in Thailand: description and forecasting of epidemics. Microbes Infect 2002;4: 699–705. pmid:12067829.
  17. 17. Thai KTD, Anders KL. The role of climate variability and change in the transmission dynamics and geographic distribution of dengue. Exp Biol Med (Maywood) 2011;236: 944–954. pmid:21737578.
  18. 18. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339: e1000097. pmid:19621072; PubMed Central PMCID: PMC2707599.
  19. 19. Schulz KF, Altman DG, Moher D, the CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8: 18. pmid:20334633; PubMed Central PMCID: PMC2844940.
  20. 20. Dankers FJWM, Traverso A, Wee L, van Kuijk SMJ. Prediction Modeling Methodology. In: Kubben P, Dumontier M, Dekker A, editors. Fundamentals of Clinical Data Science. Cham (CH): Springer; 2019. pp. 101–120.
  21. 21. Hussain-Alkhateeb L, Kroeger A, Olliaro P, Rocklöv J, Sewe MO, Tejeda G, et al. Early warning and response system (EWARS) for dengue outbreaks: Recent advancements towards widespread applications in critical settings. PLoS ONE 2018;13: e0196811. pmid:29727447; PubMed Central PMCID: PMC5935393.
  22. 22. Chen Y, Chu CW, Chen MIC, Cook AR. The utility of LASSO-based models for real time forecasts of endemic infectious diseases: A cross country comparison. J Biomed Inform. 2018;81:16–30. pmid:29496631
  23. 23. Semenza JC. Prototype Early Warning Systems for Vector-Borne Diseases in Europe. Int J Environ Res Public Health 2015;12: 6333–6351. pmid:26042370; PubMed Central PMCID: PMC4483704.
  24. 24. Teng Y, Bi D, Xie G, Jin Y, Huang Y, Lin B, et al. Dynamic Forecasting of Zika Epidemics Using Google Trends. PLoS ONE 2017;12: e0165085. pmid:28060809; PubMed Central PMCID: PMC5217860.
  25. 25. Chien L-C, Lin R-T, Liao Y, Sy FS, Pérez A. Surveillance on the endemic of Zika virus infection by meteorological factors in Colombia: a population-based spatial and temporal study. BMC Infect Dis 2018;18: 180. pmid:29665783; PubMed Central PMCID: PMC5905128.
  26. 26. Shi Y, Liu X, Kok S-Y, Rajarethinam J, Liang S, Yap G, et al. Three-Month Real-Time Dengue Forecast Models: An Early Warning System for Outbreak Alerts and Policy Decision Support in Singapore. Environ Health Perspect 2016;124: 1369–1375. pmid:26662617; PubMed Central PMCID: PMC5010413.
  27. 27. Chen Y, Ong JHY, Rajarethinam J, Yap G, Ng LC, Cook AR. Neighbourhood level real-time forecasting of dengue cases in tropical urban Singapore. BMC Med. 2018;16:129. pmid:30078378
  28. 28. Zhang H, Li Z, Lai S, Clements ACA, Wang L, Yin W, et al. Evaluation of the Performance of a Dengue Outbreak Detection Tool for China. PLoS ONE 2014;9. pmid:25170873; PubMed Central PMCID: PMC4149532.
  29. 29. Lowe R, Coelho CA, Barcellos C, Carvalho MS, Catão RDC, Coelho GE, et al. Evaluating probabilistic dengue risk forecasts from a prototype early warning system for Brazil. Elife 2016;5:e11285. pmid:26910315; PubMed Central PMCID: PMC4775211.
  30. 30. Ortiz PL, Rivero A, Linares Y, Pérez A, Vázquez JR. Spatial Models for Prediction and Early Warning of Aedes aegypti Proliferation from Data on Climate Change and Variability in Cuba. MEDICC Rev 2015;17: 20–28. pmid:26027583.
  31. 31. Ledien J, Souv K, Leang R, Huy R, Cousien A, Peas M, et al. An algorithm applied to national surveillance data for the early detection of major dengue outbreaks in Cambodia. PLoS ONE. 2019;14. pmid:30730979
  32. 32. Putra PS, Nuraini N. Modeling of dengue occurrences early warning involving temperature and rainfall factors. Asian Pacific Journal of Tropical Disease. 2017;7:385–90.
  33. 33. Hidayati R, Boer R, Koesmaryono Y, Kesumawati U, Manuwoto S. Dengue early warning model using development stages of aedes aegypti mosquito and climate information. Biotropia. 2012;19:30–41.
  34. 34. Githeko A, Ototo E, Muange P, Zhou GZ, Yan G, Sang J. The launch and operation of the malaria epidemic early warning system in Kenya. Science Publishing Group Journal. 2018;published online Aug 2: 1–12. https://doi.org/10.32392/biomed.20.
  35. 35. Wongkoon S, Jaroensutasinee M, Jaroensutasinee K. Assessing the temporal modelling for prediction of dengue infection in northern and north-eastern, Thailand. Trop Biomed 2012;29: 339–348. pmid:23018496.
  36. 36. Lowe R, Bailey TC, Stephenson DB, Jupp TE, Graham RJ, Barcellos C, et al. The development of an early warning system for climate-sensitive disease risk with a focus on dengue epidemics in Southeast Brazil. Stat Med 2013;32: 864–883. pmid:22927252.
  37. 37. Yu H-L, Yang S-J, Yen H-J, Christakos G. A spatio-temporal climate-based model of early dengue fever warning in southern Taiwan. Stoch Environ Res Risk Assess. 2011;25:485–94.
  38. 38. Lee J-S, Carabali M, Lim JK, Herrera VM, Park I-Y, Villar L, et al. Early warning signal for dengue outbreaks and identification of high risk areas for dengue fever in Colombia using climate and non-climate datasets. BMC Infect Dis 2017;17: 480. pmid:28693483; PubMed Central PMCID: PMC5504639.
  39. 39. Halide H, Ridd P. A predictive model for Dengue Hemorrhagic Fever epidemics. Int J Environ Health Res 2008;18: 253–265. pmid:18668414.
  40. 40. Eastin MD, Delmelle E, Casas I, Wexler J, Self C. Intra- and Interseasonal Autoregressive Prediction of Dengue Outbreaks Using Local Weather and Regional Climate for a Tropical Environment in Colombia. Am J Trop Med Hyg 2014;91: 598–610. pmid:24957546; PubMed Central PMCID: PMC4155567.
  41. 41. Zhang Y, Wang T, Liu K, Xia Y, Lu Y, Jing Q, et al. Developing a Time Series Predictive Model for Dengue in Zhongshan, China Based on Weather and Guangzhou Dengue Surveillance Data. PLoS Negl Trop Dis. 2016;10: e0004473. pmid:26894570; PubMed Central PMCID: PMC4764515.
  42. 42. Li Z, Liu T, Zhu G, Lin H, Zhang Y, He J, et al. Dengue Baidu Search Index data can improve the prediction of local dengue epidemic: A case study in Guangzhou, China. PLoS Negl Trop Dis. 2017;11: e0005354. pmid:28263988; PubMed Central PMCID: PMC5354435.
  43. 43. Sang S, Gu S, Bi P, Yang W, Yang Z, Xu L, et al. Predicting Unprecedented Dengue Outbreak Using Imported Cases and Climatic Factors in Guangzhou, 2014. PLoS Negl Trop Dis 2015;9. pmid:26020627; PubMed Central PMCID: PMC4447292.
  44. 44. Adde A, Roucou P, Mangeas M, Ardillon V, Desenclos J-C, Rousset D, et al. Predicting Dengue Fever Outbreaks in French Guiana Using Climate Indicators. PLoS Negl Trop Dis 2016;10: e0004681. pmid:27128312; PubMed Central PMCID: PMC4851397.
  45. 45. Guo P, Liu T, Zhang Q, Wang L, Xiao J, Zhang Q, et al. Developing a dengue forecast model using machine learning: A case study in China. PLoS Negl Trop Dis 2017;11: e0005973. pmid:29036169; PubMed Central PMCID: PMC5658193.
  46. 46. Merkord CL, Liu Y, Mihretie A, Gebrehiwot T, Awoke W, Bayabil E, et al. Integrating malaria surveillance with climate data for outbreak detection and forecasting: the EPIDEMIA system. Malar J. 2017;16:89. pmid:28231803
  47. 47. Githeko AK, Ndegwa W. Predicting Malaria Epidemics in the Kenyan Highlands Using Climate Data: A Tool for Decision Makers. Global Change & Human Health. 2001;2:54–63.
  48. 48. Githeko AK, Ogallo L, Lemnge M, Okia M, Ototo EN. Development and validation of climate and ecosystem-based early malaria epidemic prediction models in East Africa. Malar J. 2014;13:329. pmid:25149479
  49. 49. Midekisa A, Senay G, Henebry GM, Semuniguse P, Wimberly MC. Remote sensing-based time series models for malaria early warning in the highlands of Ethiopia. Malar J. 2012;11:165. pmid:22583705
  50. 50. Smith J, Tahani L, Bobogare A, Bugoro H, Otto F, Fafale G, et al. Malaria early warning tool: linking inter-annual climate and malaria variability in northern Guadalcanal, Solomon Islands. Malar J. 2017;16:472. pmid:29162098
  51. 51. Ruiz D, Poveda G, Vélez ID, Quiñones ML, Rúa GL, Velásquez LE, et al. Modelling entomological-climatic interactions of Plasmodium falciparum malaria transmission in two Colombian endemic-regions: contributions to a National Malaria Early Warning System. Malar J 2006;5: 66. pmid:16882349; PubMed Central PMCID: PMC1569851.
  52. 52. Racloz V, Ramsey R, Tong S, Hu W. Surveillance of Dengue Fever Virus: A Review of Epidemiological Models and Early Warning Systems. PLoS Negl Trop Dis. 2012;6: e1648. PubMed Central PMCID:. pmid:22629476
  53. 53. Louis VR, Phalkey R, Horstick O, Ratanawong P, Wilder-Smith A, Tozan Y, et al. Modeling tools for dengue risk mapping—a systematic review. Int J Health Geogr 2014;13: 50. pmid:25487167; PubMed Central PMCID: PMC4273492.
  54. 54. Velasco E, Agheneza T, Denecke K, Kirchner G, Eckmanns T. Social media and internet-based data in global systems for public health surveillance: a systematic review. Milbank Q 2014;92: 7–33. pmid:24597553; PubMed Central PMCID: PMC3955375.
  55. 55. Thomson M, Lyon B, Ceccato P. Climate matters in health decision-making. In: The Palgrave Handbook of Global Health Data Methods for Policy and Practice. Palgrave Macmillan, London; 2019. pp. 263–281.
  56. 56. Withanage GP, Viswakula SD, Nilmini Silva Gunawardena YI, Hapugoda MD. A forecasting model for dengue incidence in the District of Gampaha, Sri Lanka. Parasit Vectors. 2018;11:262. pmid:29690906
  57. 57. Ramadona AL, Lazuardi L, Hii YL, Holmner Å, Kusnanto H, Rocklöv J. Prediction of Dengue Outbreaks Based on Disease Surveillance and Meteorological Data. PLoS ONE 2016;11. pmid:27031524; PubMed Central PMCID: PMC4816319.
  58. 58. Lowe R, Bailey TC, Stephenson DB, Graham RJ, Coelho CAS, Sá Carvalho M, et al. Spatio-temporal modelling of climate-sensitive disease risk: Towards an early warning system for dengue in Brazil. Comput Geosci. 2011;37:371–81.
  59. 59. Rocklöv J, Tozan Y, Ramadona A, Sewe MO, Sudre B, Garrido J, et al. Using Big Data to Monitor the Introduction and Spread of Chikungunya, Europe, 2017. Emerg Infect Dis 2019;25: 1041–1049. pmid:31107221; PubMed Central PMCID: PMC6537727.
  60. 60. Ramadona AL, Tozan Y, Lazuardi L, Rocklöv J. A combination of incidence data and mobility proxies from social media predicts the intra-urban spread of dengue in Yogyakarta, Indonesia. PLoS Negl Trop Dis 2019;13: e0007298. pmid:30986218; PubMed Central PMCID: PMC6483276.
  61. 61. Wen T-H, Lin NH, Lin C-H, King C-C, Su M-D. Spatial mapping of temporal risk characteristics to improve environmental health risk identification: a case study of a dengue epidemic in Taiwan. Sci Total Environ 2006;367: 631–640. pmid:16584757.
  62. 62. Peters R, Stevenson M. Zika virus diagnosis: challenges and solutions. Clin Microbiol Infect 2019;25: 142–146. pmid:30553031.
  63. 63. RBM Partnership to End Malaria. RBM Partnership to End Malaria Annual Report 2018. Geneva; 2018. p 1–24. Available from: https://malariaworld.org/sites/default/files/RBM%20Annual%20Report%202018_EN.pdf
  64. 64. Global Partnership to Roll Back Malaria. A framework for field research in Africa: malaria early warning systems: concepts, indicators and partners. World Health Organization. Geneva; 2001. p. 1–80. Available from: https://apps.who.int/iris/handle/10665/66848
  65. 65. Thomson MC, Connor SJ. The development of Malaria Early Warning Systems for Africa. Trends Parasitol 2001;17: 438–445. pmid:11530356.
  66. 66. Runge-Ranzinger S, Kroeger A, Olliaro P, McCall PJ, Sánchez Tejeda G, Lloyd LS, et al. Dengue Contingency Planning: From Research to Policy and Practice. PLoS Negl Trop Dis 2016;10: e0004916. pmid:27653786; PubMed Central PMCID: PMC5031449.
  67. 67. Badurdeen S, Valladares DB, Farrar J, Gozzer E, Kroeger A, Kuswara N, et al. Sharing experiences: towards an evidence based model of dengue surveillance and outbreak response in Latin America and Asia. BMC Public Health. 2013;13:607. pmid:23800243
  68. 68. World Health Organization. Early detection, assessment and response to acute public health events: implementation of early warning and response with a focus on event-based surveillance: interim version. Geneva; 2014. p. 1–59. Available from: https://apps.who.int/iris/handle/10665/112667.
  69. 69. Lowe R, Barcellos C, Coelho CAS, Bailey TC, Coelho GE, Graham R, et al. Dengue outlook for the World Cup in Brazil: an early warning model framework driven by real-time seasonal climate forecasts. Lancet Infect Dis 2014;14: 619–626. pmid:24841859.
  70. 70. Dudouet P, Gautret P, Larsen CS, Díaz-Menéndez M, Trigo E, von Sonnenburg F, et al. Chikungunya resurgence in the Maldives and risk for importation via tourists to Europe in 2019–2020: A GeoSentinel case series. Travel Med Infect Dis 2020;36: 101814. pmid:32619732; PubMed Central PMCID: PMC7324928.
  71. 71. Emmanouil M, Evangelidou M, Papa A, Mentis A. Importation of dengue, Zika and chikungunya infections in Europe: the current situation in Greece. New Microbes and New Infections. 2020;35: 100663. pmid:32300479; PubMed Central PMCID: PMC7153298.