Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Assessing Interventions to Manage West Nile Virus Using Multi-Criteria Decision Analysis with Risk Scenarios

  • Valerie Hongoh ,

    valerie.hongoh@umontreal.ca

    Affiliations Groupe de Recherche en Épidémiologie des Zoonoses et Santé Publique, Faculté de médecine vétérinaire, Université de Montréal, Saint-Hyacinthe, Québec, Canada, Département de pathologie et microbiologie, Faculté de médecine vétérinaire, Université de Montréal, Saint-Hyacinthe, Québec, Canada

  • Céline Campagna,

    Affiliations Institut national de santé publique du Québec, Québec, Canada, Département de médecine sociale et préventive, Université Laval, Québec, Canada

  • Mirna Panic,

    Affiliations Institut national de santé publique du Québec, Québec, Canada, Canadian Field Epidemiology Program, Public Health Agency of Canada, Ottawa, Ontario, Canada

  • Onil Samuel,

    Affiliation Institut national de santé publique du Québec, Québec, Canada

  • Pierre Gosselin,

    Affiliations Institut national de santé publique du Québec, Québec, Canada, Ouranos, Consortium on regional climatology and adaptation to climate change, Montreal, Quebec, Canada

  • Jean-Philippe Waaub,

    Affiliation Group for Research in Decision Analysis, Montréal, Québec, Canada

  • André Ravel,

    Affiliation Département de pathologie et microbiologie, Faculté de médecine vétérinaire, Université de Montréal, Saint-Hyacinthe, Québec, Canada

  • Karim Samoura,

    Affiliations Groupe de Recherche en Épidémiologie des Zoonoses et Santé Publique, Faculté de médecine vétérinaire, Université de Montréal, Saint-Hyacinthe, Québec, Canada, Université Aube Nouvelle, Ouagadougou, Burkina Faso

  • Pascal Michel

    Affiliations Groupe de Recherche en Épidémiologie des Zoonoses et Santé Publique, Faculté de médecine vétérinaire, Université de Montréal, Saint-Hyacinthe, Québec, Canada, National Microbiology Laboratory at Saint-Hyacinthe, Public Health Agency of Canada, Saint-Hyacinthe, Québec, Canada

Assessing Interventions to Manage West Nile Virus Using Multi-Criteria Decision Analysis with Risk Scenarios

  • Valerie Hongoh, 
  • Céline Campagna, 
  • Mirna Panic, 
  • Onil Samuel, 
  • Pierre Gosselin, 
  • Jean-Philippe Waaub, 
  • André Ravel, 
  • Karim Samoura, 
  • Pascal Michel
PLOS
x

Abstract

The recent emergence of West Nile virus (WNV) in North America highlights vulnerability to climate sensitive diseases and stresses the importance of preventive efforts to reduce their public health impact. Effective prevention involves reducing environmental risk of exposure and increasing adoption of preventive behaviours, both of which depend on knowledge and acceptance of such measures. When making operational decisions about disease prevention and control, public health must take into account a wide range of operational, environmental, social and economic considerations in addition to intervention effectiveness. The current study aimed to identify, assess and rank possible risk reduction measures taking into account a broad set of criteria and perspectives applicable to the management of WNV in Quebec under increasing transmission risk scenarios, some of which may be related to ongoing warming in higher-latitude regions. A participatory approach was used to collect information on categories of concern to relevant stakeholders with respect to WNV prevention and control. Multi-criteria decision analysis was applied to examine stakeholder perspectives and their effect on strategy rankings under increasing transmission risk scenarios. Twenty-three preventive interventions were retained for evaluation using eighteen criteria identified by stakeholders. Combined evaluations revealed that, at an individual-level, inspecting window screen integrity, wearing light colored, long clothing, eliminating peridomestic larval sites and reducing outdoor activities at peak times were top interventions under six WNV transmission scenarios. At a regional-level, the use of larvicides was a preferred strategy in five out of six scenarios, while use of adulticides and dissemination of sterile male mosquitoes were found to be among the least favoured interventions in almost all scenarios. Our findings suggest that continued public health efforts aimed at reinforcing individual-level preventive behaviours combined with the application of larvicides to manage the risk of WNV infection are the interventions most acceptable and effective at reaching current management objectives now and under future theoretical transmission risk.

Introduction

West Nile virus (WNV) is a mosquito-borne flavivirus that first emerged in North America in New York City in 1999 [1,2] and in Canada in 2001 [3,4]. Most WNV infections are asymptomatic, but an important proportion can result in febrile illness with general muscle weakness (approximately 25% of infections) and in rare cases, more severe neurologic symptoms or death (less than 1% of infections) [5]. In the United States of America (US) alone, approximately 42,000 combined cases of neuroinvasive and non-neuroinvasive cases of WNV were reported between 1999 and 2015 with more than 1,700 associated deaths [6]. Over 5,200 cases were reported in Canada between 2002 and 2014, representing a much higher incidence rate relative to reports from the US (given Canada’s approximately 10 times smaller population) [7].

WNV’s emergence in the eastern Canadian province of Quebec in 2002 was linked to climatic conditions that occurred that year [8]. Vector-borne and zoonotic diseases (VBZD), such as WNV, are sensitive to changes in weather and climate [9] and incidence is anticipated to change in response to changes in climate [912]. Furthermore, multiple factors including weather are known to affect the transmission and distribution of WNV [13] and climatic projections for Quebec predict rising average temperatures (particularly in winter) and increased average precipitation [14]. As such, early preparedness and planning for current and future VBZD transmission dynamics is a key management strategy for improving public health adaptation to risks posed by climate change.

To date, WNV transmission dynamics have shown themselves to be largely unpredictable in the short term thereby increasing the need to elaborate management strategies that can cover a large range of epidemiologic scenarios [9].Human transmission of WNV in North America follows a seasonal pattern and is the result of a complex ecology of interacting species. The virus is maintained in an enzootic transmission cycle between birds and mosquitoes, primarily of the Culex genus, with occasional, dead-end infection in humans and other mammals generally appearing later in the summer season when virus amplification has reached a peak in its avian hosts and mosquito density is at a maximum [5,1517].

Due to the zoonotic nature and transmission dynamics of WNV, prevention and control opportunities should take place at a number of intervention levels, including: the avian reservoir, the mosquito vector or the human accidental host populations. Known prevention and control strategies range from preventive interventions aimed at individuals, such as the use of mosquito repellents and wearing protective clothing, to vector control interventions, including the application of larvicides or habitat modification measures to reduce mosquito abundance [1519]. Environmental control interventions aimed at the avian reservoir or the mosquito population have important operational, environmental, and social impacts. These impacts need to be accounted for above and beyond the cost of the interventions alone to ensure feasibility, acceptability and sustainability of the interventions. Although effective WNV vaccines exist for horses; a commercial human WNV vaccine does not yet exist [2022]. Research is ongoing and a number of promising candidates vaccines that have successfully undergone Phase I and Phase II clinical trials are in development; however, poor perceived cost-benefit of mass vaccination is often cited as the reason for lack of a licensed vaccine for humans at this time [2022].

Prevention and control of West Nile virus in the province of Quebec (Canada) has primarily consisted of source control of mosquito populations via the use of larvicides, integrated surveillance of humans, animals and mosquitoes, as well as sensitization of the public regarding personal protection measures (21). Uncertainties over the fluctuating yearly numbers of human cases and challenges relative to the perceived high cost of vector control activities in the context of fiscal restraint and government deficits provide ground for periodic re-assessment of the most effective risk reduction strategies. Furthermore, understanding and effectively tackling climate sensitive diseases such as WNV calls for a multidisciplinary perspective and multi-sectoral collaboration [2325]. Doing so will require transparent approaches that can keep sight of the overarching goal (i.e. reducing public health burden of disease) while taking into account multiple categories of concern, informed by a comprehensive review of available evidence and best-practices [24]. A multi-criteria decision analysis (MCDA) informed approach with multiple stakeholders can help structure reflection and aid in decision-making on the basis of multiple, potentially conflicting criteria (designed to measure specific categories of concern) [26] thereby providing a structured mechanism for multidisciplinary and multi-sectoral collaboration on a decision problem. MCDA enables the ranking of multiple interventions based on a list of stakeholder identified and weighted concerns (i.e. decision criteria) and thus allows for an appreciation of the relative strengths and weakness of various interventions under consideration.

In the current study, preventive interventions for the management of WNV were identified, assessed and ranked using a multi-stakeholder informed MCDA to document effective, favoured and acceptable interventions relating to management of the disease in Quebec under varying increasing transmission risk scenarios in order to help inform future seasonal operational decision making at the provincial level.

Materials and Methods

A participatory methodology was adapted from an existing MCDA model for Lyme disease management [27]. MCDA is a formal method that can be used to combine evidence-based information and stakeholder values to support decision-making (Fig 1) [26]. The MCDA method consists first of a ‘problem structuring’ phase. This phase describes the decision problem and identifies a list of management interventions and the important criteria that need to be taken into account when evaluating these interventions. Discussion of the proposed criteria and intervention list by participants ensures exhaustiveness and transparency. Interventions are evaluated using peer-reviewed, grey literature and available data pertaining to all of the retained criteria. Criteria are then weighted by importance by all stakeholders using a standardized form, under different epidemiological transmission scenarios. This allows stakeholders to modify the relative importance of decision criteria (e.g.: incidence reduction vs. cost), depending on the situation they are faced with (ex: low-risk scenario vs. high-risk scenario) and their perspective of the decision problem. The ‘problem structuring’ phase is richest when performed with a varied group of stakeholders, allowing for the integration of multiple concerns (i.e. criteria), and creating the opportunity to build a common understanding of the decision problem. The second stage of the MCDA process is the ‘decision analysis’ phase, where the MCDA analysis tool is used to aggregate the information collected in the first phase (i.e. intervention evaluations and criteria weights) in order to produce a relative ranking of assessed interventions.

Transmission scenarios

Interventions for the prevention and control of human WNV in the province were evaluated under current and future possible transmission scenarios of WNV in order to support governmental decision making. We constructed six scenarios to reflect potential increases in transmission under current and potential future fluctuations in transmission intensity of the disease. The scenarios themselves are hypothetical and do not reflect historical reality, nor do they reflect a scientific consensus on expected future conditions, rather these scenarios depict fictional, yet climatically plausible WNV transmission scenarios for the province of Quebec [28]. For each scenario, a combination of WNV transmission risk intensity (low, medium and high) and interventions having taken place during the current season were described (Table 1).

thumbnail
Table 1. Transmission risk scenarios assessed under the MCDA model for West Nile virus interventions in Quebec.

https://doi.org/10.1371/journal.pone.0160651.t001

Identification of stakeholders

Stakeholders (n = 15) already involved in WNV management from various levels of government, academia as well as from an existing expert committee on WNV in Quebec were invited to participate in the MCDA process in April 2014. Invited stakeholders included individuals from the National institute of public health, Ministry of health and social services, the Public Health Agency of Canada, Ministry of sustainable development, environment and the Fight against climate change, the academic sector, the Quebec center for wildlife health, companies involved in mosquito control operations, Ouranos Consortium for research in climatology and adaptation to climate change and Quebec regional public health authorities. The protocol for this project was reviewed and approved by the Ethical Committee for Health Research of the University of Montreal (Comité d’éthique de la recherche en santé, CERES) (certificate number 14-025-CERES-D). All participants gave informed written email consent for inclusion prior to participation in the study.

Identification of potential interventions

A comprehensive literature review was conducted to construct a preliminary list of interventions for discussion with stakeholders [5,17,19,2931]. Interventions including active and passive surveillance, large scale and targeted communication campaigns and various prevention and control interventions were included in this preliminary list. Interventions under development and implementable under both a short and long-term perspective were included in order to provide a range of options to cover all transmission scenarios. A baseline, status quo intervention encompassing passive surveillance of human cases and representative of what is currently done to manage WNV in the province was also included (please note that interventions will hereafter be shown in italics in the text while criteria will be shown in “quotes” to ease readability). The proposed interventions were then discussed and validated with participating stakeholders during a focus group discussion. Individual feedback was solicited from all stakeholders following the discussion by means of a Delphi survey during which stakeholders had the opportunity to suggest additional interventions previously missed [32]. Consensus was not explicitly sought during this process; rather stakeholders agreed that an intervention would be retained in the model so long as at least one stakeholder deemed it pertinent to include.

Identification of decision criteria

Drawing from previous work [27,33], a preliminary list of 15 evaluation criteria, distributed over five categories (“Public Health” criteria, “Social Impact” criteria, “Economic” criteria, “Strategic and Operational”, and “Animal and Environmental Health” criteria) was compiled by the research team. Each criterion was defined with a measurement scale (allowing for a quantitative or qualitative assessment of an intervention), including a direction of desired effect. Linear preference functions were used with all criteria and qualitative assessments were transformed into monotone ascending or descending scales depending on the direction of the desired effect [34]. The relevance of criteria and their measurement scales was discussed and validated with stakeholders. Individual feedback was also solicited via a Delphi survey [32]. Once again, consensus was not explicitly sought regarding retained criteria; rather a criterion was retained so long as at least one stakeholder deemed it pertinent. Weights of zero were permitted by stakeholders to indicate absence of importance for a given criterion during the weighting process (described in the following section).

Criteria weighting

Stakeholders were asked to weight the relative importance of criteria under all transmission scenarios. Scenarios were presented to stakeholders as hypothetical yet climatically plausible transmission scenarios meant to examine the effect of changing criteria trade-offs under different transmission intensities. For the weighting exercise, stakeholders were given a Microsoft Excel spreadsheet tool and asked to distribute 100 points across the list of criteria included in the model. The more points given to a criterion, the more important this criterion for the stakeholder, thus permitting a relative ranking of criteria. The process was repeated for each of the six scenarios by all stakeholders. Differences in assigned weights were tested between groups of stakeholders using Welch’s t-test (unequal variances t-test) in R (version 3.3.0) to test for differences in the mean category weights.

Evaluations of interventions

Assessments were performed for all interventions for all criteria using measurement scales discussed and finalized with stakeholders (see S1 Table). Evaluations were based on existing peer-reviewed evidence, grey literature and available data (see S2 Table for the results of this evaluation and supporting references in S1 Appendix). A comprehensive literature review was performed for all interventions. When data was not available for an evaluation, expert judgment was used. All information relative to the evaluations was compiled into an assessment matrix then revised and discussed by all evaluators. Assessments were further reviewed and validated by external experts with specific field or research experience.

The population specific criterion (“proportion affected”) was assessed as the estimated proportion of the population currently employing these measures for individual-level interventions in population-level analyses. Where data was incomplete, the incidence reduction criterion was assessed as either known to reduce cases or reducing contact between vectors and human hosts. The entomological risk reduction criterion was assessed with regards to having an effect on reducing the population or density of mosquitoes. Data availability and reliability of assessments was tracked to reflect the degree of certainty over provided assessment distinguishing literature based assessment versus expert opinion or field tested result.

Multi-criteria decision analysis

The evaluations of all interventions were aggregated with criteria weights and analyzed using a multi-criteria analysis tool. The PROMETHEE method (Preference Ranking Organization Method for Enrichment Evaluations) [34] was used to perform multi-criteria analysis with the D-Sight software (version 3.3.2, D-Sight company). Geometrical analysis for interactive aid (GAIA) analysis maps, available with the D-Sight software, were also used to aid in visual interpretation of results [35,36]. Two main sets of analyses were performed, one based on individual-level interventions (n = 11) and the second based on regional-level interventions (n = 10). A subset of mosquito-targeting control measures (n = 8), as well as a subset of the currently available interventions (i.e. interventions ready for deployment within the next year in the province; n = 5), were analysed separately. For the purpose of exploratory comparison, an analysis of combined individual-level and regional-level interventions was also performed. Following this, sensitivity analyses were performed on all criteria and for all stakeholders to examine the robustness of rankings and identify potentially weight-sensitive criteria.

Results

Stakeholder consultation and MCDA model construction

Twelve stakeholders (out of 15 invited) consented to participate in the study. Following presentation and discussion of the preliminary lists of interventions and criteria with stakeholders, a final list of 23 interventions (Table 2) and 18 evaluation criteria were retained (Table 3). The identified interventions included individual protective measures, mosquito source reduction measures, adult mosquito control measures, and interventions aimed at the animal reservoir. Four of the twenty-three interventions were not assessed due to insufficient information in the literature to do so (use of lethal ovitraps, reduction in abundance of the main animal reservoir, modification of animal reservoir habitat, and increased biodiversity at the peridomestic level). Although communication and surveillance interventions were explicitly recognized as important elements within a VBZD management programme by stakeholders, these interventions were not included in the current model due to concerns regarding the ability to properly assess the efficacy of these interventions under one comprehensive model. The consensus was to explore these interventions separately in a future exercise.

thumbnail
Table 2. Potential protection and control interventions for the management of West Nile virus in Quebec.

https://doi.org/10.1371/journal.pone.0160651.t002

Criteria weighting

Stakeholder weights for the criteria under all scenarios are included in the supporting information (see S3S5 Tables for the individual weighting results). The criteria deemed most important (most points attributed per criterion by stakeholders), were predominantly criteria related to the “Public Health” category, followed by the “Economic” category or the “Strategic and Operational” criteria category. In nearly all transmission scenarios, “Animal and Environmental Health” criteria ranked lowest, with fewest weights attributed by stakeholders. Within the “Public Health” category, a majority of weights were attributed to the “incidence reduction” criterion, and “physical health impact” criterion. Within the “Social Impact” category, the credibility impact” criterion received the highest weight in most scenarios. Within the “Economic” criteria category, the “government cost” criterion received the highest weight. Within the “Strategic and Operational” criteria category, the “delay” criterion was given highest weight for medium and high scenarios. Finally, in the “Animal and Environmental Health” criteria category, the “environmental impact” criterion was given the highest weight for all scenarios.

Global results

A strong level of congruence was generally observed among weights expressed by stakeholders across all scenarios. The high-risk transmission scenario analysis of regional-level interventions illustrates this (Fig 2). In Fig 2, two semi-coalitions of stakeholders can be observed consisting in one case of stakeholders 4,5,7,9 and 10 and in the second case of stakeholders 1,2,3,5,8,11 and 12. Stakeholder positions are generally all pointing in the same direction as the decision axis indicating that no stakeholders is in direct opposition to the group consensus; however slight differences between these two groups of stakeholder weights can be observed. A statistical comparison of weights (Welch’s t-test, unequal variances) revealed that these two groups of stakeholders had significant differences in weights for the Social impact category (p = 0.015) as well as the Animal and Environmental Health criteria category (p = 0.04). From an organizational standpoint, stakeholders in the 2nd coalition consist of a mix of organizations including public health, wildlife and environmental management. The 1st coalition consists of a mix of wildlife and public health related organizations. The bigger difference between these two groups may be their spatial planning mandates with stakeholders in coalition 2 having more involvement in daily field operations and stakeholders from coalition 1 being more involved at a regional planning scale, though not strictly so. Both points of view are important to take into account and despite their differences in weighting; there is a consensus with regards to recommended interventions. Stakeholder positions were seen to converge under scenarios of increasing severity. Sensitivity analyses were performed to examine the robustness of weights given by stakeholders to criteria in the models and their effect on the overall rankings. The criteria most sensitive to stakeholder weights primarily consisted of criteria from the “Public Health” category, as well as the “credibility impact” criterion, “individual cost” criterion and “government cost” criterion.

thumbnail
Fig 2. GAIA decision map for regional-level model under scenario 6 (high-risk transmission with interventions).

Vector points Act1 through Act12 represent the 12 stakeholders in the model. Points INT-8-23 represent the various Interventions under consideration in this analysis (see Table 2). The red vector indicates the group decision axis with preferred direction indicated by the red dot. Proximity of intervention points along the decision axis represents group ranking preference for these interventions. The relatively proximity of all stakeholder points in the same general direction as the decision axis indicates that all stakeholders are generally in agreement with the group decision axis, and no stakeholder is diametrically opposed to this decision. The close proximity of all stakeholders to one another furthermore indicates fairly strong consensus between stakeholders. There are two slightly divergent coalitions of stakeholders (1st group consists of stakeholders above the decision axis and the 2nd group consists of those below) indicating that these two groups have slightly different perspectives with regards to their criteria weighting, but these differences in perspective are not in conflict with the group decision axis. (Zoom = 300% and Delta = 92.2%, indicates that 92.2% of the information is conserved in the two-dimensional representation of this decision map).

https://doi.org/10.1371/journal.pone.0160651.g002

Ranking of individual-level interventions

The top four ranked personal protection interventions, inspecting window screen integrity, wearing lightly colored clothing, and eliminating peridomestic mosquito larval sites, reducing outdoor activities at peak times, were identical across all scenarios (Table 4). These rankings are based on evidence-based assessment scores combined with stakeholder assigned weights. Fig 3 shows how Inspecting window screen integrity scores high on a majority of criteria with the exception of “entomological risk reduction”, “reduction of circulating virus” and “social equality” where it received lower scores. The second and third ranked interventions, wearing light colored clothing and eliminating peridomestic larval sites, also scored highly on a majority of criteria (Fig 3).

thumbnail
Fig 3. Intervention profiles for six individual-level protection interventions.

Each bar represents one of the criteria included in the model. Values along the vertical axis indicate the scores received for the intervention on a particular criterion. Values above zero indicate good performance of the intervention for that criterion based on evaluation scores and conversely, values below zero indicate “poor” relative performance. Criteria bar color codes: red: Public Health criteria; orange: Social Impact criteria; blue: Economic criteria; purple: Strategic and Operational criteria; green: Animal and Environmental Health criteria. (Please refer to supplementary material for all other intervention profiles).

https://doi.org/10.1371/journal.pone.0160651.g003

The least favoured interventions among this subset varied slightly from one transmission scenario to another, but generally included: use of alternative technologies, human vaccination and status quo (Table 2). Examination of the profiles for the bottom ranked interventions, status quo and human vaccination, (Fig 2) shows how these interventions score poorly on most criteria including many “Public Health” criteria, a category consistently weighted highly by all stakeholders. Human vaccination in particular scores poorly over many criteria, notably “entomological risk reduction”, “physical health impact”, “social equity” (if not covered by universal health care, then some costs must be incurred by the general public for vaccination), “public acceptance”, “credibility impact”, “government cost”, “individual cost”, “delay”, and “complexity” (highly complex since licensed human vaccine not yet available).

Ranking of regional-level interventions

In the model containing regional-level management interventions (Table 5), the top three identified interventions were consistently: larvicides, vaccination of animal reservoir and modification of man-made larval sites with small variations in the order of these interventions depending on the scenarios. Examination of regional-level intervention profiles showed larvicides, vaccination of animal reservoir and modification of man-made larval sites, to be top scorers over most of the criteria, although Larvicides scored less well on the “government cost”, “complexity”, “other policy impact”, “animal health impact” and “environmental impact” criteria (see S2 and S3 Figs). The vaccination of animal reservoir intervention was found to score less well on the “incidence reduction” criterion compared to larvicides, but scored relatively well on other criteria “reduction of circulating virus” criterion in particular. The modification of man-made larval sites intervention scored less well on “Economic” criteria, “Strategic and Operational” criteria and the “Animal and Environmental Health” criteria.

The ordering of the bottom three interventions included: use of adulticides, dissemination of sterile males, and human vaccination in the low and medium-risk scenarios. For the high-risk scenarios, the bottom ranked interventions changed to include use of parasites and pathogenic microorganisms instead of human vaccination.

Ranking of mosquito-targeting and currently available interventions

Among the mosquito-targeting interventions (Table 6), the top two ranked were larvicides and modification of man-made larval sites. This was followed by use of mosquito predators in the first two scenarios and modification of natural larval sites across remaining scenarios. The bottom three ranked interventions included, in order, the use of parasites and pathogenic microorganisms, the dissemination of sterile males and the use of adulticides for the low- and medium- risk scenarios with the ordering of the last two interventions reversed for the two high-risk scenarios.

In the analysis of currently available to deploy regional-level management interventions (Table 7), the ranking did not change for any of the six scenarios and included larvicides, modification of man-made larval sites, status quo, use of parasites and pathogenic microorganisms and use of adulticides in the listed order.

Ranking of combined individual- and regional-level interventions

In the combined model of individual- and regional-level interventions (Table 8), inspecting window screens and wearing lightly colored clothing were always ranked 1st and 2nd. This was most often followed by larvicides in all but the low-risk scenario 2 where it was replaced by eliminating peridomestic larval sites. The bottom three ranked interventions most often included use of parasites and pathogenic microorganisms, dissemination of sterile males, and adulticides.

thumbnail
Table 8. Ranking of the individual-level protection and regional-level management interventions combined.

https://doi.org/10.1371/journal.pone.0160651.t008

Discussion

This study has demonstrated adaptation planning for management of WNV under various increasing transmission risk scenarios using multi-criteria decision analysis (MCDA). To the best of our knowledge, this is the first study to use MCDA for management planning of WNV. Aenishaenslin and colleagues (2013) had previously demonstrated the possibility of MCDA use for management of Lyme disease emergence in Canada and had suggested that general criteria categories exist that are suitable for VBZD management at large [27]. The categories retained in our study are consistent with previous multi-stakeholders concerted decisions that have taken place in public health over the past 20 years [33,37,38]. Our study further supports the application of MCDA for VBZD and reinforces the notion of common categories of concern to consider in VBZD management. Additionally, our study has shown how many of these concerns remain relevant under various increasing transmission risk scenarios, many of which are coherent with a change in transmission intensity that could occur with climate change. Indeed, substantial warming in higher-latitude regions will likely open up new terrain for some infectious diseases that are limited at present by low temperature boundaries [39].

The degree of concern (weights) attributed to different criteria by stakeholders was shown to vary with transmission intensity of scenarios. This was expected as we anticipated that an increasing number of reported cases in the scenarios would lead to increased concern for public health and social impact related considerations thereby triggering a trade-off among remaining criteria. A similar result was found in the Lyme disease study [27]. A priori hypotheses around economic cost trade-offs were that as WNV incidence increased, costs would become less of a concern with regards to investment in interventions. Indeed, this pattern is observed but is more apparent when scenarios 1,3,and 5 (scenarios without interventions performed during the current season) are compared together versus scenarios 2,4 and 6 (scenarios where interventions have been carried out during the current season). Despite the decreasing importance of cost under increased transmission intensity, important differences in intervention rankings were not observed. The ranking of interventions was found to vary under different scenarios and among the different models. This was also expected since changes in weights affect rankings. Intervention profiles can be examined to further understand the relative rankings of interventions independently of stakeholder assigned weights (see supplementary material for comprehensive coverage of profiles). Model rankings and interpretation are discussed in more detail in the following sections.

Individual-level protection model

The relative rankings of individual-level interventions were generally not found to vary considerably across the scenarios (low to higher risk transmission). This stability suggests specific protective behaviors that remain effective and acceptable and should continue to be promoted in communication campaigns in order to reinforce adaptive capacity under increasing transmission intensity.

The individual-level model results observed where inspection of window screens, wearing light colored clothing, eliminating peridomestic larval sites and reducing outdoor activities at peak times were highly ranked and use of alternate technologies, human vaccination and status quo were lower ranked are consistent with primary prevention messages already included in Quebec WNV communication campaigns as well as other Canadian and the US ones [4042]. These messages are also consistent with personal protection methods prescribed within integrated vector management programs in Europe [18]. The inspection of window screens in particular was the most highly ranked intervention at this level and indeed is already a common and well accepted practice in most homes in the province of Quebec [43]. As such few if any financial costs are expected to be associated with the promotion of this strategy; however, individuals without sufficient economic means may be less likely to replace or purchase window screens. Examination of the relative strengths and weaknesses of interventions via their intervention profiles (see Fig 3 and S1 Fig) illustrates how a comprehensive public health strategy can be built that addresses all concerns raised by stakeholders. For example, the second and third ranked interventions, wearing light colored clothing and eliminating peridomestic larval sites, which also ranked highly, are complementary to the inspecting window screen integrity intervention as they score well on criteria where inspecting window screens performed less well (Fig 3).

Regional-level management model

Overall, the rankings of regional-level interventions were found to vary more than individual-level interventions across the increasing transmission intensity scenarios. The positional stability of top ranked interventions here too suggests specific actions to manage WNV effectively that remain acceptable across a range of transmission dynamics. The positional change of other interventions such as vaccination or modification of natural mosquito larval sites, under the higher transmission risk scenarios suggests increased acceptability of potentially more controversial interventions under these conditions. Periodic re-evaluations are warranted as additional information becomes available for these interventions.

Evaluated regional-level interventions were primarily vector targeted with the exception of the vaccination (human and animal) and status quo (human passive surveillance) interventions. Top ranking interventions included larvicides, vaccination of animal reservoir and modification of man-made larval sites having scored highly on most criteria but with important trade-offs on other criteria. For example, Larvicides scored poorly on cost, operational complexity and environmental criteria. Mosquito control programs are costly and complex to operate as they require entomological surveillance programs, well-trained staff and infrastructure [17] and repeated application in order to maintain effectiveness [44,45]. Nevertheless, vector control remains key to effective vector borne disease management [46]. While the vaccination of animal reservoir intervention was highly ranked, the inclusion of a criterion explicitly targeting the level of circulating virus in the animal reservoir may explain the high ranking of this strategy as it is the only measure that directly acts on this aspect of transmission. A few studies have demonstrated success with this measure [4749] but for the time being, it remains a hypothetical intervention for the province of Quebec. With regards to man-made larval sites, studies have found that proximity to certain types of structures such as combined sewer overflow systems have been significantly associated with high rates of WNV infection in humans and corvids; however, construction and modification of major infrastructure can be very costly [29,50,51]. Additionally, man-made water systems such as those designed to handle sewer overflow may have negative impacts on water quality and animal health by association [52].

Mosquito-targeting and currently available management models

The top ranked mosquito interventions, larvicides and modification of man-made sites, performed well on most “Public Health” Criteria. However, these interventions had economic, environmental and operational shortcomings that would need to be addressed in any comprehensive public health strategy. In the model examining only the list of currently available interventions, the rank ordering of interventions did not change for any of the six scenarios and included larvicides, modification of man-made larval sites, status quo, use of parasites and pathogenic microorganisms and use of adulticides in this order. This stable ranking across scenarios adds to the robustness of these interventions suggesting their capacity to meet current and higher intensity transmission scenario management demands.

Combined model

In the combined model, four out of the top seven interventions included individual measures. This suggests that based on available evidence, current epidemiological levels of WNV, and values held by experts in Quebec, interventions aimed at personal level protection, source reduction or reduction of circulating levels of virus are most appropriate over habitat modification interventions and other forms of vector control and also under the higher transmission risk scenarios described in this study. These results are in agreement with the management options currently implemented in Quebec and elsewhere in North America although other forms of vector control (such as the use of adulticides) have been employed elsewhere in North America under high levels of WNV transmission [5,53].

Limitations

It must be clarified that the MCDA approach is based on a socio-constructivist paradigm and that the validity of results are not based on strict reproducibility of results, but rather representativeness of society or relevant group of experts. The validity is also intimately tied to the coherence and transparency of results that are modeling a complex system. There are limits inherent in the choice of stakeholders, but the stakeholders chosen in our exercise were meant to be relevant to the dimensions at stake within the decision problem. In our example, as a first consultation, stakeholders from public health, wildlife and environmental management responded to our invitation to participate in this exercise. These stakeholders were representative of real-life management in the context of the study (small province where such files are managed by no more than 10–12 people) although many participants were indeed involved in previous WNV outbreaks. It is likely that given a different set of stakeholders, values expressed would be different.

With regards to interventions, from our initial stakeholder validated list, four interventions were found to currently lack sufficient data for evaluation (use of lethal ovitraps, reduction in abundance of the main animal reservoir species, modification of habitat to reduce host reservoir species, and increasing biodiversity at the peridomestic level). While MCDA methods exist to deal with missing data [54], these were not explored in the current study to avoid speculating on their efficacy and acceptability. Future models should explore these interventions as data becomes available.

The exploration of multiple scenarios in the models did not yield very different rankings. While some differences in stakeholder weights were observed, convergence of stakeholder values was seen under scenarios of increased transmission severity; however, this did not strongly impact rankings. Many of the stakeholders have been working together on WNV related projects for a number of years which may in part explain the observed homogeneity in responses. A recommendation for future studies would be to include a more diverse group of stakeholders including, amongst others, front line clinicians responsible for providing care to the general population and members of the general population themselves to examine the potential variation in responses. Furthermore, to reduce workload, to explore low and high transmission scenarios first and if variations are found, to follow-up with medium transmission scenarios analyses where warranted.

Intervention evaluations were not re-assessed under the different scenarios. While many of these evaluations would likely not have changed, the social impact related evaluations might have with potential effects on rankings. However, no data were available to document this change for the current evaluation. An exploration of these and other potential changes to evaluations under different transmission scenarios in future studies may be warranted.

The PROMETHEE algorithm used in the ranking process provides a relative position for ordered interventions, therefore while general observations can be taken away from this analysis, such as individual preventive measures being preferred over regional-level interventions, the actual ranking results are valid only for the current model. In other words, middle or bottom ranked interventions should not necessarily be dismissed as being “poor”, rather they are less favoured over the top ranked interventions in the current model but still remain viable options to explore in future models or analyses as new options and information become available. Overall “poor” interventions, known to be so at the outset should not be included in the model in the first place. For this reason, it is worthwhile to explore specific subsets of interventions to further deepen our understanding of why one intervention may outperform another.

Conclusions

While integrated vector management is often the primary recommendation for VBZD control [18,19,55], multicriteria decision analysis (MCDA) can be used to further refine the selection of complementary interventions for a VBZD management programme. MCDA can integrate cost-benefit analysis type information and other categories of concern including social acceptability and animal and environmental health concerns. Additionally, the use of scenarios enables the examination of tradeoffs between intervention performance and acceptability under different conditions.

The MCDA approach provides the opportunity to not only offer an informed recommendation to decision makers, but also an opportunity to build a shared understanding of the decision problem between different disciplines and sectors thereby increasing adhesion and support of all final recommendations. Further diversifying the stakeholder composition to include various representatives of society can also contribute to this process and should be explored in future projects.

Decisions are ultimately political but must be informed and supported by the best available evidence. While the explicit ranking of possible interventions often represent the main management objective driving a comparative assessment of interventions, the rigorous MCDA process in itself provides a framework to explicitly deconstruct stakeholder expressed priorities, rendering the decision-making process more transparent and arguably richer in its ability to document trade-offs and differing perspectives.

This project showed how a vector-borne and zoonotic diseases (VBZD) management model can be created to assess intervention options for the management of West Nile virus (WNV) at both the individual- and regional-levels taking into account currently available evidence now and under future potential transmission intensity scenarios, some of which are coherent with transmission intensity that may be seen under ongoing climate change. The results confirm that prevention of WNV via individual-level prevention measures such as well maintained window screens, in conjunction with source reduction regional-level interventions, such as larvicides, were top ranked interventions consistent with expressed stakeholders perspectives and in-line with currently stated WNV management objectives in the province of Quebec. Given the depth of both the model building exercise and broad similarities in approaching public health interventions for VBZD, we conclude that this current WNV model is likely useful as a base starting point for the analysis of other mosquito-borne diseases. Further work is warranted to better understand and clarify decision making mechanisms and determinants leading to selecting effective public health interventions for other VBZD now and under climate change.

Supporting Information

S1 Appendix. References used in the assessment of management interventions.

https://doi.org/10.1371/journal.pone.0160651.s001

(DOCX)

S1 Fig. Additional individual-level protection strategy performance profiles.

https://doi.org/10.1371/journal.pone.0160651.s002

(TIF)

S2 Fig. Regional-level management intervention profiles (interventions 11–16).

https://doi.org/10.1371/journal.pone.0160651.s003

(TIF)

S3 Fig. Regional-level management intervention profiles (interventions 18–23).

https://doi.org/10.1371/journal.pone.0160651.s004

(TIF)

S1 Table. Measurement scales used to score interventions in the model.

https://doi.org/10.1371/journal.pone.0160651.s005

(DOCX)

S2 Table. Matrix of evaluation scores for the interventions in the Quebec WNV management model.

https://doi.org/10.1371/journal.pone.0160651.s006

(DOCX)

S3 Table. Stakeholder weighting results by criteria and category for the Scenarios 1& 2 (low risk transmission).

https://doi.org/10.1371/journal.pone.0160651.s007

(DOCX)

S4 Table. Stakeholder weighting results by criteria and category for the Scenarios 3& 4 (medium risk transmission).

https://doi.org/10.1371/journal.pone.0160651.s008

(DOCX)

S5 Table. Stakeholder weighting results by criteria and category for the Scenarios 5& 6 (high risk transmission).

https://doi.org/10.1371/journal.pone.0160651.s009

(DOCX)

Acknowledgments

The authors would like to sincerely thank all stakeholders and experts consulted during this project for their participation and contribution. The authors would also like to thank the Public Health Agency of Canada for funding the project and the Quebec National Institute of Public Health for its collaboration in this project. VH is funded by the Canadian Institutes of Health Research (CIHR), Ouranos Consortium for research in climatology and adaptation to climate change and the Strategic Training Program in Global Health Research, a partnership of CIHR and the Quebec Population Health Research Network (RRSPQ). This project is also nested within the IRIACC-FACE program funded by IDRC, CIHR, NSERC and SSHRC from Canada. The funding sources had no involvement in the actual research or preparation of the paper.

Author Contributions

  1. Conceived and designed the experiments: VH CC OS MP PM.
  2. Performed the experiments: VH CC OS MP.
  3. Analyzed the data: VH CC MP OS PM AR PG JW KS.
  4. Contributed reagents/materials/analysis tools: VH CC OS MP.
  5. Wrote the paper: VH CC MP OS PM AR PG JW KS.

References

  1. 1. Lanciotti RS, Roehrig JT, Deubel V, Smith J, Parker M, Steele K, et al. Origin of the West Nile virus responsible for an outbreak of encephalitis in the northeastern United States. Science. 1999;286:2333–7. pmid:10600742
  2. 2. Nash D, Mostashari F, Fine A, Miller J, O’Leary D, Murray K, et al. The outbreak of West Nile virus infection in the New York City area in 1999. N. Engl. J. Med. 2001;344:1807–14. pmid:11407341
  3. 3. Venter A. West Nile virus reaches Canada. Trends Microbiol. 2001;9:469.
  4. 4. Campbell GL, Marfin AA, Lanciotti RS, Gubler DJ. West Nile virus. Lancet Infect. Dis. 2002;2:519–29. pmid:12206968
  5. 5. Petersen LR, Brault AC, Nasci RS. West nile virus: Review of the literature. JAMA-J. Am. Med. Assoc. 2013;310:308–15.
  6. 6. Centers for Disease Control and Prevention. West Nile Virus | Features | CDC [Internet]. 2015 [cited 2015 Aug 21]. Available from: http://www.cdc.gov/features/westnilevirus/
  7. 7. Government of Canada. Surveillance of West Nile virus [Internet]. 2015 [cited 2015 Aug 21]. Available from: http://healthycanadians.gc.ca/diseases-conditions-maladies-affections/disease-maladie/west-nile-nil-occidental/surveillance-eng.php
  8. 8. El Adlouni S, Beaulieu C, Ouarda T, Gosselin P, Saint-Hilaire A. Effects of climate on West Nile Virus transmission risk used for public health decision-making in Quebec. Int. J. Health Geogr. 2007;6:40. pmid:17883862
  9. 9. Mills JN, Gage KL, Khan AS. Potential influence of climate change on vector-borne and zoonotic diseases: a review and proposed research plan. Environ. Health Perspect. 2010;118:1507–14. pmid:20576580
  10. 10. Gubler DJ. The global threat of emergent/reemergent vector-borne diseases. In: Lemon SM, Sparling PF, Hamburg MA, Relman DA, Choffnes ER, Mack A, editors. Vector-Borne Dis. Underst. Environ. Hum. Health Ecol. Connect. Workshop Summ. Forum Microb. Threats. Washington, D. C.: The National Academies Press; 2008. p. 43–64.
  11. 11. Kilpatrick AM, Randolph SE. Drivers, dynamics, and control of emerging vector-borne zoonotic diseases. The Lancet. 2012;380:1946–55.
  12. 12. Altizer S, Ostfeld RS, Johnson PTJ, Kutz S, Harvell CD. Climate Change and Infectious Diseases: From Evidence to a Predictive Framework. Science. 2013;341:514–9. pmid:23908230
  13. 13. Paz S. Climate change impacts on West Nile virus transmission in a global context. Philos. Trans. R. Soc. Lond. B Biol. Sci. [Internet]. 2015;370. Available from: http://rstb.royalsocietypublishing.org/content/370/1665/20130561.abstract
  14. 14. Plummer DA, Caya D, Frigon A, Côté H, Giguère M, Paquin D, et al. Climate and Climate Change over North America as Simulated by the Canadian RCM. J. Clim. 2006;19:3112–32.
  15. 15. Hayes EB, Gubler DJ. West Nile Virus: epidemiology and clinical features of an emerging epidemic in the United States. Annu. Rev. Med. 2006;57:181–94. pmid:16409144
  16. 16. Hayes EB, Komar N, Nasci RS, Montgomery SP, O’Leary DR, Campbell GL. Epidemiology and transmission dynamics of West Nile virus disease. Emerg. Infect. Dis. 2005;11:1167–73. pmid:16102302
  17. 17. Reisen W, Brault AC. West Nile virus in North America: perspectives on epidemiology and intervention. Pest Manag. Sci. 2007;63:641–6. pmid:17373672
  18. 18. Bellini R, Zeller H, Bortel WV. A review of the vector management methods to prevent and control outbreaks of West Nile infection and the challenge for Europe. Parasit. Vectors. 2014;7.
  19. 19. Nasci RS, Fischer M, Lindsey NP, Lanciotti RS, Savage HM, Komar N, et al. West Nile virus in the United States: guidelines for surveillance, prevention and control. Centers for Disease Control and Prevention; 2013.
  20. 20. Amanna IJ, Slifka MK. Current Trends in West Nile Virus Vaccine Development. Expert Rev. Vaccines. 2014;13:589–608. pmid:24689659
  21. 21. Brandler S, Tangy F. Vaccines in Development against West Nile Virus. Viruses. 2013;5:2384. pmid:24084235
  22. 22. Iyer AV, Kousoulas KG. A Review of Vaccine Approaches for West Nile Virus. Int. J. Environ. Res. Public. Health. 2013;10:4200. pmid:24025396
  23. 23. Rabinowitz P, Conti L. Links Among Human Health, Animal Health, and Ecosystem Health. Annu. Rev. Public Health. 2013;34:189–204.
  24. 24. Coker R, Rushton J, Mounier-Jack S, Karimuribo E, Lutumba P, Kambarage D, et al. Towards a conceptual framework to support one-health research for policy on emerging zoonoses. Lancet Infect. Dis. 2011;11:326–31. pmid:21376670
  25. 25. Karesh WB, Dobson A, Lloyd-Smith JO, Lubroth J, Dixon MA, Bennett M, et al. Ecology of zoonoses: natural and unnatural histories. The Lancet. 2012;380:1936–45.
  26. 26. Huang IB, Keisler J, Linkov I. Multi-criteria decision analysis in environmental sciences: Ten years of applications and trends. Sci. Total Environ. 2011;409:3578–94. pmid:21764422
  27. 27. Aenishaenslin C, Hongoh V, Cisse H, Hoen A, Samoura K, Michel P, et al. Multi-criteria decision analysis as an innovative approach to managing zoonoses: results from a study on Lyme disease in Canada. BMC Public Health. 2013;13:897. pmid:24079303
  28. 28. Harrigan RJ, Thomassen HA, Buermann W, Smith TB. A continental risk assessment of West Nile virus under climate change. Glob. Change Biol. 2014;20:2417–2425.
  29. 29. Labbé Y, Aubé-Maurice B, Vézina A, Boisvert J, Gingras D. Revue des mesures de prévention et de protection contre le virus du Nil occidental. Institut national de santé publique du Québec; 2006 p. 173.
  30. 30. Tran A, Sudre B, Paz S, Rossi M, Desbrosse A, Chevalier V, et al. Environmental predictors of West Nile fever risk in Europe. Int. J. Health Geogr. 2014;13.
  31. 31. Paz S, Semenza JC. Environmental Drivers of West Nile Fever Epidemiology in Europe and Western Asia—A Review. Int. J. Environ. Res. Public. Health. 2013;10:3543. pmid:23939389
  32. 32. Sawford K, Dhand N, Toribio J-A, Taylor M. The use of a modified Delphi approach to engage stakeholders in zoonotic disease research priority setting. BMC Public Health. 2014;14:182. pmid:24552445
  33. 33. Morestin F, Gauvin F, Hogue M, Benoit F. Method for synthesizing knowledge about public policies [Internet]. National Collaborating Centre for Healthy Public Policy; 2011. Available from: http://www.ncchpp.ca/docs/methodPP_EN.pdf
  34. 34. Brans JP, Vincke P. A Preference Ranking Organisation Method: (The PROMETHEE Method for Multiple Criteria Decision-Making). Manag. Sci. 1985;31:647–56.
  35. 35. De Smet Y, Lidouh K. An Introduction to Multicriteria Decision Aid: The PROMETHEE and GAIA Methods. In: Aufaure M-A, Zimányi E, editors. Bus. Intell. [Internet]. Springer Berlin Heidelberg; 2013. p. 150–76. Available from: http://dx.doi.org/10.1007/978-3-642-36318-4_7
  36. 36. Brans J-P, Mareschal B. Promethee Methods. Mult. Criteria Decis. Anal. State Art Surv. [Internet]. Springer New York; 2005. p. 163–86. Available from: http://dx.doi.org/10.1007/0-387-23081-5_5
  37. 37. Bolduc D, Ricard S, Québec I national de santé publique du. Cadre de référence en gestion des risques pour la santé dans le réseau québécois de la santé publique. [Montréal]: Institut national de santé publique du Québec; 2003.
  38. 38. Ross W, Birkwood P, Henter T, Krewski D, Muise AM, Nahas A. Health Canada Decision-Making Framework for Identifying, Assessing, and Managing Health Risks—August 1, 2000 [Internet]. Health Canada; 2000 [cited 2015 Sep 25]. Available from: http://www.hc-sc.gc.ca/ahc-asc/pubs/hpfb-dgpsa/risk-risques_tc-tm-eng.php
  39. 39. Smith KR, Woodward A, Campbell-Lendrum D, Chadee DD, Honda Y, Liu Q, et al. Human Health: Impacts, Adaptation, and Co-Benefits. In: Field CB, Barros VR, Dokken DJ, Mach KJ, Mastrandrea MD, Bilir TE, et al., editors. Clim. Change 2014 Impacts Adapt. Vulnerability Part Glob. Sect. Asp. Contrib. Work. Group II Firth Assess. Rep. Intergov. Panel Clim. Change. Cambridge, United Kingdom and New York, NY, USA: Cambridge University Press; 2014. p. 709–54.
  40. 40. Gouvernement du Québec. Portail santé mieux-être [Internet]. Se Prot. Piqûres Moustiques Tiques. 2015 [cited 2015 Jul 27]. Available from: http://www.xn--sant-epa.gouv.qc.ca/conseils-et-prevention/se-proteger-des-piqures-de-moustiques/
  41. 41. Government of Canada. Healthy Canadians [Internet]. Prev. West Nile Virus. 2015 [cited 2015 Jul 27]. Available from: http://healthycanadians.gc.ca/diseases-conditions-maladies-affections/disease-maladie/west-nile-nil-occidental/prevention-eng.php
  42. 42. CDC. West Nile virus [Internet]. Cent. Dis. Control Prev. 2015 [cited 2015 Jul 27]. Available from: http://www.cdc.gov/westnile/prevention/index.html
  43. 43. Grondin J, Corriveau R, Bolduc D, Brunel M. Virus du nil occidental: Évaluation des attitudes, Des comportements et des Connaissances populaires. Institut national de santé publique; 2003.
  44. 44. Stockwell PJ, Wessell N, Reed DR, Kronenwetter-Koepel TA, Reed KD, Turchi TR, et al. A field evaluation of four larval mosquito control methods in urban catch basins. J. Am. Mosq. Control Assoc. 2006;22:666–71. pmid:17304935
  45. 45. Hribar LJ, Fussell EM, Leal AL. Larviciding Offshore Islands Reduces Adulticidal Treatment of Populated Areas Adjacent To National Wildlife Refuges. J. Am. Mosq. Control Assoc. 2011;27:408–13. pmid:22329274
  46. 46. Townson H, Nathan M, Zaim M, Guillet P, Manga L, Bos R, et al. Exploiting the potential of vector control for disease prevention. Bull. World Health Organ. 2005;83:942–7. pmid:16462987
  47. 47. Bunning ML, Fox PE, Bowen RA, Komar N, Chang G-JJ, Speaker TJ, et al. DNA Vaccination of the American Crow (Corvus brachyrhynchos) Provides Partial Protection Against Lethal Challenge with West Nile Virus. Avian Dis. 2007;51:573–7. pmid:17626486
  48. 48. Kilpatrick AM, Dupuis AP, Chang G-JJ, Kramer LD. DNA Vaccination of American Robins (Turdus migratorius) Against West Nile Virus. Vector-Borne Zoonotic Dis. 2010;10:377–80. pmid:19874192
  49. 49. Young JA, Jefferies W. Towards the Conservation of Endangered Avian Species: A Recombinant West Nile Virus Vaccine Results in Increased Humoral and Cellular Immune Responses in Japanese Quail (Coturnix japonica). PLoS ONE. 2013;8:e67137. pmid:23825633
  50. 50. Harbison JE, Metzger ME, Hu R. Association Between Culex quinquefasciatus (Diptera: Culicidae) Oviposition and Structural Features of Belowground Stormwater Treatment Devices. J. Med. Entomol. 2010;47:67–73. pmid:20180310
  51. 51. Vazquez-Prokopec G, Vanden Eng JL, Kelly R, Mead DG, Kolhe P, Howgate J, et al. The risk of West Nile virus infection is associated with combined sewer overflow streams in Urban Atlanta, Georgia, USA. Environ. Health Perspect. 2010;118:1382–8. pmid:20529765
  52. 52. Lund A, McMillan J, Kelly R, Jabbarzadeh S, Mead DG, Burkot TR, et al. Long term impacts of combined sewer overflow remediation on water quality and population dynamics of Culex quinquefasciatus, the main urban West Nile virus vector in Atlanta, GA. Environ. Res. 2014;129:20–6. pmid:24528998
  53. 53. Ministère de la Santé et des Services sociaux. Plan d’intervention gouvernemental 2013–2015 pour la protection de la population contre le virus du Nil occidental Mise à jour—Saison 2014. 2014.
  54. 54. Greco S, Matarazzo B, Slowinski R. Dealing with Missing Data in Rough Set Analysis of Multi-Attribute and Multi-Criteria Decision Problems. In: Zanakis S, Doukidis G, Zopounidis C, editors. Decis. Mak. Recent Dev. Worldw. Appl. [Internet]. Springer US; 2000. p. 295–316. Available from: http://dx.doi.org/10.1007/978-1-4757-4919-9_20
  55. 55. Lowe A-M. Le risque relié au virus du Nil occidental au Québec et les interventions à privilégier en 2013. Institut national de santé publique du Québec; 2014 Apr p. 83.