A Quantitative and Novel Approach to the Prioritization of Zoonotic Diseases in North America: A Public Perspective

Background Zoonoses account for over half of all communicable diseases causing illness in humans. As there are limited resources available for the control and prevention of zoonotic diseases, a framework for their prioritization is necessary to ensure resources are directed into those of highest importance. Although zoonotic outbreaks are a significant burden of disease in North America, the systematic prioritization of zoonoses in this region has not been previously evaluated. Methodology/Principal Findings This study describes the novel use of a well-established quantitative method, conjoint analysis (CA), to identify the relative importance of 21 key characteristics of zoonotic diseases that can be used for their prioritization in Canada and the US. Relative importance weights from the CA were used to develop a point-scoring system to derive a recommended list of zoonoses for prioritization in Canada and the US. Over 1,500 participants from the general public were recruited to complete the online survey (761 from Canada and 778 from the US). Hierarchical Bayes models were fitted to the survey data to derive CA-weighted scores. Scores were applied to 62 zoonotic diseases of public health importance in Canada and the US to rank diseases in order of priority. Conclusions/Significance This was the first study to describe a systematic and quantitative approach to the prioritization of zoonoses in North America involving public participants. We found individuals with no prior knowledge or experience in prioritizing zoonoses were capable of producing meaningful results using CA as a novel quantitative approach to prioritization. More similarities than differences were observed between countries suggesting general agreement in disease prioritization between Canadians and Americans. We demonstrate CA as a potential tool for the prioritization of zoonoses; other prioritization exercises may also consider this approach.


Introduction
Zoonotic diseases are defined as those that are naturally transmitted between vertebrate animals and humans [1]. Zoonoses account for over half of all communicable diseases causing illness in humans [2,3]. As there are limited resources available for research, surveillance, control and prevention of zoonoses, it is necessary to prioritize diseases in order to direct resources into those with the greatest needs. While there is general consensus amongst medical and veterinary professionals for the need to prioritize zoonoses, there are numerous challenges. Firstly, zoonotic diseases vary greatly in their occurrence and in their health impact on the human and animal populations, making it difficult to compare their overall public health importance [2]. Second, there is no universal agreement on the measurable criteria by which to quantify and prioritize zoonoses, nor is there agreement on the methodologies to elicit such information [4][5][6]. Lastly, there are a number of stakeholders involved, each with their own objectives [7]. It is therefore difficult to establish a universally accepted priority list for zoonotic diseases.
A number of studies have attempted to methodically prioritize communicable diseases of public health concern [6,[8][9][10][11][12][13]. With respect to zoonoses, a framework for their prioritization has been developed in France (non-foodborne zoonoses) [14,15], Belgium (foodborne zoonoses) [16], The Netherlands (emerging zoonoses) [17] and Europe (food-producing animal diseases and zoonoses) [18]. Krause and colleagues [4] compared prioritization methodologies published between 1995 and 2005 and noted there was no uniformity in the objectives, methodological approaches, criteria for prioritization, number of criteria considered, level of standardization and weighting of criteria. Despite methodological differences, it is agreed that risk-based priority should be systematic, empirical and quantitative, easy to implement, based on good science, transparent, flexible, reproducible and informative to public policy [4,19]. In recent studies, there has been a shift towards the use of novel quantitative approaches to overcome limitations in traditional methods and to address the complexity of disease prioritization [4,6,13,16,17].
Conjoint analysis (CA) has been used in market research of consumer preferences over the last 40 years [20]. More recently, this method has gained widespread use in the health and medical setting in eliciting preferences for healthcare [21][22][23]. The theory behind CA is that any product (goods or services) can be described by a set of characteristics and the extent to which an individual places value on a product is determined by the level of those separate characteristics (part-worth of the product) and the combination of those characteristics (overall worth of the product) [21,22,24]. CA forces individuals to make decisions by presenting competing products with both desirable and undesirable characteristics and asking individuals to state a preference. In expressing preference, individuals make a trade-off between the desirable and undesirable characteristics in those products allowing researchers to determine the true value of each characteristic relative to all other characteristics. A relative weighted score for each characteristic is derived as well as an overall score for each competing product as a whole. In the context of zoonoses, each disease can be treated as a product described by a set of disease criteria (characteristics), and the value of the disease is determined by the level of each criterion and the combination of the value of those levels. The overall score for each disease can be used to rank diseases in order of priority. Although there are similar methods to CA including Maximum Difference Scaling (MaxDiff) that has an emphasis on prioritizing a list of characteristics together [24], the complexity of zoonoses requires an understanding of preferences in the face of multiple levels of multiple characteristics combined together. Conjoint Analysis allows for the exploration of interrelationships between various levels of disease characteristics together and was considered a more appropriate method for exploring preferences for disease characteristics and levels.
Current prioritization methods are limited by numerous challenges [4][5][6][8][9][10][11][12][13][14][15][16][17][18], many of these can be overcome using CA, these include: (1) generating relative weighted scores; a better representation of criteria importance compared to arbitrary scores and subjective weights, (2) eliciting preferences through choice; an intuitive approach as opposed to ranking or rating diseases, (3) considering disease criteria jointly rather than separately; a more realistic approach to decision-making that recognizes disease criteria are not equal nor independent from each other, (4) presenting zoonoses as a set of disease characteristics without identifying diseases; this forces individuals to prioritize based on science thereby reducing biases associated with disease names, these can include the potential fear of a disease name compared to a lesser-known disease or prioritizing diseases based on professional interest and/or personal gain), and (5) presenting respondents with all the information necessary to prioritize diseases; this allows for prioritization to be based on wide social participation including public participation and experts who may not be familiar with the full range of diseases. We note in particular point number 4, by choosing to conduct an unlabeled prioritization study.
Although zoonotic outbreaks are a significant burden of disease in North America [25][26][27], the systematic prioritization of zoonoses in this region has not been previously evaluated. The primary objective of this study was to describe the novel use of a quantitative method, CA, to develop a point scoring system for disease criteria considered important in determining zoonotic disease priority. The second objective was to use the CA-derived scores to develop and compare a priority list of zoonoses in Canada and the US. This paper will focus on the results of individuals from the general public; this is the first study to describe a systematic and quantitative approach to the prioritization of zoonotic diseases from a public perspective.

Focus Groups and Criteria Identification
The authors previously described the six focus groups using the nominal group technique (NGT) to inform this study [7]. Individuals from the public and medical and veterinary professionals identified 59 unique disease criteria to prioritize zoonoses. Of these, 21 were used to inform the CA experimental design (Tables 1, 2, 3). These criteria were selected on the basis of having a high mean score (derived from the NGT) across groups, identified by three of more groups, or corresponding to humanrelated criteria that were deemed important (e.g. incidence in humans scored high, therefore, incidence in animals was included despite a lower score). A number of criteria were merged together, for example, pathogenicity, immunogenicity, incubation period, communicability and mode of transmission were integrated into transmission potential. This process allowed for a more efficient number of the most relatively important disease characteristics to be included in the study.
Criteria that were difficult to quantify in measureable units (e.g. ability of the disease pathogen to mutate) or lacked scientific information for quantification (e.g. bioterrorism potential) were excluded. Criteria pertaining to public disruption, awareness and perception were also excluded. Although the professional groups acknowledged disease prioritization was often driven by public and political pressure, they agreed it should not steer the process [7]. The final 21 selected criteria could all be quantitatively measured with scientific information available in the literature.

Disease Selection and Literature Search
We identified 62 existing and emerging zoonotic and enteric diseases of public health importance in North America (Tables 4,  5). These diseases were selected using the following criteria: a) nationally notifiable to the Public Health Agency of Canada (PHAC), Canadian Food Inspection Agency (CFIA), Centers for Disease Control and Prevention (CDC) or the United States Department of Agriculture (USDA), b) internationally notifiable to the World Health Organization (WHO), the World Organization for Animal Health (OIE) or the Food and Agriculture Organization (FAO) or c) identified as a priority by PHAC at a national meeting on non-enteric zoonoses in 2009. We note that the majority of enteric diseases in this study are zoonotic; hence, the group of diseases in this paper will be referred to as zoonotic diseases. However, there are three diseases in this study that are strictly enteric diseases with no zoonotic involvement, these are, Cholera, Hepatitis A and Paralytic Shellfish Poisoning (PSP). As enteric diseases typically fall under the responsibility of the Zoonotic Division in public health (for example, PHAC and the CDC), these diseases were also included in our study.
A literature search was conducted for each zoonosis in the study. Diseases exhibiting multiple forms (e.g. acute/chronic, latent/active, encephalitic/non-encephalitic) were divided into separate syndromes and assigned approximate proportions that were informed by the literature. There were 117 separate disease syndromes identified from the 62 diseases (Tables 4, 5). The literature search obtained information for each criterion for each disease syndrome. The literature search included: a) website searches of human and animal health organizations involved in zoonotic disease prevention and control including national organizations, intergovernmental organizations, provincial organizations and academic institutions; b) reference textbooks [2,28,29] and c) PubMed cataloged peer-reviewed publications. Key search terms used included the disease criterion, the scientific and/or common name of diseases, and a combination of the two (e.g. case-fatality rate and/or rabies).
When there was conflicting information between multiple sources, the most consistent sources were used to define the disease criteria; where only two sources were available, the most recent source was used. When information was not available for North America, information available for other developed countries was used. Country-specific information for Canada and the US were collected for four criteria -disease incidence in the last five years (humans and animals) and disease trend in the last five years (humans and animals). An assumption was made that the remaining criteria were consistent between countries. When there was a gap in the literature, particularly pertaining to animalrelated criteria [30], information from corresponding humanrelated criteria were used to define animal-related criteria on the assumption that disease criteria were similar between humans and animals. Animal-related criteria, however, were not used to inform human-related criteria due to existing data for humans.

Defining Levels for Disease Criteria
Three or four levels for each criterion were defined according to the range exhibited in the literature (Tables 1, 2, 3). For example, the duration of illness in humans among diseases ranged from no duration (asymptomatic) to chronic disability (permanent deficits). For the criteria, case-fatality and disease incidence (both humans and

Survey Development and Administration
We used 21 criteria with criteria levels ranging from three to four levels that reflected the full range of each criterion across the 62 diseases to inform our survey instrument. Due to the large number of criteria, a partial-profile choice-based conjoint (CBC) survey was developed comprising of 14 choice tasks [31,32]; each choice task presented participants with five disease combinations (zoonoses 1-5) described by criteria levels for 5 of the 21 criteria. Disease criteria and levels varied between choice tasks. Participants were asked to select one zoonosis to prioritize in either Canada or the US with the objective being for policy implementation for control and prevention ( Figure 1). Surveys were administered electronically to allow for flexibility in the study design and wide participation across North America. Surveys were offered in English, French and Spanish. We used Sawtooth Table 3. Disease criteria and standardized part-worth utility values (b) for disease criteria levels by country.

Disease criteria 1 and corresponding levels
Canada

Economic and social burden on trade in animals
No cost to trade in animals 225. 21  Software CBC module v7 to create 300 survey versions to generate an efficient experimental design (D-efficiency of 908.13326 relative to a full-orthogonal design with a standard error of ,0.05 for each criterion level). The selected design ensured a number of important features of an efficient experimental design were included, these were: level balance (each criteria level appeared approximately an equal number of times across the 14 choice tasks), orthogonality (criteria levels were selected independently of other criteria levels) and a balanced overlap approach (the design permitted levels to appear more than once per choice task for a more precise measurement of interaction terms) [33]. Disease combinations were designed to be easily understood, to mimic the presentation of zoonoses in nature and to enhance informant and statistical efficiency [24,31,32,34]. Two additional fixed choice tasks were included in all survey versions to determine the reliability of responses. In these choice tasks, one zoonosis was more severe as described by the set of criteria compared to the remaining four, giving participants an incentive to select that zoonosis to prioritize. Fixed choice task 1 presented one zoonosis with the highest incidence in humans (10,000 cases), most severe illness in humans (severe clinical symptoms), highest transmission potential between humans (high), highest case-fatality in humans (80%) and the most costly economic burden in humans ($10,000 per sick individual). In comparison, the remaining four zoonoses contained a combination of lower and less severe criteria levels. Fixed choice task 2 presented one zoonosis with the most severe illness in animals (severe clinical symptoms), highest case-fatality in animals (80%), most costly socioeconomic burden in trade in animals (high cost such as culling of herds or destroying infected crops/produce), longest duration of illness in animals (chronic illness or permanent deficits) and rapid change in disease trend in the human population (new emerging disease, rapid increase over the last five years). In comparison, the remaining four zoonoses contained a combination of lower and less severe criteria levels. The fixed choice tasks tested the reliability of responses by identifying respondents who did not understand the choice task process and/ or fatigue responders. The ordering of the presentation of disease criteria within each choice task was randomized to reduce ordering bias. The fixed choice tasks were also randomized to reduce ordering bias and as an additional measure of reliability given the tasks were designed to test the reliability of respondents to understand the choice task process rather than the reliability of respondents to make the same choices consistently across the two fixed choice task sets. Sawtooth Software SSI Web v7 was used to randomly assign a survey version to each study participant.

Study Participants
The study was approved by the Research Ethics Board at the University of Guelph. The target study participants were individuals from the public in Canada and the US with no knowledge or experience in the prioritization of zoonotic diseases. Participants from both countries were recruited using an online panel through Research Now TM ; these are groups of pre-screened individuals who have expressed a willingness to participate in online surveys. Surveys were collected anonymously. All participants acknowledged an informed consent assuring confidentiality and the option to withdraw from participation without penalty. Sawtooth Software SSI Web v7 was used to screen participants through a series of demographic questions prior to survey commencement. Participants were disqualified if they did not reside in North America or were employed in the following fields: epidemiology, public health, medical sciences, veterinary sciences, infectious disease research, laboratory technician, nursing or dentistry. Quotas were used to ensure that no particular age group, gender or geographic region dominated the survey responses and that study populations were representative of the national populations in their respective countries (Table 6).

Data Analysis
We used Chi Square and Mann-Whitney tests to compare the demographic and survey characteristics of study participants to their respective national populations and to make comparisons by country. National population data for age, gender, education and region was obtained from Statistics Canada [35,36] and the US Census Bureau [37,38]. Unpaired t-tests and F-tests were used to explore differences in standardized importance scores and partworth utility values between Canada and the US. Twenty-one part-worth utility values, one for each disease criterion, were assigned to the 117 separate disease syndromes by matching the level of each disease criterion to those of disease syndromes. Partworth utility values were summed up in proportion to the relative frequency of each syndrome within a disease (Tables 4, 5) to derive an overall score for each of the 62 diseases. The overall scores were used to rank-order diseases; the higher the score, the higher the ranking on the priority list. We chose to use summed part-worth utility values over other approaches such as market simulations because we wanted to apply CA-derived scores to a set of disease using a method that is comparable to current traditional prioritization methods [4,[11][12][13][16][17][18]. We also selected this approach because our goal was to prioritize the list of diseases in ranked order, which can be achieved using the simple summed part-worth utility values approach.
Hierarchical Bayes (HB) was used to compute parameter estimates (weighted scores) from survey choice data [39]. We used Sawtooth Software CBC/HB v5.2.8 to estimate individual-level parameter estimates (b). The software uses a combination of Bayes theorem, a Monte Carlo Markov Chain (MCMC) procedure and the Metropolis/Hastings algorithm to derive parameter estimates from two distributions: an upper-level model (prior) drawn from a multivariate normal distribution representing parameter estimates at the population level and a lower-level model (posterior) described by a multinomial logit model representing parameter estimates at the individual level. Bayesian updating of probabilities using MCMC and the Metropolis/Hastings algorithm provided an iterative process to update parameter estimates drawing on the upper-level and lower-level model [39]. The final individual-level parameter estimates reflected an optimal mix of the population average and individual choices [40]. We computed 30,000 preliminary iterations before convergence was assumed (and observed) and an additional 30,000 iterations per respondent to estimate parameters.
Zero-centered part-worth utility values (b estimates) were standardized by setting the average range of the parameter values of all disease criteria to 100. Part-worth utility values represent the Figure 1. Example of one choice task set completed by each study participant. As multiple survey versions were administered randomly to each person, a different combination of disease criteria and levels was presented to study participants. The ordering of the presentation of disease criteria within each choice task was randomized to reduce ordering bias. doi:10.1371/journal.pone.0048519.g001 relative influence each criterion level had on respondent choices with higher values indicating a stronger influence on choice [24]. To estimate the influence of each criterion collectively, importance scores were calculated as a percentage by dividing the difference in range between the highest and lowest criterion level part-worth utility value by the sum of all part-worth utility value ranges across all criteria. The more variation between the levels in a criterion, the higher the importance score and the stronger the influence the criterion had on the decision to prioritize [24]. Part-worth utility values and importance scores were calculated directly using Sawtooth Software SMRT v4.22.0. We computed t-statistics by dividing the mean difference in range in part-worth utility values across each criterion by the standard error of the differences to test each disease criterion for statistical significance in the final model.
The goodness of fit of the individual-level HB models were estimated using Sawtooth Software CBC/HB v5.2.8 and are presented as a percent certainty fit and a root likelihood (RLH); both of these measures are calculated as the probability of each respondent choosing as he/she did on each choice task using a logit model fitted with the current estimates of each respondent's part-worth utility values [39]. The percent certainty and RLH both indicate how much better the model is than a chance model, as compared to a perfect model.

Demographic and Survey Characteristics
In total, 46,547 and 8,298 email invitations were sent to Canadian and US participants, respectively. Of these, 1,313 (2.8%) and 1,309 (15.8%) completed surveys were returned in 8 and 7 business days, respectively. The majority of completed surveys were in English (77.1% in Canada, 99.4% in the US) while remaining surveys were completed in French (22.9% in Canada) and Spanish (0.6% in US). Participants passed the survey if all 14 choice task sets were completed and the correct diseases were selected for both fixed choice tasks 2 . The pass rate was 58.0% in Canada (761) and 59.4% in the US (778); there was no significant difference in the pass rate between countries (x 2 = 0.59, p = 0.44). The median completion time for passed surveys was 26.9 minutes in Canada and 28.1 minutes in the US. There were 1,539 completed and passed surveys in this study.
The study population was generally representative of the national population by gender, age and geography (Table 6). There was a significantly higher response rate in the 35 to 50 years age group in the Canadian study population compared to the Canadian national population (p,0.001). A higher than expected educated population was observed in both study populations compared to their respective national populations (p,0.001 for both countries).

Model Fit
Both the Canadian and the US models had a percent certainty fit of 79.4% and a root likelihood (RLH) of 0.72. The expected percent certainty for a chance model is 0% and a perfect model is 100% while the expected RLH for a chance model is 0.2 (one divided by five disease combinations per task) and a perfect model is 1.0 [39].

Disease Criteria Importance Scores and Part-worth Utility Values
The importance scores for disease criteria indicate the degree to which each criterion contributed to the decision to prioritize (Table 7). Human-related criteria were preferred over corresponding animal-related criteria with each of the eight matching criteria exhibiting this trend in both countries. The four transmission potential criteria were ranked in the following order of preference in both countries: human-human, animal-human, humans-animals and animal-animal; thus also revealing a stronger preference for human-related criteria over animal-related criteria. Although the contribution of each disease criterion in the decision to prioritize zoonoses varied, each criterion was statistically significant (P-value ,0.05) in the final model for both countries validating the choice of appropriate criteria for assessing disease prioritization and highlighting their varying degree of importance in the overall decision.
The part-worth utility values (b) indicate the relative influence each level had on respondent choices with higher values representing a stronger degree of influence on choice (Tables 1,  2, 3). The part-worth utility value trends for the ten disease criteria with significant difference in importance score between countries (Table 7) can be broadly summarized as follows: Both groups considered case-fatality in humans and incidence of the disease in the last five years in humans to be the most influential criteria in the decision to prioritize zoonoses (Table 7). Although the US group was influenced more strongly by these criteria (P-value ,0.001 for both, Table 7), expressed by a wider range in partworth utility values between the lowest and highest levels (Tables 1,  2, 3), there was agreement between countries on the importance placed on the levels within these criteria. The same trend was observed for the criterion incidence in the last five years in animals.
The Canadian group was more strongly influenced by casefatality in animals, disease trend in the last five years in humans, transmission potential between humans, economic and social burden on trade in animals and high risk groups in humans (Pvalue ,0.01 for all, Table 7), nonetheless, there was agreement between countries on the levels of least importance (lowest partworth utility values) and levels of highest importance (highest partworth utility values) with incremental increases in part-worth utility values for the levels in between (Tables 1, 2, 3).
The criteria control measures in humans and control measures in animal did not exhibit this sequential pattern in level preference in either country, however, these criteria were statistically significant (Pvalue ,0.05) in the final models. Although preference was given to 'highly effective in reducing disease burden' (highest part-worth utility values) in both humans and animals and in both countries, there was no clear order of preference in the remaining levels (Tables 1, 2, 3). Both groups expressed next preference for the level 'not effective at all in reducing disease burden' in humans (second highest part-worth utility values). While preferences appear contradictory, this is due to the difference in preferences between individuals who choose to prioritize when the opportunity to control is attainable (highly effective control measures) and those who choose to prioritize when the opportunity to control is not viable (no control measures).
Although the strength of preference in disease criteria importance scores differed between countries, there was general agreement in the contribution of disease criteria in the decision to prioritize zoonoses (Table 7). However, marginal differences in the part-worth utility values within disease criteria (Tables 1, 2, 3) and in disease incidence and trend contributed to a unique disease priority list by country (Table 8).

Disease Priority Lists
The final ranking of diseases by their overall CA-derived scores is presented in Table 8. The higher the score, the higher the ranking on the priority list. The range in the overall scores by diseases differed between Canada and the US and correlates with the part-worth utility values derived by country (Tables 1, 2, 3). As part-worth utility values are interval data, the overall scores cannot be directly compared both within and between countries [24]. We can, however, compare the difference in disease ranking as an overall measure of proximity of diseases both within and between countries. Although differences in disease ranking were observed between countries, the majority of diseases (77%) were within ten ranked positions of each other indicating a general consensus in ranks between countries. Diseases of high priority generally exhibited high incidence, high case-fatality, severe symptoms, prolonged duration of illness, high transmission potential and are increasing or emerging in humans and animals, though it was not necessary to exhibit all of these characteristics to be prioritized (e.g., Nipah virus encephalitis, Ebola and Marburg do not occur naturally in North America while H1N1 Influenza has a low casefatality rate). Diseases of low priority generally comprised rare diseases or diseases with a large proportion of asymptomatic cases, low case-fatality, short duration of illness, mild symptoms, stable disease trend and low transmission potential in humans and animals.
Canadians considered Giardiasis, Salmonellosis, Lassa fever, Cryptosporidiosis, Toxoplasmosis and Botulism of higher priority than Americans. Conversely, Americans considered Babesiosis, Anaplasmosis, Paralytic shellfish poisoning, Coccidioidomycosis, Cysticercosis, Hantavirus pulmonary syndrome, Rocky Mountain spotted fever, Typhus, and the plague of higher priority than Canadians. This was due to regional differences in disease incidence (Anaplasmosis, Rocky Mountain spotted fever and Babesiosis do not occur naturally in Canada but are endemic in the US), differences in disease trend (Paralytic shellfish poisoning has been increasing in the US but is stable in Canada) or differences in the part-worth utility values within disease criteria by country (Tables 1, 2, 3).

Discussion
Disease prioritization has engaged the interest of numerous research groups recently [4,6,12,13,[15][16][17][18]. As zoonotic outbreaks are a significant public health burden in North America [25][26][27], and with decreasing resources available for their prevention and control, a strategic framework for the prioritization of zoonoses would be valuable. We present on the novel use of a quantitative methodology, CA, for the prioritization of zoonoses in North America. There are a number of unique features in our study that overcome constraints in traditional prioritization methodologies. Our method fits a statistical model to a robust experimental design to generate relative weighted scores for disease criteria and their levels thus overcoming the limitation of a subjective weighting process for disease criteria by expert panels [4,6,13,16,18] or an arbitrary scoring system for levels [4,6,[8][9][10][11][12][13]16,18]. Because CA generates weighted scores by assessing disease combinations rather than disease criterion separately, our scores are relative weighted scores that do not assume disease criteria are independent or of equal importance [4,6,[8][9][10][11][12][13]16,18]. The study explored preference for disease criteria, rather than actual diseases; thus, individuals were forced to prioritize on the basis of scientific information, eliminating biases associated with disease names. We elicited preferences by choice rather than by ranking or rating diseases allowing for a more intuitive approach to expressing preferences, particularly when choices are similar [17].These features of the CA trade-off methodology brings additional value to the disease prioritization process compared to traditional methods.
Perhaps the most distinct feature of this study was the ability to engage wide social participation from the public. This has not been previously conducted and understanding the perception of the public may offer healthcare professionals the opportunity to improve public education and risk communication. Our CA (H) = human-related characteristic, for example, case-fatality in humans.

2
(A) = animal-related characteristic, for example, case-fatality in animals. 3 Relative rank of disease criteria by importance scores; presented in order of importance to Canadian participants. 4 Mean importance score across respondents. 5 Standard deviation of importance scores across respondents. 6 t-statistic; d.  models indicate individuals with no prior knowledge or experience in prioritizing zoonoses were capable of producing meaningful results with a satisfactory model fit. Part-worth utility values exhibited face validity with a stronger preference for salient disease criteria levels than non-salient alternatives. The degree to which each disease criterion contributed to the overall decision to prioritize also demonstrated face validity and are consistent with findings from similar studies [4,6,13,[16][17][18]. The priority lists derived from applying part-worth utility values from the CA models to actual diseases produced a list of diseases that were reasonable, from a public perspective, for prioritization. More similarities than differences were observed in the strength of preference in disease criteria importance scores between countries suggesting general agreement in disease prioritization between Canadians and Americans. There are some limitations associated with this study. The response rate was low given the length and technical content of the survey, we also used a pre-screened panel to recruit participants; the study populations were therefore highly educated compared to the national populations and the results may not be truly representative of the general public. However, public education and risk communication has the highest impact on those who are educated and actively involved in the community, thus, our study populations likely represent the target population for public health intervention. There was a marked difference in the response rate between the Canadians and Americans, although the cause for this difference is unknown as recruitment strategies and survey design were identical between countries. It is important to note that while response rates differed dramatically, the study populations did not deviate considerably from their respective national populations or from each other in terms of gender, age and education. The difference in response rate is therefore unlikely to have resulted in substantial differences in the results between countries. Our percent certainty model fit was 79.4% for both countries; an opportunity therefore exists to improve the models further to estimate more precise part-worth utility values. Marginal changes in part-worth utility values may have a substantial impact on the relative ranking of diseases on the priority list; thus, our current priority lists are only acceptable if we accept the current model fit. Nonetheless, this is the first disease prioritization study to validate disease criteria scores against respondent choices and may serve as a standard for prioritization methodologies. We also acknowledge that not all of the 59 disease criteria identified from focus groups was considered in this study [7], the inclusion of additional criteria would have an impact on the relative ranking of diseases. The current priority lists are therefore only applicable to the 21 disease criteria in the study, in spite of this; we believe these criteria capture some of the most relatively important aspects for priority setting.
Finally, there are multiple objectives for prioritizing zoonoses (e.g. prioritizing for research, regulation, control, prevention, management, vaccination, diagnosis, cost-effective and surveillance), although we specified to participants to prioritize ''for policy implementation for the control and prevention of zoonoses'', they may have prioritized with another objective in mind. There is no way to measure this type of error and we assume that participants were consistent in their objective to prioritize. Further, we asked participants ''which of the following diseases should be prioritized'' with each choice task set, which may be ambiguous in the face of multiple competing diseases. Although we did state in the instructions (not presented) to ''select one of the five disease profiles that characterizes a disease that, in your opinion, should be prioritized above the others'', we could have reinforced this by explicitly specifying, ''which of the following diseases should receive the highest priority'' with each choice task set. We assume that participants understood the research question and were consistent in their interpretation.
In conclusion, we describe the first zoonotic disease prioritization exercise, involving public participation, in North America. Given the established quantitative approach, robust experimental design, satisfactory model fit and reasonable disease criteria level scores [24,34], we have demonstrated CA as a potential tool for the prioritization of zoonoses. Periodical updating of criteria levels to match current disease trends will allow for the revision of disease priority lists that will reflect the most current state. Finally, the application of this methodology is versatile and not limited to zoonoses or North America; thus, other prioritization exercises may consider this approach with different criteria inputs.