Attribution of country level foodborne disease to food group and food types in three African countries: Conclusions from a structured expert judgment study

Background According to the World Health Organization, 600 million cases of foodborne disease occurred in 2010. To inform risk management strategies aimed at reducing this burden, attribution to specific foods is necessary. Objective We present attribution estimates for foodborne pathogens (Campylobacter spp., enterotoxigenic Escherichia coli (ETEC), Shiga-toxin producing E. coli, nontyphoidal Salmonella enterica, Cryptosporidium spp., Brucella spp., and Mycobacterium bovis) in three African countries (Burkina Faso, Ethiopia, Rwanda) to support risk assessment and cost-benefit analysis in three projects aimed at increasing safety of beef, dairy, poultry meat and vegetables in these countries. Methods We used the same methodology as the World Health Organization, i.e., Structured Expert Judgment according to Cooke’s Classical Model, using three different panels for the three countries. Experts were interviewed remotely and completed calibration questions during the interview without access to any resources. They then completed target questions after the interview, using resources as considered necessary. Expert data were validated using two objective measures, calibration score or statistical accuracy, and information score. Performance-based weights were derived from the two measures to aggregate experts’ distributions into a so-called decision maker. The analysis was made using Excalibur software, and resulting distributions were normalized using Monte Carlo simulation. Results Individual experts’ uncertainty assessments resulted in modest statistical accuracy and high information scores, suggesting overconfident assessments. Nevertheless, the optimized item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection. Conclusions Cooke’s Classical Model with performance-based weighting provided robust uncertainty estimates of the attribution of foodborne disease in three African countries. Attribution estimates will be combined with country-level estimates of the burden of foodborne disease to inform decision making by national authorities.

item-weighted decision maker was statistically accurate and informative. While there is no evidence that animal pathogenic ETEC strains are infectious to humans, a sizeable proportion of ETEC illness was attributed to animal source foods as experts considered contamination of food products by infected food handlers can occur at any step in the food chain. For all pathogens, a major share of the burden was attributed to food groups of interest. Within food groups, the highest attribution was to products consumed raw, but processed products were also considered important sources of infection.

Introduction
Infections of humans by enteric pathogens can occur through various transmission routes including food, water, air, soil, human-to-human contact, and animal-to-human contact. Due to these various pathways, burden estimates of foodborne disease are challenging. The Foodborne Disease Burden Epidemiology Reference Group (FERG) was established by the World Health Organization (WHO) in 2007 with the aim to develop the first worldwide estimates of the burden of foodborne disease on a sub-regional level. Subregions were based on the official grouping of WHO Member States. FERG further subdivided each region into subregions based on child and adult mortality as described by [1]: very low child and adult mortality (stratum A), low child mortality and very low adult mortality (stratum B), low child mortality and high adult mortality (stratum C), high child and adult mortality (stratum D), and high child mortality and very high adult mortality (stratum E). Burkina Faso was assigned to the Africa Region (AFR), stratum D (AFRD) and Ethiopia and Rwanda to AFR, stratum E (AFRE). FERG has used Structured Expert Judgment according to Cooke's Classical Model [2] to quantify the relative impact of foodborne disease compared to other transmission routes [3]. Factors considered by the experts included the epidemiology of the foodborne hazards (microorganism or chemical) causing the disease, seasonality, patterns of food consumption, geographic regions, as well as the ecology of the hazards. According to the World Health Organization [4], 600 million cases of foodborne disease were estimated to have occurred globally in 2010. Of these, enteric pathogens accounted for 550 million cases. The African sub-regions D and E were observed to have the highest burden of foodborne disease among all sub-regions analyzed: 13,000 and 12,000 Disability Adjusted Life Years (DALYs) per 100,000 population, respectively. Global attribution estimates of foodborne disease to specific food groups were first published in 2017 for a limited number of zoonotic pathogens [5]. Such estimates are not yet available for many pathogens that have human reservoirs, such as Shigella spp. and Escherichia coli pathotypes. Building on these estimates, Li et al., estimated the global and sub-regional burden of pathogens in animal source foods [6]. Estimates of disease burden attributed to specific food groups, food types and food products are not yet available at country level.

PLOS NEGLECTED TROPICAL DISEASES
We present a study on food attribution to support the objectives of three projects aiming to improve food safety in three African countries: Hazards (in this context defined as a biological agent which may cause illness in humans) were selected for each country separately based on the goals for the individual projects as shown in Table 1. The selection of foods and hazards to be included was made jointly by project partners while designing the studies and was informed by the results from the global burden of foodborne disease study [4] and specific concerns in the target countries, as voiced by local partners. We designed a hierarchical scheme to classify foods at three levels. The first level was based on FERG and included the food groups "Beef, Ruminants' meat, Dairy, Poultry, Vegetables, Fruits and nuts, Grains and beans, Oils and sugars, and Other foods". Guided by the goals of the projects and input from experts in each country, food groups were subdivided in food types. For example, food types in the Dairy category included "Milk from cattle and Milk from other species" while food types in the Vegetables category included "Tomatoes, Leafy greens, Cabbage, Green pepper, Onions, and Other vegetables". Only food types of interest for the three projects were further subdivided in food products, representing specific consumer products. These products were defined on a country basis as necessary. For example, in Ethiopia, the Milk from cattle food type included the food products "Raw, Fermented traditional (ergo), Cottage cheese traditional (ayib), Heat treated, and Other products" while in Rwanda, this food type included the food products "Raw, Fermented traditional, Fermented industrial, Heat treated and Other products". Within each category, items were defined to be mutually exclusive and exhaustive (i.e., attribution estimates should sum to 100%). Experts were asked to provide conditional assessments for food group domains. For example, of all cases of foodborne disease due to Campylobacter, what percentage of these cases is attributable to consumption of Red meat? And for all cases attributed to Red meat, what proportion is attributable to Raw meat? For enterotoxigenic Escherichia coli (ETEC), we also elicited attribution to food groups while FERG results [5] were used for other hazards.

Point of attribution
The point of attribution for food groups, food types and food products were based on FERG who have defined this as "the point where the hazards entered the place where the foods are prepared for final consumption" [3]. For example, a person may become ill from eating a fresh green salad that was contaminated by a knife previously used to cut raw chicken. If the chicken had been contaminated with pathogens prior to entering the food preparation area, we would attribute the illness to the chicken, not the lettuce.

Cooke's classical model
The Classical Model for Structured Expert Judgment is a method to elicit, validate and aggregate expert opinion for uncertainty quantification [2,13]. The experts are asked to quantify their uncertainty about quantities of interest, by providing subjective assessments for percentiles of the distributions. Usually the 5 th , 50 th and the 95 th percentiles, denoting the best estimate (50 th percentile), along with a credible interval, provided by the 5 th and 95 th percentiles. The validation is enabled by calibration questions, whose realizations are not known to experts but are known to analysts. The objective evaluation of experts' assessments is enabled by a calibration and an information score, which are derived from the calibration questions. The calibration score measures the statistical accuracy of expert's assessments, that is, relative to expert's assessments, how discrepant the distribution of the realizations is to the expected relative frequencies of the realizations. We expect 5% of the realizations to fall below expert's 5 th percentiles and 5% of the realizations to fall above the 95 th percentiles, and 90% of the realizations to fall between the 5 th and the 95 th percentiles, with 45% above and 45% below the median. The information score denotes how concentrated expert's assessments are with respect to a background measure. A combined score, obtained by multiplying the calibration score with the information score is used to evaluate the overall performance of experts and it is also used to compute performance-based weights. Experts' three elicited quantiles are used to construct so-called minimum informative distributions, that is, distributions which don't assume any parametric form, nor other information apart from the three quantiles. The Classical Model enables the weighted combination of these distributions using various weights, which result in corresponding Decision Makers (DMs). Performance-based weights which result from experts' normalized combined score lead to a Performance-Based Decision Maker. Experts' distributions for the calibration questions can be aggregated as well, the same performance-based weights and therefore the Performance-Based Decision Maker can be evaluated using the calibration and information score, just as for any expert. Various performance-based weights can be considered in aggregating distributions, such as global weights, which are computed using all calibration questions, and item weights, which are derived for each question. Alternatively, optimized Decision Makers search for subsets of experts whose aggregated distributions lead to the best performing Performance-Based Decision Makers in terms of both the calibration and information score. Finally, equal weights can also be considered, and the performance of all resulting Decision Makers can be compared using their calibration and information score. We note that unlike data-driven models, which rely on large number of observations, expert judgment methods typically employ, due to practical constraints, a limited number of experts. The Classical Model accepts a minimum number of 4 experts and advises for at least 6 experts to participate in a study [13]. A robustness analysis provides insights into how robust DMs performance is with respect to each expert or calibration question.

Expert identification, invitation, selection, and panels
Identification of experts was carried out by the study team for each project within each country. In total, 31 experts were identified in Ethiopia, 27 in Burkina Faso and 23 in Rwanda. Of these, 11, 12 and 9 participated in the study (Fig 2). The identified experts were sent a formal letter of invitation by email to participate in the study and, if interested, complete a Qualtrics (Qualtrics, Provo, UT) survey using a link provided in the invitation letter. The survey consisted of questions pertaining to their working knowledge and experience.
Experts with appropriate domain knowledge were selected by project principal investigators and project managers based upon the survey. Here, appropriate knowledge was defined as expertise in one or more areas of diarrheal disease in humans, zoonoses, microbial food safety, water and sanitation or veterinary public health. Selected experts were sent an email informing them of their selection into the study for their appropriate country, along with biographies of

PLOS NEGLECTED TROPICAL DISEASES
Food attribution Africa the interviewers. Selected experts were asked to return a signed consent form and were also asked to recommend colleagues not yet invited as potential participants. In addition, experts who served in one of the three studies, and who had working knowledge in one or more other study countries, were invited to the subsequent studies where appropriate.
Of the 31 identified experts in Ethiopia, 18 submitted detailed background information, 17 were selected and enrolled by consent. Of these, 6 experts withdrew consent due to time conflicts or due to COVID-19 when invited for an interview. In total, 11 experts provided estimates in Ethiopia. For Burkina Faso 27 experts were identified, 14 experts submitted detailed background information, 9 experts were selected and enrolled by consent, and 3 experts carried over from the Ethiopia study. Overall, 12 experts provided estimates in Burkina Faso. For the Rwanda study, 23 experts were identified, 13 submitted detailed background information, 7 were selected and enrolled by consent and 2 experts carried over from the Ethiopia study. For Rwanda, 9 experts provided estimates (Fig 2). Experts were not remunerated for their inputs.
Expert's education is summarized in Table 2. In all countries, veterinary medicine (including veterinary public health) was the most frequent education, followed by biological sciences (including microbiology) in Burkina Faso and Rwanda but not in Ethiopia. Other backgrounds included agricultural sciences, environmental and food sciences, medical doctor, and public health (including epidemiology). All experts in Burkina Faso held PhD degrees, in Ethiopia there were 8 PhDs and in Rwanda 7. In all countries, most experts were affiliated with universities (6,8,7 in Burkina Faso, Ethiopia and Rwanda, respectively), followed by research and development institutes (6, 2 and 2). In Ethiopia, 1 expert worked for a government institute.

Expert training
A background document was prepared for each country separately and provided in advance to the experts. This document included a description of the concepts of uncertainty, variability, and probabilities, as well as an explanation of the calibration and target questions, a definition of the point of attribution, published background information and estimates on the epidemiology of the pathogens of interest. In addition, the background document detailed the procedure such as interview duration, agenda items, and an explanation on how to use Zoom software to participate in the interview and how to complete the survey instrument. All experts completed a training video on providing quantitative estimates under uncertainty (Training video for expert elicitation-YouTube).

Expert interviews
Interviews were held via the Zoom platform and had a duration of 60-90 minutes. Interviews covered welcome and introduction to the study and an interactive discussion covering the concept of uncertainty. Subsequently, experts provided assessments for the calibration questions during the interview without access to any resources. Then, interviewers walked through an example target question with the experts. After the interview, experts were allotted two weeks to research and complete answers to the target questions, which were returned by email to the interviewer. Experts were encouraged to use resources as they considered necessary while evaluating target questions.

Calibration questions
All panels included the same series of 10 calibration questions that covered three general themes regarding the continent of Africa. These questions were chosen to reflect the broad domains of the study, i.e., infectious diseases and food and referred to data from Africa, but not from any of the countries to which the study applied. We did not the experts to know any of the answers exactly, but they should be able to produce reasonable estimates based on their general background knowledge. If they were less knowledgeable in a specific domain, we expected them to acknowledge this by providing wider uncertainty intervals. Interviewers presented the questions during the interview and requested answers based solely on current expert domain knowledge. All questions were completed during the interview and answers were sent to the interviewer by email before the interview concluded. Table 3 provides some example questions, while S1 Table provides the full set of calibration questions.
If an expert participated in more than one study, their calibration assessments were re-used for all subsequent studies, reducing the burden to experts for participating in more than one study.

Target questions
Target questions consisted of combinations of hazards with food groups and food types for attribution by country. For each combination, experts were asked to attribute the incidence of foodborne disease in a typical year. Target questions differed between countries, reflecting the food-hazard combinations detailed in Table 1. Estimates were collected by an Excel spreadsheet where each individual tab represented a specific pathogen in one country. An example set of questions and the Excel instrument used to collect the data is shown in Fig 3. The instrument included two quality checks to support the experts to provide valid estimates. For each food group, type or product, the expected sum of medians was provided. The experts were instructed that the sum of the medians should be close to, but not necessarily exactly 100%. An indicator ( ���� ) was shown next to each set of uncertainty estimates. The indicator would disappear if all three estimates for a target question were completed and 5 th percentile < 50 th percentile < 95 th percentile. Strictly increasing percentiles are a theoretical requirement of the Classical Model when eliciting continuous distributions. All experts were allotted two weeks to return their target estimates, but exceptions were granted as necessary.

Data analysis
Individual expert assessments were imported into Excalibur software [14] for each country separately and evaluated using their calibration and information score. The theoretical background is described in [13]. Experts' assessments were combined using performance-based weights, both optimized and non-optimized, along with equal weights. The resulting Decision Makers' performance was evaluated and the Decision Maker with the highest combined score selected. We performed an expert and item-wise robustness analysis on the Ethiopia expert results. For expert-wise robustness analysis, each individual expert was removed from the pool of experts and the impact on the resulting item weighted optimized Decision Maker was assessed. Similarly, each calibration question (item) was removed, and the item re-weighted optimized Decision Maker's performance were evaluated.
The item weighted optimized Decision Maker's distributions were used to obtain uncertainty estimates for the attribution of foodborne disease in the three countries. The attribution in food groups is based on FERG, while attribution to food type and food products is based on expert judgment elicited for these three studies specifically.
100 percentiles from the Decision Maker's uncertainty distribution were exported from Excalibur for each target question in each of the three countries. These percentiles were

PLOS NEGLECTED TROPICAL DISEASES
Food attribution Africa imported in R software version 4.1.0 [15] and 10,000 observations were jointly sampled from each target variable whose distribution is characterized by these percentiles. A normalization procedure was applied to tuples of jointly selected observations. For example, each tuple with joint samples of food groups for a given hazard was normalized to sum to 100%. The procedure ensured that, as appropriate, the resulting sample mean attribution estimates for all hazards in all food groups, types and products summed up to 100% and provided 10,000 normalized samples of the uncertainty distribution. Normalized samples were thus obtained for each corresponding target question. While the estimates for food types or food products were elicited conditional on corresponding food groups or food types, the mean and 95% credible interval of unconditional attribution estimates were calculated for further combination with foodborne disease burden estimates.
Treemaps were constructed using the treemapify package in R [16] to provide a visual summary of the mean attribution results per hazard and per country. For completeness of the treemaps, mean food group attribution results from FERG [5] were included as appropriate.

Results
The calibration and information scores of the item weights optimized Decision Maker are plotted along with individual expert's scores in  in all three countries (Fig 4). No single expert met the threshold of a calibration score greater than 0.05, while information scores were quite high in the range of 1 to 3.5. The information score of experts in Burkina Faso was on average lower than in the other two countries. The Optimized Item Weighted Decision Maker performed best in all three countries. In all countries Decision Maker's outperformed individual experts in terms of calibration score, which did not occur at a loss of much information. In Ethiopia, the Decision Maker's calibration score was 0.7, displaying a significant improvement in statistical accuracy, with an information score of 1.8. For Burkina Faso, a calibration score of 0.4 and an information score of 1.1 were obtained. For Rwanda a calibration score of 0.7 and an information score of 2.0 were obtained. All Decision Makers were exceeding the threshold of 0.05. Having information scores higher than 1 implies that the Decision Makers were also informative relative to the uniform background.
Robustness analysis per expert and question revealed that the results were robust with respect to the pool of experts in each country and with respect to the calibration questions. Removing individual experts or questions led to only slight changes in the performance of the Decision Maker's.
Tables 4-8 present the attribution estimates per country. About ¾ of the foodborne burden by ETEC in Ethiopia was attributed to the food groups of interest (beef, dairy, poultry, and vegetables) ( Table 4). In Burkina Faso, about 40% of the foodborne burden of ETEC was attributed to the food groups of interest (poultry and vegetables) ( Table 6). Attribution of ETEC to dairy was lower in Burkina Faso (13%, Table 5) than in Ethiopia (24%, Table 4).
In Ethiopia, about 70 percent of the burden of Campylobacter, Salmonella and Shiga-toxin producing Escherichia coli (STEC) from dairy was attributed to milk from cattle (Table 5) and about ⅓ of the burden of dairy was attributed to consumption of raw cattle milk. About 60% of the burden of these pathogens in beef was attributed to red meat and about 30% to beef consumed raw. For poultry, 100% of the burden was attributed to chicken meat by the study team because meat from other poultry species is consumed very little, if at all. Of the burden of Campylobacter and Salmonella from poultry meat, 56% was attributed to chicken bought raw and prepared at home. A higher fraction of the burden of Salmonella was attributed to tomatoes than of ETEC (0.32, 0.06 respectively) which translated to a higher burden attributed to raw tomatoes (0.22, 0.04 respectively).
The attribution of the burden from vegetables to Salmonella and ETEC in Burkina Faso was similar to Ethiopia (Table 7). However, about half of the burden of Campylobacter and Salmonella from poultry meat was attributed to chicken meat consumed out of the home. For Rwanda, most of the burden of dairy for all pathogens was attributed to cattle milk (~90 percent, Table 8). Note that the burden of M. bovis was attributed 100% to dairy by the study team as this pathogen is considered to be exclusively transmitted through milk [3,5]. The attribution to raw milk varied by pathogen with the highest proportions attributed for Brucella and Cryptosporidium (61% and 63%, respectively) and the lowest for Campylobacter (0.33). A substantial proportion of the burden from dairy was also attributed to traditionally fermented cattle milk products (~20%, Table 8) and even industrially fermented products (0.06-0.20).
More detailed data including those used to create figures are provided in S2 Table.   Tree maps (Figs 5-7) graphically show the results of the experts' assessments, combined with food group attribution from FERG [5] or this study (for ETEC). Treemaps display hierarchical data as a set of nested rectangles, where the surface area is subdivided according to the proportion of disease attributed to all food groups, food types and food products included in the study. In our case, the surface area of each treemap represents all foodborne disease in a country by a hazard. The surface area is first subdivided according to the attribution to food groups. Food groups are color coded as indicated in the legend to the right of the figure. For example, 68% of all cases of illness by Brucella spp. in Rwanda were attributed to dairy (light goldenrod). Second, the surface area of dairy is further subdivided to represent the contribution of different dairy types, represented by italic font. For example, of the 68% Brucella spp. cases in Rwanda attributed to dairy, 63% were attributed to milk from cattle. Third, the surface area of milk from cattle is subdivided in five dairy products, represented by regular font. Of the 63% of Brucella spp. cases in Rwanda attributed to milk from cattle, 39% were attributed to raw milk consumption.

Discussion
Individual experts in the three countries did not provide statistically accurate assessments, as all calibration scores were lower than the significance level of 0.05. In turn, their assessments were very informative, with half of the experts' information scores being higher than 2. This suggests that experts in the study were overconfident-they provided narrow uncertainty margins for the calibration questions which did not include the realizations at the expected relative frequency. Despite having a relatively low number of experts providing assessments, the item- weighted optimized Decision Maker's in all three countries were statistically accurate. Moreover, the significantly improved statistical accuracy did not come at the cost of low informativeness as all Decision Maker's information scores were above 1. These results show the power of performance-based weighting and suggest that the range of estimates provided by the experts as a group did represent the realizations to the calibration questions at the expected relative frequencies.
Even though dedicated training sessions were offered to the experts before the interviews, these apparently were not sufficient to prevent overconfident assessments. These results are in contrast with an earlier study in the US, where a large proportion of the experts individually produced well calibrated assessments [17]. An important difference between these two studies was that the US study was performed as an in-person meeting, while the current study was conducted through remote communication. Nonetheless, it needs to be emphasized that empirical evidence suggest that a large proportion of experts do not provide well calibrated assessments. An analysis on 322 experts [13] showed that only 27% of the experts' assessments in 74 studies were calibrated, using a 0.05 threshold. By contrast, all experts in the three studies presented in this paper provided assessments leading to calibration scores smaller than 0.05. Finally, we emphasize that experts' ability to provide valid assessments under uncertainty does not appear to correlate with years of experience or domain knowledge [18]. It is notable that the uncertainty in some attribution estimates is rather small, for example attribution of ETEC transmission to grains and beans in Ethiopia, or Campylobacter transmission to cooked red meat, also in Ethiopia. For these products, experts largely agreed on the attribution estimates. The uncertainty in other attribution estimates was higher, independent of a relatively low or high mean attribution estimate. For example, the expected attribution of milk from cattle due to Campylobacter was relatively high at 70%, and the 95% uncertainty interval ranged between 6% and 100%. On the other hand, the attribution estimates for offal due to Salmonella were rather low (36%), with an attendant large uncertainty interval ranging between 3% and 92%.
Attribution results for ETEC to food groups in Ethiopia and Burkina Faso were largely similar, even though only 2 experts provided estimates for both countries. Experts were explicitly instructed that there is no evidence that animal pathogenic ETEC strains are infectious to humans [19], but nevertheless in both countries, a sizeable proportion of ETEC illness was attributed to animal source foods. We asked the experts for their rationale behind this attribution, and they considered contamination of food products by infected food handlers can occur at any step in the food chain and may contribute to transmission of pathogens with humanonly reservoirs by animal source foods. Likewise, attribution results for poultry meat and vegetables that were done in both countries were largely similar. A noteworthy exception is a higher attribution of poultry meat related illness to chicken meat consumed out of the home in Burkina Faso (~50%) vs. Ethiopia (~20%). This corresponds to a difference in the contribution of street food to chicken consumption in these countries [20,21].
To support decision making by national authorities, the attribution estimates presented in this study will be combined with national level estimates of the burden of foodborne disease from WHO FERG to present estimates of the burden of specific food groups and food products [10,11]. These data will inform priority setting of foods for control activities and can be combined with risk assessment and economic estimates to inform cost-benefit analyses. Supporting information S1