Recent public health concerns regarding commercial food products have increased the need to develop an automated method to detect food product-related health events. We developed and verified a method for the early detection of potentially harmful events caused by commercial food products. We collected data from daily internet-based questionnaires examining the presence or absence of symptoms and information about food purchased by the respondents. Using these data, we developed a method to detect possible health concerns regarding commercialized food products. To achieve this, we combined the signal detection method used in the reporting system of adverse effects of pharmaceutical products and the Early Aberration Reporting System (EARS) used by the United States Centers for Disease Control. Whiteleg shrimp (Litopenaeus vannamei), which had odds ratio and Odds(−) of 8.99 and 4.13, respectively, was identified as a possible causative food product for diarrhea and vomiting. In conclusion, this study demonstrated that food distributors can implement post-marketing monitoring of the safety of food products purchased via the internet.
Citation: Maeyashiki A, Akahane M, Sugiura H, Ohkusa Y, Okabe N, Imamura T (2016) Development and Application of an Alert System to Detect Cases of Food Poisoning in Japan. PLoS ONE 11(5): e0156395. https://doi.org/10.1371/journal.pone.0156395
Editor: Ja-an Annie Ho, National Taiwan University, TAIWAN
Received: November 26, 2015; Accepted: May 15, 2016; Published: May 27, 2016
Copyright: © 2016 Maeyashiki et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data is available at Dryad, DOI:10.5061/dryad.5pt57 (http://datadryad.org/review?doi=doi:10.5061/dryad.5pt57).
Funding: This study was supported by a Health Labour Sciences Research Grant. The author who received the funding is Tomoaki Imamura. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Hygienic management of food products is particularly important to humans. In the United States (US), the National Strategy for Biosurveillance was implemented in 2012 to enable the early detection of health concerns related to food . The Foodborne Diseases Active Surveillance Network (known as FoodNet), an emerging infection program, has implemented active epidemiological surveys to detect emerging infections; this has been done in collaboration with the US Centers for Disease Control and Prevention (CDC) .
To detect and manage food poisoning at an early stage, it is necessary to monitor the causative pathogen and the contaminated food. The incidence and distribution of food poisoning cases have increased in recent years because foods produced in one location are now sold in larger areas because of the development of efficient distribution systems. For example, in 2008 in Japan, there were several cases of food poisoning caused by frozen dumplings that had been produced in China and were contaminated with methamidophos, an organophosphorus pesticide . In 2013, there was a case where a worker intentionally added malathion, an agricultural chemical, into frozen food during the production process. These cases raised considerable attention concerning the safety of foods consumed by the general public. For the early detection of health concerns of mass-produced foods, it is necessary to implement a crisis management system together with post-marketing monitoring (PMM) of food products.
Pharmaceutical products have much stricter controls than food products, and post-marketing surveillance (PMS) programs are routinely performed for pharmaceutical products. The Uppsala Monitoring Centre of the World Health Organization and regulatory authorities have developed pharmacovigilance databases for the early detection of potential adverse effects of pharmacological products . In the Netherlands, the Netherlands Pharmacovigilance Centre Lareb maintains a self-reporting system to record the adverse effects of pharmacological products. At Lareb, a reporting odds ratio (ROR) method, which was initially developed by Meyboom et al., has been further developed and used to monitor safety issues since 2000 [5–10].
PMS not only involves the early detection of adverse effects, but it also includes the safety management of vaccines by PMM  as well as the monitoring of erroneous use and abuse of drugs . In 2010 in Japan, Ooba et al. used Japanese self-reported data to investigate the effects of under-reporting in signal detection systems based on the ROR method .
Unlike food products, pharmacological products are individual products, and if an adverse effect occurs in Japan, the patient should notify the prescribing doctor, who should then notify the Pharmaceutical and Medical Devices Agency. Regarding food products, Hepburn et al. summarized the reports on PMS of new food products in 2008 .
Several factors hinder the ability to implement post-marketing health surveys of food products. In particular, the distribution routes of food products are complex. It is also difficult to identify who purchases individual food products. For these reasons, no methods have been developed to perform PMM of food products. To address this issue, we examined whether a method that is already used clinically to detect adverse effects of pharmacological products can also be applied to the early detection of potentially harmful events related to post-marketing food products.
We have developed the Web-based Daily Questionnaire for Health (WDQH), a syndrome surveillance/early detection system, in which a daily survey records physical symptoms (health information) directly from community residents . This system is useful for the detection of outbreaks of infections in the community and for performing health surveys of community residents in relation to environmental factors [16,17].
In the current study, we used a database that combines individual daily health information with a list of foods purchased by individual people. We developed a system that sends alerts to the users if specific symptoms are associated with specific foods. We also used this system to evaluate the practicability of a food PMM for the early detection of food-related issues.
This study was approved by the Ethical Committee of Nara Medical University (approval number: 220). Written informed consent for each participant was obtained at the registration screening.
The participants were customers and their families who purchased foods through the websites of Pal-system Tokyo Consumers’ Co-operative (Tokyo) and Osaka Izumi Co-operative Society (Osaka). The databases included 1002 households in Tokyo (3128 people, including family members) and 554 households in Osaka (1925 people, including family members).
Implementation of the health survey
Health surveys were conducted according to the method of WDQH . We focused on two of the health survey items, diarrhea and vomiting, because they are strongly associated with food poisoning. With support of the Japanese Consumers’ Co-operative Union (CO-OP), the surveys were implemented in two cities, Tokyo and Osaka, in Japan between January 20 and April 30, 2011.
Subjects who consented to participate in the survey were registered and agreed to provide the dates that food products were purchased during the survey period. For the health survey, which was accessed online using a personal password, the registered subjects received daily remainder e-mails to answer questions regarding the presence or absence of symptoms in themselves and/or their family members. All participants were given a reward, equivalent to JPY 500 (approximately US$5), at the end of the survey period (S1 and S2 Files).
Data on purchased foods were obtained from CO-OP every 2 weeks. As a baseline, we obtained data on foods purchased for 4 weeks before the start of the health surveys. The data included the member identity code (ID), date of the order, list of foods ordered, Japanese Article Number code, stock keeping unit, and number of items ordered. The data were anonymized in a similar manner to the personal information.
Health information and food purchase data were linked with the subject’s ID. In the data analyses, we needed to consider two limitations of the database. First, the database did not record which family member(s) ate a specific food. Thus, the survey was performed for the family unit, and if one family member reported a specific symptom on one day, the family was regarded as being a symptomatic household. Second, although the food delivery date was recorded, the date of consumption was not. To overcome this limitation, the period of food intake was defined as the time between delivery to the expiration date of the food product, or a maximum of 30 days for frozen food products.
All statistical analyses were conducted using Microsoft Excel and Microsoft Access, as described in the following sections.
Step 1: EARS screening.
We used the CDC EARS  to identify dates on which the number of reports of diarrhea and/or vomiting increased rapidly as possible cases of food poisoning. Unlike vomiting, there is a constant number of subjects with chronic symptomatic diarrhea that is independent of the consumed food products. The daily mean proportion of all respondents who reported each symptom was 2% for diarrhea and 1% for vomiting . Thus, it is possible that symptoms of food poisoning may have been overlooked for foods sold in small quantities.
To increase the sensitivity for the detection of diarrhea, the parent group of foods was divided into 20 subgroups, and the EARS was separately applied to vomiting and diarrhea groups as well as 20 diarrhea subgroups. For the 20 subgroups, we included the condition that ≥2.5% of households needed to report the symptom for detecting a possible association between the food subgroup and the symptom. We also set the condition that at least three households needed to purchase a food that was possibly and temporally related to an episode of food poisoning.
Step 2: Screening using the signal detection method.
We first sought to detect possible signals of food poisoning and symptoms by extracting data obtained in Step 1. The World Health Organization defines a signal as information that requires further investigation and is accompanied by a possible causal relationship between a pharmacological product and adverse events that is currently unknown or has been incompletely investigated . In this study, we replaced pharmacological products with food products in the signal detection process.
For signal detection, we prepared a 2 × 2 contingency table, with food products in the rows and the absence/presence of symptoms in the columns of the table (Table 1). To calculate the frequency of events, the date of signal detection by EARS was designated as the starting point, and data were collected in 1-week units for the previous 4 weeks. If the suspected food was consumed on ≥1 day(s) in a given week, that week was classified as positive exposure to the food product. If a specific symptom was reported on ≥1 day in a given week, that week was regarded as symptomatically positive.
We used the ROR method, which was introduced by the Netherlands Pharmacovigilance Centre Lareb [6–10] for signal detection. The ROR is an ordinary odds ratio (OR) and the lower limit value (threshold value or Odds(−)) of the 95% confident interval is used as the standard for signal detection [8,10,20,21]. The Odds(−) in PMS of food products was calculated using the equations presented below . Using these equations, we calculated the OR for cases with symptoms of food poisoning attributable to a specific food product (n11/n21) and for cases without symptoms after consuming other food products (n12/n22). We then calculated the ORs for cases without symptoms who consumed other food products and cases with symptoms who consumed the specific food product. Next, we calculated the Odds(−); Odds(−) >1 was considered a positive signal.
A value of n11 = 0 indicated no combination of a specific food product and symptom. A value of n22 = 0 indicated no combination between other food products and other symptoms, including the absence of symptoms. Those cases are excluded from analyses. A value of n12 = 0 indicated no combination of a specific food product and a specific reported symptom. A value of n21 = 0 indicated no combination between other food products and the reported symptoms. For these two events, the value 0.5 was added to each cell in the contingency table.
To screen for potential signals in Step 2, we used the ROR threshold value as the reference value. After identifying food products with Odds(−) with >1 and >3 reported cases (i.e., n11) , we ranked them according to Odds(−) values in descending order and selected those products with an Odds(−) value ranked in the top 10.
Step 3: Analysis of scatterplots.
After ranking the 22 groups (vomiting and diarrhea groups and 20 diarrhea subgroups) identified in Step 2, we prepared a scatterplot of the possible causative foods by displaying the OR values for the food products ranked in the top 10. The scatterplot arranges the occurrence of each symptom per household according to the timing of food consumption relative to the symptom in a chronological order. The scatterplot was used to confirm the family structure of the affected household, age, presence or absence of simultaneous symptoms in the household, presence or absence of simultaneous diarrhea/vomiting, and correlation between the date of the symptom and time of consumption. To examine the possibility of a causal relationship between the food product and the reported symptom (Table 2), an epidemiologist with expert knowledge verified whether the scatterplot indicated an outbreak of food poisoning, with consideration of the patients’ daily symptoms. If there were sudden occurrences of symptoms or if there were other patients with symptoms in the same time of period from other families who had the same meal as multiple members in the family, we implemented further investigation, considering food poisoning. We considered sustained symptoms as noninfectious digestive symptoms.
Responses to the health survey
In total, 56,340 responses were obtained in Tokyo (174,173, including family members) and 33,596 responses were obtained in Osaka (114,881, including family members) during the index period.
Purchase of foods
Overall, during the index period, 6212 different food products were purchased in Tokyo, and 5392 different food products were purchased in Osaka.
Step 1: Screening by EARS.
EARS was performed during the survey period for all the foods and the number of days on which diarrhea or vomiting were reported, and the results are shown in Table 3. The list of possible causative foods was refined by applying the signal detection threshold in Step 1 (Table 4). Possible causative foods were extracted, and 1.9–3.3% of the foods were associated with diarrhea, and 0.3–0.4% were associated with vomiting.
Step 2: Screening of potential signals.
The food products ranked in the top 10 using the Odds(−) determined in Step 1 are shown in Tables 5 and 6 for Tokyo and Osaka, respectively. Only alerts identified using the EARS on or after February 1 were included in the analysis.
In Tokyo, the three food products with the highest Odds(−) for diarrhea were noodles, grilled pork innards, and dried whitebait. Among the 20 subgroups of food products, seasoning for boiled rice was associated with diarrhea. The three food products with the highest Odds(−) for vomiting were yogurt, rice, and frozen fish-shaped pancakes stuffed with custard (Table 5).
In Osaka, the three food products with the highest Odds(−) for diarrhea were whiteleg shrimp, salted dried kelp, and tofu. Pasta sauce was also found to be associated with diarrhea as one of the 20 subgroups of food products. The three food products with the highest Odds(−) for vomiting were canned tuna, whole tomatoes, and tomato ketchup (Table 6).
Step 3: Inspection of the scatterplots.
Scatterplots showing the associations between the food products ranked in the top 10 using Odds(−) in Steps 1 and 2 were developed to visualize the timing of symptoms in each household. These scatterplots revealed that three symptomatic households consumed whiteleg shrimp (Litopenaeus vannamei) on the same day and reported multiple episodes of diarrhea within the household. Because the OR and Odds(−) values for whiteleg shrimp are very high (8.99 and 4.13, respectively), we considered that it had a possible causal relationship to food poisoning and hence issued an alert.
Considering this possible causative relationship, we tested this food product for bacterial contamination, and microbiological tests revealed the presence of Vibrio parahaemolyticus with a most probable number of 23/g. We also surveyed all households who purchased this food product during the survey period and confirmed that there were no additional episodes.
In this study, we conducted analyses of a large number of food products to establish and validate a PMM for food products, with the goal of early detection of health concerns (e.g., food poisoning) related to specific food products. To achieve this, we collaborated with two companies that sell food products via the internet to the general public and obtained data on the food products purchased by the participants.
In 2008, Hepburn et al. described a food PMM for artificial cholesterol (Olestra), artificial sweetener (aspartame), and StarLink maize (genetically modified food) . However, they only investigated some specific foods that were not linked to symptoms of food poisoning. Furthermore, although they proposed a method for collecting data on household food purchases, they only presented the purchase records. To our knowledge, there are no previous reports of prospective food PMMs in which researchers tried to link large numbers of food products with specific symptoms.
Because of the widespread food poisoning caused by frozen dumplings made in China in 2008, initiatives aimed at the early detection of health concerns of post-marketing food products and preventing such concerns are being implemented in Japan . In response to the food poisoning event, the Ministry of Health, Labour and Welfare established the Office of Foodborne Disease Surveillance. Its mandate is to conduct retrospective studies of information obtained from autonomous communities if food poisoning affects ≥ 50 people or if the cases are widely distributed. However, this approach only includes actual cases, and the Office of Foodborne Disease Surveillance cannot make comparisons with control groups. To address this issue, we performed more detailed analyses in subjects with or without daily symptoms as recorded using the WDQH ; moreover, we analyzed data regarding food products purchased by the subjects during the index period.
The EARS used in Step 1 is suitable for short-term symptom surveillance and does not require long-term data. In PMM of a single food product, Newkirk et al. conducted the EARS with data collected by the CDC (1990–2006) to determine a potential food defense index using food poisoning data for milk and verified this index in 2007 .
Using the results of the EARS and symptoms reported on a daily basis by the subjects, we were able to determine the dates on which the number of episodes of vomiting and diarrhea increased rapidly in association with a specific food product . In the next step, we selected all events occurring in at least three households who purchased the relevant food product. The frequency of possible causative food products to be associated with diarrhea was 1.9% and 3.3% and that of vomiting was 0.3% and 0.4%, in Osaka and Tokyo, respectively. There were a constant number of subjects with chronic symptoms of diarrhea .
We also conducted further analyses to refine the overall group of food products and found that it was necessary to identify food products that were associated with symptoms, even if they were purchased in small quantities. This suggests that less-frequently purchased foods may be overlooked by the EARS, and it is sometimes difficult to detect symptoms caused by a specific food product, considering the vast number of different food products purchased in the index period. Although the sensitivity of the signal is particularly important in EARS-based analyses, false-positive results are possible in analyses involving small numbers of subjects. Thus, it was necessary to combine the EARS with other methods. To address this issue, we classified the food products into 20 subgroups for the analyses.
Using the EARS, we were able to identify possible dates on which symptoms of food poisoning occurred with a high frequency of symptoms. We were able to execute the PMM for food products by refining the list of possible causative food products for food poisoning to ≤5% of all purchased food products.
In the signal detection step (i.e., Step 2), we constructed a 2 × 2 table. This included a control group of symptomless subjects who did not purchase the specific food products. Using the screening results obtained by applying the ROR signal detection standard [8,10,20,21], we searched for possible signals of causative food products for food poisoning that were retrieved in Step 1.
After ranking the food products according to their Odds(−) for vomiting/diarrhea (Tables 5 and 6), we identified which food products were associated with large numbers of symptomatic subjects among the households that purchased the specific food product compared with households that did not purchase the food product. The food product with the highest OR was whiteleg shrimp, which indicates that the method used for signal detection of causative food products for food poisoning is practicable.
In 2006, the Wisconsin State Laboratory of Hygiene conducted a comparative control study of 86 cases and 49 controls in a community affected by an outbreak of Escherichia coli 0157. They revealed that spinach was the causative food with an OR of 82.1 . That study used the PulseNet database, in which relevant organizations share information regarding food-borne infections to rapidly detect any food-related infections and determine the source. Although our system has similar aims, our approach also allowed us to perform follow-up surveys of affected cases.
After identifying food products potentially associated with food poisoning, Step 3 involved drawing scatterplots to visually investigate the relationship between the timing of events and determine whether a final alert should be issued. Statistical relationships do not necessarily indicate a causative relationship, and they may reflect some bias in the data and factors introduced by the investigator. Thus, before concluding that the observed signal is indicative of a causative relationship, the relationship must be considered in the context of our scientific knowledge and experience. Thus, a completely warranted medical evaluation helps achieve a scientific consensus [20,24].
In the present study, whiteleg shrimp had the highest odds for food poisoning, and we issued a final alert to raise awareness that it was a possible causative agent for food poisoning. We also obtained samples of the remaining food product to conduct microbiological tests for pathogenic bacteria and hence verify the causative relationship between the food product and symptoms of food poisoning.
To our knowledge, there have been no prospective PMM of food products in which the households that purchased specific food products were linked with the symptoms of food poisoning (i.e., diarrhea and vomiting) in a family member. Using this approach, we confirmed that PMM could be implemented for food products purchased from an internet-based distributor, and food products associated with an adverse outcome were identified using a three-step screening system.
Regarding the limitations of this study, it is possible that food products purchased from other retailers might be eaten simultaneously. Additionally, all symptoms recorded in the health surveys were self-reported, and any underlying diseases in the subjects might influence their reported symptoms.
To specifically evaluate the relationship between the possible causative food products detected in this study and health concerns, it will be necessary to conduct more detailed analyses of historical data, including the subjects’ individual health conditions, foods consumed, and seasonal eating habits; confirm the presence or absence of the purchase food claims by purchasers; and test food samples before shipment. Furthermore, to increase the methodological accuracy, we should consider implementing PMM, in which multivariable analyses are conducted where sex, age, consumption of the specific food product, and region are included as explanatory variables and the presence/absence of symptoms, as the dependent variable. The PMM should also include a system to alert for residual error within three standard deviations of the actual measured value. From a practical perspective, it will be necessary to improve the methodology to allow real-time alerting of possible outbreaks.
The present study demonstrated the feasibility of implementing PMS of food products purchased via the internet. We combined the EARS with an endemic detection algorithm originally developed by the US CDC with a signal detection method that is already used for the early detection of adverse effects of pharmacological products. We believe that the PMM method used in this study represents a quantitative and qualitative method for detecting potential health issues of food products. The analytical approach used in this study could also be applied to chemically related food poisoning, not just microbial food poisoning. Finally, we wish to highlight the importance of early detection of symptoms of food poisoning that are related to a specific food product to prevent widespread cases of food poisoning.
This study was supported by a Health Labour Sciences Research Grant. We wish to thank Mr. Onitake and Mr. Minematsu of the Japanese Consumers’ Co-operative Union for their cooperation in the collection of data. We would also like to express our gratitude to Mr. Hasegawa, Ms. Ushizima and Mr. Suzuki of Mitsubishi Research Institute, Inc., which performed data collection and analysis.
Conceived and designed the experiments: MA HS YO NO TI AM. Performed the experiments: MA HS TI AM. Analyzed the data: MA HS TI AM. Contributed reagents/materials/analysis tools: MA TI. Wrote the paper: MA HS TI AM.
- 1. Centers for Disease Control and Prevention. National Strategy for Biosurveillance. Available from: http://www.cdc.gov/surveillancepractice/reports/nbs.html.
- 2. Centers for Disease Control and Prevention. Foodborne Diseases Active Surveillance Network (FoodNet). Emerg Infect Dis 1997; 3: 581–583. pmid:9368789
- 3. Sumi Y, Oode Y, Tanaka H. Chinese dumpling scare hits Japan—a case of methamidophos food poisoning. J Toxicol Sci 2008; 33: 485–486. pmid:18827448
- 4. Lindquist M, Edwards IR, Fucik H, Nunes HM, Stahl M. From association to alert—A revised approach to international signal analysis. Pharmacoepidemiol Drug Safety 1999; 8: S15–S25.
- 5. Meyboom RHB, Egberts ACG, Edwards IR, Hekster YA, de Koning GHP, Gribnau FW. Principles of signal detection in pharmacovigilance. Drug Saf 1997; 16: 355–365. pmid:9241490
- 6. Van Puijenbroek EP, Egberts ACG, Meyboom RHB, Leufkens HGM. Signalling possible drug-drug interactions in a spontaneous reporting system: Delay of withdrawal bleeding during concomitant use of oral contraceptives and itraconazole. Br J Clin Pharmacol 1999; 47: 689–693. pmid:10383548
- 7. van Puijenbroek E, van Grootheest K, Diemont WL, Leufkens HGM. Determinants of signal selection in a spontaneous reporting system. Br J Clin Pharmacol 2001; 52: 579–586. pmid:11736867
- 8. van Puijenbroek E, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf 2002; 11: 3–10. pmid:11998548
- 9. van der Heijden PGM, van Puijenbroek EP, van Buuren S, van der Hofstede JW. On the assessment of adverse drug reactions from spontaneous reporting systems: the influence of under-reporting on odds ratios. Stat Med 2002; 21: 2027–2044. pmid:12111885
- 10. van Puijenbroek E, Diemont WL, van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf 2003; 26: 293–301. pmid:12650632
- 11. Nguyen M, Ball R, Midthun K, Lieu TA. The Food and Drug Administration’s Post-Licensure Rapid Immunization Safety Monitoring program: strengthening the federal vaccine safety enterprise. Pharmacoepidemiol Drug Saf 2012; 21: 291–297. pmid:22262619
- 12. Dart RC. Monitoring risk: Post marketing surveillance and signal detection. Drug Alcohol Depend 2009; 105S: S26–S32.
- 13. Ooba N, Kubota K. Selected control events and reporting odds ratio in signal detection methodology. Pharmacoepidemiol Drug Saf 2010; 19; 1159–1165. pmid:20669233
- 14. Hepburn P, Howlett J, Boeing H, Cockburn A, Constable A, Davi A, et al. The application of post-market monitoring to novel foods. Food Chem Toxicol 2008; 46: 9–33. pmid:17950974
- 15. Sugiura H, Ohkusa Y, Akahane M, Sano T, Okabe N, Imamura T. Development of a web-based survey for monitoring daily health and its application in an epidemiological survey. J Med Internet Res 2011; 13: e66. pmid:21946004
- 16. Sugiura H, Ohkusa Y, Akahane M, Sugahara T, Okabe N, Imamura T. Construction of syndromic surveillance using a web-based daily questionnaire for health and its application at the G8 Hokkaido Toyako Summit meeting. Epidemiol Infect 2010; 138: 1493–1502. pmid:20067657
- 17. Sano T, Akahane M, Sugiura H, Ohkusa Y, Okabe N, Imamura T. Internet survey of the influence of environmental factors on human health: environmental epidemiologic investigation using the web-based daily questionnaire for health. Int J Environ Health Res 2013; 23: 247–257. pmid:22946467
- 18. Hutwagner L, Thompson W, Seeman GM, Treadwell T. The bioterrorism preparedness and response early aberration reporting system (EARS). J Urban Health 2003; 8 (Suppl 1): i89–i96.
- 19. World Health Organization. Safety of Medicines—A Guide to Detecting and Reporting Adverse Drug Reactions—Why Health Professionals Need to Take Action. Available from: http://apps.who.int/medicinedocs/en/d/Jh2992e/
- 20. Council for International Organizations of Medical Sciences. Practical Aspects of Signal Detection in Pharmacovigilance. Report of CIOMS Working Group VIII. Geneva: CIOMS; 2010.
- 21. Rothman KJ, Lanes S, Sacks ST. The reporting odds ratio and its advantages over the proportional reporting ratio. Pharmacoepidemiol Drug Saf 2004; 13: 519–523. pmid:15317031
- 22. Newkirk R, Hedberg C, Bender J. Establishing a milkborne disease outbreak profile: potential food defense implications. Foodborne Pathog Dis 2011; 8: 433–437. pmid:21114422
- 23. Wendel AM, Johnson DH, Sharapov U, Grant J, Archer JR, Monson T, et al. Multistate outbreak of Escherichia coli 0157: H7 infection associated with consumption of packaged spinach, August–September 2006: the Wisconsin investigation. Clin Infect Dis 2009; 48: 1079–1086. pmid:19265476
- 24. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf 2009; 32: 99–110. pmid:19236117