http://orcid.org/0000-0002-2791-1037
Figures
Abstract
The differential pattern and characteristics of completeness in adverse event (AE) reports generated by hospitals/clinics, pharmacies, consumer and pharmaceutical companies remain unknown. Thus, we identified the characteristics of complete AE reports, compared with those of incomplete AE reports, using a completeness score. We used Korea Institute of Drug Safety and Risk Management-Korea Adverse Event Reporting System Database (KIDS-KD) between January 1, 2016 and December 31, 2016. The completeness score was determined out of a total of 100 points, based on the presence of information on temporal relationships, age and sex of patients, AE progress, name of reported medication, reporting group by profession, causality assessment, and informational text. AE reports were organized into four groups based on affiliation: hospitals/clinics, pharmacies, consumers, and pharmaceutical companies. Affiliations that had median completeness scores greater than 80 points were classified as ‘well-documented’ and these reports were further analyzed by logistic regression to estimate the adjusted odds ratios and 95% confidence intervals. We examined 228,848 individual reports and 735,745 drug-AE combinations. The median values of the completeness scores were the highest for hospitals/clinics (95 points), followed by those for consumers (85), pharmacies (75), and manufacturers (72). Reports with causality assessment of ‘certain’, ‘probable’, or ‘possible’ were more likely to be ‘well-documented’ than reports that had causality assessments of ‘unlikely’. Serious reports of AEs were positively associated with ‘well-documented’ reports and negatively associated with hospitals/clinics.
Citation: Oh I-S, Baek Y-H, Kim H-J, Lee M, Shin J-Y (2019) Differential completeness of spontaneous adverse event reports among hospitals/clinics, pharmacies, consumers, and pharmaceutical companies in South Korea. PLoS ONE 14(2): e0212336. https://doi.org/10.1371/journal.pone.0212336
Editor: Tomasz Bochenek, Jagiellonian University, POLAND
Received: August 14, 2018; Accepted: January 31, 2019; Published: February 14, 2019
Copyright: © 2019 Oh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: As the KAERS database contains potentially identifying or sensitive patient information, ethical and legal restrictions on sharing de-identified datasets have been imposed by the Korea Institute of Drug Safety and Risk Management (KIDS). Therefore, any data requests may be sent to the following contact information of KIDS (Tel: 02-2172-6776, E-mail: kids_kd@drugsafe.or.kr).
Funding: This study was funded by the National Institute of Food and Drug Safety Evaluation, Ministry of Food and Drug Safety of South Korea (grant no: 17172-186) to JYS.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Although the spontaneous reporting system is a keystone of pharmacovigilance, the incompleteness of data is a major concern for causality assessment [1, 2]. Since causal assessment of adverse events (AEs) cannot be made without demographic or clinical information, completeness in reporting is important for the early detection of adverse drug reactions. Identifying the factors associated with reporting completeness is critical for enhancing quality management systems—an essential component of good pharmacovigilance practices [3, 4]. Higher reporting completeness is associated with the seriousness of an AE [5].
Including different reporting groups and using an integrated interpretation are important in evaluating completeness, as the reported AE will vary with the reporting group. Reports from community pharmacists are important owing to their wide distribution and geographical accessibility [6]. Although reports from pharmacists had lower completeness scores than reports from physicians, significant differences in the distribution of AEs were observed [7]. In addition, consumer reports have the potential benefit of detecting unknown signals because they provide different categories of AEs for different types of medicines, unlike reports by healthcare professionals [8, 9].
Few studies have used a large spontaneous reporting database to evaluate the characteristics of completeness by reporting group. Given the scarcity of information on the factors associated with completeness of AE reports, established reporting systems can be leveraged to describe and compare the characteristics of the different reporting groups. Hence, we compared the differences in completeness by reporting group and categorized the determinants by following four affiliations: hospitals/clinics, pharmacies, pharmaceutical companies, and consumers.
Materials and methods
This study was approved by the Sungkyunkwan University’s Institutional Review Board(SKKU-IRB-2017-09-007), which waived the requirement for informed consent.
The Korea Adverse Event Reporting System (KAERS) database
The Korea Adverse Event Reporting System was set up in 1988, and the reporting rate increased rapidly following the designation of three university hospitals as Korean Regional Pharmacovigilance Centers in 2006, and the establishment of the Korea Institute of Drug Safety and Risk Management-Korea Adverse Event Reporting System Database (KIDS-KD) in 2012 [10]. The KAERS database was developed by KIDS in 2012 to manage AE reports effectively. This computerized AE reporting system includes voluntary reporting by healthcare workers and the general public, as well as mandatory reporting by manufacturers for serious and unexpected events, using a standardized form [11]. Reports from the Korean Regional Pharmacovigilance Centers accounted for 71.5% (163,676 reports) of the KAERS database, and among these, hospitals/clinics accounted for the majority.
This database includes information on AE codes, serious AEs, suspected drug information, reporting group by profession, reporting group by affiliation, and causality assessment information. In the database, the suspect drugs are arranged in accordance with the Anatomical Therapeutic Chemical (ATC) classification system. AEs are described using the preferred terms (PTs) recommended by the World Health Organization Adverse Reaction Terminology (WHO-ART). In the future, these will be changed to terms in the Medical Dictionary for Regulatory Activities (MedDRA) according to the recommendation of the WHO [12, 13]. A serious AE corresponds to any AE resulting in a death, a life-threatening situation, an inpatient hospitalization or prolongation of existing hospitalization, a persistent or significant disability, congenital abnormality or birth defect, or other medically important conditions as defined by KIDS.
Selection of AE reports
We accessed all AE reports that were filed in the KAERS database between January 1, 2016 and December 31, 2016, and categorized the reporting group by affiliation where this information was available. Since each report consisted of combinations of drugs defined according to the ATC classification system and AEs classified according to WHO-ART, we included individual reports that comprised of combinations of drug-AEs. We then conducted an analysis based on the reporting group by affiliation.
Reporting group by affiliation and reporting group by profession
We used four different reporting groups by affiliation: hospitals/clinics, pharmacies, manufacturers, and consumers. Reporting groups were defined as follows: hospitals/clinics, if they were either medical institutions or health centers; pharmacies, if pharmacy reports were reported to the Korean Pharmaceutical Association; manufacturers if they were reported by pharmaceutical companies; and consumers, for those that were not reported through the aforementioned affiliations.
Reporting groups by profession consisted of doctors, pharmacists, nurses, and consumers. These categories were based on information provided about the occupation of the person who made the initial report on the AEs.
Characteristics of AE reports
For this study, we analyzed the frequency and proportion of patient age, sex, reporting year, date of prevalence, and reporting group by profession among hospitals/clinics, pharmacies, manufacturers, and consumers. Age consisted of six subgroups: neonates aged <28 days; infants aged 28 days to < 2 years; children aged 2 to < 12 years; adolescents aged 12 to < 19 years; adults aged 19 to < 65 years; and the elderly aged ≥65 years. In order to ensure comparability, the age subgroups were merged into three groups (<19, 19–65, and >65). For each report, if information was missing in age, sex, month, or reporting group by profession, it was classified as ‘No information’ for that particular variable.
Based on the International Conference on Harmonization (ICH) E2D Guidelines, we also classified all adverse reactions into ‘serious reports of AEs’ or ‘not serious reports of AEs’. Causality assessment was performed using the six categories classified by the World Health Organization-Uppsala Monitoring Center (WHO-UMC) criteria: certain, probable, possible, unlikely, unclassified, and unassessable [14]. This causality was primarily estimated by the adverse drug reaction monitoring team at each Regional Pharmacovigilance Center and was subsequently re-estimated and validated by KIDS healthcare professionals [15]. In this study, our research interest was not in the level of causality assessment but in the presence of causality; therefore, we merged causality information into three groups: presence of causality (certain, probable, or possible), absence of causality (unlikely), and other (unclassified or unassessable).
Completeness score for hospitals/clinics, pharmacies, manufacturers, and consumers
The completeness score comprised a total of 100 points. From the 100 points, 20 points were accorded to temporal relationships, 15 points for information on the patient, 5 points for progress of AEs, 30 points for information on prescription of medicines, 5 points for information on reporting group by profession, 5 points for causality assessment, and 20 points for informational text (S1 Appendix).
Based on information provided in the AE reports, reports were classified into two groups: ‘well-documented’ and ‘poorly documented.’ A full completeness score was given only when there were no missing values among all criteria of the completeness score assessment. We used the median score to determine the criteria for classifying reports as ‘well-documented’ or ‘poorly documented.’ Completeness scores greater than the overall median score were considered ‘well-documented’; scores below the overall median score were considered ‘poorly documented.’
Statistical analysis
The demographic and clinical characteristics of study participants were analyzed using medians, means, and standard deviations (SD) for the continuous variables. Frequencies and percentages were used to analyze the categorical variables. The chi-square test was used to compare categorical variables, and p-values < 0.05 were considered to be statistically significant. To identify the characteristics of a ‘well-documented’ AE report, we conducted a multivariate logistic regression analysis with ‘well-documented’ AE report as the dependent variable, and estimated the adjusted odds ratios (aORs) and 95% confidence interval (CI). Our multivariate logistic regression model included sex, age, serious AEs, and causality assessment as independent variables.
We also carried out a subgroup analysis for each affiliation to determine the influence of the reporting group’s profession. We conducted a multivariate logistic regression model to reveal the association between the completeness score and the sex, age, serious AEs, and causality assessment for each profession (doctors, pharmacists, nurses, and consumers). We determined if there were differences between characteristics associated with high completeness scores for each profession nested within each affiliation. We used the SAS 9.4 software (SAS Institute Inc., Cary, NC, USA) for all statistical analyses.
Results
Of the 228,939 individual reports collected in the KAERS database, 91 were excluded because of a lack of information about reporting group by affiliation (Fig 1). A total of 228,848 individual reports and 735,745 drug-AE combinations were included for analysis. About 64.9% of the reports were from hospitals/clinics (N = 148,559; combination = 280,602), 8.0% were from pharmacies (N = 18,244; combination = 77,454), 26.0% were from manufacturers (N = 59,600; combination = 371,953), and 1.1% were from consumers (N = 2,445; combination = 5,736).
We describe characteristics of the reporting groups by affiliation according to age, sex, month, and reporting group by profession, with all chi-square test results being statistically significant (Table 1). When classified by age, adults (19–65 years) comprised the largest proportion of reports for all affiliations, which was also the case for overall reports. For all variables except reporting group by profession, the proportion of criteria that had ‘no information’ was largest from manufacturers. When classified by reporting groups by profession, more than 90% were pharmacists and consumers for the affiliations of pharmacy and consumers, respectively. Nurses and doctors were the major professions when the affiliations were hospitals/clinics and manufacturer, respectively. Furthermore, from a total of 107,694 reports made by nurses, 106,494 were from hospitals/clinics, whereas, for doctors and consumers, the vast majority of the reports were made to manufacturers.
Table 2 describes the characteristics of ‘serious’ and ‘not serious’ reports of AEs and causality assessments according to the reporting group by affiliation. The overall proportion of serious AEs was 9.7%. For manufacturers, the proportion of serious AEs was higher than those of other reporting groups by affiliation (26.3%). ‘No information’ was zero for all reporting groups by affiliation. By affiliation, manufacturers had the highest proportion of reports for which causality was "unlikely", while the highest proportion of "no information" was among consumers and manufacturers.
Table 3 shows the distribution of completeness scores by affiliation. Median values of completeness scores were the highest for hospitals/clinics (95 points), followed by those of consumers (85 points), pharmacies (75 points), and manufacturers (72 points). Hospitals had the highest proportion (93.6%) in the group of completeness scores greater than the overall median, greatly exceeding all other affiliations, with the mean and median results showing similar results as the distribution. In addition, hospitals had a much higher proportion than any other affiliation of ‘well-documented’ reports that had a completeness score greater than the median value, whereas pharmacies and consumers had a very low proportion of ‘well-documented’ reports.
Table 4 presents the odds ratios for high completeness scores associated with reporting group by affiliation. Among 735,745 combinations of reports, pharmacies and consumers had too few ‘well-documented’ reports; therefore, we could not estimate the odds ratios for pharmacies and consumers. The aOR for hospitals/clinics associated with serious AEs was 0.71 (95% CI: 0.68–0.73), whereas that of manufacturers was 1.34 (95% CI: 1.31–1.38). However, the aORs associated with the presence of causality for hospitals/clinics and manufacturers were 2.41 (95% CI: 2.26–2.56) and 0.78 (95% CI: 0.74–0.83), respectively.
In the subgroup analyses of hospitals/clinics, for all professions, females had significantly decreased odds ratios of association with high completeness scores, as compared with those of males. Doctors were observed to have an increased association with completeness for serious AEs, whereas nurses showed opposite results. Moreover, for all professions, if the causality assessment was not ‘unlikely,’ the completeness scores were high (Table 5).
In the subgroup analysis of manufacturers, doctors and nurses, unlike consumers, had significant;y higher completeness scores for serious AEs. When causality was confirmed, nurses and consumers had large ORs, inferring a high completeness score, in contrast to those of doctors (Table 6).
Discussion
The omission of key information in AE reports has been a critical hurdle for the quality management of signal detection [2, 5, 16, 17]. Our study was designed to identify factors associated with completeness in AE reports. It is notable that reports that include causality assessment had 12.4 increased odds of being “well-documented”, when compared with reports where causality was assessed as “unlikely”. In addition, hospital and clinics had higher median completeness scores than those of pharmacies. Sixty-five percent of reports were collected from hospital and clinics. Nurse reports made up the highest proportion (55%) of reports from all professions, a marked difference from a previous finding in which nurse reports comprised 6.5% of all AE reports [18]. Our results demonstrated the determinants of greater data completion in different report groups by using a nationwide AE reporting system.
We found that 99.4% of pharmacy reports were poorly documented, whereas 80.9% of reports from hospitals/clinics were well-documented. Since community pharmacies are based on ambulatory settings and have less access to clinical information, including indications, only 0.1% of pharmacy reports had information on indications, which led to lower completeness in their reports. The Korean Pharmaceutical Association, which supplied the only group of AE reports from community pharmacies, was designated as a regional pharmacovigilance center in 2013 [6, 19]. Although issues regarding the quality of the reports have been raised, the association has generated the highest number of reports among 27 regional pharmacovigilance centers [20]. In contrast, hospital and clinics were more likely to have access to the demographic and clinical information of patients, resulting in high completeness.
Contrary to a previous finding reporting an association between seriousness of event and increased completeness [5], our logistic regression results were statistically insignificant. Moreover, an inverse association between the two was observed for hospitals/clinics. However, significant results were observed in the hospital and clinic subgroup analyses. Nurses had a lower chance of submitting ‘well-documented’ reports for serious AEs than non-serious AEs. This can be explained in the context that nurses are involved with medication errors in the process of administering and dispensing medications [21, 22] in nursing practice. Medication errors—preventable mistakes in prescribing or delivering medication to patients [23]—are associated with adverse drug reactions [24]. The psychological burden caused by the possible risk of legal responsibility for serious AEs might lead to intentional omissions in reporting serious AEs. Conversely, doctors had 1.3-fold increased odds of ‘well-documented’ reports for serious AEs.
In the logistic regression analyses, ORs were not estimated for pharmacies and consumers because of their highly skewed completeness scores. In the S2 Appendix, we explore the proportion of each completeness evaluation criterion that had ‘no information,’ which was included in the calculation of the completeness score. The proportion of ‘no information’ for the criterion ‘indication’ in reports filed by pharmacies and consumers was greater than 99.7% (S2 Appendix). Since this particular criterion accounts for 10 points of the total score, the completeness scores of pharmacies and consumers could not have exceeded the overall median of 95 points.
The low completeness score of manufacturer reports corroborates a study of FDA reports that also observed poor completeness of reports by pharmaceutical companies for serious adverse drug events [25]. In the manufacturer subgroup analyses, heterogeneity of the results was observed when data were classified by reporting profession. Consumer reports showed an inverse association between seriousness of AEs and completeness, unlike those from healthcare specialists, including doctors, pharmacists, and nurses. Notably, doctors were more likely to submit ‘poorly documented’ reports for cases suspected to have causal association when reporting to manufacturers. This result is discordant with the results from the hospitals/clinics subgroup analysis, in which doctors had 3.43-fold increased odds of submitting ‘well-documented’ reports that are suspected to have causal association. When reporting to manufacturers, doctors may be less obligated to complete reports. However, since AE reporting is used for performance evaluation in hospitals designated as regional pharmacovigilance centers, this may have resulted in a higher motivation for doctors to complete the AE reports [10]. Therefore, affiliatory and psychological motivations might account for the inconsistency in the magnitude of association between main and subgroup analyses of AE reports.
Our study has several strengths. By using the nationwide AEs reporting database, we generated representative results because of the inclusion of all reports from different reporting groups in South Korea. To the best of our knowledge, this is the first study identifying determinants of completeness in AE reports in South Korea. Our results have particular importance in that we used reporting groups across different affiliations and professions of reporters with distinct features [18, 26, 27]. In addition, we identified the current status of completeness in AE reporting systems using recent data.
However, our study is subject to the following limitations. First, reports from consumers and pharmacies were excluded from the logistic regression model owing to the selection of reports with scores above the overall median completeness score. Since hospitals/clinics reported the majority of such reports, the distribution of completeness scores is skewed negatively by reports from this affiliation. Our findings of characteristics associated with data completeness are restricted to hospitals/clinics, and manufacturers, and therefore, may not apply to other affiliations (consumers and pharmacies). More generalizable results could have been achieved by including more reports from consumers and pharmacies using a weighted method. Second, spontaneous reports are based on the reporters’ subjective judgements. Thus, concerns about objectivity and quality stability may arise. However, this is not a significant hurdle for our objective, as we are interested in identifying the determinants of completeness, with a focus on variables with missing values. Third, our study is based on a cross-sectional design, so no causal interpretation can be made. Moreover, the findings in the main and subgroups analyses are inconsistent. Although we interpreted the results based on previous findings and affiliatory contexts, the possibility of misinterpretation exists. Finally, there is a possibility of duplication in reports, as the KAERS database is anonymized. Such duplication could occur if multiple reports are made for the same event.
From 228,848 individual AE reports in the database, we found the differential completeness score for reporting source by affiliation to be highest for hospitals and clinics, followed by those of consumers, pharmacies, and manufacturers. Reports suspected to have causal association in the causality assessment were more likely to have better completeness in all reports by affiliation, while seriousness of AEs was not associated with completeness of reports by hospitals/clinics. Low completeness in reports by pharmacies and manufacturers could be due to the difficulty of collecting information on the details of AEs such as indication, medical history or inability to follow-up with patients after the onset of AEs. Given that pharmacists and manufacturers are equally important stakeholders of AEs as well as healthcare professionals, systematic training programs such as developing standardized coding guidelines should be provided in adverse reports to enhance completeness of reports from these affiliations. Implementation of strategies to engage healthcare professionals will be critical for enhancing completeness in spontaneous reports and early signal detection.
Supporting information
S1 Appendix. Standard of completeness scores in the Korea Adverse Event Reporting System (KAERS) database.
https://doi.org/10.1371/journal.pone.0212336.s001
(DOCX)
S2 Appendix. The number of reports having missing values in each category of the adverse event reporting form, used to calculate completeness scores, by reporting group by profession.
https://doi.org/10.1371/journal.pone.0212336.s002
(DOCX)
Acknowledgments
The authors would like to thank the Korea Institute of Drug Safety and Risk Management for their cooperation in providing access to the Korea Adverse Event Reporting System Database (KIDS-KD). This study was funded by the National Institute of Food and Drug Safety Evaluation, Ministry of Food and Drug Safety of South Korea (Grant no: 17172–186). We thank Editage for their assistance with English language editing.
References
- 1. Stephenson WP, Hauben M. Data mining for signals in spontaneous reporting databases: proceed with caution. Pharmacoepidemiol Drug Saf. 2007;16(4):359–65. Epub 2006/10/05. pmid:17019675.
- 2. Plessis L, Gomez A, Garcia N, Cereza G, Figueras A. Lack of essential information in spontaneous reports of adverse drug reactions in Catalonia-a restraint to the potentiality for signal detection. Eur. J Clin Pharmacol. 2017;73(6):751–8. Epub 2017/03/03. pmid:28251276.
- 3. Brajovic S, Piazza-Hepp T, Swartz L, Dal Pan G. Quality assessment of spontaneous triggered adverse event reports received by the Food and Drug Administration. Pharmacoepidemiol Drug Saf. 2012;21(6):565–70; discussion 71–2. Epub 2012/02/24. pmid:22359404.
- 4. Bergvall T, Noren GN, Lindquist M. vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug Saf. 2014;37(1):65–77. Epub 2013/12/18. pmid:24343765.
- 5. Durrieu G, Jacquot J, Mege M, Bondon-Guitton E, Rousseau V, Montastruc F, et al. Completeness of Spontaneous Adverse Drug Reaction Reports Sent by General Practitioners to a Regional Pharmacovigilance Centre: A Descriptive Study. Drug Saf. 2016;39(12):1189–95. Epub 2016/10/01. pmid:27688025.
- 6. Yu YM, Shin WG, Lee JY, Choi SA, Jo YH, Youn SJ, et al. Patterns of Adverse Drug Reactions in Different Age Groups: Analysis of Spontaneous Reports by Community Pharmacists. PLoS One [Internet]. 2015[cited 2018 Dec 26]; 10(7):e0132916. Available from: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132916. Epub 2015/07/15. pmid:26172050; pmcid: PMCPMC4501755.
- 7. Gedde-Dahl A, Harg P, Stenberg-Nilsen H, Buajordet M, Granas AG, Horn AM. Characteristics and quality of adverse drug reaction reports by pharmacists in Norway. Pharmacoepidemiol Drug Saf. 2007;16(9):999–1005. Epub 2007/04/26. pmid:17457794.
- 8. Blenkinsopp A, Wilkie P, Wang M, Routledge PA. Patient reporting of suspected adverse drug reactions: a review of published literature and international experience. Br J Clin Pharmacol. 2007;63(2):148–56. pmid:17274788
- 9. Aagaard L, Nielsen LH, Hansen EH. Consumer reporting of adverse drug reactions: a retrospective analysis of the Danish adverse drug reaction database from 2004 to 2006. Drug Saf. 2009;32(11):1067–74. Epub 2009/10/09. pmid:19810778.
- 10. Shin JY, Jung SY, Ahn SH, Lee SH, Kim SJ, Seong JM, et al. New initiatives for pharmacovigilance in South Korea: introducing the Korea Institute of Drug Safety and Risk Management (KIDS). Pharmacoepidemiol Drug Saf. 2014;23(11):1115–22. Epub 2014/09/25. pmid:25251052.
- 11. Soukavong M, Kim J, Park K, Yang BR, Lee J, Jin XM, et al. Signal Detection of Adverse Drug Reaction of Amoxicillin Using the Korea Adverse Event Reporting System Database. J Korean Med Sci. 2016;31(9):1355–61. Epub 2016/08/12. pmid:27510377; pmcid: PMCPMC4974175.
- 12.
WHO. Adverse reaction terminology (WHO-ART). World Health Organization. 2005.
- 13. Park HJ, Shin JY, Kim MH, Park BJ. Increased use in propofol and reported patterns of adverse events among anesthetics in Korea. Regul Toxicol Pharmacol. 2015;71(3):478–83. Epub 2015/02/11. pmid:25666110.
- 14.
Uppsala Monitoring Centre [Internet]. The use of the WHO-UMC system for standardised case causality assessment 2018. Available from: https://www.who-umc.org/media/164200/who-umc-causality-assessment_new-logo.pdf. [Cited 2018 Dec 26].
- 15. Sim DW, Park KH, Park HJ, Son YW, Lee SC, Park JW, et al. Clinical characteristics of adverse events associated with therapeutic monoclonal antibodies in Korea. Pharmacoepidemiol Drug Saf. 2016;25(11):1279–86. Epub 2016/11/03. pmid:27364925.
- 16. Shanmugapriya S, Bhuvaneswari K, Karthika P. Audit to evaluate the completeness of adverse drug reporting in an ADR monitoring centre in Tamilnadu. Indian Journal of Medical Specialities. 2017;8(4):175–8.
- 17. Sanchez-Sanchez B, Altagracia-Martinez M, Kravzov-Jinich J, Moreno-Bonett C, Vazquez-Moreno E, Martinez-Nunez JM. Evaluation of completeness of suspected adverse drug reaction reports submitted to the mexican national pharmacovigilance centre: a cross-sectional period-prevalence study. Drug Saf. 2012;35(10):837–44. Epub 2012/08/29. pmid:22924896.
- 18. Mendes D, Alves C, Batel Marques F. Nurses' spontaneous reporting of adverse drug reactions: expert review of routine reports. J Nurs Manag. 2014;22(3):322–30. Epub 2014/04/24. pmid:24754750.
- 19. Shin YS, Lee WY, Choi YH, Park B, Jee YK, Choi SK, et al. Spontaneous reporting of adverse drug events by Korean regional pharmacovigilance centers. Pharmacoepidemiol Drug Saf. 2009;18(10):910–5. pmid:19621345
- 20.
Shin JY [Internet]. The evaluation of completeness in adverse drug reaction reports and the role of pharmacists. Newsletter, Regional Pharmacovigilance Center. 2017. Available from: https://www.drugsafe.or.kr/iwt/ds/ko/community/EgovCenterDetail.do?seqNo=676. [Cited 2018 Dec 26].
- 21. Kim KS, Kwon SH, Kim JA, Cho S. Nurses' perceptions of medication errors and their contributing factors in South Korea. J Nurs Manag. 2011;19(3):346–53. Epub 2011/04/22. pmid:21507105.
- 22. Benjamin DM. Reducing medication errors and increasing patient safety: case studies in clinical pharmacology. J Clin Pharmacol. 2003;43(7):768–83. Epub 2003/07/15. pmid:12856392.
- 23. Aronson JK. Medication errors: definitions and classification. Br J Clin Pharmacol. 2009;67(6):599–604. pmid:19594526
- 24. Runciman WB, Roughead EE, Semple SJ, Adams RJ. Adverse drug events and medication errors in Australia. Int J Qual Health Care. 2003;15 Suppl 1:i49–59. Epub 2003/12/09. pmid:14660523.
- 25. Moore TJ, Furberg CD, Mattison DR, Cohen MR. Completeness of serious adverse drug event reports received by the US Food and Drug Administration in 2014. Pharmacoepidemiol Drug Saf. 2016;25(6):713–8. Epub 2016/02/11. pmid:26861066.
- 26. Avery AJ, Anderson C, Bond CM, Fortnum H, Gifford A, Hannaford PC, et al. Evaluation of patient reporting of adverse drug reactions to the UK 'Yellow Card Scheme': literature review, descriptive and qualitative analyses, and questionnaire surveys. Health Technol Assess. 2011;15(20):1–234, iii-iv. Epub 2011/05/07. pmid:21545758.
- 27. Hazell L, Cornelius V, Hannaford P, Shakir S, Avery AJ; Yellow Card Study Collaboration. How do patients contribute to signal detection?: A retrospective analysis of spontaneous reporting of adverse drug reactions in the UK's Yellow Card Scheme. Drug Saf. 2013;36(3):199–206. Epub 2013/02/28. pmid:23444232.