https://orcid.org/0000-0003-3780-3213
Figures
Abstract
Several statistical methods have been proposed to detect adverse drug reactions induced by taking two drugs together. These suspected adverse drug reactions can be discovered through post-market drug safety surveillance, which mainly relies on spontaneous reporting system database. Most previous studies have applied statistical models to real world data, but it is not clear which method outperforms the others. We aimed to assess the performance of various detection methods by implementing simulations under various conditions. We reviewed proposed approaches to detect signals indicating drug-drug interactions (DDIs) including the Ω shrinkage measure, the chi-square statistic, the proportional reporting ratio, the concomitant signal score, the additive model and the multiplicative model. Under various scenarios, we conducted a simulation study to examine the performances of the methods. We also applied the methods to Korea Adverse Event Reporting System (KAERS) data. Of the six methods considered in the simulation study, the Ω shrinkage measure and the chi-square statistic with threshold = 2 had higher sensitivity for detecting the true signals than the other methods in most scenarios while controlling the false positive rate below 0.05. When applied to the KAERS data, the two methods detected one known DDI for QT prolongation and one unknown (suspected) DDI for hyperkalemia. The performance of various signal detection methods for DDI may vary. It is recommended to use several methods together, rather than just one, to make a reasonable decision.
Citation: Jung D, Jung I (2024) A simulation-based comparison of drug-drug interaction signal detection methods. PLoS ONE 19(4): e0300268. https://doi.org/10.1371/journal.pone.0300268
Editor: Jed N. Lampe, University of Colorado Denver Skaggs School of Pharmacy and Pharmaceutical Sciences, UNITED STATES
Received: April 4, 2023; Accepted: February 25, 2024; Published: April 17, 2024
Copyright: © 2024 Jung, Jung. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The dataset used in the present study cannot be publicly shared. Qualified researchers can request access to the KAERS database through the Korea Institute of Drug Safety and Risk Management by visiting https://open.drugsafe.or.kr/original/invitation.jsp.
Funding: IJ received National Research Foundation of Korea grant by the Korean government (MSIT) (No. 2021R1F1A1060156). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Polypharmacy, the use of multiple medicines has increased as the average life expectancy and the prevalence of multimorbidity has increased [1]. Adverse events (AEs) caused by the administration of many drugs at the same time are therefore a serious concern. These suspected adverse drug reactions (ADRs) due to drug-drug interaction (DDI) can be discovered through post-market drug safety surveillance (PMS). Spontaneous reporting systems (SRSs) are databases used for PMS that include ADR reports and prescription information (e.g, sex, age, date, quantity, etc). By investigating SRS databases using data mining tools, we can identify signals and prevent the potential ADRs induced by DDI. Generally, quantitative DDI signals refer to excessive risk for a combination of two drugs compared with the risks for the individual drugs. However, the criteria used in each method to define signals are different and have various pros and cons.
Several studies have proposed approaches to detect signals indicating drug-drug interactions as well as single-drug adverse reactions. Norén et al. proposed the Ω shrinkage measure to screen for disproportional reporting indicative of suspected drug-drug interaction [2]. Gosho et al. proposed the chi-square statistic as a statistical criterion for detecting drug-drug interaction signals and compared this method with Norén et al.’s Ω shrinkage measure using a simulation study [3].The proportional reporting ratio (PRR) proposed by Evans et al [4] was used to detect signals indicating single-drug reactions, then was extended for drug-drug interactions by Wang et al. [5]. The concomitant signal score (CSS), proposed by Noguchi et al. [6], is an improved detection method using the PRR. Thakrar et al. proposed the additive and multiplicative models [7].
Some studies have reviewed the methodologies [8, 9]. Other studies applied the statistical models to real world data, and then compared the results [10–16]. Only a few studies used simulation to evaluate the performance of a few methods [3]. The main purpose of this study is to assess the performance of various detection methods for DDI through a simulation study. We evaluated the methods in terms of false positive rate and sensitivity in various scenarios. Note that our study exclusively focused on employing frequentist methods for detecting DDI signals. It’s noteworthy to mention that there exists a well-known Bayesian model named Interaction Signal Score (IntSS) [15, 16] for this purpose.
In the next section, we provide an overview of existing methods and explain how the simulation study was implemented. We also introduce the Korea Adverse Event Reporting System (KEARS) data (2017–2019) from the Korea Institute of Drug Safety and Risk Management (KIDS). In the following section, we present the performance of the signal detection algorithms for DDI from the simulation study. The DDI signal detection results using the KEARS data are also shown. We present a discussion in the final section.
2. Methods
The signal detection methods are fundamentally based on the observed AE frequencies according to exposure status of two drugs as presented in Table 1. Let r00 denote the observed reporting rate for the AE in the absence of both Drug 1 and Drug 2. Similarly, r10,r01, and r11 are the observed reporting rate (i) with Drug 1 but not Drug 2, (ii) with Drug 2 but not Drug 1, and (iii) with concomitant use of the two drugs, respectively.
2.1 Ω shrinkage method
Norén et al. [2] proposed the Ω shrinkage method. The method basically compares the observed reporting rate r11 with its expected value E[r11] estimated under the assumption that there is no interaction between the two drugs. The estimator s11 of E[r11] is given as
A shrinkage factor is considered to adjust spurious associations due to a very small value of s11 by reason of generally very rare ADRs by DDI. The Ω shrinkage measure is defined as
where n11 = n111 + n110 and α is a tuning parameter determining the shrinkage strength. Generally, α is set to 0.5. The lower limit of the 95% confidence interval for Ω can be estimated as
where ϕ(0.975) is the 97.5th percentile of the standard normal distribution. The criterion Ω25 > 0 is used to determine the DDI signal.
2.2 Chi-square statistic method
Gosho et al. [3] proposed the chi-square statistic model in order to reduce the false positive rate when events are rare. The measure of the chi-square statistic method χ is the square root of the chi-square test statistic with a correction term to adjust for false positives.
The threshold χ > 2 and χ > 2.6 is set for identifying DDI signals. These cutoff values are set based on the 95th and 99th percentiles, respectively, of the chi-square distribution with one degree of freedom.
2.3 Proportional reporting ratio (PRR)
The proportional reporting ratio (PRR) proposed by Evans et al. [4] is commonly used in disproportionality analysis to detect adverse event induced by a single drug. The PRR is the ratio of observed reporting rate with and without a drug. The PRR has been extended to drug-drug interactions [5]. First, PRRD1 for Drug1 and PRRD1 for Drug2 are defined as
where n10 = n101 + n100, n00 = n001 + n000, and n01 = n011 + n010. The PRRD1D2 for concomitant use of Drug1 and Drug2 is defined as
The lower limit of the 95% confidence interval for the PRR is used to define a signal. It is calculated as
where SD stands for the standard deviation. The SD for Drug1 is
, the SD for Drug2 is
, and the SD for the Drug1-Drug2 pair is
. The signal detection criterion is to compare the lower limit of the 95% confidence interval for a single drug and drug-drug pairs. If PRR025D1D2 > max (PRR025D1, PRR025D2), then the drug pair is considered to be a signal of DDI.
2.4 Concomitant signal score (CSS)
The concomitant signal score, proposed by Noguchi et al. [6], was shown to improve the combination risk ratio (CRR) [17], a DDI detection method using PRR. The weakness of CRR is that the lower limit of the 95% CI of PRRD1D2 overlaps with the upper limit of the 95% CI of PRRD1 or PRRD2. This is because adverse event reports involving individual drugs are more common than reports concerning the concomitant use of two drugs. The concomitant signal score (CSS) is the ratio of PRR025D1D2 and the maximum value between PRR975D1 and PRR975D2.
he signal detection criteria are (1) PRR025D1D2 > 1 and (2) CSS > 1.
2.5 Additive model
Thakrar et al. considered both an additive model and a multiplicative model for the detection of DDI signals [7]. The additive model assumes that the risk associated with a drug adds to the background risk. Under the additive assumption, no interaction is established when the excess risk associated with the drug combination is the same as the sum of the excess risks associated with each exposure in the absence of the other. The risk difference is defined as RDD1D2 = p11 –p00, RDD1 = p10 –p00, and RDD2 = p01 –p00. When RDD1D2 > RDD1 + RDD2, the signal of an interaction is detected. Using the linear probability model in the below, we test for the significance of the interaction term.
When β3 is statistically significantly greater than 0, there is a potential DDI. A positive value of β3 signifies a positive interaction, namely, an elevated risk for the combination of Drug 1 and Drug 2 compared to that expected based on the individual drugs.
2.6 Multiplicative model
The multiplicative model assumes that the risk associated with a drug multiplies with the background risk. The risk ratio is defined as , and
When
, a signal is detected. To test for the interaction effect based on the multiplicative model, we implement the log linear regression model in the below.
When β3 is statistically significantly greater than 0, there is a potential DDI. As in the additive model, a positive value of β3 indicates a positive interaction.
2.7 Simulation study
We conducted a simulation study to evaluate the performances of the methods reviewed in the previous section. We considered four different sets of scenarios. Although the basic idea for setting the parameters was adopted from the study by Gosho et al. [3], we created more various situations. While they only considered the additive assumption for an interaction effect, we considered both the additive (scenario sets 1 and 2) and multiplicative (scenario sets 3 and 4) assumptions. Under scenario sets 1 and 3, no interaction was assumed to evaluate the false positive rate, which is the proportion of signals falsely detected for an interaction effect. Under scenario sets 2 and 4, positive interaction effects were created to evaluate sensitivity, which is the proportion of signals correctly detected. In each scenario set, four different scenarios were considered. Four scenarios in set 1 all assumed no interaction based on the additive assumption, that is, . Additionally, scenario (1–1) assumed no effect of each single drug
; (1–2) assumed a positive effect of Drug 2
; (1–3) assumed the same positive effect for Drugs 1 and 2
; and (1–4) assumed a greater effect of Drug 2 than Drug 1
. Four scenarios in set 2 all assumed a positive interaction effect, i.e.,
Additional settings for each single drug effect for (2–1), (2–2), (2–3), and (2–4) were the same for (1–1), (1–2), (1–3), and (1–4), respectively. Scenario set 3 followed the structure of scenario 1 using the multiplicative assumption. All four scenarios in set 3 assumed no interaction under the multiplicative assumption, i.e.,
. Similarly, scenario set 4 followed the structure of scenario 2 using the multiplicative assumption. All four scenarios in set 4 assumed
. In addition, (3–1) and (4–1) assumed
; (3–2) and (4–2) assumed
; (3–3) and (4–3) assumed
; and (3–4) and (4–4) assumed
We generated AE count data {n001,n101,n011,n111} from binomial distributions B(n00,p00),B(n10,p10),B(n01,p01),B(n11,p11) with To prevent the AE count from being zero, n111 was generated using
following the approach described in the study by Gosho et al. [3]. Different values for {p00,p10,p01,p11} were used under different scenarios as presented in Tables 2–5. The false positive rate and sensitivity were calculated from 3000 replications.
2.8 Korea Adverse Event Reporting Systems (KAERS) data
We applied the six DDI signal detection methods to Korea Adverse Event Reporting System (KAERS) data from the Korea Institute of Drug Safety and Risk Management (KIDS) in 2017–2019. Drug information in KAERS data was documented by the ingredient names with ATC (Anatomical Therapeutic Chemical) codes (https://www.whocc.no/atc_ddd_index/), and AE details were recorded with WHO-ART (WHO Adverse Reaction Terminology) codes (https://www.who-umc.org), eliminating the need for an additional mapping process. We focused on two AEs of QT interval prolongation and hypokalemia. Known interactions for the two AEs were derived from a research report published by the Health Insurance Review and Assessment Service (HIRA) on the adverse event monitoring system, and the list of contraindications of co-medication drugs was derived from KIDS (as of Dec. 28, 2020). In addition, suspected interaction pairs for the same AEs were searched from the combinations of drug-drug-AE that have high frequencies in KAERS data. According to the HIRA research report, it has been reported that co-administration of potassium chloride and spironolactone and co-administration of tactolimus and spironolactone induce hyperkalemia. Potassium chloride is prescribed for potassium-deficiency, electrolyte imbalance, and digitalis poisoning. Spironolactone is a diuretic. Tactolimus is an intensive immunosuppressant agent used to prevent transplant rejection. Co-administration of domperiodone and amiodarone has been reported to cause QT prolongation. Prolongation of the QT interval is associated with an increased risk of development of a potentially lethal cardiac arrhythmia called torsade de pointes, a risk that increases with the administration of a QT-prolonging drugs [18]. Domperidone is a drug that increases gastrointestinal motility and is prescribed for indigestion and vomiting. Amiodarone is an antiarrhythmic drug.
3. Results
3.1 Simulation study
Tables 2 and 3 present the false positive rate and sensitivity of the six methods under the additive assumption. The false positive rate of the Ω method ranged from 0.001 to 0.055. The false positive rate of the chi-square method with threshold = 2 was similar to that of the Ω method: between 0.001 and 0.066. The false positive rate of the chi-square method with threshold = 2.6 showed the smallest variation, ranging from 0.000 to 0.029, while the PRR method showed a wide range of false positive rate, ranging from 0.006 to 0.127. The Ω method and the chi-square method with threshold = 2 controlled the false positive rate below 0.05 and had high sensitivity in most scenarios. In particular, when the number of events is small (n111 < 2), the chi-square method has a higher sensitivity than the Ω method. The CSS method showed the lower false positive rate than the Ω method or the chi-square method, but it also showed the lower sensitivity. This may be due to the lower threshold for the measure of the CSS method to reach an interaction. Comparing the additive and the multiplicative model, when there is no effect of each single drug or there is an effect of Drug2 only (scenarios (2–1) and (2–2)), the multiplicative model showed the higher sensitivity than the additive model. On the other hand, when there is an effect of Drug1 and Drug2 (scenarios (2–3) and (2–4)), the additive model showed the higher sensitivity than the multiplicative model.
Comparing scenarios (2–3) and (2–4), most methods except the multiplicative model showed the higher sensitivity in the scenario (2–4). That is, when the difference between the effects of the two drugs is large, the sensitivity increases.
The false positive rate and sensitivity of the six methods under the multiplicative assumption are presented in Tables 4 and 5. In scenario 3 comparing false positive rate, scenarios (3–1) and (3–2) are the same with scenarios (1–1) and (1–2) because they satisfy both the additive assumption and multiplicative assumption. In scenario 4 comparing sensitivity, scenarios (4–1), (4–2), and (4–3) are equivalent to scenarios (2–1), (2–2), and (2–3) because they also satisfy both the additive assumption and multiplicative assumption. It can be shown that satisfying the multiplicative assumption will imply satisfying the additive assumption, thus the additive model has the lower threshold for incidence f to reach an interaction,
The detection signals of the multiplicative model mean stronger interactions than those of the additive model. By comparison, the additive model can detect the DDI signal earlier. The multiplicative model can help determine whether there is stronger evidence of DDI after checking DDI with the additive model.
3.2 KAERS data
A total of 1,131,985 reports were taken from the KAERS. Among them, there were 1656 cases of hyperkalemia and 284 cases of QT prolongation. The observed reporting rate for the exposure status of each drug-drug pair was summarized in Table 6. The measure of the six methods for the six drug-drug-AE combinations was summarized in Table 7. A known interaction for QT prolongation, domperidone ‐ amiodarone, was detected as a signal by the Ω method, the chi-square method, the PRR method, the CSS method, and the additive model. The number of reports of domperidone ‐ amiodarone combination was very small (n11 = 18), but its reporting proportion of the AE was relatively high (4/18). The multiplicative model showed a positive interaction trend for domperidone ‐ amiodarone combination, but it did not reach the statistical significance (p-value = 0.185). For known interactions, only one drug-drug pair (domperidone- amiodarone) was detected. The reason for the reduced sensitivity may be that the number of cases is very small because it is rare to prescribe combinations that are known to have side effects when taken together.
Acetylsalicylic acid ‐ polystyrene sulfonate, a suspected interaction for hyperkalemia, was detected as a potential signal by the Ω method, the chi-square method, and the PRR method. The reporting proportion of hyperkalemia for acetylsalicylic acid ‐ polystyrene sulfonate combination was 22/459. While the combination of acetylsalicylic acid and polystyrene sulfonate was not a previously known interaction for hyperkalemia, DDI signals were detected among the suspected combinations that has a large number of AE reports. Only three of the six methods identified DDI signals.
4. Discussion
As the number of patients with chronic disease becomes more common, the co-prescription of multiple drugs has increased. Therefore, it has become more important to identify combinations of drugs that have side effects through post-market drug safety surveillance. In this article, we examined statistical methodologies for DDI signal detection. Of the six methods, the Ω shrinkage method and the chi-square method showed the best performance. The Ω shrinkage method and the chi-square method with threshold = 2 controlled the false positive rate below 0.05 and had high sensitivity in most scenarios. The chi-square method was especially effective when there were a very small number of reports for an AE. The chi-square method with threshold = 2.6 and the additive model seemed rather conservative. They rigorously controlled the FPR, but they had lower sensitivity than the Ω shrinkage method and the chi-square method with threshold = 2.
While we undertook a comprehensive simulation study and presented a real data example, there are some limitations in this study. For the simulation study, we assumed the number of cases involving both drugs (n11) to be 10,000, which is somewhat large. However, when considering the scenario of a very low reporting probability rate for AEs, such as 0.005%, it can be deemed a reasonable value. Nevertheless, it would be beneficial to perform an additional simulation study, assuming smaller values for n11. We presented the performance of the six methods based on FPR and sensitivity at a fixed threshold for each method. It is worth considering the feasibility of evaluating the methods using multiple possible thresholds through ROC curves and AUCs. For the KAERS data, we evaluated only two AEs of QT interval prolongation and hypokalemia. Additionally, we did not consider the approach involving negative controls for the real data example. As our study primarily focuses on evaluating the six frequentist methods for detecting DDI signals based on a simulation study, we believe it would be more appropriate to present real data examples in a consistent manner. Nevertheless, the utilization of negative controls would be a valuable approach for detecting DDI signals, as well-described in the study by Kontsioti et al. [16].
An important consideration is that spontaneous reporting system databases may result in bias due to underreporting. Since underreporting information is not included in the collected data, it cannot be considered in the analyses. Thus, it is likely to have an impact on DDI signal detection. None of the six methods introduced in this paper adjust for the underreporting bias. We should mention that regardless of the methods used, the underreporting bias and other research limitations of spontaneous reporting databases, such as the Weber effect [19], notoriety effect [20], ripple effec t [20], and masking effect [21], cannot be eliminated [22]. It would be very valuable to develop a DDI signal detection method that can account for the bias. Also, further simulation studies will be needed to investigate the effect of the underreporting bias.
The aforementioned methods have the advantage of being easy to apply and easy to calculate, but the sole use of individual method may not be adequate. It is recommended to use several methods together, rather than just one, to make a reasonable decision. Also, pharmacological mechanisms for drug-drug interactions should be considered. The DDI signal detection considering clinical aspects may further improve the limitations of statistical DDI signal detection methods.
References
- 1. Masnoon N, Shakib S, Kalisch-Ellett L, et al. What is polypharmacy? A systematic review of definitions. BMC Geriatrics 2017; 17(1):1–10.
- 2. Norén GN, Sundberg R, Bate A, et al. A statistical methodology for drug–drug interaction surveillance. Statistics in Medicine 2008; 27(16):3057–3070. pmid:18344185
- 3. Gosho M, Maruo K, Tada K, et al. Utilization of chi-square statistics for screening adverse drug-drug interactions in spontaneous reporting systems. European journal of Clinical Pharmacology 2017; 73(6):779–786. pmid:28280890
- 4. Evans SJ, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and Drug Safety 2001; 10(6):483–486. pmid:11828828
- 5. Wang X, Li L, Wang L, et al. Propensity score‐adjusted three‐component mixture model for drug‐drug interaction data mining in FDA Adverse Event Reporting System. Statistics in Medicine 2020; 39(7):996–1010. pmid:31880829
- 6. Noguchi Y, Aoyama K, Kubo S, et al. Improved detection criteria for detecting drug-drug interaction signals using the proportional reporting ratio. Pharmaceuticals 2021; 14(1):4.
- 7. Thakrar BT, Grundschober SB, Doessegger L. Detecting signals of drug–drug interactions in a spontaneous reports database. British Journal of Clinical Pharmacology 2007; 64(4):489–495. pmid:17506784
- 8. Heba I, Amany A, Ahmed SE, et al. Novel data-mining methodologies for detecting drug-drug interactions: a review of pharmacovigilance literature. Advances in Environmental Sciences, Development and Chemistry 2014; 20:301–14.
- 9. Noguchi Y, Tachi T, Teramachi H. Review of statistical methodologies for detecting drug–drug interactions using spontaneous reporting systems. Frontiers in Pharmacology 2019; 10:1319. pmid:31780939
- 10. Qian Y, Ye X, Du W, et al. A computerized system for detecting signals due to drug–drug interactions in spontaneous reporting systems. British Journal of Clinical Pharmacology 2010; 69(1):67–73. pmid:20078614
- 11. Tatonetti NP, Fernald GH, Altman RB. A novel signal detection algorithm for identifying hidden drug-drug interactions in adverse event reports. Journal of the American Medical Informatics Association 2012; 19(1):79–85. pmid:21676938
- 12. Noguchi Y, Ueno A, Otsubo M, et al. A new search method using association rule mining for drug-drug interaction based on spontaneous report system. Frontiers in Pharmacology 2018; 9:197. pmid:29593533
- 13. Zhang P, Li M, Chiang CW, et al. Three‐component mixture model‐based adverse drug event signal detection for the adverse event reporting system. CPT: Pharmacometrics & Systems Pharmacology 2018; 7(8):499–506. pmid:30091855
- 14. Noguchi Y, Tachi T, Teramachi H. Comparison of signal detection algorithms based on frequency statistical model for drug-drug interaction using spontaneous reporting systems. Pharmaceutical Research 2020; 37(5):1–10. pmid:32356247
- 15. Almenoff JS, MuMouchel W, Kindman LA, et al. Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiology & Drug Safety 2003; 12: 517–521. pmid:14513665
- 16. Kontsioti E, Maskell S, Dutta B, et al. A reference set of clinically relevant adverse drug-drug interactions. Scientific Data 2022; 9:72. pmid:35246559
- 17. Susuta Y, Takahashi Y. Safety risk evaluation methodology in detecting the medicine concomitant use risk which might cause critical drug rash. Japanese Journal of Pharamacoepidemiology 2014; 19(1):39–49.
- 18. Charbit B, Samain E, Merckx P, et al. QT Interval Measurement: Evaluation of Automatic QTc Measurement and New Simple Method to Calculate and Interpret Corrected QT Interval. Anesthesiology 2006; 104:255–260. pmid:16436843
- 19. Hartnell NR, Wilson JP. Replication of the Weber effect using postmarketing adverse event reports voluntarily submitted to the United States Food and Drug Administration. Pharmacotherapy 2004; 24:743–9. pmid:15222664
- 20. Pariente A, Gregoire F, Fourrier-Reglat A, et al. Impact of safety alerts on measures of disproportionality in spontaneous reporting databases: the notoriety bias. Drug Safety 2007; 30:891–898. pmid:17867726
- 21. Wang HW, Hochberg AM, Pearson RK, et al. An experimental investigation of masking in the US FDA adverse event reporting system database. Drug Safety 2010; 33:1117–1133. pmid:21077702
- 22. Noguchi Y, Tachi T, Teramachi H. Detection algorithms and attentive points of safety signal using spontaneous reporting systems as a clinical data source. Briefings in Bioinformatics 2021; 22(6):1–14. pmid:34453158