A simulation-based comparison of drug-drug interaction signal detection methods

Several statistical methods have been proposed to detect adverse drug reactions induced by taking two drugs together. These suspected adverse drug reactions can be discovered through post-market drug safety surveillance, which mainly relies on spontaneous reporting system database. Most previous studies have applied statistical models to real world data, but it is not clear which method outperforms the others. We aimed to assess the performance of various detection methods by implementing simulations under various conditions. We reviewed proposed approaches to detect signals indicating drug-drug interactions (DDIs) including the Ω shrinkage measure, the chi-square statistic, the proportional reporting ratio, the concomitant signal score, the additive model and the multiplicative model. Under various scenarios, we conducted a simulation study to examine the performances of the methods. We also applied the methods to Korea Adverse Event Reporting System (KAERS) data. Of the six methods considered in the simulation study, the Ω shrinkage measure and the chi-square statistic with threshold = 2 had higher sensitivity for detecting the true signals than the other methods in most scenarios while controlling the false positive rate below 0.05. When applied to the KAERS data, the two methods detected one known DDI for QT prolongation and one unknown (suspected) DDI for hyperkalemia. The performance of various signal detection methods for DDI may vary. It is recommended to use several methods together, rather than just one, to make a reasonable decision.


Introduction
Polypharmacy, the use of multiple medicines has increased as the average life expectancy and the prevalence of multimorbidity has increased [1].Adverse events (AEs) caused by the administration of many drugs at the same time are therefore a serious concern.These suspected adverse drug reactions (ADRs) due to drug-drug interaction (DDI) can be discovered through post-market drug safety surveillance (PMS).Spontaneous reporting systems (SRSs) are databases used for PMS that include ADR reports and prescription information (e.g, sex, age, date, quantity, etc).By investigating SRS databases using data mining tools, we can identify signals and prevent the potential ADRs induced by DDI.Generally, quantitative DDI signals refer to excessive risk for a combination of two drugs compared with the risks for the individual drugs.However, the criteria used in each method to define signals are different and have various pros and cons.
Several studies have proposed approaches to detect signals indicating drug-drug interactions as well as single-drug adverse reactions.Nore ´n et al. proposed the O shrinkage measure to screen for disproportional reporting indicative of suspected drug-drug interaction [2].Gosho et al. proposed the chi-square statistic as a statistical criterion for detecting drug-drug interaction signals and compared this method with Nore ´n et al.'s O shrinkage measure using a simulation study [3].The proportional reporting ratio (PRR) proposed by Evans et al [4] was used to detect signals indicating single-drug reactions, then was extended for drug-drug interactions by Wang et al. [5].The concomitant signal score (CSS), proposed by Noguchi et al. [6], is an improved detection method using the PRR.Thakrar et al. proposed the additive and multiplicative models [7].
Some studies have reviewed the methodologies [8,9].Other studies applied the statistical models to real world data, and then compared the results [10][11][12][13][14][15][16].Only a few studies used simulation to evaluate the performance of a few methods [3].The main purpose of this study is to assess the performance of various detection methods for DDI through a simulation study.We evaluated the methods in terms of false positive rate and sensitivity in various scenarios.Note that our study exclusively focused on employing frequentist methods for detecting DDI signals.It's noteworthy to mention that there exists a well-known Bayesian model named Interaction Signal Score (IntSS) [15,16] for this purpose.
In the next section, we provide an overview of existing methods and explain how the simulation study was implemented.We also introduce the Korea Adverse Event Reporting System (KEARS) data (2017-2019) from the Korea Institute of Drug Safety and Risk Management (KIDS).In the following section, we present the performance of the signal detection algorithms for DDI from the simulation study.The DDI signal detection results using the KEARS data are also shown.We present a discussion in the final section.

Methods
The signal detection methods are fundamentally based on the observed AE frequencies according to exposure status of two drugs as presented in Table 1.Let r 00 denote the observed reporting rate for the AE in the absence of both Drug 1 and Drug 2. Similarly, r 10, r 01 , and r 11 are the observed reporting rate (i) with Drug 1 but not Drug 2, (ii) with Drug 2 but not Drug 1, and (iii) with concomitant use of the two drugs, respectively.

O shrinkage method
Nore ´n et al. [2] proposed the O shrinkage method.The method basically compares the observed reporting rate r 11 with its expected value E[r 11 ] estimated under the assumption that there is no interaction between the two drugs.The estimator s 11 of E[r 11 ] is given as A shrinkage factor is considered to adjust spurious associations due to a very small value of s 11 by reason of generally very rare ADRs by DDI.The O shrinkage measure is defined as where n 11 = n 111 + n 110 and α is a tuning parameter determining the shrinkage strength.Generally, α is set to 0.5.The lower limit of the 95% confidence interval for O can be estimated as ffi ffi ffi ffi ffi ffi ffiffi n 111 p where ϕ(0.975) is the 97.5th percentile of the standard normal distribution.The criterion O 25 > 0 is used to determine the DDI signal.

Chi-square statistic method
Gosho et al. [3] proposed the chi-square statistic model in order to reduce the false positive rate when events are rare.The measure of the chi-square statistic method χ is the square root of the chi-square test statistic with a correction term to adjust for false positives.The threshold χ > 2 and χ > 2.6 is set for identifying DDI signals.These cutoff values are set based on the 95th and 99th percentiles, respectively, of the chi-square distribution with one degree of freedom.

Proportional reporting ratio (PRR)
The proportional reporting ratio (PRR) proposed by Evans et al. [4] is commonly used in disproportionality analysis to detect adverse event induced by a single drug.The PRR is the ratio of observed reporting rate with and without a drug.The PRR has been extended to drug-drug interactions [5].First, PRR D1 for Drug1 and PRR D1 for Drug2 are defined as where n 10 = n 101 + n 100 , n 00 = n 001 + n 000 , and n 01 = n 011 + n 010 .The PRR D1D2 for concomitant use of Drug1 and Drug2 is defined as The lower limit of the 95% confidence interval for the PRR is used to define a signal.It is calculated as where SD stands for the standard deviation.The SD for Drug1 is SD D1 ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi , the SD for Drug2 is SD D2 ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffiffi ffi ffi ffi ffi ffi ffi ffi ffi ffi  , and the SD for the Drug1-Drug2 pair is SD D1D2 ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffiffi ffi ffi ffi ffi . The signal detection criterion is to compare the lower limit of the 95% confidence interval for a single drug and drug-drug pairs.If PRR 025D1D2 > max (PRR 025D1 , PRR 025D2 ), then the drug pair is considered to be a signal of DDI.

Concomitant signal score (CSS)
The concomitant signal score, proposed by Noguchi et al. [6], was shown to improve the combination risk ratio (CRR) [17], a DDI detection method using PRR.The weakness of CRR is that the lower limit of the 95% CI of PRR D1D2 overlaps with the upper limit of the 95% CI of PRR D1 or PRR D2 .This is because adverse event reports involving individual drugs are more common than reports concerning the concomitant use of two drugs.The concomitant signal score (CSS) is the ratio of PRR 025D1D2 and the maximum value between PRR 975D1 and PRR 975D2 .

Additive model
Thakrar et al. considered both an additive model and a multiplicative model for the detection of DDI signals [7].The additive model assumes that the risk associated with a drug adds to the background risk.Under the additive assumption, no interaction is established when the excess risk associated with the drug combination is the same as the sum of the excess risks associated with each exposure in the absence of the other.The risk difference is defined as RD D1D2 = p 11p 00 , RD D1 = p 10 -p 00 , and RD D2 = p 01 -p 00 .When RD D1D2 > RD D1 + RD D2 , the signal of an interaction is detected.Using the linear probability model in the below, we test for the significance of the interaction term.
When β 3 is statistically significantly greater than 0, there is a potential DDI.A positive value of β 3 signifies a positive interaction, namely, an elevated risk for the combination of Drug 1 and Drug 2 compared to that expected based on the individual drugs.

Multiplicative model
The multiplicative model assumes that the risk associated with a drug multiplies with the background risk.The risk ratio is defined as RR D1D2 ¼ p 11 =p 00 RR D1 ¼ p 10 =p 00 , and RR D2 ¼ p 01 =p 00 : When RR D1D2 > RR D1 � RR D2 , a signal is detected.To test for the interaction effect based on the multiplicative model, we implement the log linear regression model in the below.
When β 3 is statistically significantly greater than 0, there is a potential DDI.As in the additive model, a positive value of β 3 indicates a positive interaction.

Korea Adverse Event Reporting Systems (KAERS) data
We applied the six DDI signal detection methods to Korea Adverse Event Reporting System (KAERS) data from the Korea Institute of Drug Safety and Risk Management (KIDS) in 2017-2019.Drug information in KAERS data was documented by the ingredient names with ATC (Anatomical Therapeutic Chemical) codes (https://www.whocc.no/atc_ddd_index/),and AE details were recorded with WHO-ART (WHO Adverse Reaction Terminology) codes (https:// www.who-umc.org),eliminating the need for an additional mapping process.We focused on two AEs of QT interval prolongation and hypokalemia.Known interactions for the two AEs were derived from a research report published by the Health Insurance Review and Assessment Service (HIRA) on the adverse event monitoring system, and the list of contraindications of co-medication drugs was derived from KIDS (as of Dec. 28, 2020).In addition, suspected interaction pairs for the same AEs were searched from the combinations of drug-drug-AE that have high frequencies in KAERS data.According to the HIRA research report, it has been reported that co-administration of potassium chloride and spironolactone and co-administration of tactolimus and spironolactone induce hyperkalemia.Potassium chloride is prescribed for potassium-deficiency, electrolyte imbalance, and digitalis poisoning.Spironolactone is a diuretic.Tactolimus is an intensive immunosuppressant agent used to prevent transplant rejection.Co-administration of domperiodone and amiodarone has been reported to cause QT prolongation.Prolongation of the QT interval is associated with an increased risk of development of a potentially lethal cardiac arrhythmia called torsade de pointes, a risk that increases with the administration of a QT-prolonging drugs [18].Domperidone is a drug that increases gastrointestinal motility and is prescribed for indigestion and vomiting.Amiodarone is an antiarrhythmic drug.

Simulation study
Tables 2 and 3 present the false positive rate and sensitivity of the six methods under the additive assumption.The false positive rate of the O method ranged from 0.001 to 0.055.The false positive rate of the chi-square method with threshold = 2 was similar to that of the O method: between 0.001 and 0.066.The false positive rate of the chi-square method with threshold = 2.6 showed the smallest variation, ranging from 0.000 to 0.029, while the PRR method showed a wide range of false positive rate, ranging from 0.006 to 0.127.The O method and the chisquare method with threshold = 2 controlled the false positive rate below 0.05 and had high sensitivity in most scenarios.In particular, when the number of events is small (n 111 < 2), the chi-square method has a higher sensitivity than the O method.The CSS method showed the lower false positive rate than the O method or the chi-square method, but it also showed the lower sensitivity.This may be due to the lower threshold for the measure of the CSS method to reach an interaction.Comparing the additive and the multiplicative model, when there is no effect of each single drug or there is an effect of Drug2 only (scenarios (2-1) and (2-2)), the multiplicative model showed the higher sensitivity than the additive model.On the other hand, when there is an effect of Drug1 and Drug2 (scenarios (2-3) and (2-4)), the additive model showed the higher sensitivity than the multiplicative model.
Comparing scenarios (2-3) and (2-4), most methods except the multiplicative model showed the higher sensitivity in the scenario (2)(3)(4).That is, when the difference between the effects of the two drugs is large, the sensitivity increases.
The false positive rate and sensitivity of the six methods under the multiplicative assumption are presented in Tables 4 and 5.In scenario 3 comparing false positive rate, scenarios (3-1) and (3-2) are the same with scenarios (1-1) and (1-2) because they satisfy both the additive assumption and multiplicative assumption.In scenario 4 comparing sensitivity, scenarios (4-1), (4-2), and (4-3) are equivalent to scenarios (2-1), (2-2), and (2-3) because they also satisfy both the additive assumption and multiplicative assumption.It can be shown that satisfying the multiplicative assumption will imply satisfying the additive assumption, thus the additive model has the lower threshold for incidence f to reach an interaction, The detection signals of the multiplicative model mean stronger interactions than those of the additive model.By comparison, the additive model can detect the DDI signal earlier.The multiplicative model can help determine whether there is stronger evidence of DDI after checking DDI with the additive model.

KAERS data
A total of 1,131,985 reports were taken from the KAERS.Among them, there were 1656 cases of hyperkalemia and 284 cases of QT prolongation.The observed reporting rate for the exposure status of each drug-drug pair was summarized in Table 6.The measure of the six methods for the six drug-drug-AE combinations was summarized in Table 7.A known interaction for QT prolongation, domperidone -amiodarone, was detected as a signal by the O method, the chi-square method, the PRR method, the CSS method, and the additive model.The number of reports of domperidone -amiodarone combination was very small (n 11 = 18), but its reporting proportion of the AE was relatively high (4/18).The multiplicative model showed a positive interaction trend for domperidone -amiodarone combination, but it did not reach the statistical significance (p-value = 0.185).For known interactions, only one drug-drug pair (domperidone-amiodarone) was detected.The reason for the reduced sensitivity may be that the number of cases is very small because it is rare to prescribe combinations that are known to have side effects when taken together.
Acetylsalicylic acid -polystyrene sulfonate, a suspected interaction for hyperkalemia, was detected as a potential signal by the O method, the chi-square method, and the PRR method.The reporting proportion of hyperkalemia for acetylsalicylic acid -polystyrene sulfonate combination was 22/459.While the combination of acetylsalicylic acid and polystyrene sulfonate was not a previously known interaction for hyperkalemia, DDI signals were detected among the suspected combinations that has a large number of AE reports.Only three of the six methods identified DDI signals.

Discussion
As the number of patients with chronic disease becomes more common, the co-prescription of multiple drugs has increased.Therefore, it has become more important to identify combinations of drugs that have side effects through post-market drug safety surveillance.In this article, we examined statistical methodologies for DDI signal detection.Of the six methods, the O shrinkage method and the chi-square method showed the best performance.The O shrinkage method and the chi-square method with threshold = 2 controlled the false positive rate below 0.05 and had high sensitivity in most scenarios.The chi-square method was especially effective when there were a very small number of reports for an AE.The chi-square method with threshold = 2.6 and the additive model seemed rather conservative.They rigorously controlled the FPR, but they had lower sensitivity than the O shrinkage method and the chi-square method with threshold = 2.
While we undertook a comprehensive simulation study and presented a real data example, there are some limitations in this study.For the simulation study, we assumed the number of cases involving both drugs (n 11 ) to be 10,000, which is somewhat large.However, when considering the scenario of a very low reporting probability rate for AEs, such as 0.005%, it can be deemed a reasonable value.Nevertheless, it would be beneficial to perform an additional simulation study, assuming smaller values for n 11 .We presented the performance of the six methods based on FPR and sensitivity at a fixed threshold for each method.It is worth considering the feasibility of evaluating the methods using multiple possible thresholds through ROC curves and AUCs.For the KAERS data, we evaluated only two AEs of QT interval prolongation and hypokalemia.Additionally, we did not consider the approach involving negative controls for the real data example.As our study primarily focuses on evaluating the six frequentist methods for detecting DDI signals based on a simulation study, we believe it would be more appropriate to present real data examples in a consistent manner.Nevertheless, the utilization of negative controls would be a valuable approach for detecting DDI signals, as well-described in the study by Kontsioti et al. [16].An important consideration is that spontaneous reporting system databases may result in bias due to underreporting.Since underreporting information is not included in the collected data, it cannot be considered in the analyses.Thus, it is likely to have an impact on DDI signal detection.None of the six methods introduced in this paper adjust for the underreporting bias.We should mention that regardless of the methods used, the underreporting bias and other research limitations of spontaneous reporting databases, such as the Weber effect [19], notoriety effect [20], ripple effec t [20], and masking effect [21], cannot be eliminated [22].It would be very valuable to develop a DDI signal detection method that can account for the bias.Also, further simulation studies will be needed to investigate the effect of the underreporting bias.
The aforementioned methods have the advantage of being easy to apply and easy to calculate, but the sole use of individual method may not be adequate.It is recommended to use several methods together, rather than just one, to make a reasonable decision.Also, pharmacological mechanisms for drug-drug interactions should be considered.The DDI signal detection considering clinical aspects may further improve the limitations of statistical DDI signal detection methods.