Estimating the Power of Indirect Comparisons: A Simulation Study

Background Indirect comparisons are becoming increasingly popular for evaluating medical treatments that have not been compared head-to-head in randomized clinical trials (RCTs). While indirect methods have grown in popularity and acceptance, little is known about the fragility of confidence interval estimations and hypothesis testing relying on this method. Methods We present the findings of a simulation study that examined the fragility of indirect confidence interval estimation and hypothesis testing relying on the adjusted indirect method. Findings Our results suggest that, for the settings considered in this study, indirect confidence interval estimation suffers from under-coverage while indirect hypothesis testing suffers from low power in the presence of moderate to large between-study heterogeneity. In addition, the risk of overestimation is large when the indirect comparison of interest relies on just one trial for one of the two direct comparisons. Interpretation Indirect comparisons typically suffer from low power. The risk of imprecision is increased when comparisons are unbalanced.


Introduction
In recent years, the adjusted indirect comparisons method, first suggested by Bucher et al. [1], has been widely used to compare competing treatments in the absence of direct evidence about their relative performance. [2] For instance, if two treatments B and C are compared against a common comparator, treatment A, via two distinct sets of randomized trials, this method can be used to derive an indirect estimate of the relative effect of B versus C on the basis of the direct estimates of the relative effects of B versus A and C versus A.
For the adjusted indirect method, it is generally well understood that the precision of the resulting indirect estimate of the relative effect of B versus C is lower than that of the direct estimate that would have been obtained if direct evidence from trials comparing B and C head-to-head were available. 3 Indeed, under certain assumptions, it has been established that an indirect estimate of B versus C would have to be based, on average, on four times as many trials than a direct estimate to achieve the same precision as the direct estimate. [3] These assumptions are as follows: (i) all within-study variances are (approximately) equal within and across pair-wise comparisons of treatments, (ii) between-study vari-ances are (approximately) equal across pair-wise comparisons of treatments and (iii) each pair-wise comparison of treatments includes an equal number of trials.
Wells et al. [4] have investigated in great detail the mean squared error properties of the indirect point estimation of the relative effect of B versus C by means of a simulation study. However, to our knowledge, there have been no attempts in the literature to expand the scope of this investigation to the study of the risk of overestimation as well as the properties of confidence interval estimation and hypothesis testing regarding the relative effect of B versus C.
The power of indirect comparisons to detect differences in treatment effects, if they exist, is a particularly important one for clinical practice. In settings where the direct evidence available for the comparison of B versus A is sparse relative to that available for the comparison of C versus A, we need to understand the extent to which the indirect comparison of B versus C may be underpowered. Intuitively, if the direct comparison of B versus A is under-powered, we would also expect the indirect comparison of B versus C to be under-powered, as it relies on the direct comparison of B versus A in addition to that of C versus A.
In this paper, we present the results of a simulation study that examines the performance of the following aspects concerning the indirect inference on the relative effect of B versus C: (i) overestimation associated with point estimation of the indirect estimate of B versus C (ii) coverage of confidence intervals for the relative effect of B versus C, (iii) type I error of tests of hypotheses concerning the relative effect of B versus C and (iv) power of tests of hypotheses concerning the relative effect of B versus C. Our study focuses on effects expressed on the odds ratio scale, though it could be easily extended to effects expressed on different scales. [4] We start by explaining the Bucher method. We then describe the design of our simulation study and present its results. We conclude by discussing the practical implication of the findings of this simulation study.

Adjusted indirect comparisons
In many situations, we are interested in assessing the relative effects of three different treatments -A, B and C -on the basis of randomized trials that have compared B against A and C against A, but not B against C.
In the absence of direct evidence for the comparison of B against C, the adjusted indirect method provides a convenient way to conduct inferences on the relative effect of B versus C based on the point estimates of the relative effects of B versus A and C versus A and their associated standard errors. While these relative effects can be expressed on any suitable way for the data produced by the trials of B versus A and C versus A, we briefly explain below how the method works for the case where these data are binary in nature and the relative effects are expressed on the odds ratio scale.
Let The 95% confidence interval for OR BC produced by the Bucher method can be used to test the null hypothesis H 0 : OR BC~1 versus H a : OR BC =1. If this interval precludes the value 1 (which denotes a null relative effect of B compared to C), we reject the null hypothesis and conclude that the effect of B is significantly different from that of C (based on two-sided a = 5%). However, if this interval includes the value 1, we fail to reject the null hypothesis and conclude that the data do not provide sufficient evidence in favour of a difference in the effects of the two treatments.
In practice, the use of random-effects meta-analysis is typically recommended for deriving both (

Generation of simulated data
Our simulation study was geared at the indirect comparison of two drugs B and C, which were compared head-to-head against another drug A, but not against each other. In this study, the direct comparisons of B versus A and C versus A were performed on the basis of trials with a binary outcome for each trial participant (i.e., participant experienced/did not experience the event of interest). For this reason, the true relative effects of B versus A and C versus A were expressed on the odds ratio scale as OR AB and OR AC . Similarly, the true relative effect of B versus C, which was of primary interest, was expressed on the odds ratio scale as OR BC .
Using Bucher's adjusted indirect comparison as a basis for conducting inferences on OR BC , the simulation study was concerned with answering the following questions: (I) What is the risk of over-estimation associated with the point estimation of OR BC ?
(II) What are the coverage properties of the confidence interval estimation method of OR BC ?
(III) What are the Type I error properties of the test of hypotheses H o : OR BC~1 (null relative effect of C versus B) versus H a : OR BC =1 (non-null relative effect of C vs. B)?
(IV) What are the power properties of the test of hypotheses H o : OR BC~1 versus H a : OR BC =1?
The simulation study included six different factors but was not set up as a full factorial experiment. These factors were: (1) k AB , the number of trials pertaining to the B versus A comparison; (2) k AC , the number of trials pertaining to the C versus A comparison; (3) p A , the true average event rate in the common comparator group A; (4) OR AB , the true relative effect of B versus A, quantified as an odds ratio; (5) OR AC , the true relative effect of C versus A, quantified as an odds ratio; (6) t, the between-study standard deviation, assumed constant across the comparisons B versus A and C versus A.
Given these factors, we explored the extent to which the performance of the indirect inference on OR BC would be influenced by the size of k AB and k AC , especially in situations where k AC would either be equal to 1 or larger than 1 but much smaller than k AB . However, we also considered the influence of the remaining factors on the indirect inference on OR BC .
In view of the above, we focused our attention on a limited number of combination of values for the factors p A , OR AB and OR AC , while allowing k AB to take on the values 5, 10, 25 and 100, k AC to take on the values 1 and 5, and the heterogeneity parameter t to take the following values: 0.001 (small betweenstudy heterogeneity), 0.2 (moderate between-study heterogeneity) and 0.4 (large between-study heterogeneity). The combinations of values entertained for p A , OR AB and OR AC are listed in Table 1. Given any such combination of values, the resulting simulation experiment had a factorial structure with respect to the remaining factors k AB , k AC and t.
We note the following in connection with the combination of values reported in Table 1 (See Table 1).
Knowing OR AB and OR AC allows the determination of OR BC , the true relative effect of B versus C, via the formula Þ . Using this formula, we can see that: (i) OR BC~1 for those simulation settings where OR AB~O R AC~1 :4; (ii) OR BC~1 :17 for those simulation settings where OR AB~1 :2 and OR AC~1 :4 and (iii) OR BC~1 :15 for those simulation settings where OR AB~0 :65 and OR AC~0 :75.
In addition, if p B and p C denote the true average event rates in groups B and C, respectively, we can determine the value of the former parameter from the values of p A and OR AB and that of the latter parameter from the values of p A and OR AC : Table 1 shows the resulting values of p B and p C corresponding to the combinations of values of p A , OR AB and OR AC given in Table 1 (See Table 1). Based on Table 1, we see that the simulation settings for which OR AB~O R AC~1 :4 have equal true average event rates in groups B and C and that both of these rates are higher than the true average event rate in group A. Simulation settings for which OR AB~1 :2 and OR AC~1 :4 have different true average event rates in groups B and C (with the event rate in group C being slightly higher than that in group B). Both of these rates are higher than the true average event rate in the common comparator group A. Simulation settings for which OR AB~0 :65 and OR AC~0 :75 have a higher average event rate in group C than in group B, with both of these rates being smaller than the average event rate in group A. For each combination of values for the six factors included in the simulation study, we generated 5,000 sets of k AB trials comparing B versus A and k AC trials comparing C versus A and used them as input for conducting indirect inferences on the true relative effect of B versus C. The data for each of the k AB trials consisted of counts of events and number of participants in arms A and B of that trial. Similarly, the data for each of the k AC trials consisted of counts of events and number of participants in arms A and C of that trial. For simplicity, we discuss below only the generation of data from trials comparing B versus A.
Consider the j-th trial comparing B versus A amongst the k AB trials available for this comparison. The data for this trial were generated from the following model: Here, n Aj and n Bj represent the number of participants in arms A and B of the j-th trial comparing B versus A. Under the assumption of equal numbers of participants in both arms (n Aj~nBj ), the total number of participants in the two arms was determined by sampling an integer between 20 and 500 participants.
The number of observed events in group A, e Aj , was drawn from a binomial distribution with parameters n Aj and p Aj , with p Aj denoting the trial specific event rate in group A. The parameter p Aj was drawn from a uniform distribution with support given by The observed number of events in arm B of the j-th trial comparing B versus A, e Bj , was drawn from a binomial distribution with parameters n Bj and p Bj , with p Bj denoting the trial specific event rate in group B. The value of the latter parameter was derived on the basis of p Aj (trial specific event rate in group A) and OR AB, j (trial-specific true relative effect of B versus A, expressed as an odds ratio). The natural logarithm of OR AB, j was sampled from a normal distribution with mean given by ln OR AB ð Þand variance given by t 2 , where t is the betweenstudy standard deviation. The latter specification is consistent with assuming that the relative effects of B versus C are different across trials yet similar enough to be sampled from a common distribution.
Given the data e Aj ,n Aj ,e Bj ,n Bj À Á , j~1, . . . ,k AB , generated for the k AB trials comparing B versus A, a random-effects metaanalysis based on the DerSimonian and Laird method was used to estimate log OR AB ð Þ and its associated standard error. [5] These estimates -along with similarly obtained estimates of log OR AC ð Þ and its corresponding standard error -were used as inputs for the adjusted indirect comparisons method of Bucher.

Measures of performance
The following measures of performance of the indirect inference on OR BC were considered in our simulation study: N Risk of over-estimation; N Confidence interval coverage; N Type I error; N Statistical power.
The risk of overestimation was evaluated only for those simulation settings where (i) OR AB~1 :2 and OR AC~1 :4 (hence OR BC~1 :17) or (ii) OR AB~0 :65 and OR AC~0 :75 (hence OR BC~1 :15). Given a simulation setting, this risk was computed by recording the proportion of times the indirect estimate of OR BC exceeded four different thresholds in the 5,000 simulations. The thresholds were selected to represent approximately a 20%, 30%, 50% and 75% increase in the true value of OR BC . Specifically, when OR BC~1 :17, the thresholds were taken to be 1.40, 1.52, 1.75 and 2.05, respectively. When OR BC~1 :15, the thresholds were taken to be 1.38, 1.49, 1.72 and 2.01, respectively.
The confidence interval coverage was assessed for all simulation settings. Given a setting, coverage was evaluated by recording the percentage of simulations out of 5,000 for which the 95% confidence interval of OR BC included the true value of OR BC corresponding to that setting. The type I error of the test ofH o : OR BC~1 against H a : OR BC =1 was evaluated only for those simulation settings with OR AB~O R AC~1 :4 for which the null hypothesis was true (i.e., OR BC~1 ). For each such setting, Type I error was assessed by tracking the percentage of simulations out of 5,000 which produced 95% confidence intervals for OR BC that excluded the value 1.
The statistical power of the test of H o : OR BC~1 against H a : OR BC =1 was computed only for those simulation settings with OR AB~1 :2 and OR AC~1 :4 or OR AB~0 :65 and OR AC~0 :75, for which the null hypothesis was false. For each such setting, power was expressed as the percentage of simulations out of 5,000 which produced 95% confidence intervals for OR BC that excluded the value 1.

Software Implementation
All simulations were performed using the freely available software package R 2.11.0. [6] All random-effects meta-analyses pertaining to the direct comparisons of B against A and C against A were conducted using the R package metafor (version 1.1-0).  Table 2 presents the risk of over-estimation of OR BC for simulation settings where OR AB~1 :2 and OR AC~1 :4 while Table 3 presents the same quantity for those settings where OR AB~0 :65 and OR AC~0 :75.

Risk of over-estimation
When k AC is 1, the true relative effect OR BC is often considerably overestimated. When k AC is 5, the overestimation is both less frequent and pronounced. For both of these values of k AC , the more trials are available for the direct comparison of B versus A (i.e., the larger k AB ), the smaller the risk of overestimation becomes.

Coverage
Tables 4, 5 and 6 present the empirical coverage of the 95% confidence interval estimation method of Bucher for OR BC for simulation settings where OR AB~O R AC~1 :4, OR AB~1 :2 and OR AC~1 :4, and OR AB~0 :65 and OR AC~0 :75, respectively. (The nominal coverage is 95%.) For the all of these settings, the Bucher confidence interval estimation method generally reports empirical coverage values below the nominal coverage when the between-study heterogeneity is moderate or large (i.e., t~0:2 or 0:4) -a phenomenon referred to as undercoverage. As anticipated, the undercoverage tends to be more pronounced when k AC equals 1 than when k AC equals 5. Undercoverage could either be due to bias in the estimates of OR BC or due to underestimation of SE OR BC ð Þ(which would cause the confidence interval to be artificially narrow).
When the between-study heterogeneity is small (i.e., t~0:001), the estimated coverage of is generally either greater or slightly smaller than the nominal coverage, suggesting that the method produces conservative or valid confidence intervals. In particular, coverage exceeding the nominal level indicates that the Bucher method produces overly wide confidence intervals in this scenario. Increasing the value of k AC from 1 to 5 while keeping k AB fixed has a minimal impact on the empirical coverage of the method. When the between-study heterogeneity is moderate (i.e., t~0:2) and especially large (i.e., t~0:4), the estimated coverage is generally smaller than the nominal level. Increasing the value of k AC from 1 to 5 while keeping k AB fixed results in coverages closer to the nominal level, albeit still off by as much as 5%. Table 7 displays the estimated Type I error associated with the test of H 0 : OR BC~1 versus H 0 : OR BC =1 for those simulation settings with OR AB~O R AC~1 :4. (The nominal Type I error is 5%.) For these settings, the estimated Type I error falls below the nominal Type I error when the between-study heterogeneity is small (i.e., t~0:001) but exceeds the nominal Type I error when the between-study heterogeneity is moderate or large (i.e., t~0:2or 0:4). For fixed values of k AB and k AC , the levels of the estimated Type I error increases as the between-study heterogeneity increases. These findings hold for most values of k AB and regardless of whetherk AC equals 1 or 5.   Power Tables 8 and 9 show the estimated power of the test of H 0 : OR BC~1 versus H 0 : OR BC =1 for those simulation settings with either OR AB~1 :2 and OR AC~1 :4, or OR AB~0 :65 and OR AC~0 :75. The results in these two tables show that this test is profoundly underpowered across both types of simulation settings. As expected, when k AB is kept fixed, increasing the value of k AC from 1 to 5 does result in an increase in the level of power, with the magnitude of this increase depending on the value of the betweenstudy standard deviation t. Similarly, when k AC is kept fixed, increasing the value of k AB results in an increase in the level of power. Nevertheless, these increases in power are not large enough to overcome the issue of lack of power.

Discussion
Our study demonstrates that adjusted indirect comparisons are severely affected by the power and fragility of the data in the contributing comparisons. Given the growing acceptance and increased use of indirect comparisons in health-care decisionmaking, there is a need for caution when determining the strength of evidence from indirect comparisons. In particular, health-care decision makers should carefully assess the strength of evidence within each comparison (e.g., A vs B and A vs C) to grasp the reliability of the produced indirect point estimate and confidence interval. [7] There are strengths and limitations to consider when interpreting our simulation study. Strengths of this study include the use of clinically reasonable assumptions about treatment effects and the simulation of varying scenarios of clinical importance versus statistical importance. Further, we explored inferential properties for the simplest form of indirect comparison (A vs B and A vs C). Such comparisons are present in multitude in more complex indirect comparisons and multiple treatment comparisons (MTC). To a considerable extent, our results may therefore extrapolate beyond the simulated scenarios as the underlying statistical assumptions used in MTC are similar. [8,9] The limitations of our study include the overarching issue that we used simulations rather than real data for our analysis. We investigated the impact of the number of direct comparison trials on various statistical properties of an indirect comparison while allowing the sample size of each direct trial to follow a uniform distribution from 20 to 500. This setup ensured that our simulation scenarios are representative of real-world meta-analytic situations, where trials pertaining to a direct comparison typically vary in their sample sizes. However, our ability to reproduce such situations came with a price: we were unable to assess the effect of the trial sample size on the power of an indirect comparison, due to its confounding with the other factors examined in our simulation study, such as heterogeneity. Furthermore, we assessed the risk of overestimation, confidence interval coverage and statistical power of an indirect comparison involving two treatments, but we did not examine these statistical features for the direct comparison involving the same two treatments. One reason for this is that, in practice,   indirect comparisons are performed specifically when direct comparisons cannot be performed due to a lack of direct evidence. While it is possible to expand our simulation study to include a comparison of the statistical properties of direct and indirect comparison concerning the same treatments, we chose not to pursue this here in an effort to preserve the simplicity of our findings and interpretations. We hope to address this issue in a future paper. We used the DerSimonian-Laird random-effects model which makes use of the DerSimonian-Laird estimator to estimate the between-study variation. This estimator has been known to underestimate the between-study variance.[ 10 11, 12]Thus, the undercoverage and inflation of type I error we detected in simulation scenarios with moderate or large heterogeneity may in part be caused by properties of this estimator rather than properties of the Bucher adjusted indirect comparison method. [1] The use of indirect comparisons and MTC analyses is growing in popularity in both journal publications and by health technology assessments. [2] The criticisms of both approaches is that it is not obvious where biases or errors may arise from, including issues of individual trial bias, trial-level differences across comparisons, and problems in the conduct of the indirect model. [13] Authors and readers appear to have difficulty interpreting the quality of indirect comparison meta-analysis and tools for critical appraisal do not yet exist. [14] Our study demonstrates that caution is warranted, especially in situations where low numbers of trials are included in any treatment arm. Insights from empirical studies are crucially needed to further inform this issue. Further, we hope investigate the fragility and power associated with point estimation and hypothesis testing in MTC in a near future.
In conclusion, indirect comparisons with 1 or 5 trials in one of the indirect comparison arms are consistently underpowered (power ,20%), regardless of the number of trials in the other indirect comparison arm. Results from indirect comparisons may especially become unreliable with the heterogeneity is moderate or large. Authors and readers of indirect comparisons should exercise caution and scepticism when interpreting results from indirect comparisons.