1 Introduction

With the dawn of precision medicine and targeted antibody therapies, the wish for more flexible trial designs compared to randomized controlled trials has been emphasized in both clinical research and methodology. While randomized controlled trials are still the gold standard of clinical research owing to their high internal validity, in some contexts they may be costly, slow, unethical, or simply not feasible. The term master protocols summarizes more flexible trial designs that combine features such as the addition of promising new treatment arms, the removal of ineffective treatment arms, and the combination of arms with different endpoints. A commonly requested idea is testing the same treatment in several substrata of a patient cohort. Such a master protocol, i.e., a design that unifies several strata in a single clinical trial, is called basket trial design. This unification streamlines the planning phase of the different strata and parallelizes their recruitment, resulting in an efficient use of resources. Basket trial designs are beginning to be picked up in practice. A systematic literature review conducted on February 20, 2023, in MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials found 146 oncology-related basket trials [1]. Especially in early stages of clinical research, these designs also offer statistical benefits: So-called borrowing techniques allow strata with similar responses to the treatment to share information with one another, thereby leveraging power despite small sample sizes while keeping type-I error rates only moderately inflated. The earliest publication known to us that suggests borrowing between strata was published in 2003 [2] and the last ten years showed a colorful bouquet of Bayesian and frequentist borrowing mechanisms being introduced to methodological research [3].

Borrowing usually depends on the choice of several tuning parameters. An optimal choice of tuning parameters has to offer a compromise between multiple components: achieving high power in responsive strata, keeping type-I error rates low in unresponsive strata and maintaining a good balance between these two measures across several response scenarios. This compromise can be quantified using utility functions. The optimal choice of tuning parameters can then be found by finding optimal utility function values with the help of numerical optimization algorithms. In the following document, we present the protocol of a comparison study planned to investigate utility-based optimization of basket trial designs using both simulations and numerical calculations. The basket trial design that we are considering as an example is a Bayesian design introduced by [4].

2 Methodology of utility functions in basket trial designs

In the statistical planning of clinical trials, the communication of type-I error rate (TOER) and power to stakeholders such as principal investigators, sponsors, ethical committees and regulatory authorities is essential. TOER is the probability of rejecting the null hypothesis conditional on the assumption that the null hypothesis is true. Power is the probability of rejecting the null hypothesis conditional on the assumption that some alternative hypothesis of interest holds. While these measures are purely frequentist in nature, they may also be requested during the planning of Bayesian trial designs.

In the planning of basket trial designs, this demand for TOER and power calculation is confronted with several challenges. Firstly, every stratum may have its own null and alternative hypothesis of interest so that a multitude of combinations of null and alternative hypotheses across baskets can be considered. Secondly, control of TOER in a scenario may not be possible if we want to employ borrowing in order to leverage power. [5] proved that power increase always comes at the cost of TOER inflation in the context of borrowing from external data sources whenever a uniformly most powerful test exists. While we are not aware of a formal transfer of their argument to the context of basket trials, it is plausible that the argument holds in that setting as well.

In communication with stakeholders, the best practice may be to communicate both TOER and power for each stratum across a range of plausible scenarios combining null and alternative hypotheses in different strata.

However, when searching for the optimal choice of tuning parameters of a basket trial designs we need to combine TOER and power across strata and scenarios. A natural way of combining these values is by defining an appropriate utility function. Then, optimization algorithms can be employed in order to find the optimal tuning parameter vector. Constraints, e.g. the maximally tolerated TOER values, may either be incorporated into the utility function allowing for unconstrained optimization methods or may be set up as separate inequalities, asking for constrained optimization approaches. This utility-based optimization approach has already been employed in the context of adaptive designs [6] and in the context of planning several stages of drug development [7] even though formal control of TOER is possible in these contexts. In the methodology of basket trial designs, we are aware of a first utility-based approach presented in [8]. Their approach will also be considered in the presented comparison study protocol and will be supplemented by several other optimization algorithms and utility functions.

3 Goals of this comparison study

This comparison study’s goal is to find optimal tuning parameter combinations for Fujikawa’s basket trial design in a general framework that could subsequently be generalized to other basket trial designs as well. In particular, the study is divided into three parts addressing three questions related to finding optimal parameter combinations:

1. Which type of optimization algorithm for finding the optimal tuning parameter vector should be preferred in terms of runtime and reliability?
2. What is a good definition of optimal tuning parameter vector that takes the desire for maximizing the detection probability of active strata as well constraints on TOER into account while delivering favorable results across a range of outcome scenarios? This question amounts to finding an appropriate utility function. We will use the best algorithm found in Part I, and apply it to a variety of different utility functions.
3. How does the optimal tuning parameter vector found as a result of parts I and II perform in comparison to the tuning parameter combinations suggested in [4]?

The document will begin with a brief introduction into basket trial designs in general and into Fujikawa’s design in particular. Afterwards, we will explain the true scenarios of interest in our study as well as the considered utility function utility functions and optimization algorithms. Then we will present the pre-specified plan of the three parts of the study structured using the ADEMP scheme by [9]. Part I is a simulation study for comparing different algorithms, parts II and III are comparison studies where all measures of interest can be calculated exactly.

4 Basket trial designs

A basket trial design is a clinical trial design used primarily in oncological single-arm phase II studies. It tests the same null hypothesis in several strata. In the literature, either the ensemble of all strata together is called “basket” or the strata themselves are called “baskets”. In the following, the endpoint will always be binary. While this could be any binary endpoint, we will without loss of generality refer to response to a treatment vs. no response throughout the text. Strata that have a sufficiently high true response rate to the treatment are called active, otherwise they are called inactive. In particular, the primary comparison is against a specified target rate and not against a control group. As no control group exists, the designs we consider are stratified single-arm trials and not multi-arm trials.

Consider the design from [4], which is based on an alteration of the beta-binomial model. Using the notation from [3], it can be defined as follows: Let n_i resp. r_i be the number of patients resp. responders in stratum for some number of strata I. The sampling distribution is simply the binomial distribution

where the true rate p_i follows a prior beta distribution with shape parameters ,

For analysis of data, Fujikawa et al. recommend an uninformative choice of prior distribution. We choose . By conjugacy, the usual posterior distribution would be

(1)

Now this posterior is altered by introducing a borrowing mechanism,

(2)

Here is a similarity measure defined by , where we have , , and − with the Jensen-Shannon divergence of the unaltered beta-binomial posterior distributions from Eq 1. Note that in the original publication [4], the bound for is set to . However, there is no mathematical or design-related reason to not allow values between 0 and 1. is a measure of divergence of probability distributions which implies that becomes a measure of similarity of probability distributions, which is set to 0 if the similarity is less or equal to . The Jensen-Shannon divergence is defined as

where is the mixture distribution of P and Q [4]. Here, is the Kullback-Leibler divergence defined as

where is the probability space on which P, Q or M are defined [10] and denotes the Radon-Nikodym derivative with respect to , i.e. the probability density function in case is chosen to be the Lebesgue measure. Here, denotes the natural logarithm. We use this logarithm for better comparability with Fujikawa’s results where the natural logarithm is used as well. In [11], the logarithm with base 2 is used as it implies that the Jensen-Shannon divergence ranges from 0 to 1.

The test decision whether stratum i is detected as active is based on the posterior probability of lying above a desired target rate , i.e.,

(3)

where is the vector of responses and where the posterior probability is defined with respect to the borrowing posterior .

We denote by the vector of tuning parameters. In the examples of [4], the shape parameter , the similarity cutoff and the detection threshold are chosen to be either or , but it is unclear whether this is the optimum for choosing the tuning parameters.

Concerning the relationship between and , one should note two things. Firstly, they are redundant when it comes to defining the minimal similarity for which borrowing is still allowed. Indeed, for a given minimal similarity and a given , we can choose (and analogously ) such that the function is greater than 0 if and only if .

Secondly, this implies that when choosing or greater, then borrowing is only allowed for two substrata that have identical response rates. We call this the extreme borrowing boundary . Increasing or above this boundary does not change the behavior as is either equal to 1 in case or equal to 0 in case .

The parameter space upon which we will perform optimization in this study will be , and . For the grid search algorithm (explained below), we will restrict ourselves to . This means that the parameter space will encompass the parameter suggestions by Fujikawa et al. and that we can study the behavior of the extreme borrowing boundary for higher values (approx. above ) as can be seen in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Extreme borrowing boundary of Fujikawa et al.’s tuning parameters.

Extreme borrowing boundary for different stratum counts I and per-stratum patient counts n_i and the upper bound of the parameter space for the grid search algorithm.

https://doi.org/10.1371/journal.pone.0323097.g001

5 Outcome scenarios

In order to study the performance of the tuning parameter combinations, each step of the comparison study will consider the outcome scenario sets summarized in Table 1. The outcome scenario sets differ in the number of strata (I = 3 to I = 20), in the response rates assumed for inactive and active strata (p₀ = 0.01 to 0.20, p₁ = 0.10 to 0.50) and in their total and per-stratum sample sizes (total sample sizes ranging from 72 to 480). With the aim of using scenarios that are representative of both relevant methodological research and actual clinical trials, we considered three methodological publications, [4, 12], and [11], four systematic reviews of basket trials, [13, 14, 15], and [1], and additional information from the website ClinicalTrials.gov for the study [16]. Details on the choice of scenario sets can be found in S1 File Section 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Outcome scenarios sets.

https://doi.org/10.1371/journal.pone.0323097.t001

We would like to note that all scenario sets consider equal sample sizes across all strata. This is intended to simplify simulations. However, the optimization framework can easily be generalized to unequal sample sizes, too. Basket trial designs for unequal sample sizes can be found in the literature, e.g. [17].

For the scenario sets based on actual clinical trials, the observed response rates are known. They are listed in S1 File Section 1. After optimization of the utility functions on the scenarios mentioned in Table 1, we will calculate the performance measures of part II of the comparison study not only for the scenarios listed in the table but also for the true scenarios.

6 Utility functions

Under a given true scenario and a tuning parameter vector , we define the power in a truly active stratum i as the probability

and the type-I error rate (TOER) in a truly inactive stratum i as the probability

In this context, all probabilities are “frequentist”, meaning that they are defined with respect to true binomial sampling distributions for all i without being modeled on a prior distribution. However, the test decision whether a stratum is detected to be active is made according to the “Bayesian” borrowing posterior probability as defined in Eq 3. This approach is sometimes called “pragmatic Bayesianism”.

Let be the set of active strata with respect to , the set of inactive strata. Then, we define the experiment-wise power (EWP) as the probability

Analogously, we define the family-wise error rate (FWER) as the probability

Finally, we define the expected number of correct decisions (ECD) as

Based on these functions, we define the following utility functions. Across all definitions, are penalty parameters set to 1 by default.

1. Discontinuous family-wise power-error functionusually with being the global null scenario and .
2. Expected number of correct decisionsusually with being the global null scenario.
3. Two-level family-wise power-error functionwhere is a threshold for imposing harder FWER penalty, set to 0.1 by default.
4. Two-level stratum-wise power-error functionas suggested in [8].
5. Scenario-averaged versions of the above utility functionsandwhere is a set of scenarios of interest, e.g. the set of scenarios with number of strata I from Sect 5, and are weights with , e.g. for all p.
6. Scenario-averaged utility functions with penalty of maximal TOER inflationfor l =  ewp or ecd with the maximum defined across all and , andfor l =  2ewp or 2pow, where we choose and . In order to make the penalty work, should be greater than the absolute value .

The term utility function is used as in the context of constrained optimization [18], i.e., as an objective function that is to be maximized (contrary to a loss function that would have to be minimized). In utilitarian economics and in medicine, utility may be defined as the satisfaction of individual desires by a good in general [19] or, more specifically, the improvement of individual health by a therapy, e.g. as in [20]. This meaning is not necessarily implied in the present comparison study, where the utility function is a global function across the whole trial. However, one could adapt the presented utility functions to incorporate patient utility. For example, the two-level stratum-wise power-error function could be adapted by defining , were c is the cost of mis-detecting an inactive stratum and g denotes the gains from detecting a truly active stratum. A more sophisticated adaption with stratum-specific cost and gains is already discussed in [8].

We call the first four functions single-scenario utility functions. The scenario-averaged two-level stratum-wise power-error utility function was suggested in [8], except for the fact that the authors use all possible partitions with respect to response rates whereas we allow arbitrary scenario sets. The scenario-averaged expected number of correct decisions function emulates the optimization algorithm from [11] which in turn took the algorithm from [21]. There, the detection threshold is first optimized to keep FWER with respect to the global null scenario below a threshold and then the mean ECD across all scenarios is maximized subsequently.

The scenario-averaged utility functions with penalty of maximal TOER inflation are promising as they present a good compromise between research goals and regulatory requirements. In the context of borrowing from an external data source, it is an established fact that power gains using an external data source can only come at the price of type-I error inflation [5]. (This holds in the presence of a uniformly most powerful test as is the case for binary endpoints.) This shortcoming is usually accepted as a lesser evil in the context of master protocols. However, regulation may impose a constraint on maximal TOER inflation per basket, which can be taken into account by implementing the very harsh penalty .

7 Optimization algorithms

For finding the optimal value (minimum or maximum) of a utility function , we will consider the optimization algorithms named in the following list. A brief explanation of the functionality of each optimization algorithm is presented below the list.

1. Bounded simulated annealing algorithm using the reflection approach for bounding the parameter space as suggested in [Section 6 of 22] – implemented in an R package developed for the purpose of this comparison study, using a start temperature of
   1. i. ,
   2. ii. ,
   3. iii. ,
   
   and one function evaluation per temperature step. Simulated annealing is inspired by a thermodynamic process, hence the physical term “temperature” as explained further below.
2. Unbounded simulated annealing algorithm using the “return NA” approach for bounding the parameter space – implemented in the R function stats::optim(), using the start temperature and one function evaluation per temperature step.
3. Differential evolution (DE) as implemented in the R package metaheuristicOpt, using a population size of 40, a scaling vector of 0.8 and a cross-over rate of 0.5.
4. Grey wolf optimizer (GWO) as implemented in the R package metaheuristicOpt, using a population size of 40.
5. Constrained optimization by linear approximations (COBYLA) algorithm as implemented in the R package nloptr, using stopping tolerance of 10⁻⁶ in the parameter space of and a stopping tolerance of 0 in the value space of .
6. Grid search algorithm, searching the set of all combinations of

We will allow each algorithm to run for up to 1000 function evaluations. The grid of the grid search algorithm was chosen to use 1000 function evaluations as well. In order to put focus on parts of the grid that seemed most relevant in preliminary experiments while restricting to grid dimensions , we omitted some seemingly less relevant parts: (i.e. test decision almost always positive), high resolution for (i.e. sharing only with high similarity) and (i.e. almost no sharing, similar to 0.8 and 1.0). If one of the simulated annealing algorithms, GWO, DE, or the COBYLA algorithm shows poor convergence after 1000 evaluations and runtime permits it, we will conduct one more run with 20 000 function evaluations. With this number of iterations, preliminary experiments with simulated annealing reached convergence, which were, however, based on an R package unfit for the analysis.

We will end this chapter with a brief explanation of the functionality of each optimization algorithm:

Simulated annealing, referred to as unbounded simulated annealing in this publication, is a physics-inspired metaheuristic optimization algorithm suggested by [23]. It builds upon the Metropolis algorithm that is also used in Markov chain Monte Carlo procedures. It is named simulated annealing as it mimics the procedure of annealing in metallurgy: There, a metal product is heated and then slowly cooled down in order to achieve a more homogeneous and stable structure within the product. The algorithm starts at a user-suggested or randomly selected initial parameter vector. In each step, the simulated annealing algorithm randomly suggests a new parameter vector near the old parameter vector. If the new vector has better utility, the current parameter vector is updated to be the new parameter vector. If it has worse utility, it is still replaced with a probability proportional to current “temperature”. At the beginning, the temperature is high in order to allow the algorithm to escape local optima. Following a prespecified temperature schedule, it is then slowly cooled down in order to find the global optimum. This algorithm was first shown to converge on finite parameter spaces in [24]. Due to the finite nature of computer memory, we can consider our parameter spaces as finite.

Bounded simulated annealing is a generalization of the simulated annealing algorithm to hypercubes of the form with lower and upper bounds l_i and u_i considered as a subset of . It works identically to above-mentioned unbounded simulated annealing algorithm. Whenever a suggested parameter vector’s component surpasses one boundary l_i or u_i, it is reflected along these boundaries until it lies in the respective interval . [22] suggested this modification and proved its convergence. This reflection procedure’s result can be calculated by a simple affine transformation combined with division with remainder.

Constrained optimization by linear approximation (COBYLA) is an optimization algorithm suggested by [25]. It employs a d-dimensional simplex in order supply linear approximations of the utility function without the need for calculating derivatives. New simplex vertices are suggested by using these linear approximations while dynamically adjusting search radius and punishment for constraint violations. The algorithm’s procedure is too complex to be described in detail but can be found in above-mentioned reference.

Differential evolution (DE) is a metaheuristic optimization algorithm that mimics genetic evolution [see 26, for an overview]. From a fixed number of candidate vectors (“the population”), donor vectors are generated by randomly adding the scaled differences of two vectors to a third vector (“mutation”). Then, new candidates (“offspring”) are generated by randomly replacing some vector components of the original candidates with components of the donor vectors (“crossover”). Finally, the new candidates replace the old candidates in the next generation if they perform better or equal (“natural selection”). Many improvements of this idea have been suggested as discussed by Das et al.

The grey wolf optimizer (GWO) is a metaheuristic optimization algorithm inspired by the hunting behavior and social hierarchy of grey wolves suggested by [27]. The parameter space is searched by a number of candidate vectors (“pack of wolves”) which are following the direction of the three best candidates (“alpha, beta and delta wolves”). Candidate vectors are allowed more random behavior in the beginning (“searching for prey”) and are more strictly following the best solutions in the end (“encircling the prey”). The algorithm has further been improved [28], but this improved version is not implemented in R.

Grid search is the conceptually simplest of the mentioned algorithms. For each component of the parameter vector, the user specifies a set of values of interest. Then, the algorithm simply searches all possible combinations of values for the optimal value. Grid search can be parallelized and is completely deterministic, but on the other hand, its cost grows exponentially with the number of parameters. Its precision will never be finer than the size of the mesh.

This choice of optimization algorithms is obviously not exhaustive of the abundance of available optimization algorithms. We chose these algorithms as they represent different approaches to optimization: stochastic metaheuristics with inspirations from physics, genetics and swarm behavior (simulated annealing, DE, and GWO, respectively), non-linear programming (COBYLA), and naive deterministic approaches (grid search). Both COBYLA and grid search were already applied to the optimization of clinical trial designs, see [29, 8], and [11]. Stochastic metaheuristics appear to be a good alternative as they have little requirements to the “niceness” of the targeted utility function and as they are often able to escape local minima. Availability of R implementations was also a relevant criterion in the selection of optimization algorithms.

8 Comparison protocol

The comparison study is divided into three parts with I. the goal of comparing optimization algorithms, II. the goal of comparing utility functions, and III. the goal of comparing the optimized parameter values to the parameter values suggested in [4] as described above. For each part of the study, we will apply the ADEMP scheme for describing simulation studies that was introduced in [9]. The ADEMP scheme was developed for describing simulation studies of statistical methods. In Part I, the methods of interest are optimization algorithms rather than statistical methods. However, the ADEMP scheme could still be adapted to match the best practices in benchmarking algorithms described in [30], namely “clarifying the reason for benchmarking” (aim in the ADEMP scheme), selecting an appropriate test set (data-generating mechanism in ADEMP) and reporting comparative measures of efficiency, reliability and quality of solution (performance measures in ADEMP). Some more sophisticated methods from [30] such as the choice of an exhaustive test problem set and the reporting of performance profile plots was omitted as our algorithm comparison is a quite small case study rather than a complete benchmarking of possible algorithms choices.

9 Further analyses

In an exploratory fashion, we will consider two further aspects in our comparison study. Firstly, Fujikawa’s basket trial design could be altered by replacing the Jensen-Shannon divergence by the Hellinger distance [see e.g. 32],

which has the advantage that for two beta distributions, it can be calculated from basic functions without the need for numerical integration [33]:

We will explore whether this replacement results in a speedup of the design while at the same time maintaining a similar behavior compared to Fujikawa’s design.

Secondly, we will graphically investigate the effect of borrowing on the maximal TOER in I = 2 strata. One inactive basket will be kept at a stable response rate of p₁ = 0.2, while we increase the response rate of the other basket from p₂ = 0.2 to p₂ = 1. We will then plot the response rate of p₂ on the x-axis and the TOER of basket 1 on the y-axis for different combinations of and . The resulting curves may give an overview of how borrowing affects the TOER in inactive strata.

10 Justification of simulation size and runtime

For assessing the precision of the stochastic algorithms in Part I of our comparison study, we decided to run the algorithms for a total of times. In the following Sect 10.1, we will justify the choice of this simulation size by estimating the study’s run time. For the calculation of design operating characteristics, we will use the baskexact package, which calculates exactly up to numerical imprecision whenever it is feasible. However, we use the Monte Carlo-based R package basksim [34] for estimating design characteristics in the case of large sample sizes and large stratum counts, as the baskexact package would take too long (see Sect 10.1 for a more detailed explanation). We use for the number of Monte Carlo-generated data sets in basksim. The choice of this number is justified in Sect 10.3.

For scenario sets and , Part II and Part III of our comparison study are deterministic and precise up to the precision of numerical integration. Therefore, we do not conduct several runs in these parts. In other words, the simulation size for parts II and III equals 1. For the other scenario sets, the calculation of performance measures is based on Monte Carlo-generated data sets with .

10.1 Estimation of runtime

In order to estimate the duration of our simulation, we ran a small pilot simulation: We executed the bounded simulated annealing algorithm and the grid search with 1000 iterations each for optimizing the utility function with respect to stratum counts I, per-stratum sample sizes n_i and the respective global null hypothesis using the R packages baskexact and basksim. The resulting run times are specified in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Run time of 1000 iterations in a pilot study.

https://doi.org/10.1371/journal.pone.0323097.t002

The baskexact package took to run for . This package calculates operating measures combinatorically and the runtime is hence influenced by the number of combinations, which is proportional to . Therefore, it is reasonable to assume that larger scenarios, e.g. with , would have a longer duration by several orders of magnitude more than (e.g. 10⁵), rendering any simulation infeasible. This is the reason why we will calculate the utility function for the other scenario sets with the R package basksim, which applies Monte Carlo simulation and can hence reduce run time. The execution of basksim used Monte Carlo simulations in each call to the package.

The grid search algorithm was faster than the simulated annealing algorithm by more than a factor 15 even though both algorithms ran for 1000 iterations. This is due to the fact that we used parallelization on 20 workers on our institute’s RStudio server for calculating the performance measures on the grid. Due to the sequential stochastic nature of metaheuristic optimization algorithms, parallelization of a single simulated annealing, GWO or DE run is not possible. However, we will be able to parallelize part of the simulated annealing runs; hence we will also estimate a speedup by a factor 15 for Part I. In Table 3, we show a detailed explanation of the run time of Part I. The estimated run time of 15.7 days is long. However, it is still feasible while underlining the necessity to keep the example set of utility functions as small as it is.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Estimation of the total run time for Part I of the comparison study.

https://doi.org/10.1371/journal.pone.0323097.t003

The run time of Part II of our study is shown in the following Table 4. The estimation of the benefit of parallelization appears more complicated in this case: Should we parallelize across the set of 12 utility functions or across the set of up to 11 data scenarios? However, even without any parallelization, the estimated run time is 16.9 days, which would be feasible.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Estimation of the total run time for Part I of the comparison study.

https://doi.org/10.1371/journal.pone.0323097.t004

Finally note that the grid search algorithm on I = 8 strata is slower by a factor of 3 compared to simulated annealing. Grid search using the basksim package cannot use parallelization, as the basksim package already uses parallelization internally. However, it should optimally have about the duration as the simulated annealing algorithm, as both take the same number of iterations if executed sequentially. The unwanted slowdown may be due to a suboptimal R implementation and will be further investigated before execution of the comparison study.

10.2 Monte Carlo standard error of the algorithms’ precision in Part I

In Part I of the comparison study, we are interested in the precision of partly stochastic optimization algorithms. We use to estimate this precision. How precise will our estimates be? The marginal sample standard deviation of the components of the resulting optimal parameter vectors is one of the most relevant performance measures of Part I of the comparison study. Assuming a normal distribution of the marginal optimal algorithm result components around the true optimal vector components, we want to keep the standard deviation of the marginal standard deviation of the vector components below a reasonable level. An unbiased estimator of the standard error of the sample standard deviation is given by

where is the sample standard deviation and is the gamma function for with positive real part , see S1 File Section 2, for details. For a total of 50 algorithm runs, this means that we will achieve a standard error of , i.e., the sample standard deviation will be precise up to a standard error of little more than 10%. Regarding the already long run time, this appears acceptable.

10.3 Monte Carlo standard error of the design characteristics in parts I and II

In parts I and II, we will estimate the design characteristics of basket trial designs with I = 8 using the Monte Carlo-based package basksim. We want to estimate the stratum-wise power and TOER by applying the design to simulated data sets. According to [9, Table 6], the Monte Carlo standard error of a rejection rate estimate such as power and TOER is given by

The value is maximal for , resulting in a Monte Carlo standard error of for , i.e., a standard error of less than 2%. This seems acceptable regarding the long run time in Part I of our comparison study.

11 Discussion

Utility functions are a feasible and objective way of combining operating characteristics in clinical trials, which has proved useful in different contexts. So far in the context of basket trials, optimization is usually restricted to heuristic manual tuning of parameters as in [4] or to optimizing one characteristic of interest (e.g. expected number of correct decisions) while keeping type-I error rate in one scenario controlled as in [21] and [11]. The challenge of optimizing across multiple scenarios and the choice of optimization algorithm is usually not discussed. [8] suggested two types of utility functions but their choice was also not compared to other functions. Hence, our comparison study will fill a research gap in investigating both the choice of utility functions and of optimization algorithms.

Even though the framework is intended for the optimization of both Bayesian and frequentist basket trial designs, we chose frequentist performance measures as they are most common in clinical trials and hence easy to communicate despite all well-known limitations. A useful extension of the framework could be to exchange TOER and power for probability of success while replacing the different scenarios for the true response rates with an appropriate joint prior distribution.

If it proves effective and feasible, the studied optimization framework may lay the foundation for further research on optimizing basket trials, be it extensions to unbalanced sample size in the strata, multi-stage basket trials, and different Bayesian or frequentist basket trial designs.

Supporting information

Acknowledgments

We would like to thank Lukas Baumann, Marietta Kirchner and Paul Thalmann from the Institute of Medical Biometry at the Heidelberg University Hospital, Germany, Norbert Benda from the German Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte, BfArM), and Carolin Herrmann from the Heinrich Heine University Düsseldorf, Germany, for useful comments concerning this protocol. Furthermore, we thank the anonymous reviewers and editors for their suggestions on how to improve the study protocol.