Simulation study

The following list provides a brief description of each step in our simulation study. We provide a more detailed explanation of each step in the next section.

1. Defined the parameter sets (θ_i). Each θ_i is a vector of variables used to calculate the statistical power (P) of a theoretical CRT. The following components make up θ_i and are also listed (with the values we assumed for each) in Table 1:
   1. Type I Error (α, fixed at 0.05 for all experiments)
   2. Mean cluster size (μ)
   3. Intraclass correlation coefficient ($\mathrm{\text{ICC}}=\frac{{\sigma }_{\eta }^2}{{\sigma }_{\eta }^2+{\sigma }_{\epsilon }^2}$)
   4. Between cluster variance ($\mathrm{\text{BCV}}={\sigma }_{\eta }^2$)
   5. Number of clusters needed to reach 80% power with fixed cluster sizes (C⁸⁰)
   6. Effect size (Δ, calibrated on the combination of above variables)
   Using parameters (i-v) and a type II error (β) of 0.2 in Murray’s effect size equation (defined in Methods), we calculated the minimum effect size required to find a significant result in a properly powered CRT, Δ. By jointly varying parameters (i-v) over realistic ranges of values, we created carefully calibrated hypothetical CRT settings. Then, when we simulated data from CRTs in these setting to make power estimates with clusterPower, we allowed for the actual number of clusters in the study (C^A) to be different than C⁸⁰.
2. Estimated the statistical power when cluster sizes were fixed at μ. We refer to these estimates as our fixed cluster size power estimates, ${\hat{P}}_{{\theta }_{\mathit{i}}}^F(C^A)$. To do this, we simulated hypothesis tests on 5000 unique datasets for each (θ_i, C^A) pair.
3. Estimated the statistical power when cluster sizes have variability, defined by several coefficient of variation (cv) levels. We refer to these estimates as variable cluster size power estimates, ${\hat{P}}_{{\theta }_{\mathit{i}}}^{cv}(C^A)$. To do this, we generated S = 2000 variable cluster size sets for each (θ_i, C^A, cv) combination and ran a simulation on each variable cluster size set, which output a dataset. Hypothesis testing was performed on each dataset and produced a ${\hat{P}}_{{\theta }_{\mathit{i}},s}^{cv}(C^A)$, where s = 1, …, S. The average result over all S datasets defined the ${\hat{P}}_{{\theta }_{\mathit{i}}}^{cv}(C^A)$.
4. Calculated the required number of clusters for each combination of θ_i and cv. The vectors ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{F}}}$ and ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{cv}}}$ form power curves for each θ_i along the range of C^A. Using the values just above and below P = 0.8, we interpolated the point where C^A is equal to the number of clusters required to achieve a statistical power of 80% for fixed-size cluster sets (${\hat{C}}_{{\theta }_{\mathit{i}}}^F$, which approximates C⁸⁰) and variable cluster sets (${\hat{C}}_{{\mathbf{\theta }}_{\mathbf{i}}}^{cv}$).
5. Using our results, we observed the effect of cv on the required number of clusters (${\hat{C}}_{{\theta }_{\mathit{i}}}^{cv}$).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. List of simulation parameters and their values.

https://doi.org/10.1371/journal.pone.0119074.t001

Step 1: Parameter Selection

To test the impact of cluster size variability on P, we simulated from 420 parameter sets (θ_i). This is the number of unique combinations of μ values (5), ICC values (7), BCV values (3), and C⁸⁰ values (4). Each θ_i was simulated across a range of C^A values (6 to 9, depending on C⁸⁰) to create a total of 3,255 data points.

For each θ_i parameter set, we calculated a Δ value based on α, β, μ, ICC, BCV, and C⁸⁰ according to these equations [17]: (2)

Parameter selection required balancing the desire for well-spaced values across a meaningful range for each parameter with the computational burden of the simulations. (Over 68 million simulations took approximately two weeks running in parallel in 21 threads on a 12-core Mac Pro Desktop.)

The values for α and β were set at the standard type I and type II error rates of 0.05 and 0.2, respectively.

The five values of the mean cluster size, μ, were distributed between 20 and 125. We limited the maximum μ to 125 because simulations with large cluster sizes take considerably more computational time.

The ICC quantities were seven evenly-spaced, rounded log-linear values between 0.001 and 0.2.

The three BCV values were equally spread out on a log scale, spanning from 0.01 to 1.

The four C⁸⁰ values were chosen based on the range of popular numbers of clusters for CRTs. A CRT with 10 clusters would constitute a small trial and a CRT with 60 clusters is decidedly larger. We set the maximum C⁸⁰ value at 60 because larger quantities of clusters take significantly longer to simulate.

The C^A values ensured that the power curves for each CRT ranged from near 0 to 1. Initial tests showed that a wide range (5 ≤ C^A ≤ 120) was needed to accomplish this when C⁸⁰ = 60. However, for smaller C⁸⁰ values, the statistical power reached 1 sooner and thus the larger values of C^A were unnecessary.

Step 2: Generate fixed cluster size power estimates

Prior to investigating the effect of variable cluster sizes on statistical power, we calibrated the empirical estimates of power from the clusterPower package against the formula-based estimates of power (2). Since existing CRT formulas assume equal cluster sizes, we ran one 5000-simulation fixed cluster size CRT for each (θ_i, C^A) pair. This produced fixed cluster size power estimates, ${\hat{P}}_{{\theta }_{\mathit{i}}}^F(C^A)$, where every cluster size in each CRT was fixed at μ. In Step 4, these estimates are used to compare the difference between the estimated number of clusters needed based on formulaic and simulated power calculations.

Step 3: Generate variable cluster size power estimates

Next, we generated variable cluster size sets, sets of C^A randomly-drawn cluster sizes from a negative binomial distribution with mean μ. The negative binomial distribution was chosen due to the fact that the cluster sizes from trials that study investigators have worked on have shown over-dispersion similar to that of these distributions. We fixed the coefficient of variation (cv) of the negative binomial distribution, to roughly match observed skewness in cluster sizes from published CRTs. We used three levels of variability in cluster sizes: low variance (cv = 0.5), mid variance (cv = 1.0), and high variance (cv = 1.5).

In our high variance cluster size set, the cv was fixed at 1.5, which is about twice the size of the largest cv in other CRT papers [8–10]. This ensured that our high variance cluster size power estimates represented a near-upper bound on the number of variable-sized clusters required in practice. By holding the cv constant, the size parameter of the negative binomial distribution (r) and consequently the variance, are functions of μ [18]: (3)

When cv = 1.5, draws from a negative binomial distribution with a mean of 20 yield a value of zero 18% of the time, a value of fifty or greater 12% of the time, and a value of 140 or greater 1% of the time. Because a cluster cannot consist of zero participants, we set the minimum cluster size to 3. Specifically, to obtain a mean cluster size of 20, we drew from a negative binomial distribution with a mean of 17 before adding 3 to all cluster sizes. The probability mass functions that these distributions are drawn from are shown in Fig. 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The negative binomial distributions used to draw the cluster sizes sets for each mean cluster size (μ) and coefficient of variance (cv).

Each set had between 5 and 120 clusters, which made up the sample size of a CRT.

https://doi.org/10.1371/journal.pone.0119074.g002

The number of participants in the variable cluster size sets fluctuated immensely. Since this has an effect on the estimated power of a CRT, many variable cluster size sets (S = 2000) were created for each (θ_i, C^A) pair, so that the mean cluster size of all sets would converge to μ. We ran one clusterPower hypothesis test simulation for every variable cluster size set. This generated a collection of binary outcomes (1 if the null hypothesis was correctly rejected, 0 if not), ${\hat{P}}_{{\theta }_{\mathit{i}},s}^{cv}(C^A)$ for s = 1, ⋯, S. These outcomes were averaged to create one point: (4)

Step 4: Estimate number of clusters needed with and without variability

Let $\hat{\mathit{P}}$ denote any ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{F}}}$ or ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{cv}}}$, a vector of power values for each of the θ_i across C^A, with or without variance. This vector creates a power curve that allows us to estimate the number of clusters required for that parameter set at a certain level of variability.

For each $\hat{\mathit{P}}$, we found the $\widehat{P}\left(C^A\right)$ and C^A at the points just below and above P = 0.8. ${\hat{P}}_{{0.8}^{-}}$ and C_0.8⁻ are the power and number of clusters below our point of interest. ${\hat{P}}_{{0.8}^{+}}$ and C_0.8⁺ are the power and number of clusters above our point of interest. ${\hat{P}}_{{0.8}^{-}}$ and ${\hat{P}}_{{0.8}^{+}}$ were placed in a vector and C_0.8⁻ and C_0.8⁺ into a matrix as follows: (5)

In solving this equation, we find the slope ($\widehat{a}$) and intercept ($\widehat{b}$) of a line that passed between the points (C_0.8⁻, ${\hat{P}}_{{0.8}^{-}}$) and (C_0.8⁺, ${\hat{P}}_{{0.8}^{+}}$). From there, we set P = 0.8 to find the number of clusters required to achieve sufficient power, $\widehat{C}$: (6)

Where Ĉ is the estimated number of required clusters to achieve a statistical power of 0.8. When using ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{F}}}$, $\hat{C}={\hat{C}}_{{\theta }_{\mathit{i}}}^F$, which is a simulated estimate of C⁸⁰. For ${\widehat{\mathbf{\text{P}}}}_{{\theta }_{\mathit{i}}}^{\mathbf{\text{cv}}}$, $\hat{C}={\hat{C}}_{{\theta }_{\mathit{i}}}^{cv}$, which is a value for which Murray’s formulas (2) cannot be used. For the sake of simpler notation, we will refer to these as ${\widehat{C}}^F$ and ${\widehat{C}}^{\mathit{cv}}$.

A graphical representation of this process is shown in Fig. 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. How to find the required number of clusters (Ĉ).

First, for a given set of parameters, θ_i, hypothesis test simulations are run for each level of C^A. Second, the results of the simulations are averaged into one point per level. Third, $\widehat{C}$ is calculated by interpolating a point at $\widehat{P}=0.8$ between (C_0.8⁻, ${\hat{P}}_{{0.8}^{-}}$) and (C_0.8⁺, ${\hat{P}}_{{0.8}^{+}}$.

https://doi.org/10.1371/journal.pone.0119074.g003

Additionally, we defined and compared the percentage change in the number of clusters needed to achieve 80% power between those required by the Murray equation and the variable cluster size sets as: (7)

Step 5: Analyze effects of cluster size variance

With ${\widehat{C}}^{\mathit{cv}}$, ${\widehat{C}}^F$, and C⁸⁰, we observed the effect of cluster size variance on the required number of clusters.

We compared and contrasted ${\widehat{C}}^F$ and C⁸⁰. If the numbers were similar, then the clusterPower simulations approximate the Murray equations and ${\widehat{C}}^{\mathit{cv}}$ could be used with confidence. If they were substantially different, further analysis would be required before evaluating ${\widehat{C}}^{\mathit{cv}}$.

${\widehat{C}}^{\mathit{cv}}$ was analyzed with respect to C⁸⁰. From this analysis, we observed the effects of cluster size variation on the required number of clusters.

Without cluster size variability, simulations approximate formula

The first clusterPower simulation used each unique parameter set (θ_i) across a range of sample sizes (actual number of clusters, C^A) to generate fixed cluster size power estimates (${\hat{P}}^F={\hat{P}}_{{\theta }_{\mathit{i}}}^F(C^A)$). These estimates were plotted to form the power curves represented by the black lines in Fig. 4. Using these results and the technique described in Methods Step 4, we calculated the number of clusters required to achieve a statistical power of 80% for fixed-size cluster sets (${\widehat{C}}^{\mathit{F}}$). To validate clusterPower, ${\widehat{C}}^F$ was compared to Murray’s formula-based estimate of clusters required to reach 80% power (C⁸⁰, from Equation 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The CRT power curves for two different parameter sets over four levels of cluster size variance.

When C^A = 60, both fixed cluster size power estimates (${\hat{P}}_{{\theta }_{\mathit{i}}}^F$), the solid, black lines) should equal 0.8. By looking below the point (60, 0.8), one observes that power is lost as the cluster size variance increases in both scenarios. By looking to the right of the point, the observer notices that more clusters are required to attain a statistical power of 0.8 with increased variability.

https://doi.org/10.1371/journal.pone.0119074.g004

The clusterPower simulations produced accurate, though conservative estimates. One example of this is the result from the lower left corner of Fig. 4. In this scenario, the first five θ_i parameters (type I error(α) = 0.05, mean cluster size (μ) = 75, ICC = 0.006, BCV = 0.1, C⁸⁰ = 60) require an effect size (Δ) of 0.417 when the cluster sizes are fixed. Using these parameters and a C^A = 60, we should find that the statistical power is near 80%. The average statistical power across 5,000 simulations, ${\widehat{P}}^F$, was 79.04%. Since this is smaller than 80%, from this point and the one at C^A = 80 we can interpolate to find that ${\widehat{C}}^F=61.74$. This is a slightly conservative result, in that the statistical power was less than one percent lower than expected and the required number of clusters were two greater than expected.

More than half of our simulated ${\widehat{C}}^F$ values were closer to Murray’s C⁸⁰ than in this example. In situations where C^A = C⁸⁰, the ${\widehat{P}}^F$ ranged from 75.04% to 81.44% with a median of 79.52%. The difference between ${\widehat{C}}^F$ and C⁸⁰ ranged from -1.55 to 8.28 with a median difference of 0.51 (with negative values indicating ${\widehat{C}}^F<C^{80}$). The ${\widehat{C}}^F$ and C⁸⁰ were highly correlated (R = 0.9982). A paired two-sample t-test showed that ${\widehat{C}}^F$ was significantly larger than C⁸⁰ and that the average difference is within one cluster (Estimate: 0.81, CI: (0.69, 0.92)).

While ${\widehat{C}}^F$ did not perfectly reflect C⁸⁰, the high correlation and generally conservative estimates suggest that clusterPower can be used to simulate variable cluster size CRTs.

With cluster size variability, power decreases

The subsequent clusterPower simulations used θ_i with variable cluster size sets of C^A clusters and three levels of coefficient of variance (cv = 0.5, 1.0, 1.5) to generate variable cluster size power estimates (${\hat{P}}^{cv}={\hat{P}}_{{\theta }_{\mathit{i}}}^{cv}(C^A)$). These estimates were plotted to form the power curves that form the blue, green, and orange lines in Fig. 4 (which represent cv = 0.5, 1.0, 1.5, respectively). By performing the same process for calculating ${\widehat{C}}^F$, we derived the number of clusters required to achieve a statistical power of 80% for variable cluster size sets, ${\widehat{C}}^{\mathit{cv}}$.

Using the lower left plot again as an example, one can see how power decreases and required number of clusters increases with greater cluster size variation. Looking below the ${\widehat{P}}^F$ point at (60, 0.79), we see that the power decreases as variance increases: ${\widehat{P}}^{0.5}=0.77$; ${\widehat{P}}^{1.0}=0.73$; and ${\widehat{P}}^{1.5}=0.69$. Looking to the right of the ${\widehat{C}}^F$ point at (61.74, 0.8), we see that more clusters are required in the presence of variation: ${\widehat{C}}^{0.5}=64.55$; ${\widehat{C}}^{1.0}=72.6$; and ${\widehat{C}}^{1.5}=79.82$.

This is merely one example of a consistent pattern of decreasing power as cluster size variability increased. Using paired t-tests, we observed that the difference between ${\widehat{C}}^{0.5}$ and C⁸⁰ was significantly greater than zero (Estimate: 2.19, CI:(2.05, 2.34)); that ${\widehat{C}}^{1.0}$ was significantly greater than ${\widehat{C}}^{0.5}$ (Estimate: 4.09, CI:(3.82, 4.36)); and ${\widehat{C}}^{1.5}$ was significantly greater than ${\widehat{C}}^{1.0}$ (Estimate: 4.91, CI:(4.59, 5.23)).

Within each cv level, there is substantial variability in statistical power and required number of clusters due to the large diversity of parameter sets. We can observe this by looking at the percent increase in number of clusters required by variable cluster size sets over those from the Murray equation (${\widehat{C}}^{\mathit{cv}}\%$, from Equation 7). The range for ${\widehat{C}}^{0.5}\%$ spans −2.03% to 30.83%, with a median of 7.47%; ${\widehat{C}}^{1.0}\%$ covers 2.8% to 43.27%, with a median of 21.49%; and ${\widehat{C}}^{1.5}\%$ ranges from 13.04% to 63.72%, with a median of 38.38%.

In 21 cases (5%), ${\widehat{C}}^{0.5}$ was less than both the equivalent C⁸⁰ and C^F values. However, there does not seem to be a common link between these scenarios, and thus their occurrence may be due to sampling variation from the stochastic simulations.