Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Assessing standardized contrast effects in ANCOVA: Confidence intervals, precision evaluations, and sample size requirements

  • Gwowen Shieh

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Software, Supervision, Validation, Writing – original draft, Writing – review & editing

    gwshieh@nycu.edu.tw

    Affiliation Department of Management Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Abstract

Standardized effect sizes and confidence intervals are useful statistical assessments for comparing results across different studies when measurement units are not directly comparable. This paper aims to describe and compare confidence interval estimation methods for the standardized contrasts of treatment effects in ANCOVA designs. Sample size procedures are also presented to assure that the resulting confidence intervals yield informative estimation with adequate precision. Exact interval estimation approach has theoretical and empirical advantages in coverage probability and interval width over the approximate interval procedures. Numerical investigations of the existing method reveal that the omission of covariate variables has a negative impact on sample size calculations for precise interval estimation, especially when there is disparity in influential covariate variables. The proposed approaches and developed computer programs fully utilize covariate properties in interval estimation and provide accurate sample size determinations under the precision considerations of the expected interval width and the assurance probability of interval width.

Introduction

The utility of effect sizes and confidence intervals has been strongly emphasized in several editorial guidelines and methodological implications. The standardized mean difference between two independent populations is the most frequently used effect size measure across virtually all disciplines of scientific researches. Accordingly, the intuitive formula Hedges’s [1] g or commonly known as of Cohen’s [2] d is an estimate of the population standardized mean difference and is defined as the difference between two sample means divided by their pooled sample standard deviation under homoscedasticity. Unlike the unstandardized contrasts, the effect size reporting and interpretation practices suggest that the standardized effect sizes are useful when comparing results from multiple studies using measurement instruments whose raw units are not directly comparable. Comprehensive expositions and practical uses about standardized mean difference and related effect size measures are available in Grissom and Kim [3], Kline [4], Lin and Aloe [5], Takeshima et al. [6], Zhang [7], Zhang and Heyse [8], and the references therein.

Considerable attention has also been drawn toward the development of interval estimation methods for standardized mean difference, see Odgaard and Fowler [9], Smithson [10], Steiger and Fouladi [11], Tian [12], Wu, Jiang and Wei [13], and Zou [14], among others. Standardized contrasts of treatment effects and interval procedures are presented in Kline [4], Steiger and Fouladi [11], and Steiger [15] in the context of ANOVA. Moreover, Grissom and Kim [3], Kline [4], Levin [16], and Olejnik and Algina [17] addressed the calculation and interpretation of standardized contrasts of treatment effects in ANCOVA. However, these studies did not discuss the issues of interval estimation of standardized treatment contrasts. The exceptional case of Lai and Kelley [18] presented confidence interval formula and sample size determination of standardized contrast effects for ANCOVA designs with only one covariate. Particularly, Lai and Kelley [18] focused on the special cases of ANCOVA with a single covariate and suggested that, under randomized designs, the linear contrast of covariate means is usually close to zero. Hence, the corresponding covariate quantity can be omitted in the sample variance of a linear contrast for confidence interval and sample size calculations of standardized treatment contrasts. Despite the conceptual argument and technical simplification, they did not conduct empirical study to justify the suggested procedures regarding the influence of omitted covariate variable. Hence, it is of methodological importance to perform detailed numerical examination for clarifying the adequacy of the approximate confidence interval and sample size methods. Moreover, their model setting did not cover the more involved situations with more than one covariate variable. Thus, it is of practical importance to present more accurate interval estimation and sample size procedures for ANCOVA studies with possibly diverse covariate configurations.

Although randomized experiments are preferred in the gold standard of research designs, it is still possible that randomized experiments may generates imbalanced groups in terms of sample size and baseline disparities even in large studies. The related results about the impact of covariate imbalance on power of ANCOVA studies and randomized clinical trials can be found in Ciolino et al. [19,20], Egbewale, Lewis and Sim [21], Kahan and Morris [22] and the references therein. More importantly, feasible power and sample size procedures were developed in Shieh [2325] and Tang [2628] to accommodate covariate randomness and imbalance for randomized and nonrandomized designs. It should be emphasized that a thorough and rigorous illustration of interval estimation, precision assessment, and sample size techniques for standardized contrast effects of ANCOVA has not been given in the literature.

In view of the limited findings in the literature, the goal of the current article is twofold. First, this research examines and compares exact and approximate confidence interval procedures of standardized contrast effects for ANCOVA designs with one or more covariate variables. A second aim of the present study is to extend and complement the demonstration of unstandardized contrast effects in Shieh [29] by developing exact precision and sample size procedures for standardized contrast effects under the assumption of multinormal covariate variables. Extensive numerical studies for interval estimation, precision assessment, sample size determination were conducted to illustrate the advantages of the proposed techniques over the approximate methods. In order to facilitate data analysis and design planning, computer algorithms are presented to resolve the computational issues of the suggested procedures.

Methods

Interval estimation procedures

Consider the fixed-effects ANCOVA model with G treatment groups and P multiple covariates: (1) where Yij is the score of the jth subject in the ith treatment group on the response variable, μi is the ith intercept, Xkij is the score of the jth subject in the ith treatment group on the kth covariate, βk is the slope coefficient of the kth covariate, and ϵij is the independent N(0, σ2) error with i = 1,…, G (≥ 2), j = 1, …, Ni, and k = 1, …, P (≥ 1).

In view of the importance and utility advocated in Grissom and Kim [4], Kline [5], and Levin [18], the current study focuses on the appraisals of standardized contrasts rather than the conventional or unstandardized contrasts. A standardized linear contrast of adjusted group means is defined as (2) where , ci is the linear coefficient with is the expected group response evaluated at the grand covariate means , and . Note that a linear contrast of adjusted group means is equivalent to a linear contrast of intercept parameters because .

The standard results in Rencher and Schaalje [30] show that the least square estimator for the ith intercept μi is given by (3) where , , , , and . Moreover, an unbiased estimator of the adjusted group mean is of the form (4)

Thus, a convenient estimator for the contrast effect ψ is (5)

It can be shown that the linear contrast estimator has the distribution (6) where , V = a + Q/(NTG), , and VXX = SXX/(NTG). Note that the magnitude Q represents an estimate of the degree of covariate disparity in terms of the linear contrast of standardized covariate means.

The confidence interval of the contrast effect ψ can be obtained by the standard transformation T and associated distribution: (7) where , and ν = NTGP, and t(ν) is the t distribution with degrees of freedom ν. Specifically, the 100(1 –α)% equal-tail two-sided confidence limits () of the contrast effect ψ are (8) where tν,α/2 is the upper 100(α/2) percentile of the t(ν) distribution.

Confidence intervals

A natural estimator of the standardized contrast ψ* is (9)

Note that the well-known standardized mean difference index Hedges’s [1] g or Cohen’s [2] d provides a slight overestimate of the standardized mean difference δ = (μEμC)/σ between the experimental and control groups. Similarly, the standardized contrast formula is a positively biased estimator of ψ*. Following the model assumption, , where χ2(ν) is a chi-square distribution with degrees of freedom ν. Standard derivations show that E[K–1/2] = {2−1/2Γ[(ν − 1)/2]}/Γ(ν/2) and for ν > 1 and Γ{·} is the gamma function. Because and are independent, an unbiased estimator of ψ* can be obtained as where u = Γ(ν/2)/{(ν/2)1/2Γ[(ν − 1)/2]} < 1.

Moreover, the standardized statistic T* has the distribution (10) where t(ν, λ) is a noncentral t distribution with degrees of freedom ν and noncentrality parameter λ = ψ*/V1/2. The fundamental properties and related applications of noncentral t distribution are available in Johnson, Kotz, and Balakrishnan [31, Chapter 31]. Then, the noncentrality inversion procedure of Casella and Berger [32, Section 9.2.3], Mood, Graybill and Boes [33, Section 4.2], and Venables [34] provide the exact confidence interval estimations of ψ*.

Accordingly, the upper 100(1 –α1)% confidence interval of ψ* is of the form (, ∞), in which satisfies (11) where is the observed value of T*. On the other hand, the lower 100(1 –α2)% confidence interval of ψ* is of the form (–∞, ) in which satisfies (12)

Furthermore, a 100(1 –α)% two-sided confidence interval {} of ψ* can be obtained by jointly applying Eqs 11 and 12 with α1 = α2 = α/2 so that (13)

The particular method has been demonstrated for obtaining exact confidence intervals of standardized mean difference effect size in Kline [4], Odgarrd and Fowler [9], Smithson [10], and Steiger and Fouladi [11]. Notably, the confidence interval {} does not have a convenient closed-form expression. The numerical computations of exact confidence intervals require the evaluation of the cumulative distribution function of a noncentral t variable. The corresponding SAS/IML and R computer programs for performing the exact confidence interval calculations are available as supplementary material.

It is shown in Johnson, Kotz, and Balakrishnan [31, p. 513] for large degrees of freedom that E[t(ν, λ)] ≐ λ and Var[t(ν, λ)] ≐ 1 + λ2/(2ν). The results suggest that for large samples. Hence, the standardized contrast estimator has a asymptotic distribution (14) where τ*2 = V + ψ*2/(2ν). Using the asymptotic property, the approximate 100(1 –α)% equal-tail two-sided confidence limits {} of ψ* can be obtained as (15) where and zα/2 is the upper 100(α/2) percentile of the standard normal distribution. The same technique was previously recommended in Hedges [1] to construct confidence intervals for the population standardized mean difference δ.

The functional relationship between the unstandardized and standardized contrast effects ψ and ψ* = ψ/σ suggests approximate confidence interval of ψ* can be immediately obtained by dividing the confidence limits {} of ψ by the standard error estimate . This direct division gives a approximate 100(1 –α)% equal-tail two-sided confidence interval {} of ψ* where (16)

The identical procedure was proposed in Bird [35] and Fidler and Thompson [36] for finding the confidence interval of δ from the confidence limits of mean difference μEμC, and in Kline [4] for obtaining confidence interval of standardized treatment contrast from the interval endpoints of treatment contrast in ANOVA.

It should be noted that the exact two-sided interval estimates {} are not equidistant from the standardized contrast estimate except for the special case . However, the two approximate confidence intervals {} and {} are equidistant around the estimate . The two intervals {} and {} have the half-width and HD = tν,α/2V1/2, respectively. Note that the difference between zα/2 and tν,α/2 is usually trivial for large ν. Hence, it is generally true that HDHA and the interval {} is practically within the interval {} even when the estimate is nearly zero. To contribute to the understanding of interval estimation of the standardized contrast effects, numerical appraisals are presented next to investigate the estimation behavior of the three different confidence intervals.

An example

Fleiss [37, Section 7.2] described a study comparing three methods of treating the learning disabilities of children with respiratory diseases. The response variable is the number of correct answers to a test with 15 questions, and the covariate variable is the number of correct answers to a similar test with 7 questions. It is noted that the proportions of correct answers were not close to zero or one. Hence, no further transformation was applied to transform the discrete data for satisfying the normal assumption. The adjusted mean estimates and group sample sizes are and {N1, N2, N3} = {19, 20, 20}, respectively. Also, the covariate coefficient estimate is and the sample variance is . The omnibus F test of no treatment effect is F* = 12.52 with a p-value < 0.0001. Accordingly, the null hypothesis of treatment effects is rejected and there exists statistically significant differences among the three treatments at α = 0.05.

For illustration, standardized contrast analysis is presented here to complement the conventional unstandardized contrast assessments. With {c1, c2, c3} = {–1, 1/2, 1/2}, the unstandardized and standardized contrast estimates are and , respectively. It can be shown that the linear contrast of standardized covariate means or the covariate disparity index is Q = 0.2131 and the factor V of the variance of the linear contrast is V = 0.0814. The 90% confidence intervals of the exact approach, asymptotic method, and direct division are {0.8504, 1.8827}, {0.8558, 1.8885}, and {0.8947, 1.8496}, respectively. Moreover, the corresponding 95% confidence intervals of the three procedures are {0.7519, 1.9820}, {0.7568, 1.9874}, and {0.8002, 1.9440}. These results suggest that the exact approach and asymptotic method give very similar confidence limits and they evidently differ from the narrowest interval constructed by direct division.

The computations of the exact confidence intervals can be performed with the supplemental SAS/IML and R programs. Researchers only need to provide the desired key settings to replace the prescribed exemplifying values in the computer codes. Specifically, the programming lines of the SAS supplemental file A for computing confidence interval of standardized contrast are as follows:

  1. PROC IML;
  2. *USER SPECIFICATION PORTION;
  3. *DESIGNATED ALPHA;ALPHA = 0.10;
  4. *NUMBER OF GROUPS;G = 3;
  5. *NUMBER OF COVARIATES;P = 1;
  6. *TOAL SAMPLE SIZE;NT = 59;
  7. *CONTRAST ESTIMATE;PSIH = 2.4823;
  8. *SAMPLE VARIANCE;SIGSQH = 3.2728;
  9. *FACTOR FOR THE VARIANCE OF A LINEAR CONTRAST TAUSQ;V = 0.081437;
  10. *END OF USER SPECIFICATION PORTION;

The R supplemental program D has similar specifications:

  1. ancova.scie.apx1.fun(alpha = 0.1, g = 3, p = 1, nt = 59, psih = 2.4823, sigsqh = 3.2728, v = 0.081437).

Empirical comparisons

To provide a more thorough evaluation, simulation studies were performed to examine the coverage performance of the interval procedures. On the other hand, the more involved precision assessments and sample size determinations of standardized contrast effects are presented in the Results section. For ease of explication, the model and parameter configurations of the learning disabilities study are modified and extended in this numerical examination. Specifically, Monte Carlo simulation studies of 10,000 iterations were performed with the normal distribution of the linear contrast and the chi-square distribution of the model variance . Note that the variance of a linear contrast has the simple form τ2 = σ2V and . Accordingly, the error variance estimate and covariate statistic of the learning disabilities data are set as the variance component σ2 = 3.2728 and covariate disparity Q = 0.2131. The contrast effect ψ for the contrast coefficients {c1, c2, c3} = {–1, 1/2, 1/2} has four different magnitudes so that ψ = ψ* = ψ/σ = 0, 1, 2, and 3. Also, the designated balanced designs with G = 3 have the group sample size N = 5, 10, and 20. For each replicate, the lower and upper confidence limits {} and {} were computed for the 95% and 97.5% one-sided confidence intervals. These interval limits are then combined to construct the 90% and 95% equal-tail two-sided confidence intervals. The simulated coverage probability was the proportion of the 10,000 replicates whose confidence interval contained the population standardized contrast effect. The adequacy of the one- and two-sided interval procedures is determined by the error between the simulated coverage probability and the nominal coverage probability. The results of the 12 combinations of total sample size and standardized contrast effect are summarized in Tables 1 and 2 for the two-sided confidence coefficient 1 –α = 0.90 and 0.95, respectively.

thumbnail
Table 1. The error between simulated coverage probability and nominal coverage probability of the 90% two-sided and 95% one-sided confidence intervals for the number of groups G = 3, one covariate P = 1, contrast coefficients {–1, 1/2, 1/2}, error variance σ2 = 3.2728, and balanced designs with NT = 15, 30, and 60.

https://doi.org/10.1371/journal.pone.0282161.t001

thumbnail
Table 2. The error between simulated coverage probability and nominal coverage probability of the 95% two-sided and 97.5% one-sided confidence intervals for the number of groups G = 3, one covariate P = 1, contrast coefficients {–1, 1/2, 1/2}, error variance σ2 = 3.2728, and balanced designs with NT = 15, 30, and 60.

https://doi.org/10.1371/journal.pone.0282161.t002

The simulated coverage performance in Tables 1 and 2 suggest that the exact approach is excellent in attaining the nominal confidence levels for all one- and two-sided situations. Although the sample size configurations are not large, the asymptotic method provides surprisingly accurate results and is only slightly inferior to the exact procedure. Note that the small discrepancies of the exact and asymptotic procedures are generally within the 95% simulation variability of 0.006, 0.004, and 0.003 for the nominal coverage probability of 0.90, 0.95 and 0.975, respectively. However, the simulated coverage rates of the direct division are systematically lower than the nominal level, especially for large ψ* > 1. The exact interval procedure generally yields asymmetric confidence intervals for ψ*. Consequently, the equidistant confidence limits of the direct division technique tend to have undesirable effects. For the model configurations examined here, the lower confidence limits of the one-sided upper 95% and 97.5% confidence intervals are substantially less accurate than the counterparts or upper confidence limits of the one-sided lower 95% and 97.5% confidence intervals. The only exceptions that the direct division gives acceptable coverage performance are those associated with ψ* = 0. Note that the three confidence intervals for the standardized contrast effect are asymptotically equivalent. However, when the magnitude is not zero, the direct method is substantially less accurate than the other two procedures even for large sample sizes. In view of these findings, the exact confidence interval estimation is recommended. When no statistical software is available, the asymptotic method provides a viable alternative for hand computation. However, the direct division procedure is too simple to be useful and is not appropriate for general applications. Additional assessments were conducted to demonstrate the intrinsic implications and the results suggest the same performance situations as reported here.

Results

Precision assessments and sample size determinations

Within the context of ANCOVA, Hochberg and Varon-Salomon [38] showed that the conditional (on the covariates) confidence interval procedure compares favorably with the unconditional method based on the joint distribution of the response and covariate variables. Thus, the inferential procedures of hypothesis testing and interval estimation are the same under both fixed and random formulations. The distinction between the two modeling setups, however, becomes crucial when testing power, coverage probability, and corresponding sample size calculations are to be made. To exhibit the unique and distinct precision characteristics of confidence intervals, two useful criteria have been considered in Kupper and Hafner [39] for one- and two-sample problems. The suggested precision assessments maintain the expected magnitude of interval widths and the assurance probability of interval widths within a pre-assigned threshold.

Precision assessments

When planning and conducting ANCOVA research, the actual values of the continuous measurements of response and covariate variables for each subject are available only after the observations are obtained. In addition to the randomness of normal responses, the stochastic nature of covariate variables has to be taken into account in precision analysis under the general and unconditional context of ANCOVA. It is convenient and useful to consider the covariate variables have the independent multinormal distributions (17) where μXi is a P × 1 vector and ΣX is a P × P positive-definite variance-covariance matrix for i = 1,…, G, and j = 1, …, Ni. The quantity V in can also be expressed as V = a(1 + bFX), where , b = P/(ν + 1), and FX = Q/{ab(NTG)}. The multivariate results in Anderson [40, Section 5.2.2] and Muirhead [41, Section 3.2.3] show that FX has the distribution (18) where F(P, ν + 1, ξ) is the noncentral F distribution with degrees of freedom P and ν + 1 and noncentrality parameter . It is noted in Anderson [40] that for large samples the distribution of FX given by Eq 18 is approximately valid even if the parent distribution of covariates is not normal. In addition, the robust features of the F and Hotelling’s statistics have been examined empirically and theoretically in Chase and Bulgren [42], Everitt [43], Holloway and Dunn [44], Hopkins and Clay [45], and Kariya [46,47]. Hence, the noncentral F distribution provides a robust approximation for general use.

A useful population covariate disparity index can be defined as . However, it is a common assumption for a randomized design that the covariate characteristics are identical for all treatment groups with μX1 = … = μXG = μX, and the noncentrality parameter and covariate disparity reduce to ξ = θ = 0. Accordingly, the scaled quantity f2* = ξ/NT = θ/(aNT) = θ/a* has a similar role as the signal to noise ratio f2 in ANOVA where and pi = Ni/NT for i = 1, …, G. Thus, the conventional benchmark of Cohen [2] for small, medium, and large effects of f = 0.10, 0.25, and 0.40 may serve as a general guideline for the magnitude of covariate disparity when no better basis is available. However, the merit of a specific size of disparity should be assessed and interpreted by directly comparing with the findings in related prior studies.

In order to accommodate the stochastic properties of the covariate and response variables through the joint distribution of FX and T*, the expected width is evaluated for the interval width as (19) where EF[·] and ET[·] are taken with respect to the noncentral F distribution of FX and the noncentral t distribution of T*, respectively. An alternative approach to precision appraisal of confidence intervals is the assurance probability of interval width W is not wider than the designated bound ω (> 0): (20) where G = 1 if W ≤ ω and G = 0 if W > ω.

Sample size determinations

In planning research design, optimal sample size determinations need to be conducted to achieve the designated precision requirement in interval estimation. For the current problem of precise interval estimation of standardized contrasts, it is desirable to determine the minimal sample sizes such that the expected width of a 100(1 − α)% confidence interval is within the selected bound ω (> 0): (21)

Also, one may compute the minimal sample sizes needed to guarantee, with a given assurance probability 1 − γ (< 1), that the interval width of a 100(1 − α)% confidence interval will not exceed the chosen threshold ω: (22)

Note that Lai and Kelley [18] presented approximate sample size formulas for interval estimation of standardized contrasts for balanced ANCOVA with one covariate (Ni = N and P = 1). They assumed that, under a randomized design, the linear contrast of covariate means is usually close to zero and the sample variance of a linear contrast can be approximated as τ2aσ2. Extending such a simplification to the general case of ANCOVA designs with one or more covariates, the distribution of given in Eq 10 is simplified as (23) where λa = ψ*/a1/2. The expected width and assurance probability of interval widths are modified as (24) respectively, where ETa[·] is taken with respect to the approximate noncentral t distribution t(ν, λa) of T*. These two simplified precision functions Ωa and Γa provide alternative sample size calculations for standardized contrast analysis. For comparative purposes, these approximate methods are also investigated to reveal the impact of omitted covariate effects on sample size determinations.

Numerical illustrations

The underlying behavior and relative performance of the exact and approximate procedures for precision and sample size calculations are demonstrated in the following empirical study. First, the minimum sample sizes for the expected width and assurance probability criteria are computed by the exact and approximate distribution functions under the designated model configurations. Second, the accuracy of the precision and sample size procedures is examined through Monte Carlo simulation study to clarify the influence of covariate features under a wide range of model scenarios.

Accordingly, the balanced ANCOVA models with three treatment groups G = 3, and one to five (P = 1, …, 5) normal covariate variables are used as the basis for numerical assessments. The model configurations have the designated settings: the intercept parameters {μ1, μ2, μ3} = {0.5, 0, 0}, contrast coefficients {c1, c2, c3} = {1, –0.5, –0.5}, and error variance σ2 = 1. Note that the resulting standardized contrast effect is ψ* = 0.5. Because the covariate distribution function FX depends on the group mean values and variance-covariance matrix of the multinormal covariate distributions through the covariate disparity index θ. Without loss of generality, the corresponding mean values are set as μX1 = {θ1/2, 0, …, 0}, μX2 = μX3 = 0 where 0 is a P × 1 null column vector, and ΣX = IP is the identity matrix of dimension P. Three different magnitudes of covariate disparity are considered: θ = 0, 0.25 and 0.50 to represent potential covariate characteristics in randomized and nonrandomized designs. Throughout this numerical demonstration, only balanced designs with sample size ratios {r1, r2, r3} = {1, 1, 1} are considered and the confidence level is fixed as 1 –α = 0.95. The computed total sample sizes for homogeneous covariates θ = 0 with the expected width threshold ω = 1.00, 1.25 and 1.50, and P = 1, …, 5 are presented in Table 3, while the corresponding total sample sizes associated with θ = 0.25 and 0.50 are summarized in Tables 4 and 5, respectively. The optimal total sample sizes for assurance probability 1 –γ = 0.80 with ω = 1.00, 1.25 and 1.50 of bounded confidence intervals are summarized in Tables 68 for θ = 0, 0.25 and 0.50, respectively.

thumbnail
Table 3. Computed sample size, estimated expected width, and simulated expected width for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, and covariate disparity θ = 0.

https://doi.org/10.1371/journal.pone.0282161.t003

thumbnail
Table 4. Computed sample size, estimated expected width, and simulated expected width for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, and covariate disparity θ = 0.25.

https://doi.org/10.1371/journal.pone.0282161.t004

thumbnail
Table 5. Computed sample size, estimated expected width, and simulated expected width for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, and covariate disparity θ = 0.50.

https://doi.org/10.1371/journal.pone.0282161.t005

thumbnail
Table 6. Computed sample size, estimated assurance probability, and simulated assurance probability for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, assurance probability 1 –γ = 0.80, and covariate disparity θ = 0.

https://doi.org/10.1371/journal.pone.0282161.t006

thumbnail
Table 7. Computed sample size, estimated assurance probability, and simulated assurance probability for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, assurance probability 1 –γ = 0.80, and covariate disparity θ = 0.25.

https://doi.org/10.1371/journal.pone.0282161.t007

thumbnail
Table 8. Computed sample size, estimated assurance probability, and simulated assurance probability for 95% confidence interval of standardized contrast when the number of groups G = 3, standardized contrast effect ψ* = 0.5, assurance probability 1 –γ = 0.80, and covariate disparity θ = 0.50.

https://doi.org/10.1371/journal.pone.0282161.t008

It can be seen from the results of the proposed approach in Tables 38 that larger sample sizes are required for smaller expected width threshold ω and the results for ω = 1 are almost twice as those of ω = 1.5 in the same table. Also, the assurance probability principle demands larger sample sizes than the expected width criterion for the settings examined here. It is expected that the necessary sample sizes will increase for higher assurance probability levels 1 –γ > 0.80. The computed sample sizes of the approximate method also have similar patterns with respect to the width threshold and precision criteria. However, because the omission of covariate features, the computed sample sizes do not vary with the number of covariates P and the covariate disparity θ. The estimated expected width and estimated assurance probability of the exact and approximate procedures are also listed in Tables 38.

In the second stage, simulated values of expected width and assurance probability associated with the reported sample sizes and selected parameter configurations are computed through a Monte Carlo study of 10,000 independent data sets. For each replicate, NT sets of covariate values are generated from the prescribed multinormal distribution. These values of covariates, treatment effects, and error variance σ2, in turn, determine the mean responses for generating NT normal outcomes of the ANCOVA model. Because the covariate coefficients are nuisance parameters, they are set as β1 = … = βP = 0.5 in the precision analysis. According to the differences between the estimated and simulated precision quantities of Ω and Γ in Tables 38, the exact approach results in outstanding performance for all 90 cases. In contrast, the approximate method does not provide accurate sample size calculations for most of the cases. Note that the simplified precision assessments Ωa and Γa are presumed valid under the assumption of randomized designs, such as the situations with covariate disparity θ = 0 in Tables 3 and 6. However, the estimated expected width Ωa is consistently less than the simulated expected width for all the cases in Table 3. The results in Table 6 reveal that the assurance probability calculation Γa is substantially larger than the simulated assurance probability. Therefore, the approximate method generally underestimates the sample sizes and the deficiency phenomenon is more prominent and noticeably problematic for the cases with covariate disparity θ = 0.25 and 0.50 in Tables 4, 5, 7 and 8. Consequently, the usefulness of the approximate procedures is extremely limited and the exact techniques are recommended for precision assessments and sample size determinations. Additional numerical assessments were also conducted for non-normal error t(10) and non-normal covariates: Exponential(1), Gamma(5, 5−1/2), Laplace(0, 1), Log normal(0, 1/4), and Uniform(0, 1). The results show that the precision outcomes for expected width and assurance probability are marginally affected by the violations of normal assumption. Overall, the suggested confidence interval and sample size procedures preserve reasonably good performance under various non-normal error and covariate situations. To facilitate the execution of the suggested sample size procedures, SAS/IML and R computer algorithms are presented in the supplemental file. For example, the user specifications of the supplemental SAS file B for performing sample size calculations to obtain designated expected width are as follows:

  1. PROC IML;
  2. *USER SPECIFICATION PORTION;
  3. *DESIGNATED ALPHA;ALPHA = 0.05;
  4. *NUMBER OF GROUPS;G = 3;
  5. *NUMBER OF COVARIATES;P = 5;
  6. *VARIANCE;SIGSQ = 1;
  7. *GROUP RATIOS;RVEC = J(1,G,1);
  8. *CONTRAST COEFFICIENTS;CVEC = {1–0.5–0.5};
  9. *STANDADIZED CONTRAST;PSIS = 0.5;
  10. *COVARIATE DISPARITY;THETA = 0;
  11. *DESIGNATED WIDTH;OMEGA = 1.5;
  12. *END OF USER SPECIFICATION PORTION;

Moreover, the user specifications of the supplemental SAS file C, for performing sample size calculations to ensure adequate assurance probability of achieving the desired width, are listed as

  1. PROC IML;
  2. *USER SPECIFICATION PORTION;
  3. *DESIGNATED ALPHA;ALPHA = 0.05;
  4. *NUMBER OF GROUPS;G = 3;
  5. *NUMBER OF COVARIATES;P = 5;
  6. *VARIANCE;SIGSQ = 1;
  7. *GROUP RATIOS;RVEC = J(1,G,1);
  8. *CONTRAST COEFFICIENTS;CVEC = {1–0.5–0.5};
  9. *STANDADIZED CONTRAST;PSIS = 0.5;
  10. *COVARIATE DISPARITY;THETA = 0;
  11. *DESIGNATED WIDTH;OMEGA = 1.5;
  12. *ASSURANCE PROBABILITY;AP = 0.80;
  13. *END OF USER SPECIFICATION PORTION;

The corresponding specifications of R programs for expected width and assurance probability are presented in supplemental files E and F, respectively:

  1. ancova.scie.apx2.fun(alpha = 0.05, g = 3, p = 1, sigsq = 1, rvec = c(1,1,1), cvec = c(1,-0.5,-0.5), psis = 0.5, theta = 0, omega = 1.5)

and

  1. ancova.scie.apx3.fun(alpha = 0.05, g = 3, p = 1, sigsq = 1, rvec = c(1,1,1), cvec = c(1,-0.5,-0.5), psis = 0.5, theta = 0, omega = 1.5, ap = 0.8).

It should be emphasized that the analytic complexity requires computer algorithms to compute the exact confidence intervals and optimal sample sizes under various design configurations. The developed computer programs substantially facilitate the recommended confidence interval and sample size techniques in practical applications. Users can easily alter the exemplifying settings in the programs with their designated model specifications.

Conclusions

Measures of effect size and confidence intervals are extremely useful for comparing quantitative information across different studies. This research describes and compares exact and approximate confidence intervals for standardized contrast effects in ANCOVA. The theoretical and numerical findings show that the exact approach has analytic support and empirical advantage over the other two approximate methods using asymptotic theory and direct division. The advanced aspects of precision appraisals and sample size calculations are also investigated for interval estimation of standardized contrasts. The existing study provided a shortcut under the speculation that covariate disparity is unlikely to exist in randomized designs. However, the approximate technique actually does not give reliable precision and sample size outcomes even when there is no covariate disparity. The proposed exact approach has the distinct utility of accommodating the full distributional information of normal covariates, especially the potential disparity of unequal means for both random and nonrandom ANCOVA designs. Overall, the described results of interval estimation, precision assessment, and sample size planning for standardized contrasts update and expand upon current work of ANCOVA in the literature.

Supporting information

S1 File. SAS programs for standardized contrast effects in ANCOVA designs.

https://doi.org/10.1371/journal.pone.0282161.s001

(PDF)

S2 File. R programs for standardized contrast effects in ANCOVA designs.

https://doi.org/10.1371/journal.pone.0282161.s002

(PDF)

References

  1. 1. Hedges L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128.
  2. 2. Cohen J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
  3. 3. Grissom R. J., & Kim J. J. (2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). New York, NY: Routledge.
  4. 4. Kline R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
  5. 5. Lin L., & Aloe A. M. (2021). Evaluation of various estimators for standardized mean difference in meta-analysis. Statistics in Medicine, 40, 403–426. pmid:33180373
  6. 6. Takeshima N., Sozu T., Tajika A., Ogawa Y., Hayasaka Y., & Furukawa T. A. (2014). Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?. BMC Medical Research Methodology, 14, 30. pmid:24559167
  7. 7. Zhang X. D. (2010). Strictly standardized mean difference, standardized mean difference and classical t-test for the comparison of two groups. Statistics in Biopharmaceutical Research, 2, 292–299.
  8. 8. Zhang X. D., & Heyse J. F. (2012). Contrast Variable for Group Comparisons in Biopharmaceutical Research. Statistics in Biopharmaceutical Research, 4, 228–239.
  9. 9. Odgaard E. C., & Fowler R. L. (2010). Confidence intervals for effect sizes: Compliance and clinical significance in the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology, 78, 287–297.
  10. 10. Smithson M. (2003). Confidence intervals. Thousand Oaks, CA: Sage.
  11. 11. Steiger J. H., & Fouladi R. T. (1997). Noncentrality interval estimation and the evaluation of statistical models. In Harlow L. L., Mulaik S. A., & Steiger J. H. (Eds.), What if there were no significance tests? (pp. 222–257). Mahwah, NJ: Lawrence Erlbaum.
  12. 12. Tian L. (2007). Inferences on standardized mean difference: The generalized variable approach. Statistics in Medicine, 26, 945–953. pmid:16708348
  13. 13. Wu J., Jiang G., & Wei W. (2006). Confidence intervals of effect size in randomized comparative parallel-group studies. Statistics in Medicine, 25, 639–651. pmid:16158411
  14. 14. Zou G. Y. (2007). Exact confidence interval for Cohen’s effect size is readily available. Statistics in Medicine, 26, 3054–3056. pmid:17310499
  15. 15. Steiger J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9, 164–182.
  16. 16. Levin J. R. (1997). Overcoming feelings of powerlessness in “aging” researchers: A primer on statistical power in analysis of variance designs. Psychology and Aging, 12, 84–106. pmid:9100270
  17. 17. Olejnik S., & Algina J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286. pmid:10873373
  18. 18. Lai K., & Kelley K. (2012). Accuracy in parameter estimation for ANCOVA and ANOVA contrasts: Sample size planning via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 350–370. pmid:22004142
  19. 19. Ciolino J. D., Renee’ H, M., Zhao W., Jauch E. C., Hill M. D., & Palesch Y. Y. (2014). Continuous covariate imbalance and conditional power for clinical trial interim analyses. Contemporary Clinical Trials, 38, 9–18. pmid:24607294
  20. 20. Ciolino J. D., Martin R. H., Zhao W., Hill M. D., Jauch E. C., & Palesch Y. Y. (2015). Measuring continuous baseline covariate imbalances in clinical trial data. Statistical Methods in Medical Research, 24, 255–272. pmid:21865270
  21. 21. Egbewale B. E., Lewis M., & Sim J. (2014). Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: A simulation study. BMC Medical Research Methodology, 14, 49. pmid:24712304
  22. 22. Kahan B. C., & Morris T. P. (2013). Adjusting for multiple prognostic factors in the analysis of randomised trials. BMC Medical Research Methodology, 13, 99. pmid:23898993
  23. 23. Shieh G. (2017). Power and sample size calculations for contrast analysis in ANCOVA. Multivariate Behavioral Research, 52, 1–11. pmid:28121163
  24. 24. Shieh G. (2020). Power analysis and sample size planning in ANCOVA designs. Psychometrika, 85, 101–120. pmid:31823115
  25. 25. Shieh G. (2021). Appraising minimal effect of standardized contrasts in ANCOVA designs: Statistical power, sample size, and covariate imbalance considerations. Statistics in Biopharmaceutical Research, 13, 468–475.
  26. 26. Tang Y. (2018). Exact and approximate power and sample size calculations for analysis of covariance in randomized clinical trials with or without stratification. Statistics in Biopharmaceutical Research, 10, 274–286.
  27. 27. Tang Y. (2018). A noniterative sample size procedure for tests based on t distributions. Statistics in Medicine, 37, 3197–3213.
  28. 28. Tang Y. (2020). Notes on exact power calculations for t tests and analysis of covariance. Statistics in Biopharmaceutical Research, 12, 394–396.
  29. 29. Shieh G. (2019). On sample-size calculations for precise contrast analysis in ANCOVA. Journal of Experimental Education, 87, 238–259.
  30. 30. Rencher A. C., & Schaalje G. B. (2007). Linear models in statistics (2nd ed.). Hoboken, NJ: Wiley.
  31. 31. Johnson N. L., Kotz S., & Balakrishnan N. (1995). Continuous univariate distributions (2nd ed., Vol. 2). New York, NY: Wiley.
  32. 32. Casella G., & Berger R. L. (2002). Statistical inference (2nd ed.). Pacific Grove, CA: Duxbury.
  33. 33. Mood A. M., Graybill F. A., & Boes D. C. (1974). Introduction to the theory of statistics (3rd ed.). New York: McGraw-Hill.
  34. 34. Venables W. (1975). Calculation of confidence intervals for noncentrality parameters. Journal of the Royal Statistical Society: Series B, 37, 406–412.
  35. 35. Bird K. D. (2002). Confidence intervals for effect sizes in analysis of variance. Educational and Psychological Measurement, 62, 197–226.
  36. 36. Fidler F., & Thompson B. (2001). Computing correct confidence intervals for ANOVA fixed- and random-effects effect sizes. Educational and Psychological Measurement, 61, 575–604.
  37. 37. Fleiss J. L. (2011). Design and analysis of clinical experiments (Vol. 73). New York, NY: Wiley.
  38. 38. Hochberg Y., & Varon-Salomon Y. (1984). On simultaneous pairwise comparisons in analysis of covariance. Journal of the American Statistical Association, 79, 863–866.
  39. 39. Kupper L. L., & Hafner K. B. (1989). How appropriate are popular sample size formulas? The American Statistician, 43, 101–105.
  40. 40. Anderson T. W. (2003). An introduction to multivariate statistical analysis (3rd ed.). New York: Wiley.
  41. 41. Muirhead R.J. (1982). Aspects of multivariate statistical theory. New York, NY: Wiley.
  42. 42. Chase G. R., & Bulgren W. G. (1971). A Monte Carlo investigation of the robustness of T2. Journal of the American Statistical Association, 66, 499–502.
  43. 43. Everitt B. S. (1979). A Monte Carlo investigation of the robustness of Hotelling’s one-and two-sample T2 tests. Journal of the American Statistical Association, 74, 48–51.
  44. 44. Holloway L. N., & Dunn O. J. (1967). The robustness of Hotelling’s T2. Journal of the American Statistical Association, 62, 124–136.
  45. 45. Hopkins J. W., & Clay P. P. F. (1963). Some empirical distributions of bivariate T2 and homoscedasticity criterion M under unequal variance and leptokurtosis. Journal of the American Statistical Association, 58, 1048–1053.
  46. 46. Kariya T. (1981a). A robustness property of Hotelling’s T2-test. The Annals of Statistics, 9, 211–214.
  47. 47. Kariya T. (1981b). Robustness of multivariate tests. The Annals of Statistics, 9, 1267–1275.