https://orcid.org/0000-0001-7109-7717
Figures
Abstract
Two related methods, Quasi-ANCOVA and Dual-Centered ANCOVA, have recently been suggested as a way to get greater power when analyzing data from a before and after study. Both methods use group-mean centering where the groups are the treatment and comparison groups. Group-mean centering creates a generated regressor problem. When the estimated standard errors are corrected for the generated regressor problem, there is no longer any gain in power. The corrected Quasi-ANCOVA estimates are identical to those from ANOVA and the corrected Dual-Centered ANCOVA estimates are identical to those from the differences model. These conclusions are derived analytically and also verified using a Monte Carlo simulation.
Citation: Brorsen BW, Lin H, Larzelere RE (2025) Critique of enhanced power claimed for Quasi-ANCOVA and Dual-Centered ANCOVA. PLoS ONE 20(1): e0317860. https://doi.org/10.1371/journal.pone.0317860
Editor: Surya Prakash Bhatt, All India Institute of Medical Sciences, INDIA
Received: March 15, 2024; Accepted: January 5, 2025; Published: January 22, 2025
Copyright: © 2025 Brorsen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The analysis is based on analytical derivations and a Monte Carlo simulation and so does not use data. An example copy of the R program used for the Monte Carlo simulation is included in the supplementary materials.
Funding: This research benefited from funding from NICHD grant #5 R03 HD107307 to Robert Larzelere and Hua Lin. Brorsen receives funding from the Oklahoma Agricultural Experiment Station, and United States Department of Agriculture National Institute of Food and Agriculture [Hatch Project number OKL03170]. The funders had no role in study design, analysis, decision to publish, or preparation of manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Two methods of analyzing data that have been suggested recently include Quasi-ANCOVA for randomized pre-post data [1] and Dual-Centered ANCOVA for longitudinal data over two occasions [2]. When using standard errors from ordinary least squares (OLS), these methods both appear to duplicate the corresponding estimated treatment effects of ANOVA and differences-in-differences, respectively, while generally increasing statistical power. This note argues that there is no increase in power from these methods once standard errors are corrected for the presence of a generated regressor. As Pagan [3] argued, in the presence of a generated regressor, the standard errors need to be calculated using the two-stage least squares (2SLS) formula. As noted by Greene [4] (p. 58) the key difference is that the 2SLS formula for the residual variance uses the actual value of the variable rather than the generated value. Once the 2SLS formula for the residual variance is applied, the estimated treatment effect of quasi-ANCOVA and its standard error are the same as with analysis of variance (ANOVA). Similarly, estimates and inference from Dual-Centered ANCOVA are the same as with the differences model. Thus, there is no gain in power from using either Quasi-ANCOVA or Dual-Centered ANCOVA once the correct standard errors are used.
Alternative models
First consider three basic models: (i) Analysis of Variance (ANOVA), (ii) differences, and (iii) Analysis of Covariance (ANCOVA). The ANOVA model is
(1)
where Yij1 is the posttest score (the subscript 1 indicates posttest) of the ith person that receives the jth treatment, Xj is an indicator variable for the jth treatment (X2 = 1 for those receiving the treatment and X1 = 0 for the control group), and εij is the mean-zero error term.
Many behavioral studies are longitudinal and thus provide both a pretest and a posttest. One approach to analyze such data is difference analysis (it is a special case of differences in differences when the time series has length two). The differences model is
(2)
where Yij0 is the pretest score and ϑij is a mean-zero error term that differs from εij.
The ANCOVA model relaxes the differences model’s assumption that the coefficient on the pretest score is one:
(3)
where vij is the ANCOVA error term. Huitema [1] suggests Quasi-ANCOVA as an alternative to ANCOVA in randomized studies when one or more covariate test scores are not obtained until after the treatment has begun. We limit the exposition herein to the case of a late pretest score.
Quasi-ANCOVA
Now consider the generated regressor problem created by Quasi-ANCOVA [1]. Quasi-ANCOVA modifies (3) by subtracting the group means of the pretest score from each pretest score. Following Pagan’s [3, p. 227] Model 2, Quasi-ANCOVA can be represented as a two-equation system (a recursive simultaneous equation system):
(4)
(5)
where the group means of the pretest score
are the predicted values of Eq (5) and τij0 is a random error term. Note that the same symbol is used for the error term in Eqs (3) and (4) since the error terms are equal. Since Quasi-ANCOVA uses the predictions of the second equation in calculating an explanatory variable, it creates a generated regressor problem. Note that the term
is orthogonal to the treatment effect Xj by construction (the term
is equal to the error term of (5) and with OLS, the explanatory variable Xj and the error are assumed uncorrelated). Therefore, the estimated treatment effect with Quasi-ANCOVA will be the same as with ANOVA
. As Pagan [3] argues, the standard errors for the estimate of γ1 in Eq (4) will be too small if calculated using the OLS formula. Instead, standard errors need to be calculated using the two-stage least squares formula (this is the same formula as with instrumental variables). In the 2SLS formula, the standard errors in (4) are calculated using the actual value of Yij0 rather than its predicted value
. Letting
in (4) gives
, and so the 2SLS standard deviation of the error in (4) is exactly the same as the OLS standard deviation from ANOVA in Eq (1). Thus, the estimated treatment effect and its standard error are exactly the same in Quasi-ANCOVA as in ANOVA.
Note that Pagan [3] argues that generated regressor problems can also be solved by using full information maximum likelihood (FIML). That does not work with Quasi-ANCOVA as the parameters are not all identified (the parameter γ2 cannot be estimated with FIML since Eq (5) has no exogenous variable that is not also in Eq (4) and so the order condition for identification is not met).
An alternative way to think about Quasi-ANCOVA is as a seemingly unrelated regression [4, p. 328]. Note that the added regressor in (4) is the same as the residual in (5): . Therefore, the Quasi-ANCOVA model can also be represented as
(6)
(7)
where cov(εij, τij0) ≠ 0. From (4), the Quasi-ANCOVA assumes εij = γ2 τij0 + vij and so cov(εij, τij) = cov(γ2 τij + vij, τij). Since the errors in (6) and (7) are correlated, it would appear that there could be a potential gain in efficiency from using seemingly unrelated regressions. However, since both equations have the same explanatory variables, there is no gain to using seemingly unrelated regressions in this case. That is, εij represents the residuals after regressing Yij1 on Xj. Eq (4) merely divides those residuals into two parts: a portion that covaries with the residuals in regressing Yij0 on Xj and a remaining portion. Thus, Eqs (6) and (7) provide an alternative argument to reach the same conclusion: Quasi-ANCOVA and ANOVA produce the same estimates of coefficients and standard errors when the Quasi-ANCOVA standard errors are calculated correctly.
Dual-Centered ANCOVA
Dual-Centered ANCOVA as defined by Lin & Larzelere [2] is the same as Quasi-ANCOVA except the pretest score is subtracted from the left side:
(8)
(9)
As before, is orthogonal to Xj by construction. As a result, the treatment effect with Dual-Centered ANCOVA is the same as the differences model in (2). The 2SLS standard error is also the same as the standard error in (2). Note that
must be replaced by Yij0 on both the left side and the right side of Eq (8) when calculating the standard errors for the estimate of ω1. Thus, there is also no gain in using Dual-Centered ANCOVA once the standard errors are corrected for the presence of a generated regressor.
Monte Carlo study
A Monte Carlo study in Table 1 is included to verify the analytical results. The simulation is based on the null hypothesis of a randomized pre-post design, in which the treatment effects are unbiased for Eqs (2) and (3). The simulation notation uses Eq (3) with the intercept set at 130 although its value is immaterial, the treatment effect (δ in Eq (2) and α1 in Eq (3)) is set at zero in all simulations,and varying values of α2 are used. The simulation included 1,000 replications.
Table 1 presents selected results of the Monte Carlo study. The key column of interest is the actual standard deviation of the estimated treatment effect (sd()). This column shows that the standard deviation of the estimated treatment effects for ANOVA and Quasi-ANCOVA are identical. Further, the standard deviation of the estimated treatment effects of Differences and Dual-Centered ANCOVA are also exactly the same. Thus, the Monte Carlo results in Table 1 exactly match the theory.
Note that Quasi-ANCOVA produces the treatment effects of the ANOVA model, yet it uses the residual standard deviation and the treatment effect standard error of the ANCOVA model. Similarly, dual-centered ANCOVA produces the treatment effects of the differences model, but uses the residual standard deviation and treatment effect standard error of the ANCOVA model. All of the methods produce unbiased estimates of the treatment effects under the stated assumptions. The problem with Quasi-ANCOVA and Dual-Centered ANCOVA is in the calculation of the standard error.
Interaction effects
As in Lin and Larzelere [2], an interaction term can be added to the dual-centered ANCOVA model in (8), which provides
(10)
We know from previous arguments that the coefficient on the treatment variable will be the same as in the differences model since the two added terms are orthogonal. We also know that 2SLS standard errors (where all three are replaced by Yij0 in the calculation) will need to be used for ω1. OLS standard errors can be used for estimates of γ2 and γ3 [3]. Now consider the ANCOVA model with an interaction term
(11)
From algebra (and verified with Monte Carlo simulation), α2 = γ2 + 1 and α3 = γ3. So, Dual-Centered ANCOVA provides the treatment effect of the differences model and the interaction effect from the ANCOVA model in (11), including the correct standard error for the interaction effect, but not for the treatment effect.
Discussion
Quasi-ANCOVA and dual-centered ANCOVA do not have increased power once the standard errors are corrected for the statistical problem of using generated regressors. In fact, quasi-ANCOVA is identical to ANOVA and dual-centered ANCOVA is identical to the differences model. Thus, the claim of enhanced power for these two techniques does not hold.
Given that group-mean centering can cause the standard errors reported by standard statistical packages to be incorrect, it raises the question: when does group-mean centering create a generated regressor problem? Group-mean centering is often used to disaggregate within-group fixed effects and between-group fixed effects, but can create unforeseen problems [5]. The first point that we want to make is that with both quasi-ANCOVA and dual-centered ANCOVA, the group used for the group mean is the treatment condition, which creates an identification problem. Centering around individual means does not always create an identification problem and so FIML can work in some cases.
Further, note that grand-mean centering does not create any statistical problems. Grand-mean centering is a linear transformation of an explanatory variable and so it changes nothing other than the intercept.
Hill et al. [6, p. 124] discuss three different methods of using group-mean centering:
- The demeaned predictor variable (i.e.,
) can then be included with the group mean
, resulting in
. Second, the group mean
can be included along with the unaltered, or raw, predictor xij, resulting in y = a + B1xij + B2
. Third is to simply use the demeaned value, resulting in
.
Pagan’s [3] Model 4 directly addresses Hill’s first method. He argues that OLS standard errors are asymptotically correct for B2 (there could still be a benefit from a degrees of freedom adjustment in small samples), while 2SLS would be needed for B1. With Hill’s model two, it is B1 that can use OLS standard errors and 2SLS standard errors are needed for B2. As Pagan [3] discusses, OLS standard errors can be used for Hill’s model three.
Bliese et al. [7, p. 81] suggests using random effects in the demeaned case because standard OLS packages will use an incorrect degrees of freedom. Using random effects does not solve the generated regressor problem.
Hoffman [8] also discusses baseline centering in longitudinal studies, where the values are centered around the initial value at the baseline time. Since the baseline value is not estimated, there is no generated regressor problem.
Zyphur et al. [9] consider the special case of a general cross-lagged model for panel data. In this case, group-mean centering can introduce dynamic panel bias. Dynamic panel bias is a different problem than generated regressors and so it is not addressed here.
Similarly, Rights and Sterba [5] consider the problem of random conflated slopes (slope heterogeneity) in Hill’s Model 1 and Model 2. The problem that Rights and Sterba [5] consider is distinct from the generated regressor problem, but does indicate that fixed effects can be biased and standard errors incorrect in complex multilevel models.
As Pagan [3] (p. 283) says “the use of predictors or residuals does not necessarily lead to efficiency losses or incorrect standard errors” so it is hard to develop a general rule. But, in many cases, standard computer programs will not provide the correct standard errors when using group-mean centered data. The correct standard errors can usually be calculated using the 2SLS formula for standard errors or by estimating the equations jointly using full-information maximum likelihood.
References
- 1.
Huitema B. The analysis of covariance and alternatives: Statistical methods for experiments, quasi-experiments, and single-case studies (2nd ed.). New York: John Wiley & Sons; 2011.
- 2. Lin H, Larzelere RE. Dual-Centered ANCOVA: Resolving contradictory results from Lord’s paradox with implications for reducing bias in longitudinal analyses. Journal of Adolescence. 2020; 85, 135–147. pmid:33242670
- 3. Pagan A. Econometric issues in the analysis of regressions with generated regressors. International Economic Review. 1984; 25(1): 221–247.
- 4.
Greene WH. Econometric analysis. 8th edition. London: Pearson; 2018.
- 5. Rights JD, Sterba SK. On the common but problematic specification of conflated random slopes in multilevel models. Multivariate Behavioral Research. 2023; 58(6):1106–33. pmid:37038722
- 6. Hill AD, Johnson SG, Greco LM, O’Boyle EH, Walter SL. Endogeneity: A review and agenda for the methodology-practice divide affecting micro and macro research. Journal of Management. 2021; 47(1): 105–143.
- 7. Bliese PD, Schepker DJ, Essman SM, Ployhart RE. Bridging methodological divides between macro-and microresearch: Endogeneity and methods for panel data. Journal of Management. 2020; 46(1): 70–99.
- 8.
Hoffman L. Longitudinal analysis: Modeling within-person fluctation and change. Philadelphia: Routledge; 2015.
- 9. Zyphur MJ, Voelkle MC, Tay L, Allison PD, Preacher KJ, Zhang Z, et al. From data to causes II: Comparing approaches to panel data analysis. Organizational Research Methods. 2020; 23(4): 688–716.