As Fredrickson notes, "[t]he Discovery study sample alone is too small to provide a well-powered mixed effect linear model analysis". We concur. We would add that this sample is also too small to provide a well-powered linear regression analysis—as was conducted in Fredrickson et al.'s 2013 PNAS article—with 17 predictors. However, the PNAS article remains part of the scientific record, with its Figure 2A (equivalent to Figure 1C in Fredrickson et al.'s 2015 PLoS ONE article) still uncorrected to take into account the coding error in the dataset. We also note that Fredrickson's comment in effect insists that the only relation worth considering is that between eudaimonic well-being and CTRA gene expression. This made us wonder what, if any, aspects of the 2013 PNAS article (in which hedonic and eudaimonic well-being were treated as having exactly equal importance, albeit with opposite signs) are left intact. Under such circumstances, we do not consider it to be “selective” to examine the consequences of applying Fredrickson et al.'s mixed-effects linear model to this dataset, especially when this revealed an association between CTRA gene expression and (only) hedonic well-being. We also wonder why Fredrickson chose to focus her comment on the standard errors of this reanalysis, which are necessarily higher (although, we would argue, not excessively so) than those of the N = 122 study, because the sample size is smaller. Of more interest, it seems to us, is the radical differences in parameter estimates between the two studies, which border on being totally opposite to each other, as well the almost entirely different results obtained from the mixed-effects linear model with the Discovery sample compared to the original regression-based model used in the 2013 PNAS article.

Walker (2016), who completed an independent reanalysis of Fredrickson et al’s (2013) PNAS paper, pointed out that a mixed-effects linear model is no magic bullet for fixing an inadequate sample size. We agree with his argumentation. Fredrickson et al.'s (2015) model requires the estimation of 69 regression coefficients and 53 variance parameters. In these circumstances, it seems to us that neither 122, nor 198, nor 229, nor indeed 305 (Discovery + Confirmation + Generalization) is likely to be an adequate sample size, as any such model is likely to be overfitted with a sample size that does not run into the thousands. Yet, given that the largest part of Fredrickson et al.'s 2015 article concentrated on the Confirmation study, the apparent conclusion is that while N = 76 was an inadequate sample size, N = 122 was just fine. This would seem to require a very careful a priori power analysis to have been performed; however, at no point in this entire process has any power analysis been published.

Continuing on the topic of the sample size, we consider that the invocation of a total of 229 participants from two new samples lacks relevance. This number is composed of the N = 122 Confirmation sample (using the same psychological variables as the N = 76 PNAS article) and the N = 107 Generalization sample. But the latter used completely different outcome variables (based on Ryff's model of psychological well-being), which has not been shown to be measuring equivalent constructs to the MHC-SF. Furthermore, the analysis strategies used for these two samples were different, as documented by Fredrickson et al. on p. 14 of their PLoS ONE article. It therefore seems rather optimistic to suggest that these two studies are both measuring the construct of eudaimonic well-being. (Of course, Fredrickson and colleagues have still not even demonstrated that the MHC-SF measures two, rather than three, dimensions of well-being; indeed, in both our PNAS and PLoS replies, we have produced multiple pieces of empirical and theoretical evidence to the contrary). Additionally, these two new samples were (necessarily) analysed separately, so that there was no possibility of using this “combined sample size” of 229 to avoid the problems of overfitting mentioned previously. Pooling outcomes from two separate analyses cannot extract a signal, post hoc, from the noise of each sample. Thus, the viability of Fredrickson et al.’s mixed-effects linear model must stand or fall on the power of the N = 122 Confirmation sample.

In summary, we feel that Fredrickson's comment is missing two crucial elements. First, we strongly believe that a power analysis, showing why N = 122 is sufficient for the mixed-effects linear model (given that everyone agrees that N = 76 is inadequate), should have been performed. Second, the apparent change of hypotheses from the first (PNAS, 2013) article—which demonstrated, as hypothesised, associations between CTRA gene expression and both hedonic and eudaimonic well-being—to the second (PLoS ONE, 2015) article—in which associations between CTRA gene expression and hedonic well-being were neither hypothesised nor found—should have been explained.

Nicholas J. L. Brown
Douglas A. MacDonald
Manoj P. Samanta
Harris L. Friedman
James C. Coyne

Reference
Walker JA (2016) The opposing effects of hedonic and eudaimonic happiness on gene expression is correlated noise. Manuscript submitted for publication. Preprint available at http://biorxiv.org/conten...