P-value selection method

P-curvers are instructed to include the p-value for the hypothesized outcome [8]. Modern researchers, however, tend to include more than one outcome measure or propose a broad hypothesis that can encompass multiple outcome measures. For instance, in a paper included in the p-curve analyses of the power posing debate [21,22], Huang et al. hypothesized that power posing—nonverbal displays characterized by expansive and open postures—affects the poser’s "thought and behavior" 24, p. 97. Huang et al. operationalized and tested their hypothesis using a series of assessments, including an abstraction task and a measure of risk orientation. Which p-value is the p-curver to include, the p-value for the abstraction task or for the risk measure? When multiple p-values are reported for a stated hypothesis, p-curvers are instructed to establish a rule (e.g., "pick the first p-value reported, pick the last, pick the highest, pick the lowest") and to follow that rule for selecting p-values for all papers in the sample [25]. For Simonsohn and colleagues, which p-value is extracted from a particular study is not considered beyond being clearly specified and applying the rule consistently: If there is evidential value, the distribution of p-values from a sample of studies produces a right skew. However, we question whether it is of no consequence which p-value from a specific study is included in the p-curve analysis.

We submit that such an inclusion rule is prone to produce an idiosyncratic sample of p-values that can influence the results of the p-curve analysis. Most p-curvers tend to use the rule "include the first p-value reported" (e.g., [26–29]), but other p-curvers have used the p-value associated with the largest sample [30] or selected a p-value at random [31], as discussed in [32]. Moreover, as mentioned, modern researchers often include and report multiple outcome measures. So, which p-value do authors typically report first? The most significant? The one they p-hacked? The most "interesting"? There is no consensus for the order in which authors report the findings from a set of significant outcomes.

The arbitrary order authors use to report their findings, combined with the inclusion rule used by p-curvers, results in any significant p-value perhaps having an equal chance of inclusion in the p-curve analysis. This arbitrariness can have a dramatic impact on the results of a p-curve analysis. Consider the hyperstylized example presented in Table 1, in which three studies each present the results from three outcome measures. The three p-values arose from three different hypothesized outcome measures, and the order in which they were reported in the results section was arbitrary. One effect from each study is highly significant (p = .001) and the other two effects are closer to the p < .05 criterion: p = .025 and p = .049. The p-curver is instructed to include only one p-value from each study. There are 27 possible permutations of p-values that could be included in a p-curve analysis. Given both the arbitrary nature of the authors’ reporting order and the researcher’s selected inclusion rule, 3.7% of the time (1 out of 27) a p-curver would extract the three p = .001 values, and the resulting p-curve analysis would produce a right-skewed distribution and support for the conclusion that evidential value is present. Alternatively, if or when authors present their findings in a different order and/or p-curvers use a different selection rule, 3.7% of the time the p-curver would include the three p = .049 values, which would produce a strong left-skewed distribution and the conclusion that the exact same set of studies reflects the absence of evidential value (and perhaps strong evidence of p-hacking).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Hypothetical p-values reported in an arbitrary order in three different papers.

https://doi.org/10.1371/journal.pone.0305193.t001

The arbitrarily selected p-values also impact the estimated power of the sample of studies under consideration. The estimated true power determines the "shape" of the p-curve: the larger the true effect, the more right skew that is expected to be evident (both when p-hacking is present and absent; see Fig 1 from [8]). Consider the p-values from the original Carney et al. paper from the power posing literature [33]. Carney and colleagues examined whether power posing would affect the poser’s physiological processes, behavior, and psychological state. Carney et al. reported, in order, the effect for testosterone (p = .045), cortisol (p = .009), riskiness (p = .049), power feelings (p = .003) [33], and thus contributed a single p-value to Simmons and Simonsohn’s analysis, the p = .045 for testosterone [22]. The p = .045 indicates that the power of this study was .52, but if Carney et al. had happened to mention first, say, cortisol, the power from the same study would be estimated at .75. Thus, when a study reports two p-values, say, one at .009 and one at .045, the arbitrary selection method not only determines the distribution of p-values, but the magnitude of the power estimated from the p-curve analysis.

Purpose of the studies

The goal of this research was to investigate the impact that different combinations of p-values reported across and within studies have on the results of a p-curve analysis, and thus whether p-curve provides reliable or meaningful insight into whether a set of findings has evidential value. Using recent p-curve analyses from the (a) power posing phenomenon [21,22] and the (b) HPA axis debate [23], we investigated whether the conclusions from the p-curve analyses were dependent on which specific p-values were selected for analysis.

The power pose debate

Carney et al.’s seminal article investigated the impact of power posing on the poser’s mental and physiological state. They found that participants who posed using "expansive" versus "contractive" poses (i.e., limbs close to the body) exhibited higher testosterone levels, reduced cortisol levels, greater feelings of power, and made more risky decisions [33]. Their findings sparked a resurgence of interest in the impact of power poses (as first observed by Riskind and Gotay [38]), as evident by scores of power posing studies (for an expansive list, see [39]), one of the most viewed TedTalks [40], and the considerable attention it received in the popular media (e.g., [41–43]).

Confidence in their findings began to erode when a commentary concluded that, based on a p-curve analysis, power posing "ought to be treated as hypotheses currently lacking in empirical support" 22, p. 690 In their analysis of the power posing literature, Simmons and Simonsohn tested whether the 33 papers summarized by Carney et al.’s qualitative review [44] provided evidence in support of the power posing effect. Simmons and Simonsohn extracted 31 p-values from those papers, and after removing the non-significant p-values (specifically, two-tailed p > .05), 24 p-values were analyzed. They concluded that the distribution of those p-values revealed that evidential value was absent because there was no evidence of right skew and there was evidence of flatness [22] (categorized in this research as "underpowered"). In response to those findings, Cuddy et al. conducted a p-curve analysis of their own, which included an additional 20 power posing papers, and concluded that evidential value was present in an aggregated p-curve across different types of outcome variables, including for subsets of outcomes, specifically, for "feelings of power," and "EASE" (Emotion, Affect, and Self-Evaluation), but not for "non-EASE" outcomes (e.g., pain control, hormones, and risk-taking) [21].

Data preparation

The first step was to create the databases of relevant p-values. The disclosure table for both analyses is presented in the (S3 and S5 Files).

For the Simmons and Simonsohn sample, in addition to the 24 significant p-values reported originally [22], inspection of the set of articles produced an addition of 17 significant p-values. This sample, however, included 42 p-values rather than 41 p-values. Whereas p-curve excludes non-significant p-values from all analyses, non-significant values potentially hold informational value in IPA. For instance, Simmons and Simonsohn did not include a significant p-value for Experiment 1 from Welker et al. [45] because the first reported analysis, the omnibus test, was not significant. However, Welker et al. reported a second set of simple effects that would be included in an IPA. Simmons and Simonsohn [22] "non-significant" notation thus provides informational value through its ability to inhibit a set of permutations that include a significant p-value. To replicate Simmons and Simonsohn’s analyses, non-significant values were excluded from the reported analyses (except for a non-significant value for the aforementioned Welker et al. paper). Results of the IPA that include non-significant values are reported in the (S4 File).

For the Cuddy et al. p-curve (aggregate) analysis, its database contained 42 significant p-values [21], and 48 additional significant p-values were identified and included. This sample, however, was only computationally feasible after removing two studies ([46, 47], Experiment 2]. We eliminated these two studies because they would have multiplied the number of permutations by 36 and increased the number of permutations to over 2.1 billion. These studies were selected for omission because they both included (a) a large number of p-values (both papers reported six usable p-values) and (b) p-values that, by their exclusion, would contribute to a more conservative test for the presence of evidential value (i.e., the omitted p-values would have strongly supported the presence of evidential value; specifically, for Hao et al., the values were ps = .0009, .0067, .0092, .0039, .0164, .0285 [47]; for Nair et al., the values were ps = .016, .015, .0107, .0053, .0149, .0013 [46]). An IPA that excluded four additional p-values identified as “outliers” by Simmons and colleagues [48] produced nearly identical results and is presented in the (S4 File).

All told, for Simmons and Simonsohn’s [22] p-curve sample, the IPA script’s iterations of the 42 p-values produced 2,592 possible combinations of p-values, and for the larger Cuddy et al. [21] sample, the IPA script’s iteration of the 78 p-values produced 59,719,680 combinations.

If it is of no consequence which p-value is extracted from a specific study, then the results of the Simmons and Simonsohn [22] IPA would reveal little variation across the 2,592 combinations of p-values, and the results would align clearly with the p-curve analysis presented by Simmons and Simonsohn. The same would hold for the Cuddy et al. [21] p-curve analysis, such that it would demonstrate little variation across the 59,719,680 combinations. Alternatively, if it is of consequence which p-value is extracted from a particular study, the IPA would produce widely varying combinations of p-values, some of which support the presence of evidential value and some of which do not.

Simmons and simonsohn sample

Table 2 (top) and Fig 2 presents the results from Simmons and Simonsohn’s [22] analysis and the results from the IPA. To demonstrate the possible range of outcomes of the IPA, we identified the permutation that was associated with the least and most right skew. We identified the extreme permutations by using one of p-curve’s estimate of right skew, z_Full. (Other methods to identify the extreme permutations are discussed in the S4 File) The permutations associated with the highest and lowest scores are presented in Table 2 as “most left skewed” and “most right skewed.” We identified the median permutation by finding the median z_Full value. To further display the variability in the permutations, we also included permutations that were +/- 1SD removed from the median permutation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Simmons and Simonsohn (2017) [22] p-curve reported results, with most right skewed, most left skewed, and median IPA p-curve permutations.

Percentage in each p-value bin are included in the table below the graph.

https://doi.org/10.1371/journal.pone.0305193.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Reported results and the results of the iterated p-curve analysis using Simmons and Simonsohn’s [22] sample and Cuddy et al.’s [21] sample.

https://doi.org/10.1371/journal.pone.0305193.t002

Cuddy et al. sample

The results for the Cuddy et al. [21] sample are presented in Fig 3 and Table 2 (bottom).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cuddy et al.’s (2018) [21] p-curve reported results, with most right skewed, median, and most left skewed IPA p-curve permutations.

Percentage in each p-value bin are included in the table below each graph.

https://doi.org/10.1371/journal.pone.0305193.g003

IPA percentages

The Simmons and Simonsohn [22] sample produced conclusions that changed across permutations: 24.57% of permutations provided support for evidential value, 30.97% generated an inconclusive finding, and 44.44% generated an underpowered result. Alternatively, the Cuddy et al. [21] sample provided a more consistent conclusion: 100% of permutations concluded that right-skew was present and did not support the absence of evidential value.

The HPA debate

Over the past quarter century, various theories have produced competing predictions regarding the relation between trauma and the body’s stress response (i.e., HPA reactivity). On the one hand, it may be that traumatic experiences produce a hyper-reactive stress response. From this perspective, repeated exposure to traumatic events (e.g., death of a loved one) generates a more pronounced stress response (e.g., a more vigilant HPA response) to future threats [50,51]. The body thus becomes “calibrated” to respond to additional stressors with hyper-reactivity. On the other hand, it may be that such traumatic events produce a hypo-reactive stress response [52,53]. This view submits that traumatic experiences do, indeed, initially produce a hyper-reactive response; however, over time, the body compensates for this hyper-reactivity by downregulating its resources, resulting in an under-responsive system. Thus, when exposed to future stress, an HPA hypo-reaction results.

When traditional meta-analyses investigated these competing models, they produced inconsistent conclusions (e.g., [54,55]). To “resolve” the discrepancies between these competing models, Hosseini-Kamkar et al. [23] used p-curve to evaluate the evidence for the relation between HPA and trauma. HPA reactivity was exclusively operationalized by cortisol levels, such that greater levels of cortisol were assumed to be associated with a stronger HPA response. Hosseini-Kamkar et al. operationalized stress in one of two ways: (a) traumatic experiences and (b) adverse life experiences (ALE), in which trauma was considered to be exposure to acute adverse events (e.g., death in the family, loss of a job) and adverse life experiences were stressors that persisted throughout development (e.g., poverty, history of childhood maltreatment).

Hosseini-Kamkar et al. completed four p-curve analyses: two for the literature that examined the relation between cortisol and trauma (i.e., one p-curve of the hyper-reactivity literature and one p-curve of the hypo-reactivity literature) and two for the literature that examined the relation between cortisol and adverse life experiences (i.e., one p-curve of the hyper-reactivity literature and one p-curve of the hypo-reactivity literature) [23]. As summarized in Table 3, Hosseini-Kamkar et al. found different patterns for the relation between HPA and traumatic and adverse life experiences: Whereas the relation between traumatic experiences and hyper-reactivity was inconclusive, the relation between traumatic experiences and hypo-reactivity showed evidential value. The authors concluded that the opposite pattern held for trauma: The literature examining the relation between traumatic events and hyper-reactivity contained evidential value, but the hypo-reactivity literature did not. In short, Hosseini-Kamkar et al. concluded that traumatic events are associated with a lessened cortisol response whereas adverse life experiences are paired with a greater cortisol response.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of Hosseini-Kamkar et al. [2,3] conclusions.

https://doi.org/10.1371/journal.pone.0305193.t003

Data preparation

IPAs were conducted to evaluate the reliability of Hoesinni-Kamkar and colleagues’ [23] p-curves. The first step was to create a database of the relevant p-values. We added 33 p-values to the original database of 22 p-values for the hyper-reactivity and trauma analysis, 20 p-values to the original database of 16 p-values for the hypo-reactivity and ALE analysis, and 20 p-values to the original database of 15 p-values for the hyper-reactivity and ALE analysis. We did not conduct an IPA for the hypo-reactivity and trauma association because the number of p-values (k = 95) would generate a computationally prohibitive number of permutations (135,444,234,240). The disclosure table for each analysis is presented in the (S3 and S5 Files).

As an important note, Hoesinni-Kamkar et al.’s [23] p-curve analyses included several p-values that violated Simonsohn et al. inclusion standards [8]. For instance, according to Simonsohn et al., the original authors’, not the p-curver’s, stated hypothesis determines which p-values should be included in the p-curve analysis. Hoesinni-Kamkar et al., however, included several p-values that violated this rule. For example, Hoesinni-Kamkar et al. included a p-value from Mrug et al. [56] for the correlation between family income and cortisol [23]. However, Mrug et al.’s stated hypothesis was the attenuated interaction of the impact of sleep and gender on cortisol [56]. The three IPAs reported herein reflect Hoesinni-Kamkar et al.’s inclusion standards, in which p-values were included that related to cortisol regardless of the original authors’ reported hypothesis [23]. To further investigate whether these violations of Simonsohn et al.’s coding rules affected the IPA results, a second set of IPAs were conducted that met Simonsohn et al.’s p-curve inclusion standards. These analyses are reported in the (S4 File).

Trauma and hyper-reactivity

Hoesinni-Kamkar et al. reported that their results did not meet either condition for the first and second test of evidential value, z_Half-score = -1.12, p = .13; z_Full = -1.62, p = .05 [23]. Alternatively, of the 13,824,000 IPA permutations, 29.37% met the conditions of the first test that support the presence of evidential value, and 53.41% met the conditions of the second test to indicate support of evidential value.

Regarding flatness, Hoesinni-Kamkar et al. reported that their results did not indicate support of the absence of evidential value, z_Full = -1.16, p = .03 [23]. For the IPA, 45.27% of the permutations met the conditions for the first test and 3.89% met the conditions of the second test to indicate support of the absence of evidential value.

Can P-curve produce informative results?

Simmons and Simonsohn’s conclusion regarding the presence of evidential value for power posing was definitive, that power poses ought to be “treated as a hypothesis currently lacking in empirical support” [22]. However, the results of the IPA revealed that the statistical method used to produce this conclusion was not reliable. The larger question remains whether p-curve (or related analyses) can ever be used to generate an informative conclusion. Simonsohn and colleagues [8] provided the following description for when and how p-curve could be used:

1. We envision that most applications of p-curve will involve assessing the evidential value of the core findings put forward in a set of studies, whatever those findings were. For instance, if a p-curver were interested in assessing the evidential value of the Top 10 most cited empirical articles published in this journal, her statistical results of interest would be those testing the stated hypotheses in those articles, whatever those hypotheses might be. (p. 543)

Based on the findings of this paper, p-curve in these situations would not produce informative findings. P-curve analyses are subject to produce idiosyncratic results because of the uncertainty regarding which specific p-value would be included in the analysis given the possibility of multiple outcome measures and/or a broad hypothesis. In the sample of studies included in the power posing IPAs, for instance, the average number of p-values per study from the Simmons and Simonsohn [22] sample was 1.57 (SD = 1.00), for the Cuddy et al. [21] sample, it was 1.77 (SD = 1.06), and for Hosseini-Kamkar et al.’s [23] sample, collapsing across analyses, it was 2.66 (SD = 1.79), most likely producing the variability in p-curve estimates evidenced in this very analysis.

There does, however, remain a narrow set of situations when using p-curve would not be subject to the issue raised in this article. P-curve might be used when a study meets three conditions: (a) the outcome has been specifically hypothesized (as consistent with Simonsohn et al.’s [8] standards), (b) the hypothesis is identical across each study and paper, and (c) the same outcome measure is used in every study and paper. For example, a p-curve analysis not subject to the issues raised herein may be an investigation into the existence of precognitive abilities (e.g., [57]), in which people have an awareness of a future event from which they do not otherwise have the ability to anticipate. As called on by Bem, researchers have repeatedly replicated his test of “retroactive facilitation of recall” (e.g., [58]), in which participants are able to recall expected, but not-yet-practiced words better than those words not expected to be practiced. Such tests all utilized identical methods, hypotheses, and outcome measures. Given a large sample of studies, such a p-curve would not be subject the limitations raised in this paper, but may continue to be subject to the various statistical issues that have been identified elsewhere (e.g., [14–19]).

The path to determining the presence or absence of a phenomenon optimally lies in experimental investigation and meta-analyses. For the power posing literature, for instance, an experimental replication using a larger sample failed to reproduce Carney et al.’s [22] original finding [59]. However, a meta-analysis of 48 power posing studies concluded that it was the absence of a contractive pose, rather than an expansive power pose per se, which was responsible for the affective and behavioral changes [39]. Indeed, the power pose literature would be served by an influx of registered replications and/or large-scale registered report studies (e.g., [60,61]).

Simonsohn et al.’s robustness tests

The robustness tests reported in Simmons and Simonsohn [22], Cuddy et al. [21], and Hosseini-Kamkar et al. [23], each reported conclusions that matched those of the main analysis. The robustness test, as developed by Simonsohn et al. [8], and which take the form of (a) excluding the highest or lowest p-value from the sample, (b) using an alternative p-value for the included analysis, or (c) replacing one outcome measure for another, does not adequately address the issue raised in this paper.

Specifically, a robustness test that examines a second set of p-values from a possible set of 59,719,680 permutations (for the Cuddy et al. database [21]) does not adequately represent the possible range of results that could be produced from a p-curve analysis. For the Cuddy et al. sample, for instance, whereas IPA considers 100% of the possible permutations, Simonsohn et al.’s robustness test [8] considered 0.000003% of them. A robustness test would not address the issue even if each paper only reported two p-values. For instance, an IPA of Simmons and Simonsohn disclosure table, which included 32 primary p-values and 13 robustness p-values [22], would produce a set of 768 possible p-value permutations, of which only two would be considered using the robustness test (0.26% of the possible permutations).