Potential costs and benefits of a norm against causal language

Describing results using causally ambiguous statistical language could serve several useful functions. First, researchers often predict statistical associations between variables based on hypothesized causal mechanisms. When predicted associations are found in a nonexperimental study, causally ambiguous statistical language permits researchers to describe their findings accurately while hedging against the possibility that the hypothesized causal mechanism does not exist. Second, causally ambiguous statistical language could prevent readers from incorrectly assuming that a causal relation has been definitively demonstrated [2, 3]. Relatedly, when studies have potential policy implications, avoiding causal language could engender healthy skepticism of policy recommendations that rely on a causal interpretation of the findings.

A possible unintended consequence of the norm, however, can occur when the authors research questions and analytic approaches do not match the interpretation of the policy implications of the results. For example, if an author’s stated interest is in determining whether children’s early academic skills have an “effect” on later academic achievement, but they conduct an analysis that only includes two variables, there would be a mismatch between the stated goal and the analytical approach because it does not rule out plausible alternatives, therefore likely producing biased estimates and interpretations due to potential confounders. This analytic approach would upwardly bias the estimated effect of children’s early academic skills if instead of selecting covariates based on their hypothesized causal relation to the key explanatory variable and dependent variable of interest [1, 4–9] , a researcher includes a more limited set of covariates based on common practice within the field. A separate problem may arise if a researcher overly-adjusts their model by adding in variables that are simply of interest to them. If one of these variables is a potential mediator, then the estimates may be downwardly biased in an over-adjusted model. If the researcher then uses causally ambiguous statistical language to explain their results from these alternative models, for example concluding that “children’s early academic skills have an effect on later academic achievement, which has important policy implications”; this may permit studies that seem causal to be published without being held to the same standards of evidence as studies that make causal claims explicit [10]. The researcher might address a skeptical reviewer’s concerns by explaining in the discussion section that no causal conclusions can be drawn from nonexperimental studies although the value of the study lies in its contribution towards understanding causal mechanisms. In addition, the researcher might explain that they chose not to consider alternative models because the study’s goals were predictive and not causal. In this process, the researcher would have failed to rule out plausible alternative models. Consequently, the study might avoid the stringent standards of causal evidence to which it would have been subjected if it had explicitly made causal claims.

In contrast an author that transparently states their interest in estimating the causal effect of children’s early academic skills on their later academic achievement, it will be clear to an informed reader that an analytical approach with no adjustment for potential confounders, including a variety of explanatory cognitive, emotional, and contextual variables, will be vulnerable to omitted variables bias, likely in the upward direction) . Models that attempt to adjust for such confounds might be subject to further robustness checks or falsification tests that could detect the presence of and perhaps establish the direction and range of plausible magnitudes of additional bias in these estimates [11–13]. These models might even make quantitative predictions that could be tested in a subsequent experiment [14]. This is why it is recommended that authors identify their research goals (i.e., description, prediction, association, or causal inference) and analytic plans (i.e., predictive model, causal model) to reduce the ambiguity in the interpretation of the results [15–17].

Estimates from a predictive study may differ from causal effects in a variety of (sometimes predictable) ways. These differences do not pose a problem if the end goal of a study is statistical prediction per se. Prediction is often a useful goal in and of itself [15] and our critique of the use of causally ambiguous statistical language does not apply to work that seeks to build models for purely predictive purposes. Still, based on researchers’ descriptions of their findings [1, 16, 17] and the mismatch between the methodological approaches taken in most psychological research and approaches designed to maximize prediction for applied purposes, we note that many, if not most, nonexperimental studies in psychology are intended to inform theory about causal mechanisms underlying relations between variables.

Supporting this possibility, many nonexperimental studies use prescriptive language when discussing policy implications of their findings, despite avoiding causal language when describing their results. Critically, such prescriptive language implicitly relies on causal interpretation of the findings [18, 19]. For example, a study that demonstrates a “predictive” relation between early skills and later achievement might also recommend intervening on early skills to improve later achievement; yet such an intervention could only achieve the stated goal if the relation between early skills and later achievement were causal, and not fully explained by confounding factors influencing both early skills and later achievement. Thus, a researcher can satisfy the norm against explicit use of causal language while still relying on an implied causal inference, contrary to the purpose of the norm.

Grosz and colleagues label this approach a motte-and-bailey strategy, “in which researchers profit from the more interesting but difficult to defend causal interpretation of their effect (the bailey), but once challenged, they retreat to the almost trivial yet difficult to attack descriptive finding (the motte)” [1]. In this case, the stated purpose of the study (perhaps even stated in the title) is prediction, which is a different analytical task that may be easier to defend, but the theoretical and applied issues around which the study is framed upon depend on the assumption that causal information is conveyed by the findings—a harder to defend position. Following the previous example, a researcher may write a paper in which they regress later achievement on earlier skills, focusing on the importance of prior knowledge for later learning and on the importance of early intervention in the abstract, introduction, and discussion sections (the bailey), but caution in the discussion section that, because the study was not a randomized experiment, the reader should not draw causal conclusions (the motte). In doing this, the author might use the motte-and-bailey strategy to communicate a preferred causal interpretation without ruling out plausible alternatives.

Additional ways psychologists may implicitly communicate a preferred causal interpretation while hedging to avoid criticism for making causal claims based on correlational data include 1) designing studies for which results are far more interesting or important if they are causally informative and 2) focusing more on their preferred causal interpretation than on ruling out plausible alternatives. For example, consider a hypothetical study that regresses children’s achievement test scores on previous participation in a program designed to improve parent-child interactions, along with a rich set of baseline measures of child and parent skills and contexts, estimating a relation between program participation and adjusted achievement test scores of 0.10 SD. This finding would not be very useful for predictive purposes, nor would it be a useful way to model the process through which some families came to participate in the program and others did not (because test scores are measured after program participation, and thus may be caused by participation). However, if the findings are interpreted as informing readers about the effects of program participation on achievement test scores, a causal interpretation, they are interesting and policy relevant Moreover, the method used may focus more on the author’s preferred causal interpretation than on ruling out plausible alternatives, for example by failing to probe the robustness of results to the inclusion of child prior test scores in the model. For these reasons, avoiding the use of causal language is unlikely to prevent readers from treating results as if they are intended to be interpreted as causally informative (because of point 1 above), yet the most plausible set of causal estimates will frequently go unreported (because of point 2).

Importantly, neither Grosz and colleagues nor we assume that researchers deliberately mislead readers by using causally ambiguous statistical language. The use of causally ambiguous statistical language is likely a reaction to both the norm against straightforward causal interpretations of nonexperimental research and researchers’ beliefs that their work is causally relevant in some way. Still, there is a contradiction between denying causal informativeness of some statistical predictor and arguing for intervention on the same variable based on its predictive power. Such recommendations are often given in ways that imply stronger evidence for the relevant causal pathway than warranted by the method [16] consistent with the hypothesis that the use of causally ambiguous statistical language may be an example of “strategic ambiguity” [19, 20]. For these reasons, avoiding the use of causal language completely will not plausibly prevent readers from treating results as if they are intended to be interpreted as causally informative. Regardless of its origins, the use of causally ambiguous statistical language may allow authors to avoid punishment for violating the norm against straightforward causal language while still arguing for the policy-relevance of their claims.

In such cases, it is unrealistic to ask authors to avoid mentioning policies or causal concepts; not only would the paper make little sense without them, but descriptions of the ways in which causal estimates are identified are essential for judging the paper’s contribution. Yet current norms require authors to hedge against causal interpretations even in such cases. For these reasons, some have argued that psychologists should consider using straightforward causal language in descriptions of nonexperimental research findings, a proposal to which we will return in the discussion.

Current study

The goal of the current study is to test whether describing nonexperimental psychological research findings using causally ambiguous statistical language influences perceptions of the quality and policy-relevance of the research. We hypothesize that causally ambiguous statistical language raises judgments of study quality. Further, because causally ambiguous statistical language allows authors to emphasize policy-relevance without violating the norm against straightforward causal language, we hypothesize that causally ambiguous statistical language increases receptivity towards policy-relevant conclusions that are based on causal interpretations of the results. Such findings would imply that the status quo limits incentives for psychologists analyzing nonexperimental data to aspire to causal estimation. In addition, contrasting estimates from increasingly well-controlled models to account for confounding variables or conduct sensitivity and robustness tests often yields smaller associations between the key dependent and independent variables. When this is the case, a straightforward causal framing of the reduced estimates speaks to the robustness of the relationship between the key dependent and independent variables. However, it may be detrimental for the perceived quality and policy-relevance of one’s work if policy makers perceive the study to be less conclusive.

The study focused on potential reviewers, editors, and citers of psychological research—namely, graduate students, postdoctoral researchers, and faculty in psychology. If causally ambiguous statistical language raises perceived study quality to these readers without reducing perceived policy-relevance, this practice may increase the frequency of nonexperimental research studies being interpreted causally without increasing the quality of methods for causal inference within psychology.

Participants

Based on the power analysis, we planned to recruit 100 Ph.D. students, postdoctoral scholars, and faculty in psychology, education, and human development and family studies departments by posting a link on social media sites frequented by psychologists and via Twitter. Colleagues posted a link to the website, so that the involvement of the last author (who has previously posted about possible costs of the norm against using causal language in nonexperimental psychology on Twitter) would be less obvious to participants. In all, 142 individuals were recruited via Twitter, the Psychological Methods Discussion Facebook page, and the Cognitive Development Society listserv. Laypeople were not included in the study for two reasons. First, the public likely consumes psychological research findings primarily via the media, which often does not respect the norm against use of causal language to describe results of nonexperimental studies [4]. Second, among lay adults, the tendency to infer causation from psychological research findings does not appear to be strongly influenced by the strength of the research design [21].

Descriptive statistics for the full sample of participants are shown in Table 1. The full sample included participants who reported being 22 to 67 years old (M = 33.20, SD = 8.96). On average, the participants had completed 4.13 statistics courses (SD = 2.74). Most participants were recruited from the Cognitive Development Society listserv (47%), followed by the Psychological Methods Discussion Facebook group (28%), and Twitter (25%). Most participants were female (54%) and worked in the United States of America (59%). Our participants consisted of academic faculty (32%), post-doctoral researchers (20%), PhD students (33%), Masters students (1%), and undergraduates (1%). Most were from psychology (80%) encompassing a variety of interdisciplinary subfields, as shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics for the demographic variables of the full survey sample.

https://doi.org/10.1371/journal.pone.0286403.t001

We preregistered that we would attempt to recruit 100 participants and exclude participants who were younger than 18 years old (n = 1), responded to the first two questions about one of the abstracts in under 30 seconds (n = 12), responded incorrectly to the question of how many abstracts were causally worded (our manipulation check, n = 79), were not in a field related to psychology (n = 2), and were not (1) enrolled in a psychology, education, or human development Ph.D. program, (2) a postdoctoral researcher in psychology, or (3) a psychology faculty member. Of the original 142 participants only 55 met these criteria.

Many participants failed the manipulation check (55%) which asked: “How many of the two abstracts you just read employed explicitly causal language in their description of the results?” with three responses (0) Neither (n = 11), (1) Only one (n = 63), (2) Both (n = 53), and some participants did not respond (n = 15). The manipulation check question appeared after participants were asked to rate the quality of the design and analysis of the study and the policy conclusions made in each abstract. The purpose of this question was to check how many participants had comprehended that one study had employed explicit causal language as we intended them to do. Although most participants did interpret the use of explicit causal language as intended, 53 participants believed both abstracts employed explicit causal language meaning that they either did not attend sufficiently to the abstracts or that they viewed words like “predict” as implying causation to some degree. We suspect the latter explanation may be correct (indeed, a recent study reported that about half of respondents view the word “predict” as a weak to moderate root word for implying causation [17]), and thus this manipulation check was not a strong measure of participant attention. In either case, the manipulation would be less likely to work for such participants, so excluding participants who failed the manipulation check would thus plausibly be predicted to result in larger estimates. However, across different inclusion criteria, estimates of the effect of causally ambiguous language are consistent in direction and magnitude. The Descriptive statistics for the 55 participants who met these criteria are shown in S1 and S2 Tables in S1 File. We stopped sending out our survey link once responses to our posts surpassed 100 participants. We ended data collection with more responses (n = 142), because of a larger than anticipated set of responses after posting to a listserv, ending data collection after we realized we had surpassed 100 participants.

Procedure

Each participant received two of the four abstracts—the causally-worded abstract for one study (e.g., scientific reasoning study) and the causally ambiguous abstract for the other study (e.g., reading achievement study). We randomized which study was presented in its causal version and whether the causal or causally ambiguous abstract was presented first, resulting in four conditions (see Table 1 and S1 Table in S1 File). Participants read the first abstract and answered Questions 1 and 2 for that abstract, then did the same for the second abstract. Next, participants answered the manipulation check, followed by the demographic questionnaire, which was labeled “optional”. We did not predict any effects involving the demographic questionnaire variables.

After participants read each abstract, they were asked the two questions. Question 1 asked “Based on the limited information available in this abstract, how would you rate the quality of the design and analysis of the study described?”. Participants answered using the following scale: 5 = Very high quality, 4 = High quality, 3 = Moderate quality, 2 = Low quality, 1 = Very low quality. Question 2 asked “Based on the limited information available in this abstract, how strongly does the study support the conclusion in the final sentence, quoted below:”. Participants responded to question 2 using the following scale: 5 = Very strongly, 4 = Strongly, 3 = Moderately, 2 = Weakly, 1 = Very Weakly/Not at all. The abstract and questions were shown on the same page, allowing participants to refer to the abstract as they evaluated it.

After participants read both abstracts they were asked the following question as a manipulation check: “How many of the two abstracts you just read employed explicitly causal language in their description of the results?” which participants answered on a scale of 0 = Neither, 1 = Only one, and 2 = Both. Participants were not allowed to go back to change their answer after the manipulation check. Then, participants were given a brief demographic questionnaire about academic status, field of study, and psychology subfield of study followed by six optional questions about personal background characteristics such as age, sex, and English language learner status. At the end of the survey we asked participants to “Please leave any feedback (for example, about how to improve the survey). We take your feedback very seriously”. All questions are provided as they were asked verbatim in the preregistration document, Appendix B. (available via https://osf.io/2v8mg/).

Analysis

As pre-registered, we estimated random intercept models, with responses nested within participants, using maximum likelihood estimation. We use the random intercept model to allow for the variance to be clustered at the individual level since we have repeated observations improving the precision of our estimates. In addition, this analysis allows us to model a different intercept for each individual when assessing the differential impact of causal language v. causally ambiguous language on their ratings of study quality and policy conclusion. Data and analysis code are available at https://osf.io/2v8mg/. To estimate the effect of causally ambiguous statistical language wording on the key outcome measures, we regressed responses to the questions about study quality and policy relevance on an indicator of whether the abstract was causally worded, along with indicators for the order in which the abstract was seen, and which hypothetical study was described in the abstract. Our preregistered hypotheses pertained to the effect of causal (vs. causally ambiguous statistical) language. However, for clarity, we present the impacts of the reverse-coded treatment, causally ambiguous statistical (vs. causal) language.

For each rating question—perceived quality and perceived support for policy conclusions—we estimated the effect of causally ambiguous statistical language in a series of four models. Model 1 included the full study sample and both abstracts and provides the most precise estimate of the treatment effect. Model 2 included the full study sample (n = 142) but only included data from the first abstract viewed by each participant. This model was included to test whether results were sensitive to order effects. Because in the real world, readers may not usually judge the merits of two abstracts that differ in their use of causal language one after the other, the estimate based on participants’ ratings of the first abstract only is plausibly the most externally valid estimate. However, it is less precise because it uses only half the available data. Model 3 included the first 100 participants regardless of whether they met the inclusion criteria (n = 100) as an robustness check because our preregistration plan was to stop collecting data once we gathered 100 participants. Finally, Model 4 included only the preregistered sample that met all the inclusion criteria, we refer to this model throughout the results section as representing the best test of our hypotheses (n = 55). In the supplementary materials we provide several additional robustness checks.

Effects of causally ambiguous statistical language on perceived quality

Following our preregistration plan we first conducted confirmatory analyses to estimate the impact of causal framing on participants’ ratings of study quality. Table 4 indicates that results for the key parameter estimate, “causally ambiguous statistical language”, were quite similar and highly significant across the four models. Participants reliably rated abstracts worded in causally ambiguous statistical language as of higher quality than those worded in causal language, with estimates of the difference ranging from .34 to .53 points on the 5-point Likert scale. Expressed in standardized effect sizes, these estimates range from .48 to .80 (in the preregistered Model 4). For the full sample, results were similar when both responses were included (Model 1) as to when only the first abstract response was included (Model 2), suggesting that the within-participants design did not elicit a contrast effect that exaggerated these estimates.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Participant ratings of quality of design and study.

https://doi.org/10.1371/journal.pone.0286403.t004

Results using all available observations (Model 1) are displayed in Fig 1. As can be seen in the figure, most participants gave a different response in the two conditions, with the most common response patterns being a 4 or a 3 when studies were described in causally ambiguous statistical language, with a rating of 2 or 3 when studies were described in causal language. The higher amount of area shaded in red indicates that the most common pattern was to rate causally worded abstracts as of lower quality.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Changes in ratings by condition for questions 1.

Note. N = 142. This panel compares ratings from question 1: “Based on the limited information available in this abstract, how would you rate the quality of the design and analysis of the study described? “. Lines represent differences in participant rating of quality between abstracts that have statistical language and causal language. Gray lines indicate the number of participants that rated abstracts with different languages the same. Blue lines indicate the number of participants that rated abstracts with causal language higher than abstracts with statistical language. Red lines indicate the number of participants that rated abstracts with statistical language higher than abstracts with causal language.

https://doi.org/10.1371/journal.pone.0286403.g001

Effects of causally ambiguous statistical language on perceived support for policy conclusions

We also conducted confirmatory analyses to estimate the impact of causal framing on participants’ ratings of strength of support for policy conclusions. Table 5 indicates that results for the key parameter estimate, again “causally ambiguous statistical language”, were similar across the four models. Participants rated abstracts that used causally ambiguous statistical language as supporting policy conclusions as strongly, or more strongly, than causally worded abstracts in Model 1. Estimates in Models 2–4 were similar to Model 1 in magnitude but statistically nonsignificant and ranging from .16 to .25 points on the 1–5 Likert scale. The preregistered Model 4 generated the largest point estimate, but it was not statistically significant, because over half of participants were excluded. Expressed in standardized effect sizes, these estimates range from .16 to .27. The estimate was significantly greater than 0 in the full sample using all available observations (Model 1). Fig 2 shows that more participants gave lower ratings to causally worded abstracts than to statistically worded abstracts, but many participants shifted in both directions. For the full study sample, the estimate was similar whether both responses were included (Model 1) compared to when only the first abstract response was included (Model 2), suggesting that testing effects did not affect the magnitude of the estimate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Changes in ratings by condition for question 2.

Note. N = 142. This panel compares ratings from question 2:“Based on the limited information available in this abstract, how strongly does the study support the conclusion in the final sentence, quoted below: “We conclude that raising the quantity and quality of (resources for third grade reading OR early science instruction) is an important way to raise children’s educational prospects in the 21st century.”. Lines represent differences in participant rating of quality between abstracts that have statistical language and causal language. Gray lines indicate the number of participants that rated abstracts with different languages the same. Blue lines indicate the number of participants that rated abstracts with causal language higher than abstracts with statistical language. Red lines indicate the number of participants that rated abstracts with statistical language higher than abstracts with causal language.

https://doi.org/10.1371/journal.pone.0286403.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Participant ratings of information support for policy conclusion.

https://doi.org/10.1371/journal.pone.0286403.t005

Exploratory subgroup analyses

Because we used a convenience sample, external validity was a major concern. We thus estimated the effects of causally ambiguous statistical language for two kinds of subgroups: 1) the source from which a participant accessed the survey, and 2) participants’ occupational status (S3-S7 Tables in S1 File). Estimated effects of causally ambiguous statistical language on ratings of study quality went in the same direction across source subgroups, although they were larger in the PsychMD and cogdevsoc subgroups than in the Twitter subgroup (S3 Table in S1 File). The effect of causally ambiguous statistical language on ratings of support for policy conclusions was only statistically significant in the cogdevsoc subgroup and was close to 0 in the other groups (S4 Table in S1 File). The cogdevsoc subgroup represents researchers in various psychology subfields including developmental, educational, cognitive, and social/personality psychology subfields (S5 Table in S1 File). Estimates for both questions were similar for psychology faculty, postdocs, and Ph.D. students (S6 and S7 Tables in S1 File).

Exploratory participant self-reports

To probe participants’ thinking, we read through participants’ feedback provided at the end of the survey. Although we did not ask participants to explain the reasoning behind their ratings, some participants self-reported their reasoning in an open-ended request for feedback at the end of the survey. Of the 142 participants, 22 provided feedback on the survey. Of these, 10 were substantive critiques of the contents of the abstracts, and the remaining comments provided helpful critiques of the structure of the survey or compliments.

We conducted an exploratory analysis of participant feedback to probe their reasoning about the quality of abstract design and support for policy implications. For example, of the 10 critiques of abstract content, three participants thought that the conclusion for both abstracts claimed causality and two of them stated these claims were wrong; two other participants discussed confounding variables, and one participant believed that the quality of both studies was inadequate to answer a causal question but adequate to answer a predictive question. Consistent with the idea that causally ambiguous statistical language can communicate causal information, three critiques expressed concerns about the causal conclusions of both abstracts. One participant was concerned about the norm against causal language being violated and therefore rated the abstract about reading achievement written with causal language as lower quality. However, the same participant found that the other abstract about scientific reasoning written in causally ambiguous language made policy conclusions that were higher quality due to a larger sample size; thus, the norm against causal language did not appear to preclude them from endorsing the causal inferences made by the policy conclusions. Lastly, other factors such grammar and run-on sentences, informed one participant’s ratings of study design quality. This exploratory analysis offers converging evidence that several participants based their decisions in part on perceptions of the extent to which the abstract explicitly described findings as causally informative.

Why are authors making policy recommendations that are not supported by the ambiguous causal language of the results?

Recommendations for policy and practice that are based on nonexperimental research are common in educational psychology [16]. Concerned researchers have advocated for policies that restrict discussions of practical and policy implications that extend beyond the evidence from the data [17–20]. For example, the authors warn against making recommendations that generalize study results to populations that were not included in the sample [21–23]. We agree with the authors that policy recommendations should be appropriately limited in scope.

However, as noted in the introduction section, nonexperimental psychology contains frequent examples, most obviously studies attempting to estimate the effects of programs or policies in the absence of random assignment, for which reporting associations without discussing policy or causation makes little sense. This is not an argument for naïvely assuming that all our estimates have a clearly understood causal interpretation: Rather, we maintain that for researchers to test theory and provide policy-relevant empirical evidence, authors of non-experimental studies should explicitly state their intention to test hypotheses about causal mechanisms, attempt to rule out plausible alternative hypotheses, and calibrate the strength of the causal evidence to the strength of the policy conclusion. Merely replacing causal language with causally ambiguous statistical language is no substitute for these difficult but essential steps. Researchers should consider using causal language, when the assumptions of an analysis are explicit, and sources of bias recognized. This is much more useful scientific endeavors than sweeping those details under the veil of statistically ambiguous language and proceeding boldly with the inference/policy recommendations. We thank an anonymous reviewer for making this point.. We encourage future research on how causal language can best be employed to communicate researchers’ communication about psychological research in ways that address the underlying theory under consideration but do not convey overconfidence about causal implications.

Why does causally ambiguous statistical language increase ratings of study quality and support for policy conclusions?

The reasons why participants rated abstracts with similar research designs and conclusions as having lower quality and being less supportive of policy implications when they were written with causal language as opposed to causally ambiguous statistical language are not clear. Possible explanations include that readers view the use of causal language to describe results of nonexperimental studies as unfamiliar or as a norm violation. Several participants responded to an open-ended prompt for feedback on the study with reference to the norm, suggesting this was an important factor contributing to the observed effects. Another possible reason is that causal language makes plausible alternative explanations more salient to the reader. Indeed, some participants noted that failing to control for additional confounds made the hypothetical studies inadequate to answer a causal question. Finally, the negative effect on perceived support for policy recommendations may reflect a halo effect, wherein any of the above factors make readers view a nonexperimental study described with causal language as less trustworthy and less useful in a variety of ways.

Regardless of the mechanisms, it is concerning that participants rated an abstract with the same research design and conclusion as being as or more supportive of policy recommendations when it was worded in accordance with the norm against causal language. Whether the support for policy recommendations is viewed differently due to the salience of methodological limitations or a halo effect, the conclusion drawn is the same, readers favored a non-experimental design more when the language used was causally ambiguous. This is an important consideration for discussions around optimal norms pertaining to the use of causal language in nonexperimental psychology.

Implications: Should we encourage causal language?

Perhaps a useful practice for improving causally informative statistical analysis in psychology would be to encourage psychologists testing causal theories to use causal language throughout their manuscripts. Pearl and Mackenzie asserted that statisticians’ reluctance to incorporate causal language into their work limited the types of questions they could answer and resulted in longstanding apparent paradoxes [11]. Hernán describes the problem of specifying a model without a clear causal question in mind using the example of estimating the association between drinking wine and heart disease: If the researcher’s ultimate goal were to test the statistical association between these variables, then measuring and statistically adjusting for likely confounders would not be necessary [9, 10]. However, as Rutter [24] argued in response to the argument that nonexperimental studies provide important information about associations that have no causal meaning, “it is difficult to see why anyone would be interested in statistical associations or correlations if the findings were not in some way relevant to an understanding of causative mechanisms” (p. 377). Therefore, although retreating to the motte (the claim that the researcher is only interested in associations) may protect them from criticism in the short-term, the ultimate goal of understanding how wine consumption influences heart disease would be better served by a straightforward discussion of the assumptions required to address the causal question of interest: What is the impact of wine on heart disease? For this reason, several have argued that psychologists’ reluctance to explicitly talk about causality limits the questions they can address directly, and results in seemingly contradictory findings, stifling cumulative progress in psychology [1, 13, 14, 24, 25].

The possible benefits of encouraging psychologists to specify their causal theories explicitly include clarity to readers and clarity to writers. Causal language can make the link between research methods and theory more transparent to support the responsible use of causal methods to test the causal theories. Most findings in nonexperimental psychology are consistent with multiple possible theories[8, 26, 27], often with different policy implications. Encouraging researchers testing causal theories to use causal language throughout the manuscript can help clarify the assumptions, which, along with these findings, led the writer to a particular set of theoretical and policy conclusions.

This approach is used in other fields where results from studies with strong quasi-experimental designs are often described with straightforward causal language as well [28]. Further, use of causal language could make both writers and readers more alert to potential biases in estimates of causal effects (e.g., over-estimation of effect size due to the presence of unobserved confounding variables). Careful thinking about these details is not a fool’s errand: Researchers have identified several instances in which, through careful design and analysis, nonexperimental analyses have yielded empirical estimates very similar to impacts estimated from randomized experiments [21, 29].

Although we find these arguments compelling, it does not necessarily follow that causal language will improve standards or practices around the presentation of causal evidence in psychology. For the benefits of causal language to outweigh the costs, using such language must cause researchers to think differently. Without stronger norms around causal inference (e.g., different training, longer manuscripts with more robustness checks and falsification tests), dissolving the norm against straightforward causal language in psychology could backfire, resulting in stronger claims without stronger evidence. Journals might consider incentives (e.g., badges of recognition) for researchers using rigorous methods for causal inference. Also, to counter “hedge-drift”—wherein key study limitations are relegated to the middle of long discussion sections, after the strongest claims have already been made—journals might require important limitations and caveats to any major claims in the paper to appear in titles and abstracts [22, 30, 31].

Limitations

The generalizability of these results is limited in three ways. First, our convenience sample is not representative of psychological researchers in general or in any subfield. While subgroup analyses suggested that participants from different recruiting sources, in different research stages and different psychological subfields, provided similar ratings of study quality, it is possible that certain subfields have different norms. The exploratory subgroup analyses suggested that support for the conclusion in the final sentence of the abstract was only significantly affected within participants recruited from one of the three sources (see S4 Table in S1 File).

Second, we only tested two different types of abstracts with the term “predict” reflecting the causally ambiguous condition. A potential issue with using the term “predict” is that individuals may have interpreted it as reflecting a study exclusively concerned with predictive modelling. This likely explains the high error rate on our manipulation check. Future work might benefit from using words such as “associate”, and “link” which Haber and colleagues [17] identified as being more common and causally ambiguous than the term “predict”–although at least half of reviewers rated all three of these words as suggesting a causal link. Whether this would improve nonexperimental research on net would depend on the extent to which these words increase perceived study quality yet continue to convey some degree of causal information to the reader. Finally, perhaps most important for guiding best practices related to causal language is the extent to which wording affects the author’s thinking about causal assumptions and interpretations, which we hope future work will measure.

Third, perhaps with stronger study designs, a different pattern of finding would emerge. On average, participants found the abstracts to somewhat weakly support the policy conclusions across the two conditions (mean rating = 2.16 on a 5-point scale). Creating an important limitation of this experiment in that it is difficult to make a judgment about whether participants were overly harsh in their ratings of abstracts described with straightforward causal language, overly generous in their ratings of abstracts described with causally ambiguous statistical language, or both. This is a difficult question that should bear strongly on the implications of these findings.

One potential objection to our conclusions is that it was not the causal language, but rather the inability of the methods described in these hypothetical abstracts to support causal claims, which raters penalized in their ratings of study quality and support for policy conclusions in the causal language condition. However, we do not view these as alternative explanations: The use of causal language was penalized because it makes clear that the method does not provide strong support for the conclusions. To be clear, we do not hypothesize that our findings would generalize to studies with stronger designs for causal inference.