Fig 1.
Overview of 121 emphasis framing experimental studies in the field of climate and environmental politics, economics and psychology published between 2007 and 6/2020.
Note: Panel designs (i.e., repeated measurements for the same study participants at two or more points in time) are used to study whether framing effects vary over time (e.g. how long the effect of a one-time exposure lasts). Competing frames are used to emphasize competing arguments in a debate (e.g., pro and contra climate mitigation messages).
Fig 2.
Partisan Sub-Group effects are not robust.
Points indicate estimated treatment effects for sub-group framing effects by partisanship. The y-axis displays the estimated sub-group treatment effects estimated using LASSOplus that allows for all possible covariate interactions. The x-axis displays the estimated sub-group effects using OLS and not allowing for covariate interactions, equivalent to a difference-in-means test. The solid black line displays the 45 degree line, with points falling on this indicating identical estimates for the different methods of estimating sub-group effects. As many studies have multiple treatments and outcomes the number of points displayed is greater than the number of studies re-analyzed (for further details, see Methods).
Fig 3.
Third, we systematically analyzed those 121 articles by coding each of the articles according to the following criteria: a) Significant main treatment effect: To what extent did the experiment report any type of significant main treatment effect? If the study included multiple treatment groups, and at least one of these had a significant effect on the outcome variable, we coded the study as reporting significant main effects. If no main effect was reported, we marked this category as not applicable. b) Significant heterogeneous treatment effect: To what extent did the experiment report any type of significant heterogeneous treatment effect? If the study included multiple treatment groups, and at least one had a significant heterogeneous effect for population subgroups, we coded the study as reporting significant heterogeneous effects. If no heterogeneous effect was reported, we marked this category as not applicable. c) Comparative research design: To what extent did the experiment use a comparative research design? If the study focused on more than one country case, we coded the study as a comparative research design. d) Case: In which countries were the experiment(s) conducted?. e) Panel research design: To what extent did the experiment use a panel research design? We coded the study as panel research design if the study was conducted at multiple points in time (at least two data collection waves). f) Experimental design setting: What type of experimental design did the experiment use? We coded whether the study used a field-, survey- or lab-experimental design or a combination of those experimental design types. g) Competing frames: To what extent did the study use different, competing frames? We coded studies as using competing frames if they used one-sided messages and employed frames that emphasize competing arguments and subsets of information. h) Method used: To what extent did the study use an advanced statistical method to check for the results’ robustness? We coded studies using an advanced computational method if they employed LASSOplus, LASSO, Ridge Regression, or Kernel regularized least squares. i) Sample type: To what extent did the study use a convenience/population non-representative sample not or a non-convenience/population representative sample? We coded studies as convenience/non-representative sample if they study did not use a probability-based, stratified or controlled quota sampling methods to aim at representing the target population. Most of the time convenience samples in our review had a sample size of below 500 and were based on student samples. Population representative/non-convenience samples were mostly larger (n >1000). j) Published data: To what extent did the study make the data publicly available? We only coded studies as publicly available material if the authors had deposited the data in a public repository, such as Harvard Dataverse.
Table 1.
Population, intervention, outcome and study design (PIOS) criteria for the inclusion and exclusion of articles.