Issue framing

VAAs are intended to make politics more comprehensible for the general public (e.g., [2], [16]). The way political issues are perceived depends on the problem definitions that are put forward by different frames [17]. When VAA developers formulate issue statements, inevitably they offer users a frame. For instance, consider the positively phrased statement, “The nuclear power station in Borssele should stay open” [18]. This formulation is likely to activate different associations and considerations compared to an alternative negative statement, such as “The nuclear power station in Borssele should be closed” (see [19] for a discussion of polarity effects). Likewise, the header above the VAA statements can also carry a certain problem definition. The statement about the nuclear power station can for example be placed under the heading Environment, as well as under Energy supply. The choice for either one of these frames probably highlights different aspects of the issue.

Different headers and different ways of phrasing attitude statements are examples of framing. Over the past decades, scholars have gathered a vast amount of evidence of such framing choices affecting public opinion (e.g, [4], [20], [21]). Within political communication, these framing effects have been classified into two broad categories (see [21], [22], [23], [24]). Equivalence framing, often found in psychological research and behavioral economics literature, exists of “two logically equivalent (but not transparently equivalent) statements of a problem [that] lead decision makers to different options” ([25]: 36). The famous experiment by Tversky and Kahneman [26] is often cited in this context, which demonstrated that people are inclined to be risk averse if a dilemma is framed in terms of gains and tend to be risk taking if the same dilemma is framed in terms of losses. Positive and negative wording alternatives for policy statements, like the one above about the nuclear power station at Borssele, are another example of equivalence framing.

The second category of framing research is called emphasis framing, and is often studied within the field of media effects. Here, a more relaxed definition of framing is applied, supposedly better reflecting the “complexity of everyday communication environments, where attitude formation is likely driven by an interplay of complementary or competing frames” ([24]: 5). Many studies on forms of emphasis framing use the definition of framing by Gamson and Modigliani ([27]: 143), who define framing to be a “central organizing idea or story line that provides meaning to an unfolding strip of events, weaving a connection among them. The frame suggests what the controversy is about, the essence of the issue.” For example, [28] found that a news story framing the entry of Bulgaria and Romania to the EU as a financial opportunity resulted in a more positive opinion towards the economic consequences of this choice as compared to the same news story emphasizing the monetary risks.

Literature on emphasis framing has often been criticized for using a definition of framing which is too relaxed and inclusive (see e.g., [17], [29], [30]). First of all, emphasis framing operationalizations are context-specific which decreases the generalizability of potential framing effects. Second, and more importantly, emphasis framing studies have been criticized because they confound real emphasis framing, that is, placing emphasis on a specific topic, with different or additional information about that topic [12]. [31] demonstrated in a meta-analysis that when controlling for the amount and content of information the effects of emphasis framing are very limited.

In order to meet these two points of criticism, the current field experiment investigates the effect of changing the header that comes with the VAA question, consistently comparing a condition with no header above the question to a second condition with a right-wing frame (e.g., finance) and a third condition with a left-wing frame (e.g., nature and environment). This way we study a very minimalist form of emphasis framing that is not prone to the critique that more information is offered.

Second, we aim to generalize our findings beyond the level of the individual item by applying the same type of left-wing and right-wing frames across items. This left-right continuum is the dominant way political views are organized in many Western societies (e.g., [32], [33]), and it is (therefore) also used in VAA practice to classify the statements and to organize the voting advice [8]. The left-right distinction is also apparent in a citizen’s organization of political information; e.g., [34] show that people associate political themes such as environment, social security, and unemployment with left-wing parties, whereas themes like taxes, immigration, and crime are more often linked to right-wing parties. In a similar way, other research shows that the left-right divide can also be applied to the content of arguments about issues in political discourse. For example, left-wing parties favor policies that increase government spending and induce growth, while right-wing parties support policies inducing lower spending, balanced budgets, and lower inflation (see, e.g., [35], [36], [37]). Moreover, partisan cleavages can even be used to predict positions on new issues, such as on the EU integration [38]. In sum, these studies indicate that certain standpoints across a wide variety of political themes can be classified clearly on a left-right continuum. By using such different types of headings as issue frames in our VAA statements, we aim to study emphasis framing in such a way that generalizations beyond the level of individual items can be made.

The general finding in previous studies on emphasis framing is that people tend to adapt their opinion in the direction of a frame: If a frame motivates people to think positively about a course of action, they are inclined to agree with it; if a frame draws attention to negative aspects or consequences, people will more likely oppose (e.g., [6], [39], [40]). Therefore, we hypothesize that in a VAA context users follow the direction of the frame:

1. H1. Right-wing headers lead to more right-wing opinions than conditions without a header above the question and left-wing headers lead to more left-wing opinions as compared to conditions without a header.

Political sophistication

An important issue is whether framing affects people’s reported attitudes across-the-board, or whether framing effects are systematically smaller or larger in size for particular groups of respondents. Dual-route theories of information processing such as the Elaboration Likelihood model (see [14], [15]) or Krosnick’s satisificing-optimizing model [13], have proposed that people who are motivated to think about an issue (e.g., people that are interested in an issue or for which the issue is personally relevant), as well as people with higher cognitive capacities (e.g., the higher educated) tend to process information about an issue more deeply. By doing so, they will probably be less susceptible to superficial characteristics of the way the information is conveyed, such as frame choices. And, although evidence for this claim is not always observed [41], several studies have indeed found framing effects to be related by variation in educational level [42], knowledge (see [39], [43], [44]) and the importance of the issue [45].

Comparable to these theories from the domains of social psychology and survey methodology, research in political decision making has indicated that one’s degree of political sophistication affects the way political information is organized and processed [46]. Political sophistication is related to both motivational factors, such as political interest, as well as to measures that are indicative of one’s cognitive capacities, such as educational level. Hence, political sophistication is a factor that combines the type of characteristics that are also deemed important indicators of attitude strength within social psychology and survey methodology (see [13], [47]). Therefore, we will test political sophistication as a moderator of possible effects of framing in the typical political attitude context of VAAs. In line with literature from various fields, we predict that:

1. H2. Political sophistication moderates the framing effect such that the lower one’s political sophistication is, the stronger one is affected by framing variation.

Design and materials

A randomized field experiment was carried out during the campaign for the 2014 Dutch municipality elections in Utrecht, a large Dutch city. For this experiment, we collaborated with the Voting Advice Application Kieskompas, which is the second most used VAA in the Netherlands (after Stemwijzer). The project description for this study was approved by the research director of the research institute UiL OTS of Utrecht University, by the registry and by the mayor of the city of Utrecht, by the director of Kieskompas.nl and a blanket approval of the Ethical Committee of Social Sciences at the Free University.

Kieskompas Utrecht was online for 30 days, during which 41,505 people accessed the website. People who visited the website were randomly assigned to one of five versions of the tool; they could stop filling out the VAA at any time they liked. We will elaborate on the five VAA versions below.

The first VAA version was developed according to the standard procedure at Kieskompas (see for a discussion on this procedure: [8]). The VAA consists of 30 political attitude statements about various political issues and there were no headers presented above the questions (see Fig 1 for an example question).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example of a question from the benchmark version without headings.

The question reads “The parking charges in Utrecht can be raised” with a response scale ranging from “completely agree” to “completely disagree” added with a “no opinion”-option.

https://doi.org/10.1371/journal.pone.0212555.g001

Based on this first benchmark version, four alternative versions (version 2–5) of the VAA were created. These versions were manipulated according to a 2 x 2 experimental design of which one of the factors was the manipulation of issue framing that we report on here (with the levels: left-wing and right-wing), and the other manipulation concerned question polarity (with the levels: positive or negative wordings). As this second manipulation of question polarity did not interact with the manipulation of issue framing, we decided to report the effect of this factor elsewhere [19].

Issue framing was manipulated by providing a header above the VAA statements. For each statement in versions 2 to 5, there was a header above the question and for a little more than half of these statements (N = 17), we were able to vary these headers between experimental conditions. To ensure ecological validity, we only used headers that VAA developer Kieskompas would normally use. We classified these existing headers the VAA builders generally used in their tools into two groups of left-wing and right-wing frames. We define left-wing headers to be headers that are associated with left-wing political parties (e.g., nature and environment, see Fig 2) and right-wing headers to be headers that are associated with right-wing political parties (e.g., Transportation, see Fig 3). As such we classified the headings nature and environment, social policy and culture & education to be left-wing frames, as these and comparable themes were clearly associated with left-wing parties in the study of associative ownership by [34]. Moreover, the themes finances, economy, safety, transportation, and building & living, were classified to be right-wing headers; the first three themes were associated with right-wing parties in the study by [34], the latter ones turned out to be suitable themes to distinguish between parties running in the Utrecht elections based on their party programs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example of a question with a left-wing heading “Natuur en milieu” (Nature and environment).

The question reads “The parking charges in Utrecht can be raised” with a response scale ranging from “completely agree” to “completely disagree” added with a “no opinion”-option.

https://doi.org/10.1371/journal.pone.0212555.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Example of a question with a right-wing heading “Mobiliteit” (Transportation).

The question reads “The parking charges in Utrecht can be raised” with a response scale ranging from “completely agree” to “completely disagree” added with a “no opinion”-option.

https://doi.org/10.1371/journal.pone.0212555.g003

There were various reasons why we could not manipulate all statements in the VAA. One reason was that about half of the items did not scale onto the left-right axis, but on a progressive-conservative axis. Another reason was that sometimes we were unable to formulate relevant left- and right-wing headers for a statement.

In our experiment, the left- and right-wing frames were distributed across the different conditions in such a way that each VAA version approximately included an equal number of both left- and right-wing frames. This was done to assure that if the headers would influence people’s opinions in a consistent direction, the mean effect on the voting advice would be evened out. S1 Appendix provides an overview of the experimental materials.

Procedure

When entering the Kieskompas Utrecht website, visitors were first asked to complete some demographic questions before answering the VAA questions (see S1 Appendix). Next, visitors indicated their opinions to 30 VAA statements on a five-point agree-disagree scale supplemented with a “no opinion” option. The voting advice, which followed after some optional questions about voting intentions, was based on the match between the VAA user’s answers and the stances of the political parties (see [8] for an elaboration).

Measures

The dependent measure in our study was formed by the mean answers to the 17 manipulated attitude questions. We recoded these answers in such a way that a higher score indicates a more right-wing answer. Non-substantive answers were excluded from the analyses (a similar procedure was used in e.g., [19] and [41]).

To measure political sophistication, our moderator variable, we created an additive index combining a question about political interest with a question on educational level (compare: [19, 48]). This measurement of political sophistication was preferred to asking factual knowledge questions (as employed in e.g., [49]), because employing factual questions might have left respondents with the impression that they had to pass some sort of test in order to obtain a voting advice. In order to combine the questions on political interest (measured on a five-point scale) and educational level (measured on a seven-point scale) into one additive measure in which each measure receives an equal weight, we created two new variables with an equal number of scale points. We chose to recode education into 3 scale points, similar to the classification used by CBS (the Dutch Central Bureau of Statistics) and therefore recoded political interest into 3 scale points too. To create an index of political sophistication, we then combined these two variables into a new additive variable (compare: [19]).

Participants

While 41,505 respondents accessed the Kieskompas website, the actual number of valid responses included in the analysis was far less. This is because only 28,741 users (69.25%) filled out the optional demographic questions on the first page of the tool that we used to measure political sophistication. Moreover, we cleaned our data, removing VAA users who are under 18 (and hence not eligible to vote), users who took less than two minutes to fill out the VAA, and users exposing straight-lining behavior providing the same answer to all political attitude questions in the tool. These cleaning criteria were also applied in a previous VAA study [19], and led to the exclusion of 1,337 respondents, leaving 27,404 respondents for the main analysis. We also re-ran the analyses described below in a data file that was not cleaned. Results for the full data file were comparable to what we report in the results section.

The mean age of respondents was 37.08 (SD = 13.96). Most VAA users were well-educated (78.4% had a Bachelor or Master degree), and most were fairly interested in politics (on a five-point scale: M = 3.3; SD = .84). The male-female ratio was about equal (Males: 49.3%; Females: 50.7%). A randomization check showed that VAA users in the various conditions did not differ significantly on age (F(4, 29353) = 0.79; p = .54), gender (χ²(4) = 3.62; p = .46), political sophistication (χ²(16) = 17.09; p = 38), educational level (χ²(24) = 14.59; p = .93) or interest in politics (F(4, 29015) = 0.66; p = .62). This means that there is no reason to assume that any possible framing effects are caused by a priori differences between groups.

Manipulation check

As our manipulation of issue framing was quite subtle (see Figs 1–3), we checked whether the VAA users were likely to have noticed the headings above the statements. If so, we expected more processing time for the conditions with a header as compared to the benchmark version without headers. To test this, we compared the total time spent on the VAA between the benchmark version (without headers) and the experimental versions (including headers). The total time spent on the VAA is, of course, a rough measure of processing time, as it incorporates not only the time spent on the VAA statements, but also the time on the result screen and the additional demographic questions. Hence, if we were to find time differences between the benchmark version and the manipulated VAA versions (with the only difference that they include a header) for this noisy measure, this would be a strong indication that VAA users noticed the headers. Results show that VAA users spent about 16.5 seconds longer on the VAA when there were headers above the question (M_{no heading} = 451.69; M_heading = 468.25; z = 3.60; p < .001). VAA users thus spent approximately half a second longer per statement when headings are included, which is quite substantive when imagining that the answering of questions is a very fast process that takes about 3 seconds per question [50]. Therefore, there is no reason to assume that VAA-users might have missed the headers.

Analyses

To test our hypotheses, we analyzed the VAA users’ answers in a multi-level model. In the model, the mean answer was estimated for questions without a header (i.e., questions related to the benchmark version). Subsequently, deviations from this mean answer were calculated for questions with a left-wing header, a right-wing header, for respondents with higher political sophistication, and the interactions between both left-wing and right-wing frames and political sophistication. To ensure a clean comparison between these different versions, we added question polarity as an additional term to the fixed part of the model. This final model fitted the data much better than a model that only included the issue framing effect (z = 111; p < .001).

The random part of the model accounts for the hierarchical structure of the data. As each respondent in our sample (N = 27,404) answered various manipulated questions (up to N = 17), each observation is nested within respondents and items at the same time. To account for this, the models distinguish between-item variance (e.g., because one item elicits more right-wing answers than another statement), between-respondent variance (e.g., because one respondent has more right-wing opinions than another respondent does), and a residue of error variance. Hence, cross-classified models are in operation (see also [51], [52]). For a formal representation of this model, we refer to S2 Appendix.

If there is an effect observed, we also classify the effect size by providing Cohen’s d. Cohen’s d is a measure for the effect size that relates the difference between conditions to the standard deviation [53]. In multi-level models such as the ones we have applied in the current research there are different variance components that can all be used separately or in combination with one another to calculate the effect size. In the current research we have used the between-item variance for calculating the effect size.

Explorations

First, we checked if our results may be due to a specific selection of left- and right-wing headings. As has been explained in the Method section, some of the left- and right-wing headers were derived from a previous study [34] and others from the party programs of the parties running in the 2014 Utrecht municipality elections. S3 Appendix includes an analysis of the results based on only those headings for which we had a motivation based on [34]. As can be read from the Appendix, results were comparable to what we have reported in Table 1.

Second, we analyzed the data for each statement separately to check whether there are individual statements that caused the overall effect to be in a different direction than expected. S4 Appendix displays the results of these models. As can be seen, 10 out of 17 statements show a pattern similar to the overall results: both left-wing and right-wing frames lead to more left-wing opinions for VAA users with lower levels of political sophistication. This implies that the overall pattern found is not bound to a specific pair of left- and right-wing frames, but really occurs for a wide range of headings (nature & environment vs. transportation; nature & environment vs. finance; culture and education vs. finance; nature and environment vs. building and living). In addition, there are two items (social policy vs. safety; social policy vs. finance) for which a reversed pattern is shown, in the sense that both the left-and right-wing frames lead to more right-wing opinions, but again, just for the those VAA users with lower levels of political sophistication. The remaining five questions do not show main effects of framing nor in interaction with political sophistication. Therefore, it seems unlikely that a specific pair of frames caused these results to occur and explaining the results probably lies in the interaction between the question content and the frame. We will return to this in the discussion section.