Introduction

The central aim of behavioral sciences such as social psychology is to explain and predict behavior [1]. Thus, behavioral sciences have to produce knowledge (i.e., develop and test theories and models) that can be applied to explain and predict behavior. The next step in this quest is to develop consistent and parsimonious theories with testable models that consist of measurable concepts (e.g., attitudes, intentions, control beliefs), and finally, to explain or predict behavior. In order to increase explanatory power, researchers develop theories with the help of additional constructs that explain how attitudes influence behavior (for a broader overview on attitude research see [2]). However, the increases in the amount of variance that can be explained with each additional concept are often modest, and theory building suffers from high interdependencies between explanatory concepts, or in other words, low discriminant validity (e.g., [3]). These interdependencies could, of course, result from true correlations between the concepts. But what if they are nothing more than methodological artefacts that obstruct a clear view of the true interdependencies of the underlying theoretical constructs?

We argue that the method bias that often results from behavioral science’s strong reliance on introspective measures is a common critical methodological issue that deserves attention. Many measures in social psychology are based on participants’ verbal statements that require at least some degree of introspection. For example, participants have to reflect on what social norms are, reflect on their own intentions, recall their own values, or evaluate objects verbally. In contrast to other rather direct measures (e.g., attainment tests, observations, and reports of past behavior), introspective measures are more indirect because they involve evaluative and cognitive processes that are open to many more individual influences (e.g., self-presentation) and, thus, are much more prone to influences that are unrelated to the measured construct. If these influences are similar across different measures due to the use of the same method of measurement (e.g., introspective measures), common method bias can occur. We can only speculate which of the potential causes of method bias listed by Podsakoff and MacKenzie [4] have the strongest influence and distort the answers of introspective measures. But most likely common rater effects that help to present a desirable and consistent self (e.g., the consistency motif, social desirability, and implicit theories) probably have a strong influence on introspective measures as they demand introspection by design. In this respect, the evaluation of an object as requested by common attitude measures is a prototypical case of such introspective measures.

Thus, and because attitudes play a central role in explaining behavior [5, 6], we use the concept of attitude to exemplify common method bias. Furthermore, as presented in textbooks (e.g., [7]), the attitude concept provides a perfect platform for validation studies because attitudes can be expressed in different ways: A person can articulate a verbal statement that demands reflection (e.g., “The natural environment is important to me!”), express a certain emotion (e.g., laugh or cry), or act in a particular way (e.g., donate money to an environmental organization). Operationalizing all three forms of attitude expressions and using them in a methodologically more balanced way are the prerequisites of a more desirable heterogeneous measurement approach (i.e., multitrait-multimethod matrix; [8]). However, attitude research relies almost exclusively on introspective, or more specifically, evaluative verbal statements.

More specifically, our study is based on measures of environmental attitude of which a considerable number of different measures exist (e.g., [9]). Most such introspective measures that are used to predict ecological behavior can be divided into either specific measures of attitudes toward such ecological behaviors or more general attitudes toward the environment. One introspective measure of a general attitude toward the environment, that is, the New Ecological Paradigm (NEP), has been used extensively for more than two decades (e.g., [10]).

We were able to find only one corroborated environmental attitude measure that was not based on introspective verbal statements, that is, the General Ecological Behavior scale (GEB; e.g., [11]). We introduce the GEB as an example of a methodologically distinct behavior-based attitude measurement approach that, like the NEP, was designed to measure environmental attitude. By administering both measures to a representative sample, we demonstrate the differences the GEB brings to a setting dominated by introspective attitude measures. However, by no means are we claiming that either of the two attitude measures is inferior to the other measure in general. It is merely the ubiquitous use of introspective measures in some subdisciplines of psychology that we claim is problematic. Hence, our goal is to provide a discussion of how a more diverse mixture of methods will provide an opportunity to foster theory development, especially in the behavioral sciences. We achieve this goal by showing—with the help of our example from environmental psychology—how method bias can be reduced and how validity can be increased.

Even though our study focuses on strong empirical interdependencies between theoretically distinct concepts (i.e., low discriminant validity), the validation of measures with meaningful related criteria (i.e., convergent validation) is even more important for any measure. Thus, and in order to draw a more complete picture, we also test the convergent validity of our example measures. With respect to behavioral science, the concepts and their measures have to converge with meaningful criteria, which is foremost overt behavior. Thus, convergent validity plays a crucial role on a methodological level and seems to be a recurring problem. Especially in attitude research, convergent validity (i.e., correlations between attitudes and corresponding behaviors) is disappointingly low or even nonexistent (e.g., [12, 13]), which also might be a driver for researchers to develop models and theories with more concepts. As most of the measures of these concepts are introspective measures, this strategy potentially also fosters the problems of increasing numbers of barely distinguishable concepts and measures with low discriminant validity.

Overall, this study makes the argument that many of the interrelations of several introspective measures but also the discordance between introspective and behavioral measures are due to introspection, which leads to common method bias. Please note that we begin each of the following parts of the manuscript by exploring convergent validity, which is not the main focus of our study but is the more common and more important part with respect to validation. After exploring convergent validity, we get to the main concern of this study, that is, discriminant validity. We decided to use this order because it actually reflects the common order of validation studies even though our focus is on the second part.

Materials and methods

Ethical permission is provided by the Scientific Advisory Board of DIW Berlin.

Procedure

In order to compare the validity of the NEP and GEB in our study, we related each scale to the validation items. The validities of the two environmental attitude scales were assumed to increase as their correlations with the five ecological behaviors and intentions increased. Thus, for a high validity, we expected positive relations of the NEP and GEB with the pro-environmental life aspects and no significant correlations with the 11 unrelated life aspects. We calculated a simple validity score for each of the instruments by adding 1 point for all correlations that were in the expected direction (marked with a “+” in Table 1) and subtracting 1 point for all correlations that were not in the expected direction for the convergent items (marked with a “-” in Table 1). For the discriminant statements, we added 1 point for a nonsignificant correlation and subtracted 1 point for a significant correlation. The last column includes a comparison of correlations from dependent samples according to Eid and Gollwitzer [32].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Correlations of the NEP and GEB with the validation items.

https://doi.org/10.1371/journal.pone.0192907.t001

Results

The mean for the NEP score was M = 3.81 (scale range 1–5; SD = 0.51), which replicated the common finding of a strong endorsement of the NEP [10]. The mean for the GEB was M = 0.07, SD = 0.80 (Logits), and the NEP was positively correlated with the GEB (r = 0.19, p < 0.001). The NEP was positively correlated with only two and even negatively correlated with one of the five ecological behaviors and intentions (see Table 1). The GEB was correlated with four of the five behaviors and intentions in the expected way. Four of the five convergent cognitive statements about life aspects were positively related to the NEP, whereas all five of them were positively related to the GEB. Please note that correlations below .3 are usually interpreted as small effects, whereas correlations of .3 or higher are considered good [33]. Table 1 shows that only two of the convergent validation items had correlations with the GEB above .3. However, the magnitudes of the correlations between the predictors of ecological behavior such as the NEP and GEB with actual behavior were usually in the lower ranges. One reason is the broad and heterogeneous class of behaviors that are considered ecological behaviors. Thus, within the domain of ecological behavior, the correlations of the GEB as a general measure of environmental attitude with the convergent validation behaviors were relatively substantial [13].

Furthermore, the NEP was correlated with five of the 11 discriminant cognitive statements about life aspects (e.g., the statements about the importance of health and religion), whereas the GEB was correlated with only one of the 11 discriminant cognitive statements (see Table 1). On the basis of our scoring system, the NEP scored six out of 21 points, whereas the GEB scored 18 out of 21. As an additional measure, we included a test to compare the correlations from dependent samples [32]. Using this test, we found six significantly different correlations and four more marginally significant different correlations that all indicated differences in favor of the GEB.

The GEB includes two items (i.e., car use and buying organic food) that are similar in content to the convergent validation items. In order to account for this fact and to show the robustness of our findings, we ran an additional Rasch analysis without these two items. As expected, the new person scores were almost perfectly correlated (r = .98) with the person scores from the original adaptive test. Also as expected, the correlations between the person scores that were based on the new 48-item version and the validation items showed exactly the same pattern as the original results in Table 1.

Discussion

Our exemplary comparison between an introspective attitude measure (i.e., the NEP) and a behavioral attitude measure (i.e., the GEB) revealed two major weaknesses of the introspective attitude measure. First, our data showed low convergent validity for the NEP, thus replicating the commonly identified, often discussed, and never solved attitude-behavior gap that has been linked to classical introspective attitude measures. Second, but most important for this study, our results indicated low discriminant validity for the NEP, thus referring to a common method bias that has been shown in particular for introspective attitude measures [4]. Even though our competitive behavior-based attitude measure (i.e., the GEB) also showed some weaknesses because it did not show a perfect validation pattern, it nevertheless showed significantly better convergent and discriminant validity than the introspective attitude measure. By employing a widely used introspective attitude measure (i.e., the NEP), we were able to provide a solid replication of the problems (attitude-behavior gap and common method variance) involved in the use of introspective measures in general. Beyond the methodologically undesirable focus on just one measurement approach (e.g., [4, 8]), the corroborated gap between introspective attitude measures and behavior remains an unresolved problem for introspective attitude measures [34].

However, aspects of convergent validity—as the foremost aim of validation—have already been discussed widely elsewhere (e.g., [12, 14]). In this study, we focused more on discriminant validity as an important basis for distinctive measures that help to foster consistent and cumulative theoretical progress. Without discriminant validity, concepts are hard to distinguish. This can hinder the development of models with interrelated concepts because concepts become interchangeable, and model development that is based on these concepts becomes more arbitrary. By testing and comparing the discriminant validity of an exemplary introspective measure of environmental attitude with another environmental attitude measure that is not based on introspection, we found some evidence for common method bias as a cause for the low discriminant validity of widely used introspective measures. Even though we have no empirical evidence for which of the causes of the method bias were at work in our study, we argue that the most likely sources are common rater effects (e.g., the consistency motif, social desirability, and implicit theories) because they help to present a desirable and consistent self [4]. The reason for this might lie in the fact that introspective measures are much more prone to desirable and consistent self-presentation because they rely on evaluative and cognitive processes that easily allow for it. Thus, independent of the latent construct, the motive to consistently represent one self might lead to an equalization of answers on introspective measures, thus resulting in a measurement component that leads to common method bias and to low discriminant validity.

Low discriminant and convergent validities are especially problematic when introspective measures are used to predict overt behavior in applied research or when such measures are used in costly panel studies (e.g., the SOEP) that aim to provide meaningful information to a broad scientific community. Thus, even at this basic methodological level of measurement validity, it seems worthwhile to work on developing measures with significantly better discriminant validity and to consider more than just convergent validity. This is especially important in the field of environmental psychology because environmental attitude is frequently used to predict ecological behavior (for an overview, see [14]).

At least in the domain of environmental attitude, we were able to identify an alternative measure that was based on the Campbell paradigm, that is, the GEB. Beyond the promising results of our rather basic comparison study on construct validity, other studies using the GEB have already provided results on its good external validity with respect to actual behavior [22, 35, 36]. Even though the GEB is not based on independently observed behavioral responses and is instead based on self-reported behavior, its outcome corresponds well with actual behavior and even the ecological impact of the behavior [37]. More specifically, its criterion validity has been shown with an ecologically strong criterion on the individual level, that is, electricity consumption [25]. Furthermore, because the Campbell paradigm is based on the Rasch model, all the benefits that come with the Rasch model, such as adaptive testing, can be used and applied (e.g., [38]). For instance, the possibility to link two test of the same construct with only a few items that overlap, repeated measurement becomes much more convenient (e.g., [35]).

Using only single validation items that we correlated with our two measures is certainly a limitation of our study. It is not possible to calculate Cronbach’s alpha for single-item measures, and thus, we were not able to report the reliability of our single-item measures. Furthermore, it is common practice in psychometrics to assume that measures based on more items usually produce more reliable results than measures with fewer items (all else being equal). However, this view has been contested, and single-items measures are often quite acceptable measures as well [39]. Furthermore, other validation criteria such as observed behavior would strengthen our study further.

Finally, we would like to outline a way to shift from introspective measures to more diverse measurement. We showed that introspective and behavior-based items possess different response properties, which lead to different outcomes with respect to validation criteria. However, they are linked with respect to content when they refer to the same attitudinal object because each kind of item legitimately represents one aspect of the attitude concept, respectively (e.g., [7]). A person can articulate his or her attitude with a verbal statement (i.e., introspective item) or behave in a particular way and report this behavior (i.e., behavior-based item). Furthermore, according to the Campbell paradigm, articulating a verbal statement can also be interpreted as a performance on the same scale as the performance of self-reported ecological behaviors from the same attitudinal domain (e.g., ecological behavior). Thus, theoretically, it is possible to use both introspective and behavior-based items in the same scale. However, an empirical test needs to be applied to determine whether and how well this works for any set of items. Indeed, it has been shown in a methodological study that introspective and behavior-based items that address attitude toward the environment can be scaled together reliably [40].

The technical key to testing the fit of different items empirically—and thus to exchange introspective items for behavior-based ones—is the Rasch model. The ability of the Rasch model to utilize different sets of items to measure the same constructs (e.g., [35]) enables the researcher to adapt and optimize item sets without losing comparability across the different item sets. That is, a set with a focus on introspective items and a set that focuses on behavioral items that both refer to the same attitudinal object can be linked with only a few overlapping items. Technically, linking two such scales is a common and straightforward procedure that is described in textbooks and articles on Item-Response-Theory of which Rasch models are a part (e.g., [41, 42]). By exchanging introspective items for behavior-based items, reducing the common method bias is possible and enables a more diverse measurement on the item level.

Our exemplary comparison showed that the present focus on introspective measures might hinder the progress that is needed in theory development by producing spurious findings and that shifting toward more diverse measurement methods might ease this burden—at least in environmental psychology. However, environmental psychology is an applied subdiscipline of social psychology, and as such, it provides a typical example of a behavioral science that overuses introspective measures. Thus, the chances are good that our proposal to shift the focus from introspective measures to more diverse measurement methods will be useful in all behavioral sciences that are currently dominated by introspective measures.

Acknowledgments

We wish to thank Florian Kaiser for helping to determine the item parameters for our adaptive version of the GEB; Jane Zagorski for her language support; as well as the editors and two anonymous reviewers who helped to improve our manuscript substantially.