On Accurately Assessing Self-Reported Emotions

Psychologists prefer to use established, published measures of constructs. As a result, many psychologists have chosen to use the Positive and Negative Affect Schedule (PANAS) [7] to measure self-reported positive and negative affect when testing whether their manipulation influenced emotions. Indeed, the PANAS, which was originally published in the Journal of Personality and Social Psychology (JPSP), has been cited over 18,000 times (from Google Scholar, 22 July 2015), and is the most cited paper in JPSP behind Baron and Kenny’s [8] mediation-moderation paper (according to Google Scholar, 22 July 2015).

Often, research uses the PANAS to measure responses to situational manipulations of emotion, and often null effects are reported. Because emotions are only probabilistically linked to measures of emotion, it is inappropriate to assume that a null effect on a self-report measure means that emotional processes were not involved in the psychological phenomena of interest. That is, the (often poor) measurement of emotion should not be conflated with emotion itself. Emotion can be involved in driving a phenomenon despite null effects on measures of self-report.

Many psychological scientists use poor measures of emotion to discount emotion prematurely. In fact, we are scarcely immune from the allure of premature discounting, having done so in the not-so-distant past [9]. In this study, inspired by Terror Management Theory, participants were assigned to groups based on their ostensible painting preferences (minimal groups) and either prompted to write about their own death or watching television. They then completed the PANAS. We found no increase in negative affect or decrease in positive affect in the mortality salience condition, as measured by the PANAS, and argued that this suggested that affect was not involved in the effects of mortality salience. Similar flawed logic has been used in numerous published papers.

The way in which the field dismisses emotion based on self-report alone demonstrates the necessity for more comprehensive scrutiny of emotional processes in mainstream psychology. Emotion is a primary influence on psychological processes and behavior. It need not be controlled for; rather, it should always be considered and measured appropriately. Researchers do not administer cognitive measures in an attempt to “rule out” cognition as a factor in a psychological process (because cognition is assumed to play a role) and emotional/motivational processes are at least as fundamental as cognition. If an organism is behaving, it can safely be assumed it is motivated and emoting. Measuring emotion should be directed at identifying the contribution of emotion to the behavior.

As we have already alluded to, emotions involve more than self-reported feelings. Although assessing self-reported emotional responses may be appropriate (and accurate) in some cases, one of the most widely used measures of self-reported emotional experience, the PANAS [7], is fraught with problems that may lead to it failing to yield support for the role of affective processes as resulting from and being involved in psychological phenomena.

First, consider a bit of history about the development of the PANAS. When Watson and colleagues [7] developed it, they were intent on arriving at a measure that would provide statistically independent measures of positive versus negative affect, “that is, terms that had a substantial loading on one factor but a near-zero loading on the other” (p. 1064) [7]. So they began with a long list of emotion labels, and then discarded all words that did not fit the statistical independence of positive versus negative affect. Perhaps consequently, they arrived at a measure of positive affect that does not include many common positive emotion words such as happy or joyful, but instead includes items such as alert, active, and attentive that may not always be positive in valence (e.g., a person could be alert, active, and attentive during threat). Similarly, the measure of negative affect does not include many common negative emotion words such as sad, down, grieved, and angry. Watson and colleagues [10] later suggested that their measures should be referred to as positive activation and negative activation because the emotion words on the scale were activated (i.e., high arousal) positive and negative emotions. This is partly accurate and explains why sadness-related words are missing but it fails to explain why some positive affects are missing (e.g., joyful) and why less common activated negative affect words are used for some concepts (i.e., irritated is included, rather than angry).

It would seem that the PANAS may be limited only to those contexts where activated emotion states are involved, indicating the need for other validated measures of emotion. Yet, even the use of the PANAS in these contexts could be problematic. Firstly, recent factor analytic studies have revealed that the factor structure of the PANAS is not maintained under emotion-evoking manipulations. Instead, items load on additional factors reflecting discrete emotions. For example, from before, to during, to after taking an exam, the PA items clustered into three discrete positive emotions, “joy”, “interest”, and “activation” [11]. Whereas when participants were angered, PANAS items loaded on three factors: the PA items loaded on the first factor, the anger items on the second factor, and the fear/anxiety items on the third factor [12].

Another issue is that PA scores have been found to increase during anger (an approach-motivated state that is negative in valence), suggesting that the positive activation scale does not always measure positivity, but may be sensitive to approach motivation [12]. In support of the idea that PA reflects approach rather than pure positivity, recent research has revealed that the PANAS is contaminated by a large first factor that could be referred to as evaluation or social desirability [13]. This research has revealed that responses to PANAS positive activation words (e.g., proud) are highly positively correlated with responses to positive antonyms of those words (e.g., humble). When this first factor is statistically removed, anger correlates with positive activation, as revealed in other studies [12,14].

Other Measures of Self-Reported Emotions

The PANAS is not the only way to measure self-reported emotions, yet other questionnaires are also commonly based on dimensional models, similar to the PANAS. Some of these questionnaires use multiple items to assess the broad dimensions of positive and negative affect, such as the Brief Mood Introspection Scale [15], or use single items to assess broad dimensions of valence, arousal, and dominance, such as the self-assessment manikin [16].

Some affect measures that do focus on discrete emotions have been developed for clinical purposes. The Profile of Mood States (POMS) [17] is a measure of emotion blends (i.e., anger-hostility, tension-anxiety, depression-dejection) and other affective states (e.g., fatigue-inertia; vigor-activity; confusion-bewilderment). It is a scale that must be purchased from the publisher in order to be used for research purposes. Since it was produced in 1971, the POMS has been used primarily in research and applied settings with psychiatric outpatients, medical patients, and in sports psychology. The Multiple Affect Adjective Checklist (MAACL) [18] and its revised version (MAACL-R) are comprised of subscales that are designed to assess self-reported Anxiety, Depression, Hostility, Positive Affect, and Sensation Seeking. Like the POMS, the MAACL has been used primarily in clinical psychology and psychiatry, and it has been used to assessed trait affect or day-to-day changes in affective states. Both the POMS and MAACL lack subscales for certain basic emotions (e. g., disgust). Furthermore, their usefulness in assessing the full range of emotional experience is limited by their focus on maladaptive affect.

Other studies examining situationally induced emotions have used single items to measure the discrete emotions evoked [19]. The use of single items to measure emotions has proven effective in these published studies, but generally speaking, methodologists express concerns over the use of single items to measure psychological constructs. Short measures are more likely to contain error variance [20] because they cannot benefit as much from aggregation across multiple items [21]. The use of a single item to measure a discrete emotion may not accurately capture an individual’s subjective emotional state because the individual may interpret the item differently at that moment.

In support, a review of personality questionnaires ranging from two to eight items in length found a positive correlation of r = .77 between questionnaire length and reliability [22]. If self-report questionnaires of emotions have low reliability, the chances of observing correlations between specific emotions and other measured constructs is reduced, because the correlation between two measures is limited by the reliability of each measure. In addition, because unreliable measures contain more measurement error, the determination of effect size between the unreliable measure and other measures will be more difficult than if reliable measures were used. Short measures of emotion, however, may prove to be more valid, as they assist in reducing participant boredom and fatigue [23]. Some research has suggested that the optimal trade-off between reliability and validity occurs with two to four items [24].

Other measures of discrete emotions have been produced, but these discrete emotions measures do not measure a larger set of emotions. For example, measures have been developed to assess the trait tendency to experience various discrete emotions, such as proneness to anger, awe, compassion, disgust, embarrassment, envy, gratitude, guilt and shame, happiness, and pride [25, 26, 27, 28, 29, 30, 31, 32, 33, 34]. However, these questionnaires measure traits and are not designed to measure state emotions.

Self-report questionnaires designed to assess state emotions exist for a few emotions, such as anxiety [35], pride [34], and shame and guilt [36]. But these state emotion questionnaires, like the trait emotion questionnaires, stand in isolation of other emotions and are not part of a questionnaire assessing a broad array of emotions. The current research was designed to provide a state measure of a wide array of emotions.

A single instrument that assesses discrete emotions relatively comprehensively is important for several reasons. One is that the same manipulation may evoke primarily one emotion in a particular individual, but a different emotion in another. For example, negative feedback on an essay might be intended to increase anger, and may in fact elevate anger in a majority of participants. However, if this manipulation evokes primarily sadness in a subset of participants, this would be missed if the researcher used only a measure of the single emotion of anger. Additionally, individuals may experience mixed emotions, which may have very different effects than the same emotion when not blended with other emotions. For example, anger is usually an approach-related emotion. However, when anger is blended with anxiety, it may be associated with withdrawal motivation [37]. A single instrument that assesses the broad range of emotions would allow researchers to identify when emotions are relatively pure versus when they are blended.

Theories of Basic Emotions

In designing our Discrete Emotions Questionnaire (DEQ), we aimed to include emotions that are considered "basic" by prominent emotion theories. These theories of emotion were developed by Paul Ekman [3, 38], Carroll Izard [4, 5], Jaak Panksepp [6], and Phillip Shaver and colleagues [39]. Although none of these theorists consider their lists to encompass the entire range of possible emotions, they (and we) consider these viewpoints on the structure of emotions a useful starting point. These various theories have considerable overlap in the basic emotions proposed, even though they were arrived at using diverse methods, including facial expressions of adult humans (Ekman), behaviors of human infants and children (Izard), direct brain stimulation and behavioral responses of non-human animals (Panksepp), and sorting of emotion words by adult humans [39]. Below we consider each emotion for which we included a subscale in the DEQ.

The DEQ is predicated on a discrete, versus a dimensional, approach to emotions. Regarding the ‘debate’ between discrete approaches and dimensional approaches, we believe that both have value. Discrete emotions can be categorized along dimensions, including valence (positivity/negativity), arousal (high/low), and motivational direction (approach/avoid). However, discrete emotions also possess other characteristics that do not fall neatly into dimensions, such as specific action tendencies, subjective feelings, evoking situations, and cognitive appraisals. Whether emotions are viewed as discrete or dimensional, people can still name specific emotion types and having a measure that allows them to distinguish between these different emotion types is useful and important. The emotions measured by the DEQ are listed below.

Anger is often regarded as a negative, high arousal emotion that is associated with approach motivational tendencies [40, 41, 42]. Neurally, anger involves testosterone, substance P (a neuropeptide that acts as a neurotransmitter and neuromodulator), the medial hypothalamus, amygdala, and the periaqueductal gray [6, 43, 44]. The full facial expression of anger involves the muscles of the brow moving inward and downward; the eyes fixed in a hard stare; and the nostrils and wings of the nose expanded [4, 5, 45]. When laypersons sort emotion words into groupings, the Anger cluster includes words like anger, rage, irritation, and exasperation [39].

Disgust is also regarded as a negative, high arousal emotion, but it is associated with withdrawal motivational tendencies. Disgust is considered a basic emotion by Ekman and Izard but not by Panksepp, who views disgust as a sensory affect. Shaver et al. [39] found that the two disgust-related words (disgust, revulsion) included in their emotions list clustered with the anger words, forming a branch along with contempt. Thus, if disgust is a basic emotion, researchers may need to devote additional effort to finding words that capture the construct. Neurally, disgust uniquely involves the inferior frontal gyrus/anterior insula relative to other emotions [46]. The full-face expression of disgust involves the brows being drawn down and together and a wrinkled nose. The upper lip is also pulled up and the lower lip is pulled downward. The tongue may be pushed forward out of the mouth [4, 5, 45].

Fear is often regarded as a negative, high arousal emotion that is associated with withdrawal motivational tendencies. Neurally, fear involves corticotrophin releasing factor, adreno-cortico-trophic-hormone, cholecystokinin, central and lateral amygdala, anterior and medial hypothalamus, PAG, and the lower brain stem [6]. The full-face expression of fear involves the eyebrows lifted and slightly pulled together; the eyes opened wide; and the corners of the mouth retracted straight back with the mouth slightly open [4, 5, 45]. When laypersons sort emotion words into groupings, the Fear cluster includes words like horror, alarm, terror, and fear [39].

Anxiety is often regarded as a negative, high arousal emotion that is most likely associated with behavioral conflict [47]. Anxiety is not characterized as a "basic" emotion by emotion theorists such as Ekman, Izard, or Panksepp. Similarly, Shaver et al. [39] found that anxiety words (e. g., anxiety, nervousness, tenseness) form a branch within the Fear cluster. However, many areas of research point to the theoretical and empirical importance of distinguishing anxiety from fear. Fear is often evoked by discrete, acute threats, whereas anxiety is evoked by vague, potential threats [48, 49]. Neurally, anxiety is not as well understood as fear, but the brain systems that contribute to anxiety overlap with those that contribute to fear. One key neural difference between fear and anxiety is that the peptide corticotropin-releasing factor (CRF) has a special role in anxiety [49]. The facial expression of anxiety is distinguished from other facial expressions of emotion by eye darts and head swivels that presumably increase the spatial area seen by the eyes [50]. When laypersons sort emotion words, the anxiety branch includes words like anxiety, nervous, and tense [39]. For the purposes of the DEQ, we saw value in attempting to distinguish anxiety from fear, although we expected that these constructs would show some overlap.

Sadness is often conceptualized as a negative, low arousal emotion; it appears to be mostly associated with the approach motivational system. In Panksepp's system, the affect that maps onto sadness is called PANIC/GRIEF. Although this terminology may be surprising to human researchers, PANIC/GRIEF, like human sadness, is a response to loss (particularly the infant animal's separation from the mother), and characterized by distress vocalizations (crying). Neurally, PANIC/GRIEF (i.e., sadness) involves endorphins, corticotrophin releasing factor, cortisol, glutamate, midbrain PAG, medial diencephalon, ventral septal area, preoptic area, bed nucleus of stria terminalis, cingulate gyrus, amygdala, and hypoththalamus [6]. In human neuroimaging studies, sadness uniquely activates middle frontal gyrus and head of the caudate/subgenual anterior cingulate cortex [46]. The full facial expression of sadness involves the inner corners of the brows being drawn obliquely upward and together, with the eyes begin slightly narrowed; the corners of the mouth are pulled downward and the chin may be pushed up and quiver [4, 5]. When laypersons sort emotion words, the Sadness cluster includes words like suffering, sadness, depression, and disappointment [39].

Joy or happiness is conceptualized as a positive emotion that could be associated with a variety of intensities of approach motivation; most theoretical perspectives are silent or mixed regarding where joy sits on the arousal dimension. Joy is described as a basic emotion by Ekman and Izard, and joy words form a major cluster in the analysis by Shaver et al. [39]. The conceptually closest emotion in Panksepp's [6] system may be PLAY, which seems similar to the joy posited by other theories. When laypersons sort emotion words, the Joy cluster includes words like cheerfulness, joy, enthusiasm, and contentment [39].

Most theories concerned with basic emotions have only carefully considered one or two positive emotions, and we believe that this is unfortunate for emotion science. Of course, theorists, including Ekman [51], have discussed many other positive emotions but research has not delved too far into these discrete positive emotions. One reason for the difficulty in distinguishing discrete positive emotions may be the overly dominant influence of valence on self-reports. In a stark demonstration of this, Pettersson and Turkheimer [13] found that a large first factor, which they refer to as "evaluation" confounds self-reported moods. They created a measure, based on the PANAS, by adding words of opposite meaning but identical valence (for example, proud was matched with humble, and determined was matched with easy-going). To assess evaluation, the PANAS items and matched antonyms were rated on how desirable it would be for a person to endorse each item. Separate participants rated the items on how much they felt these words applied to themselves, in general. When the first principal component was extracted from participants' self-endorsement of the items, it correlated r = .96 with the evaluation of the items, strongly suggesting that individuals endorsed the items based on valence, rather than semantic content. Similar results have been found in early research [52]. The evaluation component may be equivalent to social desirability [53, 54, 55]. In creating the DEQ, we considered that it may be important to control for this large evaluation component, in order to accurately measure other discrete emotions, particularly positive emotions.

In addition to including a Joy/Happiness subscale (which we suspect would be equivalent to evaluation, or valence) on the DEQ, we aimed to include subscales for two other positive emotional states that have received much theoretical and empirical attention. Broadly speaking, these two states can be thought of as low- and high-approach positive affect. We refer to these subscales as Satisfaction and Desire, respectively. These concepts align with perspectives that examine appetitive or pre-goal positive states as being distinct from consummatory or post-goal positive states [56], or "wanting" as distinct from "liking" [57]. The positive affective states of high-approach (desire) and low-approach (satisfaction) are associated with different neural structures and neurochemicals [6, 56, 57, 58]. Low-approach positive affect involves a small area within the nucleus accumbens and the posterior half of the ventral pallidum, brain regions which are sensitive to opioids and endocannabinoids [57]. High approach positive affect (SEEKING, in Panksepp's system) involves the amygdala, nucleus accumbens, and frontal cortex [59]. The neurotransmitter dopamine, produced in the mesolimbic and mesocortical dopamine circuits, activates these regions. Pregoal, high-approach-motivated positive affect (desire) presumably assists in promoting reward acquisition, whereas postgoal, low-approach-motivated positive affect (satisfaction) presumably assists in promoting reward enjoyment.

The research we review concerning the brain regions, hormones, and neurotransmitters involved in discrete emotions is primarily based on non-human animal evidence, which is typically more invasive and therefore more precise than the research that can be conducted with humans. There is evidence regarding brain regions involved in discrete emotions that is obtained from functional magnetic resonance imaging (fMRI) of humans lying supine while experiencing emotions. A supine posture has been found to reduce certain physiological responses associated with certain emotions [60, 61, 62]. This fMRI evidence presents a more mixed picture as to whether there is any discrete emotions specificity with brain regions. That is, some meta-analyses have suggested there is no support [63], whereas other meta-analyses have suggested there is support for discrete emotions being linked to specific neural circuits [46]. Hamann’s [64] interpretation of this literature is prescient, “… although neuroimaging studies have identified consistent neural correlates associated with basic emotions and other emotion models, they have ruled out simple one-to-one mappings between emotions and brain regions, pointing to the need for more complex, network-based representations of emotion” (p. 458).

Because the DEQ is intended primarily for use in measuring self-reported emotions among lay participants, we were careful to ensure that it was made up of words that are 1) clearly understood by lay English speakers (as opposed to psychological or emotion scientists), and 2) actually used by ordinary individuals in describing their emotional experiences. Instead of developing the DEQ from an existing list of emotion words, we preferred to build the instrument on a foundation of words that are spontaneously used and understood by lay persons. Thus, Study 1 used a bottom-up, open-ended method to collect words used by individuals to name the emotions they experienced in situations that would, according to emotion theory, evoke the emotions of interest: Anger, disgust, fear, anxiety, sadness, joy, desire, and satisfaction.

Method

Materials.

We created story prompts based on the themes that have been identified in emotional accounts in past research. Smith and Lazarus [65] identified "core relational themes" for anger, guilt, sadness, and fear/anxiety. Shaver et al. [39] analyzed emotional stories and identified the prototypical themes for fear, sadness, anger, joy, and love. The story prompts for Study 1 were based on these themes from past research (Table 1). It is important to note that the prompts avoided the use of emotion words. Instead, they focused on the prototypical characteristics of events that tend to produce the target discrete emotions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Study 1: Prompts Used to Elicit Emotional Recall.

https://doi.org/10.1371/journal.pone.0159915.t001

Manipulation checks.

To determine whether the emotion manipulation was successful, a judge who was blind to condition coded each story for whether it included each of the eight emotion themes (non-exclusive, S1 Data). Results appear in Table 2. To assess reliability, a second judge coded a random sample of 15% of the stories. Inter-rater reliability was good, Cohen’s Kappa = 0.73.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Study 1: Story Themes Identified by Coding.

https://doi.org/10.1371/journal.pone.0159915.t002

Results

We performed a count of the words generated for each condition. Different grammatical forms of a word (e. g., anger, angry) were counted as a single word. Table 3 shows the most common words generated in response to each prompt. Words were included in the table if the word was generated at least twice for “Best Word” or “Vernacular,” or at least three times for “Other Words.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Study 1: Most Frequently Generated Words for Each Condition.

https://doi.org/10.1371/journal.pone.0159915.t003

From this list, we created a preliminary word list for the emotion instrument, with six items per emotion category. This preliminary list included the most frequently generated words for each emotion prompt which overlapped minimally with words generated in response to the other emotion prompts. For the categories for which participants did not generate at least six common words, we supplemented the list with items from Shaver et al. (1987) that fell near the participant-generated words in a cluster analysis, and that were rated as being good exemplars of emotion (rated at least 3.00 on a 4-point scale where 1 = I definitely would not call this an emotion and 4 = I definitely would call this an emotion) [38].

Method

Results

Using Statistica Version 12 (StatSoft, Inc.), the data (S2 Data) were subjected to a factor analysis using maximum likelihood with varimax (raw) rotation; cases with missing values were rejected on pairwise basis (Table 4). Initially, the number of factors allowed was 8, because we predicted that 8 factors would result. This analysis yielded five factors with Eigenvalues greater than 1.00. The first factor had an Eigenvalue of 24.92, and all of the positively valenced words loaded negatively on this factor. The second factor had an Eigenvalue of 5.58, and all of the fear and anxiety words loaded positively on this factor (except “shock,” which had a loading of 0.44). The third factor had an Eigenvalue of 2.14, and all of the anger words loaded positively on this. The fourth factor had an Eigenvalue of 2.43, and all of the sad words loaded on this factor. The fifth factor had an Eigenvalue of 1.58, and all of the disgust words loaded on this factor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Study 2: Exploratory Factor Analysis Results.

https://doi.org/10.1371/journal.pone.0159915.t004

These results suggested that the negative emotion words differentiated into specific emotions, with the exception of fear and anxiety. However, the positive emotion words on our preliminary scale loaded on the large first factor, which, based on past research [13], is equivalent to evaluation/valence.

Composite variables (subscales) were created by taking the mean of the four items with the highest loadings on each factor and small cross-loadings on other factors (shown in bold in Table 4). Subscales were not created for the positive affect items, because these did not load on separate factors. To examine whether the subscales were sensitive to our emotion manipulation, one-way analyses of variance (ANOVA) were conducted with the emotion manipulations as independent variables and the mean ratings for each emotion category as the dependent measures (Table 5). These results showed that these “best four” items were elevated by the expected manipulations, with the exception of the anger subscale and the anxiety subscale. For the anger subscale, the anger prompt elevated ratings significantly more than any except the fear prompt. The increase in the mean for the anger prompt compared to the fear prompt was not quite statistically significant (p = .06). For the anxiety subscale, the anxiety prompt elevated ratings more than any prompts except the anger and fear prompts. Means for the anger prompt did not differ from those for the anxiety prompt (p = .73). Furthermore, means for the fear prompt elevated ratings on the anxiety subscale more than any other prompt, including anxiety (p = .03). These results suggested that our preliminary scale discriminated between all negative emotions except anxiety and, to some extent, anger. Based on these results, we conducted Study 3 with a more controlled emotion manipulation and additional positive emotion words.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Study 2: Mean Responses on Preliminary Subscales in Response to Emotional Recall Prompts.

https://doi.org/10.1371/journal.pone.0159915.t005

Method

Materials.

The stimuli were composite stories created from typical memories submitted by participants in Study 1 and Study 2. Two scenarios were created for each emotion category (Table 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Study 3: Scenarios Used to Elicit Emotion.

https://doi.org/10.1371/journal.pone.0159915.t006

Results

The data (S3 Data) were subjected to the same factor analysis parameters as used in Study 2. Our original conceptual prediction was eight factors or emotion subscales. However, Study 2 revealed fear and anxiety to be highly related and to load on the same factor. Thus, in Study 3, we set the factor analysis to have seven factors (Table 7). The first factor had an Eigenvalue of 25.02 and all of the positive items loaded negatively on this factor. The second factor had an Eigenvalue of 8.65 and all of the fear and anxiety items loaded on this factor. The third factor had an Eigenvalue of 3.71 and all of the anger items loaded on this factor. The fourth factor had an Eigenvalue of 2.73 and all of the sadness items loaded on this factor. The fifth factor had an Eigenvalue of 1.99 and all of the desire items loaded on this factor. The sixth factor had an Eigenvalue of 1.47 and all of the disgust items loaded on this factor. The seventh factor had an Eigenvalue of 0.94 and all of the relaxation items loaded on this factor; however these items also cross-loaded on the first factor. Although we began this research thinking that low approach positive affect might be best captured by a word like “satisfaction,” the factor analysis did not reveal that “satisfaction” loaded separately from other positive words. Because the words that did load (somewhat) separately from happiness were associated with and included the word “relaxation,” we labelled this last factor “relaxation.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Study 3: Factor Loadings.

https://doi.org/10.1371/journal.pone.0159915.t007

Composite variables (subscales) were created by taking the mean of the four items with the highest loadings on each factor (shown in bold in Table 7). For all of the negative affect subscales, these words were identical to those used in Study 2. To examine whether these subscales were sensitive to our emotion manipulations, one-way ANOVAs were conducted with the emotion scenarios as independent variables and the average ratings for each subscale as the dependent measures (Table 8). These results showed that average ratings on the subscales were elevated by the intended manipulations, with a few exceptions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Study 3: Mean Responses on Subscales in Response to Emotional Scenarios.

https://doi.org/10.1371/journal.pone.0159915.t008

For the fear subscale, only fear scenario 2 elevated ratings more than any other manipulation. For the anxiety subscale, ratings following anxiety scenario 1 did not significantly differ from fear scenario 2, and ratings following anxiety scenario 2 did not differ significantly from fear scenario 1 or 2. For the desire subscale, only desire scenario 1 elevated ratings more than any other scenario. For the relax subscale, ratings following relax scenario 2 differed only marginally (p = .07) from ratings following desire scenario 1.

These results showed that each of the subscales was sensitive to manipulations of the intended discrete emotion, although some of the emotional scenarios were more effective than others. The only exception was the anxiety subscale, which was sensitive to both anxiety and fear scenarios.

Method

Results

To examine whether these subscales were sensitive to our emotional picture manipulation, one-way analyses of variance (ANOVA) were conducted with the picture sets as independent variables and the average ratings on each emotion subscale as the dependent measures (Table 9). These results showed that ratings on the discrete emotion subscales were elevated by the expected picture sets, with minor exceptions. Although ratings on the fear subscale were elevated more by the fear pictures than by the anxiety pictures, ratings on the anxiety subscale were elevated equally by fear and anxiety pictures. Also, ratings on the desire subscale were only marginally higher for desire pictures compared to relaxation pictures (p = .06). Reliability for all subscales was high (Cronbach’s αs > 0.80, Table 10).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Study 4: Mean Responses on Subscales in Response to Emotional Photographs.

https://doi.org/10.1371/journal.pone.0159915.t009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Subscale Reliabilities,

https://doi.org/10.1371/journal.pone.0159915.t010

Given the strong predictions derived from past theory and research, we thought the use of controls for family-wise error rate was unnecessary. However, when Bonferroni corrections were applied to the ANOVAs (.05/7 = .0071), the p value needs to be less than .007 to be acceptable. In all cases, the p values were acceptable. For Studies 2, 3, and 4, all p values for the ANOVAs were less than .001. Study 1 did not use ANOVA.