https://orcid.org/0000-0003-3156-336X
Figures
Abstract
As perceivers, we need to understand context to make social judgments about emotion, such as judging whether emotion is appropriate. We propose a graphic novel-like method, the emotion storyboard, for use in research on social judgments of emotion. Across two studies, participants were randomly assigned to read emotion storyboards or written vignettes to compare the efficacy of the emotion storyboard to that of vignettes in studies on social judgments of emotion. In Study 1, undergraduates (N = 194) answered comprehension questions and rated story clarity and immersion. Participants also made social judgments of emotion by rating main character emotion control and appropriateness of intensity. To further compare the efficacy of the methods, in Study 2, Amazon Mechanical Turk workers (N = 213) answered comprehension questions while response times were recorded, rated clarity, answered a race manipulation check, and rated main character emotion type appropriateness. Overall, emotion storyboards resulted in greater clarity ratings, greater race manipulation check accuracy, and in some instances, enhanced comprehension and comprehension response times relative to vignettes. In emotion storyboards, main character emotion was rated more controlled and more appropriate in intensity, but not different in emotion type appropriateness, than in vignettes. Overall, the method offers a new method of examining social elements of emotion that enhances comprehension and maximizes experimental efficiency.
Citation: McCormick-Huhn K, Shields SA (2021) The emotion storyboard: A method to examine social judgments of emotion. PLoS ONE 16(4): e0249294. https://doi.org/10.1371/journal.pone.0249294
Editor: Hedwig Eisenbarth, Victoria University of Wellington, NEW ZEALAND
Received: July 19, 2020; Accepted: March 15, 2021; Published: April 2, 2021
Copyright: © 2021 McCormick-Huhn, Shields. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files have been shared through the Open Science Framework and are available at: https://osf.io/6c5bs/.
Funding: This work was supported by a Superior Teaching and Research Award from The Pennsylvania State University’s College of the Liberal Arts, awarded to K.M-H. URL: https://la.psu.edu/current-students/graduate-students/student-resources/funding-opportunities/superior-teaching-and-research-star-award The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Emotion is social. Emotion is often created in response to social elicitors, affected and regulated by others, and understood by the self through others’ reactions. Therefore, as perceivers, we need to understand the person’s context (i.e., the particular social setting) and the other people involved in the person’s situation to understand and make judgments about their emotion. In experimental work, vignettes are commonly used to assess social dimensions of hypothetical others’ emotion. As an alternative, we propose a new graphic novel-like method, the emotion storyboard, to assess social judgments of emotion in context while enhancing comprehension of content. Although cartoons are commonly used to depict emotion in psychological studies with children [1] and graphic novel formats are used to maximize comprehension in other disciplines [2], to the best of our knowledge, the graphic novel-like format is new in the psychological study of adult emotion. In the present research, we compare the effectiveness of emotion storyboards and vignettes as methods for examining social evaluations of emotion in context.
Emotion is social
In recent years there has been a move from conceptualizing emotions as internal, private states to a perspective that recognizes emotion as social [3–6], in that emotion is affected by context and by the actions and reactions of other social actors within the context. Contextual cues affect both perceived type and intensity of emotion in oneself [7] and in others [8]. For instance, the happiness and pride of a face expressing happiness were rated as greater in the context of achieving a high exam grade than an average exam grade [9]. Simply the presence of other faces expressing emotion while looking at a target face influenced ultimate judgments of a target face’s emotion [10], even when the other faces were presented subliminally [11]. The study of emotion, therefore, requires methods that include social contextual information. We propose the emotion storyboard, by combining illustrations and text, may be well-positioned to examine emotion research questions about emotion that are affected by social context.
Measuring emotion as social and social judgments of emotion
A variety of methods can be used to examine emotion as a social process, including face and full body photographs and morphs, audio, vignettes, vignettes with photographs, videos, and innovative methods, such as the Amsterdam Dynamic Facial Expression Set [12]. Different types of emotion-relevant questions, however, require different methods. For example, to depict emotion in context over an extended time-course, researchers often rely on vignettes or videos [13].
Vignettes are written descriptions of hypothetical events. Vignettes have long been used to examine social judgments of emotion. Social judgments of emotion include people’s perceptions, beliefs, stereotypes, and attitudes about hypothetical third-party others [14–16]. Vignettes can be particularly effective for examining social judgments of emotion because their hypothetical nature and third-party focus allow for depersonalized and unthreatening responding [15]. Additionally, vignettes can provide social contextual information that affects evaluations, such as whether emotion is under control and authentic [17], whether emotion is appropriate and typical [18], and the status of the person experiencing the emotion and their likely action tendencies [19]. Vignettes are also used to induce emotion or to stimulate people’s predictions of how they might act in a given situation. There are, of course, limitations to using vignettes, such as disconnection from social experiences in emotion induction and detachment from real-world consequences in hypothetical decision-making situations [20, 21]. In the present studies, we focused on the first type of vignette method: using vignettes to examine people’s social judgments about emotion.
Why a graphic novel approach?
In the present studies we were most interested in comparing vignettes used for the perception and social evaluation of third-party others’ emotion in context with our proposed graphic novel-like method, the emotion storyboard. In graphic novel formats, illustrations and text are combined to convey stories. Our method, the emotion storyboard, uses a graphic novel-like format to visually represent emotion in context. In psychology, cartoons have long been used to measure emotion recognition and understanding in typically developing children [1, 22, 23] and children with autism spectrum disorders and other emotion-processing difficulties [24]. Cartoons have also been used to examine Theory of Mind reasoning [25] and to examine emotion cross-culturally [26]. To our knowledge, graphic novel-like formats using illustrations and text to create story plots have not been used as an alternative to vignettes to examine adult emotion.
The graphic novel has gained both popularity and legitimacy in public discourse [27] and is used in other academic fields for research and teaching [28]. The American Medical Association’s Journal of Ethics, for example, devoted a special issue to the study of graphic medicine (i.e., use of visual narratives; [29]). Graphic novel methods have been used to teach concepts in medicine [28], business [30], history [31], literacy [32], and secondary school [33]. Studies in these domains have demonstrated that graphic novels and illustrated content resulted in greater recall [30], enhanced test performance and reading behavior [34], greater reading motivation [2], and comparable decision-making while requiring less reading time [35] than text-only formats. Overall, graphic novel formats have resulted in greater memory of content, greater ease of processing, and greater motivation to engage with content, than text alone. Findings from other disciplines suggest that emotion storyboards could enhance comprehension of content relative to written vignettes in research on social judgments of emotion.
A new method for obtaining judgments about others’ emotion: The emotion storyboard
In the present studies, we examined the relative effectiveness of emotion storyboards and vignettes for study manipulations of social emotion. We also conducted exploratory analyses for questions of interest that we did not have specific hypotheses for. In Study 1, we investigated whether the emotion storyboard method could convey information about social context comparable to the vignette method, while enhancing comprehension of information. We hypothesized a difference in effectiveness of the two methods on comprehension, clarity, and immersion. Our exploratory analyses in Study 1 examined whether the two methods yielded differences in social judgments of emotion, specifically character emotional control and appropriateness of emotional intensity.
In addition, we examined two mechanisms that could possibly explain greater comprehension for the emotion storyboard than vignette. First, because graphic novels have yielded greater ease of processing than text formats [35], we reasoned this could be due to the graphic presentation being clearer and less confusing than text. If so, greater clarity may explain enhanced comprehension. Second, based on findings of greater reported motivation to read graphic novels than text [30], we reasoned people might experience greater feelings of immersion or absorption in the graphic novel reading process. If so, greater immersion could also explain enhanced comprehension. Therefore, we predicted that participants would pass comprehension checks at a higher rate and rate the stories as clearer and more immersive in emotion storyboard than vignette conditions. We also predicted that clarity and immersion would mediate the relationship between format and comprehension. Finally, we explored how the emotion storyboard method would fare relative to the vignette method for evaluative judgments that are affected by social context, namely, perceptions of emotion as under control [17] and emotional intensity as appropriate [18].
In Study 2, we examined whether emotion storyboards may offer a way to maximize accuracy relative to vignettes for identifying social group memberships (e.g., race; socioeconomic status; gender) in studies about social emotion judgments. In vignettes it can be difficult to represent social group memberships without being too subtle (e.g., using names stereotypical of particular social groups). To test this possibility, we examined participants’ accuracy on a race manipulation check in emotion storyboards compared to only subtle name manipulations in vignettes. In Study 2, we again examined comprehension and clarity. We also explored whether emotion storyboards resulted in quicker comprehension question response time and lower overall story reading time compared to vignettes. Additionally, in Study 2 we conducted an exploratory test of the relative effectiveness of the two methods for evaluative judgments of emotion other than emotion expression. Specifically, in line with feeling rules [36], we investigated evaluations of the appropriateness of the emotion type for the given situation [37]. Pilot studies as well as Study 1 and Study 2 were between-subjects designs.
Materials
Emotion storyboards and vignettes depicted a main character experiencing anger in the workplace, an emotion-relevant situation that has been the topic of studies on social judgments of emotion [19, 38]. Our goal was to compare a narrative vignette of the type commonly used to describe a person’s emotional experience [e.g., 17, 19] to the emotion storyboard. We developed a written vignette by describing a character becoming angry due to a situation at work, and then worked with a graphic artist to translate the written vignette into panels for the emotion storyboard version.
A graphic artist developed characters for the emotion storyboards and held constant gendered facial features (e.g., brow ridges) that are confounded with emotion expression type [39].
For Study 1, each method was compared across two scenarios that involved a character experiencing workplace anger to ensure that participants’ judgments were not specific to features of the particular social situation. To select our two scenarios, we drew on reports of two surveys conducted with employees in business workplaces [40, 41] and selected three situations from the reports to pilot: taking credit for a coworker’s work, spreading a rumor about a coworker, and stealing a coworker’s lunch. Study 1 materials depicted a White man character (see Fig 1).
For Study 2, we used the rumor vignette and emotion storyboard with the White man character from Study 1. We also used that same vignette and emotion storyboard with an Asian man character. To manipulate race in the vignette, we used Anglo and Chinese names, Andrew and Xiang, respectively, that were used previously in vignette research [42]. For the emotion storyboard, the White man character was modified to have Asian features, while keeping other properties of the face (e.g., brow ridges; eyebrow position) constant [39] (see Fig 2).
Pilot studies
Four pilot studies were conducted to test equivalence of scenarios, accurate perception of main character’s gender and race, and type and intensity of emotion expressed by the main character in the final panel of the storyboard.
Scenario pilot study
We first used a between-subjects design to pilot three scenarios (N = 247) to examine perceptions of how justified someone would be to feel angry in the various situations. Participants were 247 university undergraduates (125 women, 121 men, and 1 transgender person; ages 18–49, M = 19.71, SD = 3.72). The racial/ethnic composition of participants in the sample was predominantly White (77.3%; Asian/Asian-American, 6.9%; Black, 4.5%; multiracial, 3.6%; Latinx, 3.2%; Middle Eastern, 1.6%; Native American/Alaskan, 1.2%; Native Hawaiian or Pacific Islander, 0.4%; Southeast Asian, 0.4%; other, 0.9%). For the present study, we selected two scenarios that were rated equivalently on justification of the main character’s anger, (t(165) = 1.12, p = .266, d = 0.17, 95% CI [-0.13, 0.48]), on a scale of 1 (Not justified at all) to 7 (Very justified): a character at work learning that a rumor had been spread about them (M = 5.28, SD = 1.39), and a character discovering that a coworker was stealing their lunch from the break room (M = 5.02, SD = 1.59).
Character gender and race in emotion storyboard pilot studies
Next, we conducted pilot studies to ensure gender and race of the main character in the emotion storyboards were accurately identified by at least 80% of the sample. For Study 1, participants were university undergraduates (N = 17; 9 women, 8 men; ages 18–25, M = 20.18, SD = 1.67). The racial/ethnic composition of participants in the sample was predominantly White (64.7%; Asian/Asian-American, 11.8%; Black, 11.8%; Latinx, 5.9%; Southeast Asian, 5.9%). Participants provided open-ended responses that identified the main character as White (94%) and man (100%).
For Study 2, a sample of 18 university undergraduates were recruited for a pilot study (8 women, 9 men, and 1 other; ages 19–39, M = 20.78, SD = 4.67). The racial/ethnic composition of participants in the sample was predominantly White (61.1%; Black, 22.2%; Asian/Asian-American, 11.1%; Latinx, 5.6%). Participants provided open-ended responses that identified the main character as Asian (83%) and man (100%).
Anger expression in emotion storyboard pilot study
University undergraduates in a psychology class participated in a pilot study, with a between-subjects design (N = 25; demographics not collected). Participants rated only the final panel, showing the anger expression of the character without text. The emotional intensity of a number of emotions was rated on a scale of Not at all (1) to Very (7). Piloting was conducted to ensure that the character was rated as more than moderately angry (indicated by a rating greater than 4 on a scale of Not angry at all (1) to Very angry (7); metric selected to account for perceptual overlap of facial emotion expressions when presented in a decontextualized manner [43]). For Study 1, participants who rated the White man main character (N = 7) perceived him as a good deal angry (M = 4.86, SD = 0.38), as somewhat disgusted (M = 2.86, SD = 1.46), somewhat sad (M = 2.43, SD = 1.13), slightly surprised (M = 2.14, SD = 1.07), and close to not at all afraid (M = 1.43, SD = 0.79) and to not at all happy (M = 1.29, SD = 0.49). For Study 2, participants who rated the Asian man main character (N = 5) perceived him as a good deal angry (M = 4.80, SD = 1.10), as moderately disgusted (M = 3.80, SD = 0.45), somewhat surprised (M = 3.60, SD = 1.67), and close to not at all sad (M = 1.40, SD = 0.55), to not at all afraid (M = 1.40, SD = 0.55) and to not at all happy (M = 1.20, SD = 0.45).
Methods: Study 1
Design
The design of the study was a 2 (format: emotion storyboard vs. vignette) X 2 (scenario: rumor vs. lunch stealing) between-subjects randomized design.
Participants
An a priori power analysis for between-subject group comparisons in G*Power [44] revealed a sample size of 171 people was needed to detect a moderate effect size (d = 0.50) at .90 power. The Institutional Review Board (IRB) at The Pennsylvania State University’s Office of Research Protections approved the study. Psychology undergraduates in the United States participated online, remotely through a university-hosted site for course credit. As approved by the IRB, participants read a consent form before the study began that included a closing message that stated, “Your participation implies your voluntary consent.” Due to the online nature of the study, participants did not sign the consent form but instead indicated their consent by advancing to the next page. All participants chose to continue to the online study, indicating their consent. Pre-participation instructions asked participants not to complete the study on a phone to ensure participants viewed stimuli similar in size. We recruited 203 participants to account for potential exclusions. The final sample had 194 participants (96 women and 98 men; ages 18–30, M = 19.11, SD = 1.53) after exclusions (6 removed for completing the study on a phone, 3 more removed for failing an attention check). The racial/ethnic composition of participants in the sample was predominantly White (71.7%; Asian/Asian-American, 12.4%; Latinx, 5.7%; multiracial, 4.1%; Black, 3.6%; Middle Eastern, 1.5%; Native Hawaiian or Pacific Islander, 0.5%; Native American/Alaskan, 0.5%; other, 0.5%).
Statistical reporting
For all data analyses, significance was determined using an alpha of p < .05. All analyses for pilot studies, Study 1, and Study 2, were conducted using SPSS Statistics [45].
Procedure
Participants completed the study online and were told the purpose of the study was to examine people’s thoughts about social situations in the workplace. Participants were told they would read a brief story about a main character named Andrew. The design of the study was between-subjects, in which participants read one of two scenarios, in either emotion storyboard or vignette form (see Fig 1). In the rumor scenario, participants read about coworkers Paul, Dan, and the main character, Andrew. Paul tells Dan a rumor about Andrew, Dan then tells Andrew the rumor, and Andrew later confronts Paul. In the lunch stealing scenario, participants read about coworkers Dan, Margaret, Paul, and the main character, Andrew. Andrew expresses to Dan, and later to Dan and Margaret that someone keeps stealing his lunch. Margaret suggests it may be Paul and Andrew confronts Paul. For the emotion storyboard conditions, instructions also included an illustration of Andrew. After reading a scenario, participants answered comprehension and emotion manipulation check questions, rated story clarity and immersion, and rated character emotional control and appropriateness of emotional intensity.
Measures
Participants responded to the following items (see S1 Appendix; scales created for this study unless noted).
Manipulation checks
Emotion type/intensity.
To ensure that the main character was judged as experiencing anger across format and scenario, we measured anger as a manipulation check. We also measured additional emotions (disgust, sadness, surprise, and fear) that could be relevant for the character in the scenarios. Five single-items assessed perceptions of how angry, sad, afraid, surprised, and disgusted the main character was when saying something to Paul (the wrongdoer) on 7-point Likert scales ranging from 1 (Not at all) to 7 (Very).
Dependent measures
Comprehension.
Comprehension was measured with two items, “Why was the main character angry?”; “Who did the main character confront?” (rs (192) = 0.29, p < .001). Each item had four possible multiple-choice responses, later coded as correct or incorrect. Comprehension scores represented the proportion of correct answers.
Clarity
Clarity was measured with four items on 7-point Likert scales ranging from 1 (Not at all) to 7 (Very), “How easy was it to follow what was happening between the characters in the story?”; “How clear was the story you read?”; “How confusing was the story you read?” (reverse-coded, R); “How difficult was it to follow what emotions the characters in the story were feeling?” (R) (Cronbach’s α = .86).
Immersion.
Immersion was measured with four items on 7-point Likert scales ranging from 1 (Not at all) to 7 (Very), “How absorbed in the story were you?”; “How interesting was the story you read?”; “How immersed in the story were you?”; “To what extent did you identify with any of the characters in the story?” (Cronbach’s α = .86).
Exploratory measures
Emotional control.
Emotional control was measured with an existing measure [17] of three items on 7-point Likert scales ranging from 1 (Not at all) to 7 (Very much), “How much self-control did the main character demonstrate?”; “How much were the main character’s feelings kept ‘in check’?”; “How much composure did the main character demonstrate?” (Cronbach’s α = .86).
Appropriateness of emotional intensity.
Appropriateness of emotional intensity was measured with an existing measure [37] of five items on 7-point Likert scales ranging from 1 (Strongly disagree) to 7 (Strongly agree), “The emotions shown by the main character were too extreme (R)”; “Most people would not have been so emotional at certain points as the main character was.: (R); “The main character was too emotional.” (R); “I think the main character was emotionally out of control” (R); “I think that the main character had too much emotion for clear thinking” (R) (Cronbach’s α = .89).
Results: Study 1
For the manipulation checks, dependent measures, and exploratory measures, we conducted 2 (format: emotion storyboard vs. vignette) X 2 (scenario: rumor vs. lunch stealing) between-subjects ANOVAs. To examine whether clarity or immersion mediated the effect of format (coded: 0 = vignette, 1 = emotion storyboard) on comprehension, we conducted two mediation analyses using the bootstrapping procedure, with 5,000 bootstrap samples and 95% confidence intervals, of the PROCESS macro for SPSS [46]. Correlations among variables are shown in Table 1. Means and standard deviations by format are listed in Table 2.
Manipulation checks
Emotion type/intensity.
For anger intensity, differences based on format were nonsignificant, p = .240, ηp 2 = .01, 95% CI [0.00 to 0.05] (emotion storyboards M = 5.66, SD = 1.26; vignettes M = 5.87, SD = 1.22). Differences based on scenario were nonsignificant (p = .400, ηp2 = .004, 95% CI [0.00 to 0.04]). Differences based on the interaction between format and scenario were also nonsignificant (p = .259, ηp2 = .01, 95% CI [0.00 to 0.05]).
For fear, disgust, surprise, and sadness, differences based on format were nonsignificant (see S2 Appendix for further reporting of manipulation check results for emotions other than anger). For fear, disgust, and surprise, differences based on the interaction of format and scenario were also nonsignificant. Perceptions of the main character’s sadness differed based on the interaction between format and scenario (F(1, 190) = 4.16, p = .043, ηp2 = .02, 95% CI [0.00 to 0.08]). Simple effects testing of the interaction revealed there was an effect (F(1, 190) = 13.93, p < .001, ηp2 = .07, 95% CI [0.02 to 0.15]) in the emotion storyboard format of the main character being perceived as sadder in the rumor situation (M = 3.42, SD = 1.68) than in the lunch stealing situation (M = 2.23, SD = 1.32).
Dependent measures
Comprehension.
There was an effect of format on comprehension, F(1, 190) = 4.38, p = .038, ηp2 = .02, 95% CI [0.00 to 0.08], such that participants’ comprehension was greater in emotion storyboard than vignette conditions (see Table 2 for means and standard deviations). Differences based on scenario were nonsignificant (p = .225, ηp2 = .01, 95% CI [0.00 to 0.05]). Differences based on the interaction between format and scenario were also nonsignificant (p = .324, ηp2 = .01, 95% CI [0.00 to 0.04]).
Clarity.
There was an effect of format on clarity, F(1, 190) = 12.80, p < .001, ηp2 = .06, 95% CI [0.01 to 0.14], such that participants rated the stories as clearer in emotion storyboard than vignette conditions (see Table 2). Differences based on scenario were nonsignificant (p = .637, ηp2 = .001, 95% CI [0.00 to 0.03]). Differences based on the interaction between format and scenario were also nonsignificant (p = .848, ηp2 = .00, 95% CI [0.00 to 0.01]).
Immersion.
Differences in immersion based on format were nonsignificant (p = .607, ηp2 = .001, 95% CI [0.00 to 0.03]; see Table 2). Differences in immersion based on scenario were nonsignificant, (p = .069, ηp2 = .02, 95% CI [0.00 to 0.07]). Differences based on the interaction between format and scenario were nonsignificant (p = .747, ηp2 = .001, 95% CI [0.00 to 0.02]).
Exploratory variables
Emotional control.
There was an effect of format on emotional control, F(1, 190) = 17.87, p < .001, ηp2 = .09, 95% CI [0.02 to 0.17], such that the main character was rated as higher in emotional control in emotion storyboard than vignette conditions (see Table 2). Differences based on scenario were nonsignificant (p = .229, ηp2 = .01, 95% CI [0.00 to 0.05]). Differences based on the interaction between format and scenario were also nonsignificant (p = .761, ηp2 = .001, 95% CI [0.00 to 0.02]).
Appropriateness of emotional intensity.
There was an effect of format on appropriateness of emotional intensity, F(1, 190) = 16.83, p < .001, ηp2 = .08, 95% CI [0.02 to 0.16], such that the main character’s emotional intensity was rated more appropriate in emotion storyboard than vignette conditions (see Table 2). There was also an effect of scenario on appropriateness of emotional intensity, F(1, 190) = 12.93, p < .001, ηp2 = .06, 95% CI [0.01 to 0.14], such that the character’s emotional intensity was rated more appropriate in the rumor situation (M = 5.23, SD = 1.04) than lunch stealing situation (M = 4.60, SD = 1.33). Differences based on the interaction between format and scenario were nonsignificant (p = .619, ηp2 = .001, 95% CI [0.00 to 0.03]).
Discussion: Study 1
Manipulation check findings suggest the main character’s anger when confronting the wrongdoer was rated equivalently across conditions. However, the unexpected interaction of the main character in the emotion storyboard being rated as sadder in the rumor than lunch-stealing scenario may suggest that visual depictions from previous panels in the emotion storyboard affected the judgment of the main character’s emotion when he was confronting the wrongdoer. Alternatively, this interaction may have been a Type I error due to the number of tests conducted.
Study 1 findings suggest the emotion storyboard method enhances comprehension and clarity, but not immersion, of story content relative to vignettes. Contrary to predictions, neither clarity nor immersion mediated the effect of format on comprehension. Comprehension was, however, weakly correlated with clarity and not associated with other measures (see Table 1). It remains possible, given comprehension in both conditions was relatively high, that clarity may mediate the emotion storyboard’s effectiveness on comprehension if comprehension questions were more difficult. Therefore, in Study 2, we examined the relationship between comprehension and clarity again for the two methods with three comprehension questions. One of the comprehension questions used in Study 1 about confrontation may have been ambiguous in the emotion storyboard conditions, given that the main character confronts several characters about the situation that evoked their anger, whereas only one interaction is labeled as confrontation in the vignette. In Study 2, we replaced this question with a more neutrally-worded question assessing how the character discovered who the wrongdoer was. We also added a question asking how many total characters were portrayed in the story.
Findings that emotion storyboards were clearer than vignettes may suggest that emotion storyboards are easier to process for participants. To examine this possibility in Study 2 we also compared participants’ overall time spent reading the story in the two format conditions. To explore the potential efficiency of the emotion storyboard method, we also examined the average response times of participants to the comprehension questions.
Additionally, participants evaluated the main character as expressing emotion with more emotional control and more appropriate emotion intensity in emotion storyboard than vignette conditions, suggesting that the emotion storyboard method allows for needed experimental control in depictions of expressions of emotion. To explore whether the two methods differ in broader emotion judgments, not specifically about emotion expression, in Study 2 we also examined evaluations of the main character’s appropriateness of emotion type for the given situation.
In Study 2, we also compared the efficacy of two methods to convey information about a character’s social group membership because such information is often manipulated in research on social judgments about emotion. To compare the methods on this dimension, we examined whether experimental manipulations of race in emotion storyboard resulted in greater accuracy on a race manipulation check compared to subtle name manipulations in vignettes.
Method: Study 2
We chose to use only the rumor scenario for Study 2 because it was the longer of the two scenarios from Study 1. We examined accuracy on a race manipulation check, comprehension, clarity, story reading time, average comprehension question response time, and appropriateness of emotion type experienced by the main character.
Design
The design of the study was a 2 (format: emotion storyboard vs. vignette) X 2 (character race: Asian man vs. White man) between-subjects randomized design.
Participants
An a priori power analysis for between-subject group comparisons in G*Power [44] revealed a sample size of 171 people needed to detect a moderate effect size (d = 0.50) at .90 power. The study was approved by the IRB at The Pennsylvania State University’s Office of Research Protections and involved the same process for consent as in Study 1. Workers on Amazon Mechanical Turk (MTurk) in the United States participated online. All participants chose to continue to the online study, indicating their consent. Pre-participation instructions asked participants not to complete the study on a phone to ensure participants viewed stimuli similar in size. We recruited 226 participants to account for potential exclusions. The final sample had 213 participants (96 women and 117 men; ages 18–68, M = 35.52, SD = 11.33) after exclusions (8 removed for completing the study on a phone, 5 more removed for failing an attention check). The racial/ethnic composition of participants in the sample was predominantly White (71.8%; Asian/Asian-American, 8.9%; Black, 8.9%; Latinx, 6.6%; multiracial, 0.9%; Middle Eastern, 0.5%; Native Hawaiian or Pacific Islander, 0.5%; Native American/Alaskan, 1.9%).
Procedure
As in Study 1, the design was between subjects. Participants read either an emotion storyboard or a vignette, with either an Asian man (Xiang) or a White man (Andrew) as the main character (see Fig 2). For the Asian man vignette condition, participants read about a main character named Xiang. For the Asian man emotion storyboard condition, participants read about and viewed a depiction of an Asian man character referred to by the name Xiang. We selected Anglo and Chinese names, Andrew and Xiang, respectively, that were used previously in vignette research [42]. In Study 2, only the rumor scenario from Study 1 was used. Participants read about coworkers Paul, Dan, and the main character, Xiang/Andrew. Paul tells Dan a rumor about Xiang/Andrew, Dan then tells Xiang/Andrew the rumor, and Xiang/Andrew later confronts Paul. For the emotion storyboard conditions, instructions also included an illustration of Xiang or Andrew. Participants completed the study online through MTurk and the procedure was the same as in Study 1.
Dependent measures
Race manipulation check.
The accuracy of the race manipulation check was measured with one, open-ended item at the end of the study, “What do you think was the race/ethnicity of the main character who became angry?” Each response was coded as correct or incorrect. Responses that identified the correct racial group or an ethnicity within the correct racial group (e.g., Chinese) were coded as correct. Responses that were incorrect, unsure, or did not provide a race were coded as incorrect. There were two responses, one in an emotion storyboard condition and one in a vignette condition, for which coding was less clear. For the emotion storyboard response, the participant provided two groups, one of which was the correct racial group. For the vignette response, the participant labeled the White main character as “Not Hispanic.” We opted to code both responses as correct. The pattern of results was unchanged when these two responses were coded as correct or as incorrect. All free-text responses to this measure are available in the Study 2 dataset.
Comprehension.
Comprehension was measured with three items, “Why was the main character angry?”; “How did the main character discover that it was Paul who spread a rumor about him?”; “How many characters were there in the story you read?” (Cronbach’s α = .63). Each item had four possible multiple-choice responses, later coded as correct or incorrect. Comprehension scores represented the proportion of correct answers.
Exploratory measures
Reading time.
Reading time was measured by calculating how long participants remained on the story page before clicking the arrow to advance the online survey.
Comprehension response time.
Comprehension response time was measured by calculating the average time that participants remained on the question page before clicking the arrow to advance to the next question, across the three comprehension questions.
Appropriateness of emotion type.
Appropriateness of the emotion type was measured with an existing measure [37] of four items on 7-point Likert scales ranging from 1 (Strongly disagree) to 7 (Strongly agree), “The main character’s emotions were exactly the kinds that were called for.”; “I think the types of emotions that the main character felt were normal.”; “The emotions displayed by the main character were wrong.” (R); “I would not have shown the types of emotions that the main character displayed.” (R) (Cronbach’s α = .80).
Results: Study 2
We conducted 2 (format: emotion storyboard vs. vignette) 2 (character race: Asian man vs. White man) between-subjects ANOVAs for all measures. To examine whether clarity mediated the effect of format (coded: 0 = vignette, 1 = emotion storyboard) on comprehension, we conducted a mediation analysis using the bootstrapping procedure, with 5,000 bootstrap samples and 95% confidence intervals, of the PROCESS macro for SPSS [46].
For the exploratory measure of reading time, participants with reading times two standard deviations above the average reading time across condition were excluded from the analysis (all five participants excluded were in vignette condition; no reading times were two standard deviations below the mean). Data were not normally distributed for overall reading time as assessed by Shapiro-Wilk’s test (for White man main character vignette condition p = .045; for all other conditions ps < .001). Neither a log transformation nor a square root transformation resulted in conditions with normally distributed data. Therefore, we conducted the ANOVA without a transformation because ANOVAs can be relatively robust with deviations from normality [47].
For comprehension response time, participants with an average response time two standard deviations above the average across condition were excluded from the analysis (seven participants in vignette condition, one participant in the emotion storyboard condition; no average response times were two standard deviations below the mean). Data were not normally distributed for comprehension response time as assessed by Shapiro-Wilk’s test (ps < .001 in the four cells). Neither a log transformation nor a square root transformation resulted in conditions with normally distributed data. Thus, we conducted the ANOVA without a transformation.
For all measures, correlations among variables are shown in Table 3. Means and standard deviations by format are listed in Tables 4 and 5.
Dependent measures
Race manipulation check.
Accuracy at identifying the main character’s race differed based on format, F(1, 209) = 5.99, p = .015, ηp2 = .03, 95% CI [0.001 to 0.08], such that participants provided the correct race of the character at a higher rate in emotion storyboard than vignette conditions (see Table 4 for means and standard deviations). Differences based on main character race were nonsignificant (p = .368, ηp2 = .004, 95% CI [0.00 to 0.04]). Differences based on the interaction between format and main character race were also nonsignificant (p = .153, ηp2 = .01, 95% CI [0.00 to 0.05]).
Comprehension.
Differences in comprehension based on format were nonsignificant, F(1, 209) = 0.07, p = .785, ηp2 = .00, 95% CI [0.00 to 0.01] (see Table 4). Differences based on main character race were nonsignificant (p = .535, ηp2 = .002, 95% CI [0.00 to 0.03]). Differences based on the interaction between format and main character race were also nonsignificant (p = .231, ηp2 = .01, 95% CI [0.00 to 0.05]).
Clarity.
Clarity differed based on format, F(1, 209) = 6.19, p = .014, ηp2 = .03, 95% CI [0.001 to 0.09], such that participants rated the stories as clearer in emotion storyboard than vignette conditions (see Table 4). Clarity also differed based on main character race F(1, 209) = 6.71, p = .010, ηp2 = .03, 95% CI [0.002 to 0.09], such that participants rated the stories as clearer when they read about an Asian man than a White man. Clarity differed based on the interaction between format and main character race as well, F(1, 209) = 4.56, p = .034, ηp2 = .02, 95% CI [0.00 to 0.07]. Simple effects testing of the interaction revealed there was an effect (F(1, 209) = 11.11, p = .001, ηp2 = .05, 95% CI [0.01 to 0.12]) in the vignette format of the stories being rated as clearer with the Asian man main character (M = 6.08, SD = 1.04) than with the White man main character (M = 5.28, SD = 1.61).
Exploratory measures
Reading time.
Differences in reading time based on format were nonsignificant, F(1, 204) = 2.43, p = .121, ηp2 = .01, 95% CI [0.00 to 0.06] (see Table 5 for means and standard deviations). Differences based on main character race were nonsignificant (p = .556, ηp2 = .002, 95% CI [0.00 to 0.03]). Differences based on the interaction between format and main character race were also nonsignificant (p = .663, ηp2 = .001, 95% CI [0.00 to 0.03]).
Comprehension response time.
Average comprehension response time differed based on format F(1, 201) = 5.03, p = .026, ηp2 = .02, 95% CI [0.00 to 0.08], such that participants responded to comprehension questions more quickly in emotion storyboard than vignette conditions (see Table 5). Differences based on main character race were nonsignificant (p = .454, ηp2 = .003, 95% CI [0.00 to 0.03]). Differences based on the interaction between format and main character race were also nonsignificant (p = .424, ηp2 = .003, 95% CI [0.00 to 0.04]).
Appropriateness of emotion type.
Differences in appropriateness of emotion type based on format were nonsignificant, F(1, 209) = 1.38, p = .241, ηp2 = .01, 95% CI [0.00 to 0.04] (see Table 4). Differences based on main character race were nonsignificant (p = .332, ηp2 = .01, 95% CI [0.00 to 0.04]). Differences based on the interaction between format and main character race were also nonsignificant (p = .119, ηp2 = .01, 95% CI [0.00 to 0.06]).
Discussion: Study 2
Study 2 findings suggest the emotion storyboard method does enhance the accuracy of participants’ performance on a race manipulation check relative to the vignette method. And, as in Study 1, emotion storyboards were rated as higher in clarity than vignettes. In contrast to Study 1, differences in comprehension scores for story content in Study 2 were nonsignificant across method. However, clarity was moderately correlated with comprehension in Study 2 (see Table 3) and the indirect mediation of clarity for format on comprehension was significant, though weak in effect, in Study 2. Overall, high comprehension and clarity ratings suggest effects of clarity and format on comprehension may be more pronounced if story content was more complex and comprehension questions were still more difficult.
Unexpectedly, for clarity, an interaction and main effect of main character race emerged that suggest, especially for participants reading vignettes, that reading about an Asian character resulted in higher perceptions of clarity than reading about a White character did. This finding could be due to our predominantly White participants being especially attentive upon encountering a novel name. Alternatively, this interaction may have been a Type I error due to the number of tests conducted.
Across methods, reading time was equivalent for the story content, however, average comprehension question response time was faster for participants who read the story in emotion storyboard than in vignette. Such findings suggest the emotion storyboard method is able to maximize efficiency without damaging comprehension performance.
Additionally, judgments about the appropriateness of emotion type did not differ across methods. Patterns suggest that both the emotion storyboard and vignette adequately conveyed that the situation was one in which anger would be an appropriate response for the main character.
Discussion: General
Overall, findings suggest the emotion storyboard is comparable to the vignette method, and has some advantages for assessing evaluations of third-party others’ emotion. Findings suggest emotion storyboards enhance clarity, and in some instances, comprehension and comprehension response time, relative to vignettes when conveying stories of the same complexity. Although effect sizes are small to medium for comprehension, comprehension response time, and clarity, and thus for some of the measures somewhat lower than our sample had adequate power to detect, even small gains on these dimensions could be meaningful for enhancing presentation to participants of information pertaining to social elements of emotion and social context. Findings are in line with effects found in studies comparing graphic novels to text in other disciplines [30, 35].
The emotion storyboard method also facilitated greater accuracy on a race manipulation check than the vignette method. On a practical level, using the emotion storyboard to manipulate race would limit the number of participants excluded for failing such a manipulation check (14% as opposed to 27%). Findings suggest emotion storyboards offer a method that can enhance participants’ accuracy when identifying social group memberships (e.g., race; socioeconomic status; gender) in studies on social judgments of emotion, relative to vignette manipulations. As with any method, researchers should be attentive when using the emotion storyboard that they are not relying on stereotypical depictions that reinforce essentialist thinking about social group members. One way the emotion storyboard method is able to do so is through its ability to control for features that are confounded with emotional expressions, such as gender and facial dominance cues [39].
Contrary to predictions, emotion storyboards and vignettes did not differ on immersion. Immersion ratings were around the mid-point of the scale for both emotion storyboards and vignettes, indicating that participants found the brief nature of each only moderately immersive. This finding reflects discussion [48] of the difficulties of fully immersing participants in vignettes without engaging all of people’s senses. Stronger effects of immersion and interest, however, might emerge in longer and more complex narratives. Indeed, research comparing longer graphic novel selections to text selections found graphic novels resulted in greater interest in learning more about the focal topic [49], and comparison of graphic novels to traditional novels found participants reported greater interest and enjoyment for the graphic novel than traditional novel [50].
Clarity explained effects of the emotion storyboard on comprehension inconsistently across studies. Future research should test other possible mechanisms that may explain the emotion storyboard’s enhancing effect on comprehension and comprehension response time. For instance, multimedia formats that fuse words and pictures can enhance learning and reduce cognitive load [51]. As others have discussed [51], research on the dual-code theory [52] suggests that people have two separate channels for processing verbal/auditory and visual/pictorial information, each of the channels has a limited capacity [53], and meaningful learning requires cognitive processing on several levels, for example through attention, organization, and integration [54]. One possibility is that emotion storyboards result in greater comprehension or faster response times on comprehension questions than vignettes, due to the multimodal nature of their design. Emotion storyboards could result in processing content through both channels, while, relative to emotion storyboards, vignettes could overload the verbal/auditory channel. In line with this possibility, second language learners had greater comprehension after reading graphic novels than text alone, irrespective of their pre-existing English language reading abilities or cognitive learning styles [49]. We did not find, however, that participants read the story more quickly in emotion storyboard format than vignette format. Such a finding might emerge, due to for instance reduced cognitive load, in a test with longer stories.
Another possibility is that the mutually-reinforcing combination of visual images and text in the emotion storyboard aids comprehension and comprehension response time. For instance, related pictures and text promoted greater attention and recall of health behaviors than text only [55]. Similarly, children who viewed a congruent picture rather than an incongruent picture while hearing a story narration had better comprehension of story details [56]. Eye-tracking data with these children revealed that they processed the congruent picture in a manner that maximally integrated narration and image. Additionally, the spatial organization of the emotion storyboard resembles a diagram and such a formation may enhance problem-solving through organization of information in spatially adjacent and logical locations [57]. The mutually-reinforcing nature of the emotion storyboard may also enhance participants’ confidence in their comprehension as evidenced by quicker response times to comprehension questions in emotion storyboard conditions, without sacrificing accuracy [58]. Future research could examine the deliberation process involved in responding to such questions using methods such as cursor-tracking [59].
It also remains possible, given comprehension and comprehension response times in both conditions were relatively high, that clarity may most robustly mediate the emotion storyboard’s effectiveness on such outcomes if comprehension questions were more difficult, if questions had free-recall rather than forced choice options, or if time was extended between reading the stories and answering comprehension questions. Additionally, power analyses were conducted for between-subjects effects rather than for mediation effects. Another possibility is that clarity and comprehension could be further enhanced in emotion storyboards relative to vignettes if the characters were depicted as more visually distinct from one another. In both studies, depictions of the characters’ clothing and physical features looked relatively similar. Future studies using emotion storyboards could pilot and use characters that have been rated as visually distinct from one another.
In vignettes, clarity was enhanced for participants who read about the Asian character as opposed to the White character. This unexpected finding could suggest that a feature of the vignettes, such as encountering a name that was potentially more novel to our predominately White participants (“Xiang” rather than “Andrew”), may have made participants especially attentive to the story and thereby affected perceptions of story clarity.
Findings suggest both emotion storyboards and vignettes are effective at conveying anger at a particular intensity and at conveying emotion as appropriate for a given situation. Although participants who read vignettes envisioned the anger of the main character as similarly appropriate in emotion type to those who read emotion storyboards, participants who read vignettes rated the character’s anger as less controlled and less appropriately intense than those who read emotion storyboards. These findings reveal when examining social judgments of emotion, subtle changes in nonverbal depictions can affect judgments. The unexpected interaction that emerged for the intensity of sadness manipulation check in the emotion storyboard conditions also highlights such effects. When judging the main character’s emotion in Study 1 when he confronted the rumor spreader, but not the lunch thief, participants who read the emotion storyboard made a more holistic judgment of that confrontation by including sadness they had seen the character express in a previous panel. Future research should meticulously examine visual and written cues about context and emotion that may be embedded in stimuli and may differentially affect such judgments relevant to emotion expression. These findings, in concert with greater clarity, greater accuracy of character race, and in some instances, enhanced comprehension and comprehension response time, for emotion storyboards than vignettes, suggest promise of the emotion storyboard to provide more emotional nuance through nonverbal depictions than vignettes may be able to feasibly include without sacrificing comprehension. For measures where effects of emotion storyboards and vignettes were not different, such as immersion, vignettes did not demonstrate an advantage over emotion storyboards. Thus, results suggest emotion storyboards are comparable to vignettes on a number of indicators and have advantages to vignettes on some indicators.
Limitations and future directions
One limitation is that we only tested anger within the workplace context. Future directions should examine the effectiveness of emotion storyboards with other emotions, other contexts, other scenarios, and main characters with other social group memberships. Another limitation is that we tested some variables (e.g., scenario, race manipulation check) in only one study, which prevented us from testing the replication of their effects. Also, instead of a design comparing a “typical” narrative vignette and emotion storyboard, we could have created a vignette by verbally describing each of the illustrations in the emotion storyboard. Indeed, a future study could focus on specific written descriptions of the main character’s emotional expression (e.g., “scowling”, “arms crossed”) instead of descriptions of their emotion experience (e.g., “angrily”). Future research could examine cognitive processing constraints and other measures of relative effectiveness between emotion storyboards and vignettes as the level of written detail of the emotional expression is increased.
Additionally, we examined social interactions and emotional exchanges at only one level of complexity. Future research should compare the emotion storyboard and vignette methods with increasing complexity. To be sure, there may be elements of story complexity that might be more effectively conveyed in written vignette than in emotion storyboard. However, there are also elements of story complexity, such as time shifts or representation of multiple perspectives, that might be more rapidly and effectively conveyed through emotion storyboard than written vignette. Given potential cognitive processing constraints of providing thorough information about the context and other social actors [60], which may add both length and complexity to written text, it is possible the emotion storyboard method may be especially useful in instances in which story content is high in complexity. Indeed, guidelines for researchers using vignettes recommend minimizing vignette complexity as a design feature that should guide research practice [61]. One researcher suggested that vignettes should make three or fewer changes to the story of a vignette to prevent participant confusion [62]. Our findings of greater clarity and, although less consistent, findings of enhanced comprehension and comprehension response time for emotion storyboards than vignettes in these relatively simple social interactions, suggest that the emotion storyboard method may be even more effective than vignettes in depicting complex, multi-person emotion exchanges. If so, the emotion storyboard method could allow researchers the opportunity to examine questions that may have previously been avoided with written vignettes due to complexity constraints.
Another limitation of emotion storyboards is the requirement of a graphic artist for development. Although the cost of employing a graphic artist is likely less than the cost of producing high-quality videos [48], the cost of the method is higher than that of written vignette. However, once developed the illustrations can be easily customized and adapted to depict other contexts and emotional qualities using computer software without additional cost. Researchers interested in finding a graphic artist to develop emotion storyboards could advertise with their university or college art departments to find interested art students to work with. Alternatively, researchers could contract graphic artists for freelance work from online platforms.
A strength of emotion storyboards is that they enable researchers to selectively include and emphasize particular features of the social context. For instance, reactions of other characters to a main character expressing emotion can be easily included or excluded in the emotion storyboard according to the researcher’s goals. Specific content can be emphasized relative to other content through enlargement of a certain panel or through allocating multiple panels to a particular concept. In contrast, researchers using vignettes may have difficulty controlling which specific features of the social context participants attend to during the reading process. Future research is needed to examine how participants process content in emotion storyboards, using methods such as eye-tracking to connect basic-level processing to attention, comprehension, deliberation, and other relevant outcomes.
The emotion storyboard also provides researchers with strong experimental control of nonverbal elements in the social context. On the macro level, researchers can easily control specific nonverbal emotion information participants see and do not see, such as a character’s neutral expression prior to an emotion-relevant event. Across domains in social psychology, the emotion storyboard method offers researchers a way to easily convey and control complex social exchanges and story lines visually that may be cumbersome and confusing to convey via written text alone. On the micro level, the emotion storyboard method allows researchers to visually depict tone in emotion. Artistic techniques in graphic novels such as steam coming out of the ears of a character experiencing anger or sweat droplets depicted on a character experiencing nervousness can be used as metaphorical depictions to communicate emotion-relevant features, such as intensity of emotion. Although metaphorical and other descriptions of emotional tone can be similarly used in written form, a visual depiction may be a more efficient method than written vignette (as indicated, for example, by enhanced comprehension and comprehension response time) to convey particular concepts. In the emotion storyboards used in the present studies, sharing the rumor while covering one’s face, for instance, may have communicated additional felt emotion of the sharer, such as guilt. The act of covering one’s mouth with a hand may have also conveyed the manner in which the information was shared, for instance quietly. Future directions should establish and examine elements of emotional tone in visual depictions of social emotion.
Emotion storyboards may also offer a less intrusive way than other methods to examine how social group memberships affect emotion judgments. In vignettes it can be both difficult to represent various social group memberships without being too subtle (e.g., using names stereotypical of particular social groups) and without eliciting reactance from participants by being too explicit (e.g., “William, a Black man…”). Videos and photographs, unless digitally altered, are affected by confounds between gendered and racialized facial features and emotion expression [39]. By controlling for these features, emotion storyboards offer a more direct way to examine in future studies how stereotypes affect perception and evaluation of emotional behavior. Future directions could compare the effects of social group membership when manipulated through the emotion storyboard and vignette on judgments prone to stereotypes, such as perceptions of having an emotional disposition [38]. Future research could also compare the effects of controlled facial features in emotion storyboards to naturally occurring facial features in pictures or videos that affect judgments of emotion, such as affiliation and dominance perceptions [39]. Additionally, future research could use the emotion storyboard method to examine questions about cultural effects on social judgments of emotion, such as perceptions of emotion intensity [63, 64].
Additional future directions for the emotion storyboard method could include comparing the method’s effectiveness to other methods used to examine social judgments of emotion. Other such methods include videos, such as Movie for the Assessment of Social Cognition [65], and virtual reality techniques, such as Virtual Assessment of Mentalising Ability [66]. Further, the emotion storyboard method may have applicability beyond the study of social judgments of emotion. Future studies could examine the relative effectiveness of using emotion storyboards in other domains that commonly use vignettes, such as in tests of emotional intelligence and situational judgment tests used for personnel selection [e.g., 67, 68].
Conclusions
Using a format inspired by graphic novels, the emotion storyboard method offers a useful way to examine social judgments of emotion. The emotion storyboard is comparable to the vignette method, with the advantage of enhancing clarity, accuracy of manipulation checks for social group memberships such as character race, and in some instances, comprehension and comprehension response time. The method enables researchers to visually convey, control, and emphasize elements of complex social exchanges. It further allows for visual inclusion of emotional nuance and emotional tone through nonverbal depictions. The method has implications for the study of social phenomena, such as stereotyping research.
Acknowledgments
We thank Michael Przybys for creating the illustrations and John McCormick-Huhn for feedback on an early draft of the manuscript. Part of this research was conducted while author Kaitlin McCormick-Huhn was at The Pennsylvania State University.
References
- 1. Pons F, Harris PL, Rosnay MD. Emotion comprehension between 3 and 11 years: Developmental periods and hierarchical organization. Eur J Dev Psychol. 2004;1(2):127–52.
- 2. Shimada H, Kitajima M. Why do illustrations promote comprehension of manuals? Effects of motivation and elaboration. Jpn J Educ Psychol. 2008;56:474–486.
- 3. Hareli S, Rafaeli A. Emotion cycles: On the social influence of emotion in organizations. Res Organ Behav. 2008;28:35–59.
- 4. Parkinson B. Emotions are social. Brit J Psychol. 1996;87(4):663–83. pmid:8962482
- 5.
Parkinson B, Fischer AH, Manstead AS. Emotion in social relations. Cultural, group, and interpersonal processes. New York: Psychology Press; 2005.
- 6. Van Kleef GA. How emotions regulate social life: The emotions as social information (EASI) model. Curr Dir Psychol Sci. 2009;18(3):184–8.
- 7.
Manstead AS, Fischer AH. Social appraisal. In: Scherer KR, Schorr A, Johnstone T, editors. Appraisal processes in emotion: Theory, research, application. New York: Oxford University Press; 2001. pp. 221–232. https://doi.org/10.1037/1528-3542.1.1.51 pmid:12894811
- 8. Hareli S, David S. The effect of reactive emotions expressed in response to another’s anger on inferences of social power. Emotion. 2017;17:717–427. pmid:28080088
- 9. Hareli S, Zohar E, David S, Lasalle M, Hess U. Seeing what you ought to see: The role of contextual factors in the social perception of achievement emotions. Motiv Emotion. 2014;38(4):600–8.
- 10. Mumenthaler C, Sander D. Social appraisal influences recognition of emotions. J Pers Soc Psychol. 2012;102(6):1118–35. pmid:22288528
- 11. Mumenthaler C, Sander D. Automatic integration of social information in emotion recognition. J Exp Psychol Gen. 2015;144(2):392–9. pmid:25688908
- 12. Van Der Schalk J, Hawk ST, Fischer AH, Doosje B. Moving faces, looking places: Validation of the Amsterdam Dynamic Facial Expression Set (ADFES). Emotion. 2011;11(4):907–20. pmid:21859206
- 13. Grandey AA, Fisk GM, Mattila AS, Jansen KJ, Sideman LA. Is “service with a smile” enough? Authenticity of positive displays during service encounters. Organ Behav Hum Dec. 2005;96(1):38–55.
- 14. Alexander CS, Becker HJ. The use of vignettes in survey research. Public Opin Quart. 1978;42(1):93.
- 15. Hughes R. Considering the vignette technique and its application to a study of drug injecting and HIV risk and safer behaviour. Sociol Health Ill. 1998;20(3):381–400.
- 16.
Parrott WG, Hertel P. Research methods in cognition and emotion. In: Daleigsh T, Power T, editors. Handbook of cognition and emotion. West Sussex: Wiley; 1999. pp. 3–19.
- 17. Zawadzki MJ, Warner LR, Shields SA. Sadness is believed to signal competence when displayed with passionate restraint. Soc Psychol. 2013;44(3):219–30.
- 18. Wong YJ, Steinfeldt JA, Lafollette JR, Tsao S-C. Mens tears: Football players evaluations of crying behavior. Psychol Men Masculin. 2011;12(4):297–310.
- 19. Tiedens LZ, Ellsworth PC, Mesquita B. Sentimental stereotypes: Emotional expectations for high-and low-status group members. Pers Soc Psychol B. 2000;26(5):560–75.
- 20. Collett JL, Childs E. Minding the gap: Meaning, affect, and the potential shortcomings of vignettes. Soc Sci Res. 2011;40(2):513–22.
- 21. Parkinson B, Manstead ASR. Making sense of emotion in stories and social life. Cognition Emotion. 1993;7(3–4):295–323.
- 22. Larsen JT, To YM, Fireman G. Children’s understanding and experience of mixed emotions. Psychol Sci. 2007;18(2):186–91. pmid:17425541
- 23.
Shields S. A. & Stern R. M. Emotion: The perception of bodily change. In: Spiegel IM, Pliner P, Blankstein KR, editors. Advances in the study of communication and affect. New York: Plenum Press; 1979. Vol. 5.
- 24. Silver M, Oakes P. Evaluation of a new computer intervention to teach people with autism or asperger syndrome to recognize and predict emotions in others. Autism. 2001;5(3):299–316. pmid:11708589
- 25. Gallagher H, Happé F, Brunswick N, Fletcher P, Frith U, Frith C. Reading the mind in cartoons and stories: An fMRI study of ‘theory of mind’ in verbal and nonverbal tasks. Neuropsychologia. 2000;38(1):11–21. pmid:10617288
- 26. Masuda T, Ellsworth PC, Mesquita B, Leu J, Tanida S, Van de Veerdonk E. Placing the face in context: Cultural differences in the perception of facial emotion. J Pers Soc Psychol. 2008;94: 365–381. pmid:18284287
- 27.
Lopes P. Demanding respect: The evolution of the American comic book. Philadelphia: Temple University Press; 2009.
- 28.
Czerwiec M, Williams I, Squier SM, Green MJ, Myers KR, Smith ST. Graphic medicine manifesto (Vol. 1). University Park: Penn State Press; 2015.
- 29. Olmstead T, Green MJ, editors. Graphic medicine and health care ethics [Special issue]. Am Med Assoc J Ethics. 2018;20(2).
- 30. Short JC, Randolph-Seng B, Mckenny AF. Graphic presentation: An empirical examination of the graphic novel approach to communicate business concepts. Bus Commun Q. 2013;76(3):273–303.
- 31. Cromer M, Clark P. Getting graphic with the past: Graphic novels and the teaching of history. Theor Res Soc Educ. 2007;35(4):574–91.
- 32.
Bakis M. The graphic novel classroom: Powerful teaching and learning with images. Thousand Oaks: Corwin Press; 2011.
- 33.
Brozo WG, Moorman G, Meyer C. Wham! Teaching with graphic novels across the curriculum. New York: Teachers College Press; 2013.
- 34. Junhasavasdikul D, Sukhato K, Srisangkaew S, Theera-Ampornpunt N, Anothaisintawee T, Dellow A. Cartoon versus traditional self-study handouts for medical students: CARTOON randomized controlled trial. Med Teach. 2017;39(8):836–43. pmid:28539065
- 35. Nalu A, Bliss JP. Comics as a cognitive training medium for expert decision making. In Hum Fac Erg Soc P. Sep 2011;55(1):2123–2127. Los Angeles: SAGE Publications.
- 36. Hochschild AR. Emotion work, feeling rules, and social structure. Am J Sociol. 1979;85(3):551–75.
- 37. Warner LR, Shields SA. Judgements of others’ emotional appropriateness are multidimensional. Cognition Emotion. 2009;23(5):876–88.
- 38. Brescoll VL, Uhlmann EL. Can an angry woman get ahead? Status conferral, gender, and expression of emotion in the workplace. Psychol Sci. 2008;19:268–275. pmid:18315800
- 39. Adams RB, Hess U, Kleck RE. The intersection of gender-related facial appearance and facial displays of emotion. Emot Rev. 2015;7(1):5–13.
- 40. McLaughlin J. Ever Eaten a Reality Sandwich or Actually Put Your Ducks in a Row? Of Course Not. American Express OPEN Wonders Why We Keep Talking That Way at Work. Businesswire. 2017 June 15 [Cited 2020 December 23]. Available from: https://www.businesswire.com/news/home/20170615005558/en/Eaten-Reality-Sandwich-Put-Ducks-Row-Not.
- 41. Daily Mail Reporter. Office niggles that drive us to ’desk rage’: Rude clients and computer crashes top list of things that make workers angry. Daily Mail. 2013 August 19 [Cited 2020 December 23]. Available from: https://www.dailymail.co.uk/news/article-2397643/Office-niggles-drive-desk-rage-Rude-clients-crashes-list-things-make-workers-angry.html.
- 42. Zhao X, Biernat M. “Welcome to the US” but “change your name”? Adopting Anglo names and discrimination. J Exp Soc Psychol. 2017;70:59–68.
- 43. Aviezer H, Hassin RR, Ryan J, Grady C, Susskind J, Anderson A, et al. Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychol Sci. 2008;19(7):724–32. pmid:18727789
- 44. Faul F, Erdfelder E, Lang AG, Buchner A. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007;39(2):175–91. pmid:17695343
- 45.
Corp IBM. IBM SPSS statistics for Macintosh. Version 20.0. Armonk: IBM Corp; 2011.
- 46.
Hayes AF. Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. Guilford Publications; 2017.
- 47. Blanca MJ, Alarcón R, Arnau J, Bono R, Bendayan R. Non-normal data: Is ANOVA still a valid option?. Psicothema. 2017; 29(4): 552–7. pmid:29048317
- 48. Aguinis H, Bradley KJ. Best practice recommendations for designing and implementing experimental vignette methodology studies. Organ Res Methods. 2014;17(4):351–71.
- 49. Wong SWL, Miao H, Cheng RW-Y, Yip MCW. Graphic novel comprehension among learners with differential cognitive styles and reading abilities. Read Writ Q. 2016;33(5):412–27.
- 50. Jennings KA, Rule AC, Vander Zanden SM. Fifth graders’ enjoyment, interest, and comprehension of graphic novels compared to heavily-illustrated and traditional novels. Int Elect J Elem Educ. 2014;6:257–274.
- 51. Mayer RE, Moreno R. Nine ways to reduce cognitive load in multimedia learning. Educ Psychol. 2003;38(1):43–52.
- 52. Paivio A, Csapo K. Picture superiority in free recall: Imagery or dual coding? Cognitive Psychol. 1973;5(2):176–206.
- 53. Sweller J. Instructional design. Aus Educ Rev. 1999.
- 54.
Mayer RE. The promise of educational psychology: Vol. 2, Teaching for meaningful learning.2 Upper Saddle River: Prentice Hall; 2002.
- 55. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: A review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61(2):173–90. pmid:16122896
- 56. Takacs ZK, Bus AG. How pictures in picture storybooks support young children’s story comprehension: An eye-tracking experiment. J Exp Child Psychol. 2018;174:1–12. pmid:29857060
- 57. Larkin JH, Simon HA. Why a diagram is (sometimes) worth ten thousand words. Cognitive Sci. 1987;11:65–100.
- 58. Papesh MH, Goldinger SD. Memory in motion: Movement dynamics reveal memory strength. Psychon B Rev. 2012;19(5):906–13. pmid:22711579
- 59. Hehman E, Stolier RM, Freeman JB. Advanced mouse-tracking analytic techniques for enhancing psychological science. Group Process Interg. 2014;18(3):384–401.
- 60.
Sweller J. Cognitive load theory, learning difficulty, and instructional design. Learn Instr. 1994;4(4):295–312. https://doi.org/10.1016/0959-4752(94)90003-5
- 61. Finch J. The vignette technique in survey research. Sociology. 1987; 21(1):105–14.
- 62. Barter C, Renold E. The use of vignettes in qualitative research. Soc Res Updat. 1999;25(9):1–6.
- 63. Kleinsmith A, De Silva PR, Bianchi-Berthouze N. Cross-cultural differences in recognizing affect from body posture. Interact Comput. 2006;18(6):1371–89.
- 64. Matsumoto D, Ekman P. American-Japanese cultural differences in intensity ratings of facial expressions of emotion. Motiv Emotion. 1989;13(2):143–57.
- 65. Dziobek I, Fleck S, Kalbe E, Rogers K, Hassenstab J, Brand M, et al. (2006). Introducing MASC: a movie for the assessment of social cognition. J Autism Dev Disord. 2006; 36(5): 623–36. pmid:16755332
- 66. Canty AL, Neumann DL, Fleming J, Shum DH. Evaluation of a newly developed measure of theory of mind: The virtual assessment of mentalising ability. Neuropsychol Rehabil. 2017; 27(5): 834–70. pmid:26095322
- 67. Motowidlo S. J., Dunnette M. D., & Carter G. W. (1990). An alternative selection procedure: The low-fidelity simulation. J Appl Psychol. 1990; 75(6): 640–7.
- 68. Lievens F, Sackett PR. Video-based versus written situational judgment tests: A comparison in terms of predictive validity. J Appl Psychol. 2006; 91(5): 1181–8. pmid:16953779