Theories of emotion

The two approaches to the analysis of emotion are the dimensional theory of emotion and the theory of basic emotions [22]. The difference between these two approaches is that emotions are either described as independent dimensions [23] or discrete entities [24]. In the dimensional approach, Russell (1980) [23] proposed a circlex model of emotion, which showed that each emotion could be arranged in a circle controlled by two orthogonal dimensions in space: valence and arousal [25–28]. The position of each emotion on the quadrant reflects different amounts of valence and arousal traits [27, 29]. The valence dimension is associated with a person’s subjective feeling, ranging from displeasure to pleasure. The arousal dimension is associated with the energy of a person’s subjective feeling, ranging from sleep to excitement [28].

The theory of basic emotions suggests that human emotions are composed of a limited number of basic emotions [30]. Each basic emotion has its proprietary neural circuits which are structurally different [24, 25, 31, 32]. Although the idea of basic emotions is commonly accepted, there is no consensus on the exact number of basic emotions. Plutchik (1962) [33] proposed eight primary emotions (anger, fear, sadness, disgust, surprise, anticipation, trust, and joy). Ekman (1992) [24, 34] proposed seven basic emotions (fear, anger, joy, sad, contempt, disgust, and surprise), but later changed to six (happiness, anger, sadness, fear, disgust, and surprise). Izard [35] proposed seven basic emotions (fear, anger, happiness, sadness, disgust, interest, and contempt). Recent studies examining facial expressions, neural mechanisms, and brain imaging suggest that the number of basic emotions could be further reduced to four (fear, anger, joy, and sadness) [36–40]. As an exploratory study of the tone-emotion relationship, we adopt the framework of four basic emotions (anger, fear, happiness, and sadness) in the current study.

How emotions affect speech acoustics

There is ample evidence that different emotions result in distinct acoustic characteristics [2, 5–7, 41–54]. Physiologically, the sympathetic nervous system is aroused by emotions such as anger, fear, or happiness, resulting in a higher heart rate and blood pressure, a dry mouth, and occasionally muscle tremors [55, 56]. Consequently, speech is loud, fast, and has intense high-frequency energy. On the other hand, sadness arouses the parasympathetic nervous system. Heart rate and blood pressure decrease, salivation increases, and speech is produced slowly and with little high-frequency energy. These physiological changes are reflected in amplitude, energy distribution across the frequency spectrum, frequency of pauses, and duration. For example, higher arousal associated with excitement, fear, and anger have been shown to generate higher mean F0 [7, 9, 57], higher mean amplitude [58–61], and shorter duration [56].

The influence of specific emotions on speech acoustics varies across studies. When compared with a neutral tone of voice, an angry voice has a higher mean F0, a wider or similar F0 range, a higher mean amplitude, and a shorter duration [2, 5–7, 41–51, 54]. A fearful voice shows a higher mean F0, a narrower, wider, or similar F0 range, a higher or lower mean amplitude, and a shorter duration [2, 5–7, 41–51, 54]. A happy voice has a higher mean F0, a wider or similar F0 range, a higher or equal mean amplitude, and a shorter or longer duration [2, 5–7, 41–54]. A sad voice has a lower or similar mean F0, a narrower or wider or similar F0 range, a lower mean amplitude, and a longer duration [2, 5–7, 41–54]. In sum, some emotions have fairly consistent acoustic features, whereas other emotions are more variable. The variability, however, is consistent with the idea that emotion is sociocultural in nature, i.e., there are cross-linguistic and cross-cultural differences in the acoustic manifestation of emotions [62]. The variability is also consistent with the observation that features of emotions vary across speakers, sexes, and contexts [63].

Several studies compared the acoustic characteristics of emotions between tonal and non-tonal languages. Ross, Edmondson, and Seibert (1986) [64] examined acoustic characteristics of neutral, happy, sad, angry, and surprising emotions using Thai, Taiwanese, Mandarin (all tonal languages), and English (a non-tonal language). They found greater F0 variations in English compared to the tonal languages, suggesting that non-tonal languages have a greater degree of freedom in using F0 to convey emotions. In contrast, no significant difference was found in duration or amplitude between the tonal and non-tonal languages.

Anolli, Wang, Mantovani, and De Toni (2008) [65] investigated acoustic differences among happy, sad, angry, fear, scornful, prideful, guilty, and shameful emotions in Mandarin and Italian (a non-tonal language). They found that emotions were characterized by significant variations in F0 and amplitude for Italian but not for Mandarin. In contrast, duration varied significantly among the emotions for Mandarin but not for Italian. Since Italian is a syllable-timed language, it is also likely that the less variation of syllable duration in Italian reflects the language-specific prosodic structure.

Wang, Lee, and Ma (2016, 2018) [46, 66] examined acoustic correlates of angry, fear, happy, sad, and neutral emotions in Mandarin and English. Semantically-neutral declarative sentences were embedded in different contexts to elicit angry, fear, happy, sad, and neutral emotions. Comparable English sentences were constructed with a direct translation of the Mandarin sentences. Acoustic analysis showed that F0 variations among the emotions were significantly greater in English than in Mandarin. In contrast, duration variations were significantly greater in Mandarin than in English.

Studies using other tonal languages also show more restricted F0 variations for emotions in a tonal language (Chong, Kim, and Davis, 2015 [67] for Cantonese; Luksaneeyanawin, 1998 [68] for Thai). To our knowledge, the only exception to this pattern is Li, Jia, Fang, and Dang’s (2013) [69], who showed greater F0 variations associated with emotions in Mandarin compared to Japanese, which is a non-tonal language that uses lexical pitch accent extensively.

Regarding the effect of emotions on specific lexical tones, Chao (1933) [70] noted that Mandarin uses successive additional tones and edge tones to implemet the intonation for emotions (see Liang and Chen, 2019 [71], for an illustration). Li, Fang, and Dang (2011) [44] examined how emotions affect the F0 and duration of Mandarin utterances ranging from one to fourteen syllables. The results showed that anger and disgust were associated with an additional falling tone, and happiness and surprise were associated with an additional rising tone. Non-neutral emotions resulted in a different F0 range, register, contour, or duration. For example, happiness and surprise were associated with a higher F0 range and higher register, whereas sadness and disgust were associated with a reduced F0 range and lower register.

In sum, most studies comparing tonal and non-tonal languages show that F0 variations associated with emotions are greater in non-tonal languages. This suggests that lexical tones constrain the availability of F0 for emotions in tonal languages. In contrast, amplitude or duration variations associated with emotions appear to be greater in tonal languages [14, 64, 65], suggesting that amplitude or duration may be used to compensate for the restricted use of F0 in conveying emotions. Studies on Mandarin further showed that emotions shape F0 and duration characteristics of Mandarin tones.

How emotions affect speech perception

The speech perception literature shows that emotions affect speech perception at various levels of processing. Mullennix, Bihon, Bricklemyer, Gaston, and Keener (2002) [1] examined how variations in emotions and talker voice affect spoken word recognition in English. They presented pairs of names (e.g., Todd-Tom) produced by either the same or different talkers, and with the same or different emotions. The participants’ task was to judge whether the names in a pair were the same or different. The results showed that variations in emotion slowed down judgments of both the names and talker voices, indicating emotion affected perception of consonants and talker characteristics. Kitayama and Ishii (2002) [72] and Ishii et al. (2003) [73] presented words spoken in a pleasant or unpleasant tone of voice. While ignoring the emotional tone, listeners were asked to judge whether the word meaning was pleasant (e.g., grateful, satisfaction) or unpleasant (e.g., complaint, dislike). The results showed that emotion variations slowed down judgments of word meaning. Nygaard and Lunders (2002) [52] examined how emotions (happy, neutral, and sad) affect the perception of homophonic words (e.g., die/dye). They found that selection of word meaning was compromised by the emotion of the words. Nygaard and Queen (2008) [53] presented happy (e.g., cheer), sad (e.g., upset), or neutral (e.g., chair) words spoken with consistent, inconsistent, or neutral emotions. Listeners were asked to repeat the words they heard. The results showed that listeners responded more quickly when the meaning of the words matched the emotions.

Similarly, research on tonal languages show the impact of emotions on speech perception, and the effect is further modulated by tonal language experience. Singh, Lee, and Goh (2016) [74] examined how changes in emotion and Mandarin tone affect consonant recognition, and how consonant changes affect emotion recognition and Mandarin tone identification. For Mandarin-speaking listeners, variations in Mandarin tone and emotion made consonant recognition less accurate. Consonant variations also made Mandarin tone identification and emotion recognition less accurate. Consonants and prosody (Mandarin tone and emotion) affect each other to the same extent. In contrast, for English-speaking listeners, consonant recognition was affected by prosodic variation to a different degree than prosody recognition was affected by consonant variations. That is, the effects of emotions and lexical tones on segmental perception depend on tonal language experience.

Liang and Chen (2019) [71] examined how emotions and tonal language experience affect Mandarin tone perception. Four Chinese pseudo words (i.e., mong, ging, ra, bü) were created, and each had four lexical tones variations. Each syllable-tone combination was embedded in the middle of carrier phrases. The syllable immediately before the pseudo words was manipulated to create four tonal contexts (i.e., chu1, du2, xie3, lian4). For instance, mong1 was embedded in carrier phrases (1) zhi3 chu1 [mong1] zhe4ge0 zi4 “Please point out the word [mong1]”; (2) wo3 hui4 du2 [mong1] zhe4 ge4 zi4” I can read the word [mong1]”); (3) wo3 hui4 xie3 [mong1] zhe4 ge4 zi4” I can write the word [mong1]”; (4) wo3 xiang3 lian4 [mong1] zhe4 ge4 zi4” I can practice the word [mong1].” All stimuli were produced with an angry, happy, sad, or neutral emotion. Mandarin listeners and Dutch-speaking learners of Mandarin were asked to identify the Mandarin tone of the pseudo words. The results showed that stimuli produced with the neutral emotion resulted in higher accuracy than those produced with non-neutral emotions. However, only the angry voice resulted in significantly lower accuracy relative to the neutral voice for both groups of listeners. In addition, Tone 4 was identified more accurately than Tone 1 in the angry voice.

In sum, research on speech perception shows that emotions affect the perception of segmental phonemes, talker voices, and lexical tones. The effect of emotion on lexical tone perception depends on both stimulus characteristics and tonal language experience. Particularly relevant to the current study, Liang and Chen’s (2019) [71] findings further demonstrated that emotions affect lexical tone perception to different degrees depending on specific Mandarin tones.

How emotions are perceived in speech

In addition to understanding emotion’s effect on speech perception, we also explore how emotions themselves are perceived in speech. A common approach to studying emotion recognition is to recruit professional actors to produce speech materials with different emotions. Listeners are then asked to identify the emotions of the stimuli [2, 4, 7, 50, 75–77]. Studies using non-tonal languages showed that sad and angry voices are easier to identify than fearful and happy voices [2, 4, 7, 50, 75–77]. Studies using Mandarin have reported similar findings: negative emotions such as sadness [41, 46, 66] and fear [43] are easier to identify than positive emotions such as happiness [41, 43]. It has been suggested that negative emotions are prioritized in vocal communication because they convey warnings in situations of attack, loss, and danger. Consequently, negative emotions need to be communicated more effectively to ensure human survival [43, 78, 79]. In contrast, positive emotions such as happiness are usually expressed through additional communication channels (e.g., facial expression), which may explain why a happy voice is identified with lower accuracy when only the vocal channel is used [7, 43, 50].

There is limited evidence on how lexical tones affect the perception of emotions. Wang, Ding, & Gu (2012) [80] investigated emotion recognition from Mandarin sentences by native and non-native speakers. Mandarin words with various tones (qi4che1 “car”, zhao4pian4 “picture”, xin1fang2 “new house”, dian4nao3 “computer”, and xue2xiao3 “school”) were embedded in a semantically neutral carrier phrase (zhe4 shi4 ta1 de0 [target word] “This is his [target word]”). The sentences were recorded with six emotions (happiness, fear, anger, sadness, boredom, & neutral) and presented to listeners for emotion recognition. The results showed that native listeners had an overall higher accuracy than non-native listeners, but both groups recognized sadness with the highest accuracy and boredom with the lowest accuracy. Since tones were not systematically manipulated, it is not clear whether the results could inform the effect of lexical tone on emotion recognition.

Wang and Lee (2015) [41] and Wang and Qian (2018) [47] constructed sentences composed exclusively of a particular Mandarin tone (e.g., wang1 bin1 xing1qi1tian1 xiu1 fei1ji1 “Wang Bin fixed the airplane on Sunday” or with a mixture of different tones (e.g., wo3 bu4gan3 xiang1xin4 zhe4 shi4 zhen1de0 “I cannot believe this is true”). The sentences were recorded with various emotions. It was hypothesized that emotions would be recognized less accurately in the Tone 1-only sentences because of the restricted F0 variation imposed by the (level) tone. An alternative hypothesis was that the restricted F0 variation in the Tone 1 sentences would have allowed emotions to surface more easily, thus facilitating emotion recognition. However, the results showed that emotions were recognized equally well regardless of tonal composition; that is, the restricted F0 variation imposed by the level tone did not compromise or facilitate emotion recognition. Emotional recognition seems quite robust irrespective of specific lexical tones.

Benefit of context

The presence of context can alter a listener’s interpretation of a speech sound [81]. Such perceptual adaptation forms the basis of speaker normalization [82, 83]. In speech audiometry, a carrier phrase is typically included in a word recognition task to provide a cue for the listeners to focus their attention on the target words [84–86]. Previous studies showed that word recognition accuracy is typically higher when embedded in a carrier phrase [84, 85, 87, 88]. The presence of a carrier phrase is particularly helpful under challenging listening conditions. For example, Lynn and Brotman (1981) [85] found that word identification in a carrier phrase was 10% more accurate than in isolation in the presence of speech-shaped noise. Since lexical tones produced with emotions are likely to deviate from the citation form, the presence of a context is likely to help listeners retrieve the intended tones more effectively.

In the current study, we examine Mandarin tone identification and emotion recognition in two contexts: when the target syllables are embedded in a carrier phrase (in context), and when the target syllables are extracted from the carrier phrase (in isolation). Note that the syllables in the “isolation” condition were not produced in isolation; rather, they were excised from the carrier phrase. We predict that Mandarin tone identification would be less accurate when the target syllables were extracted from the carrier phrase. This is because the citation form of a tone is likely to be altered due to the influence of neighboring tones [89]. Without the carrier phrase, it would be challenging for listeners to recover the tone. Furthermore, the carrier phrase provides information about talker characteristics such as speaking F0 range, which has been shown to facilitate tone identification from the multi-talker input [90]. We also predict that emotion recognition would be less accurate when the target syllables are presented in isolation. Since the talkers who recorded the stimuli were instructed to produce emotions for the entire utterance, the presence of emotions when preceded by a carrier phrase should facilitate emotion recognition from the target syllables.

The present study

The above review shows that both acoustics and perception of speech are shaped by emotions. Emotions affect Mandarin tone identification to different degrees depending on specific tones [71], but specific Mandarin tones do not seem to affect emotion recognition differently [41, 47]. To further clarify the interaction between lexical tones and emotions in speech, this study examines the acoustics and perception of Mandarin tones produced with various emotions, and the perception of emotions embedded in Mandarin tones. Following Liang and Chen (2019) [71], we use syllables produced with four Mandarin tones in a sentence-medial position. Extending Liang and Chen (2019) [71], we use multiple speakers of both sexes to record the stimuli. We also examine both the acoustics and perception of Mandarin tones and emotions. Finally, we examine the perception of Mandarin tones and emotions in two contexts: when the target syllables are presented with the carrier phrase, and when the target syllables are extracted from the carrier phrase.

Based on prior research, we predict that the acoustics of emotions would be affected by specific Mandarin tones. Liang and Chen’s (2019) [71] findings lead us to predict that the accuracy of Mandarin tone identification would be affected by emotions to different degrees depending on specific emotions and Mandarin tones in the stimuli. Following findings from Wang and Lee (2015) [41] and Wang and Qian (2018) [47], we predict that emotion recognition would remain robust irrespective of the specific Mandarin tones in the stimuli. Finally, we predict that Mandarin tone identification and emotion recognition would be less accurate when the target words are extracted from the carrier phrase.

Method

Results

Fig 1 shows the F0 contours of the four Mandarin tones produced with the five emotions. The F0 contours were time-normalized and averaged over speakers of the same sex. Specifically, for each token, F0 was measured every 10% from the beginning (0%) to the end (100%) to obtain 11 data points. Each of these 11 points was then averaged over all speakers of the same sex.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. F0 contours of target syllables as a function of emotion, Mandarin tone, and talker sex.

https://doi.org/10.1371/journal.pone.0283635.g001

The F0 contours of the Mandarin tones appear to be consistent with traditional descriptions of the Mandarin tones in citation form: Tone 1 is flat, Tone 2 is falling then rising, Tone 3 (which is in a non-final position in the carrier phrase) is falling, and Tone 4 is falling but in a higher register. The F0 plot is meant to show that the Mandarin tones were produced as intended. For quantitative analysis of the F0 contours, a more rigorous approach such as Functional Data Analysis should be taken [94, 95].

To evaluate the acoustic difference between the emotions, for each of the four acoustic measures (mean F0, F0 range, mean amplitude, and duration), a linear mixed-effects model was built for each of the four measures separately using R 3.6.3 (R Core Team, 2021) [96]. Mandarin tone (T1, T2, T3, and T4), emotion (ANGRY, FEAR, HAPPY, SAD, and NEUTRAL), and the tone-emotion interaction were entered as fixed effects. Talker, talker sex, syllable type, and repetition were entered as random effects.

Mean F0.

Fig 2 shows the mean F0 of the target syllables as a function of emotion and Mandarin tone. Fig 1 suggests that the overall F0 contours of the Mandarin tones produced with the four emotions are similar to those of NEUTRAL; therefore, we calculated mean F0 as a summary measure for quantitative comparisons. The linear mixed-effects model revealed significant main effects of Mandarin tone, χ²(3, N = 8) = 973.95, p < .001, emotion, χ²(4, N = 8) = 1749.66, p < .001, and tone-emotion interaction, χ²(12, N = 8) = 70.18, p < .001. Post hoc pairwise comparisons (Tukey adjusted) were conducted to disentangle the interaction (Fig 2). It was found that ANGRY had the highest mean F0 and NEUTRAL had the lowest. The ranking of FEAR, HAPPY, and SAD varies on specific Mandarin tones. Full output of the model is available in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Boxplot showing the mean F0 of target syllables as a function of Mandarin tone and emotion.

(*p < .05).

https://doi.org/10.1371/journal.pone.0283635.g002

F0 range.

Fig 3 shows the F0 range of the target syllables as a function of emotion and Mandarin tone. The linear mixed-effects model revealed significant main effects of Mandarin tone, χ²(3, N = 8) = 779.23, p < .001, emotion, χ²(4, N = 8) = 123.02, p < .001, and their two-way interaction, χ²(12, N = 8) = 114.64, p < .001. The results of post hoc pairwise comparisons are also shown on the figure. There does not appear to be a consistent pattern in the ranking of the emotions. Full output of the model is available in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Boxplot showing the F0 range of target syllables as a function of Mandarin tone and emotion.

(*p < .05).

https://doi.org/10.1371/journal.pone.0283635.g003

Mean amplitude.

Fig 4 shows the mean amplitude of the target syllables as a function of emotion and Mandarin tone. The main effects of Mandarin tone, χ²(3, N = 8) = 121.1, p < .001, and emotion, χ²(4, N = 8) = 1801.44, p < .001 were observed, but not their interaction χ²(12, N = 8) = 3.92, p = .98. Post hoc pairwise comparisons showed that ANGRY has the highest mean amplitude and NEUTRAL has the lowest. No difference was observed across SAD, HAPPY, and FEAR. Full output of the model is available in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Boxplot showing the mean amplitude of target syllables as a function of Mandarin tone and emotion.

https://doi.org/10.1371/journal.pone.0283635.g004

Duration.

Fig 5 shows the duration of the target syllables as a function of emotion and Mandarin tone. Similar to the mean amplitude, the main effects of Mandarin tone, χ²(3, N = 8) = 32.5, p < .001, and emotion, χ²(4, N = 8) = 639.74, p < .001 were significant but their interaction χ²(12, N = 8) = 14.2, p = .29 was not. Post hoc pairwise comparisons indicate SAD has the longest duration than all other emotions. Full output of the model is available in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Boxplot showing the duration of the target syllables as a function of Mandarin tone and emotion.

https://doi.org/10.1371/journal.pone.0283635.g005

Summary and discussion

Emotions leave a mark on the acoustic characteristics of Mandarin tones. Consistent with previous studies [2, 5–7, 41–53], findings from our acoustic analyses support the observation that emotions shape the acoustic characteristics of speech for both tonal and non-tonal languages [63–65, 97]. Our inclusion of all four Mandarin tones produced by talkers of both sexes further revealed that the impact of emotions varies depending on specific Mandarin tones. ANGRY has the highest mean F0 and mean amplitude. SAD has the longest duration. In contrast, we did not observe a systematic difference in F0 range.

Table 2 summarizes our findings compared to previous research. There are similarities but also discrepancies. Methodological differences such as talkers (amateurs in previous studies vs. professional actors in the current study) and materials (sentences in previous studies vs. syllables extracted from a carrier phrase in the current study) are likely to have contributed to the discrepancies. We used professional actors in the current study because they are typically more proficient in producing the desired emotions [7, 63]. The presence of the intended emotions was verified in the current study with an independent emotion judgment task. As for materials, since the syllable is the tone-bearing unit in Mandarin, our choice to analyze syllables instead of sentences allowed us to examine the effect of emotions on specific Mandarin tones systematically.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of findings from Experiment 1 regarding the effect of emotion relative to the neutral emotion.

https://doi.org/10.1371/journal.pone.0283635.t002

Among the acoustic measures, mean F0 and F0 range appear to yield the most consistent results between previous research and the current study. ANGRY, FEAR, and HAPPY consistently resulted in a higher F0. However, the utility of this measure is difficult to evaluate because of the lack of specificity in the predictions. For example, previous research showed that FEAR and SAD could result in a wider, comparable, or narrower F0 range in Mandarin tones compared to the neutral emotion. Although the current study showed the same results, no consistent patterns could be extracted without taking into consideration specific Mandarin tones.

Among the emotions, ANGRY consistently results in a higher mean F0, a greater F0 range, and a higher mean amplitude. This finding appears to support the proposal that negative emotions are conveyed more effectively in vocal communication to ensure human survival [43, 78, 79]. If an emotion results in relatively stable acoustic changes in lexical tones, recognition of that emotion is likely to be more robust. However, the other two negative emotions FEAR and SAD did not result in a similar pattern of consistent acoustic changes. It has yet to be determined if ANGRY has a special status among the negative emotions.

Method

Results

Mandarin tone identification.

Fig 6 shows the accuracy of Mandarin tone identification as a function of emotion, Mandarin tone, and context. Overall, Mandarin tones were identified more accurately in context, and the effect of emotion appeared to be much more variable when the target syllable was presented in isolation. For example, all four Mandarin tones were identified quite well in NEUTRAL regardless of context, but the presence of other emotions compromised the identification of isolated Mandarin tones disproportionately compared to those presented in context.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Boxplot showing the Mandarin tone identification accuracy as a function of Mandarin tone, emotion, and context.

https://doi.org/10.1371/journal.pone.0283635.g006

To evaluate these observations statistically, a mixed-effect logistic regression model was fitted to the Mandarin tone identification data. Mandarin tone (T1, T2, T3, and T4), emotion (ANGRY, FEAR, HAPPY, SAD, and NEUTRAL), context (in isolation and in context), and the tone-emotion interaction were entered into the model as fixed effects. Talker sex, syllable type, and repetition were entered as random effect. The dependent variable was the binary Mandarin tone identification (i.e., correct and incorrect). Full output of the model is available in S2 Table.

All main effects were significant: Mandarin tone, χ²(3, N = 36) = 83.12, p < .001; emotion, χ²(4, N = 36) = 486.38, p < .001; and context, χ²(1, N = 36) = 1534.71, p < .001. The tone-emotion interaction was also significant: Mandarin tone-emotion, χ²(12, N = 36) = 492.89, p < .001. Post hoc pairwise comparisons indicate that tone identification accuracy varied across different emotions (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of pairwise comparisons for Mandarin tone identification accuracy as a function of emotion.

The symbol > indicates a significant difference (p < .05) and = indicates no significant difference.

https://doi.org/10.1371/journal.pone.0283635.t003

To examine the specific types of Mandarin tone identification errors, Tables 4 and 5 show confusion matrices of Mandarin tone identification responses for syllables presented in isolation (Table 4) and in context (Table 5). To facilitate interpretation of the confusion patterns, the most common error for each Mandarin tone that exceeds 20% is highlighted in . For syllables presented in isolation (Table 4), Mandarin tones produced with NEUTRAL were rarely misidentified as other tones. There appears to be a response bias where Mandarin tones, particularly Tones 1 and 3, were most misidentified as Tone 4 when presented in an ANGRY tone of voice. Similarly, there appears to be a response bias for tones to be identified as Tone 1 when presented in a FEAR tone of voice. In contrast, the errors for Mandarin tones produced with HAPPY or SAD did not show distinct bias patterns.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Confusion matrices of Mandarin tone identification responses for syllables presented in isolation across emotions.

The most common error that exceeds 20% for each emotion is highlighted in gray.

https://doi.org/10.1371/journal.pone.0283635.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Confusion matrices of Mandarin tone identification responses for syllables presented in context across emotions.

https://doi.org/10.1371/journal.pone.0283635.t005

For syllables presented in context (Table 5), Mandarin tone identification was remarkably accurate. There were only three instances where the accuracy fell below 90%, and only one of those was below 80%. In terms of confusion patterns, there was only one error that approached 20%, where an ANGRY Tone 1 was misidentified as Tone 4. There were no other dominant errors. In sum, the presence of the carrier phrase effectively neutralized the negative impact of emotions on Mandarin tones identification from isolated syllables.

Emotion recognition

Fig 7 shows the accuracy of emotion recognition as a function of emotion, Mandarin tone, and context. Overall, accuracy appears higher for target syllables presented in context than in isolation. Importantly, the four emotions appear to be affected by context to different degrees. For example, FEAR appears to be identified disproportionately worse in isolation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Boxplot showing the emotion recognition accuracy as a function of Mandarin tone and context.

https://doi.org/10.1371/journal.pone.0283635.g007

A mixed-effect logistic regression model with Mandarin tone (T1, T2, T3, and T4), emotion (ANGRY, FEAR, HAPPY, and SAD), context (in isolation and in context), and the tone-emotion interaction as fixed effects, talker sex, syllable type, and repetition as random effects were constructed. The dependent variable was the binary emotion recognition (correct and incorrect). Full output of the model is available in S3 Table.

All main effects were significant: Mandarin tone, χ²(3, N = 36) = 92.92, p < .001; emotion, χ²(1, N = 36) = 775.08, p < .001; and context, χ²(1, N = 36) = 1843.42, p < .001. The tone-emotion interaction was also significant: χ²(9, N = 36) = 327.95, p < .001. Post hoc pairwise comparisons indicate that emotion recognition accuracy varied across different Mandarin tones (Table 6). For example, ANGRY was recognized most accurately in all but Mandarin Tone 3, whereas FEAR was recognized least accurately in all Mandarin tones.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Summary of pairwise comparisons for emotion recognition accuracy as a function of Mandarin tone.

The symbol > indicates a significant difference (p < .05) and = indicates no significant difference.

https://doi.org/10.1371/journal.pone.0283635.t006

To examine the types of emotion recognition errors made by listeners, Tables 7 and 8 show confusion matrices of emotion recognition responses for syllables presented in isolation (Table 7) and in context (Table 8). To facilitate the interpretation of the confusion patterns, the most common error for each emotion that exceeds 20% is highlighted in .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Confusion matrices of emotion recognition responses for syllables presented in isolation.

The most common error that exceeds 20% for each emotion is highlighted in gray.

https://doi.org/10.1371/journal.pone.0283635.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Confusion matrices of emotion recognition responses for syllables presented in context.

https://doi.org/10.1371/journal.pone.0283635.t008

For isolated syllables (Table 7), emotion recognition accuracy ranged from 21% to 93%. There appears to be a response bias where SAD stimuli were most commonly identified as FEAR in Tone 1 and 4. Similarly, there appears to be a response bias where FEAR stimuli were commonly identified as HAPPY in Tones 1 and 2, but as SAD in Tones 3 and 4. In contrast, ANGRY and HAPPY stimuli did not result in any dominant bias patterns.

For syllables presented in context (Table 8), emotions were recognized more accurately, ranging from 85% to 99%. In terms of confusion patterns, the only error that exceeded 10% was the SAD response to the FEAR stimulus in Tone 1 (15%). Because of the high accuracy, none of the emotions resulted in any dominant bias patterns.

Summary and discussion

As predicted, Mandarin tone identification accuracy varied across emotions, i.e., not all tones were identified equally well for different emotions. We replicated Liang and Chen’s (2019) [71] finding that Tone 4 is identified more accurately than Tone 1 in ANGER. Our results further revealed additional contingencies on emotion (Table 3). Tone identification accuracy also varied across contexts. All tones were identified more accurately when the target syllables were embedded in the carrier phrase, but not all tones were identified equally well across the two contexts (Table 3). For syllables presented in isolation, confusion analyses showed that Tone 4 was the most common error for the ANGRY stimuli, and Tone 1 was the most common error for the FEAR stimuli (Table 4). For syllables presented in the carrier phrase, confusion analyses did not reveal any notable patterns except that Tone 4 was a common error for ANGRY Tone 1 stimuli (Table 5).

The hypothesis that ANGRY stimuli would result in less or more accurate tone identification depending on how listeners interpret the acoustic signal was not supported by our data. Although ANGRY resulted in the greatest acoustic changes (as shown in Experiment 1), Mandarin tones produced with ANGRY were not identified disproportionately worse. Similarly, although ANGRY was associated with the most consistent acoustic changes (as shown in Experiment 1), Mandarin tones produced with ANGRY were not identified disproportionately better. With the extensive interactions between tone, emotion, and context, the current data is inconclusive as to how listeners parsed the acoustic signal into distinct sources.

Regarding emotion recognition, accuracy of emotion recognition varied across Mandarin tones, i.e., not all emotions were recognized equally well in different Mandarin tones (Table 6). However, ANGRY was recognized most accurately in three of the four Mandarin tones. Like Mandarin tones, emotions were recognized more accurately when the target syllables were embedded in the carrier phrase. Unlike Mandarin tones, the order of emotion identification accuracy was more consistent between in isolation and in context (Table 6), i.e., ANGRY was recognized better than HAPPY, followed by SAD, and FEAR was recognized least accurately. For syllables presented in isolation, confusion analyses showed that the most common error for SAD stimulus was FEAR response (Table 7). For syllables presented in context, confusion analyses did not reveal any notable error patterns (Table 8).

Taken together, data from Mandarin tone identification and emotion recognition revealed several similarities. Mandarin tones identification accuracy varied across emotions, just as emotions recognition accuracy varied across Mandarin tones. In addition, Mandarin tone identification and emotion recognition were both more accurate with syllables presented in the carrier phrase. The presence of the carrier phrase facilitated both Mandarin tone identification and emotion recognition, especially for those tones and emotions that were identified poorly in isolated syllables. Finally, for both Mandarin tone identification and emotion recognition, there were distinct error patterns in isolated syllables, but barely any notable error patterns in context.

There are also differences between Mandarin tone identification and emotion recognition. First, the accuracy of identifying specific Mandarin tones varied greatly across different emotions (Table 3), but the accuracy of recognizing specific emotions was more consistent across different Mandarin tones. For example, ANGRY was consistently identified better than most other emotions, and FEAR was consistently identified worse than all other emotions (Table 6). In other words, how well a Mandarin tone is identified depends heavily on specific emotions, but how well an emotion is identified does not depend as much on specific Mandarin tones. At first sight, this finding appears to suggest that emotion recognition is more robust than Mandarin tone identification. However, the accuracy of emotion recognition (ranging from 21% to 93%) was not higher than Mandarin tone identification (40% to 98%) in isolated syllables. The accuracy in the carrier phrase also seemed comparable: emotion recognition (85% to 99%); Mandarin tone identification (78% to 100%). Emotions do not seem to be inherently easier to identify compared to Mandarin tones. Rather, our data suggest that their mutual influence is asymmetrical.

Furthermore, Mandarin tone identification and emotion recognition differed in their interaction with context. The accuracy of identifying specific Mandarin tones varied substantially between the two contexts (Table 3), but the accuracy of recognizing specific emotions was quite consistent between the two contexts (Table 6).