One channel, two forms of expression

Emotional vocalizations and emotional speech prosody share much in common; both consist of dynamic acoustic patterns that vary in perceived pitch, energy, quality, and temporal sound properties that progressively gain significance over time [8]. There is evidence that both forms of expression exploit a common set of acoustic features that allow listeners to differentiate and attribute a range of discrete emotion states to the speaker [4,9]. These mappings allow ‘basic’ emotions such as anger, fear, happiness and many other meanings to be successfully detected in the voice at levels far exceeding chance, both within and across cultural boundaries [10–15].

Still, it seems obvious that saying, “I don’t know what to do” in a sad voice, or beginning to sob uncontrollably, are quite different exemplars of “sadness” that will have distinct perceptual and interpersonal effects [16]. At the production stage, vocalizations allow greater flexibility and acoustic variability in their expression than speech-embedded emotions, which are constrained by simultaneous demands on linguistic production [17,18]. As a result, nonverbal signals are believed to capture preferential attention [19,20] and encode emotions with greater perceptual clarity (i.e., reduced sensory ambiguity), promoting higher recognition rates than prosody when formally measured in experimental tasks [2,4]. Vocalizations also appear to be processed with greater automaticity than speech [3,21,22], promoting fast and efficient recognition of emotions that are minimally hampered by conscious deliberation or simultaneous cognitive load [23]. The apparent primacy of emotional vocalizations in perception could reflect the fact that nonverbal signals emanate from a biologically primitive, reflexive call system adapted by humans (and other species) for survival, whereas emotions in speech are cognitively mediated and under significant voluntary control [24–26]. As such, processes of socialization—including cultural preferences, language familiarity, and other forms of learning—are believed to exert stronger effects on how emotional prosody is recognized in the vocal channel than nonverbal expressions [10,13,27,28].

On the time course of vocal emotion recognition

But how does the form of vocal expressions alter the path to emotion recognition over time, and in what way(s) do these operations depend on listener experience (e.g., previous exposure or familiarity with particular event types)? Surprisingly, direct comparisons of vocalizations and speech prosody are still few. One way to infer how and when vocal emotions are recognized is to measure event-related brain potentials (ERPs) evoked by a vocal stimulus. When listeners are presented prosody in their native language, there is evidence that the brain differentiates discrete emotional qualities of speech beginning 200 milliseconds post-onset of an utterance [3,29], at least in sufficient terms to facilitate deeper cognitive analysis of the event’s contextual meaning at later timepoints (see [30] for a recent discussion). ERP studies that have presented vocalizations tend to report even earlier cortical responses, with emotion-related changes in activity often beginning 100ms post-onset of vocalizations [21,31]. These studies suggest that perceptual and cognitive operations for ‘recognizing’ emotion from voices proceed rapidly, but that vocalizations are differentiated at an earlier time point by the neurocognitive system than prosody. This claim is reinforced by an ERP experiment that compared the two expression types directly: vocalizations of anger, sadness, and happiness produced earlier and more qualitatively distinct brain responses in listeners than when these emotions were expressed prosodically in native-like pseudo-utterances [3].

Another way to illuminate the time course of vocal emotion recognition is by “gating” auditory stimuli, i.e., presenting time-limited excerpts of a vocal stimulus (e.g., the first 200 or 400 milliseconds), to gauge their effects on perception and behaviour. Using gated prosodic stimuli in a novel voice-face priming paradigm, Pell and colleagues [32,33] concluded that English listeners require at least 400ms of speech exposure to prime decisions about an emotionally-related face, suggesting that discrete meanings of prosody are activated and implicitly “recognized” from ~400ms of acoustic information (see also [34]). Interestingly, when emotions were expressed in a foreign language (Arabic), English listeners required prolonged exposure to the prosodic input (>600ms) for priming to occur [35]. These findings suggest that language familiarity is a critical factor governing the recognition of speech-embedded emotions and alters its recognition time course [36].

Other experiments have probed these questions using an auditory gating paradigm adapted from [37], whereby participants judge vocal emotion expressions presented in time- or structure-based increments which always increase in duration over the course of the study (e.g., participants render forced-choice emotion judgements after hearing the first 100ms, 200ms, or 300ms of the same event). This approach allows researchers to estimate how much acoustic input listeners need to form stable categorical representations of emotion based on a specific stimulus duration, by comparing “gate-to-gate” increases in recognition accuracy, while ensuring that acoustic details that promote recognition always build up incrementally from shortest to longest event duration. Studies that have gated emotional prosody in languages such as English, Hindi, or Swedish show that speech-embedded emotions are accurately recognized at notably different latencies or “speeds” [36,38–41]. While patterns vary from study to study due to differences in how gates were defined, most gating studies conclude that listeners need at least 400–500ms of emotional prosody for recognition to begin to stabilize (“emotion identification point”), meaning that participants can identify the target meaning at this timepoint and do not change their mind at longer exposures [38,39,41]. The time course for recognizing specific emotions from prosody varies markedly: anger, sadness and (sometimes) fear tend to be isolated earliest from ~500–800ms of acoustic information, whereas other emotions (e.g., happiness, disgust, interest) often require 1–2 seconds of speech cues to identify [36,39,41]. Moreover, certain speech-embedded emotions, particularly happiness, seem to depend heavily on linguistic structure and can only be isolated when listeners integrate acoustic cues provided toward the end of an utterance [10,40,42].

Subsequent experiments that have gated vocalizations report that stable emotion representations can typically be formed after hearing ~250–350ms of acoustic input, much earlier than for prosody [43–45]. Castiajo & Pinheiro (2019) observed large increases in target hit rates for 10 different emotional vocalizations that lasted between 200–300ms, with fastest recognition of amusement (laughter) and slowest recognition of fear. However, this literature is again marked by many methodological differences related to the number and type of emotions studied and the way that vocalizations were gated for presentation (e.g., 33ms vs. 100ms intervals). Moreover, studies have not gated vocalizations and speech-embedded emotions in a unitary manner for evaluation by the same participants to directly compare if the time course of recognition differs as a function of their expressive form. To address these issues, our study undertook a direct test of how emotion recognition unfolds from vocalizations vs. speech prosody using the gating paradigm to determine whether nonverbal stimuli are recognized “more efficiently” (i.e., at an earlier timepoint) than speech prosody, as suggested by recent findings [3,5].

As a secondary goal, we sought to shed light on how familiarity with a language influences the recognition of speech-embedded emotions. According to Dialect theory, socially-constructed forms of emotional communication, such as prosody, are shaped by cultural “styles” which provide listeners an advantage to recognize emotions expressed by native (‘ingroup’) speakers [46]. In the only emotional gating study to implement a cross-cultural design, Jiang and colleagues [36] required groups of English Canadian and Indian listeners to recognize four emotions (anger, fear, happiness, sadness) from both English and Hindi pseudo-utterances gated to six exposure durations (200ms, 400ms, 500ms, 600ms, 700ms, full utterance). Results showed that recognition accuracy was significantly higher for native prosody in each group (ingroup advantage), emphasizing the importance of language experience on emotional prosody recognition [7]. In addition, emotion identification points occurred earlier in time for native prosody, irrespective of whether the non-native language was considered foreign to the listener (Canadians judging Hindi) or the listener’s second language/L2 (Indians judging English). Interestingly, the Indian participants’ ability to recognize emotions in L2-English was positively associated with their English proficiency level, although this relationship is not consistently reported elsewhere in the literature (cf. [47–49]. These findings motivate a deeper look at the impact of language familiarity—i.e., whether emotions are expressed in a listener’s native, second, or a completely foreign language—on both the accuracy and time course of emotional prosody recognition [7,15].

Aim of the current study

Here, we investigated how emotions are recognized from vocalizations versus speech prosody using an adapted version of Jiang et al.’s [36] cross-cultural gating paradigm. Our new design simultaneously allowed us to evaluate the impact of language familiarity on recognition performance within and between two distinct groups: native speakers of Mandarin-Chinese and Arabic, all of whom were proficient second-language speakers of English. Based on above, we predicted that recognition accuracy would be higher and stabilize at earlier timepoints for vocalizations than for native prosodic expressions in each group; emotion-specific recognition trajectories in each condition were likely to vary, but these patterns should be more similar between groups when judging vocalizations than emotional prosody, which is shaped by linguistic and cultural variables to a greater extent [7]. For speech-embedded emotions, it was expected that language familiarity would enhance accuracy and speed of emotional prosody recognition when each listener group judged their native language, consistent with the ingroup advantage (native > foreign and L2-English). We further speculated that high proficiency in L2-English would enhance emotional prosody recognition over foreign prosody [36,47,49], although there is no clear precedent for comparing these measures simultaneously in native, L2, and foreign language contexts.

Participants

Fifty young adults judged all vocal expressions presented in the study; assuming medium effect sizes, this sample size achieves power exceeding.95 to detect differences in event type if mixed ANOVAs had been employed (this estimation may be considered the lower bound of actual power achieved in the study using linear mixed effects models, [50]). Participants were recruited from the greater Montréal region on the basis of speaking Arabic (n = 25, 15F/10M, Mean Age = 21.5 ± 3.4, Mean Education = 15.6 years ± 3.8) or Mandarin-Chinese (n = 25, 18F/7M, Mean Age = 22.2 ± 4.2, Mean Education = 15.6 years ± 2.4) as a native language, and for having high proficiency in English as a second language. Participants were students or recent immigrants who moved to Canada as adults and had lived in Montréal for less than five years (Median duration in Canada: Arab group = 18 months, range = 4–58 months; Chinese group = 12 months, range = 1–24 months). Most participants arrived in Canada between the ages of 19–21. Arab participants were born and raised in several Arabic-speaking countries (Syria, Jordan, Bahrain), although the majority (20/25) spoke variants of Levantine Arabic. Chinese participants were born and raised in different regions of mainland China (Shanghai, Beijing, Shenzhen).

All participants learned English in school as a second language (L2-English) from an early age (Mean Age of English onset: Arab = 5.7 years ± 3.4; Chinese = 8.1 years ± 3.3). L2-English proficiency was characterized through a series of self-report measures gathered at the onset of the study; all participants in each group rated their ability to speak and listen in English as high (group means ranged between 7.7–9.5 on a 10-point proficiency scale, Table 1). Many participants in each group knew additional languages (e.g., French, Farsi, Cantonese). To enter the study, it was verified that no participant had any knowledge of the language designated as “foreign” for that group in the experiment (i.e., Arabic for Chinese participants; Chinese for Arab participants). All participants reported normal hearing. Recruitment began on 07/21/2016 and ended on 05/26/2017. Voluntary written consent was obtained prior to the study, which was approved by the Faculty of Medicine Institutional Review Board, McGill University.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Demographic characteristics of participants in the Chinese and Arab groups.

https://doi.org/10.1371/journal.pone.0327529.t001

Materials

The stimuli were digital auditory recordings of vocally expressed emotion ~1–3 seconds in duration, taken from established inventories used actively in the literature [15,27,51–53]. All recordings were elicited in laboratory settings using lay speakers/actors and then perceptually validated by different groups of listeners. The study included two distinct event types: nonverbal vocalizations and speech-embedded emotional expressions (henceforth referred to as vocalization and speech prosody). Speech prosody could be further divided by language of expression (Arabic, Mandarin, English), which differed in familiarity to each perceiver group (defined as native, L2-English, or foreign). Vocalizations and prosody in each language communicated one of four discrete emotions: anger, fear, sadness, or happiness (Fig 1A,1B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Experimental design and procedure.

(A) Stimuli expressed one of four emotions (anger, fear, happiness, sadness) through speech prosody or vocalizations. Each stimulus was cut from vocal onset and presented in five “gate” (G) duration blocks: 200ms, 400ms, 500ms, 600ms, or the full expression (GFull). (B) Language familiarity conditions as a function of event type for Mandarin and Arabic perceiver groups. (C) Trial sequence showing timing and two mouse-click responses (emotion choice and confidence rating). Blocks always presented stimuli in order of increasing acoustic duration, separately for speech and vocalizations.

https://doi.org/10.1371/journal.pone.0327529.g001

Vocalizations took the form of growls or shouts (anger), cries (fear), sobbing or wailing (sadness), and laughter or contentment sounds (happiness). All stimuli were nonverbal in nature, although some resembled sustained vowels or nasal sounds found in most languages (“aah” or “mmm”). Stimuli in the speech prosody condition were emotionally-inflected pseudo-utterances composed of 7–11 syllables/characters produced by native speakers of Arabic, Mandarin-Chinese, or English [15,53]. Pseudo-utterances, which mimic linguistic properties of a language while restricting emotion to the speaker’s prosody, have been used in previous gating studies [36,38,39] and broadly in the prosody literature (see [8] for an overview). Pseudo-speech stimuli in each of the three language conditions were constructed, recorded, and perceptually validated using virtually identical procedures, but involving native speakers-listeners of only the target language (see [53]) for details). To mimic the prosody condition, vocalizations were selected from three separate corpora produced by French Canadian, European Portuguese, and British English speakers [27,51,52]. This decision was meant to increase generalizability of results to diverse speakers and allow finer selection of vocalizations that matched perceptual features of the speech stimuli to the extent possible.

Perceptual data from the original studies were used to select a controlled set of angry, sad, fearful and happy exemplars suitable for directly comparing the effects of vocal event type on the time course of emotion recognition. All selected vocalizations achieved high consensus about the target emotion according to the original study design (accuracy > 5 times chance). Vocalizations too brief for effective gating or much longer than the speech stimuli were excluded. In the case of vocalizations representing “happiness”, we noted that emotional meaning labels validated in the original inventories often varied (e.g., “pleasure”, “contentment”, “amusement”, “happiness”). As these terms likely refer to a broader range of positive emotions that can be communicated nonverbally than in speech [54], we initially selected two subsets of “happy” vocalizations to compare with speech prosody in an exploratory manner: happiness-amusement, characterized uniquely by laughter sounds; and happiness-pleasure, composed of vocal sounds of pleasure or contentment (e.g., “mmm”, “aah”). Ten unique vocalizations were selected for anger, fear, and sadness and 20 unique items for happiness (10 amusement, 10 pleasure). This totalled 50 vocalizations, produced by a variety of speakers (7–10 speakers/ emotion, half female, half male, 18 distinct voices in total). Selected vocalizations varied naturally in duration with a median of 1448ms (range = 715–2376ms) prior to gating.

Prosodic exemplars of each emotion were selected for one female and one male native speaker of each language, who each produced six distinct pseudo-utterances to convey each emotional target (anger, fear, sadness, happiness, 12 items/emotion). This process yielded 48 unique prosodic expressions/language (2 speakers x 4 emotions x 6 utterances). Items were again selected for having high emotion recognition rates when judged by a native listener group in the original study (minimum 3x chance accuracy based on a seven forced-choice task). In addition, items were chosen to mitigate gross differences in native emotion recognition accuracy across language sets to the extent possible (Mean emotional target recognition range by native listeners: Arabic = 55–74%, Mandarin = 64–82%, English = 64–80%). A total of 144 utterances (3 languages x 48 items) were selected for gating in the speech prosody condition. The speech stimuli were roughly similar in duration (Median = 1493ms, range = 834–2900ms) to items in the vocalization condition.

Experimental design and procedures

How does emotion recognition unfold from vocalizations?

Accuracy.

Separate group analysis first looked at recognition accuracy for vocalizations (LMM: HuScore (Vocalization) ~ Emotion + Gate + Emotion * Gate + (1| Subject)). Accuracy of each group depended on Emotion type (Arab: F = 287.81; df = 4, p < 0.001; Chinese: F = 275. 61, df = 4, p < 0.001), Gate duration (Arab: F = 64.46, df = 4, p < 0.001; Chinese: F = 50.52, df = 4, p < 0.001), and Emotion x Gate duration (Arab: F = 6.43, df = 16, p < 0.001; Chinese: F = 5.78, df = 16, p < 0.001; S3A Table). Fig 2A,2B shows that while emotional vocalizations had distinct recognition trajectories over time, the accuracy of Chinese and Arab listeners was similar in qualitative and quantitative terms. When accuracy at successive gate durations was compared, data show that anger (growls or shouts) was recognized at high levels based on 200ms of acoustic information (G200) and improved minimally as exposure increased. Happiness-amusement improved when stimulus duration increased from 200–400ms (ps < .001), without further improvements between gates to the full stimulus. Fear (screams) and sadness (sobs) also improved significantly between 200–400ms, and then again between G400 to the end of the stimulus (all ps < .001), suggesting a more incremental buildup of recognition for these signals. Happiness-pleasure was identified very poorly by both groups, and while accuracy improved between 200ms and the end of the utterance (ps < .002), recognition remained below chance performance levels for this emotion even when listeners heard the full stimulus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Emotion recognition accuracy (Hu scores) as a function of perceiver group, event type, and stimulus duration.

Top panel: Recognition of gated vocalizations by the A) Chinese and B) Arab participants. Bottom panel: Recognition of gated native speech prosody by the C) Chinese and D) Arab participants.

https://doi.org/10.1371/journal.pone.0327529.g002

Limiting stimuli to 200ms yielded superior recognition of anger versus all other vocalizations. Based on 400ms excerpts, anger, fear, and happiness-amusement were recognized at roughly similar rates by each group, exceeding sadness and happiness-pleasure. Virtually no changes occurred in the 400–500ms time range for either group. When stimuli lasted 600ms, only happiness-pleasure was recognized at inferior levels (Arab participants displayed a somewhat prolonged “anger” detection advantage over some vocalizations in the 200–600ms time range, Fig 2B). Recognition of ungated vocalizations (GFull) was highest for anger, fear, and sadness, followed by happiness-amusement, with markedly inferior detection of happiness-pleasure. The fact that pleasure/contentment sounds, in contrast to laughter, were not reliably identified as “happiness” by either group irrespective of exposure time suggests an artefact of our forced-choice design for this category (S1 Table shows that these expressions were labelled as “neutral” approximately half of the time). For this reason, happiness-pleasure was not considered in further analyses; “happiness” in all subsequent models referred solely to amusement/laughter sounds.

Latency.

Analysis of EIPs (omitting happiness-pleasure) considered the time needed to form a stable representation of the four vocalizations between groups, controlling for differences in individual event duration (LMM: EIPtime (Vocalization) ~ Group + Emotion + Group* Emotion + GFullDuration + (1 | Subject)); S3B Table). EIPs varied by Emotion (F = 9.16, df = 3, p < 0.001) and Group x Emotion (F = 4.04, df = 3, p = 0.007). Fig 3A,3B shows that all vocalizations were reliably isolated by listeners in both groups within a narrow ~300–500ms time window. Overall, fear (M = 500ms) required significantly more acoustic exposure to isolate than happiness-amusement (M = 355ms, p < 001), anger (M = 394ms, p < .001), and sadness (M = 407ms, p = .019). The interaction was explained by small group differences in the significance of emotion-specific contrasts, and evidence that Arab listeners required less time to detect happiness (laughter) than the Chinese (p = .041). There was no main effect of perceiver Group on recognition latencies for vocalizations (p = .25, ns).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Recognition latency (Emotion Identification Points in milliseconds) as a function of perceiver group and event type.

Top panel: Recognition of vocalizations by the A) Chinese and B) Arab participants. Bottom panel: Recognition of native speech prosody by the C) Chinese and D) Arab participants.

https://doi.org/10.1371/journal.pone.0327529.g003

How does emotion recognition unfold from native speech prosody?

Do listeners recognize emotions better from vocalizations than their native language?

Analyses then broadly examined whether vocalizations promoted a recognition advantage in accuracy or speed over native prosody through direct group comparisons (LMM for Accuracy: HuScore (GFull) ~ Group + EventType + Group*EventType + (1 | Subject)+ (1 | Emotion); LMM for Latency: EIPtime ~ Group+ EventType + Group*EventType + GFullDuration + (1 | Subject)+ (1 | Emotion); S5A Table). The model performed on accuracy considered Hu scores at a single gate, GFull, when listeners had all acoustic information available to promote recognition of emotional targets.

Accuracy.

Performance differed significantly by Event type (F = 25.95, df = 1, p < 0.001), pointing to superior accuracy to detect emotional vocalizations (M = 0.69) than native prosody (M = 0.59) overall. A Group x Event type interaction (F = 9.90, df = 1, p < 0.001) qualified that while the group trends were similar (vocalization > native prosody), this pattern was significant for the Arab listeners (p < .001) but not the Chinese (p = .169, Fig 4A). When listeners heard expressions in their entirety, there were no Group differences in the ability to recognize emotions in either the vocalization or the native prosody condition (ps > .191).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Recognition of vocalizations vs. native prosody by Chinese and Arab participants across emotion types.

A) Accuracy (Hu score) of each group when ungated expressions were presented (GFull condition). B) Corresponding Emotion Identification Points (in milliseconds) by group and event type.

https://doi.org/10.1371/journal.pone.0327529.g004

Does linguistic experience influence emotional prosody recognition?

At a final step, separate group analyses considered patterns of emotion recognition in the three speech prosody conditions defined by the familiarity of each language to each listener group (native, L2-English, foreign), collapsed across emotion types (LMM: HuScore ~ Familiarity + Gate + Familiarity*Gate + (1 | Subject) + (1 | Emotion), S6A Table). These models will reveal if representations underlying emotional prosody are facilitated by (native and/or second) language experience and when this knowledge comes into play during event processing [36].

Accuracy.

For Chinese listeners, accuracy depended on Gate duration (F = 147.3, df = 4, p < 0.001) and language Familiarity (F = 333.27, df = 2, p < 0.001) in the absence of an interaction. Emotional prosody recognition improved significantly within two principal time windows: between 200–400ms and when listeners had access to all acoustic information in the utterance (G600-GFull); these effects were observed consistently across speech contexts (native, L2, foreign) and were mirrored by the Arab group. Irrespective of stimulus exposure, Chinese listeners identified emotions more accurately from native prosody (Mandarin) than L2-English or foreign prosody (Arabic), exemplifying an ‘ingroup’ recognition advantage for native over non-native forms of prosody [7]. Accuracy in the L2-English vs. foreign language conditions did not differ at any time point (Fig 5A). Arab listeners displayed a unique pattern: accuracy depended on Gate duration (F = 186.25, df = 4, p < 0.001), language Familiarity (F = 52.77, df = 2, p < 0.001), and their interaction (F = 5.42, df = 8, p < 0.001). When prosodic stimuli were short (200–600ms), Arab listeners isolated emotions better in the foreign language, Mandarin, than from native (Arabic) or L2-English prosody, which did not differ. When the full utterance became available, Arab listeners showed marked improvements in the native and L2-English conditions, resulting in similar accuracy levels in the three speech prosody contexts (Fig 5B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Effects of language familiarity on emotional prosody recognition.

Top panel: Accuracy (Hu score) of the A) Chinese and B) Arab participants over time to recognize prosody in their native language, English as a second language (L2), and in a foreign language. Bottom panel: Corresponding Emotion Identification Points (in milliseconds) for the C) Chinese and D) Arab participants by language familiarity.

https://doi.org/10.1371/journal.pone.0327529.g005

To further contextualize these findings, we directly compared Group accuracy at GFull after recoding the speech prosody conditions according to the language of expression (Arabic, English, Mandarin) irrespective of its relevance/familiarity to a particular perceiver group (LMM: HuScore (GFull) ~ Language + Group + Language*Group + (1 | Subject) + (1 | Emotion). Recognition of full prosodic expressions did not differ overall by Language, but differed by Group (F = 5.59, df = 1, p = 0.02) and Group x Language (F = 28.73, df = 2, p < 0.001). Chinese listeners displayed significantly better recognition of Mandarin emotions from full utterances (Mandarin > English = Arabic), whereas Arab listeners displayed no advantage across languages for full utterances (Mandarin = English = Arabic). Emotional prosody in Arabic and English were recognized more accurately by Arab vs. Chinese listeners, whereas the opposite was true for Mandarin (Chinese > Arab, S6B Table).

Vocalizations promote faster emotion recognition

Our data provide within-subjects verification that listeners tend to be more accurate, but more notably, require substantially less time to detect basic emotions communicated by vocalizations than prosodic features of their native language. Listeners required approximately half the exposure time on average to form a stable impression of the speaker’s emotion from vocalizations (Mean = 417ms) than from native prosody (Mean = 765ms). Thus, it can be said that listeners in our study achieved an accurate sense of the intended meaning conveyed by vocalizations more efficiently, i.e., in a much shorter time period, than for speech prosody. In evolutionary terms, it is thought that vocalizations were functionally purposed to communicate rapid, ‘honest’ details to conspecifics about emotional events necessary for survival which require minimal conceptual elaboration [18,28]. Our findings similarly suggest that the primary communicative function and main ecological benefit of vocalizations is to transmit acoustic information that allows listeners to construct a categorical representation of the speaker’s emotion state in an expedited manner [23,60,61]. Reducing the latency of emotion recognition processes would accelerate adaptive action tendencies associated with discrete emotional signals [1], especially for long-range calls when visual cues about the behavioral context are obscured [62], allowing earlier predictions and reactions in the emotional situation [5].

Anger vocalizations (e.g., shouts) displayed an early detectability advantage and were almost fully recognized from 200ms bursts [45], extending claims that these vocalizations “maximize” the perceived strength and formidability of a speaker when compared to angry speech [6,63]. Increasing stimulus exposure to 400ms resulted in near peak accuracy for all vocalization types with the exception of sadness/crying (which improved somewhat beyond 400ms). These patterns underscore that structural differences within the first 200–400ms of vocalizations (beginning <200ms) provide adequate perceptual information to discriminate and form abstract categorical representations of the emotion qualities communicated by nonverbal acoustic signals [60,61]. Although conducting acoustic analyses was beyond the scope of our study, vocalizations are known to exhibit more nonlinear (aperiodic) acoustic features, and exploit a restricted portion of the acoustic space associated with “roughness” perception that is not used by speech [5,6,18], which could partly explain why these signals are recognized so rapidly. Calculation of EIPs confirmed that the functional significance of all vocalizations was firmly established within a narrow ~300–500ms time window [43] for both Chinese and Arab listeners, who displayed minimal performance differences—despite the fact that vocalizations were extracted from recording databases produced by speakers of various European languages. Arab/Chinese listeners only differed in their recognition latencies for happiness/laughter, although isolation points for this emotion were still rapid (300–400ms). These data bolster claims that nonverbal vocalizations possess robust ‘universal’ acoustic elements [64] that reveal their functional significance to perceivers 300–500ms post-onset of the vocalization. This process can be achieved without prior experience and shows little cultural variation, except possibly for positive emotions [11,12].

Effects of language on emotional prosody recognition

Emotional prosody was marked by group differences in accuracy, timing, and unique recognition trajectories in each native language, pointing to a more pronounced impact of linguistic and socio-cultural variables on how emotional representations are constructed in speech. When speech excerpts were limited (200–600ms), Chinese listeners displayed superior detection of anger and sadness from native prosody, while Arab listeners displayed a slight advantage for sadness [15,36,41]. The Arab group was notably less accurate to recognize emotions from native prosody when stimulus exposure was short (<600ms). Prosody recognition significantly increased and stabilized only when listeners were fully exposed to native pseudo-utterances (G600-GFull), yielding moderately high emotion hit rates in each group that are comparable to the literature (~ 3x chance accuracy level, e.g., [10,13]). Chinese listeners recognized all emotions with similar accuracy when all acoustic details were available, whereas Arab listeners were more accurate for certain emotions (happiness > anger = fear > sadness).

The timing (EIP) data show that emotions were isolated from prosody at vastly different rates, ranging from 300–1100ms in Mandarin compared to 800–1000ms in Arabic (EIPs in each language were always shortest for sadness and anger and longest for happiness, cf. [36,39,40]). An important conclusion that can be drawn from these data is that emotional prosody recognition is interdependent on language context and structure [13], as suggested by emotion dialect theory [7,46]. Results also show that refining impressions of a speaker’s emotional state from speech often benefits from phrase-final acoustic information [40–42]. In contrast, we found little evidence that event details beyond 600ms post-onset of emotional targets aided recognition of vocalizations for our paradigm.

Our data highlight two critical time periods which may be crucial for extracting details about vocal emotion expressions following event onset: an early window (~0–400ms), which promotes full recognition of nonverbal vocalizations and appears to allow rough differentiation of emotionality/highly salient emotional qualities communicated by speech prosody [3,21,31,43]; and a late extended window (400–1000ms+), which is necessary to monitor and consolidate cues that encode the emotional meaning of prosodic expressions as speech unfolds [36,38,39,41]. The late integration window also serves to incorporate semantic information relevant to the speaker’s emotion state [65,66] and considers phrase-final acoustic cues which impact on how emotional prosody is contextually interpreted [10,41,42]. This processing scheme ensures rapid bottom-up detection of the categorical relevance of motivationally salient vocal signals from acoustic-perceptual information in the early processing interval (including trait impressions of dominance, attractiveness, etc. [67]). At the same time, it explains the gradual emergence of stable prosodic representations in speech which tended to build up over longer sampling periods in our data. On average, the EIP data show that the emotional significance of speech prosody—at least for basic negative emotions such as anger, fear and sadness—is reliably established ~500–800ms post-onset of an utterance [36,38–40]. However, unlike vocalizations, these estimates vary considerably across items and languages and can be substantially longer when more socially-constructed emotions, including happiness, are studied (EIPs > 1–2 seconds) [39,41].

Theories of speech perception have proposed that auditory-perceptual integration of basic linguistic units (segments vs. syllables) is accomplished by distinct brain mechanisms that sample acoustic information over different time scales (25–80ms windows for segments, 150–300ms windows for syllables [68]). Along these lines, different forms of vocal signals (vocalizations, prosody) may reveal emotional meanings at unique timepoints owing to distinctly adapted procedures for sampling emotionally relevant acoustic variation over different time scales. Arguably, recognizing emotions in speech depends on a broader analysis window that roughly aligns with major perceptual units that promote linguistic comprehension, such as syllabic units [8,39]. As listeners appraise emotional speech cues, they must also distribute perceptual and cognitive resources to extract linguistic meaning from the acoustic signal; the conscious nature and complexity of this process may contribute to why listeners build more gradual representations of the speaker’s emotion state from prosody when compared to vocalizations.

Vocal communication serves critical functions that regulate human social dynamics, affecting group synchronization, cohesion and how social networks are formed and maintained [62]. Although vocalizations may act as effective long-ranging signals that propagate over distance and are decoded rapidly [5,62], emotional speech tends to serve more “short-ranging” functions selected to modulate within-group behavior and enhance individuating information [69,70]. Thus, humans undoubtedly learn that the antecedents of emotional events that are typically expressed in fluent speech are rarely as urgent as those signalled by vocalizations (see [20] for a discussion). Also, experience would reveal that it is more common for people to intentionally manipulate and misrepresent emotional cues to serve their own interpersonal goals [28,71]. Acquired knowledge of these form-function relationships could explain the more variable and sustained analysis period associated with emotional prosody recognition compared to vocalizations, as it is believed that the human capacity to communicate using language, and to communicate emotions in language, co-evolved [72].

On the ingroup advantage for emotional prosody recognition

Several variables are likely to shape how speech-embedded emotions, as well as volitional productions of vocalizations such as laughter [73,74], are assigned value over time and “recognized” in daily interactions; these include cultural preferences/display rules and inferences about the strategic “pragmatic” goals of the speaker in the context of emotional communication [28,75]. Of main interest here, there is mounting evidence that language experience familiarizes people with emotional dialects or vocal expressive “styles” used by particular groups [46], allowing more precise and more rapid construction of emotional prosody representations in listeners’ native vs. a foreign language (ingroup advantage [7]). Our study provides a unique view on this issue, as we employed a fully crossed presentation design involving participants who had no familiarity with the foreign language (Mandarin vs. Arabic) but who all had shared proficiency in English as a second language acquired from childhood, allowing graded effects of familiarity to be evaluated in each group for the first time.

Our findings replicated the ingroup advantage for only one of our two groups: Chinese listeners were more accurate and required less time to name emotions in Mandarin than the other two languages (native > foreign = L2-English, [10,47,76]). This relationship was established after hearing only 200ms speech excerpts and did not temporally evolve (cf. [36]). Presumably, Chinese listeners isolated emotional meanings more efficiently in Mandarin because these stimuli abided by culturally acquired norms of expression familiar to the participants; since performance in L2-English and Arabic (foreign) did not differ, this effect is unlikely to be traced to basic problems in phonological encoding for non-native stimuli.

In contrast, there was no evidence that Arab participants presented an ingroup advantage in accuracy or speed at any stimulus duration (see [77] for similar conclusions using a fully crossed design). Moreover, unlike the Chinese participants, performance of the Arab group evolved as a function of acoustic exposure in each language. When excerpts were brief (200–600ms), Arab listeners were unexpectedly superior in Mandarin for both accuracy and speed (foreign > native = L2-English). However, emotion recognition accuracy for full utterances was comparable in the three languages (Arabic = Mandarin = L2-English). These patterns suggest that Arab listeners relied heavily on utterance-final acoustic cues to form stable impressions of emotion in Arabic and L2-English, and ultimately, they achieved higher recognition rates in these two familiar language contexts than Chinese listeners. However, their path to recognition and how they dynamically integrated emotion-related cues in each language was clearly distinct.

Counter-intuitively, our results point out that emotional prosody may be recognized more efficiently at times in a completely unfamiliar language, overriding any advantages conferred by experience with emotional stylings or ‘dialects’ used by native speakers [46]. Although we strived to match our prosodic recordings in the three languages along key dimensions, it seems nonetheless clear that Mandarin speakers supplied representative acoustic cues to emotion that were more perceptually distinct at earlier stages in their productions than either the Arabic or English speakers, allowing recognition to stabilize more quickly in this language context for both native (Chinese) and foreign (Arab) listeners, irrespective of familiarity. This observation is at odds with claims that emotion recognition abilities are linked to the distance between the expresser and perceiver culture [7]. However, given the sustained time course of emotional prosody analysis, the (bottom-up) perceptual advantage we initially observed for Mandarin prosody disappeared for Arab listeners as acoustic information unfolded over Arabic and English utterances. These patterns remind us that the task of communicating emotions both within and across linguistic boundaries depends not only on top-down factors, i.e., acquired cultural knowledge and contextual expectations about a speaker’s vocal behaviour as described by Dialect Theory [46]—it is fundamentally driven by the dynamic quality of the input, i.e., how well speakers execute universally shared ‘affect programmes’ [78] that guide emotional speech recognition at different timepoints, and how well acoustic cues facilitate emotional “inference rules” shared by the speaker-listener [13]. In daily life, speakers encode emotion with different levels of intensity [79,80] and not all individuals are adept at strategically encoding recognizable emotion states in speech [8]. As this research moves forward, the dynamic interplay of stimulus-related features and various knowledge sources that are brought to bear on the act of emotional speech recognition, especially in the case of prosody, will come increasingly to light.

A key novelty of our report was that each perceiver group judged vocalizations and speech prosody in their native, second, and in a foreign language in a fully within-subjects design. While no definitive claims can yet be made, our findings supply no clear evidence that knowing a second language benefited emotional prosody recognition, despite the high English proficiency of our participants. Performance in the L2-English condition mirrored the pattern of foreign prosody (Chinese group) or native prosody (Arab group; cf. [29] for data on English and Hindi). Bhatara et al. [48] reported that L2-English proficiency in a group of French listeners had no effect on prosody recognition (for negative emotions) or seemed to interfere in this process (for positive emotions). Other studies suggest that emotional prosody recognition from single words is equally accurate and rapid in one’s native language and L2-English [49]. Our data call for continued monitoring of the relationship between L2 proficiency, emotion recognition, and how acquiring “prosodic-pragmatic competence” in a second language is influenced by different forms of instructional practice [81].