1.1. Vocal emotion recognition

Emotions are defined as “episodic, relatively short-term, biologically-based patterns of perception, experience, physiology, action and communication that occur in response to specific physical and social challenges and opportunities” [2, p. 468]. Emotions can be communicated in several ways, for example through vocalizations, i.e., vocal emotion expressions. The recognition of vocal emotion expressions depends on an interplay between different factors [3]. The person expressing an emotion produces different lower-level features (distal cues), which are registered by the listener and integrated into higher-level perceptual features (proximal cues). The higher-level features are then interpreted by the listener according to implicit rules (cognitive processing). The rules for interpreting high-level features are influenced by the individual’s knowledge and sociocultural context [3]. Furthermore, individuals’ psychobiological architecture, including for example the auditory system’s functioning, also affects vocal emotion expression and perception [3]. Vocal emotion expressions can be divided into verbal prosody and non-verbal vocalizations [4,5]. Prosody consists of modulations of acoustic characteristics such as frequency, amplitude, and duration that occur over longer timescales than the words in speech (suprasegmentally) [4,6], while non-verbal vocalizations consist of brief, non-linguistic sounds such as laughter, expressing happiness [4]. The expression of emotional prosody is likely affected to a high degree by socioculturally grounded norms of communication, while nonverbal expressions are mainly shaped by involuntary physiological changes [3].

Non-verbal vocalizations are generally recognized with higher accuracy than emotional prosody [3,7], but accuracy for different emotions varies for both emotional prosody [8–11] and non-verbal vocalizations [12–16]. The emotional prosody of anger, fear, sadness, and happiness have consistently been found to be recognized with above-chance accuracy and are seldom confused with other emotions [3,8–11]. Recognition of emotions in speech can also be supported by speech understanding, such that recognition is higher when the meaning of speech, in addition to emotional prosody, indicates the speaker’s intended emotion expression [17]. The ability to recognize others’ vocal emotion expressions play an important role in social interaction, and difficulties in vocal emotion recognition can therefore influence individuals’ participation and subjective well-being [1].

1.2. Acoustic characteristics of vocal emotions

Different emotions have different acoustic characteristics, and vocal emotion recognition depends on the listeners’ ability to perceive the acoustic features of emotions [3]. In speech, anger, happiness, fear, and surprise have, for example, been described as having high mean amplitude and relatively high pitch [10,11,18–20], while sadness has been described as having low pitch and low amplitude [10,19]. For non-verbal vocalizations, anger has been shown to have relatively low mean pitch, compared to several other emotions, while fear has low pitch variation and sadness has slightly lower mean intensity compared to several other emotions [12,16]. Some studies have also examined which acoustic parameters contribute to accurately recognizing emotions [e.g., 9,12,16]. For example, it was shown that accurate recognition of anger, happiness and fear in speech was positively associated with pitch (logarithmic F0) and loudness (estimated intensity) [10]. Another example is that recognition of the non-verbal vocalization of anger was found to be negatively associated with pitch, while recognition of fear was positively associated with pitch [12]. Examining which acoustic characteristics distinguish different emotion categories may provide further insights into vocal emotion recognition.

1.3. Prevalence and causes of hearing loss

Hearing loss is among the leading causes of disability globally [20,21]. It leads to difficulties in communication, which can contribute to social isolation and reduced quality of life [1]. It has been estimated that 18% of the population in Sweden have a hearing loss [22]. The prevalence of hearing loss increases with age and is around 24% in middle-aged-to-older individuals (ages 55–74) [22].

Hearing loss is commonly determined by the pure-tone average, which is measured across several frequencies) [23]. The specific frequencies included in measuring PTA varies, but it is common to include the frequencies of 500, 1000, 2000, and 4000 Hz, which is referred to as PTA-4 [23]. According to one description, degrees of hearing loss a PTA between 26–40 dB HL constitutes a mild hearing loss, between 41–60 dB HL a moderate hearing loss, between 61–80 dB HL a severe hearing loss, and >81 dB HL a profound hearing loss) [21]. The majority of individuals with hearing loss have mild-to-moderate hearing loss [21].

Hearing loss can occur by any damage to the auditory system [24]. The most common cause of hearing loss is peripheral sensorineural hearing loss, which refers to damages to the cochlea and the auditory nerve [25–27]. This commonly occurs due to the negative effects of aging on the auditory system, which is referred to as age-related hearing loss (presbycusis; [20]). Age-related hearing loss is particularly associated with elevated high-frequency hearing thresholds of 2000 Hz or higher [24].

1.4. Perceptual effects of peripheral hearing loss

Hearing loss leads to reduced hearing sensitivity [28]. Peripheral sensorineural hearing loss further leads to loss of dynamic range, reduced frequency selectivity, and poorer pitch perception [25,26]. The reduced hearing sensitivity makes sounds become less audible in specific frequency bands. A loss of dynamic range means the quiet sounds may be inaudible while loud sounds are perceived similarly as by normal hearing individuals. Reduced frequency selectivity makes it difficult to distinguish between sounds of different frequencies [25–26].

1.5. Hearing aids

The use of hearing aids is the most common intervention for individuals with mild-to-moderate hearing loss [21,29]. It improves their listening ability and verbal communication [21,29]. Despite its benefits, hearing aids does not normalize auditory perception [25,28]. Linear amplification (used mostly in older hearing aids) involves amplification of all sounds regardless of level. This restores the audibility of quiet sounds, however, loud sounds may become too loud [25]. Modern hearing aids use non-linear amplification and compression. This involves the selective amplification of quieter sounds, which resolves the problem of sounds becoming too loud [28]. However, evidence does not support hearing aids, including modern hearing aids, restoring frequency selectivity and pitch perception [25].

1.6. The impact of hearing loss on vocal emotion recognition

Vocal emotion recognition is reduced in individuals with hearing loss in general [30–34] including in mild-to-moderate hearing loss [31–33]. For example, poorer recognition for individuals with mild-to-severe hearing loss compared to individuals with normal hearing have been found in young and middle-aged individuals (ages 22–74; [31]) and in middle-aged to older adults (50–90 years old; [31,33].

Further, Goy et al. found that individuals with mild-to-moderate hearing had better recognition of sadness compared to anger, fear, and happiness, but not surprise [31].

While the studies presented here show that individuals with hearing loss, including mild-to-moderate hearing loss, have poorer vocal emotion recognition, questions remain regarding its causes. For example, low frequency hearing loss contributed only marginally to emotion prosody recognition in individuals with mild-to-moderate hearing loss, while high frequency thresholds did not contribute at all [30]. In addition, Goy et al. did not find a significant association between the PTA and emotion recognition in individuals with mild-to-moderate hearing loss [31]. However, Legris et al., examining recognition of non-verbal vocalizations, did find a significant negative association between PTA and recognition accuracy in this group [33]. As such, the association between hearing sensitivity and vocal emotion recognition in individuals with hearing loss might differ between emotions in speech and nonverbal expressions. Further, the evidence so far shows that the use of hearing aids does not normalize vocal emotion expressions, neither in speech [31,35], nor non-verbal vocalizations [33]. Goy et al [31] found small, non-significant, improvements in the recognition of some emotions, particularly fear, [31], and Legris et al. found overall improvements but with remaining significantly poorer recognition compared to normal hearing individuals [33]. It is unclear how important different factors, such as reduced hearing sensitivity, deficits in psychoacoustic abilities, and changes in cognitive processing are for explaining vocal emotion recognition deficits in individuals with mild-to-moderate hearing [32].

Several aspects of hearing aid signal processing may influence vocal emotion recognition. In the current study, the audibility of the sound provided by hearing aid amplification is investigated. This is achieved through linear amplification based on individuals’ hearing thresholds using the Cambridge formula [36]. The Cambridge formula aims to restore loudness perception across the entire speech spectrum for individuals with hearing impairment. An advantage of the Cambridge formula compared to other formulae for linear amplification (e.g., NAL-R) is that it provides slightly larger gains for higher frequencies (>2000 Hz; [37]), which could be particularly beneficial for individuals with age-related hearing loss.

Furthermore, the ability to recognize emotions is related to the processing of the acoustic characteristics of different emotion categories, therefore relating recognition as well as how different specific emotions are mistaken for other emotions in individuals with mild-to-moderate hearing loss to the acoustic characteristics of different emotions may provide further insight into vocal emotion recognition in this group.

1.7. Aims

In this study, we examine recognition accuracy for different emotions (i.e., which emotions are the easiest or most difficult to accurately recognize) and patterns of confusion (i.e., which emotions are mixed up when misrecognized) for individuals with normal hearing and for individuals with hearing loss (listening with and without linear amplification), and relate performance to acoustic analyses, the aims are to gain a deeper understanding of;

• how the effect of hearing loss affects emotion recognition (by comparing performance of participants with and without hearing loss), and;
• how emotion recognition is affected by linear amplification (by examining performance of participants with hearing loss using linear amplification).

The results regarding performance will be related to acoustics in order to examine whether vocal emotion acoustics provide clues to better understand the effects of hearing loss and amplification on the recognition of different emotions, in relation to known perceptual consequences of sensorineural hearing loss.

In the stage 1 registered report protocol for this article [38], we made several predictions based on the research that was available to us at that time. In line with general guidelines for registered reports and open science practices, we have not changed our predictions, and this article focuses on examining the predictions we originally made.

Based on the literature, we predicted that:

1. Individuals with hearing loss will have poorer recognition compared to normal hearing individuals for emotions expressed verbally, regardless of acoustic features and regardless of the use of linear amplification, and for non-verbal vocalizations when linear amplification is not used
2. Individuals with and without hearing loss will not differ in accuracy for non-verbal vocalizations when linear amplification is used (This prediction is contrary to the findings of Legris et al [33] of which we were not aware at the time the predictions were formulated (Stage 1 of the registered report)..
3. Vocal emotions, which are more distinct in terms of acoustic parameter measures, will be recognized with higher accuracy for both groups, but emotions that are distinguished mainly based on frequency parameters will be less accurately recognized by the hearing loss group.
4. The more distinct the frequency-related acoustic parameters of an emotion are, the better that emotion will be recognized when linear amplification is used compared to when not.
5. Patterns of confusion will differ between individuals with and without hearing loss for both verbally and non-verbally expressed emotions.

2.1. Participants

Using a 2 x 6 mixed design (group x emotion), 80% power, 5% significance level, correlation between repeated measures of 0.5 and with N=56 participants (a reasonable number of participants from a recruitment point of view), we would be able to detect an effect as small as η²=.02 (f=0.14); a next to small effect size. Power analysis was performed using G*Power version 3.1.9.7 [39].

Twenty-six native Swedish-speaking participants, aged 40–75, with mild-to-moderate, bilateral, symmetric sensorineural hearing loss (PTA4, based on 0.5, 1, 2, and 4k Hz, of 30–60 dB HL), who have been using hearing aids for at least one year, and 34 age-matched native Swedish speaking participants with self-reported normal hearing were recruited. We originally intended to recruit at least 28 individuals from both groups. Approximately equally many men and women were included in both groups. The audiometric profiles of participants with hearing loss and information about their hearing aids, were planned to be obtained through the audiological clinics, from participants who gave their consent. However, this was not practically solved, so instead audiometric profiles of all participants were obtained through air-conduction pure-tone audiometry by the first author. Since an association between general cognitive ability (G) and emotion recognition ability have been established [40], the subtest Matrices from the Swedish version of WAIS-IV, which is strongly correlated with G, was used for all participants [41]. Before being invited to participate in the study, interested individuals filled out an online questionnaire, including questions about health problems, hearing-aid use, different diagnoses, educational attainment, age, gender, and native language. The survey contained questions regarding whether individuals suffered from hyperacusis, neurological disorders affecting the brain (e.g., multiple sclerosis or epilepsy), severe tinnitus (which is perceived to cause impairment and disability), developmental psychiatric disorders (e.g., ADHD, autism spectrum disorders or intellectual disability), mood and anxiety disorders (e.g., social anxiety disorder or depression), the experience of great difficulties in identifying and describing one’s own emotions. In addition, individuals with hearing loss also responded to the question of whether conductive problems (exemplified by ear infections, damage to the ear drum, fluid in the ear, and otosclerosis) were the cause of their hearing loss.

2.2. Material and procedure

2.3. Analyses

2.3.1. Acoustic analyses.

For each recording of verbal and non-verbal emotions, the acoustic parameters of the Geneva Minimalistic Parameter Set (GeMAPS, [45]) were extracted using the OpenSmile toolkit v.2.3 [46]. The GeMAPS consists of a set of acoustic parameters that have been proposed as a standard for different areas of automatic voice analysis, including the analysis of vocal emotions. The parameters of GeMAPS have been shown to be of value for analyzing emotions in speech in previous research [45] and are described in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of the acoustic parameters of the GeMAPS as discussed in Eyben et al. [45].

https://doi.org/10.1371/journal.pone.0322867.t001

To control for baseline parameter differences between speakers’ voices, z-scores were calculated for each parameter and for each sentence, using the means and standard deviations of each speaker’s neutral voice [8]. For non-verbal vocalizations, the same procedure was followed, with the exception that the means and standard deviations of non-verbal vocalizations were used as reference.

3.1. Demographic data

Demographic data and hearing thresholds of the participants are presented in Table 2. The mean age of the hearing loss group (M=63, SD=8) was higher than for the normal hearing group (M=52, SD=8), t(39)=4.42, p<.001, d =1.38. Therefore, to control for the potential influence of age on group differences, age was included in all statistical analyses including group as a factor. The distribution of gender χ2=0.51, p=.48, and degrees of educational attainment, χ2=1.47, p=.69, did not differ between the two groups. Nor did they differ in estimated general cognitive ability (G), as measured through the Matrices subtest from WAIS-IV, t(39)=0.26, p=.79, d =0.08. In Fig 1, the average audiograms for both groups, based on air-conduction pure-tone audiometry, are presented.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Participant demographic data, hearing thresholds (PTA-4), and scaled points on the Matrices subtest from WAIS-IV.

https://doi.org/10.1371/journal.pone.0322867.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Audiograms for the normal hearing group and for the hearing loss group.

Mean air conduction thresholds for the normal hearing and normal hearing loss group, including 95% confidence intervals. HL = Hearing Level.

https://doi.org/10.1371/journal.pone.0322867.g001

3.2. Acoustic characteristics of emotions in sentences

In the sentences (Table 3), anger was characterized by a narrow formant 1 bandwidth, high loudness, and high alpha ratio (more energy in lower frequencies). Happiness was characterized by a high pitch and high formant 1–3 frequencies. Fear was characterized by low shimmer, high HNR, and high spectral slope 0–500 Hz. Sadness was characterized by low formant 1–3 relative energies (low in comparison to the other emotions). Lastly, surprise was characterized by loudness, and low HNR, low shimmer and low Spectral Slope 0–500 and 500–1500 (low relative to the other emotions).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Means of Z-scores (using each speaker’s neutral sentences’ means and standard deviations as reference) for acoustic parameters for different emotions expressed verbally, measured across speakers, divided by acoustic features (acoustic domains). Standard deviations presented within brackets.

https://doi.org/10.1371/journal.pone.0322867.t003

3.3. Acoustic differences between emotions in sentences

Findings from PCA showed that frequency-related and amplitude-related parameters generated one component each, while spectral-balance-related parameters generated two components. The frequency component comprised high values for pitch, Frequency Formants 1, 2, and 3, and low values for the Bandwidth of Formant 1. The amplitude component comprised high values for shimmer and loudness, and low values for HNR. From the spectral-balance-related parameters, the first component comprised high values for the amplitude of Formants 1, 2, and 3, while the second component comprised high values for Alpha Ratio, Spectral slope 0–500 Hz, Spectral slope 500–1500 Hz, and low values for Hammarberg index. Values for the four components, divided by emotion, are presented in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Means (standard deviations) for the four components describing acoustics of sentences, divided by emotion.

https://doi.org/10.1371/journal.pone.0322867.t004

To determine which emotions are the most distinct based on acoustic parameters, we calculated the differences between each emotion and all other emotions (first part of prediction 3, Table 5). The analysis revealed that surprise is the most distinct emotion. Additionally, we conducted a similar analysis focusing on frequency parameters, which showed that happiness is primarily distinguished by these frequency parameters (second part of prediction 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Differences between emotions expressed as differences of components’ mean values. The TOTAL section describes the total difference between emotions when considering all four components (by Euclidean distances).

https://doi.org/10.1371/journal.pone.0322867.t005

The multinominal regression analyses indicated that the components significantly contributed to the prediction of emotions (χ² = 64.37, p < 0.001, Nagelkerke R² = 0.34). When the frequency component alone contributed to discrimination, it was specifically for distinguishing happiness from both anger and sadness. In both cases, increased values on the frequency component increased the odds for happiness (B = 0.659, p = 0.050, OR = 1.93 and B = 1.081, p = 0.004, OR = 2.95, respectively). In the same way, the amplitude component alone contributed to distinguishing fear from both happiness and surprise. Increased values on the amplitude component increased the odds for fear (B = 0.825, p = 0.045, OR = 2.28 and B = 1.846, p < 0.001, OR = 6.34, respectively). The contribution of spectral-balance components was only present when used in combination with frequency or amplitude components. In general, higher values for spectral-balance components contributed to discrimination of anger and sadness from both fear and surprise (Spectral-Balance 1) and happiness from surprise (Spectral-Balance 2). Notably, surprise could always be distinguished (Table 6, see). For further details see SI Appendix in Supporting information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Odds ratios for significant coefficients in multinomial logistic regression models with each emotion as the reference emotion, using the four components from the PCA analyses as independent variables (only significant odds ratios are shown).

https://doi.org/10.1371/journal.pone.0322867.t006

3.4. Acoustic characteristics of non-verbal vocalizations

In Table 7, the means and standard deviations of z-scores for all acoustic parameters for the different non-verbal vocalizations is presented. With reference to which parameters have the highest or lowest z-scores for each emotion, in relation to the average for all other emotions, anger is characterized by low pitch, low formant 1–3 frequencies, low shimmer, and high loudness. Happiness is characterized by high jitter, high shimmer, a low Harmonics-to-Noise ratio, low Spectral slope 0–500 Hz, and high Spectral slope 500–1500 Hz. Fear is characterized by low jitter, a narrow formant 1 bandwidth, high Spectral slope 0–500 Hz, and low Spectral slope 500–1500 Hz. Sadness is characterized by high pitch, high formant 1 frequency, high formant 2 frequency (as is interest however which has the same value), and high alpha ratio. Finally, interest is characterized by a high formant 3 frequency, a broad formant 1 bandwidth, low loudness, high Harmonics-to-Noise ratio, low alpha ratio, high Hammarberg index, and high formant 1–3 relative energies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Means of Z-scores (using all of each speaker’s non-verbal vocalizations’ means and standard deviations as reference) for acoustic parameters for different non-verbal vocalizations, measured across speakers, divided by acoustic features (acoustic domains). Standard deviations are presented within brackets.

https://doi.org/10.1371/journal.pone.0322867.t007

3.5. Recognition accuracy

Descriptive data for recognition accuracy for different emotions, for both stimulus types, and with the hearing loss group listening with as well as without amplification is presented in Table 8. Included in the table is also Rosenthal’s pi (47) for the sake of comparison with other studies which included different numbers of response options. No participant aborted any trials in the emotion recognition task, therefore, there is no missing data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Proportions of accuracy for different emotions, divided by group, listening condition and type of emotion expression. Means (standard deviations) and Rosenthal’s proportion index (Π) are presented in separate columns.

https://doi.org/10.1371/journal.pone.0322867.t008

3.6. Effects of hearing loss

All main effects and interaction effects from the statistical behavioural analyses are summarized in S1 Table in Supporting information. For the analyses of sentences when the hearing loss group listened without amplification, there was a significant main effect of Emotion, F(4,152)=2.69, p=.033, η²_p=.07. Pairwise comparisons showed that accuracy for sadness and surprise was higher than for anger, fear, happiness, all p <0.001. There was a significant main effect of Group, F(1,38)=15.24, p <0.001, η²_p=.29. Accuracy measured across emotions was 18 percentage points higher for the normal hearing group. There was a significant interaction between Emotion and Group, F(4, 152)= 3.68, p=0.007, η²_p=.09. Pairwise comparisons showed that the hearing loss group had significantly lower recognition accuracy compared to the normal hearing group for fear, (p=.001), surprise, (p=0.011), and happiness, (p <0.001), but not for anger (p=0.469), or sadness (p=0.245). For the hearing loss group recognition of fear was lower compared to recognition of sadness (p <0.001), recognition of happiness was significantly lower than recognition of anger (p=0.022), sadness (p <0.001), and surprise (p <0.001), while for the normal hearing group recognition of happiness was significantly lower than recognition of surprise (p=.004). There was no significant effect of the covariate of Age, F(1,38)=0.31, p=0.578. η²_p=.009, but there was a significant interaction between Emotion and Age, F(4,152)=2.45, p=0.048, η²_p=.06. Because of this interaction we also performed the analysis without age as a covariate. In this analysis, there were still main effects of Group, F(1, 39)=19.54, p <0.001, η²_p =.33, and Emotion, F(4, 156)=22.38, p <0.001, η²_p =.37, but the interaction between Emotion and Group, F(4, 156)=2.32, p=0.059, η²_p =.06, was no longer significant. This was most likely due to the group difference for sadness being larger, p=0.048, a 10-percentage point difference when not controlling for age compared to 7 percentage points when controlling for age. Similarly, the difference for anger was larger, p=0.10, with an 11 percentage point difference when not controlling for age compared to 7 percentage points when controlling for age. In summary, the HL group had poorer recognition of emotions expressed in sentences compared to the NH group, regardless of whether they listened with or without the aid of linear amplification.

For sentences, when comparing the normal hearing group with the hearing loss group using linear amplification, there was a significant main effect of Group, F(1, 38)= 5.62, p=0.023, η²_p=.13. Accuracy was 10 percentage points higher for the normal hearing group. There was also a significant main effect of Emotion, F(4, 152)=2.49, p=0.045, η²_p=.06. Accuracy measured across groups for surprise was significantly higher than for anger, fear, happiness (all p <.001), and sadness (p=0.028) accuracy for sadness was higher than for anger (p=0.006) and happiness (p=0.025). There was no significant interaction between Group and Emotion, F(4,152)=0.90, p=0.464, η²_p=.023, and no significant effect of the covariate Age, F(1,38)=0.30. p=0.585, η²_p=.008. The differences between the normal hearing and the hearing loss groups for different emotions expressed in sentences, when not controlling for age, are illustrated in Fig 2A and Fig 2B. In sum, similarly to when the hearing loss group listened without linear amplification, they exhibited poorer overall recognition of emotions expressed in sentences when using linear amplification. However, the overall difference in recognition between the two groups was smaller, and there were no significant differences in accuracy for specific emotions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Recognition of emotions expressed in sentences and non-verbal vocalizations for the hearing loss group and for the normal hearing group.

Recognition of different emotions for both groups, with the hearing loss group (HL) listening to: A) emotions expressed in sentences without amplification; B) HL listening to emotions expressed in sentences with linear amplification; C) HL listening to non-verbal vocalizations without linear amplification, and; D) HL listening to non-verbal vocalizations with linear amplification Significant differences between groups (pairwise comparisons): *p <0.05, ** p <0.01, ***p <0.001. Error bars represent 95% Confidence Intervals.

https://doi.org/10.1371/journal.pone.0322867.g002

For the analyses of non-verbal vocalizations when the hearing loss group listened without amplification, there was no significant main effect of Emotion, F(3.84, 146.06)=2.05, p=0.090, η²_p=.05, but there was a significant main effect of Group, F(1,38)= 25.18, p<0.001, η²_p=.40. Accuracy was 22 percentage points higher for the normal hearing group. There was a significant interaction between Emotion and Group, F(3.84, 146.06)= 2.59, p=0.041, η²_p=.06. The hearing loss group had significantly lower recognition accuracy for anger (p<0.001), fear (p=0.003), sadness (p=0.022), and interest (p<0.001), but not for happiness (p=0.264). In the hearing loss group, recognition of happiness, was significantly higher compared with accuracies for anger (p<0.001), fear (p<001), sadness (p=0.005), and interest (p=0.025), while recognition accuracy of interest was significantly higher than anger (p=0.016) and fear (p=0.036). In contrast, there were no significant differences in recognition accuracy between emotions for the normal hearing group. The covariate of Age was not significant, F(1,38)=0.15, p=0.704, η²_p=.004. The interaction between Emotion and Age was nearly significant, F(3.84, 146.06)=2.451, p=0.051, η²_p=.06. To explore the meaning of this nearly significant interaction, we again performed the analysis without age as a covariate. The differences between groups for different emotions differed depending on whether age was included as a covariate or not. For example, when controlling for age recognition accuracy for interest was 24 percentage points lower for the HL group, but 17 percentage points lower when not controlling for age, and accuracy for sadness was 19% lower for the HL group when controlling for age, but 26 percentage points lower when not controlling for age. In summary, the HL group had poorer recognition of all specific nonverbal emotion expressions, except happiness, when listening without linear amplification compared to the NH group.

For non-verbal vocalizations when the hearing loss group listened with linear amplification, the main effect of Group was significant, F(1, 38)= 10.30, p=0.003, η²_p=.21, accuracy was 10 percentage points higher for the normal hearing group compared to the hearing loss group. The main effect of Emotion was not significant, F(3.74, 141.97)=0.76, p=0.547, η²_p=.020. The interaction between Emotion and Group was significant, F(3.74, 141.97)= 2.80, p=0.031, η²_p=.07. Pairwise comparisons showed that the hearing loss group had significantly lower recognition accuracy compared to the normal hearing group for anger (p=0.014) and sadness (p=0.002), but not for fear (p=0.180), interest (p=0.527) or happiness (p=0.857). Similar to what was the case for non-amplified listening, recognition accuracy of happiness for the hearing loss group was significantly higher than accuracy of anger (p<0.001) fear (p=0.026) and sadness (p<0.001), while recognition accuracy for interest was significantly higher than for anger (p<0.001), fear (p=0.025), and sadness (p<0.001). There were no significant differences in recognition accuracy between emotions for the normal hearing group. Neither the effect of the covariate of Age, F(1,38)=0.65, p=0.425, η²_p=.02, nor the interaction between Emotion and Age, F(3.74,141.97)= 1.06, p=0.376, η²_p=.03, were significant. The differences between the groups for different non-verbal vocalizations, without age as a covariate are presented in Fig 2C and Fig 2D. Thus, the HL group had poorer overall recognition of non-verbal vocalizations when listening with linear amplification, as was the case for listening without linear amplification. However, in contrast to when listening without linear amplification, the HL group did not have significantly poorer recognition of fear or interest when listening with linear amplification.

3.7. Effects of amplification

For the analyses of the effects of linear amplification on recognition of emotions in sentences, the main effect of Listening condition was not significant, F(1, 20)=3.10, p=0.093, η²_p=.13. The main effect of Emotion was significant, F(4, 80)=18.27, p <0.001, η²_p=.48. Pairwise comparisons showed that recognition accuracy for surprise measured across listening conditions was higher compared to, anger, fear, and happiness (all p <0.001), while accuracy for sadness was higher than for anger (p=0.014), fear (p=0.008), and happiness (p <0.001). The interaction between Emotion and Listening condition was significant, F(4, 80)=4.31, p=0.003, η²_p=.18. Pairwise comparisons showed that linear amplification significantly improved recognition accuracy for happiness (p<0.001, with a nearly 18-percentage point improvement), but not for anger (p=0.535), fear (p=0.204), sadness (p=0.102), or surprise (p=0.153). The effect of amplification on emotions expressed in sentences is presented in Fig 3A.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Effects of amplification on emotion recognition for the hearing loss group.

Recognition of emotions by the hearing loss group listening with and without amplification for: A) Emotion expressed in sentences, and; B)Non-verbal vocalizations. Significant differences between listening conditions (pairwise comparisons): *p <0.05, ** p <0.01, ***p <0.001. Error bars represent 95% Confidence Intervals.

https://doi.org/10.1371/journal.pone.0322867.g003

For the analyses of the effects of linear amplification on the recognition of nonverbal vocalizations, the main effect of Listening condition was significant, F(1, 20)=16.80, p <0.001, η²_p=.46, there was a 10-percentage point improvement in recognition accuracy with linear amplification. The main effect of Emotion was also significant, F(4, 80)= 14.20, p<0.001, η²_p=.42. Pairwise comparisons showed that accuracy for happiness was significantly higher than accuracy for fear (p=0.014), anger and sadness (both p <0.001), and that accuracy for interest was significantly higher than for anger (p <0.001), fear (p=0.009), and sadness (p=0.011). The interaction between Emotion and Listening condition was significant, F(4, 80)=2.66, p=0.038, η²_p=.12. Pairwise comparisons revealed that linear amplification significantly improved recognition accuracy for anger (p=0.004), fear (p=0.004), and interest (p=0.010), but not for sadness (p=0.480) or happiness (p=0.428). The effect of linear amplification on the recognition of emotions expressed non-verbally are presented in Fig 3B. In summary, for the HL group, linear amplification significantly improved the recognition of happiness expressed in sentences, the non-verbal vocalizations of anger, fear, and interest.

3.8. Differences in confusion matrices

Examining the confusion matrices, presented in Fig 4, we note that the patterns of confusion were similar between the two groups although the hearing loss group more frequently misrecognized different emotions. The most common confusion, for both groups, pertaining to emotions expressed verbally, was the confusion of happiness for surprise, 34% without amplification and 31% with amplification, respectively for the hearing loss group, and 23% for the normal hearing group. For non-verbal vocalizations the most common confusion, for both groups, was the confusion of sadness for fear, 26% without and 27% with amplification, respectively for the hearing loss group, and 7% for the normal hearing group. Thus, the difference between the groups in the study sample regarding confusions was more quantitative than qualitative, and the tendency was for linear amplification to reduce the frequency of confusions but not the patterns.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Confusion Matrices.

Distributions of responses for different emotions in percentages.

https://doi.org/10.1371/journal.pone.0322867.g004

4.1. Limitations

There are limitations to the present study including some departures from the stage 1 registered report. We were not able to include the number of participants suggested by our power analyses and it resulted in reduced power to detect small effects. The primary reason was that several individuals with self-reported normal hearing did not fulfill the strict study criteria for normal hearing and were therefore excluded. However, a sensitivity analysis performed in G*Power version 3.1.9.7 [39] indicated that we would still be able to show significance for relatively small effect sizes (η²_p ≥.032). To find more participants with normal hearing according to the study’s criteria, we extended the age span from 50–75–40–75. Consequently, there were more normal hearing participants in the lower age span and fewer in the higher age span compared to the hearing loss group. Because of this, age was included as a covariate in analyses including group as a factor. The need to include age as a covariate, most likely further reduced the ability to identify effects. Despite this, examination of obtained effects showed that we were able to identify the effects of relevance to our predictions. With larger sample size, one could potentially have identified effects of linear amplification for more emotions. Future studies should replicate and build upon the present study by including more participants to overcome the limitations related to sample size. In summary, we diverged from the original plans for a larger sample size and age-matched groups primarily to maintain our relatively strict criteria for normal hearing.

Furthermore, the number of nonverbal vocalizations included for each emotion in the recognition task was relatively few, limiting the ability to generalize about the recognition of different emotions expressed non-verbally. Due to the limited number of nonverbal vocalizations, we were not able to perform a PCA and nominal regression analyses, preventing us from drawing conclusions about which acoustic features significantly differentiate between emotions.

Another, minor deviation from the stage 1 registered report is that audiograms were not possible to retrieve from medical journals, and instead we included tone audiograms also for individuals with hearing loss in our testing procedure.

In addition, we did not include a neutral condition in the recognition task. In the validation study [44], which was used to select stimuli for this study, neutral sentences were used to familiarize the participants with the voices of each speaker.

Moreover, the definition of sensorineural hearing loss (i.e., type of hearing loss) is based on self-reports. This makes arguments regarding the links between recognition in the hearing loss group and frequency-related acoustic characteristics of different emotions less reliable, as diminished pitch perception and frequency-selectivity are characteristic of sensorineural hearing loss. Future research should evaluate sensorineural hearing loss based on clinical diagnosis rather than self-reporting. It is important to note, however, that the participants’ audiograms showed good low-frequency thresholds, suggesting that conductive losses did not significantly affect their hearing. Additionally, their audiograms showed a pattern of sloping high-frequency hearing loss, indicating a sensorineural hearing loss of presbycusis origin.

Finally, we examined the effect of linear amplification, which is different from what is used in most hearing aids. Since hearing aids usually include different forms of compression, the dynamics can be affected, and thereby also the hearing level and perceived prosody. Typical hearing aids may affect the ability to perceive emotions expressed in speech in a negative manner compared to linear amplification. Thus, inferences about the usefulness of common hearing aids are limited.