Limitations of common emotion recognition paradigms

Difficulty of emotion perception

Hypotheses

Based on the summarized emotion recognition literature, we formulated the following hypotheses to examine person-specific (age and sex of observer and actor) and stimulus-specific (valence and arousal of the displayed expression) variables as predictors for subjective and objective difficulty in emotion perception:

1. Hypothesis 1: Age and sex of the actor have an effect on the objective and subjective difficulty experienced by the observer. We hypothesized that observers would experience more difficulty when presented with expressions from older actors (due to reduced muscle control) as compared to younger ones, and from male actors (due to lower expressiveness) in comparison to female actors.
2. Hypothesis 2: Age and sex of the observer have an effect on the observer’s objective and subjective difficulty in emotion expression perception. We expected to observe difficulty following a quadratic function across age because of developmental and deterioration effects at the two ends of the age spectrum. We also expected that difficulty would be increased for male observers compared to female observers (due to the latter’s advantage in emotion recognition).
3. Hypothesis 3: Valence and arousal of the stimuli, as subjectively perceived by a participant, have an effect on subjective and objective difficulty in emotion perception of these stimuli. We expected to see negative quadratic relationships for both valence and arousal in subjective and objective difficulty.

Participants

In total 658 volunteers took part in an online survey in German. The study was approved by the Ethics Committee at the Charité –Universitätsmedizin Berlin and was conducted according to the principles of the Declaration of Helsinki. All participants provided written informed consent for participation. Participants were included if they were either female or male native German speakers and not undergoing psychotherapy or taking psychoactive medication (to exclude clinical groups with possible emotion perception deficits, e.g., in bipolar disorder, depression, schizophrenia; [45,46]) at the time of the study. No upper age restrictions were included in the initial inclusion criteria, but few data sets from participants older than 60 were collected. To avoid spurious age effects due to the high leverage of data points from a small number of the oldest participants (oldest 79) in the planned regression models, in the final sample we included only participants who were 60 years old or younger (see S1 Fig in the S1 File for a histogram of age in the final sample). The participants were informed that their participation (and completion of the experiment) would allow them to enter a raffle with 50-euro prizes. After the data collection, 3 participants were randomly selected and awarded the prizes (via bank transfers).

Materials

The present study used a set of 480 four-second long videos with 12 actors (6 male, 6 female, age range: 21–64, mean age: 35.92) in which an actor’s face, centrally presented on a grey background, changed from a neutral expression to display an emotion and then returned to the neutral expression. These videos were a part of a larger set produced for research purposes at the film studio of the Humboldt-Universität zu Berlin (for a detailed description, see [48]). The actors were provided with the target emotion label, situations in which it is likely to occur in real life, and physiological changes associated with it. The video clips were then validated by independent judges and yielded high emotion recognition rates and good believability. The emotional facial expressions in the subset used in this study spanned over 40 different emotion categories (Supplementary Material, S1 Table in S1 File), including Ekman’s six basic emotions and 34 complex emotions. These categories were selected in previous research based on their frequency in everyday life and their even distribution across valence and arousal [49] and in the present study they assured a wide range of emotional expressions across the stimuli set. We included both basic and blended expressions in order to ensure that our results would generalize beyond the basic emotions. However, the original labels (i.e., the target emotion actors were instructed to express) was not used in this study because we were not interested in “correctness” of emotion recognition, but rather in the difficulty of emotion perception.

Procedure

Testing was carried out in German on the soscisurvey.de platform [50]. Each participant rated 12 videos randomly chosen from the pool of 480 videos. First, participants rated valence and arousal on scales anchored by pictures of the respective Self-Assessment-Manikins [51]. Second, participants provided ratings on the Basic Emotions and Interest (BEI) scales: happiness, sadness, fear, anger, surprise, disgust, interest. These scales were chosen to represent readily understandable emotions spanning positive (e.g., happiness), negative (e.g., disgust), and neutral/ambiguous (e.g., surprise) valence, higher (e.g., anger) and lower (e.g., sadness) levels of arousal, and higher (e.g., interest) and lower (e.g., fear) complexity. Together, we judged these scales to be good candidates for dimensions which could approximate the larger emotion space. Finally, participants rated the subjective difficulty of making the BEI ratings. All ratings were given on continuous visual sliding scales ranging from “not X at all” to “completely X”, where X was the rated emotion word. Responses closer to “completely X” were reflecting higher perceived intensity of emotion X in the rated video. For the difficulty rating the extreme ends of the scale were “very easy” to “very difficult”. All rating responses were encoded between 1 and 101.

Measures

The dependent variables were self-rated difficulty (SRD) and objective difficulty (OD). The SRD measure was the subjective difficulty of making the BEI ratings provided by the participants. The OD measure was calculated as the Euclidean distance from an observer’s rating to the consensus of the population on a given video. The population consensus was calculated as the centroid in the seven-dimensional BEI rating space of each video. This distance therefore reflects how similar an individual observer’s perception of a particular video was to the average perception of that video. Because according to the literature discussed above (see 1.2.4) it seemed likely that demographic-based subgroups in the population have different consensus (e.g., females and males could exhibit in-group agreement but between-group differences; see 1.1.3), we had to first test whether observer sex and age predicted significantly the BEI ratings (we call this test a consensus check). For that, the BEI ratings served as dependent variables in models with observer sex and age as predictors.

Data analysis

Consensus check.

Prior to the hypotheses testing, we built a model for each BEI item with the rating as the dependent variable and observer age and observer sex as independent variables to check for their potential systematic effects, which was important for the calculation of the OD (see 2.4.). Eq 1 shows this for the happiness rating as an example. The happiness rating for a video k rated by observer i is estimated by an intercept β₀, equal to the mean happiness rating across videos and observers, plus a term β₁ for observer sex and a term β₂ for observer age. Additionally, the model contains a random effect , which adjusts the intercept β₀ for each video k and therefore accounts for the repeated occurrence of the videos.

(1)

Confirmatory analyses.

Hypothesis 1 was examined with two mixed-effects models (Eqs 2 and 3), one for each difficulty measure (OD and SRD) as the dependent variable. Fixed effect variables were actor sex and actor age. A random intercept for observer was specified to account for the repeated measurements of each observer i. Since the variables of interest were actor-related and each video stimulus was nested within actor, no random effect for the multiple appearances of video stimuli was included in this model. We instead assume that the fixed effects of actor sex and actor age already account for the common variance of a given video.

(2)(3)

In a similar way, two mixed-effects models (Eqs 4 and 5) were specified for hypothesis 2 with observer sex, observer age and squared observer age as fixed effects, and a random intercept term for the rated video . The simple and the squared observer age variables were mean-centered to reduce collinearity of these predictors. Again, a random intercept term for observer could not be easily specified due to fitting difficulties; however, we continue using observer age and sex, both nested under that variable. While this way of modeling does leave repeated measurements of the observers in the data, their influence should be accounted for by the fixed effects.

(4)(5)

Hypothesis 3 was examined with two mixed effects models using valence and arousal as linear and squared independent variables (Eqs 6 and 7). Random intercepts for video and observer were included in the model to account for the repeated measurements of observers and the repeated occurrence of rated videos. Because the squared terms (e.g., valence²) can be seen as interaction terms (valence x valence), all constitutive terms (e.g., valence) should be included in the models, but not interpreted [52]. For this reason, we first built the models without the squared terms, and then fitted models with additional squared terms. Valence and arousal entered the model as Z-standardized variables to reduce collinearity between the linear and squared terms and to ensure comparability of estimated coefficients. An additional multiple testing correction was applied for the subsequent inclusion and testing of the squared term (see next section).

(6)(7)

Consensus check

For the calculation of our objective difficulty measure (OD) it was important to know if observer sex and age significantly influenced their BEI ratings (see 2.4.).

The analysis revealed no significant effect of observer’s age. However, as expected based on the previous literature, statistical tests revealed an influence of observer’s sex on the rating dimensions. A significant influence of observer sex was found for all emotion dimensions except “happy” (Supplementary Material, S2 Table in S1 File). The coefficients of the observer sex term are negative, indicating that women rate these dimensions systematically lower than men do. Therefore, in all the planned analyses including observer’s sex as a predictor we use the OD measure calculated as the distance from a separate centroid for each sex (to reflect the sex-specific consensus in the BEI ratings). In the process of this analysis, 171 video clips were left with ratings by less than 3 male observers; These videos were excluded.

Confirmatory analyses

Table 1 shows models examining the relationship between the two difficulty measures and actor sex and age according to hypothesis 1. For SRD a significant positive effect was found for actor age, indicating that with actor age the perceived difficulty for observers increases. For the OD measure a positive effect was found for actor sex and age, showing that OD increases with increasing actor age and is higher for female actors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Mixed effects models for subjective and objective difficulty predicted by actor sex and age and a random effect for observer.

Exact p values are shown in the brackets next to the unstandardized estimates. 95% Confidence Intervals are shown in square brackets.

https://doi.org/10.1371/journal.pone.0269156.t001

Hypothesis 2 was tested with the models depicted in Table 2. A positive association of SRD with female observer sex and a negative quadratic effect for observer age (Table 2(1)) were found. The latter indicates that the youngest and oldest observers in the sample tended to rate items as less difficult than the middle age group. OD likewise showed a positive association with female observer sex but no significant association with observer age (Table 2(3)).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Mixed effects models for self-rated difficulty and objective difficulty predicted by observer sex and age and a random effect for video.

Exact p values are shown in the brackets next to the unstandardized estimates. 95% Confidence Intervals are shown in square brackets.

https://doi.org/10.1371/journal.pone.0269156.t002

Table 3 shows models estimating the influence of valence and arousal ratings on SRD and OD according to hypothesis 3. For SRD (Table 3(1)) negative squared effects of valence and arousal were observed. This means that on average stimuli rated on the low or high ends of the valence or arousal scales are reported as easier than stimuli with ratings towards the middle on these scales. Comparison of the coefficients for the squared valence and arousal terms indicate a steeper slope for valence than for arousal. In the OD model (Table 3(3)) a negative coefficient for the squared valence term was found, which indicates lower OD for extreme ratings on the valence scale in contrast to higher OD for ratings towards the center of the scale. However, the squared arousal predictor has a positive coefficient. This indicates a lower OD for stimuli rated as medium-level in arousal as opposed to stimuli rated towards the extreme ends of the scale.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Mixed effects models for self-rated and objective difficulty predicted by valence and arousal ratings and random effects for video and observer.

Exact p values are shown in the brackets next to the unstandardized estimates. 95% Confidence Intervals are shown in square brackets.

https://doi.org/10.1371/journal.pone.0269156.t003

Exploratory analyses

Exploratory analyses were performed to complement the confirmatory results and provide further grounds for interpretation of the data. Because of the exploratory nature no p-values are provided.

Difficulty measures in relation to valence and arousal.

The relationship of both difficulty measures with valence and arousal was plotted for mean values for each video clip (Fig 1) which provides a simple way of visualization that the mixed effects models do not allow for easily. We fit simple linear models to all four relationships (Supplementary Material, Eqs 12–15). Models including both a linear and a squared term for valence showed a good fit, indicated by adjusted R² values of 0.37 and 0.51 for dependent variables SRD and OD respectively (Fig 1A and 1B). We also fit models containing a linear and a squared term for arousal and SRD and OD respectively as the dependent variable to the data. However, the fit was poor as indicated by the very low adjusted R² values of 0.05 and 0.01. Graphs c and d in Fig 1 show these much less obvious patterns in the data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Plots visualizing the relationship of valence and arousal with both difficulty measures.

Data points are averages over videos. The ribbon depicts the 95% confidence interval. A clear negative curvilinear relationship can be seen between valence and SRD (a). A similar relationship exists for valence and OD (b), although, the curve is less symmetrical with the high valence region featuring the lowest OD values. For the arousal measure the data does not exhibit such a visually clear relationship with the OD or SRD measure; this is also indicated by the low adjusted R² values (c,d).

https://doi.org/10.1371/journal.pone.0269156.g001

Feature importance comparisons of all variables.

Feature importance scores are difficult to interpret in raw form but can be used as an intuitive guide to the relative impact of different predictors. Here, we present importance relative to the most important predictor (Fig 2), such that a score of 1 indicates the most powerful predictor, and a score of .5 indicates that a given predictor carries half the predictive power of that most powerful predictor. Feature importance calculations show that the valence rating is the strongest predictor for SRD (Fig 2A), followed by the happy rating with 74% of the importance of valence. Most other predictors improve accuracy by 56% (interested rating) to 42% (disgusted rating) as much as valence does. The exceptions are the predictors actor age (30%), observer sex (10%) and actor sex (7%), which seem to be of rather low importance for the SRD prediction. For the OD measure the best predictor is the happy rating, followed closely by the interest rating (93% of happy rating) and the valence rating (79% of happy rating). Most other predictors improve accuracy by about 62% (fearful rating) to 50% (arousal rating) of the happy rating predictor. However, the predictors of lowest importance are all person-specific variables: observer age (36%), actor age (31%), actor sex and observer sex (6%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

Feature importance ranking for the prediction of SRD (a) and OD (b). The importance is expressed as a proportion of the importance of the strongest predictor and displayed in decreasing order from top to bottom. For the SRD measure the valence and happy ratings show a high importance, while all of the observer- and actor-specific variables except observer age show a particular low importance. For the OD measure the happy rating is the most important predictor with the interest and valence ratings following up close in importance.

https://doi.org/10.1371/journal.pone.0269156.g002

Age and sex of actor

We predicted that both actor’s sex and age influence subjective and objective difficulty in emotion perception. In line with hypothesis 1 and previous research [29], we found that emotions displayed by older actors were rated on average as more difficult to judge (subjective difficulty) and that ratings for older actors were more dispersed (individual ratings farther away from the group’s consensus), which indicates more disagreement between observers (objective difficulty). On the other hand, an effect for actor’s sex was only found in the model predicting objective difficulty and, surprisingly, it yielded higher difficulty linked to female actors. While the literature describes greater facial expressivity in women [31,32], it is unclear how this relates to the observers’ difficulty in emotion perception. In a categorical emotions framework, greater expressivity should result in stronger expressions of the primary emotion, and therefore a clearer signal that can be more easily interpreted. However, in a multidimensional emotion perception framework greater expressivity may result in stronger expression of all the emotions (primary and secondary), which could render the signal more difficult or ambiguous to decode. Results of the additional exploratory models (Supplementary Material, Eqs 8 and 9) are in fact consistent with this possibility; They show that on average 23.93 more rating points (Supplementary Material, Table S3.1 in S1 File) were spent describing the expressions of female actors and that the standard deviation across ratings was 2.23 units higher for female actors (Supplementary Material, Table S3.2 in S1 File). This indicates that female actors were perceived by the observers as expressing a stronger but also more complex emotional signal.

Age and sex of observer

According to hypothesis 2, the observer’s age and sex should influence both subjective and objective difficulty. We reasoned that difficulty should follow a quadratic function to account for developmental [37–39] and aging effects [40,41] on both sides of the age spectrum. However, we found no quadratic effect of age on the objective difficulty and a negative quadratic effect for the subjective difficulty. Younger and older observers within the age range of our sample rated themselves as having on average less difficulty than the middle-aged observers, while in the objective difficulty model there was no effect for the same age coefficients. Although a difference in significance patterns between the models does not mean a significant difference between the results, this is consistent with the idea that younger and older people might overestimate their perceptual abilities, which has been discussed before in metacognition literature [60].

Female observers reported having more subjective difficulty in providing the Basic Emotions and Interest (BEI) ratings and also exhibited higher objective difficulty. The former falls in line with studies showing that males exhibit overconfidence, i.e., more confidence than justified by their own abilities, in stock trading [61], test taking [62] and the usage of technology [63]. It is therefore likely that the same effect carries over to the domain of emotion recognition, which may underlie the differences between men and women in our study. By contrast, it is unexpected that women also scored lower on the objective difficulty measure, since based on the literature pointing at the advantage of females in emotion recognition [42] we expected also less difficulties in this group in emotion perception. One explanation might be that female observers see more subtle signals in facial expressions (which have a greater role in a multidimensional perception framework than in recognition tasks) but that these perceived secondary signals may not be generally agreed upon in this group, thus resulting in more deviation from the general consensus (i.e., higher objective difficulty).

To further this observation, we conducted exploratory analyses (Supplementary Material, Eqs 9 and 10), which revealed that the standard deviation across rating scales was more than one unit higher for female than male observers (Supplementary material, Table S4.2 in S1 File). This hints at greater disagreement among females. Moreover, women assigned on average 2.57 more rating points on the BEI scales for the basic emotions (angry, happy, sad, disgusted, surprised, interested, fearful), which is the only subset of our stimuli where the “target” emotion’s label (according to the stimuli set) can be found in the rating scales (Supplementary Material, Table S4.3 in S1 File). This is in line with the literature that suggests a subtle advantage of women for emotion recognition [3,4]. Also, this rating pattern of greater mean response to target emotions with a higher standard deviation in females has been previously reported [11].

Therefore, our results in part reproduce previous findings (greater target emotion attribution for women) and in part introduce novel and unexpected findings (greater objective difficulty for women). Since the higher objective difficulty in females was manifested thanks to the multidimensional emotion framework introduced in this study, this should be an indicator that the traditional emotion recognition paradigms might not be sufficient for the detection of some of the perceptual differences between the sexes. Thus, future research should also include novel approaches (like the multidimensional emotion framework) to address the bigger picture.

Stimulus valence and arousal

We predicted an influence of valence and arousal ratings on both difficulty measures. For the subjective difficulty measure as the dependent variable, both valence and arousal exhibited a negative quadratic relationship, indicating that stimuli rated on the extreme ends of the valence and arousal scales were also perceived as less difficult. The standardized coefficients revealed that the influence of valence is almost four times stronger than that of arousal (Table 3(2)). In the objective difficulty model, valence showed again a negative quadratic relationship. Arousal, on the other hand, followed a positive quadratic function, meaning that high and low arousal expressions led to higher values on the objective difficulty measure, i.e., more disagreement between observers (Table 3(4)). However, the magnitude of the quadratic effects for arousal and valence are similar.

In related research, we found that arousal ratings are correlated with the displacement from a neutral face (e.g., distance between the placement of the corners of the lips in a happy expression and in a neutral expression; [64]). It has also been shown before that facial expressions of high-intensity emotional states are often misclassified in terms of valence (an expression arising from a positive experience perceived as one of negative valence and vice-versa) in the absence of further information, such as body posture [65,66]. What we observe in the model for the objective difficulty measure could be similar: There might be some level of arousal above which discrimination of facial expressions worsens. On the other hand, too little deviation of the face from a neutral expression may also result in low arousal ratings and potentially also provide an observer with little information on the emotional meaning, thus making it difficult to recognize the displayed expression.

Further data exploration on video clip averages (Fig 1) also revealed negative quadratic relationships between valence, arousal, and both difficulty measures, except for the positive quadratic relationship of arousal and objective difficulty (Fig 1D). Valence seemed to be more predictive in terms of explained variance of self-reported (37%) and objective difficulty (51%) than arousal (5% and <1%, respectively). Importantly, these results show that valence is the predominant predictor for both difficulty measures.

The asymmetry of the curves in Fig 1A and 1B shows that high valence stimuli are the least difficult, even less than stimuli from the very low end of the valence spectrum. It is known that within basic emotion paradigms stimuli from the happiness category show the highest recognition rates and often produce ceiling effects [3,67,68]. One could argue that this pattern might be an artefact of the BEI dimensions, where happiness is the only definite positive emotion (in contrast to angry, sad, fearful, and disgusted being negative, and interested and surprised being neutral/ambiguous). In this limited set, the range of choice for positive emotions is decreased which could be reflected in the observers’ difficulty perception (measured as self-reported difficulty) and also in the dispersion of overall ratings (objective difficulty). We believe, however, that the shape of the curves reflects the true relationship of valence and difficulty, because it is highly stable across various subsets of the data. For example, if the data are split into the basic emotions and the remaining “complex” emotions (thus removing pure “happiness” from the latter), the observed effect of valence holds in both subsets. Moreover, this effect persists even when only looking at stimuli of a single emotion category (as defined in the used stimuli set). For example, for the “surprised” category, which is not easily classified as either positive or negative, the individual stimuli are still distributed along the same curves. This strongly indicates that this effect is neither driven by the influence of individual stimulus categories, nor by the choice of dimensions in the BEI ratings, nor an interaction of those. It seems to rather reflect an underlying phenomenon. Further research is needed to understand the meaning and causes of this effect in more detail, but it indicates an interesting sort of nonlinearity in the detection of emotions with extreme valence.

Effect sizes and feature importance of predictors

Marginal R² values express the variance explained by the fixed effects. The marginal R² values in all models testing person-specific variables for their influence on both difficulty measures (Tables 1 and 2) were close to zero and thus represent very small effects [69]. Conversely, marginal R² values for models with valence and arousal (Table 3) were in the range of moderate sized effects. Together, these imply a limited influence of person-specific factors and larger effects of stimulus-specific factors on emotion perception.

Similarly, the feature importance analyses (Fig 2) showed that the best predictors for subjective difficulty were ratings of overall valence and the scale “happy”, whereas for objective difficulty, overall valence, “happy”, and “interested” dominated, emphasizing once again the importance of a general pleasure dimension for emotion perception. For both difficulty measures, the person-specific measures showed low importance with the exception of observer age in the subjective difficulty model. This implies that the impact of these person-specific features may be swamped by the difficulty differences between emotions, but that observer age may still have an important impact on how difficult emotion ratings are perceived.

The rating of interest was the third strongest predictor for subjective and second strongest predictor for objective difficulty. The interest dimension may be selectively indicating expressions of “social emotions”, i.e., emotions that are directed at another person, in contrast to those which do not require the presence of an interaction partner. Social emotions are usually more subtle in their display and more dependent on context, which may make them more difficult to evaluate. Future work should investigate whether social emotions are in general more difficult and whether they can be separated by ratings of interest from non-social emotions to confirm this. Another explanation for the high importance of the interest dimension might be that interest ratings also differentiate between negative and positive emotions and thus act as a proxy for emotion valence. In fact, in our data valence and interest ratings are moderately (r = .40) and strongly (r = .63) correlated on the individual rating and video level, respectively (Supplementary Material, S5 and S6 Tables in S1 File).

Limitations and future directions

One potential limitation of this study is the sample size and the ratio of both sexes in relation to the calculation of the objective difficulty measure. We calculated the ground truth for each video clip from our sample of observers by averaging all ratings for a clip within groups of males and females, which demonstrates how the difficulty of emotion perception can be captured in a multidimensional emotion framework and without assigning an a priori ground truth. Nonetheless, the precision of this estimate is dependent on the sample size. Due to our study design each individual video clip was only rated by a small group of the total observer pool: On average 11 observers rated a video clip, which was further broken down for some analyses into male and female observers. However, because we were only interested in effects across videos, a low observer count on individual clips should not systematically bias our results. In fact, the decision to have observers randomly distributed over many video clips instead of having all observers rate the same few video clips adds to the generalizability of our results as they were observed over a wide array of emotions and actors. Thus, we opted for maximizing external validity of our design and results. Nevertheless, one could argue that because each participant rated a subset of the stimuli, this could introduce inter-subject bias in the data. While we cannot eliminate this possibility entirely, we argue that our design is suited for maximizing the external validity of the study and that such a bias is likely limited and accounted for in our results. Specifically, we considered maximum random effects structure in our models (including random slopes nested within subjects). However, these models did not converge. A likely reason for this is that, as we assume and explain for each hypothesis, the fixed effects already account for that variance (i.e., inter-subject bias), thus rendering our models suitable for testing the stated hypotheses (for a detailed comment, see Supplementary Material).

Further, one could argue that the concept of difficulty in our study is task-dependent due to the limited number of rating dimensions and as such cannot be generalized beyond the stimuli set used here. In this view, the difficulty experienced by the participants in the task is a consequence of having to “deconstruct” a given (sometimes complex) emotion expression into maximally seven proposed dimensions (the BEI ratings).

Although we cannot rule out for certain that with the chosen dimensions we covered only a limited part of the emotion space, it is likely that this part is rather broad as it has been previously shown that merely four dimensions (valence, dominance, arousal, unpredictability) can explain 75% of the variance in a space of 144 categories [9]. As long as the seven BEI ratings are not all limited to a particular niche part of the emotion space, which seems unlikely, our results should generalize over different stimuli sets. Moreover, even if some part of the experienced difficulty could be explained by the characteristics of the task, it is important to note that it was the same across participants: All were equally likely to be exposed to potentially less (basic) or more difficult (complex) emotions. Since the variables of interest in the study were person- and stimulus-specific across different emotions (more and less difficult), our results and conclusions are informative regardless of whether the task itself could add to the perceived difficulty in emotion perception.

Another possible limitation is the use of only one positive rating dimension–happiness. This could potentially influence the ratings of positive videos, which would all likely include high-intensity choices on the happiness scale. By contrast, the negative videos would be rated higher on the remaining non-positive six dimensions. Although it is true that our BEI ratings include more of obviously negative (sadness, anger, fear, disgust) than positive (happiness) emotions, there are two emotions with non-obvious valence (surprise, interest). In fact, the positive videos in this study were rated as combinations of happiness, surprise, and interest in various intensities. Moreover, even if the unbalanced dimension set in terms of valence would have an effect on the ratings between positive and negative videos, our analyses of the effects of person-specific predictors on emotion perception difficulty would not be influenced by this, as the stimuli of different valence were equally distributed across participants. Finally, the stimulus-specific analyses (of perceived valence and arousal), though exposed to this potential limitation, showed a largely resisting pattern of results across different subsets of the data (as explained in 4.3.). Together, these suggest that the observed effects in this study are not substantially influenced by the limited number of positive dimensions in the ratings. Nonetheless, future research should include more of clearly positive emotions as rating dimensions to reject the possibility that the chosen set could considerably affect the observed results.