2.1. Traditional machine learning models of emotion representation

The development of systems equipped with emotional intelligence may be traced back to the early stages of computer science. Early explorations in this field were mostly theoretical, as researchers considered the possibility of algorithms capable of understanding human emotions [26]. An important milestone in this process was the creation of ELIZA [27], a software project designed in the 1960s with the ability to simulate a dialogue with a psychotherapist [28]. The 1990s witnessed the emergence of affective computing [29] as an acknowledged field, with Rosalind Picard’s significant contributions at the MIT Media Lab further shaping its development. Picard et al. [30] developed algorithms to correctly identify human emotions by analyzing facial expressions, voice modulations, and physiological signals.

An extensive array of models for the representation of emotions has been previously presented [4,5]. These models often classify emotions into predetermined and distinct categories, such as anger, anxiety, fear, happiness, and sadness [6]. While these methods offer a direct perspective on emotions, they are inadequate for capturing the subtle and complex aspects of human emotional experiences [7–9]. The discrete categorization fails to capture the dynamic and intricate nature of emotions, which are better understood as including many aspects such as valence, activation, and control [10–13]. In response to the limitations of category models, dimensional models have emerged as a viable alternative, proposing that emotions may be represented over a range of continuous dimensions [14,15]. Nevertheless, even these multidimensional models may not fully encompass the interconnectedness of emotions and the influence of external and internal factors on emotional experiences [16,17]. The study conducted by [31] provides a comprehensive overview of the present condition and progress made in dimensional emotion models. The work of [32] examined both deep learning and machine learning models for the classification of six primary emotions.

2.2. Deep learning-based emotion recognition techniques

Over time, significant advancements have been made in this field. Noteworthy accomplishments encompass the integration of EI into virtual assistants [33,34], social robots [35,36], and mental health support tools [37,38]. The emergence of deep learning and neural networks has greatly expedited the discipline, facilitating more accurate identification of emotions and implementation of response systems. In general, the shift from rule-based systems to artificial intelligence equipped with emotional intelligence has been marked by increasing complexity and capability. As this discipline advances, the possible applications and outcomes of emotionally intelligent artificial intelligence are vast and are bound to impact the course of technological advancement and human-computer interaction [39,40].

The study [41] proposed a multi-tiered hybrid representation approach that combines superficial visual characteristics with deep semantic elements to accurately identify emotions in images. A separate investigation [42] was carried out to predict multi-modal dimensions emotional states using the AVEC2017 dataset. To implement an emotional feature learning model, this framework utilized Bidirectional Long Short-Term Memory (Bi-LSTM). Additionally, it can effectively consider the impact of previous and following emotional characteristics on the present results. In their study, Chowdary, Nguyen, and Hemanth [43] undertook a thorough examination of deep learning-based methods for recognizing facial emotions. The evaluation included many deep learning paradigms, including Convolutional Neural Networks (CNN), Residual Networks (ResNet), and MobileNet, that have demonstrated remarkable performance in the field of face emotion identification. In their study, Zhang et al. [44] introduced a pre-training framework called EmotionCLIP. This framework aimed to extract visual emotion representations from both spoken and non-verbal interactions. By employing topic-aware contextual encoding and emotion-guided contrastive learning, EmotionCLIP directed the model to focus on both non-verbal and vocal emotional cues. The study [45] introduced a comprehensive speech emotion representation model called emotion2vec. The emotion2vec model had achieved superior performance compared to the benchmarks on the IEMOCAP dataset [46]. The emotion2vec model had shown continuous improvement in emotion identification across datasets in 10 different languages. Finally, the systematic review [3] covered emotion recognition technologies, which specifically examined research on the identification of emotions using physical signals, physiological signals, and eye movement tracking. The work of [47] investigated the application of deep learning techniques for classifying emotions in short texts. Then, Chutia and Baruah [48] presented a systematic literature review of the literature published between 2013–2023 related to text-based emotion classification. Qian et al. [49] conducted a comprehensive evaluation of ten different state-of-the-art deep learning models for emotion classification using a facial expression dataset.

Emerging in the field of emotion recognition are applications based on GNN. For instance, Gao et al. [50] introduced a graph reasoning-based framework for emotion identification. The proposed methodology utilized Graph Convolutional Neural Networks (GCN) and Long Short-Term Memory Networks (LSTM) to collect and analyze data derived from physiological signs, behavioral patterns, and social interactions. Khalid and Sano [51] investigated the application of social graph networks in predicting emotions. They employed graph structures derived from communication records to assess the interactions between users. Their results emphasized the importance of social interactions in forecasting emotional conditions. Recently, Liu et al. [52] performed a comprehensive analysis of the growing practice of using GNNs for emotion identification based on Electroencephalogram (EEG) data. Building upon a unified framework for graph creation, this study analyzed and classified existing approaches and proposed potential directions.

Recent studies have explored hybrid paradigms for multimodal emotion recognition, with two prominent directions: dynamic graph construction and adversarial adaptation. UA-DAAN [53] proposes an uncertainty-aware dynamic adversarial adaptation network for EEG-based depression recognition. It leverages adversarial learning to align feature distributions across domains, focusing on single-modality (EEG) domain shift mitigation. However, it requires specialized adversarial modules that increase model complexity and computational cost. WDANet [54] introduces a Wasserstein distribution-inspired dynamic adversarial network for cross-domain EEG emotion recognition, adopting similar adversarial alignment but still limited to single-modality scenarios. Both methods prioritize domain adaptation over unified spatial-temporal modeling, making them less suitable for multi-modal sequential data. Another notable recent advance is EmoSavior [55], which focuses on depression recognition and intervention via multimodal physiological signals and large language models (LLMs). EmoSavior integrates LLMs to enhance semantic understanding of emotional cues, particularly in clinical contexts, and leverages multimodal physiological signals for comprehensive emotion modeling.

Bearing the above-mentioned analysis in mind, two types of deep learning models were utilized as based models in this study. First, the GAT [20] is taken as the base graph component, modified with enhanced attention weights (E-GAT) to prioritize emotionally salient features. Second, the Bi-LSTM [56] model follows the standard architecture with two hidden layers, used for modeling temporal dependencies in sequential data.

3.1. Graph model construction

3.1.1. Nodes-emotions.

The proposed graph model is created by each emotion state being represented as a node in a graph, as shown in Fig 2. Various emotional emotions included in text, audio, picture, and video samples can be encapsulated by these nodes. Emotions such as happiness, sorrow, rage, and fear are common components in a fundamental emotion recognition model. More sophisticated models may incorporate a wider spectrum of emotions or multidimensional representations such as valence and arousal. Each node possesses certain attributes that incorporate the information of its matching emotion state.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Overarching structure of the proposed graph model (E-GAT).

Interpretability details: 1) Nodes: Represent emotional states (text, audio), with attributes encoding feature vectors (red dashed lines link nodes to feature vectors); 2) Edges: Blue solid lines represent dynamic relationships between emotional states; 3) Weights: Edge weights indicate relationship strength—higher weights mean stronger correlation; 4) Temporal Adaptability: Black dashed lines denote feedback loops, illustrating that emotional states evolve over time.

https://doi.org/10.1371/journal.pone.0339946.g002

As seen in Fig 2, the proposed graph model is constructed by representing each emotional state as a node in a graph. These nodes can depict worldwide emotional emotions conveyed through several forms of media including text, voice, photos, and video. A standard emotion recognition model may consist of nodes representing emotions such as happiness, sorrow, anger, and fear. Elaborate models may include a wider range of emotional states or integrate dimensional representations such as valence and arousal. It is crucial to acknowledge that every node possesses unique characteristics that encode the specific emotional state it reflects.

3.1.3. Weights-strength of relationships.

The weights assigned to the edges in the graph model are crucial for representing the strength of relationships between different types of signals within the multi-modality dataset. These weights are calculated based on the differences between the features extracted from each modality, reflecting the intensity of the correlation or the likelihood of transitioning from one emotional state to another. The calculation of these weights can be formalized as follows:

Given a set of nodes V representing different modalities, and an edge e_ij connecting nodes i and j, the weight w_ij of the edge is determined by the inverse of the distance between the feature representations of the two modalities. As shown in Eq. (1), this can be mathematically expressed as:

(1)

where f_i and f_j are the feature vectors of the signals represented by nodes i and j, respectively, and is a distance metric, such as Euclidean distance, that measures the dissimilarity between the two feature vectors. This approach ensures that edges connecting nodes with more distinct features have higher weights, indicating a stronger relationship in the context of emotion representation.

To address the limitation of Euclidean distance in capturing nonlinear interactions, we introduce a learnable metric layer that integrates both feature similarity and semantic correlation. Specifically, the edge weight between node i and node j is calculated as (as shown in Eq. (2)):

(2)

where:f_i, f_j denote the feature vectors of i and j; and are learnable parameters (with d being the dimension of feature vectors), used to model nonlinear semantic correlations between modalities via the sigmoid activation ; is the cosine distance to capture normalized feature similarity; is a small constant to avoid division by zero.

3.2. Graph attention network

As shown in Fig 2, the GATv2 model [57] is exploited to extract information from the graph-structured data, where nodes represent different modalities and edges represent the relationships between them.

As provided in Eq. (3) and Eq. (4), the GATv2 model can be described by the following equations:

(3)

(4)

where is the attention coefficient between node i and j for the k -th attention head, a^k is the learnable attention parameter, W^k is the weight matrix for the k -th attention head, h_j is the feature vector of node j, and is the set of neighbors of node i. The activation function is applied to the aggregated features to produce the output features for each node. The GATv2 model uses attention mechanisms to weigh the importance of different nodes’ features, allowing the model to focus on the most relevant information when making predictions about emotional states.

In general, the graph model (E-GAT) used in this study adopts an undirected semantic graph structure. Each node is characterized by a feature vector with a shape of (300,). The graph exhibits a variable input shape that adapts to the length of the input text: for tweets in the SemEval-2018 dataset, the number of nodes N ranges from 10 to 128, and during batch processing, all graphs are padded to a uniform structure with a maximum of 128 nodes per sample to facilitate consistent model training. The E-GAT architecture consists of two stacked GAT layers: the first layer takes an input dimension of 300 and maps it to an output dimension of 128, incorporating 8 attention heads and a dropout rate of 0.2, while the second layer accepts an input dimension of 128, transforms it to an output dimension of 64, uses 4 attention heads, and also maintains a dropout rate of 0.2. Finally, the graph model outputs node embeddings with a shape of (N, 64), and these node embeddings are averaged to generate a graph-level feature vector with a shape of (64,)—this vector serves as the input for integration with the subsequent Bi-LSTM module to support further temporal feature learning and emotion recognition tasks.

3.3. Bi-LSTM model

As shown in Fig 3, the Bi-LSTM model [58] is adopted to enhance the GATv2 component. The Bi-LSTM model processes the multi-modal emotional data by extracting relevant features from each modality. The original data is transformed into a format suitable for the graph model. For instance, audio signals are converted into spectrograms to represent their frequency components over time, while images are transformed into sequences of facial landmarks, and text data is tokenized into sequences of words.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. The architecture of the introduced Bi-LSTM model.

This component consists of LSTM models, fully-connected layer, and softmax layer.

https://doi.org/10.1371/journal.pone.0339946.g003

As shown in Eq. (5) and Eq. (6), the Bi-LSTM model can be described by the followings:

(5)

(6)

where h_t is the hidden state at time step t, x_t is the input at time step t, W_xh and W_hh are the weights connecting the input to the hidden layer and the previous hidden state to the current hidden state, respectively, and b_h is the bias term. The output layer y_t is calculated by multiplying the hidden state by the weight W_hy and adding the bias b_y. The Bi-LSTM model captures both the forward and backward contexts in the sequence data, providing a comprehensive view of the temporal dynamics within each modality.

In general, this work utilizes an enhanced GATv2 model by integrating Bi-LSTM model to effectively extract information from data that is naturally organized as a graph. The proposed methodology is very suitable for accurately identifying the nuances included in emotional datasets. The stochastic gradient descent (SGD) [59] is employed throughout the training process to effectively decrease the loss function, thereby enhancing the performance of the model. The choice of the loss function is tailored to the specific characteristics of the emotion detection task. As shown in Eq. (7), the cross-entropy loss is used in this study, which is defined as:

(7)

where y_i is the true label and p_i is the predicted probability for class i. The loss function guides the model to minimize the difference between predicted and actual emotional states, thereby enhancing the model’s performance in emotion recognition.

The Bi-LSTM model is configured with two hidden layers featuring bidirectional connections, which utilize tanh activation functions and sigmoid recurrent gates; its shape parameters are defined as follows: the input shape is specified as (, , ) = (32, 128, 64), where corresponds to the maximum number of graph nodes and input_dim matches the output dimension of the E-GAT module, the hidden layer dimension is set to 128 units per direction (resulting in a total of 256 units per layer), and the output shape is (, ) = (32, 11) with 11 representing the number of emotion classes for the SemEval-2018 dataset. For training parameters, the model is trained for 50 epochs with an early stopping mechanism that activates if the validation loss plateaus for 5 consecutive epochs, employs a dropout rate of 0.2, uses the Adam optimizer with a learning rate of 1e-4 and weight decay of 1e-5, and adopts categorical cross-entropy as the loss function for single-label classification tasks or binary cross-entropy for multi-label mixed emotion classification tasks.

During the training process, the proposed model experiences iterative improvement. This is accomplished by entering the feature vectors linked to each node and adjusting the model’s parameters to reduce the difference between expected and resultant results. The process involves the distribution of information across the nodes and edges, which is crucial in effectively representing the complex interaction among different emotional states. The use of this approach enables the model to acquire knowledge from the interrelatedness of emotional expressions and experiences, therefore improving its capacity to accurately identify and categorize emotions.

4.1. Datasets

This work employed the SemEval-2018 dataset [60] to categorize emotions conveyed in textual data. This dataset consists of tweets collected between 2016 and 2017. The collection includes tweets that correspond to 11 distinct emotional categories: anger, anticipation, disgust, fear, pleasure, love, optimism, pessimism, sorrow, surprise, and trust. The dataset includes texts in three distinct languages; however, for the purpose of this study, only the tweets written in English were used for analysis. The present work integrated the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVEDESS) [61] to analyze audio and face expression data. The repository is a well-established collection of multimodal content including emotive speech and song performances by 24 professional performers. In order to achieve a high level of emotional authenticity and establish strong test-retest intrarater reliability, the actors deliver lexically-matched utterances in a neutral North American accent. The dataset is well recognized for its usefulness in analyzing emotional expressions‌‌ in both spoken and musical forms, rendering it a valuable asset for research in the domain of affective computing.

To further verify the model’s ability to integrate text, audio, and vision simultaneously, we introduce the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [62]. This dataset includes 32,285 video clips extracted from YouTube, with each clip annotated for emotional valence and arousal. For consistency with our emotion classification task, we map the continuous valence labels to 5 discrete categories: strong negative, weak negative, neutral, weak positive, and strong positive. We use the official train/validation/test split (80%, 10%, and 10%) and extract features for each modality: text, audio, and vision. The text is cleaned, truncated or padded to 128 tokens and encoded by BERT-base-uncased into 768-dimensional embeddings with mixup augmentation (=0.2); the audio is resampled to 16 kHz, converted to 40 × 100 spectrograms, augmented with 15 dB Gaussian noise, and flattened into 4 000-dimensional vectors; the vision features are extracted from facial landmarks via ImageNet-pre-trained ResNet50, yielding 2 048-dimensional vectors after ±10 % random cropping and horizontal flipping with coordinates normalized to [0,1]; hyper-parameters are tuned through 10 independent runs of 10-fold cross-validation on the training set, and early stopping on the validation set (5 epochs).

In addition, to evaluate the robustness of the proposed approach, we add Gaussian white noise (SNR = 5dB, 10dB, 15dB) to the audio signals of RAVDESS to simulate real-world background noise.

4.2. Evaluation metrics

The investigation evaluates the effectiveness of the proposed graph model using accuracy and the F1-score as the main metrics, as shown in Eq. (8) and Eq. (9). Traditional in classification tasks, these metrics provide a holistic assessment of the model’s performance. Accuracy quantifies the ratio of accurate predictions to the total, while the F1-score combines precision and recall to provide a single measure of accuracy that considers both false positives and false negatives.

(8)

where TP denotes true positive, TN true negative, FP false positive, and FN false negative are respectively represented in the context.

(9)

where precision and recall are provided as following equations Eq. (10) and Eq. (11):

(10)

and

(11)

To note that the 10-fold cross-validation strategy was exploited to evaluate the proposed approach in a robust way.

4.3. Experimental results

4.3.1. Experimental results on the SemEval-2018 dataset.

The findings obtained from our graph-based approach are really promising. When the proposed approach is used to the SemEval-2018 dataset, Table 1 displays the results obtained in the field of textual emotion categorization.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The outcome of the proposed model on SemEval-2018 dataset (%).

https://doi.org/10.1371/journal.pone.0339946.t001

Table 1 presents the detailed performance of the proposed model across 11 distinct emotion categories in the SemEval-2018 dataset. From the results, the model demonstrates robust overall performance, with an average Accuracy of 62.4%, average precision of 71.9%, average recall of 71.6%, and average F1-score of 71.7%. Notably, the model achieves the highest accuracy in recognizing “Trust” (68.2%) and “Optimism” (67.7%), which may be attributed to the clear semantic patterns and relatively distinct vocabulary associated with these positive emotion categories, allowing the graph-based approach to effectively model inter-word dependencies. For emotionally nuanced categories like “Disgust” (accuracy: 54.5%) and “Sadness” (accuracy: 55.2%), the model still maintains strong precision (74.1% and 75.3%) and Recall (73.8% and 74.4%), indicating that while fewer samples may be correctly classified overall for these categories, the model minimizes both false positives and false negatives when making predictions. Additionally, even for categories with moderate accuracy, the F1-scores remain above 68%, highlighting the model’s balanced performance between precision and recall.

To address the lack of statistical validation for our model’s performance advantage over state-of-the-art methods [63–66], we conducted paired t-tests (with a significance level of ) on the SemEval-2018 dataset. These tests compared the proposed model across 10 independent runs of 10-fold cross-validation. Additionally, we calculated 95% confidence intervals (CI) for both accuracy and F1-score to quantify the variability of each method’s performance, providing a more comprehensive view of result reliability. The performance with statistical validation are provided in Table 2:

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance on SemEval-2018 with statistical validation. 95% CI is calculated via 10-fold cross-validation to reflect result variability.

https://doi.org/10.1371/journal.pone.0339946.t002

The statistical results confirm the robustness of our model’s performance: the 95% CI of our model’s accuracy (60.7–64.1%) does not overlap with that of BERT+GCN (57.2–60.6%), and the p-values demonstrate that the observed performance improvement is statistically significant, rather than a product of random chance.

To further understand the model’s classification behavior and identify areas for improvement, we generated a confusion matrix for the SemEval-2018 dataset, as shown in Fig 4. This matrix maps the relationship between true emotion categories and predicted categories. By examining this matrix, we can quantify misclassification rates and identify which emotional categories are most frequently confused.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Confusion matrix of the proposed model on SemEval-2018.

Interpretability and decision process insights: 1) Color intensity corresponds to the number of samples (darker shades = more samples); 2) Diagonal elements: Correct classifications; 3) Off-diagonal elements: Misclassifications; 4) Overall balance: for all categories confirm the model’s ability to distinguish nuanced emotions, with confusion patterns ‌‌aligning with human emotional perception.

https://doi.org/10.1371/journal.pone.0339946.g004

4.3.2. Experimental results on the RAVDESS dataset.

Table 3 presents the performance of the proposed model and baselines with 95% CI and p-values from paired t-tests. For the overall “Emotion” accuracy, the 95% CI of our model (86.4–89.4%) does not overlap with those of Bi-LSTM+Attention (85.1–88.3%), Attention Fusion (79.8–83.4%), and other baselines, confirming statistically significant improvements (p < 0.05). For “Facial Emotion” accuracy, our model (74.5–77.9%) outperforms Attention Fusion (73.1–76.7%) with p = 0.032, and for “Speech Emotion” accuracy, our model (84.2–87.2%) outperforms VQ-MAE (82.5–85.7%) with p = 0.028.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Performance of the competing methos on the RAVDESS dataset with statistical validation.

https://doi.org/10.1371/journal.pone.0339946.t003

Moreover, to evaluate the performance of the proposed approach on the realistic conditions, Table 4 provides the results of the proposed approach on RAVDESS dataset with added Gaussian white noise.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Performance of the proposed approach on realistic scenario datasets (RAVDESS with noisy data).

https://doi.org/10.1371/journal.pone.0339946.t004

4.3.3. Experimental results on the CMU-MOSEI dataset.

Table 5 presents the statistical results for CMU-MOSEI. The proposed model’s accuracy (95.5–97.1%) and F1-score (95.3–96.9%) have non-overlapping 95% CIs with baselines, and p-values < 0.001, confirming statistically significant improvements.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Performance of the competing method on the CMU-MOSEI datast with statistical validation.

https://doi.org/10.1371/journal.pone.0339946.t005

Finally, to reduce dependence of the proposed approach on labeled data, we respectively use 10% and 20% of labeled data from SemEval-2018, and generate pseudo-labels for unlabeled data using the model trained on labeled data. The pseudo-labels with confidence > 0.8 are retained to expand the training set, and the model is retrained iteratively for 3 rounds. Table 6 compares semi-supervised and supervised performance of the proposed approach.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Performance of semi-supervised vs. supervised learning on SemEval-2018.

https://doi.org/10.1371/journal.pone.0339946.t006

The proposed approach has shown superior performance compared to the state-of-the-art methodology used for artificial emotion recognition tasks, including emotion classification, facial emotion recognition, and speech emotion recognition, on the RAVDESS dataset [61]. Currently, these methods epitomize the highest level of deep learning methodology in these three fields. Yet, the proposed graph-based approach has obtained results that are competitive and closely match the best results in all three tests. These findings indicate that although there is potential for enhancement, the graph-based model presents a strong alternative that deserves more investigation and fine-tuning in the domain of emotional computing.

4.3.4. AUC-ROC scores.

Table 7 presents macro-averaged AUC-ROC scores for all datasets. The proposed model achieves higher AUC-ROC scores than baselines, confirming its ability to distinguish between emotional classes even for imbalanced categories.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Macro-averaged AUC-ROC scores of the competing methods on three datasets.

https://doi.org/10.1371/journal.pone.0339946.t007

4.4. Ablation experiments

To evaluate the contribution of each core component (E-GAT, Bi-LSTM, learnable edge weight mechanism) in the proposed framework, we conduct ablation experiments on SemEval-2018 (text-only) and CMU-MOSEI (multi-modal) datasets. All variants use the same hyper-parameters and 10-fold cross-validation to ensure fair comparison. The following variants are tested in the ablation study. Baseline 1: Vanilla GAT + LSTM (fixed edge weights via Euclidean distance). The vanilla GAT uses 2 layers (8 attention heads for the first layer, 4 for the second) with no enhanced attention weights. The LSTM is unidirectional. Baseline 2: E-GAT + LSTM (fixed edge weights via Euclidean distance). Retains the E-GAT’s enhanced attention weights but uses fixed edge weights instead of the learnable mechanism. The LSTM is unidirectional. Baseline 3: Vanilla GAT + Bi-LSTM (learnable edge weights). Uses the unidirectional vanilla GAT but replaces the LSTM with Bi-LSTM (bidirectional context modeling) and adopts the learnable edge weight mechanism.

As shown in Table 8, the E-GAT (vs. Vanilla GAT) improves accuracy by 3.7% (SemEval-2018) and 3.9% (CMU-MOSEI), confirming the effectiveness of enhanced attention weights for emotional feature extraction. Bi-LSTM adds 1.4% (SemEval-2018) and 1.9% (CMU-MOSEI) accuracy, validating the value of bidirectional temporal context modeling. Learnable edge weights contribute 3.9% (SemEval-2018) and 5.1% (CMU-MOSEI) accuracy, as they capture nonlinear modality interactions more effectively. Finally, the proposed model achieves the highest performance across all metrics, demonstrating synergistic effects between E-GAT, Bi-LSTM, and the learnable edge weight mechanism.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Ablation experiment results (%).

https://doi.org/10.1371/journal.pone.0339946.t008

4.5. Robustness evaluation

To validate the model’s cross-modal generalization under challenging real-world conditions, we conduct three sets of evaluations: cross-corpus transfer, speaker independence, and low-resource multilingual adaptation, as shown in Table 9.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Real-World Robustness Evaluation Results (%).

https://doi.org/10.1371/journal.pone.0339946.t009

• Cross-corpus evaluation. We train the model on RAVDESS (audio-visual, 8 emotional categories) and test on CMU-MOSEI (text-audio-vision, 5 emotional categories) to evaluate transferability across multi-modal corpora. The model achieves 79.2% accuracy, with a 7.7% drop compared to in-corpus testing on CMU-MOSEI (96.3%). This confirms the model’s ability to generalize to unseen multi-modal datasets with different emotional category definitions.
• Speaker-independent evaluation. On RAVDESS, we split the 24 speakers into training (20 speakers) and testing (4 unseen speakers) sets to evaluate speaker independence. The results are: Overall emotion recognition accuracy: 82.5% (drop of 5.4% vs. speaker-dependent testing (87.9%)). Facial emotion recognition accuracy: 70.3% (drop of 5.9% vs. speaker-dependent (76.2%)). Speech emotion recognition accuracy: 80.1% (drop of 5.6% vs. speaker-dependent (85.7%)). And the minimal accuracy drop indicates the model’s robustness to unseen speakers, a critical requirement for real-world applications.
• Low-resource multilingual evaluation. We use 10% labeled data from SemEval-2018 Spanish and French subsets (low-resource settings) and fine-tune the English-pre-trained model. The results are: Spanish 51.3% accuracy and French 50.7% accuracy. This demonstrates the model’s ability to adapt to low-resource multilingual settings, expanding its applicability to global scenarios.