Deep learning-based facial expression recognition

Facial expression recognition (FER) has been extensively studied in affective computing and computer vision. Early FER methods mainly depended on handcrafted facial descriptors, such as local binary patterns, histogram of oriented gradients, Gabor features, and scale-invariant features, followed by traditional classifiers including support vector machines, random forests, or shallow neural networks [23,24]. These methods are relatively simple and interpretable, but their performance is often limited by the representation ability of manually designed features, especially under complex conditions such as illumination changes, pose variations, and expression ambiguity.

With the development of deep learning, convolutional neural networks (CNNs) have become the mainstream framework for FER. Compared with handcrafted methods, CNN-based models can automatically learn hierarchical facial representations from input images and have achieved improved performance on benchmark datasets such as CK + , FER2013, RAF-DB, AffectNet, and Oulu-CASIA [25,26]. Representative architectures, including VGG, ResNet, Inception, DenseNet, and Xception, have been widely adopted as feature extractors for facial expression classification. In addition, attention mechanisms and multi-scale feature fusion strategies have been introduced to enhance expression-related regions and capture subtle facial muscle variations [27].

Although deep learning-based FER methods have significantly improved recognition accuracy, many existing models still rely on relatively deep backbones or additional complex modules. This may increase computational cost and limit their deployment in real-time human-computer interaction scenarios. Therefore, efficient FER models that can balance recognition accuracy, robustness, and inference speed remain worthy of further investigation.

Lightweight neural networks for real-time recognition

Lightweight neural networks have been extensively investigated for real-time and resource-constrained visual recognition tasks. Unlike conventional deep CNNs that improve accuracy mainly by increasing depth, width, or structural complexity, lightweight networks aim to reduce parameters and floating-point operations while preserving sufficient representation capability. MobileNet is one of the most representative lightweight architectures. It introduces depthwise separable convolution to decompose standard convolution into depthwise and pointwise operations, significantly reducing computational cost [28,29]. MobileNetV2 further introduces inverted residual blocks and linear bottlenecks, improving the efficiency and feature representation of compact models. MobileNetV3 combines neural architecture search and lightweight attention mechanisms to improve the accuracy-efficiency trade-off [30].

Besides MobileNet-based models, other lightweight architectures have also been proposed. ShuffleNet uses pointwise group convolution and channel shuffle operations to improve information exchange across channels while reducing computation [31]. EfficientNet adopts compound scaling to balance network depth, width, and input resolution, and its lightweight variants have been applied to mobile visual recognition tasks [32]. More recently, MobileNetV4 has further improved mobile vision models through efficient convolutional designs and universal inverted bottleneck structures, providing a stronger balance between accuracy and runtime efficiency on mobile and edge devices [33,34]. These lightweight networks provide an effective foundation for real-time FER. In the context of FER, lightweight networks are particularly valuable because facial expression recognition often needs to process continuous image or video frames. In applications such as online interaction, driver monitoring, and embedded affective perception, the model must respond quickly while using limited computational resources. Therefore, several studies have explored lightweight FER models based on MobileNet, ShuffleNet, EfficientNet-lite, and depthwise separable convolution [35,36]. These methods demonstrate that compact backbones can reduce inference cost and improve deployment feasibility.

Nevertheless, directly applying general lightweight classification networks to FER may overlook the region-specific nature of facial expressions. Discriminative expression cues are often concentrated in local areas such as the eyes, eyebrows, cheeks, and mouth. In compact networks, aggressive downsampling may weaken these subtle details, while generic backbones usually lack explicit modeling of regional importance. Therefore, lightweight FER models should not only reduce computational complexity but also enhance task-specific local expression representation. This motivates the use of MobileNetV4-Conv-S as an efficient backbone, together with a simple regional reweighting strategy for facial expression features.

Occlusion-robust and multi-task facial expression recognition

Robustness is a key issue in FER, especially under unconstrained real-world conditions. Facial images collected in practical scenarios are often affected by head pose changes, illumination variation, motion blur, low image quality, and partial occlusion. Among these factors, occlusion is particularly important because it may directly cover expression-related facial regions. For example, masks may hide the mouth and chin, glasses may affect the eye region, and hands or hair may block local facial structures. Since different expressions depend on different facial areas, occlusion can cause incomplete or misleading expression information.

To improve occlusion robustness, researchers have explored several strategies. Data augmentation methods simulate occlusion during training by randomly masking facial regions, encouraging the model to learn more robust representations [37]. Part-based methods divide the face into local regions or patches and extract regional features separately, so that the model can still use visible regions when some areas are corrupted. Attention-based methods attempt to assign higher weights to informative facial regions and reduce the influence of occluded or irrelevant areas [38]. Region attention networks further show that adaptively modeling the importance of facial regions can improve FER performance under pose variation and occlusion [39]. These studies demonstrate the importance of regional modeling for robust FER. However, some occlusion-robust methods introduce additional branches, landmark guidance, complex attention maps, or heavy feature fusion modules [40]. Although these designs can improve robustness, they may also increase model complexity and reduce inference efficiency. For real-time FER, it is more practical to design a lightweight mechanism that can estimate regional importance without relying on pixel-level occlusion labels or complex restoration procedures. Efficient attention mechanisms, such as ECA-style one-dimensional convolution, provide a useful idea for generating adaptive weights with low computational overhead [41]. Based on this idea, regional reweighting can be used to enhance visible and reliable facial regions while suppressing low-quality regions.

In addition to robustness, multi-task learning has also been widely studied in affective computing. Facial expression analysis is closely related to action unit detection, valence-arousal estimation, engagement recognition, and emotion intensity prediction. By learning related tasks jointly, multi-task models can obtain more informative feature representations and improve generalization [42,43]. Most existing FER methods, however, still focus on categorical emotion classification. They assign each input face to a discrete emotion class, such as happiness, sadness, anger, fear, surprise, disgust, or neutral [44,45]. This categorical setting is simple and effective, but it cannot fully describe the gradual nature of emotional expression. In real interactions, the same emotion category may appear with different activation degrees. For example, a slight smile and a strong joyful expression may both be classified as happiness, but their emotional intensities are clearly different [46].

Continuous affective models, such as valence-arousal representations, provide a more fine-grained description of emotional states. Nevertheless, they usually require additional annotations and may increase training difficulty. For practical FER, a feasible solution is to combine expression classification with an auxiliary intensity regression task. However, jointly modeling categorical expressions and continuous affective states introduces a distinct design conflict in lightweight FER. Conventional multi-task architectures often employ high-dimensional shared representations, task-specific feature-fusion layers, or separate multi-layer prediction branches, which increase the number of parameters, FLOPs, memory access, and inference latency. In contrast, aggressively compressed lightweight backbones may discard subtle facial variations required for stable continuous regression, making it difficult to preserve both fine-grained affective representation and real-time efficiency on edge hardware. To address this trade-off, OQ-FERNet reuses a single compact representation produced by the lightweight backbone and regional reweighting module, and applies only shallow task-specific output layers for expression classification and emotion intensity regression. This shared-feature design avoids duplicating dense feature extraction branches while retaining complementary categorical and quantitative supervision.

Overall architecture of OQ-FERNet

The overall architecture of the proposed Occlusion-aware Quantitative Facial Expression Recognition Network (OQ-FERNet) is illustrated in Fig 1. The model is designed for lightweight and robust facial expression recognition in real-time human-computer interaction scenarios. As shown in the figure, OQ-FERNet consists of three main components: a MobileNetV4-Conv-S backbone, an occlusion-aware regional reweighting module, and a quantitative emotion prediction head. Given an aligned facial image, the backbone first extracts compact facial expression features with low computational cost. When sequential video frames are available, the architecture additionally supports an auxiliary frame-difference pathway for incorporating adjacent-frame variation cues. Then, the occlusion-aware regional reweighting module divides the feature map into upper-face, middle-face, and lower-face regions, and adaptively adjusts their contributions to enhance reliable expression cues. Finally, the quantitative emotion prediction head uses the reweighted features to predict both the facial expression category and the corresponding emotion intensity score. For static-image input, the reweighted representation is directly used for prediction, whereas for sequential video input, the lightweight frame-difference feature can be conditionally introduced to supplement the current-frame representation without requiring a recurrent network or a computationally intensive temporal architecture. Since the empirical evaluations in this study are conducted on the isolated facial images of RAF-DB and AffectNet-7, the reported results correspond to the static-image pathway, while the frame-difference pathway constitutes a completed auxiliary architectural extension for incorporating short-term variation cues when sequential video frames are available. Through this design, OQ-FERNet aims to achieve an effective balance among recognition accuracy, occlusion robustness, and inference efficiency.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. OQ-FERNet model overall architecture diagram.

https://doi.org/10.1371/journal.pone.0359554.g001

Lightweight expression feature extraction backbone

To meet the requirements of real-time facial expression recognition in human-computer interaction scenarios, MobileNetV4-Conv-S is adopted as the lightweight feature extraction backbone of OQ-FERNet. Compared with conventional deep convolutional networks, MobileNetV4-Conv-S has the advantages of low parameter complexity, reduced computational cost, and efficient feature representation. These properties make it suitable for real-time or edge-device facial expression recognition tasks.

Given an aligned facial image X, the backbone first extracts shallow facial texture features through the initial convolutional layers:

(1)

where F₀ denotes the shallow expression feature, and represents the initial shallow convolution operation. Since facial expressions are usually caused by local muscle movements, shallow features contain important edge, texture, and local deformation information. Therefore, preserving fine-grained facial details in early stages is important for recognizing subtle expression cues around the eyes, eyebrows, mouth corners, and lips.

In the deeper stages, MobileNetV4-Conv-S mainly relies on the Universal Inverted Bottleneck (UIB) structure for efficient feature transformation. Let be the input of the l-th feature extraction stage. The output of the UIB module can be formulated as:

(2)

where denotes the point-wise convolution for channel expansion, denotes the depth-wise convolution with kernel size , and denotes the point-wise convolution for channel projection and fusion. The term represents the residual connection, which is used only when the input and output feature dimensions are consistent. Through this inverted bottleneck design, the network first enhances feature representation in a higher-dimensional channel space, then captures local spatial patterns using depth-wise convolution, and finally compresses the channels to reduce computational cost.

Depth-wise separable convolution is an important factor that enables the lightweight property of MobileNetV4-Conv-S. For a feature map with spatial size , input channels , output channels , and convolution kernel size , the computational cost of standard convolution can be expressed as:

(3)

In contrast, the computational cost of depth-wise separable convolution is:

(4)

It can be observed that depth-wise separable convolution decomposes standard convolution into spatial filtering and channel fusion, thereby greatly reducing the computational burden of the backbone. This design allows MobileNetV4-Conv-S to extract effective facial expression features with lower complexity, which is consistent with the efficiency requirement of real-time FER.

After several UIB-based feature extraction stages, the backbone can generate multi-stage facial expression representations:

(5)

where shallow features contain more local texture and fine-grained deformation information, while high-level features provide stronger semantic representation. In this study, the final high-level feature is used as the basic expression feature for subsequent processing. Although the lightweight backbone provides efficient feature extraction, different facial regions may contribute unequally under occlusion, pose variation, or local expression degradation. Therefore, regional modeling is further performed on to enhance reliable facial cues and reduce the interference of occluded or low-quality regions.

Occlusion-aware regional reweighting module

In practical facial expression recognition scenarios, facial images are often affected by masks, glasses, hands, hair, head pose variation, or other forms of partial occlusion. These factors may cause incomplete or unreliable local expression information. Since different facial regions contribute differently to expression recognition, directly using the global feature extracted by the backbone may introduce interference from occluded or low-quality regions. To address this issue, an occlusion-aware regional reweighting module is designed to adaptively model the contributions of the upper-face, middle-face, and lower-face regions.

Let denote the basic facial expression feature extracted by the lightweight backbone, where C, H, and W represent the channel number, height, and width of the feature map, respectively. Since the input face image has been detected and aligned, the spatial structure of the feature map still maintains a certain correspondence with facial regions. Therefore, is divided along the height dimension into three regional features:

(6)

where , , and represent the upper-face, middle-face, and lower-face features, respectively. The upper-face region mainly contains expression cues from the eyebrows, eyes, and eye corners; the middle-face region reflects local variations around the nose, cheeks, and nasolabial area; and the lower-face region contains important information from the mouth corners, lips, and chin. This regional division enables the model to describe different facial areas separately without requiring additional pixel-level occlusion annotations.

To obtain a compact representation for each facial region, global average pooling is applied to each regional feature. The regional descriptor can be written as:

(7)

where denotes the descriptor of the i-th facial region, and represents global average pooling. This operation compresses the spatial information of each region and reflects the overall response strength of the corresponding facial area. The three regional descriptors are then organized into a regional feature sequence:

(8)

To generate adaptive regional weights with low computational overhead, an ECA-style one-dimensional convolution is introduced. Compared with a multi-layer perceptron, the 1D convolution can model local dependencies among regional descriptors with fewer parameters. The regional weights are obtained as:

(9)

where denotes the one-dimensional convolution with kernel size k, represents the Sigmoid activation function, and denotes the adaptive weights of the three facial regions. A larger weight indicates that the corresponding region contains more reliable or discriminative expression information, while a smaller weight suggests that the region may be affected by occlusion, blur, or low-quality features.

Based on the generated regional weights, each facial region is reweighted as follows:

(10)

where denotes the reweighted feature of the i-th region. Through this operation, visible and informative regions are enhanced, while occluded or unreliable regions are suppressed. Finally, the reweighted regional features are concatenated according to their original spatial order to obtain the occlusion-aware enhanced expression feature:

(11)

where denotes the facial expression feature after regional reweighting. This module does not require additional occlusion masks or complex occlusion restoration operations. Instead, it adaptively adjusts regional contributions at the feature level, enabling the model to focus more on reliable expression cues. Compared with directly using the global backbone feature, the reweighted feature provides a more robust representation for subsequent facial expression classification and emotion intensity prediction.

Quantitative emotion prediction head

After occlusion-aware regional reweighting, the model obtains the enhanced facial expression feature . This feature has been adaptively adjusted according to the reliability of different facial regions, providing a more stable representation for subsequent emotion prediction. Conventional facial expression recognition methods usually output only discrete emotion categories, which makes it difficult to describe the intensity differences within the same expression class. To provide more fine-grained emotional information, a quantitative emotion prediction head is designed to jointly perform facial expression classification and emotion intensity estimation.

First, global average pooling is applied to the reweighted feature to convert the spatial feature map into a compact expression representation:

(12)

where g denotes the global facial expression feature vector, and represents the global average pooling operation. The prediction feature is formulated according to the input type. For an isolated facial image, the pooled representation is directly used for subsequent prediction, namely h = g. When temporally continuous video frames are available, the prediction head additionally supports a lightweight frame-difference pathway that incorporates short-term variation between adjacent frames. This pathway is an auxiliary architectural extension for sequential video input rather than a heavy temporal modeling component based on recurrent networks or Transformers.

Let and denote the global features of the current frame and the previous frame, respectively. The frame-difference feature is calculated as , which represents the lightweight temporal variation between adjacent frames. To explicitly fuse the current-frame expression representation and the temporal variation cue, a concatenation-based fusion strategy followed by linear projection is adopted:

(13)

where denotes feature concatenation along the channel dimension, and are learnable projection parameters, and denotes a nonlinear activation function. The fused feature contains both the current-frame facial expression representation and the lightweight frame-level temporal variation information. Accordingly, the prediction head follows when sequential video features are available and h = g when the input consists of an isolated facial image. Since RAF-DB and AffectNet-7 provide static facial images rather than temporally adjacent frame sequences, the classification and regression experiments reported in this study adopt the static-image formulation h = g.

In the classification branch, the joint feature h is fed into a fully connected layer, followed by a Softmax function to obtain the probability distribution over facial expression categories:

(14)

where p denotes the predicted expression probability distribution, and and are the learnable weight and bias of the classification branch, respectively. This branch is used to predict the basic facial expression category of the input face image or video frame.

In the emotion intensity regression branch, the same joint feature h is fed into another fully connected layer to estimate the quantitative emotion intensity score:

(15)

where s denotes the predicted emotion intensity score, and and are the learnable weight and bias of the regression branch, respectively. This branch further describes the activation degree of the facial expression beyond the discrete category prediction, enabling the model to estimate not only which emotion is expressed but also how strongly it is expressed.

To jointly optimize facial expression classification and emotion intensity estimation, a multi-task loss function is adopted. The classification task is optimized using cross-entropy loss, while the emotion intensity regression task is optimized using mean squared error loss in the final experiments. The total loss is defined as:

(16)

where denotes the facial expression classification loss, denotes the emotion intensity regression loss based on mean squared error, and is a balancing coefficient that controls the contribution of the regression task. Through this multi-task learning strategy, OQ-FERNet can preserve categorical expression recognition ability while learning emotion intensity variation. Compared with classification-only FER models, the proposed quantitative emotion prediction head provides more fine-grained emotional outputs, making the model more suitable for emotion understanding in real-time human-computer interaction scenarios.

Datasets and preprocessing

To evaluate the effectiveness of the proposed OQ-FERNet, two widely used in-the-wild facial expression recognition datasets, RAF-DB and AffectNet, are adopted in this study. RAF-DB is used to evaluate facial expression classification performance under unconstrained conditions. It contains facial images with variations in pose, illumination, occlusion, age, gender, and expression ambiguity, which makes it suitable for testing the robustness of facial expression recognition models in real-world scenarios. AffectNet is used to evaluate both facial expression classification and quantitative emotion prediction. In addition to discrete expression labels, AffectNet provides continuous affective annotations, including separate valence and arousal scores. In this study, the arousal annotation is used to construct a scalar supervision target for emotion intensity estimation. The valence–arousal coordinates are not mapped to an intensity vector, and no pre-existing standalone expression-intensity indicator is extracted from the dataset metadata.

For both datasets, seven basic expression categories are selected, including neutral, happiness, sadness, surprise, fear, disgust, and anger. For RAF-DB, the official training and testing split is used. For AffectNet, the seven-class setting is adopted by excluding the contempt category, so that the expression categories are consistent with RAF-DB. For clarity, this seven-class setting of AffectNet is denoted as AffectNet-7 in the following experimental results. The manually annotated subset of AffectNet is used, and the training and validation samples corresponding to the selected seven expression categories are adopted for model training and evaluation.

Before being fed into the network, all facial images are processed using the same preprocessing pipeline. First, face detection and a standard five-landmark-based similarity affine alignment are performed for both RAF-DB and AffectNet-7. Specifically, the centers of the two eyes, the nose tip, and the two mouth corners are detected and mapped to predefined locations in a canonical facial template through translation, in-plane rotation, and isotropic scaling. The aligned facial regions are then cropped and resized to pixels. This procedure reduces variations in facial position, scale, and moderate in-plane rotation, thereby improving the spatial consistency of the upper-face, middle-face, and lower-face regions used by the regional reweighting module. Explicit three-dimensional face frontalization is not applied in this study. Therefore, although the landmark-based affine alignment alleviates moderate pose and rotation variations, it cannot completely compensate for extreme out-of-plane head poses or inaccurate landmark localization under severe occlusion. The aligned facial images are normalized using the mean and standard deviation of the training set. During training, data augmentation strategies are applied to improve model generalization, including random horizontal flipping, random cropping, color jittering, and random erasing. During testing, only resizing and normalization are used to ensure stable evaluation.

For the quantitative emotion activation regression task on AffectNet, the arousal annotation is adopted as the regression target. Let denote the original arousal value of the i-th sample. The normalized regression target is obtained through linear transformation as , where . A larger value of indicates a higher level of emotional activation, while a smaller value represents a lower activation level. The valence annotation is not used in this task because it reflects the positive-negative direction of emotion rather than activation intensity. Therefore, the regression branch predicts a single normalized emotion activation score instead of a two-dimensional valence-arousal representation.

Implementation details

All experiments are implemented using PyTorch. The proposed OQ-FERNet and all baseline models are trained under the same preprocessing, data augmentation, and optimization settings to ensure fair comparison. The input image size is fixed to . The batch size is set to 64, and the total number of training epochs is set to 100. AdamW is used as the optimizer with an initial learning rate of and a weight decay of . A cosine learning rate decay strategy is adopted to gradually reduce the learning rate during training. For the classification task, cross-entropy loss is used to optimize the facial expression category prediction branch. For the emotion intensity regression task, mean squared error loss is adopted as the default regression loss. The total training objective is defined as a weighted combination of classification loss and regression loss, where the balancing coefficient is set to 0.5 unless otherwise specified. Dropout with a probability of 0.3 is used in the prediction head to reduce overfitting. The models are trained on a workstation equipped with an NVIDIA GPU. The software environment includes Python 3.10, PyTorch 2.0, CUDA 11.8, and cuDNN. During inference efficiency evaluation, the batch size is set to 1 to simulate real-time prediction. The inference time and frames per second are measured after a warm-up stage to avoid unstable timing caused by GPU initialization.Since RAF-DB and AffectNet-7 are still-image facial expression recognition datasets, the optional frame-difference branch is not activated in the reported experiments. For all single-image evaluations, the prediction feature is directly set as h = g, and no temporal difference feature is used. Therefore, the reported classification accuracy, quantitative emotion prediction results, model complexity, FLOPs, inference time, and FPS are all obtained under the static-image setting. The frame-difference feature is only retained as an optional extension for video-based real-time human-computer interaction scenarios.

Evaluation metrics

The evaluation metrics include both recognition performance and model efficiency. For facial expression classification, Accuracy, Precision, Recall, and F1-score are used. Accuracy measures the overall classification correctness, while Precision, Recall, and F1-score provide a more comprehensive evaluation of class-level prediction quality, especially when the data distribution is imbalanced. Considering the class imbalance in RAF-DB and AffectNet-7, Precision, Recall, and F1-score are calculated using macro averaging across the seven expression categories. Accordingly, the reported values correspond to Macro Precision, Macro Recall, and Macro F1-score. For quantitative emotion prediction, Mean Absolute Error (MAE) and Mean Squared Error (MSE) are used to evaluate the difference between the predicted emotion intensity score and the ground-truth intensity label. Root Mean Squared Error (RMSE) is additionally reported to measure the prediction error in the same numerical scale as the normalized regression target. A lower regression error indicates that the model can estimate emotion intensity more accurately. To account for stochastic variation, all performance-related experiments are independently repeated using five fixed random seeds, namely {0,1,2,3,4}. The dataset partitions, preprocessing procedures, training settings, and optimization parameters remain unchanged across runs. Unless otherwise specified, the results are reported as mean standard deviation over the five runs.

To evaluate model complexity and real-time inference capability, the number of parameters, floating-point operations (FLOPs), single-image inference time, and frames per second (FPS) are reported. The number of parameters reflects the storage cost of the model, while FLOPs measure the computational complexity. Inference time and FPS are used to evaluate whether the model satisfies the efficiency requirement of real-time facial expression recognition. Parameters and FLOPs are deterministic structural measurements, whereas inference time and FPS are obtained through repeated runtime measurements after GPU warm-up.

Baseline models

To evaluate the performance of the proposed OQ-FERNet, several representative baseline models are selected for comparison. The baseline methods include both conventional convolutional networks and recent facial expression recognition models. VGG16 and ResNet18 are adopted as standard CNN-based baselines to provide comparisons with commonly used deep feature extractors. EfficientNet-B0 and ShuffleNetV2 are selected as efficient convolutional models, which are used to evaluate the accuracy-efficiency trade-off of OQ-FERNet. In addition to these general visual recognition backbones, several recent FER methods are included for comparison. CNN-DET [47] is selected as a hybrid deep learning model that combines CNN-based feature extraction with an ensemble classifier. EA-Net [48] is adopted as an attention-enhanced ensemble model, which integrates multiple CNN backbones and attention modules for facial emotion recognition. FERMam [49] is included as a recent lightweight dual-source and multi-scale fusion framework for FER. POSTER-Var [50] is selected as a recent probabilistic FER model based on variational inference, which is used to compare fine-grained expression representation ability. For fair comparison, the original architectural designs and prediction mechanisms of all baseline models were retained. The common preprocessing, training, evaluation, and multi-seed protocols described in the Implementation Details and Evaluation Metrics subsections were consistently applied to all available implementations. No additional model-specific hyperparameter tuning was performed to favor any individual baseline.

Comparison results

Table 1 presents the classification performance comparison of different models on the RAF-DB and AffectNet-7 datasets. OQ-FERNet achieves the highest results across all evaluation metrics on both datasets, indicating that the proposed model has effective classification capability and stable recognition performance in facial expression recognition scenarios. On RAF-DB, OQ-FERNet obtains an Accuracy of , a Precision of , a Recall of , and an F1-score of . Compared with the conventional CNN baselines VGG16 and ResNet18, OQ-FERNet improves Accuracy by 12.70 and 10.21 percentage points, respectively. Compared with EfficientNet-B0 and ShuffleNetV2, the Accuracy improvements are 7.78 and 11.10 percentage points, respectively. These results show that the proposed model can learn more discriminative facial expression representations than basic convolutional and lightweight recognition networks. Compared with CNN-DET on RAF-DB, OQ-FERNet improves Accuracy by 3.42 percentage points and also achieves higher Precision, Recall, and F1-score values. Compared with the stronger models FERMam and POSTER-Var, OQ-FERNet achieves Accuracy improvements of 0.99 and 0.36 percentage points, respectively. Although the margin over POSTER-Var is relatively moderate, the narrow standard deviations confirm that this performance gain reflects a consistent improvement rather than training noise. Furthermore, while achieving superior recognition performance, OQ-FERNet is specifically designed for lightweight and real-time inference. On AffectNet-7, OQ-FERNet obtains an Accuracy of , a Precision of , a Recall of , and an F1-score of . Compared with VGG16 and ResNet18, OQ-FERNet improves Accuracy by 9.48 and 7.57 percentage points, respectively. Compared with EfficientNet-B0 and ShuffleNetV2, the Accuracy improvements are 5.98 and 8.47 percentage points, respectively. The lower performance on AffectNet-7 than on RAF-DB reflects the greater difficulty of AffectNet-7, which contains more diverse facial appearances, pose variations, illumination changes, expression ambiguity, and label noise. Compared with CNN-DET on AffectNet-7, OQ-FERNet improves Accuracy by 4.90 percentage points. Compared with FERMam and POSTER-Var, OQ-FERNet also achieves higher classification performance, with Accuracy improvements of 1.94 and 0.41 percentage points, respectively. These results indicate that OQ-FERNet can maintain better recognition performance under more challenging in-the-wild conditions. The classification results on both datasets verify the effectiveness of OQ-FERNet and show that occlusion-aware regional reweighting and quantitative prediction design can improve the representation ability for different expression categories.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Classification performance comparison on RAF-DB and AffectNet-7 (mean standard deviation over 5 runs).

https://doi.org/10.1371/journal.pone.0359554.t001

Table 2 presents the quantitative emotion prediction performance of different models on AffectNet-7. Compared with the baseline models, OQ-FERNet achieves the lowest MAE, MSE, and RMSE values, indicating that the proposed model can estimate emotion intensity more accurately. Specifically, OQ-FERNet obtains an MAE of , an MSE of , and an RMSE of , outperforming VGG16, ResNet18, EfficientNet-B0, and ShuffleNetV2 under the same evaluation setting. Among the baseline models, EfficientNet-B0 achieves relatively better quantitative prediction performance, with an MAE of , an MSE of , and an RMSE of . Compared with EfficientNet-B0, OQ-FERNet reduces MAE by 0.046, MSE by 0.038, and RMSE by 0.055. This improvement suggests that the occlusion-aware regional reweighting module can provide more reliable facial representations for emotion intensity estimation by enhancing informative facial regions and suppressing less reliable local features. Compared with ShuffleNetV2, OQ-FERNet also achieves a clear reduction in prediction error, showing that the proposed model can maintain quantitative prediction capability while preserving lightweight inference. These results demonstrate that OQ-FERNet is not limited to discrete facial expression classification, but can also provide fine-grained emotion intensity estimation. This ability is important for real-time human-computer interaction scenarios, where the system needs to understand not only the emotion category but also the strength of the expressed emotion.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Quantitative emotion prediction performance on AffectNet-7 (mean standard deviation over 5 runs).

https://doi.org/10.1371/journal.pone.0359554.t002

As shown in Table 3 and Fig 2, the proposed OQ-FERNet achieves a favorable balance between recognition accuracy and computational efficiency. Table 3 provides the detailed numerical comparison in terms of parameters, FLOPs, model size, inference time, and FPS, while Fig 2 visualizes the accuracy-efficiency trade-off among different FER models on RAF-DB. In the figure, models located closer to the upper-left region indicate higher recognition accuracy with lower computational cost, and the bubble size represents the inference speed. VGG16 has the highest computational complexity, with 138.36M parameters, 15.47G FLOPs, and a model size of 527.80 MB, while its inference speed is only 78 FPS. Although ShuffleNetV2 achieves the lowest FLOPs and the highest FPS, its recognition accuracy is much lower than that of OQ-FERNet, indicating that excessive lightweight design may weaken facial expression representation capability. FERMam and POSTER-Var obtain relatively strong recognition performance, but they require substantially higher computational costs and show lower inference speed than OQ-FERNet. In contrast, OQ-FERNet achieves an accuracy of 93.12% on RAF-DB with only 4.18M parameters, 0.26G FLOPs, and 16.10 MB model size. Its inference time is 2.81 ms, corresponding to 356 FPS, which satisfies the requirement of real-time facial expression recognition. These results indicate that OQ-FERNet does not improve recognition performance by increasing computational complexity. Instead, by combining lightweight feature extraction with occlusion-aware regional reweighting and quantitative prediction design, the proposed model achieves a better trade-off between accuracy and efficiency. Therefore, OQ-FERNet is more suitable for real-time human-computer interaction scenarios where both recognition performance and inference speed are required.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Model complexity and inference efficiency.

https://doi.org/10.1371/journal.pone.0359554.t003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Accuracy-efficiency comparison of different FER models on RAF-DB.

https://doi.org/10.1371/journal.pone.0359554.g002

For robustness evaluation, a controlled synthetic occlusion protocol is designed on aligned RAF-DB images. Three occlusion conditions are considered, including eye occlusion, mouth occlusion, and random occlusion. Eye occlusion and mouth occlusion are generated by masking the corresponding facial regions after alignment, while random occlusion is generated by applying masks at randomly selected locations within the facial area. The same occlusion generation strategy is applied to all models to ensure a fair and reproducible comparison.

Table 4 presents the robustness comparison of different models under synthetic occlusion on RAF-DB. The no-occlusion results are consistent with the RAF-DB classification results reported in Table 1. Under the no-occlusion setting, OQ-FERNet achieves an accuracy of , outperforming all compared models. When different types of occlusion are introduced, the proposed model still maintains the highest accuracy, reaching under eye occlusion, under mouth occlusion, and under random occlusion. Compared with VGG16, ResNet18, EfficientNet-B0, and ShuffleNetV2, OQ-FERNet shows a much smaller performance degradation, indicating that the proposed model is more stable when local facial information is partially missing. Compared with CNN-DET and EA-Net, OQ-FERNet also achieves higher accuracy under all occlusion settings. In particular, the average drop of OQ-FERNet is only , which is lower than CNN-DET and EA-Net by 4.63 and 3.73 percentage points, respectively. Compared with FERMam and POSTER-Var, OQ-FERNet also obtains a smaller average drop, showing better robustness against synthetic occlusion. These results suggest that the occlusion-aware regional reweighting module can reduce the negative influence of unreliable facial regions and improve the stability of expression recognition under degraded visual conditions. Fig 3 further visualizes the average regional weights generated by the ORR module under different occlusion conditions. Under the no-occlusion setting, the weights of the upper-face, middle-face, and lower-face regions are relatively balanced, indicating that the model jointly uses information from different facial regions. Under eye occlusion, the upper-face weight decreases to 0.18, while the lower-face weight increases to 0.47. This indicates that the model suppresses the occluded upper-face region and relies more on visible regions. Under mouth occlusion, the lower-face weight decreases to 0.19, while the upper-face weight increases to 0.46, showing that the model shifts attention away from the occluded mouth region. Under random occlusion, the three regional weights remain relatively balanced, reflecting the adaptive response of the model to uncertain local information loss. The results provide consistent evidence for the effectiveness of the ORR module. The accuracy comparison shows that OQ-FERNet achieves the smallest performance degradation under occlusion, while the regional weight visualization explains how the model adjusts the contributions of different facial regions. These results demonstrate that the proposed regional reweighting mechanism can adaptively suppress unreliable occluded regions and enhance informative visible regions, thereby improving the occlusion robustness of facial expression recognition. Although the synthetic occlusion protocol provides a controlled evaluation of regional information loss, it relies on the accuracy of facial alignment. Landmark localization errors, extreme head poses, or large facial rotations may reduce the correspondence between predefined regions and actual facial components, which may affect the robustness evaluation. In addition, RAF-DB and AffectNet-7 contain naturally occurring occlusions as part of their unconstrained variations; however, their standard annotations do not provide independent labels for specific natural occlusion categories, such as masks, glasses, hair, or hand occlusions. Therefore, the current evaluation focuses on controlled synthetic occlusion to ensure reproducibility, while future work will investigate adaptive facial normalization, learnable regional modeling, and independently annotated natural-occlusion benchmarks to further improve generalization under realistic conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Robustness comparison under synthetic occlusion on RAF-DB.

https://doi.org/10.1371/journal.pone.0359554.t004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Regional weight visualization of the ORR module on RAF-DB under different synthetic occlusion conditions.

https://doi.org/10.1371/journal.pone.0359554.g003

Fig 4 shows the normalized confusion matrices of OQ-FERNet on RAF-DB and AffectNet-7. The rows represent the ground-truth expression labels, and the columns represent the predicted labels. Since the matrices are normalized by rows, each diagonal entry indicates the proportion of correctly classified samples within the corresponding ground-truth category. On RAF-DB, most samples are concentrated along the diagonal, indicating that OQ-FERNet can effectively distinguish the seven basic facial expression categories. The model achieves high diagonal proportions for Happiness, Surprise, Anger, and Neutral, with values of 97.2%, 93.6%, 93.4%, and 93.1%, respectively. This suggests that OQ-FERNet can capture discriminative facial cues for expressions with relatively clear visual patterns. For RAF-DB, the relatively lower diagonal entries for Fear and Disgust indicate that these two categories are more difficult to distinguish than Happiness and Surprise. Disgust is partly confused with Anger, while Fear is partly confused with Surprise, reflecting the visual similarity between these expression pairs. This phenomenon is consistent with the fact that negative expressions often share subtle local facial movements, such as changes around the eyes, nose, and mouth corners. On AffectNet-7, the diagonal entries are generally lower than those on RAF-DB, which reflects the greater difficulty of AffectNet-7 under in-the-wild conditions. Although Happiness still shows a relatively high diagonal proportion, categories such as Fear, Disgust, Sadness, and Anger exhibit more visible off-diagonal confusion. In particular, Fear is more likely to be confused with Surprise, and Disgust is more likely to be confused with Anger. This indicates that AffectNet-7 contains more challenging samples with larger variations in pose, illumination, expression intensity, and annotation ambiguity. The confusion matrices further support the quantitative results reported in Table 1. RAF-DB shows stronger class-wise separability, while AffectNet-7 presents more complex inter-class confusion. Nevertheless, OQ-FERNet maintains clear diagonal dominance on both datasets, demonstrating that the proposed occlusion-aware regional reweighting and quantitative prediction design can enhance discriminative facial expression representation under different data conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Normalized confusion matrices of OQ-FERNet on RAF-DB and AffectNet.

https://doi.org/10.1371/journal.pone.0359554.g004

Ablation study

Table 5 presents the ablation results of OQ-FERNet on RAF-DB and AffectNet-7. When both regional reweighting and quantitative emotion prediction are removed, the model obtains the lowest classification performance, with accuracy and F1-score on RAF-DB, as well as accuracy and F1-score on AffectNet-7. This result shows that directly using the basic facial expression feature without regional enhancement and quantitative supervision is insufficient for robust expression representation. After introducing the regional reweighting module, the accuracy increases to on RAF-DB and on AffectNet-7. The MAE and MSE on AffectNet-7 are also reduced to and , respectively. These results indicate that region-level feature adjustment can enhance informative facial regions and suppress less reliable local features, thereby improving both expression classification and emotion intensity estimation. When the quantitative emotion prediction head is removed, the model achieves accuracy and F1-score on RAF-DB, as well as accuracy and F1-score on AffectNet-7. Since this variant does not output emotion intensity, MAE and MSE are not applicable. Compared with this variant, the complete OQ-FERNet improves the accuracy from 91.62% to 93.12% on RAF-DB and from 66.38% to 68.32% on AffectNet-7. This demonstrates that quantitative emotion prediction provides useful auxiliary supervision for learning more fine-grained affective representations. The complete OQ-FERNet achieves the best classification results on both datasets and obtains the lowest MAE and MSE values on AffectNet-7, reaching and , respectively. These results verify the effectiveness of the occlusion-aware regional reweighting module and the quantitative emotion prediction head. The ablation study shows that the two components are complementary and jointly contribute to improving the classification accuracy, robustness, and fine-grained emotion prediction capability of OQ-FERNet.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Ablation study of OQ-FERNet.

https://doi.org/10.1371/journal.pone.0359554.t005

Sensitivity analysis

Table 6 presents the sensitivity analysis of the multi-task loss weight . The parameter controls the relative contribution of the emotion intensity regression loss in the total training objective. When is set to 0.1, the model obtains Accuracy and F1-score on RAF-DB, as well as Accuracy and F1-score on AffectNet-7. As increases from 0.1 to 0.5, the classification performance improves steadily on both datasets. Specifically, the Accuracy increases from 92.31% to 93.12% on RAF-DB and from 67.28% to 68.32% on AffectNet-7. This indicates that an appropriate regression supervision signal can help the model learn more fine-grained facial expression representations. When is further increased to 0.7 and 1.0, the regression errors on AffectNet-7 continue to decrease, with MAE decreasing from 0.246 to 0.241 and MSE decreasing from 0.101 to 0.098. However, the classification performance begins to decline. For example, when is set to 1.0, the Accuracy decreases to on RAF-DB and on AffectNet-7. This suggests that an excessively large regression weight may make the model overly biased toward emotion intensity estimation, thereby weakening its discriminative ability for discrete expression classification. Therefore, is selected as the default setting in this study. Under this setting, OQ-FERNet achieves the best classification performance on both RAF-DB and AffectNet-7, while still maintaining low MAE and MSE values for quantitative emotion prediction. The results show that a balanced multi-task optimization strategy is beneficial for improving both facial expression classification and emotion intensity estimation.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Sensitivity analysis of multi-task loss weight.

https://doi.org/10.1371/journal.pone.0359554.t006