1.1 Research gap

The study addresses the following research gaps:

• Protocol gap: SER performance is often reported using random splits with possible speaker overlap, while speaker-independent evaluation is less consistently adopted, limiting conclusions about real-world generalization [1].
• Fair comparison gap: Classical baselines (e.g., MFCC+SVM) and compact deep models are frequently evaluated with different preprocessing and split protocols, making direct comparison unreliable [2].
• Reproducibility and deployment gap: Many studies do not report multi-seed robustness or provide an end-to-end pipeline that can be validated on unseen user-provided audio, limiting practical verification and transparency.
• Integration and interpretation gap: Established handcrafted, deep, and hybrid components are less commonly evaluated together under identical data handling, repeated seeds, and both speaker-overlapping and speaker-independent protocols, making the source and robustness of performance gains difficult to interpret.

1.2 Contributions

The main contributions of this work are as follows:

• A controlled benchmark is provided for MFCC+SVM, a log-mel CNN, and a lightweight hybrid model under the same waveform standardization, label space, metrics, and software pipeline.
• A simple, CPU-compatible fusion implementation combines pooled handcrafted acoustic descriptors with penultimate-layer CNN embeddings and evaluates whether the complementary representations provide consistent gains.
• Random and actor-wise (speaker-independent) protocols are reported side by side to quantify the effect of speaker overlap on apparent performance.
• Five-seed mean ± standard deviation reporting and raw plus normalized confusion matrices are used to expose variability and class-wise error patterns rather than relying on a single headline score.
• A CPU-oriented Streamlit interface supports WAV upload, live prediction, and export of evaluation artifacts using the same preprocessing assumptions as the experiments.

1.3 Nature of novelty

The contribution of this study is methodological and practical rather than the invention of a new fundamental classifier. MFCCs, CNNs, SVMs, feature concatenation, and Streamlit are established techniques. The novelty lies in evaluating them as one controlled, protocol-aware framework in which the same standardized waveform feeds classical, deep, and hybrid branches; random and actor-wise splits are reported side by side; all results are repeated over five seeds; class-wise errors are retained as raw and normalized confusion matrices; and the trained artifacts are exposed through a CPU-oriented interface that uses the same preprocessing assumptions. This integrated design makes speaker leakage, robustness, and deployment reproducibility directly observable rather than hidden behind a single performance value.

Accordingly, the paper does not claim architectural novelty for the individual components. It contributes an experimentally transparent and lightweight reference design for interpreting representation fusion and speaker-independent generalization on CREMA-D.

As shown in Fig 1, the SER workflow uses two parallel representations derived from the same preprocessed waveform: handcrafted acoustic statistics and a log-mel spectrogram. The detailed protocol-aware processing path, including the three comparable pipelines and evaluation flow, is summarized in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. SER framework integrating MFCC–SVM, log-mel CNN, hybrid fusion, protocol-aware evaluation, and CPU inference.

https://doi.org/10.1371/journal.pone.0355238.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed SER framework with preprocessing, MFCC–SVM, log-mel CNN, hybrid fusion, and protocol-aware evaluation.

https://doi.org/10.1371/journal.pone.0355238.g002

3.1 Overall pipeline

This study implements a protocol-aware SER pipeline for fair comparison among three systems: (i) MFCC+SVM, (ii) a log-mel CNN, and (iii) a hybrid model that concatenates handcrafted features with CNN embeddings before final SVM classification. All systems share the same waveform standardization, class definitions, and evaluation metrics so that differences in performance can be attributed primarily to modeling choices rather than preprocessing mismatch [3,13,21].

As illustrated in Fig 2, each input waveform is converted into two synchronized representations. The handcrafted branch produces pooled utterance-level descriptors, while the deep branch produces a fixed-size log-mel spectrogram for CNN-based representation learning. The hybrid model then fuses the handcrafted vector with the penultimate-layer CNN embedding and performs final classification using an SVM.

The method intentionally uses established components so that the central experimental question can be isolated: whether complementary handcrafted and compact deep representations provide consistent gains when preprocessing, split construction, metrics, and repeated-run reporting are held constant. Therefore, the methodological contribution is the controlled design and validation of the integrated pipeline rather than a claim that MFCC, CNN, SVM, or feature concatenation is individually novel.

3.2 Preprocessing

All utterances are mono-sampled and resampled to a fixed sampling rate of f_s. In order to maintain the consistent length of input, the waveform is center-cropped or zero-padded to a fixed length, which results in a sequence.

(1)

This fixed-duration policy reduces variability arising from leading/trailing silence and recording-length differences, while also enabling efficient batch processing in deep models. Such normalization is particularly important in CNN-based SER, where convolutional feature learning can be sensitive to input alignment and temporal scale [20]. In speech-processing pipelines, center cropping is commonly applied to maintain uniform time-frequency resolution and stable batching in neural networks, while reducing bias toward early silence segments in an utterance. Resampling and fixed-length normalization are likewise standard practices for achieving consistent input representations [25]. Furthermore, fixed-duration inputs help reduce variability in spectrogram statistics caused by silence and recording length, which can affect the behavior of downstream classifiers [26].

3.3 Feature extraction

3.3.1 Handcrafted branch: MFCC statistics.

Features of handcrafted computation are calculated based on Mel-Frequency Cepstral Coefficients (MFCCs), which is one of the established families of speech descriptors and has been extensively relied on the traditional SER baselines [4]. MFCCs are obtained at short-time frames, and the result is a coefficient matrix, denoted as where T is the amount of frames and K is the amount of coefficients. Statistics are calculated to obtain a fixed length description of each coefficient dimension k:

(2)

The final handcrafted vector is denoted . Statistical pooling is employed due to the fact that it gives a concise summary of all the data and is efficient to classical classifiers but also captures global distributional clues like spectral shape and affect-related articulation differences. These descriptors also provide an interpretable baseline for comparing representation learning approaches [21]. Beyond MFCCs, prior SER work frequently uses curated paralin- guistic feature sets that include energy, spectral, and voice-quality descriptors designed for affective speech analysis [24]. Such handcrafted descriptors are still widely used as strong baselines because they are computationally light and provide interpretable cues linked to prosody and voice quality [26].

3.4 Models

3.4.1 Baseline 1: MFCC + SVM.

Support Vector Machines (SVMs) are a strong classical baseline for high-dimensional features and remain widely used in speech classification and benchmarking due to their robustness and simplicity [5]. Given handcrafted features f_MFCC, a linear SVM learns a margin-maximizing separator by solving:

(6)

where C controls the trade-off between margin and misclassification. For multi-class SER, a one-vs-rest strategy is typically employed [14]. This baseline serves as an efficient reference point and is especially relevant under CPU constraints, where low-latency inference may be required for interactive deployment.

3.4.2 Baseline 2: Log-mel CNN.

A convolutional neural network (CNN) is trained on log-mel spectrograms to learn discriminative time–frequency patterns associated with emotion. The classifier outputs and softmax probabilities:

(7)

The standard multi-class cross-entropy is:

(8)

CNN-based SER is motivated by the ability of convolutional filters to learn localized spectro-temporal patterns—such as harmonics, formant shifts, spectral energy variations, and temporal dynamics—that correlate with emotional expression [19,20]. However, these models can remain sensitive to speaker mismatch and confusable emotion pairs, which motivates the robust training strategies described next. CNNs operating on spectrograms were first popularized in broader audio-classification benchmarks and were subsequently adopted in SER because they can efficiently capture local spectro-temporal structure while remaining relatively lightweight for deployment [6,27]. Nevertheless, deep spectrogram-based models may still overfit dataset-specific speaker or channel cues when evaluated using random splits, reinforcing the importance of speaker-independent testing [28,29].

3.5 Robust training strategies

The CNN branch is augmented with SpecAugment-style time and frequency masking to improve robustness and reduce overfitting. The masking operation removes contiguous temporal regions or mel-frequency bands, encouraging invariance to local distortions and reducing dependence on narrow spectral cues [8]. Although originally proposed for automatic speech recognition, this perturbation is now widely used in speech representation-learning pipelines because of its simplicity and effectiveness [8,21]. Focal loss is also used to emphasize hard and confusable emotion classes, such as neutral versus sad or fear versus disgust, by down-weighting easy examples and assigning greater influence to difficult samples [9,13]. Mixup regularization further improves generalization by training on convex combinations of inputs and labels, which promotes smoother decision boundaries and reduces reliance on speaker-specific artifacts [10,21]. Model parameters are optimized with Adam, which adapts per-parameter learning rates using first- and second-moment estimates and is commonly used in deep speech pipelines because of its stable convergence [15].

3.6 Proposed hybrid fusion model

The proposed hybrid model combines complementary information from handcrafted descriptors and deep embeddings. After training the CNN, an embedding vector is extracted from the penultimate layer:

(11)

This is fused with handcrafted features using concatenation:

(12)

An SVM is then trained on the fused vector to perform final classification. Handcrafted statistics capture stable utterance-level spectral and prosodic properties, whereas CNN embeddings encode learned time-frequency patterns; their combination can therefore reduce failure cases of either representation alone [4,5,20]. The fusion operation is deliberately simple and should not be interpreted as a new mathematical fusion mechanism. Its value in this study is as a controlled, low-compute integration point for testing whether complementary representations yield consistent gains under both speaker-overlapping and speaker-independent conditions. The design is also suitable for CPU deployment because the compact CNN functions as a feature extractor while final classification is handled by a lightweight SVM.

3.7 Evaluation protocol and reporting

All models are evaluated under the same random and actor-wise protocols and reported using Accuracy, UAR, Macro-F1, and confusion matrices. Results are aggregated across five random seeds as mean ± std to reduce dependence on a single split. In addition, Fig 2 summarizes the complete experimental flow from standardized input processing to protocol-aware evaluation.

4.1 Dataset

All models are evaluated on the CREMA-D speech emotion dataset, which contains 7,442 utterances spoken by 91 actors and annotated into six discrete emotions: angry, disgust, fear, happy, neutral, and sad [3]. CREMA-D is frequently used in SER because it provides high speaker diversity and standardized categorical labels, enabling controlled within-dataset recognition experiments as well as speaker-independent generalization analysis. All recordings are processed as mono waveforms, and labels are maintained in an identical six-class space across all methods, so that performance differences arise from modeling decisions rather than label-processing variations.

4.2 Preprocessing and input standardization

Each of the audio files is subjected to a single preprocessing pipeline to create a fairness be- tween approaches. The signals are initially converted to mono and resampled to a fixed rate of sampling frequency f_s. Resampling is a standard requirement for reproducible time–frequency analysis because STFT and mel-filterbank resolution depend on sampling rate [25]. The next step is to apply a fixed period to any utterance: longer utterances are center-cropped and shorter utterances are zero-padded. This produces equal-length waveforms, reduces variability caused by differing recording durations and leading/trailing silence, and guarantees consistent tensor shapes for CNN training and evaluation.

To reduce sensitivity to amplitude differences and channel variability, we apply consistent normalization choices at the feature stage (Section 4.3). This “same-input-policy” ensures that both handcrafted features (MFCC statistics) and spectrogram inputs are derived from the iden- tical standardized waveform, preventing confounds where one model benefits from a different preprocessing pipeline.

4.3 Feature representations

Handcrafted branch (MFCC statistics): For the MFCC+SVM baseline and the handcrafted component of the hybrid model, MFCCs are computed over short-time frames as standard speech descriptors [4]. To obtain a fixed-dimensional utterance representation, frame-wise MFCC sequences are summarized via statistical pooling (mean and standard deviation per co- efficient), yielding compact features suitable for efficient SVM learning and CPU inference. In addition to MFCCs, prior paralinguistics literature frequently highlights curated acoustic descriptors (energy, spectral, and voice-quality cues) as strong interpretable baselines for af- fective speech analysis, motivating the use of pooled utterance-level statistics in classical SER comparisons [24,26].

Deep branch (log-mel spectrogram): For deep models, waveforms are transformed into log-mel spectrograms using STFT, mel filter banks, and logarithmic compression. The mel scale is widely used in audio modeling because it provides perceptually motivated frequency spacing and works effectively with convolutional filters [30,31]. To reduce amplitude/channel variability, per-feature normalization (z-score) is applied. Finally, the spectrogram is center- cropped/padded to a fixed number of frames so that the CNN receives inputs with identical dimensions.

4.4 Evaluation protocols

To obtain a reliable and protocol-aware assessment, all models are evaluated under two complementary split strategies. Both splits use the same preprocessing and feature extraction settings (Table 1) so that performance differences reflect protocol strictness and model design rather than configuration changes. Following common reproducibility guidance, random-split (potentially speaker-overlapping) performance is clearly separated from speaker-independent performance [13,21].

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Key hyperparameters used in preprocessing, feature extraction, and CNN training.

https://doi.org/10.1371/journal.pone.0355238.t001

• Random split: Utterances are divided into training and testing partitions using stratification by emotion class where applicable. Random splits are common because they are simple and often yield strong performance numbers, but they can allow the same speaker to appear in both train and test sets, which may inflate performance through speaker leakage [13]. Therefore, random-split results are interpreted as an optimistic estimate of within-dataset performance.
• Actor-wise (speaker-independent) split: Speakers are strictly separated between training and test sets so that no speaker identity overlap occurs across partitions. This protocol prevents leakage and provides a more realistic estimate of generalization to unseen speakers [21]. Actor-wise splitting is implemented using group-based partitioning, ensuring that all utterances from a given actor remain in a single split, consistent with standard group-aware evaluation practices in ML [32].

Across both protocols, Accuracy, Unweighted Average Recall (UAR), and Macro-F1 are reported. These are standard SER metrics; UAR and Macro-F1 are particularly informative when emotions are confusable or class-wise separability is uneven [22,23]. In addition, confusion matrices (raw and normalized) are reported to analyze systematic confusions between emotion pairs that may be hidden by scalar metrics alone [22].

4.5 Repeated runs and reporting

To reduce sensitivity to a single partition or initialization, each experimental setting is repeated across five different random seeds and summarized using mean ± std. Multi-run reporting improves the reliability of empirical comparisons and helps distinguish consistent gains from seed-specific variance [21]. For each seed, confusion matrices are also generated to support qualitative error-pattern analysis.

4.6 Implementation and runtime setting

All experiments are implemented in Python for reproducibility and ease of deployment. Feature extraction is implemented using librosa [11]. Classical models (SVM) and splitting utili- ties are implemented using scikit-learn, which provides standardized implementations for supervised learning and evaluation [33]. The deep learning components are implemented using TensorFlow/Keras [34,35], and CNN training uses the Adam optimizer due to its stable convergence and adaptive learning-rate behavior [15].

All experiments are executed on CPU-only hardware (no GPU acceleration). This reflects realistic deployment constraints and aligns the experimental environment with the end- to-end Streamlit inference interface used for interactive testing on unseen.wav files [36]. CPU-oriented evaluation also highlights the practicality of efficient feature representations and lightweight model designs for real-world SER integration.

4.7 Robust training and regularization

CNN-based SER can degrade under speaker mismatch and confusable emotion categories. To improve generalization and mitigate overfitting, widely adopted robustness strategies are incorporated during CNN training:

• SpecAugment-style masking: Time and frequency masking are applied directly to log-mel spectrograms, encouraging invariance to local occlusions and reducing reliance on narrow cues [8].
• Focal loss: Focal loss down-weights easy samples and emphasizes harder examples, improving learning under systematic confusions [9].
• Mixup: Mixup trains on convex combinations of inputs and labels, smoothing decision boundaries and typically improving generalization [10].

Together, these strategies provide complementary benefits (invariance, hard-sample emphasis, and boundary smoothing) and are commonly recommended to improve reliability under distribution shift and to reduce overfitting to speaker overlap [13,21].

4.8 Hyperparameter disclosure

For reproducibility, the key hyperparameters used throughout the pipeline are explicitly reported, including sampling rate , fixed crop duration (or fixed spectrogram frame length), mel-bin resolution , frequency range, and CNN training configuration (optimizer and robustness settings). These settings are kept identical for both random and actor-wise protocols so that performance differences primarily reflect protocol strictness and model design rather than tuning changes.

6.1 Overall performance (mean±std over 5 seeds)

Table 2 reports mean±std results over five random seeds for both evaluation protocols. Across all metrics (Accuracy, UAR, and Macro-F1), the proposed hybrid fusion model consistently achieves the best performance in both the random and actor-wise settings. This indicates that combining handcrafted statistics with learned CNN embeddings provides complementary in- formation that improves classification stability.

On the random split, the hybrid model reaches 62.16% ± 0.99% Accuracy and 62.03% ± 0.94% Macro-F1, outperforming MFCC+SVM (56.00% ± 1.10% Accuracy) and the log-mel CNN baseline (51.74% ± 1.31% Accuracy). On the actor-wise split, overall scores decrease for all models due to speaker mismatch, but the hybrid model remains strongest with 58.27% ± 1.21% Accuracy and 58.09% ± 1.36% Macro-F1.

This protocol gap is expected because speaker-independent testing is a stricter evaluation setting that reduces the benefit of speaker overlap present in random splits. UAR and Macro-F1 are reported in addition to Accuracy because SER often exhibits class confusability and uneven per-class separability, where aggregate Accuracy alone may not adequately reflect class-balanced behavior [22,23].

6.2 Confusion matrix analysis (per-class errors)

To examine class-wise behavior beyond aggregate metrics, Figs 3 and 4 present representative confusion matrices (raw counts and normalized percentages) for the random and actor-wise protocols, respectively. Confusion matrices are included because they explicitly show which emotion pairs are systematically confused and whether error patterns become more pronounced under speaker-independent evaluation. This form of per-class analysis complements Accuracy by revealing imbalance-like behavior and asymmetric confusions that are common in multi-class recognition tasks [22,23].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Random-split confusion matrices for SVM, CNN, and Hybrid using raw and normalized values.

https://doi.org/10.1371/journal.pone.0355238.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Actor-wise confusion matrices for SVM, CNN, and Hybrid using raw and normalized values.

https://doi.org/10.1371/journal.pone.0355238.g004

Overall, stronger diagonal dominance is observed in the random split, whereas the actor-wise split exhibits increased off-diagonal mass, indicating reduced generalization to unseen speakers. Across models, the hybrid system generally produces cleaner diagonals and fewer confusions than the individual baselines, which is consistent with the performance improvements reported in Table 2. Confusion matrices are widely used in diagnostic evaluation because they expose per-class error structure and systematic confusions that may be hidden by aggregate scores alone [40].

6.3 Comparison of SVM vs CNN vs hybrid

The three evaluated systems represent complementary modeling choices. MFCC+SVM pro- vides a strong, computationally light baseline that benefits from stable utterance-level statis- tics, and it remains competitive under CPU-only inference. The log-mel CNN learns time– frequency patterns directly from spectrogram structure, but its performance degrades more strongly under speaker-independent testing, suggesting higher sensitivity to speaker/channel variation and to limited training data. This behavior aligns with the protocol gap seen between the random and actor-wise settings in Table 2.

The hybrid fusion model consistently performs best because it combines two complementary information sources: (i) handcrafted descriptors that summarize stable spectral and prosodic statistics, and (ii) CNN embeddings that capture discriminative high-level time-frequency cues. Relative to MFCC+SVM, the hybrid model improves Accuracy by 6.16 percentage points on the random split (62.16 vs. 56.00) and 5.72 points on the actor-wise split (58.27 vs. 52.55). Relative to the log-mel CNN baseline, the gains are 10.42 points and 12.18 points, respectively. These results do not establish universal algorithmic superiority; rather, they show that within the controlled CREMA-D protocol, the integrated representation is more stable than either branch alone and that the advantage persists when speakers are separated between training and testing.

The confusion matrices (Figs 3–4) further support this conclusion. Compared to the stan- dalone baselines, the hybrid system generally shows a stronger diagonal and reduced off- diagonal spillover, especially under actor-wise testing, which is the more realistic deployment scenario. Therefore, the proposed fusion design provides a practical accuracy/robustness trade-off while remaining compatible with CPU-based real-time inference.

6.4 Deployment and practical application (streamlit)

To improve usability and practical reproducibility, an optional Streamlit-based deployment is developed to provide both (i) access to research outputs and (ii) live inference through a simple web interface. The application offers a unified dashboard for comparing SVM, CNN, and the proposed Hybrid model under both random and actor-wise protocols, while also supporting downloads of the exact outputs used in the study (CSV tables and figures) and testing on new WAV inputs. Such qualitative inspection is important in affect recognition because emotion classes can overlap acoustically, and confusion patterns often reveal model biases (e.g., systematic neutral–sad confusions) more clearly than any single scalar metric [40].

The interface is organized into a model-status panel indicating available model/split artifacts, a results section summarizing 5-seed mean ± std performance along with per-seed metrics, a confusion-matrix viewer for inspecting raw-count and normalized (%) matrices with download options, and a dedicated WAV-upload section for live prediction on unseen audio. The application also supports direct export of outputs for thesis writing and reproducible analysis, including aggregated performance results (mean ± std) in CSV format, per-seed evaluation metrics in CSV format, confusion-matrix summary figures for random and actor-wise protocols in PNG format, and bulk confusion-matrix data (e.g., zipped images or saved numeric arrays) for downstream analysis and reproducible visualization.

For live prediction, a user uploads a WAV file and the application applies the same preprocessing pipeline used during experimentation (e.g., fixed sampling rate and feature configuration), then returns a predicted emotion label and confidence scores. This enables demonstration and hands-on testing on unseen samples without command-line execution.

The deployment is designed to reduce train–test mismatch and preserve repeatable outputs by reusing the same preprocessing and feature-extraction settings applied during training, loading saved model checkpoints and stored evaluation artifacts (including seed-wise results and confusion matrices), and providing downloadable outputs in standard formats (CSV/PNG/arrays) for independent verification and thesis integration. The browser-based interface also improves accessibility by allowing non-technical users to inspect evaluation results and run live predictions without code modification. Overall, this deployment layer serves as a bridge between research and implementation, improving accessibility while preserving a reproducible experimental workflow.