Participants

Twenty healthy subjects (gender distribution: 10 males, 10 females, mean age = 23, standard deviation = 1.31, age range: 20–26 years) participated in the study. All participants were right-handed, had normal or corrected-to-normal vision, and had no history of neurological or psychiatric disorders. Three subjects were excluded from the study due to poor optical signal quality, resulting in a final cohort size of 17 subjects (10 males, 7 females). The study protocol was approved by the Ethics Committee of İstanbul Medipol University, İstanbul, Türkiye. All subjects gave verbal consent to the experimenter prior to the experiment. Experiments were conducted between 01/11/2022 and 01/03/2023. The experiments were carried out in accordance with the Declaration of Helsinki.

Experimental protocol

The experimental protocol involved passive viewing of emotional pictures on a computer screen in a dimly-lit, quiet room while fNIRS recordings were taken with a 22 channel fNIRS system from the PFC of each participant. The participants were informed about the experimental protocol at the onset of each experimental session and they were requested to sit relaxed and avoid head movements during the experiment. The emotional pictures used in this study were selected from the International Affective Picture System (IAPS) database, which provides standardized pictures with predefined valence and arousal scores [36]. Sixty stimuli were chosen from the IAPS database, consisting of 30 pleasant and 30 unpleasant pictures. The pleasant pictures had a mean valence score of 7.37 ± 1.61 and arousal score of 6.40 ± 2.2 while the unpleasant pictures had a mean valence score of 2.32 ± 1.56 and a mean arousal score of 6.42 ± 2.2. Selected IAPS stimuli pictures were: (a) pleasant: 8200, 5260, 8206, 8080, 8470, 8490, 8492, 8370, 8499, 8501, 8380, 8260, 5833, 8400, 5470, 8030, 8161, 8034, 8163, 8170, 8300, 7405, 2034, 8180, 5621, 8185, 8186, 4220, 5629, 8190; (b) unpleasant: 6021, 3212, 3213, 6540, 6415, 9620, 9630, 9250, 9635.1, 9252, 8485, 3110, 6570.1, 6834, 2683, 3005.1, 9921, 6212, 9412, 9413, 3400, 9163, 9810, 9300, 9050, 6370, 9321, 6250.1, 9075, 3195.

Each experimental session began with a 40-second baseline recording, followed by two consecutive blocks consisting of positive valence emotional stimuli trials, a congruent Stroop trial block, two consecutive blocks consisting of negative valence emotional stimuli trials, and an incongruent Stroop trial block. This sequence of stimuli presentation was repeated 5 times to ensure robust data collection (Fig 1). Stimuli blocks which involved presentation of pleasant pictures were labelled as ‘positive’ while stimuli blocks which involved presentation of unpleasant pictures were labelled as ‘negative’. Hence, the experimental protocol involved 10 blocks of positive valence stimuli and 10 blocks of negative valence stimuli. Each emotional stimuli block lasted for 12 seconds where three consecutive images with similar valence and arousal scores were displayed for 4 seconds each. Each Stroop block lasted for 30 seconds, consisting of six congruent or incongruent trials presented for 5 seconds each. Within the congruent Stroop trial blocks, each trial consisted of presentation of a color name where the ink color of the word matched its meaning, while for the trials presented within the incongruent Stroop blocks, the ink color and word meaning did not match. All stimulus blocks were separated by 20-second rest intervals during which subjects were asked to focus on the asterisk symbol presented on the screen.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Experimental Protocol.

Each experimental session consisted of presentation of positive and negative valence emotional stimuli blocks besides congruent and incongruent Stroop trial blocks. The depicted stimuli presentation cycle was repeated 5 times. Stroop trial blocks were placed in between different consecutive emotional stimuli blocks in order to wash out cognitive and emotional effects of previously presented emotional stimuli block.

https://doi.org/10.1371/journal.pone.0325850.g001

Stroop task blocks were presented in between consecutive emotional stimuli blocks with different valences in order to i) eliminate the potential after-image residual emotional effects of previously presented emotional stimuli block, ii) engage participants cognitively, and iii) assess their attentional processes and interference effects. More specifically, Stroop trial blocks were introduced in between consecutive emotional stimuli blocks with different valences to enable a washout effect of the potential after-image residual emotional effects of previously induced affective state and to ensure that participants enter each new emotional condition from a relatively neutral baseline. Engaging participants in a cognitively demanding task such as the Stroop (i.e., color–word matching) was considered to redirect the attentional resources away from the previously encountered emotional stimuli, thereby “washing out” residual emotional processing. This approach would improve the discriminability of subsequent emotional responses by reducing confounding influences from previously presented emotional stimuli of different valences. Moreover, the Stroop task was considered to enhance attentional engagement and cognitive control through task-switching, which would assist the participants in remaining consistently focused throughout the session—minimizing potential confounds such as habituation. By including Stroop tasks in this manner, we follow a strategy used in other psychophysiological studies where short, cognitively neutral interventions effectively reset the participant’s internal state before the next experimental condition. This approach is consistent with the findings of Visalli et al. (2023), who showed that the proactive- and reactive-control demands of an intervening emotional-priming Stroop task effectively abolish residual affective interference [37].’

The selected unpleasant and pleasant pictures were labelled as ‘positive’ or ‘negative’ valence stimuli based on their standardized IAPS valence scores. Subjective ratings of each presented picture were also collected after the experiments by using a Likert Scale to check whether i) the subjects produced arousal behavior in accordance with IAPS database scores and ii) their emotional responses could be represented by IAPS database. IAPS subjective ratings of arousal for each picture were obtained with the Likert scale, using a 9-point version [36]. This scale ranged from 1 to 9, representing states between low and high arousal conditions. Indeed, a good correspondence between IAPS arousal scores and the subjective arousal rating scores were found.

fNIRS data acquisition and preprocessing steps

fNIRS data were collected using a NIRSport system (NIRx Medical Technologies, LLC, Berlin, Germany), with 8 light sources and 7 detectors strategically positioned over the prefrontal cortex (PFC) forming 22 channels. The light sources emitted continuous wave near-infrared light at wavelengths of 760 nm and 850 nm. Channel forming optodes were placed 3 cm apart to ensure optimal signal penetration and coverage of the target brain regions was ensured with 22 channels. Sampling rate of the system was 7.81 Hz, and data were recorded continuously throughout the experiment. A schematic diagram of the probe design is presented in Fig 2 where EEG 10–20 coordinates corresponding to each optode is provided.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Probe configuration and montage design.

Each red dot corresponds to a light source while each blue dot represents a detector. Channels are formed between adjacent source-detector pairs, represented with yellow lines and labelled accordingly. Location of all channels are projected onto their EEG 10-20 coordinates.

https://doi.org/10.1371/journal.pone.0325850.g002

Preprocessing of fNIRS raw data was carried out by use of customized MATLAB scripts (Mathworks, Natick, MA, USA) and Homer3 toolbox [38]. The light intensity signals of each channel were converted to optical density (OD) values using the hmrR_Intensity2OD function. The OD data underwent motion correction using the hmrR_MotionCorrectSplineSG function with a parameter setting of p = 0.99 and a frame size of 10 seconds. Following motion correction, the OD data were converted to concentrations of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) using the hmrR_OD2Conc function where the differential pathlength factor (DPF) was selected as 1 for each wavelength. Bandpass filtering was applied with high and low cutoff frequencies of 0.01 Hz and 0.5 Hz to remove low-frequency drifts and high-frequency noise induced by heartbeat. Additionally, correlation-based signal improvement (CBSI) was applied to the bandpass-filtered data to further reduce motion artifacts and improve signal quality. Upon completion of preprocessing steps, HbO time series data of each channel were truncated into negative and positive stimuli blocks with a pre-stimulus baseline of 3 seconds and a post-stimulus duration of 10 seconds resulting in 25 second time traces for each stimulus condition. Neutral state data were obtained by truncating the rest interval HbO time series data of each channel into 20 second segments. Channel specific HbO signals corresponding to each stimulus condition were detrended prior to further analysis. Each emotional condition (i.e., negative, positive, neutral) had a total size of 3740 samples (22 channels*17 subjects *10 trials).

Feature extraction and classification

Twenty hemodynamic time domain features were extracted from the HbO time traces of each channel and stimulus block. The extracted features were 1) maximum, 2) range (i.e., amplitude difference between maximum and minimum signal values), 3) signal to noise ratio, 4) peak time, 5) skewness, 6) kurtosis, 7) variance, 8) median, 9) root mean square, 10) area under the curve and step response characteristics obtained with ‘stepinfo’ function implemented in MATLAB 2017B (The Mathworks Inc., Natick, MA, USA) which included 11) rise time, 12) settling time, 13) settling min, 14) settling max and 15–20) coefficients of a fifth-degree polynomial which was fitted to each channel and stimulus specific truncated time series HbO data.

The extracted features were utilized to train three supervised machine learning algorithms: Support Vector Machine (SVM), Ensemble (Subspace KNN), and k-Nearest Neighbors (kNN). These algorithms were selected based on their ability to handle small sample sizes effectively, their demonstrated accuracy in previous studies involving similar fNIRS features, and their low computational cost. Additionally, they have shown robust performance in classifying emotion-related data in neuroimaging research. Overall, as shown in Fig 3, the data analysis pipeline consists of several key stages, including conversion of optical signals into HbO data, preprocessing of HbO signals, feature extraction, and classification, which ultimately allowed prediction of each emotional state (i.e., positive, negative and neutral).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Overview of the Data Analysis Pipeline.

The pipeline consists of four main stages: (1) Conversion of raw optical signals into relative changes in HbO concentration using Modified Beer-Lambert Law; (2) Preprocessing of HbO signals through Correlation-Based Signal Improvement (CBSI) and a Butterworth bandpass filter; (3) Feature Extraction, where hemodynamic time-domain features are extracted(e. g. mean, variance, kurtosis…); and (4) Classification through machine learning algorithms which include k-Nearest Neighbors (KNN), Ensemble, and Support Vector Machine (SVM).

https://doi.org/10.1371/journal.pone.0325850.g003

For all three algorithms, labelled data underwent a random partitioning process where 80% of the dataset was preserved for training, 10% for validation, and 10% for testing. Classification performances of each algorithm were evaluated with 10 runs of a tenfold cross-validation procedure for two class and three class classification of positive, neutral and negative states.

All classification models were developed using MATLAB’s Classification Learner toolbox. Each classification method was optimized through a grid search process. The Ensemble method (Subspace kNN) combined multiple kNN models trained on different subsets of features. This approach resulted in a more reliable and generalizable model compared to using a single kNN classifier [39].

Briefly, the Ensemble (Subspace kNN) method combined decisions obtained from multiple kNN models, where each base kNN model was trained on a randomly selected subset of the original feature space. The subspace feature set utilized for training each base kNN classifier involved 10 randomly selected features. 30 base kNN learners were trained with 10 randomly selected features and the individual predictions of each kNN classifier were combined to form a final classification outcome through a majority voting approach. Selecting a random feature subset for each of the 30 base kNN learners allowed for some overlap across the feature subsets while ensuring sufficient diversity. Training multiple base kNN learners on smaller feature subspaces and obtaining a final decision for the Ensemble through a majority voting approach mitigated the risk of overfitting across the learners while reducing each classifier’s exposure to noise or less relevant features [39]. For each base kNN learner model, the initial data set consisted of 11,220 labelled samples (22 channels * 10 trials * 17 participants * 3 classes) before splitting into training, validation and testing sets. Specifically, each of the three emotional conditions (i.e., positive, negative, neutral) contributed 3,740 samples, derived from 17 participants * 22 channels * 10 trials. These condition-specific samples are concatenated, yielding a total of 11,220 rows with 11 columns (10 features + 1 label column) in the final feature matrix. For each base kNN learner of the Ensemble (Subspace kNN) model, the input feature matrix underwent an 80:10:10 random split for training, testing and validation purposes. Consequently, although each base kNN learner in the Ensemble spanned only a randomly selected subspace of features and each learner was trained on 8,976 samples (i.e., 80% of the samples are reserved for training) through the cross-validation procedure, ensuring a consistent and comprehensive use of the available data.

For the SVM algorithm, a 3^rd degree polynomial kernel was chosen. The hyperparameters were configured with a box constraint of 1 and automatic kernel scaling option (i.e., MATLAB automatically chooses the optimal bandwidth for the kernel while balancing margin maximization and misclassification tolerance). A “One‐vs‐One” multiclass scheme was implemented so that each of the three emotion classes would be distinguished by pair-wise binary SVMs. The choice of a cubic kernel (rather than a linear or quadratic one) was motivated by the need to capture complex, nonlinear relationships among the 20 fNIRS time‑domain features while avoiding overly flexible, higher‐order polynomial surfaces, which may result in overfitting. The cubic kernel’s flexibility was considered to be useful for distinguishing closely related states such as neutral vs. positive—while the box constraint and cross‑validation controlled for overfitting.

A single kNN model was trained by using all the features in the dataset and a relatively small number of neighbors (k = 1). The choice of a low number of neighbors allows kNN algorithm to be more sensitive to local variations in feature space. By focusing on only the nearest neighbor, the kNN method could capture the subtle distinctions among samples at its best.

Accuracy, precision, sensitivity, specificity, and F1-score of each algorithm were evaluated as the performance metrics by comparing the actual and predicted labels of validation data [25]. For a statistical comparison of the classification performances of all algorithms, the mean and standard deviation of each performance metric were computed over multiple runs for each algorithm (Table 1). For each performance metric, a one-way repeated measure ANOVA (main effect: classifier type) was initially performed to evaluate whether any significant difference existed among the performances of the three classification algorithms. Post hoc analysis was performed using paired t-tests, with the Bonferroni corrected significance threshold set at p < 0.05 for the following contrasts: SVM > kNN, SVM>Ensemble, Ensemble>kNN. Each performance metric (e.g., accuracy, specificity, recall, precision, and F1-score) was evaluated across the algorithms to determine whether any algorithm demonstrated statistically significant superiority in performance compared to the others.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Classification performances of Ensemble, SVM, and kNN algorithms. For each performance metric, the mean value across all runs and standard deviation of the mean value is reported in percentages (%).

https://doi.org/10.1371/journal.pone.0325850.t001

Performance comparison

Our results indicated that SVM performed statistically significantly better than the Ensemble (Subspace kNN) and the kNN algorithms in terms of accurate classification of the three emotional states tested in the present study. The superior performance of SVM in both two-class and three-class classification scenarios suggests that SVM is particularly well-suited for fNIRS-based emotion recognition tasks. This result can be related to several factors associated with the nature of fNIRS data and the characteristics of the SVM algorithm.

fNIRS datasets display high-dimensional, nonlinear characteristics besides involving inherent complex hemodynamic patterns that are associated with the neural processing of emotional stimuli [7,17,46,47]. SVMs are particularly effective in handling high-dimensional, complex datasets, such as fNIRS signals, primarily due to their ability to efficiently manage dimensionality and nonlinearity issues through their margin-based learning principle and the kernel trick property. Specifically, the margin-based principle seeks the optimal hyperplane that maximizes the geometric margin (i.e., distance) between classes [48]. By relying only on the support vectors that are formed between data points closest to the decision boundary, the margin-based principle makes SVM naturally robust to noisy data points, as the majority of the data do not influence positioning of the decision boundary. Hence, outliers or noisy points have no effect on the decision boundary once the margin is established. In contrast, kNN based classification approaches are distance-based methods which weigh decisions from all samples equally. This property causes decisions obtained from kNN based approaches to be more affected by noisy data points or sample size imbalances as they treat all data points equally.

fNIRS signals exhibited non-linear relationships between features and emotional states due to the intricate hemodynamic responses in the brain [29,49]. With kernels, SVMs can model very complex, nonlinear decision boundaries, even in very high dimensions. In our study, we used a kernel-based SVM which allowed for mapping input features into higher-dimensional spaces where non-linear patterns became linearly separable [50]. The kernel trick feature of SVM is especially advantageous for fNIRS data, where complex, overlapping patterns of HbO features can arise from different cortical regions and physiological states [17,29]. Consequently, even subtle distinctions among emotional responses may become easier to identify. Meanwhile, KNN based approaches have no inherent mechanism for learning nonlinearity as their decisions rely solely on local distance between neighbor data points.

From a dataset size perspective, SVM tends to handle relatively small sample sizes robustly, partly due to its focus on maximizing margins [48,51]. Margin maximization prevents overfitting by ensuring that only the critical points closest to the boundary define the classification model through their geometric arrangement, making it resilient even in sparse, high-dimensional feature spaces. This feature makes SVM highly resistant to overfitting—particularly valuable in neuroimaging datasets such as fNIRS, where sample sizes are limited, and data complexity is high. Ensemble Subspace kNN can likewise mitigate overfitting by aggregating base learners, yet if certain random feature subsets are less discriminative, those base learners may yield weaker local decision boundaries. These marginal drops in performance across multiple base learners may have diminished the Ensemble’s overall accuracy when compared to the performance of SVM.

In summary, the SVM classifier’s margin-based optimization, coupled with its robust kernel methods and inherent noise resilience might have provided advantage over both single kNN and Ensemble (Subspace kNN) classifiers. This combination effectively harnessed the complex, nonlinear characteristics of fNIRS data, underscoring SVM’s suitability and reliability in emotion recognition tasks based on neurophysiological signals.

From an application standpoint, the consistently strong performance of SVM highlights a computationally efficient yet highly accurate approach to fNIRS-based emotion recognition. In scenarios demanding real-time classification—such as human–computer interaction or clinical neurofeedback—an SVM classifier can be deployed on modest hardware due to its relatively fast inference time once trained [10,29]. Furthermore, the high specificity across all algorithms reinforces the conclusion that the extracted time-domain features are indeed physiologically meaningful for emotional state differentiation. SVM’s particular advantage in handling nonlinear separability suggests that researchers might explore alternative kernel functions or hyperparameter tuning to further refine classification for more granular or overlapping emotional categories [17,44].

To sum up, our results indicate that SVM’s margin-focused, kernel-enabled approach is particularly well-suited to complex, high-dimensional fNIRS data obtained for emotion recognition. This method’s robustness, interpretability, and moderate computational cost underscore its feasibility for diverse real-time and clinical applications, pushing the envelope of single-modality brain–computer interfaces in detecting and classifying emotional states with both high accuracy and specificity.

Limitations of the study and future directions

The sample size of obtained data was relatively small when compared to other application areas of machine learning (e.g., automation, finance). Nonetheless, previous emotion BCI studies conducted with fNIRS, and EEG modalities also trained their classification algorithms with similar sizes of training data. Future work will involve testing the proposed methodology on a larger subject cohort and higher number of labelled trials. We will also test the discriminative power of FNIRS-derived hemodynamic features obtained during a variety of naturalistic emotional states.

Deep learning (DL) approaches have shown promise for enhancing the accuracy of fNIRS-based BCI systems in classification, prediction or decoding of phenomenal states, when substantial amount of training data is available [52]. As a major strength, DL algorithms have the ability to capture the nuanced patterns in hemodynamic activity by use of advanced techniques applied for optimization of network architectures [53]. Indeed, previous work has successfully integrated DL models with both fNIRS and EEG data [54]. Since training DL models with training data sample size less than 5000 is reported to increase the risk of overfitting, our study did not incorporate DL algorithms because each emotional state was represented by a relatively small dataset [52]. Future studies will focus on evaluating DL methods for this classification problem using larger cohorts and data augmentation strategies to address the limitations posed by the small sample size of our study.

In clinical contexts, reliably detecting subtle shifts in emotional valence or arousal using fNIRS could support personalized mental health interventions, biofeedback for emotional regulation, or real-time monitoring in psychiatric settings [19,21]. For human–computer interaction, incorporating an SVM-based emotional state decoder can lead to more adaptive and responsive interfaces, enabling applications such as personalized learning environments or user-tailored entertainment systems [3,4]. As larger datasets become available, comparing SVM’s margin-based strength to the feature-learning capacity of deep neural networks would be an important next step. This could confirm whether SVM remains advantageous in data-limited contexts or whether neural architectures eventually surpass SVM once sufficient fNIRS data are amassed for stable training [10,52].