Fig 1.
Conceptual illustration of the affective music BCI.
(A) During calibration, the user is exposed to automatically generated patterns of affective music, and brain activity is measured simultaneously via EEG. EEG patterns are extracted and used to build a user-specific emotion model. (B) During online application, the obtained model is used to continuously translate the user’s brain activity into affective music, thereby closing the loop through continuous affective brain interactions.
Fig 2.
Schematic illustration of the automatic music generation system.
(A) mapping of valence and arousal onto music structural parameters (a), state-machine to generate MIDI patterns (b), virtual instruments to translate MIDI patterns into sound (c). (B) Exemplary synthesized musical trajectory through affective space (valence-arousal-model according to [38]).
Fig 3.
Signal processing and information flow during calibration (top row) and online application (bottom row) of the system. Note, the dual use of the automatic music generation system in both cases for providing either open-loop stimulation (calibration) or closed-loop feedback (application). The black dashed lines indicate model-specific parameters (filter parameters, standardization and baseline parameters, and classifier parameters) that were obtained during calibration, and utilized during online application.
Fig 4.
BCI study experiment protocol (A) and trial structure (B). (A) Each session began with a questionnaire, followed by a calibration phase and subsequent modeling of the emotion classifier. Afterwards, the main part of the experiment, the online application phase, was conducted. The experiment ended with a second questionnaire. (B) A single trial started with a resting period of 15 sec in which no music feedback was presented to the participant. The participant‘s EEG was recorded in the background, and the average score was used to assign, on every trial, one of the two tasks (modulate towards happy or sad) to the participant. The subject then performed the task for a duration of 30 sec (action period).
Fig 5.
Listening study perceptual ratings.
(A) Interpolated valence- and (B) arousal-ratings averaged across subjects. (C) Distribution of per-subject Pearson correlation coefficients between perceptual ratings and the music generation system‘s parameter settings for valence and for arousal.
Table 1.
The music generation system’s parameter settings (target valence/arousal) and corresponding perceptual ratings for valence and arousal (MEAN±SD, n = 11).
Table 2.
Offline model performance results.
Offline model performance based on 100-times-10-fold cross-validation for all subjects (P01-P05) and sessions (S01 and S02). Offline decoding performance is expressed in form of percentage of correctly classified observations per class (happy and sad), overall correctly classified instances (ACC) and via the area under receiver operator curve (AUC).
Fig 6.
Average modulation performance.
The bars represent the mean difference between the model score of the action period minus the corresponding resting period, averaged across trials (Eq 11) separated in sessions (S01: A and B; S02: C and D) and task (→ happy: A and C; → sad: B and D). Numbers in parentheses denote the number of trials for the respective participant/task/session combination and the asterisks (*) those combinations with significant modulation performance.
Table 3.
Correlation coefficients, across all participant-session combinations for task ‘modulate towards happy’ (→happy) and task ‘modulate towards sad’ (→sad), between performance measures (resting mean (RM), action mean (AM), total deviation (TD)) and mood assessment ratings (for the six significant mood scales of 14 total ratings made by participants).
Fig 7.
Brain activity modulations during online application in task → happy averaged across participants P01, P04, and P05.
Music feedback modulations toward the happy state were accompanied by significant power decrease in beta band over frontal areas (a), as well as an increase in gamma power over the right hemisphere (b). The highlighted channels indicate Bonferroni-corrected statistically significant modulations comparing action and resting periods (action minus rest), (p < 0.05).
Fig 8.
Granger-causal information flows during task →happy in session S02 jointly computed for P01, P04, P05 (participants with good modulation performance).
indicates causal flow from feedback to brain activity features (top row) and
from brain activity features to feedback (bottom row). The g-causalities are statistically tested for significance (p < 0.01) and Bonferroni-corrected.