A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts
Fig 3
Cross prediction analysis shows that responses are generalizable between unimodal and multimodal stimulus information, with stronger generalizability for visual information compared to auditory.
A) Model performance for audio-only (A) test sets with A-only training data (x-axis) or audiovisual (AV) training data (y-axis), calculated as the linear correlation between predicted and actual held out EEG test data. Features for this model included phonological features, the acoustic envelope, and pitch. Each dot represents an individual electrode for an individual EEG subject (64 channels x 11 participants). Dashed black line = unity line; red line = regression line. B) Model performance for visual-only (V) test set with V-only training data or AV training data. Features for this model included the 10 Gabor PCs and scene cuts. Similar model performance was observed for both within- and cross-condition predictions, though this relationship was stronger between V and AV. C) Model performance for comparing visual only responses using either scene cuts or only Gabor feature representations with the individual feature in the audiovisual condition. D) Normalized correlation coefficient between each EEG channel for audiovisual and visual only conditions and audiovisual and audio only conditions. Overall, single trial bandpass filtered EEG (input to the model) was more correlated between the AV and V only conditions as compared to the AV and A only conditions, suggesting a strong influence of visual information on the EEG signals.