A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts
Fig 2
Comparison between auditory and visual feature models to predict EEG.
A) Scatter plot showing the model comparison (r-value) between AV and A conditions, where the original (AV) movie trailer was present, or an audio only version (A). In both cases, the same set of auditory features were used to predict the data. Each gray dot represents the encoding model performance in a single channel in a single participant. Channel FT8 is shown for subject MT0033 with the corresponding weight matrices for each condition (A condition: top left, AV condition: bottom right) and associated correlation value for the channel. B) Grand average correlation values for AV and A condition plotted on topographical map and averaged across all participants (n = 11). Topography of selective channels was similar for AV and A. C) Average difference in prediction performance for AV and A for all participants. D) Average correlation between weights between acoustic and linguistic features from A and AV across all participants. The receptive field structure was similar over temporal, frontal, and central sensors. E) Scatter plot showing the model comparison (r-value) between AV and V conditions, where the original (AV) movie trailer was present, or a visual only version (V). Visual feature model used a combination of 10 Gabor wavelet filter principal components (PCs) and scene cut (SC) information. Each gray dot represents the encoding model performance in a single channel in a single participant. An example channel, P4, is shown for subject MT0029 with the corresponding weight matrices for each condition and associated model performance (correlation value) for the channel. F) Grand average correlation values for AV and V condition plotted on topographical map and averaged across all participants (n = 11). Similar spatial distribution of good model performance was observed regardless of condition. G) Same as C for visual feature models fitted with AV and V data. H) Average correlation between weights between visual features for V and AV across all participants. Receptive field structure was most similar over occipital sensors.