Fig 1.
PID of audiovisual speech processing in the brain.
(A) Information structure of multisensory audio and visual inputs (sound envelope and lip movement signal) predicting brain response (MEG signal). Ellipses indicate total mutual information I(MEG;A,V), mutual information I(MEG;A), and mutual information I(MEG;V); and the four distinct regions indicate unique information of auditory speech Iuni(MEG;A), unique information of visual speech Iuni(MEG;V), redundancy Ired(MEG;A,V), and synergy Isyn(MEG;A,V). See Materials and methods for details. See also Ince [15], Barrett [21], and Wibral and colleagues [22] for general aspects of the PID analysis. (B) Unique information of visual speech and auditory speech was compared to determine the dominant modality in different areas (see S1 Fig for more details). Stronger unique information for auditory speech was found in bilateral auditory, temporal, and inferior frontal areas, and stronger unique information for visual speech was found in bilateral visual cortex (P < 0.05, FDR corrected). The underlying data for this figure are available from the Open Science Framework (https://osf.io/hpcj8/). Figure modified from [15, 21, 22] to illustrate the relationship between stimuli in the present study. FDR, false discovery rate; MEG, magnetoencephalography; PID, partial information decomposition.
Fig 2.
MI between auditory and visual speech signals.
(A) To investigate PID in “AV congruent” condition, first MI between auditory speech and visual speech signals was computed separately for matching and nonmatching signals. MI for matching auditory-visual speech signals shows a peak around 5 Hz (red line), whereas MI for nonmatching signals is flat (blue line). The underlying data for this figure are available from the Open Science Framework (https://osf.io/hpcj8/). (B) Analysis of PID is shown for “AV congruent” condition in which both matching and nonmatching auditory-visual speech signals are present on the same brain response (MEG data). Two external speech signals (auditory speech envelope and lip movement signal) and brain signals were used in the PID computation. Each signal was band-pass filtered, followed by Hilbert transform. MEG, magnetoencephalography; MI, mutual information; PID, partial information decomposition.
Fig 3.
Redundancy and synergy revealed by PID for attention-modulated speech processing (“AV congruent” condition).
Redundant and synergistic information of matching audiovisual speech signals in the brain compared to nonmatching signals are shown. Each map (matching or nonmatching in each information map) was firstly yielded to regression analysis using speech comprehension and then transformed to standard Z maps and subtracted. (A) Redundant information is localized in left auditory and superior and middle temporal cortices. (B) Synergistic information is found in left motor and bilateral visual areas (Z-difference map at P < 0.005). The underlying data for this figure are available from the Open Science Framework (https://osf.io/hpcj8/). PID, partial information decomposition.
Fig 4.
Redundancy and synergy in congruence effect.
Comparison between conditions of matching versus nonmatching audiovisual speech signals in “AV congruent” condition entails both attention and congruence effects. To separate this effect, we additionally analyzed contrast for congruence (“AV congruent” > “All incongruent”) first. (A) Redundancy for congruence effect is observed in left inferior frontal region and pSTG/S and right posterior middle temporal cortex (Z-difference map at P < 0.005). (B) Synergistic information for congruence effect is found in superior part of somatosensory and parietal cortices in left hemisphere (Z-difference map at P < 0.005). The underlying data for this figure are available from the Open Science Framework (https://osf.io/hpcj8/). pSTG/S, posterior superior temporal gyrus/sulcus.
Fig 5.
Redundancy and synergy in attention effect.
Redundancy and synergy in attention (“AV congruent” > “All congruent”) are analyzed. Further, to explore whether this effect is specific to “AV congruent” condition (not because of decreased information in “All congruent” condition), we extracted raw values of each information map at the local maximum voxel and correlated it with speech comprehension accuracy across subjects. (A) Redundancy for attention effect was observed in left auditory and temporal (superior and middle temporal cortices and pSTG/S) areas and right inferior frontal and superior temporal cortex (Z-difference map at P < 0.005). (B) Synergistic information for attention effect was localized in left motor cortex, inferior temporal cortex, and parieto-occipital areas (Z-difference map at P < 0.005). (C) Redundancy at the left posterior superior temporal region in “AV congruent” condition was found to be positively correlated with speech comprehension accuracy (R = 0.43, P = 0.003). However, this redundant representation was not found for left motor cortex where synergistic information was represented (R = 0.21, P = 0.18). (D) Synergy at the left motor cortex in “AV congruent” condition was also positively correlated with speech comprehension accuracy across subjects (R = 0.34, P = 0.02). Likewise, synergistic representation was not found to be related to comprehension in the left posterior superior temporal region where redundant information was represented (R = 0.04, P = 0.81). This finding suggests that redundant information in the left posterior superior temporal region and synergistic information in the left motor cortex in a challenging audiovisual speech condition support better speech comprehension. The underlying data for this figure are available from the Open Science Framework (https://osf.io/hpcj8/). N.S., not significant; pSTG/S, posterior superior temporal gyrus/sulcus.