Fig 1.
Example trial and experimental design.
(A) In a spatial ventriloquist paradigm, participants were presented with synchronous audiovisual (AV) signals originating from four possible locations along the azimuth. The visual signal was a cloud of white dots. The auditory signal was a brief burst of white noise. Participants localized either the auditory or the visual signal (n.b. for illustrational purposes the visual angles of the cloud have been scaled in a non-uniform fashion in this scheme). (B) The four-factorial experimental design manipulated (1) the location of the visual (V) signal (−10°, −3.3°, 3.3°, 10°) (2) the location of the auditory (A) signal (−10°, −3.3°, 3.3°, 10°), (3) the reliability of the visual signal (high versus low standard deviation of the visual cloud), and (4) task-relevance (auditory versus visual report).
Fig 2.
Histograms of response deviations (across-subjects mean fraction ± standard error of the mean [SEM]) as a function of (i) task relevance (i.e., auditory versus visual report) (ii) audiovisual disparity, and (iii) visual reliability.
If participants were able to locate the task-relevant auditory or visual signal precisely, the histogram over response deviations would reduce to a delta function centered on zero. The histograms of response deviations for auditory report indicate that a spatially disparate visual signal biases participants’ perceived sound location in particular when the visual signal is reliable. In each panel, stimulus symbols (i.e., auditory: loudspeaker; visual: cloud of dots) indicate the location of the task-relevant signal (centered on zero) and the task-irrelevant signal (centered on the discrepant spatial location). The data used to make this figure are available in file S1 Data.
Table 1.
Model parameters (across-subjects mean ± standard error of the mean) and fit indices of the three computational models.
Fig 3.
Bayesian Causal Inference model and cortical hierarchies.
(A) Participants were presented with auditory and visual spatial signals. We recorded participants’ psychophysical localization responses and fMRI BOLD responses. (B) The Bayesian Causal Inference model [2] was fitted to participants’ localization responses and then used to obtain four spatial estimates for each condition: the unisensory auditory (ŜA,C=2) and visual (ŜV,C=2) estimates under full segregation (C = 2), the forced-fusion estimate (ŜAV,C=1) under full integration (C = 1), and the final spatial estimate (ŜA, ŜV) that averages the task-relevant unisensory and the forced-fusion estimate weighted by the posterior probability of each causal structure (i.e., for a common source: p(C = 1|xA, xV) or independent sources: 1 − p(C = 1|xA, xV). (C) fMRI voxel response patterns were obtained from regions along the visual and auditory hierarchies (V, visual sensory regions; A1, primary auditory cortex; hA, higher auditory area; IPS, intraparietal sulcus). (D) Exceedance probabilities index the belief that a given spatial estimate is more likely represented within a region of interest than any other spatial estimate. The exceedance probabilities for the different spatial estimates are indexed in the length of the colored areas of each bar (n.b. the y-axis indicates the cumulative exceedance probabilities). The data used to make this figure are available in file S1 Data.