Fig 1.
The five tasks and stimuli in our experiment.
(A) In unisensory localization trials, participants were presented with either a visual stimulus coming in one out of three reliability levels (UV task) or an auditory stimulus (UA task), and were asked to provide a location estimate for the stimulus. In the bisensory trials, participants were simultaneously presented with one visual and one auditory stimulus that either shared or did not share the same location. In the bisensory causal judgment (BC) task, participants were asked to report whether the two stimuli shared the same location; in the bisensory localization tasks, participants were asked to provide a location estimate for either the visual (BV) or auditory (BA) stimulus. (B) The visual stimulus’s location in each trial is sampled uniformly within , while the auditory stimulus’s location is sampled from discrete values
with equal probability. In bisensory trials, we force the visual stimulus location to be identical to the auditory stimulus (C = 1) with probability 0.5 in each trial. (C) The visual stimulus is a Gaussian luminance spot that comes in one out of three possible reliability levels (defined by the luminance spot’s radius). The open-source speaker icon was adopted from Wikimedia Commons at https://commons.wikimedia.org/wiki/File:Speaker_Icon.svg.
Fig 2.
Behavior in unisensory visual (UV) and auditory (UA) tasks.
Each column shows participants’ data (error bars; mean ± SEM across N = 15 participants) and model fits (ribbons) for the “vanilla” Bayesian observer model, for different summary statistics. Rows correspond to UV trials (in three visual reliability levels) and UA trials. (A) Full response distributions stratified by true stimulus location into 7 evenly spaced stimulus bins (colors); (B) Mean location estimation bias (vertical axis), evaluated separately for each of the 7 stimulus bins (error bars are located at the bin centers along the horizontal axis); (C) Standard deviation (SD) of stimulus location estimates (vertical axis), evaluated separately for each of the 7 stimulus bins (horizontal axis).
Fig 3.
The generative process and Bayesian observer models for our tasks.
(A) For unisensory tasks UV and UA, observers have access to a noisy measurement x of the stimulus s being presented. They perform Bayesian inference to obtain the posterior mean stimulus value , which is corrupted by Gaussian motor noise with SD
and occasional, uniformly distributed lapses with lapse rate
to reach the final location estimate response rS, for
. (B) For bisensory tasks BC, BV, and BA, observers consider two candidate hypotheses—that the auditory or visual stimuli either share (C = 1) or do not share (C = 2) the same location—and perform Bayesian inference accordingly. In the BC task, the posterior probabilities of either hypothesis
are processed by a causal inference strategy: Model Selection (MS), Model Averaging (MA), or Probability Matching (PM) (MS and MA are equivalent strategies for BC); and corrupted by occasional uniformly distributed lapses to reach a category response rC. In BV and BA tasks,
for i = 1,2 and the posterior mean location estimates under each hypothesis
are processed by a causal inference strategy and corrupted by Gaussian motor noise and occasional uniformly distributed lapses, ultimately reaching a location estimate response rS. (C) In the above process, the shape of the sensory noise function
influences both the generation of noisy observations and Bayesian inference, while the prior over stimulus location p(s) influences the latter. The causal inference strategy is only relevant for bisensory tasks. For simplicity, this figure does not depict further modeling elements: the binary prior over causal inference hypotheses psame, the auditory range recalibration parameter
, the rescalings applied to visual sensory noise for medium- and low-reliability trials
, nor the unisensory-to-bisensory rescaling parameters on sensory noise
. See text for more details. The open-source dice icon was adapted from Wikimedia Commons at https://commons.wikimedia.org/wiki/File:Dice_simple_flat.svg.
Fig 4.
Semiparametric fits on unisensory data.
Different colors denote different human participants. (A) Fitted shapes for each participant, where points denote the pivot locations and their corresponding function values. Inner plot: the same plot over the full stimulus range
; the smaller stimulus range used in the outer plot is denoted by a black rectangle in the inner plot. (B) Fitted
shapes. Inner plot: the same plot over the full stimulus range
. (C) Fitted p(s) shapes. Inner plot:
over the full stimulus range
. (D) Other fitted parameters. Box plots denote median and interquartile range, and whiskers the full range, with dots being individual participants.
Fig 5.
The semiparametric model fitted jointly on UV and UA data for all participants.
Subplot notations are identical to Fig 2.
Fig 6.
The lifted-semiparametric fits on all tasks, assuming the PM causal inference strategy, visualized on BC, BV, and BA data.
All response distributions are constructed from data (error bars) and model (ribbons) predictive samples; mean±SEM across participants. Rows correspond to different visual reliability levels. (A) BC response distributions, denoting the relative frequency of responding that the two stimuli share the same source (“same” responses), as a function of binned true stimulus location disparities. Columns are stratified into ‘center’ and ‘periphery’ based on the sum of the two true stimuli locations (absolute value below or above the corresponding median across all BC trials). (B) BV (blue) and BA (green) response distributions, denoting the mean estimation bias as a function of binned true stimulus location disparities. Columns are stratified into ‘left’, ‘center’ and ‘right’, based on the sum of the two true stimuli locations (25th and 75th percentiles across all BV or all BA trials).
Fig 7.
Model comparison results (BIC) for parametric models fitted on UV+UA tasks data.
The bars report difference in BIC between the model and the best-performing model (Exp-GaussianLaplace), with higher values for worse-fitting models. Error bars are bootstrapped 95% CI.
Fig 8.
The Exp-GaussianLaplace parametric model fitted jointly on UV and UA data for all participants, in which the auditory range recalibration factor is a free parameter.
The layout and color schemes are identical to Fig 5.
Fig 9.
Model comparison results (BIC) for Probability Matching (PM) parametric models fitted on all tasks.
For the LiftedSemiparam-PM model, many of its parameters were fitted in the earlier semiparametric fits. Its BIC score including such parameters is shown in white, and the BIC score excluding them is shown in gray.
Fig 10.
The Exp-GaussianLaplace-PM parametric model fitted on all tasks, visualized on UV + UA data.
UV + UA response distribution legends are identical to Fig 2.
Fig 11.
The Exp-GaussianLaplace-PM parametric model fitted on all tasks, visualized on BC, BV, and BA data.
Response distribution notations are identical to Fig 6.
Fig 12.
Responses of one example participant and their Exp-GaussianLaplace parametric model fitted jointly on UV and UA data.
(A) UV and UA data, bias and SD of responses. (B) BC data, relative frequency of responding “same” over stimulus location disparity. (C) BV data, mean estimation bias over stimulus location disparity. (D) BA data, mean estimation bias over stimulus location disparity. The subfigure-stratification and trial-binning processes for each task is identical to its group-level figure counterpart, but now with points denoting human data and lines denoting model predictions.
Fig 13.
Illustrative examples of the sensory noise families and prior parametric families used in our parametric Bayesian observer models.
All shapes are defined over the range .
Table 1.
Summary of models fitted. Numbers in parentheses convey the number of free parameters incurred by each modeling component (for sensory noise, the number of free parameters is multiplied by 2 due to separate ). The auditory range recalibration parameter is only systematically varied in the unisensory fits (*), and always free in bisensory fits. The semiparametric model is fitted only on unisensory data (**), while its fitted sensory noise and prior functions are then used for LiftedSemiparam models fitted to bisensory data (***). The acronyms MS, MA, and PM stand for model selection, model averaging, and probability matching causal inference strategies, respectively.
Fig 14.
Unisensory model recovery results.
Rows correspond to different simulated datasets (groups of 15 simulated participants), each named after their ground-truth generative model. Within each row, the columns are different candidate models fitted to the same simulated group (row and column orders are identical, hence omitted). Within each row, the matrix entries are NLL/AIC/BIC differences between each fitted model to the ground-truth model (the row’s diagonal entry, which is always the baseline of 0). Color indicates whether the NLL/AIC/BIC difference is positive (green) or negative (pink), visualized in log scale (colorbar). Successful model recovery implies that the diagonal entries have lowest AIC/BIC within their row, and thus all matrix entries should be positive (green). (A) NLL differences. (B) AIC differences, (C) BIC differences.
Fig 15.
Unisensory parameter recovery results for the Exp-GaussianLaplace model.
Panels correspond to different free parameters in the model (a total of 14). Data points denote the 15 simulated participants. In each panel, the horizontal axis denotes the ground-truth generative parameter values for simulated participants. The vertical axis denotes the recovered parameter values. The green solid line denotes unity (perfect parameter recovery).