The relay model explains multisensory processing dynamics

To evaluate the plausibility of the relay model, we tested whether its predictions align with empirical RTs reported in Miller’s benchmark dataset (data estimated from Figure 1 in [4]; see Methods). As a key advancement in understanding multisensory processing, the model assumes that perceptual decision processes comprise latent sequential processing stages. For example, an auditory RT is modelled as the sum of two stages (e.g., RT_A = T_A1 + T_A2).

To constrain the model, we fitted IG distributions to the overall RTs in unisensory auditory and visual conditions (Table 1; see Methods). This distribution is characterised by two parameters: the mean (μ) and the shape parameter (λ). We estimated these parameters by minimising the root mean squared error (RMSE) between empirical and model CDFs. These unisensory parameters were held fixed to constrain the modelling of the multisensory condition.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Data summary and best-fitting unisensory distribution parameters.

https://doi.org/10.1371/journal.pcbi.1013320.t001

To partition the overall RT into stage-specific latent components, we introduce the RT share parameter (Fig 2a). It expresses the contribution of the first stage as a proportion of the overall RT. For instance, if RT_A is on average 400 ms and the first stage (T_A1) lasts 80 ms, then T_A1 contributes 20% of the total RT. The variability of RTs across trials is partitioned analogously.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Predictions of the relay model.

a) RT share. In the relay model, the RT share denotes the proportion (%) of the overall RT attributed to the first stage: 20% (left) and 50% (right). While these schematic diagrams illustrate this partitioning for a single trial, at the distribution level, the RT share parameter specifies how the entire RT distribution (both its mean and variability) is divided into two sequential stages for each modality. The predicted multisensory RT distribution is then formed by summing the outcomes of the independent races at each stage. Schematic diagrams correspond to the model architecture introduced in Fig 1c. b) Modelling based on data from Miller [4]. The relay model is constrained by the RT distribution for auditory (A) and visual (V) signals (Table 1). The shaded region depicts relay model predictions for RT shares ranging from 0% to 50%. The empirical RT distribution for redundant audiovisual (AV) signals is best fit with an RT share of 22.6%. Circles represent the same data shown in Fig 1b. c) RSE and d) RMI violations as a function of RT share. The relay model predicts RSEs that exceed those of Raab’s race model, including systematic RMI violations. Predictions decline symmetrically for RT shares above 50% (e.g., RT shares of 20% and 80% yield identical predictions).

https://doi.org/10.1371/journal.pcbi.1013320.g002

We formalised this partitioning approach using the IG distributions fitted to the unisensory conditions. Specifically, we exploit a property of the IG distribution that allows for mathematically consistent partitioning, whereby the overall RT can be split into sequential stages using the RT share parameter, while each latent stage remains IG-distributed. This property preserves analytical tractability, allowing us to model the latent processing stages without resorting to numerical simulations. For simplicity, we used a single RT share parameter for both modalities; however, the model can be extended to support distinct RT shares per modality, which would reflect differences in sensory latencies. The RT share thus expresses the relative contribution of the first of two IG-modelled processing stages, thereby shaping how the system transitions across sensory modalities and stages.

To predict RTs in multisensory conditions, the relay model assumes a non-competitive relay architecture (Fig 1c; Eq 4). During the first gating stage, unisensory processes (e.g., T_A1, T_V1) race to completion. The winner then initiates the second decision stage, triggering a race between the second stage processes (e.g., T_A2, T_V2). The winner of this second race determines the behavioural response. Crucially, this architecture enables cross-modal initiation: for example, a visual stage 1 winner can trigger an auditory stage 2 process. This feature distinguishes the relay model from the classical race model.

To examine how the RT share affects model behaviour, we simulated the relay model across RT share values from 0% to 50% (Fig 2b). As the two stages add up to 100%, this range encompasses the full dynamic range (values greater than 50% are symmetric). At 0% RT share, the relay model reduces to the classical race model (Fig 1a; Eq 2), which predicts an RSE of 48 ms. In contrast, the relay model predicts an amplification of the RSE up to 72 ms (1.5 × the race model) at 50% RT share (Fig 2c). The race model also predicts no violations of the RMI, but with an RT share of 50%, the relay model predicts RMI violations in the order of 10 ms (Fig 2d). Thus, the RT share, as a single interpretable parameter that implements the relay architecture, enables the model to capture the key effects observed in human multisensory behaviour.

To evaluate the model’s fit to empirical data, we estimated the RT share by minimising the RMSE between empirical and predicted multisensory CDFs (Fig 1b). For parsimony, we again constrained auditory and visual RT shares to be equal. The best-fitting RT share was 22.6% (RMSE = 0.023), corresponding to first-stage components of 89.9 ms (SD = 31.4 ms) for audition, and 90.6 ms (SD = 21.3 ms) for vision. To compare the model fit to the empirical data, we computed RSE and RMI violations at the ten quantiles (as opposed to over the whole distribution as in Fig 2c, d). The best-fitting model predicted an RSE of 68 ms (empirical: 67 ms) and RMI violations of 8 ms (empirical: 10 ms). The slight attenuation of the predicted violation is most likely attributable to the constraints of the unisensory fits; specifically, the IG distribution does not perfectly capture the extreme fast tail of the empirical RT distributions, where RMI violations are most prominent (see fastest quantiles in Fig 2b). Yet, overall, the model captures the distributional shape of the AV condition with high accuracy.

To further validate the relay model, we compared its performance against a “shifted” race model, an alternative ad-hoc formulation that can also be used to explain RMI violations with a single free parameter. In this model, the classical race model prediction is simply shifted in time by a free parameter, δ, effectively translating the entire distribution to faster RTs. Conceptually, the shift may indicate a non-specific alerting effect, where the presence of multisensory signals elevates baseline arousal and accelerates RTs. The shifted race can account for RMI violations in Miller’s data [4] with a shift of 19.2 ms. However, it provides a poorer fit to the dataset (RMSE = 0.032) compared to the relay model (RMSE = 0.023), which thus shows better explanatory power.

In summary, with only one additional degree of freedom beyond the fits obtained from unisensory conditions, the relay model successfully reproduces a hallmark signature of multisensory integration: larger RSEs than predicted by the race model, including RMI violations. These findings demonstrate that a simple relay architecture offers a parsimonious and powerful framework for modelling the dynamics of multisensory processing.

Predictions of multi-stage relay models

While our initial analyses focused on a two-stage relay architecture, the model is not inherently restricted to two processing stages. In principle, multisensory decision-making could involve three or more stages, with each stage comprising a race between modalities (Fig 3a). This raises a key question: how many relay stages account for empirical multisensory RTs best?

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Multi-stage relay models.

a) Number of stages. Relay models with two and three sequential stages are shown. Each stage passes a start signal to the next, with the overall RT evenly distributed across stages. The two-stage schematic diagram corresponds to the architecture introduced in Fig 1c. b) Modelling based on data from Miller [4]. Greyscale lines depict predicted cumulative RT distributions for increasing numbers of stages, from two up to ten. Circles represent the same data shown in Fig 1b. c) RSE and d) RMI violations as a function of the number of relay stages. Relay models with three or more stages predict RSEs and RMI violations that far exceed empirical observations. Predictions from Raab’s race model (one stage) are indicated by red circles for reference.

https://doi.org/10.1371/journal.pcbi.1013320.g003

To address this, we simulated relay architectures with one to ten stages, using the same unisensory IG fits to constrain each model (Fig 3b). In every simulation, the total RT was evenly divided across N stages, such that each stage was allocated 1/N of the total RT. Accordingly, the RT share was held fixed at this proportion (i.e., not optimised). This design enabled us to assess how increasing the stage count affects predicted RSEs and RMI violations, while holding model complexity and parameterisation constant.

Simulation results show that both predicted RSEs and RMI violations increase with stage count, but with diminishing returns (Fig 3c,d). The most substantial gain occurs between the race model and the two-stage relay model. Adding more stages yielded progressively smaller gains. Crucially, higher stage counts led to increasingly exaggerated predictions. For example, the ten-stage model predicted an RSE of 156 ms and RMI violations of 84 ms, which far exceed empirically observed values.

To evaluate these architectures quantitatively, we compared predicted to empirical multisensory CDFs (Fig 3b). Because the RT share was fixed (rather than optimised) and all models shared identical unisensory parameters, this provided a fair, complexity-controlled comparison. The two-stage relay model emerged as the best configuration, yielding the lowest RMSE (RMSE = 0.030), more than twice as accurate as the one-stage race model (RMSE = 0.085). Adding further stages degraded model performance (e.g., three-stage RMSE = 0.102; ten-stage RMSE = 0.399), consistent with their overestimation of RSE and RMI effects.

Crucially, the assumption of evenly distributed stages is not a strict requirement of the relay architecture. In a relay with more than two stages, the RT shares can be asymmetric. Indeed, perceptual decision-making can be thought of as asymmetric stages of sensory encoding, evidence accumulation, and motor execution, where the vast majority of RT is dedicated to the accumulation of evidence [32,48,60]. For instance, an alternative three-stage relay configuration could consist of a brief initial stage (10%), a longer middle stage (80%), and a brief final stage (10%). This 10-80-10 distribution yields predictions much closer to the two-stage model (RMSE = 0.039). Therefore, while a two-stage relay is sufficient to capture the current data, there are likely situations where asymmetric, multi-stage relays can be modelled successfully.

In summary, while the relay architecture can, in principle, support an arbitrary number of stages, the two-stage configuration strikes the optimal balance. It captures observed multisensory enhancements without inflating predictions beyond empirical observations, though unequal multi-stage models remain a viable architecture.

Temporal dynamics of multisensory decisions

Having established that the two-stage relay model captures both RSE and RMI violations under standard conditions, we next examined its ability to capture multisensory behaviour under conditions of temporal asynchrony. Specifically, we simulated SOA conditions, in which one sensory signal precedes the other (Fig 4a). As a hallmark manipulation in multisensory research, SOA manipulations provide a stringent test of multisensory models by probing how response dynamics adapt when stimuli are no longer presented simultaneously [e.g., 38,61,62].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Temporal dynamics in multisensory processing.

a) Stimulus onset synchrony (SOA). Negative SOAs represent visual lags relative to auditory signal onset (left). An SOA of 0 ms represents synchrony between the auditory and visual onsets (centre). Positive SOAs represent auditory lags relative to visual onset (right). b) Modelling based on data from Miller [38] (n = 1). Circles show mean RTs with redundant signals (left) and the RSE (right) as a function of SOA. Dashed lines represent unisensory RTs, adjusted for SOA (e.g., the SOA is added to auditory RTs for positive SOAs). Red lines show predictions from Raab’s race model and grey lines the relay model, which fitted the empirical RTs best with an RT share of 13.5%. c) Identical analysis and modelling based on data from Otto et al. [37] (n = 10). The relay model fitted the empirical RTs best with a mean RT share of 10.7 ± 2.2% (model fitting was performed at the participant level). Data points show group averages; error bars represent bootstrapped 95% confidence intervals.

https://doi.org/10.1371/journal.pcbi.1013320.g004

We used the benchmark dataset from Miller [38], which systematically varied SOAs between auditory and visual signals from -167 ms (auditory lead) to +167 ms (visual lead) in 33 ms increments (Fig 4b). This dataset exhibits two characteristic features. First, because any deviation from synchrony introduces a delay, the fastest multisensory responses occur under synchronous conditions (SOA = 0 ms). Second, the largest RSE is observed when the SOA compensates for the difference in unisensory RTs, a phenomenon introduced earlier as the principle of congruent effectiveness [37]. In Miller’s data, visual RTs were ~100 ms slower than auditory RTs. Accordingly, the largest RSE was observed when the auditory signal lagged by ~100 ms. While the classical race model captures these key characteristics, it systematically underestimates RSE magnitudes across SOA conditions.

To accommodate temporal asynchrony in the relay model, we implemented a simple model extension. Since evidence accumulation cannot begin prior to stimulus onset, we incorporated a temporal delay into the processing of the lagging signal, equal to the experimentally defined SOA. Thus, at stage 1, processing for each modality begins with its signal onset. For example, if the auditory signal is delayed by , the stage 1 completion time becomes . Critically, we retained the relay model’s core feature of cross-modal initiation: stage 2 processing is initiated by the first modality to complete stage 1. If stage 2 is initiated before the delayed signal is physically present, processing of that modality is further delayed until its onset. In the model, we account for this by adding a corresponding residual delay at stage 2. This mechanism becomes increasingly important at larger SOAs, where the lagging modality may be partially or fully excluded from processing. When the SOA exceeds the combined duration of both stages, the model response effectively reduces to the unisensory condition of the leading signal.

We tested whether this model extension could fully reproduce the empirical data. First, we constrained the model by fitting its unisensory parameters (Table 1). With these parameters fixed, we optimised the model’s single free parameter, the RT share, by minimising the RMSE between predicted and observed RSEs across SOA levels. The best-fitting RT share was 13.5% (RMSE = 2.5 ms), corresponding to first-stage durations of 31.2 ms (SD = 7.6 ms) for auditory, and 47.0 ms (SD = 12.4 ms) for vision. Overall, the best-fitting model produced an excellent fit to all empirical data points.

As it is difficult to generalise model performance from a single participant, we applied the same analysis to a second dataset from Otto et al. [37], which employed a redundant signal paradigm with 10 participants (Fig 4c). Here, the SOA varied from -60 ms (auditory lead) to +60 ms (visual lead) in 30 ms increments. Consistent with the previous analysis, the fastest multisensory RTs occurred under the synchronous condition. Moreover, the largest RSE was observed when the SOA compensated for the difference in unisensory RTs. Here, visual RTs were on average ~30 ms slower than auditory RTs. Accordingly, the largest RSE occurred for an SOA of 30 ms. Again, the race model predicted these temporal dynamics but failed to capture the magnitude of the empirical RSE.

We employed the same model analysis as above, with all fitting performed using RT distributions at the participant level. As a first step, we constrained the model with the IG parameters estimated in unisensory conditions using Quantile Maximum Probability Estimation (QMPE; [63,64]; Table 1). We then fitted the relay model with its single free parameter, RT share, to the RT quantiles across the six multisensory SOA conditions. The mean estimated RT share was 10.7 ± 2.2%, with a mean Negative Log-Likelihood (NLL) of 368.1. A sign-flipping permutation test (10,000 iterations) confirmed that this RT share was significantly different from zero (p = 0.002). Crucially, with only this single degree of freedom at the multisensory level, the model predicted the observed RSE across all SOA conditions, always falling within the bootstrapped 95% confidence intervals.

Thus, the relay model successfully captured both qualitative and quantitative features of the datasets from Miller [38] and Otto et al. [37]. It predicted the fastest RTs at synchronous onset, identified the SOA that maximised the RSE, and closely matched the RSE magnitudes across all SOA conditions (Fig 4b,c). By incorporating a two-stage architecture that allows for cross-modal initiation, the relay model offers a mechanistically grounded account of how temporal asynchrony shapes multisensory decision-making, thereby seamlessly transitioning between unisensory and multisensory conditions.

Manipulation of signal strength

To further evaluate the relay model, we analysed data from a redundant signals paradigm with varying signal strength [37]. Such manipulations are established tools known to systematically modulate both the mean and the variability of unisensory RTs [65]. In general, stronger signals yield faster and more consistent RTs, whereas weaker signals produce slower and more variable responses. Critically, race models predict that signal strength manipulations have a similar effect on the RSE as SOA manipulations, because unisensory RTs need to be equally effective to maximise the RSE (principle of congruent effectiveness). In addition, such manipulations are also expected to affect the RSE because, according to race models, it is the variability of unisensory RTs that drives the effect (variability rule). Consequently, it is not surprising that signal strength manipulations also have strong empirical effects on the RSE [37,61,66], making them a stringent test of the model’s predictive power.

In the dataset by Otto et al. [37], signal strength was manipulated in both modalities (Fig 5a). We analysed three auditory levels (weak, medium, strong; A_W, A_M, A_S) and two visual levels (medium, strong; V_M, V_S). The dataset also included a third, weak visual signal, however, performance in this condition was not at ceiling and thus cannot be used in the modelling approach employed here, which assumes ceiling performance (see Methods). The signal strength manipulations produced substantial effects on unisensory RTs, with changes in the mean up to ~100 ms and in the variability up to a factor of two. Using a model fitting strategy at the participant level, we estimated the unisensory parameters using QMPE [63,64]. For each signal strength condition, we fitted the two unisensory parameters, yielding a total of ten parameters across five RT distributions (Table 1). As in the approaches above, these parameters were held fixed and used to constrain the modelling of the multisensory conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Signal strength manipulation.

a) Unisensory RT distributions. Signal strength systematically modulated unisensory RTs, with strong effects on both mean and variability. Empirical data are represented by quantiles (circles) and IG fits (dashed lines; for best-fitting parameter values, see Table 1). b) Multisensory RT distributions. Signal strength also modulated multisensory RTs. The relay model was fitted in the condition with both modalities at the medium level (dashed line). The best-fitting estimate of the RT share parameter was 12.9% ± 3.1%. Keeping this parameter value fixed, the relay model predicted the RT distributions for the remaining five combinations of signal strength (solid lines). c) RSE across signal strength combinations. A near-perfect correspondence was found between predicted and empirical RSEs. Data taken from [37].

https://doi.org/10.1371/journal.pcbi.1013320.g005

The unisensory conditions were combined using a 3 x 2 factorial design, yielding six multisensory conditions ranging from the weakest (A_WV_M) to the strongest combination (A_SV_S; Fig 5b). In a first step, we fitted the relay model with a single free parameter (RT share) to the RT quantiles of the multisensory condition with both modalities at the medium level (A_MV_M). The mean estimated RT share was 12.9% ± 3.1% (mean NLL = 93.4), which was significantly different from zero (permutation test, p < .001). In a second step, we generated model predictions for the remaining five multisensory conditions. Crucially, these predictions were derived with no additional free parameters, relying solely on the best-fitting RT share from the medium condition (A_MV_M) and the IG parameters in the underlying unisensory conditions (Table 1). The model predicted all empirical distributions with high fidelity.

To summarise this model analysis, we examined the modulation of the RSE for the different signal strength combinations (Fig 5c). In line with the strong modulation of unisensory RTs, the empirical RSE was significantly modulated by approximately a factor of two, ranging from 28 ± 3 ms (A_SV_S) to 55 ± 5 ms (A_WV_M; permutation test, p < .001). This modulation was accurately predicted by the relay model (as fitted above). A correlation analysis revealed a near-perfect correspondence between the empirical and predicted RSEs (r = .99, permutation test, p = .002; Fig 5c). This result demonstrates that a single, invariant relay mechanism is sufficient to explain the complex pattern of facilitation across varying levels of signal strength without requiring additional parameter adjustments, highlighting its strong explanatory power across a second key experimental manipulation.

Ethics Statement

The analysis carried out in this study involves model fitting of non-contentious anonymous secondary human data and did not involve any new participant data collection (see Data). Ethical approval for data reuse was given by the University Teaching and Research Ethics Committee (UTREC) at the University of St Andrews (0740 - PS-0740-852-2025).

Data

The data presented in Fig 1b, along with the modelling results in Figs 2 and 3, are based on Experiment 1 from Miller [4]. In this experiment, 74 participants completed a redundant signals task involving three stimulus conditions: a visual stimulus (an asterisk at the centre of the screen), an auditory stimulus (a 780-Hz tone), or both stimuli simultaneously. Participants were instructed to press a key as quickly as possible upon detecting any stimulus. Each participant completed two blocks of 40 trials, with each block containing 10 trials per stimulus condition as well as 10 catch trials (no stimulus). For analysis, RTs for each condition within each block were rank-ordered and then averaged across blocks and participants [4,90]. This yielded three cumulative group RT distributions (auditory, visual, and audiovisual), each based on 1480 trials. As raw data are not available from the original publication, we extracted group-level data using graphical digitisation (Figure 1 in [4]). The figure was digitised, and the pixel-to-millisecond scaling was calibrated using a known 100 ms interval. RT values were then read directly from the scaled x-axis coordinates of the plotted points.

The data underlying Fig 4b are taken from Miller’s follow-up study [38]. Two participants completed the same redundant signals task as described above. Crucially, this study introduced temporal asynchronies between the auditory and visual stimuli. Audiovisual trials were divided into 11 conditions: a synchronous condition, five with the visual stimulus leading (SOAs from −33 to −167 ms), and five with the auditory stimulus leading (SOAs from 33 to 167 ms), all in 33 ms increments. Each participant completed 40 blocks of trials. Each block contained 170 trials, including 10 trials per stimulus conditions (13 conditions: auditory, visual, and 11 audio-visual SOAs) and 40 catch trials (no stimulus). We used data only from participant B.D., whose responses showed significant RSEs in 7 of the 11 SOA conditions. By contrast, participant K.Y.’s data showed significant RSEs in only three conditions, a finding that Miller described as “somewhat atypical in view of the consistency with which RSEs are obtained” [38]. Summary statistics were reported in the original publication for each condition, which included the mean, standard error, and median of RTs, as well as the magnitude of the RSE (Table 1 in [38]). The analysed data comprised 400 trials per condition.

The data underlying Fig 4c are based on Experiment 1 from Otto et al. [37]. Ten participants performed a redundant signals task involving a visual stimulus (random dot motion), an auditory stimulus (a 294-Hz tone), or both stimuli. For redundant conditions, the experiment featured five SOA conditions ranging in increments of 30 ms from -60 ms to +60 ms. The experiment was conducted in four blocks per participant. Following the original protocol, we removed outliers and retained 40 valid trials per block and condition. To combine data across the four blocks, we used Vincent averaging [90]. While Otto et al. [37] originally analysed group-level data, we analysed the data at the participant level. This approach provides a stricter test of the relay model, as it requires the model to capture the RT distributions of individual participants rather than a smoothed aggregate. Overall, the dataset comprised 1600 trials per condition.

The data underlying Fig 5 are based on Experiment 2 from Otto et al. [37]. Twenty participants performed a similar redundant signals task as described above. Here, the experiment manipulated auditory and visual signal strength in a 3x3 design. Auditory signals were weak (A_W; 49.5 dB), medium (A_M; 50.5 dB), and strong (A_S; 53 dB). Visual signals varied in motion coherence at levels weak (V_W; 20%), medium (V_M; 30%), and strong (V_S; 50%). For redundant conditions, the experiment tested all nine multisensory combinations (e.g., A_WV_S or A_SV_M). Standard tests of the race model as well as our modelling approach require near-ceiling performance, implying that signals are detected on every trial. As the weak visual condition (V_W) exhibited a high miss rate (>20%), we did not include conditions involving this level in our analysis. Data collection occurred over two sessions on separate days. As the dataset did not permit linking participant data across days, we followed the original protocol and treated each block as an independent sample. For each block, we removed outliers and retained the last 50 valid trials in each condition. Overall, the dataset comprised 2000 trials per condition. Data underlying all analyses can be accessed from [91].

Unisensory RT distributions

Race models allow for parameter-free predictions of the RSE directly from the observed unisensory RT distributions. Basic predictions, such as those from Raab’s race model (probability summation, Eq 2) or the RMI (Eq 3), can be computed from empirical data without distributional assumptions (e.g., [92,93]). However, generating more complex predictions, such as those required by the relay model, requires assuming a particular RT distributional form.

For all simulations, we modelled unisensory RTs using the IG distribution (also known as Wald distribution; [94]). The IG distribution is well-suited for modelling non-negative, positively skewed data such as RTs [34,35]. Consequently, we assumed that unisensory RTs are IG-distributed with mean μ and shape parameter λ:

(5)

For the group-level RT distributions extracted from [4], we estimated the IG parameters for each unisensory condition separately: (μ_A,λ_A) for the auditory condition, and (μ_V, λ_V) for the visual condition. Parameters were optimised using MATLAB’s fmincon function, minimising the RMSE between the empirical and the simulated distributions at the 10 available quantiles.

For the SOA data extracted from [38], we estimated (μ_A,λ_A) and (μ_V, λ_V), using the method of moments approach [95]. Here, the fitting procedure minimised the RMSE between the observed and predicted descriptive statistics (mean and standard deviation). Although [38] reported standard errors, we converted these to standard deviations (using n = 400) to directly fit the standard deviation of the simulated distribution. The analytic mean was directly computed from μ and the standard deviation from .

For both the SOA and the signal strength datasets from Otto et al. [37], unisensory parameter estimation was performed at the individual participant level using Quantile Maximum Probability Estimation (QMPE; [63,64]). For each condition, we computed five quantiles from the raw RT distributions, defining six bins. Parameters were optimised by minimising the negative log-likelihood (NLL) of the observed bin counts using MATLAB’s fmincon function. For the SOA dataset [37], IG parameters (μ, λ) were fitted independently for auditory and visual conditions, resulting in four unisensory parameters per participant. For the signal strength dataset [37], IG parameters were fitted independently for each intensity level (five conditions: A_S, A_M, A_W, V_S, and V_M), resulting in ten unisensory parameters per participant.

Table 1 reports the best-fitting IG parameters for all unisensory conditions. These parameters were held fixed and used to constrain all subsequent modelling of multisensory conditions, including race model predictions and relay model simulations.

Stage-based partitioning of unisensory RTs

The relay model assumes that unisensory RTs are composed of two (or more) sequential processing stages (Fig 1c). Accordingly, the total RT can be expressed as

(6)

where is the index of stages.

To estimate the contribution of these (latent) stages, we implemented an approach that partitions the overall RT into stage-specific processing times. This approach relies on a key property of inverse Gaussian (IG) distributions: under specific conditions, the sum of IG-distributed random variables is itself IG-distributed. Specifically, if

(7)

describes the processing time at (latent) stages, then the total unisensory RT remains IG-distributed with mean μ and shape parameter λ (Eq 5). This property arises because the IG parameters are scaled with the weights , such that the convolution preserves the original distributional form. While typically used as a summation rule, we exploit this property in reverse: it allows us to decompose a single, empirically fitted IG distribution into (latent) constituent stages. As a result, the unisensory RT distributions, which are overall described by IG distributions (Table 1), can equivalently be expressed as the sum of two or more (latent) IG-distributed processing stages, with the mean and variance partitioned via the weights .

For model fitting and simulation purposes in the multisensory context, we imposed two constraints on the partitioning approach. First, we used normalised weights that sum to one:

(8)

Thus, weights can be interpreted as the proportion of the total RT contributed by a specific stage (in terms of both mean and variance), which we refer to as the RT share (Fig 2a). For example, indicates that stage 1 accounts for 20% of the total RT. Stage 2 would then account for the remaining 80%.

Second, for simplicity, we assumed that the same RT share configuration applies to both audition and vision. For example, if the RT share at the first auditory stage is 20%, we assumed the same RT share for the corresponding visual stage. This simplification facilitates model fitting and interpretation but can be relaxed in future experiments aimed at isolating modality-specific stage dynamics.

Relay model

The basic relay model consists of two sequential race units (Fig 1c). Using the partitioning approach established above, for stage , let the auditory and visual processing times be distributed as:

(9)

where represents the RT share of stage . Because this partitioning preserves the IG form, the corresponding probability density functions (PDFs) and cumulative distribution functions (CDFs) can be expressed analytically. We denote the PDFs as and , and the CDFs as and , respectively.

In the multisensory condition, stage is completed as soon as either modality finishes processing. The duration of stage is therefore given by:

(10)

The corresponding CDF, denoted , can be computed analogously to Raab’s race model (Eq 2), assuming statistical independence between auditory and visual processes within each stage. It can be analogously expressed as

(11)

where denotes the survival function. The corresponding PDF, denoted , is given by:

(12)

As the relay model assumes cross-modal initiation, the total multisensory is given by the sum of the durations of the sequential stages (Eq 4). For analytical tractability, we assume independence between stages, such that the durations of successive stages are treated as independent random variables. To compute the CDF of , we convolved the CDF of the first stage () with the PDF of the second stage ():

(13)

For simulations shown in Fig 3, we extended the relay model to more than two stages. The overall CDF for a multi-stage relay model was computed iteratively: starting with the CDF of the first stage, we successively convolved it with the PDFs of the additional stages. Each iteration follows the same computation as in Eq 13.

To provide a simple alternative to the relay model, we also fitted a shifted race model, a variant of the race model in which the predicted RT is shifted by a constant , acting as a single free parameter. The corresponding CDF is given by:

(14)

where denotes the probability of a multisensory response by time t given a shift δ, and is the survivor function of modality evaluated at the shifted time.

To account for SOA manipulations [37,38], we extended the two-stage relay model to incorporate onset lags. We defined the lag, τ, as the time difference between the onsets of the leading and the lagging signal. This lag is not a free model parameter, instead it is directly determined by the experimentally defined SOA condition. Adding τ can affect both relay model stages (). At stage 1, τ introduces a temporal disadvantage for the lagging modality. For example, if the visual signal is delayed by τ, the duration of stage 1 is given by:

(15)

As described above, the first stage initiates the second stage at time . Intuitively, if stage 2 begins before the delayed signal is presented (), that modality enters the race with a residual temporal disadvantage (equal to ). Continuing with the example of a delayed visual signal, the duration of stage 2 is therefore given by:

(16)

To describe the effect of temporal delays between signals, we divided the model predictions into two scenarios depending on the time at which stage 2 is initiated. In the first, early initiation scenario, stage 2 begins before the lagging signal has been presented (). In this case, stage 1 is driven entirely by the leading signal. For example, if the visual signal lags by , the contribution of the early initiation scenario to the overall model CDF is:

(17)

Here, denotes the probability density of the stage 1 processing time of the leading (auditory) modality. The stage 2 term follows Eq 11 but is adapted to reflect that visual processing can only begin after the lag has elapsed. In other words, stage 2 begins immediately for the leading modality (after time ), whereas the lagging modality can only start after its signal onset (at time ), resulting in a residual temporal disadvantage.

In the second, late initiation scenario, stage 2 begins after the lagging signal has been presented (). In this case, stage 1 is driven by a race between both modalities. Continuing with the example where the visual signal lags by , the contribution of the late initiation scenario to the overall model CDF is:

(18)

Here, denotes the probability density of the stage 1 completion time under a race between both modalities, which can be computed following Eq 12, adapted to account for the lag in the visual modality:

(19)

Regarding the stage 2 term, because both signals are present at time , processing starts immediately for both modalities, and Eq 11 applies directly. The total predicted CDF is then given by the sum of the two components:

(20)

RSE and RMI violations

To inspect the key features of the predictions derived from the relay model, we measured the RSE as well as violations of the RMI (Figs 2 and 3). To quantify these effects, we used geometric measures on the level of CDFs analogous to approaches described elsewhere [93,96].

Model fitting

In addition to model simulations, we fitted the relay model to empirical data from [4,37,38]. As outlined above, the relay model was constrained by the unisensory RT distributions estimated for each dataset (Table 1). To account for the observed multisensory RTs, the model included a single free parameter: the RT share of stage 1, constrained to the interval [0, 0.5].

To fit the relay model to the multisensory RT distributions from [4], we compared empirical quantiles to the corresponding quantiles predicted by the relay model CDF (Eq 13). The RMSE between the empirical and model CDF was minimised by adjusting the RT share parameter. Optimisation was conducted using MATLAB’s fminbnd function. All fitting procedures, including parameter recovery simulations, can be accessed from [91].

Similarly, we fitted the shifted race model to the same data (Eq 14). Here, the RMSE between the empirical and model CDF was minimised by adjusting the shift parameter δ. The parameter search was constrained to ensure physical plausibility. The upper bound was fixed at 500 ms, while the lower bound was dynamically set for each condition as to ensure that the shifted times remained strictly positive. Optimisation was performed using MATLAB’s fmincon function.

To fit the relay model to the multisensory RTs from [38], we used the relay model adapted for temporal asynchronies (Eq 20). Miller [38] defined the RSE as the difference in mean RT between the faster unisensory and the multisensory condition. To generate model predictions, we computed representative RT distributions for each condition and SOA by evaluating the inverse CDF (quantile function) from Equation 20 at 200 evenly spaced probability values. Using these predicted RTs, we calculated the RSE analogously to Miller [38]. We estimated the optimal RT share by minimising the RMSE between the empirical and predicted RSEs across the 11 SOA conditions. Optimisation was performed using MATLAB’s fminbnd function.

The RT share parameter was fit to the multisensory conditions in both the SOA and signal strength datasets from Otto et al. [37] using QMPE [63,64]. For the SOA dataset, we fitted the RT share parameter by minimising the joint NLL summed across all six SOA conditions (-60, -30, 0, 30, and 60 ms). This approach ensures that a single multisensory RT share accounts for the entire range of temporal asynchronies. For the signal strength dataset, we applied a stricter test of model generalisability. To test the model’s predictive validity, we fitted the RT share parameter only to the congruent medium-intensity condition (A_MV_M). This fitted RT share was then used to predict RTs for remaining five signal strength combinations (e.g., A_SV_S or A_WV_M) without further optimisation, effectively treating these conditions as out-of-sample tests.