Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Audio-motor but not visuo-motor temporal recalibration speeds up sensory processing

  • Yoshimori Sugano ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    sugano@ip.kyusan-u.ac.jp (YS); J.Vroomen@uvt.nl (JV)

    Affiliation Department of Industrial Management, Kyushu Sangyo University, Fukuoka, Japan

  • Mirjam Keetels,

    Roles Conceptualization, Investigation, Methodology, Resources, Validation, Writing – review & editing

    Affiliation Department of Cognitive Neuropsychology, Tilburg University, Tilburg, the Netherlands

  • Jean Vroomen

    Roles Conceptualization, Methodology, Resources, Validation, Writing – original draft, Writing – review & editing

    sugano@ip.kyusan-u.ac.jp (YS); J.Vroomen@uvt.nl (JV)

    Affiliation Department of Cognitive Neuropsychology, Tilburg University, Tilburg, the Netherlands

Abstract

Perception of synchrony between one's own action (a finger tap) and the sensory feedback thereof (a visual flash or an auditory pip) can be recalibrated after exposure to an artificially inserted delay between them (temporal recalibration effect: TRE). TRE might be mediated by a compensatory shift of motor timing (when did I tap?) and/or the sensory timing of the feedback (when did I hear/see the feedback?). To examine this, we asked participants to voluntarily tap their index finger at a constant pace while receiving visual or auditory feedback (a flash or pip) that was either synced or somewhat delayed relative to the tap. Following this exposure phase, they then performed a simple reaction time (RT) task to measure the sensory timing of the exposure stimulus, and a sensorimotor synchronization (SMS) task (tapping in synchrony with a flash or pip as pacing stimulus) to measure the point of subjective synchrony between the tap and pacing stimulus. The results showed that after exposure to delayed auditory feedback, participants tapped earlier (~21.5 ms) relative to auditory pacing stimuli (= temporal recalibration) and reacted faster (~5.6 ms) to auditory stimuli. For visual exposure and test stimuli, there were no such compensatory effects. These results indicate that adjustments of audio-motor synchrony can to some extent be explained by a change in the speed of auditory sensory processing. We discuss this in terms of an attentional modulation of sensory processing.

Introduction

Precise and flexible control of action is of crucial importance for human behavior in everyday life. Especially timing is critical in actions like catching a ball, dancing, driving in traffic, or playing a musical instrument. Smooth action requires a proper order of movements with correct timings that can be learned with practice. However, even after these skills have been learned, they should be modifiable in order to adapt to rapid changes in environmental conditions [1, 2, 3] as well as to gradual change like growth in body size [4] that give rise to changes in neural transmission time. From this point of view, it can be argued that sensorimotor learning is a continuous recalibration process of when and how motor commands should be issued [1, 5, 6].

In order to study these adaptive changes in timing, one can introduce an artificial delay between an action and the sensory consequence (the feedback) of that action [7]. A study by Stetson et al. [8] demonstrated that perception of synchrony between a voluntary action and the feedback thereof is shifted after exposure to delayed sensory feedback. This shift is usually referred to as sensorimotor temporal recalibration effect (TRE) because the shift is, presumably, induced to reduce the timing error between events that normally co-occur. It has been shown that the sensorimotor TRE occurs for one's own delayed vocalization [9, 10], or viewing one's own delayed hand [11]. Furthermore, it has also been reported that it occurs if a visual stimulus precedes rather than follows a voluntary action [12, 13]. Thus, sensorimotor TRE is ubiquitous and robust phenomenon (for review, [2, 14]).

Theoretically, sensorimotor TRE might be obtained via at least two different, though not mutually exclusive mechanisms: a change in motor timing ("when did I touch the surface?" or "when did I move my finger?"), or a change in the perceptual latency of the sensory feedback signal ("when did I hear the sound or see the flash"). Fig 1 illustrates how these shifts in perceived timing might induce a change in the perception of synchrony after exposure to delayed feedback. The mechanism underlying sensorimotor TRE is not clear. Previous researches suggest that sensorimotor TRE also occurs across sensory modalities (i.e., amodal) and might be controlled by rather central (cognitive) processes [15, 16, 17, 18, 19]. However, Sugano et al. [20] have shown that the magnitude of sensorimotor TRE is greater for audio-motor than visuo-motor pairings. And they also have shown that visuo-motor TRE transfers to audio-motor domain, but not vice versa. These results suggest that modality-specific mechanisms might be involved in the sensorimotor TRE.

thumbnail
Fig 1. Predictions about changes in mean asynchrony and reaction time (RT).

The model is adapted from Aschersleben and Prinz [21] with some modification. Each horizontal line represents a timeline for sensory timing, brain timing, and motor timing. Thin lines with an arrow that connects the three timelines represents a processing course between them. Distances among three timelines reflect processing latency. (A) Mean asynchrony (upper panel) and RT (lower panel) before exposure. (B) Exposure to delayed feedback during voluntary tapping. (C, D) Mean asynchrony and RT after exposure to delayed feedback: The perceived asynchrony might be reduced by a shift in the motor component (C) or the sensory component (D). (E) Predictions about mean asynchrony and mean RT after exposure to synchronous or delayed feedback. Values in the figure are examples of real values to illustrate the direction of a shift.

https://doi.org/10.1371/journal.pone.0189242.g001

TRE has also been demonstrated between pure sensory modalities like sound, vision and touch. These sensory adaptations to asynchronies have been found with artificial stimuli like flashes and beeps [22, 23, 24, 25, 26], and more natural stimuli like audio-visual speech [27, 28, 29]. Like sensorimotor TRE, both single amodal mechanism [30] and modality-specific mechanisms [31] are advocated so far.

Support for the existence of modality-specific mechanism comes from event-related potential (ERP) studies which have shown that the amplitude of early components in the ERP are modulated after exposure to visuo-motor delay [32] or audio-motor delay [33]. These results suggest that a modality-specific sensory shift might play a role during sensorimotor TRE. Furthermore, for audio-visual TRE, there is also evidence that perceptual latency of visual and/or auditory stimuli might change after exposure of audio-visual temporal asynchrony [31, 34, 35]. For example, after exposure to audiovisual asynchronies, Harrar and Harris [31] found a change in visual RT whereas Navarra et al. [34] reported a change in auditory RT. Di Luca et al. [35] reconciled this discrepancy by showing that visual RT changed if the audio-visual stimuli were presented from the same spatial position, whereas the auditory RT changed if the stimuli were not co-located. Note though that others reported that adaptation to audio-visual asynchrony does not depend on the spatial discrepancy of the audio-visual exposure stimuli [36], or on whether sounds were heard via speakers or headphones [22].

To date, there is no research that directly addresses whether perceptual latency changes following sensorimotor TRE. This was the motivation of the present research. In previous studies, it has been reported that tapping in synchrony with an external pacing signal (sensorimotor synchronization: SMS) provides a viable measure of sensorimotor TRE [17, 20, 37]. The common finding is that exposure to a fixed delay between a voluntary action (a finger tap) and an external sensory feedback signal (a flash or a tone) increases the natural anticipation tendency in a subsequent SMS task. Observers exposed to a delayed rather than synced feedback thus tap earlier, presumably to compensate the lingering effect of adaptation to the delay. Fig 1 shows a schematic illustration of how the timing of taps in the SMS task is modulated after delayed feedback. The model assumes that the movement execution time is not changed, as in principle it cannot compensate the delay. That is, even if a participant delays his/her movement execution to catch up the delayed feedback, it only results in delaying the feedback further. Rather, it is more likely that latency for tap detection or perceptual latency for stimulus is changed as it is effective for compensation (Fig 1C and 1D). Evidence from ERP studies support this assumption, because early modulations of visual and auditory evoked potential after exposure to delayed feedback have been demonstrated [32, 33].

The mean asynchrony is thought to correspond with the point of subjective simultaneity (PSS) between a tap and the pacing stimulus [20, 21, 38, 39, 40] (but see [41] for a different perspective of mean asynchrony. see [42] for review). However, with this task, one cannot tease apart a shift in the motor component (Fig 1C, upper) from a shift in the sensory component of the pacing stimulus (Fig 1D, upper). Therefore, we introduced a simple RT task along with the SMS task to separate the motor component from the sensory component. The rationale for using a simple RT task is as follows. Generally, a change in RT is thought to reflect a change in perceptual latency, but not a change in movement execution time [43, 44, 45, 46, 47, 48]. Specifically, as we assume that the movement execution time is not affected by the feedback delay, any change in RT should reflect a change in perceptual latency (Fig 1C and 1D). Moreover, earlier studies have shown that RT can be used to measure a change in perceptual latency after cross-sensory temporal recalibration [31, 34, 35]. It has also been suggested that the RT task and the temporal order judgement task are based on the same internal mechanism [49], thus giving further rationale of the use of RTs to measure a change in perceptual latency. We used the term "perceptual" and "sensory" as interchangeable here, because a simple RT task contains only two processes: the stimulus detection and the response execution (e.g., [50]). The stimulus detection is a low-level process that can be regarded as a sensory process.

It is of note that different interpretations of the mechanism underlying the asynchrony exist. Wohlschläger and Koch [41] suggests that the asynchrony is a result of a perceptual underestimation of the empty inter-onset intervals (IOIs). Also, others have suggested that periodic finger tapping is driven by an oscillatory motor activity that is coupled to a sensory driving oscillator (pacing stimuli) (e.g., [51]), in which the asynchrony can be a consequence of a phase lag caused by a detuning of coupled oscillators [52] (for review [42]). If it is the case that an exposure to delayed feedback further induces the underestimation of IOIs and/or further detunes the oscillatory motor activity from the sensory driving oscillator, it may give rise to a larger asynchrony than exposure to synchronous feedback. In fact, a slowing-down of the tapping tempo after exposure to delayed feedback is often observed in self-paced tapping (e.g., [53, 54]). As an aftereffect, it may lead to an acceleration of the tapping tempo during a subsequent SMS task and may give rise to an exaggeration of the asynchrony. Therefore, if one uses an SMS task to measure perception of synchrony, one should check if the inter-tap interval is not different across conditions during the SMS task.

One might suspect that the use of two different tasks to measure the same timing property, i.e., perceptual latency, is problematic because the SMS task relies on anticipation while the RT task relies on reaction. However, the SMS task contains a feedback-based error correction process of the asynchrony between tap and pacing stimulus (e.g., [42]), and so it involves a reaction to the current pacing stimulus. In the same vein, an RT task involves anticipation of the upcoming stimulus as is well-known from the foreperiod effect (e.g., [55]). Thus, in essence it appears that both tasks involve anticipation and reaction of upcoming stimuli.

One might also argue that the perceptual latency measured via an SMS task is beat-based, whereas latency measured via an RT task is non-beat-based. In fact, it has been shown that beat-based timing and non-beat-based timing involve different neural circuits (e.g. [56, 57, 58]). At present, we cannot say for certain if both tasks truly reflect the same timing property. However, we are optimistic regarding this for the following reasons. Firstly, the neural circuit for beat-based and the non-beat-based timing mechanism work in parallel via neural interconnections of the thalamus, pre-SMA/SMA, and the cerebral cortex [59]. Secondly, electrophysiological evidence has demonstrated that a visuo-motor TRE is accompanied by a change in early components of visual evoked potentials [32], and an early modulation of auditory evoked potentials accompanies a speed-up of auditory RT [60]. These findings might reinforce the validity of our methodology.

We had two motives in the present study. One was to test whether sensorimotor TRE contains a sensory component, and the second one was to examine whether this was different for the auditory than the visual modality. Regarding the first motive, we expected the mean asynchrony during the SMS task to increase (i.e., greater anticipation error) after exposure to delayed feedback. The extent to which this shift can be attributed to a shortening of the sensory component should be observable in the RT task. Here, we thus expected to observe faster RTs after delayed feedback that would diminish the artificial delay. Fig 1 shows a schematic illustration of how RTs might be modulated. If only the motor component shifts, then RTs do not change (Fig 1C, lower). However, if there is a shift in the sensory component (e.g., lowering the detection threshold), then RTs should become faster (Fig 1D, lower).

Regarding our second motive, consistent with our previous research [20, 37], we expected that the TRE would be greater following delayed auditory feedback than delayed visual feedback because sounds are also usually more potent to induce sensorimotor TRE. In general, audition is more precise than vision in temporal processing [3, 61, 62, 63, 64, 65]. Moreover, a stronger sensorimotor coupling has been suggested in audition than in vision [42, 66, 67]. This may lead to finer adjustment of sensory and/or motor timing in front of the delay than in vision, as the delay might be attributed to the timing error of the audio-motor system itself that should be adjusted [37].

Method

Participants

Seventeen students from Kyushu Sangyo University and one of the authors (Y.S.) participated in the experiment from November 2015 to January 2016 (one female, mean age 22.9 years ranged from 18 to 43 years, all were right-handed). All participants had normal hearing and normal or corrected-to-normal vision. Written informed consent was obtained from each participant. The experiment was approved by the Local Ethics Committee of Kyushu Sangyo University, and followed the declaration of Helsinki.

Stimuli and apparatus

Participants sat at a desk in a dimly lit booth looking at a 17-inch CRT monitor running with 100-Hz refresh-rate at approximately 60 cm viewing distance. The visual stimulus was a 1 cm white square (30 ms duration, 9 cd/m2) with a black background (0 cd/m2) on the CRT monitor. The auditory stimulus was a 2000 Hz pure tone pip (30 ms duration with 2 ms rise/fall slope) presented via headphones (Sony MDR-CD900ST) at 79 dB(A). A 1-cm red square (30 ms duration, 3 cd/m2) and a 2250-Hz pure tone pip (30 ms duration with 2 ms rise/fall slope at 80 dB(A)) were used for catch trials (see Design and procedure). White noise was continuously presented via headphones at 60 dB(A) to mask the faint sound of mouse-presses. A special gaming mouse (Logitech G300) was used to obtain high temporal resolution (2 ms polling interval). Stimulus presentation and response detection were controlled by E-prime software running on a general PC/AT personal computer (Dell Precision T3400). The timing of stimulus presentation and response detection was verified by a multiple-trace oscilloscope.

Design and procedure

Two within-subject factors were used: Exposure modality (visual vs. auditory) and exposure delay (50 ms vs. 150 ms). Exposure modality determined whether the feedback after a tap was auditory or visual, and exposure delay determined whether the sensory feedback was delayed (150 ms) or subjectively synchronous (50 ms). The trials for each of these 4 possible combinations were presented in blocks and these blocks were presented in counter-balanced order across participants.

Fig 2 shows a schematic illustration of the experimental procedure for a block of trials. Each block consisted of a long exposure phase (~1 min) followed by fifteen test trials that were each preceded by a short top-up exposure (~9 min in total). Each of the 4 possible blocks was presented once over 2 days. The exposure modality was the same during a day, and the exposure delay varied. There was a 5-min rest between blocks. The 2 days were separated by at least 24 hours (24 days maximum).

thumbnail
Fig 2. Experimental procedure for one block.

Participants first experienced a long exposure phase (~1 min) in which they made 80 voluntary taps while receiving auditory or visual feedback (a pip or a flash) at delay of either 50 ms or 150 ms. Immediately after that, they received 15 test trials that each consisted of a short top-up exposure phase (7 voluntary taps with the same feedback as in the long exposure phase), a reaction time (RT) task in which they reacted as fast as possible to 3 pips or flashes, and a sensorimotor synchronization (SMS) task in which they tried to tap in synchrony with the last 7-out-of-9 pips or flashes. During the long exposure and the short top-up, participants counted the number of deviants (i.e., a red flash or a high-pitched tone) to ensure that they properly attend to the stimuli. At the end of the long exposure or the top-up, they reported the number of deviants.

https://doi.org/10.1371/journal.pone.0189242.g002

During the long exposure phase, participants made 80 voluntary mouse-presses without pacing stimuli with their index finger of the dominant hand, trying to maintain an inter-tap interval (ITI) of ~750 ms. Participants were instructed to maintain this tempo during the whole phase. Each mouse-press was followed by a sensory feedback (a flash on the CRT display or a tone via headphones, depending on the condition), delivered at either 50 ms (subjectively synchronous) or 150 ms (delayed). These values were chosen because they yielded consistent TRE in previous reports [17, 20, 37], and because the TRE of 150 ms is still in the range were subjects adapt to the delay (i.e., it becomes unnoticeable after a certain number of exposures). In order to ensure that participants pay attention to the modality in which feedback was delivered, they counted the number of deviants (i.e., a red flash or a high-pitched tone) during exposure. At the end of each exposure phase, they reported the number of deviants via a computer keyboard.

Following the long exposure phase, testing started. Each test trial consisted a short ‘top-up’ re-exposure phase (7 voluntary taps with the same feedback as in the long exposure phase), immediately followed by a simple RT task, and then followed by a SMS task. The order of the two tasks was chosen to minimize a possible carryover effect of the well-known slow-down in tapping tempo with delayed feedback (e.g., [53, 54]). During the RT task, participants were to press the mouse as quick as possible when they saw a flash or heard a click. Each stimulus was delivered three times at random ISIs between 1500 to 3500 ms. If the RT was shorter than 100 ms or longer than 700 ms, it was regarded as a mistake, and repeated to obtain at least one response.

Immediately after the RT task, participants then did the SMS task. During the SMS task, they tried to press a mouse in synchrony with an external pacing stimulus (the same flash or the tone as presented during the exposure phase). The pacing stimuli were delivered 9 times at a constant inter-stimulus interval (ISI) of 750 ms. Participants skipped the first two pacing signals to get into the rhythm, and then tried to sync their mouse-presses with the following 7 pacing stimuli. If an asynchrony (i.e., time difference between a pacing stimulus and a tap) was greater than +375 ms (the tap lagged the pacing stimulus by 375 ms) or smaller than -375 ms (the tap preceded the pacing stimulus by 375 ms), it was regarded as a mistake. If there were more than two mistakes in a trial, then the SMS task was repeated.

Participants had short practice session before the experiment to get accustomed to the experimental tasks. They did several test trials with feedback about their performance (i.e., tapping rate during the top-up exposure, mean asynchrony during the SMS task, and reaction time in the RT task), in which points were given depending on their performance. If they earned enough points, the practice session was finished. Then, they did extra three test-trials without performance feedback to be familiar with the main session. Testing lasted about 1.5 hour over two days.

Results

Mean RT and asynchrony

Trials from the practice session were excluded from further analysis. The asynchrony during the SMS task was defined as the difference in onset time between a tap and a pacing stimulus. A negative asynchrony means that the tap precedes the pacing stimulus (the well-known anticipation error, for review [42]). Missing responses and abnormal asynchronies (smaller than -300 ms or larger than +150 ms) were eliminated from analysis (0.7%). RT was measured from stimulus onset. Missing responses and abnormal RTs (shorter than 100 ms or longer than 450 ms) were also eliminated from analysis (3.0%). In addition to this screening, asynchronies and RTs that were outside 1.5 times the interquartile range below or above the appropriate quartile were also treated as outliers on a per-participant basis, and they were also eliminated from the analysis (2.4% for asynchrony and 5.8% for RT).

TAs were averaged over trials for each experimental condition and each participant. The RT distribution was positively skewed, and median instead of mean RT was calculated. Table 1 shows the group-averaged mean asynchrony and RT for each experimental condition. The individual data were entered into a repeated-measures ANOVA with exposure modality (MV vs. MA) and exposure delay (sync vs. delay) as within-subjects factors.

thumbnail
Table 1. Mean asynchrony and simple RT in milliseconds for each experimental condition.

https://doi.org/10.1371/journal.pone.0189242.t001

As the distribution of median-RTs was significantly different from normal (Shapiro-Wilk normality test, W = 0.96, p < 0.05), a reciprocal transformation was applied. After this transformation, the distribution was not significantly different from normal (W = 0.98, p = 0.232). These transformed RTs were used in all statistical tests. Although it should be better to report back-transformed values of the mean of transformed RTs in Table 1 (see [68], p.338), we reported simple mean of RTs instead, because it is convenient. Also, the two values were highly correlated (r ~ = 1.0, p < 0.001) and thus provided the same conclusion.

The ANOVA on the asynchronies revealed significant effects of exposure modality, F(1, 17) = 26.0, p < 0.001, exposure delay, F(1, 17) = 11.9, p < 0.01, and an interaction of the two, F(1, 17) = 17.2, p < 0.001. The first one indicates that the mean asynchrony was more negative in MA (-98.3 ms) than MV (-55.9 ms), which is the usual pattern in this task (e.g., [17, 20, 37, 42]). In order to examine the interaction, the asynchronies were analyzed separately for each exposure modality and were entered into a repeated-measures ANOVA with exposure delay (sync vs. delay) as a within-subjects factor. These ANOVAs revealed that the effect of exposure delay was significant for the MA condition, F(1, 17) = 46.8, p < 0.001 (the mean asynchrony was more negative (~21.5 ms) after exposure to delayed auditory feedback than synced one: -109.0 ms vs. -87.5 ms), but not for the MV condition, F(1, 17) < 0.001, p = 0.989 (the mean asynchrony was not significantly different (~0.1 ms) after exposure to delayed visual feedback from synced one: -55.8 ms vs. -55.9 ms).

As we mentioned in the introduction, there is a possibility that the observed TRE may be exaggerated under the delayed feedback condition by an acceleration of tapping tempo during the SMS task. If this is indeed the case, the mean asynchrony would progressively increase with taps. However, we did not find such a tendency by running the following additional analysis. Repeated-measures ANOVAs per modality on the mean asynchronies with exposure delay and taps as within-subjects factors showed that exposure delay x tap interaction was not significant in both MV, F(6, 96) = 0.49, p = 0.814, and MA, F(6, 96) = 1.23, p = 0.296, meaning the size of the TRE did not change across taps. Thus, we believe that the exaggeration of TRE might be negligible, if any.

The ANOVA on the RTs revealed a significant effect of exposure modality, F(1, 17) = 18.0, p < 0.001, and it interacted with exposure delay, F(1, 17) = 4.8, p < 0.05. The mean RT was significantly shorter (~14.6 ms) for auditory (176.3 ms) than visual stimuli (191.0 ms), which is in line with earlier findings (e.g., [69, 70, 71]). In order to examine the interaction, RTs were separated by exposure modality, and entered into a repeated-measures ANOVA with exposure delay (sync vs. delay) as a within-subjects factor. These ANOVAs revealed that the effect of exposure delay on the RT was significant for the MA condition (the mean RT was ~5.6 ms faster after exposure to delayed than synced auditory feedback: -179.1 ms vs. 173.6 ms), F(1, 17) = 5.6, p < 0.05, but not for the MV condition (no significant difference (~0.9 ms) in the mean RT after delayed versus synced visual feedback: 191.4 ms vs. 190.5 ms), F(1, 17) = 0.7, p = 0.414.

To further analyze RTs, we fitted an ex-Gaussian distribution on the response time distribution [55, 72, 73, 74, 75, 76, 77]. The ex-Gaussian is made of a convolution of a Gaussian with an exponential function that is characterized by three parameters: mu, sigma, and tau, which are thought to reflect different perceptual and/or cognitive processes [55, 76, 77]. For each participant, we thus obtained estimates of mu, sigma, and tau. These estimates were then entered into repeated-measures ANOVAs per exposure modality with exposure delay (sync vs. delay) as a within-subjects factor. As the distribution of mu and tau were significantly different from normal (Shapiro-Wilk normality test; W = 0.96, p < 0.05; W = 0.92, p < 0.001; respectively), the mus were applied by a reciprocal transformation and the taus were applied by square-root transformation before ANOVA. The analyses showed a main effect of exposure-delay on the tau of MA-RT (14.7 ms for delayed feedback vs. 20.2 ms for synced feedback), F(1,17) = 14.0, p < 0.01.

Fig 3 shows the group-averaged histograms and the group-averaged ex-Gaussian functions to fit the reaction times for each participant and each condition. As is clearly visible, only for MA-RTs (Fig 3B) tau became smaller after exposure to delayed rather than synced feedback (14.7 vs. 20.2 ms, respectively), while mu (163.1 vs. 164.5 ms) and sigma (20.0 vs. 20.8 ms) were unchanged. As the tau parameter mainly affects the tail of the distribution, smaller tau means that slower RTs were sped-up. Exposure to delayed auditory feedback thus induced larger anticipation responses in tapping and faster RTs, whereas exposure to delayed visual feedback did not change these measures.

thumbnail
Fig 3. Group-averaged histograms and group-averaged ex-Gaussian functions that are fitted to reaction times for each participant and each experimental condition.

(A) Visual RT, and (B) Auditory RT. Estimated parameters of a fitting curve are also shown.

https://doi.org/10.1371/journal.pone.0189242.g003

Variability of RT and asynchrony

It has been suggested that the sensitivity to temporal asynchrony between different sensory modalities (i.e., sound and light) is lowered in the initial stages of the build-up of TRE [24, 78, 79]. The temporal window of crossmodal integration is thus first widened to accommodate a discrepancy of timing (see also [80]). To examine this, we analyzed the variability in SMS and RTs after exposure to delayed feedback. We assumed that tapping variability is inversely correlated with the sensitivity of timing (the less sensitive, the more variable tapping is) [42, 81, 82].

We calculated a mean within-trial standard deviation (WSD) of asynchrony as a measure for variability for each participant, which is defined as an averaged within-trial standard deviation of asynchrony over trials. In the following all statistical tests, they were given logarithmic transformation to let them normally distributed. For RTs, as there was only three RTs within one trial, we calculated a quartile deviation (QD) over all RTs. The group-averaged asynchrony-WSDs and RT-QDs are shown in Table 2.

thumbnail
Table 2. Mean within-trial standard deviation (WSD) of asynchrony and mean quartile deviation (QD) of simple reaction time (RT) in milliseconds for each experimental condition.

https://doi.org/10.1371/journal.pone.0189242.t002

Mean asynchrony-WSDs and RT-QDs were entered into a repeated-measures ANOVA with exposure modality (MV vs. MA) and exposure delay (sync vs. delay) as within-subjects factors. The ANOVA on the asynchrony-WSDs revealed that the main effect of modality was significant, F(1, 17) = 38.3, p < 0.001, as well as the delay x modality interaction, F(1, 17) = 9.9, p < 0.01. The main effect of exposure modality indicated that asynchrony-WSD was significantly larger in MV (38.2 ms) than MA (29.9 ms). Subsequent repeated-measures ANOVAs per modality with exposure delay as within-subjects factor revealed that the effect of exposure delay was not significant in both MA, F(1, 17) = 2.4, p = 0.139, and MV, F(1, 17) = 2.2, p = 0.160. The ANOVA on the RT-QDs, on the other hand, revealed that the main effect of modality was significant, F(1, 17) = 9.0, p < 0.01, indicating that it was significantly larger in MA (17.7 ms) than MV (13.9 ms).

Discussion

Our first motive was to examine whether adjustments in the perception of motor-sensory synchrony involve shifts in sensory timing. The TRE may contain both a shift of sensory timing and a shift of motor timing (Fig 1). However, to date, there is no research that directly addresses to what extent the sensory shift contributes to the TRE. We used a simple RT task to estimate the perceptual latency, and used a SMS task to estimate the size of TRE.

Our second motive was to examine if the TRE was different for auditory versus visual feedback. Temporal processing is usually more precise in audition than vision, and there might be stronger audio-motor than visuo-motor connections, and so we predicted a stronger MA-TRE than MV-TRE.

In order to examine this, participants tapped their index finger at a constant pace while receiving visual or auditory feedback (a flash or pip) that was either synced with the tap or somewhat delayed. Following an exposure phase, they then performed a simple RT task to measure the sensory timing of the exposure stimulus, and a SMS task to measure the point of subjective synchrony between finger taps and the pacing stimuli. The results showed that after exposure to delayed auditory feedback, participants reacted faster (~5.6 ms) to auditory stimuli and tapped earlier (~21.5 ms) relative to auditory pacing stimuli (= MA-TRE). For visual exposure and test stimuli, there was no difference after exposure to synced versus delayed feedback in RTs and tapping. Taken together, these results thus show, for the first time, that recalibration of audio-motor synchrony can partly be explained by a change in the speed of auditory sensory processing. This result fits previous reports on audio-visual temporal recalibration that also reported that exposure to audio-visual asynchrony is accompanied by shifts in auditory latency [34, 35].

Our results are also consistent with previous finding about auditory dominance over vision in sensorimotor TRE. In general, sounds appear to be more potent than visual stimuli to induce (or erase) temporal recalibration. For example, MV-TRE following exposure to delayed visual feedback is erased when it is mixed with synchronous auditory feedback [37]. Our findings can also explain our earlier finding that MV-TRE transfers to the MA domain, but that MA-TRE does not transfer to the MV domain. No transfer of the MA-TRE to MV domain would imply that the MA-TRE is mediated by a shift of sensory component (change in perceptual latency), whereas the MV-TRE might be mediated by a more amodal mechanism or a shift of movement-related timing [20].

Speed-up of auditory RT

An important question is how a speeding-up of perceptual latency can accompany sensorimotor TRE. For cross-sensory TRE, several mechanisms have been proposed like: (1) adjustment of a criterion for synchrony, (2) adjustment of a sensory threshold, (3) a widening of the temporal window for cross-sensory integration [2, 14], and (4) adaptation of neuronal population to specific temporal intervals [83, 84]. In principle, all these models can be applied to sensorimotor TRE as well, but among these models, only the threshold adjustment model makes clear predictions about a change in perceptual latency (see also [35]). If so, then how can a sensory threshold be adjusted? In the next sections, we discuss several possible mechanisms by which a sensory threshold can be adjusted.

One potential mechanism to cause an adjustment of sensory threshold could be an attentional shift in time [85, 86, 87, 88]. If a participant notices a delay between an action and a sensory feedback thereof, attention may be reallocated to the delayed feedback as it indicates that something unnatural occurs. Indeed, it has been shown that if a sound is presented at irregular timings within a regular sound sequence, it automatically captures one's attention [89]. Moreover, there is a robust evidence showing that our sensorimotor system can detect subliminal (i.e., cannot be noticed) timing perturbation of regular sequence of sound during SMS [90]. In addition, attention to a sensory stimulus can enhance the processing time of that stimulus (the law of prior entry [91, 92]; see [2] for review). Both behavioural (e.g., temporal order judgement (TOJ) task) and electrophysiological studies (e.g., event-related potential response: ERP) have shown that an attended sensory stimulus is indeed processed faster [92, 93, 94, 95], and delayed sensory feedback might thus capture attention, thereby speeding-up the sensory processing of that stimulus [35]. This hypothesis also fits with electrophysiological studies that have shown that delayed auditory feedback after one's own action elicits enhanced neural responses (P2 and N300), that is correlated with one's awareness of that delay [96]. These authors have suggested that the enhanced P2 is related to attention to the delayed auditory stimuli.

The analysis by ex-Gaussian fitting to RT distribution gives additional support of the attentional shift explanation. The tau of MA-RT was significantly smaller for delayed feedback (14.7 ms) than for synced feedback (20.2 ms). As the tau parameter is thought to reflect more central, attention demanding (analytic) processes [77, 97, 98] (but see [76]), this also favors the attentional shift explanations. Harrar et al. [77] reported a similar result in which both tau and sigma of unisensory RT distribution were changed after exposure to audio-visual asynchrony.

A weak point of this attentional explanation is that if attentional resources were allocated more to delayed than synced feedback, delayed feedback would also have a detrimental effect on the movement control, because attention for temporal features are thought to be shared with the execution of a movement [99]. This might result in more variable (i.e., less controlled) tapping in the delayed feedback condition than in the synced condition ([80] for similar discussion). Earlier study about sensorimotor TRE indeed showed this pattern [37], but not in the current study. In order to examine this further, future studies might instruct participants to attend to their own movement (i.e., tap) or to the feedback (i.e., flash or tone) and then measure whether a speed-up of RT is obtained irrespective of attentional direction.

It is of interest to note that shifts in perceived timing have also been reported in the literature on intentional binding [80]. Intentional binding refers to the phenomenon that the timing of a voluntary action and the sensory feedback thereof attract each other in time. In the study of Haggard [80], participants were asked to press a key at a time of their own choice, which triggered a beep after 250 ms. Participants were asked to judge the perceived time of their keypress or of the beep relative to a rotating clock. Result showed that the perceived time of the action shifted 15 ms (6% shift) later in time (towards the beep), and the perceived time of the beep shifted 46 ms (18.4% shift) earlier in time (towards the action). Although the relationship between intentional binding and sensorimotor TRE still remains elusive [12], a lot of similarities have been reported so far. For example, both decrease if the feedback comes several hundreds of milliseconds after the action (e.g., ~400 ms [8, 15, 80], but see [100, 101] who reported that intentional binding still occurs as long as 4 sec). Both are tightly connected to perception of causality between action and feedback [13, 102] and to a sense of agency [13, 103]. Both occur across various sensory modalities [15, 16, 104]. Both occur even if the sense of body ownership is weak [11, 105, 106]. Finally, both occur even if the feedback signal comes before the action [12, 107] (but see [13] who have shown that if TRE was measured by interval estimation, it was limited to the movement-lead side). Considering these similarities, intentional binding and TRE might reflect the same perceptual and/or cognitive process: temporal grouping. Intentional binding reflects the immediate perceptual bias in time, and sensorimotor TRE is the aftereffect of it [8, 108]. However, some insisted the opposite relationship: the sensorimotor TRE is underlying mechanism that causes intentional binding [109]. More research is needed to elucidate the relationship between intentional binding and sensorimotor TRE in future.

Auditory dominance in sensorimotor TRE

Why is it that auditory, but not visual delayed feedback induced TRE and sped-up RTs? (see also [15, 20, 37, 110]). One factor might be that auditory stimuli have more power to capture one's attention than visual stimuli [111, 112, 113, 114, 115], and possibly attention is also of importance in TRE [116, 117, 118]. For example, Heron et al. [116] argued that attention to the feedback delay enhances audio-visual TRE. Ikumi and Soto-Faraco [117] has shown that if participants were exposed to staggered audio-visual sequences, the direction of TRE depended on whether participants attended to AV grouping or VA grouping. Tsujita and Ichikawa [118] reported that awareness to a visual delay is even necessary for visuomotor TRE to occur.

Furthermore, temporal resolution is better for the auditory modality than the visual one [37, 61, 62, 63]. Such a superiority of temporal processing of audition over vision could lead to a lower delay-detection threshold in audition than vision [16, 119], resulting in auditory dominance over vision in sensorimotor TRE, which might imply that sensorimotor TRE can be affected by a top-down factor such as an attentional shift.

It might also be that there is a stronger coupling of perception and action in the MA domain than in the MV domain. It has been suggested that the auditory system is more strongly coupled to the motor system than the visual system [120, 121, 122]. Strong audio-motor coupling could then produce stronger "intentional binding" [80] for MA than MV. However, this was not the case in data reported by Engbert et al. [104], who found that the amount of compression between action and feedback by intentional binding was equivalent across audio-motor, visuo-motor and motor-tactile pairs. However, they reported a modality effect where the estimated interval was shorter for audio-motor than visuo-motor or motor-tactile interval. They interpreted this modality effect as "the interval prior to auditory stimulus is… retrospectively compressed in subjective time" (p.700). It might reflect a strong coupling between movement and auditory processing [42, 66], which is based on strong neural connectivity between motor cortex and auditory cortex [67].

Unlike earlier studies [20, 37, 40], we failed to obtain a reliable MV-TRE. One possibility for the absence of MV-TRE is that performing the MA condition erased the subsequent MV-TRE, even though they were separated by at least 24 hours. Indeed, Sugano et al. [37] have shown that preceding MA execution erased the subsequent MV-TRE if MA and MV were alternated within one experimental session. To check this possibility, we did an additional mixed-model ANOVA on MV-asynchrony with the order of modality (MA first vs. MV first) as a between-subjects factor and the exposure delay as a within-subjects factor. This ANOVA showed no significant effect of modality order, F(1, 16) = 1.9, p = 0.190, nor an interaction of two factors, F(1, 16) = 0.59, p = 0.455, indicating that the execution order of modality did not affect the size of MV-TRE (i.e., difference of MV-asynchrony between delay and sync). The absence of the MV-TRE thus cannot be explained by erasure via the preceding MA condition. A more likely possibility might be that the MV-TRE was so fragile that it did not survive the intervening RT task (which preceded the SMS task). This fits data from a pilot study of ours [123] in which we did obtain a significant MV-TRE (16.3% shift of mean asynchrony between sync and delay) under the procedure that the order of SMS and RT task was reversed (i.e., the SMS task preceded the RT task). In the pilot study, MA-TRE was also bigger (33.4% shift) than the present study, and the auditory RT was significantly shorter by 6.5 ms after exposure to motor-auditory delays.

Cue reliability and predictability

The reliability of timing information in the motor, visual, and auditory domains should affect TRE [34, 35]. In general, the modality that conveys more reliable information dominates the other modality [124, 125, 126] (for review, [14, 127]). It is also well-known that auditory perception, in general, has a finer temporal resolution than visual perception [3, 61, 62, 63, 64, 65], and this was also found here in that the tapping variability in MA was smaller than in MV (~29.9 ms in MA versus ~38.2 ms in MV).

With this in mind, one might expect that visual timing is more malleable than auditory timing. However, the present results indicated that the shift in motor timing was largest (it was estimated by subtracting the shift of perceptual latency (5.6 ms shift in the MA) from the total size of TRE (e.g., 21.5 ms shift in the MA), which gave 15.9 ms motor-shift in the MA), followed by auditory timing (which is reflected by the RT shift, a 5.6 ms shift), whereas the shift of visual timing was smallest (0.9 ms shift) (see Table 1).

A key concept that might reconcile this discrepancy is the predictability of when a sensory signal is to occur. If the timing of a sensory signal is more predictable, the more adaptable it could be. Rohde et al. [40] found support for this explanation and argued that predictability is necessary for visuomotor TRE to occur. Moreover, Vercillo et al. [128] showed that inaccurate perception of timing prevents sensorimotor TRE to occur. Supporting evidence also comes from studies of the intentional binding. It is well known that predictability of sensory feedback enhances the intentional binding [80, 129]. Perhaps, whether or not events are controllable might be more important for sensory-motor TRE than the cue reliability, which might make predictability play a key role in sensory-motor TRE [40].

Variability of asynchrony and RT

The non-significant difference in variability of asynchrony between sync and delay is in line with our earlier study [17, 20] (but see [37]). It supports the notion that the sensitivity to temporal asynchrony does not change after exposure to delayed sensory feedback. Lesser tapping variability in MA than MV was also compatible to earlier findings indicating that tapping is more stable for auditory pacing stimuli than visual pacing stimuli [20, 37, 121, 122, 130] (for review [42]). However, the greater variability in MA-RT than MV-RT is not in line with earlier researches, which showed that visual RT is more variable than auditory one [46, 131, 132]. One possible explanation might be that the perceived strength of auditory stimuli is weaker than that of visual stimuli in the present study (e.g., additional white noise during the experiment masks auditory stimuli), which causes more variable RT in MA than in MV [46].

Conclusion

Our results demonstrate that delayed auditory feedback to finger taps not only induced a shift in the perception of audio-motor synchrony, but it also sped-up auditory latency that (partly) compensated the delay. Delayed visual feedback did not induce similar shifts in the perception of motor-visual synchrony or visual latency, presumably because timing information in the visual domain is less precise and salient. These results thus demonstrate the brain maintains unity between the senses partly by shifting them in time.

Supporting information

S1 Dataset. CSV files for asynchronies and RTs in the present study.

Raw data, processed raw data in which outliers are eliminated, and mean (median) and WSD (QD) of asynchrony and RTs are included. See 'README.txt' for descriptions of each CSV file.

https://doi.org/10.1371/journal.pone.0189242.s001

(ZIP)

Acknowledgments

This work was supported by Kyushu Sangyo University, Tilburg University and JSPS KAKENHI Grant Number 26380998. We thank the editor and two anonymous reviewers for their helpful comments and suggestions on a previous version of this manuscript.

References

  1. 1. Shadmehr R, Smith MA, Krakauer JW. Error Correction, Sensory Prediction, and Adaptation in Motor Control. Annu Rev Neurosci. 2010;33: 89–108. pmid:20367317
  2. 2. Keetels M, Vroomen J. Perception of synchrony between the senses. In: Murray MM, Wallace MT, editors. The neural basis of multisensory processes. Boca Raton (FL): CRC Press/Taylor & Francis; 2012. pp. 147–178.
  3. 3. Parsons BD, Novich SD, Eagleman DM. Motor-sensory recalibration modulates perceived simultaneity of cross-modal events at different distances. Front Psychol. 2013;4. pmid:23549660
  4. 4. Nijhawan R. Visual prediction: psychophysics and neurophysiology of compensation for time delays. Behav Brain Sci. 2008;31: 179–239. pmid:18479557
  5. 5. Wolpert DM, Ghahramani Z, Jordan MI. An internal model for sensorimotor integration. Science. 1995;269: 1880–1882. pmid:7569931
  6. 6. Perruchoud D, Murray MM, Lefebvre J, Ionta S. Focal dystonia and the Sensory-Motor Integrative Loop for Enacting (SMILE). Front Hum Neurosci. 2014;8. pmid:24999327
  7. 7. Cunningham DW, Billock VA, Tsou BH. Sensorimotor adaptation to violations of temporal contiguity. Psychol Sci. 2001;12: 532–535. pmid:11760144
  8. 8. Stetson C, Cui X, Montague PR, Eagleman DM. Motor-sensory recalibration leads to an illusory reversal of action and sensation. Neuron. 2006;51: 651–659. pmid:16950162
  9. 9. Yamamoto K, Kawabata H. Temporal recalibration in vocalization induced by adaptation of delayed auditory feedback. PLoS One. 2011;6: e29414. pmid:22216275
  10. 10. Yamamoto K, Kawabata H. Adaptation to delayed auditory feedback induces the temporal recalibration effect in both speech perception and production. Exp Brain Res. 2014;232: 3707–3718. pmid:25106757
  11. 11. Keetels M, Vroomen J. Exposure to delayed visual feedback of the hand changes motor-sensory synchrony perception. Exp Brain Res. 2012;219: 431–440. pmid:22623088
  12. 12. Rohde M, Ernst MO. To lead and to lag—forward and backward recalibration of perceived visuo-motor simultaneity. Front Psychol. 2012;3. pmid:23346063
  13. 13. Rohde M, Greiner L, Ernst MO. Asymmetries in visuomotor recalibration of time perception: does causal binding distort the window of integration? Acta Psychol. 2013;147: 127–135.
  14. 14. Vroomen J, Keetels M. Perception of intersensory synchrony: a tutorial review. Attention, Perception, Psychophys. 2010;72: 871–884.
  15. 15. Heron J, Hanson JVM, Whitaker D. Effect before cause: supramodal recalibration of sensorimotor timing. PLoS One. 2009;4: e7681. pmid:19890383
  16. 16. Sugano Y, Keetels M, Vroomen J. Adaptation to motor-visual and motor-auditory temporal lags transfer across modalities. Exp Brain Res. 2010;201: 393–399. pmid:19851760
  17. 17. Sugano Y, Keetels M, Vroomen J. Concurrent sensorimotor temporal recalibration to different lags for the left and right hand. Front Psychol. 2014;5. pmid:24624098
  18. 18. Tsujita M, Ichikawa M. Non-retinotopic motor-visual recalibration to temporal lag. Front Psychol. 2012;3. pmid:23293610
  19. 19. Yarrow K, Sverdrup-Stueland I, Roseboom W, Arnold DH. Sensorimotor temporal recalibration within and across limbs. J Exp Psychol Hum Percept Perform. 2013;39: 1678–1689. pmid:23565742
  20. 20. Sugano Y, Keetels M, Vroomen J. The build-up and transfer of sensorimotor temporal recalibration measured via a synchronization task. Front Psychol. 2012;3. pmid:22807921
  21. 21. Aschersleben G, Prinz W. Delayed auditory feedback in synchronization. J Mot Behav. 1997;29: 35–46. pmid:20037008
  22. 22. Fujisaki W, Shimojo S, Kashino M, Nishida S. Recalibration of audiovisual simultaneity. Nat Neurosci. 2004;7: 773–778. pmid:15195098
  23. 23. Vroomen J, Keetels M, de Gelder B, Bertelson P. Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cogn Brain Res. 2004;22: 32–35.
  24. 24. Navarra J, Soto-Faraco S, Spence C. Adaptation to audiotactile asynchrony. Neurosci Lett. 2007;413: 72–76. pmid:17161530
  25. 25. Keetels M, Vroomen J. Temporal recalibration to tactile-visual asynchronous stimuli. Neurosci Lett. 2008;430: 130–134. pmid:18055112
  26. 26. Takahashi K, Saiki J, Watanabe K. Realignment of temporal simultaneity between vision and touch. Neuroreport. 2008;19: 319–322. pmid:18303574
  27. 27. Vatakis A, Navarra J, Soto-Faraco S, Spence C. Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments. Exp Brain Res. 2008;185: 521–529. pmid:17962929
  28. 28. Roseboom W, Arnold DH. Twice upon a time: multiple concurrent temporal recalibrations of audiovisual speech. Psychol Sci. 2011;22: 872–877. pmid:21690312
  29. 29. Yuan X, Bi C, Huang X. Multiple concurrent temporal recalibrations driven by audiovisual stimuli with apparent physical differences. Attention, Perception, Psychophys. 2015;77: 1321–1332.
  30. 30. Hanson JVM, Heron J, Whitaker D. Recalibration of perceived time across sensory modalities. Exp Brain Res. 2008;185: 347–352. pmid:18236035
  31. 31. Harrar V, Harris LR. The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res. 2008;186: 517–524. pmid:18183377
  32. 32. Stekelenburg JJ, Sugano Y, Vroomen J. Neural correlates of motor-sensory temporal recalibration. Brain Res. 2011;1397: 46–54. pmid:21600564
  33. 33. Toida K, Ueno K, Shimada S. An ERP study on temporal recalibration for delayed auditory feedback. Proceedings of the the 15th International Multisensory Research Forum. Amsterdam, the Netherlands; 2014. p. 216.
  34. 34. Navarra J, Hartcher-O’Brien J, Piazza E, Spence C. Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proc Natl Acad Sci U S A. 2009;106: 9169–9173. pmid:19458252
  35. 35. Di Luca M, Machulla T, Ernst MO. Recalibration of multisensory simultaneity: cross-modal transfer coincides with a change in perceptual latency. J Vis. 2009;9(12): 1–16.
  36. 36. Keetels M, Vroomen J. No effect of auditory-visual spatial disparity on temporal recalibration. Exp Brain Res. 2007;182: 559–565. pmid:17598092
  37. 37. Sugano Y, Keetels M, Vroomen J. Auditory dominance in motor-sensory temporal recalibration. Exp Brain Res. 2016;234: 1249–1262. pmid:26610349
  38. 38. Aschersleben G. Temporal Control of Movements in Sensorimotor Synchronization. Brain Cogn. 2002;48: 66–79. pmid:11812033
  39. 39. Kennedy JS, Buehner MJ, Rushton SK. Adaptation to sensory-motor temporal misalignment: instrumental or perceptual learning? Q J Exp Psychol. 2009;62: 453–469.
  40. 40. Rohde M, van Dam LCJ, Ernst MO. Predictability is necessary for closed-loop visual feedback delay adaptation. J Vis. 2014;14(3): 1–23.
  41. 41. Wohlschläger A, Koch R. Synchronization error: An error in time perception. In: Desain P, Windsor L, editors. Rhythm perception and production. Swets & Zeitlinger Lisse, The Netherlands; 2000. pp. 115–127.
  42. 42. Repp BH. Sensorimotor synchronization: a review of the tapping literature. Psychon Bull Rev. 2005;12: 969–992. pmid:16615317
  43. 43. Sternberg S, Knoll RL. The Perception of temporal order: fundamental issues and a general model. Atten Perform IV. 1973; 629–685.
  44. 44. Sanford AJ. Attention bias and the relation of perception lag to simple reaction time. J Exp Psychol. American Psychological Association; 1974;102: 443–446.
  45. 45. Lupp U, Hauske G, Wolf W. Perceptual latencies to sinusoidal gratings. Vision Res. 1976;16: 969–972. pmid:948887
  46. 46. Ulrich R, Stapf KH. A double-response paradigm to study stimulus intensity effects upon the motor system in simple reaction time experiments. Percept Psychophys. 1984;36: 545–558. pmid:6535100
  47. 47. Jaśkowski P, Pruszewicz A, Świdzinski P. VEP latency and some properties of simple motor reaction-time distribution. Psychol Res. 1990;52: 28–34. pmid:2377722
  48. 48. Jaśkowski P. Reaction time and temporal-order judgment as measures of perceptual latency: the problem of dissociations. In: Aschersleben G, Bachmann T, Müsseler J, editors. Cognitive Contributions to the Perception of Spatial and Temporal Events. North-Holland; 1999. pp. 265–282.
  49. 49. Cardoso-Leite P, Gorea A, Mamassian P. Temporal order judgment and simple reaction times: evidence for a common processing system. J Vis. 2007;7: 1–14.
  50. 50. Schmidt RA, Lee TD. Motor control and learning: a behavioral emphasis. 5th ed. Champaign, IL: Human kinetics; 2011.
  51. 51. Large EW, Herrera JA, Velasco MJ. Neural Networks for Beat Perception in Musical Rhythm. Front Syst Neurosci. 2015;9. pmid:26635549
  52. 52. Pikovsky A, Rosenblum M, Kurths J. Synchronization: a universal concept in nonlinear sciences. Cambridge university press; 2001.
  53. 53. Pfordresher PQ, Palmer C. Effects of delayed auditory feedback on timing of music performance. Psychol Res. 2002;66: 71–79. pmid:11963280
  54. 54. Pfordresher PQ, Bella SD. Delayed auditory feedback and movement. J Exp Psychol Hum Percept Perform. 2011;37: 566–579. pmid:21463087
  55. 55. Luce RD. Response times: their role in inferring elementary mental organization. Oxford University Press; 1986.
  56. 56. Teki S, Grube M, Kumar S, Griffiths TD. Distinct neural substrates of duration-based and beat-based auditory timing. J Neurosci. 2011;31: 3805–3812. pmid:21389235
  57. 57. Grahn JA, Brett M. Rhythm and beat perception in motor areas of the brain. J Cogn Neurosci. 2007;19: 893–906. pmid:17488212
  58. 58. Merchant H, Harrington DL, Meck WH. Neural basis of the perception and estimation of time. Annu Rev Neurosci. 2013;36: 313–336. pmid:23725000
  59. 59. Teki S, Grube M, Griffiths TD. A unified model of time perception accounts for duration-based and beat-based timing mechanisms. Front Integr Neurosci. 2012;5. pmid:22319477
  60. 60. Sugano Y, Stekelenburg J, de Schipper F, Vroomen J. ERP correlates of sensory facilitation after exposure to delayed auditory feedback. Proceedings of the 33rd Annual Meeting of the International Society for Psychophysics (Fechner Day 2017). Fukuoka, Japan; 2017. p. 118.
  61. 61. Grondin S. Duration discrimination of empty and filled intervals marked by auditory and visual signals. Percept Psychophys. 1993;54: 383–394. pmid:8414897
  62. 62. Grondin S. Timing and time perception: a review of recent behavioral and neuroscience findings and theoretical directions. Attention, Perception, Psychophys. 2010;72: 561–582.
  63. 63. Grondin S, Meilleur-Wells G, Ouellette C, Macar F. Sensory effects on judgments of short time-intervals. Psychol Res. 1998;61: 261–268. pmid:9870293
  64. 64. Grahn JA. See what I hear? Beat perception in auditory and visual rhythms. Exp Brain Res. 2012;220: 51–61. pmid:22623092
  65. 65. Hove MJ, Fairhurst MT, Kotz SA, Keller PE. Synchronizing with auditory and visual rhythms: an fMRI assessment of modality differences and modality appropriateness. Neuroimage. 2013;67: 313–321. pmid:23207574
  66. 66. Thaut MH, Kenyon GP, Schauer ML, McIntosh GC. The connection between rhythmicity and brain function. IEEE Eng Med Biol Mag. IEEE; 1999;18: 101–108.
  67. 67. Grahn JA, Rowe JB. Feeling the beat: premotor and striatal interactions in musicians and nonmusicians during beat perception. J Neurosci. 2009;29: 7540–7548. pmid:19515922
  68. 68. Howell DC. Statistical methods for psychology. 7th editio. Wadsworth Pub Co; 2010.
  69. 69. Elliott R. Simple visual and simple auditory reaction time: a comparison. Psychon Sci. 1968;10: 335–336.
  70. 70. Niemi P. Stimulus intensity effects on auditory and visual reaction processes. Acta Psychol. 1979;43: 299–312.
  71. 71. Jaśkowski P, Jaroszyk F, Hojan-Jezierska D. Temporal-order judgments and reaction time for stimuli of different modalities. Psychol Res. 1990;52: 35–38. pmid:2377723
  72. 72. Ratcliff R, Murdock BB. Retrieval Processes in Recognition Memory. Psychol Rev. 1976;83: 190–214.
  73. 73. Ratcliff R. A theory of memory retrieval. Psychol Rev. 1978;85: 59–108.
  74. 74. Ratcliff R. Group reaction time distributions and an analysis of distribution statistics. Psychol Bull. 1979;86: 446–461. pmid:451109
  75. 75. Ratcliff R. Methods for dealing with reaction time outliers. Psychol Bull. 1993;114: 510–532. pmid:8272468
  76. 76. Matzke D, Wagenmakers E-J. Psychological interpretation of the ex-Gaussian and shifted Wald parameters: a diffusion model analysis. Psychon Bull Rev. 2009;16: 798–817. pmid:19815782
  77. 77. Harrar V, Harris LR, Spence C. Multisensory integration is independent of perceived simultaneity. Exp Brain Res. 2016;235: 763–775. pmid:27872958
  78. 78. Navarra J, Vatakis A, Zampini M, Soto-Faraco S, Humphreys W, Spence C. Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cogn Brain Res. 2005;25: 499–507.
  79. 79. Winter R, Harrar V, Gozdzik M, Harris LR. The relative timing of active and passive touch. Brain Res. 2008;1242: 54–58. pmid:18634764
  80. 80. Haggard P, Clark S, Kalogeras J. Voluntary action and conscious awareness. Nat Neurosci. 2002;5: 382–385. pmid:11896397
  81. 81. Vorberg D, Wing A. Chapter 4 Modeling variability and dependence in timing. In: Heuer Herbert, Keele SW, editors. Handbook of Perception and Action. Academic Press; 1996. pp. 181–262.
  82. 82. Pesavento MJ, Schlag J. Transfer of learned perception of sensorimotor simultaneity. Exp Brain Res. 2006;174: 435–442. pmid:16688417
  83. 83. Roach NW, Heron J, Whitaker D, McGraw P V. Asynchrony adaptation reveals neural population code for audio-visual timing. Proc R Soc B Biol Sci. 2011;278: 1314–1322.
  84. 84. Heron J, Aaen-Stockdale C, Hotchkiss J, Roach NW, McGraw P V, Whitaker D. Duration channels mediate human time perception. Proc R Soc B Biol Sci. 2012;279: 690–698.
  85. 85. Coull JT, Nobre AC. Where and when to pay attention: the neural systems for directing attention to spatial locations and to time intervals as revealed by both PET and fMRI. J Neurosci. 1998;18: 7426–7435. pmid:9736662
  86. 86. Coull JT, Frith CD, Büchel C, Nobre AC. Orienting attention in time: behavioural and neuroanatomical distinction between exogenous and endogenous shifts. Neuropsychologia. 2000;38: 808–819. pmid:10689056
  87. 87. Coull JT. fMRI studies of temporal attention: allocating attention within, or towards, time. Cogn Brain Res. 2004;21: 216–226.
  88. 88. Coull JT, Nobre AC. Dissociating explicit timing from temporal expectation with fMRI. Curr Opin Neurobiol. 2008;18: 137–144. pmid:18692573
  89. 89. Hughes RW, Vachon F, Jones DM. Auditory attentional capture during serial recall: violations at encoding of an algorithm-based neural model? J Exp Psychol Learn Mem Cogn. 2005;31: 736–749. pmid:16060777
  90. 90. Repp BH. Compensation for subliminal timing perturbations in perceptual-motor synchronization. Psychol Res. 2000;63: 106–128. pmid:10946585
  91. 91. Titchener EB. Lectures on the elementary psychology of feeling and attention. Macmillan; 1908.
  92. 92. Spence C, Shore DI, Klein RM. Multisensory prior entry. J Exp Psychol Hum Percept Perform. 2001;130: 799–832.
  93. 93. Hillyard SA, Münte TF. Selective attention to color and location: an analysis with event-related brain potentials. Percept Psychophys. Springer; 1984;36: 185–198.
  94. 94. McDonald JJ, Teder-Sälejärvi W a, Di Russo F, Hillyard S a. Neural basis of auditory-induced shifts in visual time-order perception. Nat Neurosci. 2005;8: 1197–1202. pmid:16056224
  95. 95. Vibell J, Klinge C, Zampini M, Spence C, Nobre a C. Temporal order is coded temporally in the brain: early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task. J Cogn Neurosci. 2007;19: 109–120. pmid:17214568
  96. 96. Toida K, Ueno K, Shimada S. Neural Basis of the Time Window for Subjective Motor-Auditory Integration. Front Hum Neurosci. 2016;9. pmid:26779000
  97. 97. Balota DA, Spieler DH. Word frequency, repetition, and lexicality effects in word recognition tasks: beyond measures of central tendency. J Exp Psychol Gen. 1999;128. 32–55. pmid:10100390
  98. 98. Kieffaber PD, Kappenman ES, Bodkins M, Shekhar A, O’Donnell BF, Hetrick WP. Switch and maintenance of task set in schizophrenia. Schizophr Res. 2006;84: 345–358. pmid:16563700
  99. 99. Buhusi C V, Meck WH. Relative time sharing: new findings and an extension of the resource allocation model of temporal processing. Philos Trans R Soc Lond B Biol Sci. 2009;364: 1875–1885. pmid:19487190
  100. 100. Humphreys GR, Buehner MJ. Magnitude estimation reveals temporal binding at super-second intervals. J Exp Psychol Hum Percept Perform. 2009;35: 1542–1549. pmid:19803655
  101. 101. Buehner MJ, Humphreys GR. Causal binding of actions to their effects. Psychol Sci. 2009;20: 1221–1228. pmid:19732384
  102. 102. Desantis A, Roussel C, Waszak F. On the influence of causal beliefs on the feeling of agency. Conscious Cogn. 2011;20: 1211–1220. pmid:21396831
  103. 103. Moore JW, Obhi SS. Intentional binding and the sense of agency: a review. Conscious Cogn. 2012;21: 546–561. pmid:22240158
  104. 104. Engbert K, Wohlschläger A, Haggard P. Who is causing what? The sense of agency is relational and efferent-triggered. Cognition. 2008;107: 693–704. pmid:17825813
  105. 105. Wohlschläger A, Engbert K, Haggard P. Intentionality as a constituting condition for the own self—and other selves. Conscious Cogn. 2003;12: 708–716. pmid:14656512
  106. 106. Engbert K, Wohlschläger A, Thomas R, Haggard P. Agency, subjective time, and other minds. J Exp Psychol. 2007;33: 1261–1268.
  107. 107. Yabe Y, Goodale MA. Time flies when we intend to act: Temporal distortion in a go/no-go task. J Neurosci. 2015;35: 5023–5029. pmid:25810531
  108. 108. Cai M, Stetson C, Eagleman DM. A neural model for temporal order judgments and their active recalibration: a common mechanism for space and time? Front Psychol. 2012;3. pmid:23130010
  109. 109. Wenke D, Haggard P. How voluntary actions modulate time perception. Exp Brain Res. 2009;196: 311–318. pmid:19471909
  110. 110. Timm J, Schönwiesner M, Sanmiguel I, Schröger E. Sensation of agency and perception of temporal order. Conscious Cogn. 2014;23: 42–52. pmid:24362412
  111. 111. Meck WH. Attentional bias between modalities: effect on the internal clock, memory, and decision stages used in animal time discrimination. Ann N Y Acad Sci. 1984;423: 528–541. pmid:6588813
  112. 112. Bertelson P, Aschersleben G. Temporal ventriloquism: crossmodal interaction on the time dimension 1. Evidence from auditory–visual temporal order judgment. Int J Psychophysiol. 2003;50: 147–155. pmid:14511842
  113. 113. Aschersleben G, Bertelson P. Temporal ventriloquism: crossmodal interaction on the time dimension 2. Evidence from sensorimotor synchronization. Int J Psychophysiol. 2003;50: 157–163. pmid:14511843
  114. 114. Allman MJ, Teki S, Griffiths TD, Meck WH. Properties of the internal clock: first- and second-order principles of subjective time. Annu Rev Psychol. 2014;65: 743–771. pmid:24050187
  115. 115. Leiva A, Parmentier FBR, Andrés P. Distraction by deviance comparing the effects of auditory and visual deviant stimuli on auditory and visual target processing. Exp Psychol. 2015;62: 54–65. pmid:25270560
  116. 116. Heron J, Roach NW, Whitaker D, Hanson JVM. Attention regulates the plasticity of multisensory timing. Eur J Neurosci. 2010;31: 1755–1762. pmid:20584179
  117. 117. Ikumi N, Soto-Faraco S. Selective attention modulates the direction of audio-visual temporal recalibration. PLoS One. 2014;9. pmid:25004132
  118. 118. Tsujita M, Ichikawa M. Awareness of temporal lag is necessary for motor-visual temporal recalibration. Front Integr Neurosci. 2016;9. pmid:26778983
  119. 119. Fujisaki W. Effects of delayed visual feedback on grooved pegboard test performance. Front Psychol. 2012;3. pmid:22408631
  120. 120. Fraisse P. Multisensory aspects of rhythm. In: Walk R, Pick H Jr., editors. Intersensory Perception and Sensory Integration. Springer US; 1981. pp. 217–248.
  121. 121. Repp BH, Penel A. Auditory dominance in temporal processing: new evidence from synchronization with simultaneous visual and auditory sequences. J Exp Psychol Hum Percept Perform. 2002;28: 1085–1099. pmid:12421057
  122. 122. Repp BH, Penel A. Rhythmic movement is attracted more strongly to auditory than to visual rhythms. Psychol Res. 2004;68: 252–270. pmid:12955504
  123. 123. Sugano Y, Keetels M, Vroomen J. Perceptual latency of sensory stimuli is changed with motor-auditory but not with motor-visual temporal recalibration. IEICE Technical Report. 2015;115: 71–74.
  124. 124. Welch RB, Warren DH. Immediate perceptual response to intersensory discrepancy. Psychol Bull. American Psychological Association; 1980;88: 638–667.
  125. 125. Ernst MO, Bülthoff HH. Merging the senses into a robust percept. Trends Cogn Sci. 2004;8: 162–169. pmid:15050512
  126. 126. Burge J, Girshick AR, Banks MS. Visual-haptic adaptation is determined by relative reliability. J Neurosci. 2010;30: 7714–7721. pmid:20519546
  127. 127. Chen L, Vroomen J. Intersensory binding across space and time: a tutorial review. Atten Percept Psychophys. 2013;75: 790–811. pmid:23709064
  128. 128. Vercillo T, Burr D, Sandini G, Gori M. Children do not recalibrate motor-sensory temporal order after exposure to delayed sensory feedback. Dev Sci. 2014;18: 703–712. pmid:25444457
  129. 129. Moore J, Haggard P. Awareness of action: inference and prediction. Conscious Cogn. 2008;17: 136–144. pmid:17306565
  130. 130. Kolers PA, Brewster JM. Rhythm and responses. J Exp Psychol Hum Percept Perform. 1985;11: 150–167. pmid:3159834
  131. 131. Giray M, Ulrich R. Motor coactivation revealed by response force in divided and focused attention. J Exp Psychol Hum Percept Perform. 1993;19: 1278–1291. pmid:8294892
  132. 132. Stauffer CC, Haldemann J, Troche SJ, Rammsayer TH. Auditory and visual temporal sensitivity: evidence for a hierarchical structure of modality-specific and modality-independent levels of temporal information processing. Psychol Res. 2012;76: 20–31. pmid:21461936