The authors have declared that no competing interests exist.
Conceived and designed the experiments: AS JCC. Performed the experiments: AS. Analyzed the data: SM AS LDK. Wrote the paper: JCC AS SM LDK.
Behavioral and neuropsychological research suggests that delayed actions rely on different neural substrates than immediate actions; however, the specific brain areas implicated in the two types of actions remain unknown. We used functional magnetic resonance imaging (fMRI) to measure human brain activation during delayed grasping and reaching. Specifically, we examined activation during visual stimulation and action execution separated by a 18-s delay interval in which subjects had to remember an intended action toward the remembered object. The long delay interval enabled us to unambiguously distinguish visual, memory-related, and action responses. Most strikingly, we observed reactivation of the lateral occipital complex (LOC), a ventral-stream area implicated in visual object recognition, and early visual cortex (EVC) at the time of action. Importantly this reactivation was observed even though participants remained in complete darkness with no visual stimulation at the time of the action. Moreover, within EVC, higher activation was observed for grasping than reaching during both vision and action execution. Areas in the dorsal visual stream were activated during action execution as expected and, for some, also during vision. Several areas, including the anterior intraparietal sulcus (aIPS), dorsal premotor cortex (PMd), primary motor cortex (M1) and the supplementary motor area (SMA), showed sustained activation during the delay phase. We propose that during delayed actions, dorsal-stream areas plan and maintain coarse action goals; however, at the time of execution, motor programming requires re-recruitment of detailed visual information about the object through reactivation of (1) ventral-stream areas involved in object perception and (2) early visual areas that contain richly detailed visual representations, particularly for grasping.
Grasping an object no longer in view requires the motor system to access stored information about the size, shape, position, and orientation of the intended object to guide and preshape the hand. Memory-guided grasping is distinct from immediate grasping, where accurate information about relevant object properties is continuously provided by the visual system to the motor system on a moment-to-moment basis.
Considerable research has investigated the neural substrates of immediate actions. In an influential model of perception and action
Far less is known about the neural substrates of memory-guided actions, although neuropsychology and behavioral studies have suggested they may rely on somewhat different mechanisms than immediate actions. Neuropsychology has revealed a double dissociation between the ability to perform immediate vs. delayed actions. Specifically, a patient, DF, with visual form agnosia and bilateral lesions within LOC
Taken together, this evidence has been used to argue that memory-driven actions rely more on the ventral visual stream than immediate actions because of different computational requirements
However, not all research supports this model. Some behavioral findings have been used to argue against a strict dichotomy
We used neuroimaging in healthy normal participants to investigate the contributions of the dorsal and ventral streams during initial object presentation, an 18-s delay, and execution of grasping or reaching actions. This long-delay design enabled us to clearly distinguish activation related to visual stimulation/encoding, memory maintenance during the delay period, and memory retrieval at the time of execution, thus providing a richer characterization of dorsal and ventral stream brain regions to delayed actions.
Due to the limited number of slices that could be collected in a 2-s volume acquisition time on the high-field MRI scanner we employed, data from two groups were collected with two different slice orientations, one to collect frontal and parietal and data (n = 11) and one to collect also occipital and temporal cortex data (n = 9) (
(
We used high-field (4-Tesla) functional magnetic resonance imaging (fMRI) to measure the blood-oxygenation-level dependent (BOLD) signal
Although behavioral research has shown that brief delays of 2 s
Importantly, aside from the brief period of visual illumination, participants remained in complete darkness, except for a small light emitting diode (LED), which was too dim to allow vision of anything else within the scanner bore. Reaction time (RT) and accuracy for Go responses were collected when the participant released a key placed on the torso.
In sum, the 2×2 factorial design, with factors of task cue (Reach or Grasp) and execution cue (Go or Stop) led to four trial types: Grasp Go, Grasp Stop, Reach Go and Reach Stop. Because Go and Stop periods were only distinguished at the end of the delay period, prior to that the data were grouped into just two categories, Grasp and Reach.
There were 16 trials per experimental run, with each of the four trial types presented in counterbalanced order for a run time of ∼11 min. Participants completed on average 106 trials (min 64 trials, max 160 trials).
3D objects were presented on the original “grasparatus” grasping apparatus developed within our lab
The participants lay in the magnet with their heads and torsos tilted (∼30 deg.) to allow direct viewing of the target area without the use of mirrors. The right upper arm was restrained with an arm brace to restrict shoulder movements, but allowed for full motion of the elbow and wrist. The grasparatus was placed above the participant’s hips to allow comfortable reaching and grasping from a starting position on the chest where the right index finger was placed over a key-press.
We are not able to use an MR-compatible eye tracker with the head-tilted configuration because the eyelids droop in that posture, occluding too much of the pupil to detect eye position; nevertheless, our subjects were highly experienced fMRI participants who were accustomed to the requirement to maintain fixation throughout fMRI runs. Moreover, eye tracking outside the scanner showed that a similar group of subjects could successfully fixate throughout a run with negligible eye movements and no differences between grasping and reaching conditions
To localize visuomotor areas, we included runs of immediate grasping and reaching without visual feedback (open loop) in 9 of 11 participants of Group A (for 2 of 11 participants, insufficient time was available to include these localizer scans). To localize the object-selective lateral occipital cortex, we obtained data comparing objects vs. scrambled objects
Scanning was done in a 4-Tesla whole-body MRI system (Siemens-Varian) at the Robarts Research Institute (London, ON, Canada) employing a single-channel transmission-reception cylindrical birdcage radiofrequency whole-head coil. Functional volumes were collected using a T2*-weighted, segmented gradient-echo echoplanar imaging (19.2 cm field of view with 64×64 matrix size for an in-plane resolution of 3 mm; repetition time, TR = 1 s, with two segments/plane for a volume acquisition time of 2 s; time to echo, TE = 15 ms; flip angle, FA = 45 deg.). Each volume was made up of 15 contiguous slices of 5-mm thickness. For Group A, slices were angled at approximately 15 deg. from axial to cover entire parietal cortex and superior frontal cortex, and for Group B, slices were angled at approximately 30 deg. from axial to cover the entire parietal cortex, entire occipital cortex, posterior temporal cortex and posterior frontal cortex (
The imaging data were preprocessed and analyzed using Brain Voyager software (BV QX 1.10, Brain Innovation, Maastricht, The Netherlands). Anatomical volumes were transformed into standard stereotaxic space
Data were analyzed using a random effects (RFX) general linear model (GLM) with separate predictors for each phase of each trial type: Reach-Visual, Reach–Delay, Reach–Go, Reach-Stop, Grasp-Visual, Grasp-Delay, Grasp-Go, Grasp-Stop. The Visual, Go, and Stop predictors were modeled with a single-volume (2-s) rectangular wave, and the Delay predictors were modeled with a 9-volume (18-s) rectangular wave that was sustained throughout the delay period. Each predictor was convolved with a standard hemodynamic response function (HRF; Boynton model). Although the visual stimulus lasted less than 2 s (250 ms), we found that the predicted HRF for a 2-s Visual predictor demonstrated a good fit with visual response seen in the event-related average time courses across brain areas, perhaps because sensorimotor processing continued beyond visual stimulation (e.g., iconic memory, visual encoding, motor planning). Similarly, the predicted HRF for the Go predictor demonstrated a good fit with the execution response seen in the event-related average time courses across brain areas. Time course data were z-transformed prior to analysis. Since conditions were distributed equally within runs (and thus had the same standard deviation), the beta weights reflect the magnitude of activation.
During the action phase in the Go conditions, the hand actions sometimes caused a distortion of the magnetic field observed as negative spikes during the first volume of each event. These motion artifacts occurred abruptly and without the standard hemodynamic lag (∼ 5 s) and response profile. To adequately account for the variance due to these artifacts, the GLM included two single-volume spike predictors of no interest (one for Reach-Go and one for Grasp-Go) that peaked one volume after the go cue (corresponding with the spikes that can be observed as for example in the LOC time courses in
Activation maps show group activation in aIPS, identified by Grasping>Reaching in Group A and in LOC, identified by Objects>Scrambled in Group B (RFX GLM contrasts; t >3 for aIPS; t >4.5 for LOC, corrected). Activation is superimposed on the group-averaged anatomical slices. Although group data is presented to illustrate the average location of the areas, time course data and statistical comparisons between conditions were performed on regions of interest extracted from individual subjects based on data from independent localizer runs. In individuals, areas aIPS (
To correct for the problem of multiple comparisons, we used Brain Voyager’s cluster-level statistical threshold estimator plug-in. This algorithm uses Monte Carlo simulations (1000 iterations) to estimate the probability of clusters of a given size arising purely from chance. Because the minimum cluster size for a corrected p value is estimated separately for each map (based on smoothness estimates), cluster sizes can vary across different comparisons. Nevertheless all the clusters reported have a minimum size of at least 6× (3 mm)3 = 162 mm3. All post hoc contrasts between conditions were performed using Brain Voyager’s region-of-interest general linear model (ROI-GLM) feature with RFX to compute statistical significance based on beta weights within each region.
We used RFX-GLM contrasts to (a) identify, at the group level, key regions for which we had hypotheses (including those derived from independent localizers); and (b) perform an exploratory search of activations for key experimental contrasts. To show the patterns of data across the phases and tasks, we have plotted event-related average time courses and performed statistical comparisons between critical conditions in each phase.
To avoid issues with analyses that are non-independent with respect to the selection criteria, we have used several approaches. Where possible, we have used selection criteria independent from the key experimental contrasts. Where that was not possible, we have used square brackets to flag contrasts that are non-independent of the means used to select the regions so the reader is aware of those cases. Averaged Talairach coordinates are provided in
Talairach Coordinates | |||
X | Y | Z | |
Left aIPS | −39 | −34 | 45 |
Right aIPS | 37 | −37 | 52 |
Left LOC | −47 | −66 | 1 |
Right LOC | 46 | −65 | 2 |
Left Overlap Region | −51 | −58 | 1 |
Right Overlap Region | 50 | −55 | 4 |
Left cuneus | −5 | −79 | 14 |
Right cuneus | 3 | −82 | 7 |
Left SPOC | −4 | −84 | 30 |
Pre-SMA | 0 | 11 | 53 |
Left PMd | −28 | −13 | 55 |
Right PMd | 23 | −9 | 49 |
Left IPS | −38 | −49 | 46 |
−27 | −55 | 45 | |
Right IPS | 25 | −51 | 45 |
Right precuneus | 12 | −73 | 43 |
20 | −70 | 36 | |
3 | −70 | 43 | |
Midline cuneus | −2 | −74 | 14 |
Left LOC/MTG | −47 | −62 | 2 |
Right LOC/MTG | 41 | −66 | 8 |
Pre-SMA/SMA | −5 | −3 | 47 |
Left PMd | −25 | −18 | 63 |
Right PMd | 24 | −13 | 61 |
Left mid-precentral sulcus | −45 | −1 | 40 |
Left insula | −37 | 2 | 13 |
Left M1 | −44 | −21 | 55 |
Left IPS | −47 | −35 | 55 |
−32 | −49 | 53 | |
Right IPS | 30 | −51 | 56 |
Left SII | −60 | −24 | 26 |
Right SII | 57 | −19 | 28 |
Left lateral SPOC | −14 | −79 | 36 |
Right lateral SPOC | 9 | −75 | 35 |
Right precuneus | 6 | −72 | 45 |
4 | −60 | 52 | |
3 | −50 | 58 | |
Left posterior cingulate sulcus | −11 | −32 | 43 |
Right posterior cingulate sulcus | 9 | −32 | 52 |
Left LOC/MTG | −53 | −62 | −1 |
Right LOC/MTG | 51 | −55 | 4 |
Anterior calcarine sulcus | 0 | −66 | 9 |
Posterior calcarine sulcus | 0 | −80 | 1 |
Left thalamus | −13 | −20 | 14 |
Right thalamus | 12 | −16 | 14 |
Superior cerebellum | −2 | −55 | −8 |
Left M1 | −39 | −19 | 51 |
−34 | −33 | 53 | |
SMA | −3 | −15 | 55 |
The contrasts used to define these areas are indicated with letters from A to H. Area abbreviations: aIPS, anterior intraparietal sulcus; LOC, lateral occipital complex; SPOC, superior parietal occipital sulcus; SMA, supplementary motor area; PMd, dorsal premotor; M1, primary motor; IPS, intraparietal suclus; SII, secondary somatosensory; MTG, middle temporal gyrus.
The behavioural data confirmed that the participants were executing the task correctly with respect to the “Go vs. Stop” conditions. The RT data revealed no significant differences between grasping and reaching conditions (grasping RT = 321.6 ms +/−9.8 (SE); reaching RT = 322.7 ms +/−10.6 (SE); t(10) = 0.896, p = 0.39 two-tailed).
Area aIPS was localized at the junction of the intraparietal and postcentral sulci using an independent contrast of Immediate Grasping>Immediate Reaching (t>3) from separate runs in 9 out of 11 participants of Group A. Time courses and post hoc analyses showed that in both hemispheres, aIPS showed significant activation relative to the intertrial interval for all three phases of the sequence (
These results demonstrate that human aIPS, like its putative macaque homologue, AIP, is sensitive to visual presentation (without simultaneous action), visuomotor delays, and reach and grasp actions in the dark (without simultaneous target vision or visual feedback). Moreover, the visual responses and grasp-selectivity during vision suggest that human aIPS is not merely activated by the greater somatosensory stimulation and motor demands of grasping vs. reaching (as for example is S1). Our results also strengthen the case for homologies between human aIPS and macaque AIP (
Area LOC was identified inferior to the junction of the inferior temporal sulcus and lateral occipital sulcus using an independent contrast of Intact Objects>Scrambled Objects (t>4.5) in all 9 participants of Group B. Although activation of LOC during visual stimulus presentation was expected, we were surprised to observe a robust reactivation of LOC at the time of action (larger for Go than Stop trials), even though participants were in complete darkness (
In addition to the ROI analyses above, we also conducted group voxelwise contrasts to further examine activation across these areas and others.
Although the ROI approach is valuable in evaluating responses in well-known areas, it neglects the possibility that in large areas such as LOC, only a subregion of the ROI may be activated in the experimental contrast. To investigate the pattern of activation within and around the LOC, we performed a voxelwise group analysis of activation overlap. As shown in
Activation maps show group activation from 9 participants of Group B on the group-averaged anatomical for different contrasts. Maps show significant effects for action reactivation (t >3, k = 10 voxels) in yellow and for the Object>Scrambled contrast (t >3, k = 9 voxels),in blue. The Venn diagram shows effects of overlapping activation colors. The time course is shown from the region of overlap of all three contrasts. Contrasts listed in square parentheses are expected given the criteria used to select the region. The blue dot indicates the Talairach coordinates of the overlapping region in the present study. Other dots indicate the Talairach coordinates of LOTV from previous studies: Pink = Amedi et al. 2001; Black = Amedi et al. 2002; Red = Tal & Amedi 2009).
Given the action reactivation observed in the vicinity of LOC, we wondered whether even early visual cortex (EVC) might reflect the same phenomenon. Although we did not have retinotopic mapping data for our participants, we identified the region of occipital cortex near the calcarine sulcus that showed the highest response for visual stimulus presentation [(Grasp and Reach)>Baseline, Group B], (
The statistical maps show areas activated during the visual phase of delayed grasping in 9 participants of Group B (t>5, corrected). The white lines represent the calcarine sulcus from each of the nine participants while the black straight line approximates the calcarine sulcus on the group-averaged anatomical. Although retinotopic mapping data were not available, we selected a visually activated region in each hemisphere (blue box) that was near and slightly above the calcarine sulcus, where the representation of the lower visual field in area V1 would be expected, and extracted the time courses for further analysis.
Recent evidence has shown that the human superior parieto-occipital cortex (SPOC) is involved in the arm transport component of reaching
SPOC was identified in Group A (n = 9) using a contrast of Immediate Grasping+Immediate Reaching>Baseline (t>8.5, corrected).
Another area of interest is the dorsal premotor cortex, PMd, which is involved in both the transport and grip components of reach-to-grasp actions
Area PMd was identified in Group A (n = 9) using a contrast of Vision phase Grasping+Vision phase Reaching>Baseline (t>5, corrected).
The use of a delay paradigm allowed us to qualitatively evaluate the relative contribution of visual stimulation and motor execution (and/or memory recall) in reach to grasp actions. As shown in the activation map in
Activation maps show the relative contribution between the visual and action predictors in a group voxelwise analysis (Group A, n = 11) shown on the group-average anatomical image for areas in which a gradient was observed (A) and other areas (B). For each voxel in which the visual and action predictors together accounted for a significant proportion of the variance (R >0.3), the figure shows the relative contribution index, computed the following contrast of beta weights: (+GG+RG-GV-RV)/(+GG+RG+GV+RV). Voxels with only responses to the action cues (grasping and reaching) appear dark blue; areas with only responses to the visual cues (both grasping and reaching) appear dark yellow; areas with comparable visual and motor responses appear in the intermediate range of the spectrum (light blue – light yellow). The time course for aIPS is the same as that in
The relative contribution map (as in
In addition to enabling the separation of vision- and action-related activity, the long delay period allows us to identify areas with sustained delay period activation. When we contrasted delay period activation against the intertrial baseline [Grasp Delay+Reach Delay)>ITI], only three foci survived cluster threshold correction: pre-SMA, SMA and the central sulcus, presumably M1 (
Areas SMA and M1 (second row) were identified in the subjects from Group A using the contrast Delay phase Grasping+Delay phase Reaching>Baseline (t>3.5, corrected).
A delayed action paradigm enabled us to examine the role of ventral- and dorsal-stream brain areas in visual processing, memory maintenance, and action execution requiring memory recall. First and most strikingly, we found that two putatively visual sensory areas – LOC and EVC – were not only activated during visual stimulus presentation as expected but were reactivated at the time of action
Recent functional neuroimaging studies have provided growing support for an intriguing proposal with a long history including suggestions by Karl Wernicke
Although most studies of memory reactivation have focused on the perceptual system, there have been some suggestions that similar reactivation processes are at play for memory-driven actions. For example, the recall of action phrases reactivates somatosensory and motor regions, particularly following performance of the action
In an influential explanation for the neuropsychological and kinematic differences between immediate and delayed actions, Goodale and colleagues
We propose that at the time of the initial stimulus presentation, information enters the visual system predominantly through the primary visual pathway and activates areas in the ventral stream (including LOC) and the dorsal stream. As this visual response decays away, sustained responses are maintained in some areas of the dorsal stream (including SPOC, aIPS, PMd and the preSMA).
One framework that may be helpful in considering the types of sensorimotor processing occurring in later phases is the distinction between motor planning and motor programming (as an analogy, one may make a
Another useful distinction is between motor planning and motor control
We suggest that our data and model clarify the explanation of impairments in delayed grasping across a wide range of neuropsychological patients. For example, we suggest that the preservation of many dorsal stream areas within DF
The proposed model derived from this fMRI data has been reinforced by “virtual lesion” data using transcranial magnetic stimulation (TMS) to LOC and aIPS during immediate and delayed actions
Our hope is also that a more detailed model will help to reconcile previously discrepant findings. For example, though we did not compare our delayed grasping activation to that for immediate grasping, our results are fully consistent with past fMRI studies showing strong recruitment of dorsal stream areas in both tasks
One under-appreciated possibility in this literature is that types of information required by delayed actions and the areas involved may differ depending on task or stimuli. For example, the case for the recruitment of the ventral stream in delayed actions seems to be particularly strong for the role of size in grasping. Indeed, the strongest behavioral evidence for an abrupt transition to “ventral-stream mode” comes from size illusions in grasping
One remaining question is how brain areas in the two streams share information at the time of action. Some clues come from an event-related potential (ERP) experiment
A related question is which areas of the frontoparietal dorsal stream network receive information from LOC and EVC. One candidate region is aIPS. The macaque brain contains direct connections between inferotemporal (IT) cortex and AIP
Another remaining question is what information is stored in various brain areas during the delay interval. Several dorsal-stream areas showed significant delay period activation that was comparable for grasping and reaching. Specifically sustained delay-period activation was observed in SMA, pre-SMA, and M1 (in voxelwise analyses) along with aIPS and PMd (in the ROI analysis). Although the delay-period activation in aIPS is consistent with its putative homology with macaque AIP
In considering neural processing during the delay period, it is important to note that neural coding may not necessarily be manifested as increased BOLD activation (relative to the intertrial interval). For example, pattern classifiers applied to fMRI activation during memory delays
Although our data did not include enough trials to apply pattern classification, information about the upcoming trial type may be decodable during the delay period as it is during a planning period in which the stimulus remains visible
In sum, these data have led to a detailed, testable model of how delayed actions may rely on brain areas within the ventral and dorsal visual streams. Indeed, two follow-up experiments using TMS
We are very grateful to Tutis Vilis, Adrian Aldcroft and Mary-Ellen Large for providing localizer data from a subset of the participants in our study, to Ken Valyear for technical support, and to Mel Goodale for helpful discussions.