Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Reading positional codes with fMRI: Problems and solutions

Reading positional codes with fMRI: Problems and solutions

  • Kristjan Kalm, 
  • Dennis Norris
PLOS
x

Abstract

Neural mechanisms which bind items into sequences have been investigated in a large body of research in animal neurophysiology and human neuroimaging. However, a major problem in interpreting this data arises from a fact that several unrelated processes, such as memory load, sensory adaptation, and reward expectation, also change in a consistent manner as the sequence unfolds. In this paper we use computational simulations and data from two fMRI experiments to show that a host of unrelated neural processes can masquerade as sequence representations. We show that dissociating such unrelated processes from a dedicated sequence representation is an especially difficult problem for fMRI data, which is almost exclusively the modality used in human experiments. We suggest that such fMRI results must be treated with caution and in many cases the assumed neural representation might actually reflect unrelated processes.

Introduction

One of the most important features of human cognition is the ability to bind individual events into a sequence. Almost any complex task requires us to remember not only the individual elements but also the order in which they occurred. For example, two tasks such as starting a car and stopping it might share the same events but in different order. All computational models of sequence processing acknowledge this distinction between the representations of items in memory and the representation of the order in which they occur [1, 2]. The view that item’s position in the sequence is encoded separately and independently of their identity has been also suggested by decades of research in human behaviour and animal neurophysiology.

Neurons in the monkey prefrontal cortex (PFC) have been found to be selective for each position in a learned sequence [36]. Fig 1 gives an example of a simple positional code showing the responses of position-sensitive neurons from monkey supplementary motor area as observed by [7]. Other research on animal neurophysiology has suggested that the hippocampus encodes the position of items in a sequence [6, 810], with some authors proposing the existence of ‘time cells’ tracking the temporal position of items in a sequence [11, 12]. From hereon we refer to such neural representation of the item’s position in the sequence as positional code. The extensive literature on the neural representation of the positional code is summarised in Table 1.

thumbnail
Fig 1. Sequence representation and temporal position.

(A) Representation of two sequences as mappings between item codes and temporal position codes. (B) Left: representation of temporal position in a 7-item sequence. The variance around positional signal is coded in terms of the darkness of the circle. Right: the order position is retrieved by reinstating each positional code which then cues the associated item. (C) Examples of temporal position selective neurons from [7]. From left to right: pre-supplementary motor area neuron selective for 1st position, supplementary eye field neuron selective for 2nd position, and supplementary motor area neuron selective for the 3rd position in the serial object task.

https://doi.org/10.1371/journal.pone.0176585.g001

However, interpreting a neural signal tracking the positional code suffers from a major methodological problem: items in different positions necessarily differ along other dimensions too. For example, in a memory task, memory load will be greater at position three than position two. Changes in neural activity that are sensitive to memory load might therefore give the appearance of coding position. An item in position n will always be associated with a load of n items. Any neural index of load will therefore consistently be in a different state for items in different positions. An item in position n also occurs at a later time than item n − 1. Sensory adaptation might change the neural response to items as the sequence progresses. Such a signal could also masquerade as a positional code. Any or all of these factors might therefore lead to a differential neural response which would correlate with the position of an item in a sequence, but which might play no role in determining how the brain codes temporal position. In their analysis of how we can measure information in the brain, [42] made a contrast between “cortex as receiver” and “experimenter as receiver”. There may be ways in which we as experimenters can decode neural states to recover information about temporal position, but what we would like to do is to identify specifically those neural representations that the cortex uses to represent temporal position and to drive behaviour.

In this paper we show that dissociating positional ‘read-out’ from a neural positional code is an especially difficult problem for fMRI data. We show that fMRI data acquired from sequentially presented stimuli suffer from several confounds. First, we show that with any sequence processing task there are experimental variables which are collinear with the positional signal (e.g. memory load, sensory adaptation, etc.) and which can serve as a positional code. Second, we show how interference between stimulus representations, task phases, and measurement modalities can also lead to a similar positional read-outs indistinguishable from a dedicated positional code. Importantly these correlated effects do not simply result in a univariate change in signal that varies across sequence position but also change the pattern of information that can be read out by multivariate methods.

The problem of interpreting a positional signal is especially relevant since neural data on human sequence processing comes almost exclusively from fMRI studies (Table 1). Our simulations and experimental data show that results from fMRI experiments studying the positional code must be treated with caution. Specifically, in many cases the assumed positional code might actually reflect processes which are correlated with position in the sequence instead.

Positional code from collinear processes

Any signal tracking the position of an item in a sequence will be collinear with a number of cognitive processes:

  • Memory load—signal for position n will always co-occur with a memory load of n items when storing a sequence. Any neural index of load will therefore always reflect the progression of sequence.
  • Sensory adaptation—neural responses in the human sensory cortex have been shown to monotonically decrease as a response to sequentially presented stimuli [4345]. Any signal that monotonically changes over sequence positions can be used to read out position-like code.
  • Reward—in most animal studies the subject is rewarded after successfully attending or recalling a sequence. This means that the next item in a sequence is always closer to the reward. Neurons tracking the temporal proximity of reward have been described in both monkey and rodent studies [6, 11, 18].
  • Passage of time—signal for position n always occurs after the signal for position n − 1.

All these processes represent a change in the cognitive state of the participant throughout the processed sequence, and hence will necessarily be collinear with any positional code. It follows that in the analysis of experiments on temporal order it is necessary to distinguish between a dedicated positional code and a positional read-out from collinear processes.

Next, we provide two examples of positional read-out based on human fMRI data. In the first example we show how sensory adaptation in the sensory cortices can be interpreted as a positional signal. In the second example we show how differences in retinotopic activation over the course of the sequence can similarly provide positional read-out. In the final part of the section we provide simulations which explore whether it is possible to develop methods to subtract the effects of such collinear processes from sequentially obtained data.

Sensory adaptation

Sensory adaptation across sequence positions has been observed in a number of fMRI studies of sequence processing as a decreasing univariate signal over positions [4345]. Note that an inverse trend, where the univariate signal increases over sequence positions, has also been observed [33]. The latter most likely reflects the attenuation of the BOLD signal in response to sequentially presented stimuli as reported in other fMRI studies on human STM [46, 47]. However, the direction of the univariate change is unimportant as any consistent change over sequence positions will permit position decoding.

Here we used two human fMRI datasets obtained with a sequence processing task [32, 33] to carry out a classification analysis of item position in a sequence. In both cases we chose the sensory cortex of the presented stimuli as a region of interest (ROI): in the first experiment the sequences were presented auditorily [32] and in the second visually [33]. Since in both experiments sensory areas responded differentially to sequence positions (Fig 2A) linear classification analysis can be used to predict the position of the items significantly above chance (Fig 2B). However, in both cases the signal changes were uniform across all voxels in the anatomical region suggesting not a dedicated positional code, but sensory adaptation or change in measurement noise. Sensory adaptation thus serves as a clear example how a monotonically changing signal can be read out by an experimenter as a positional code.

thumbnail
Fig 2. Sensory adaptation in the sensory cortex and decoding order position.

(A) Uniform signal change over 3-item sequences in sensory brain areas averaged across participants. Data from visual regions V1, V2, pericalcarine, and lateraloccipital regions is from [33]. Data from auditory areas Heschl’s gyrus (HG) and superior temporal sulcus (STS) is from [32]. (B) Distribution of average linear classification accuracy values of item position in V1 region across participant’s from [33]. Bar charts display the average classification accuracy across participants by comparing the known positions (labels) to the predictions made by the classification algorithm. Bars show the proportion of predicted values for each position. Correct classifications are represented with a darker bar. Error bars show the standard error of the mean. The red line depicts the chance level classification accuracy 1/3.

https://doi.org/10.1371/journal.pone.0176585.g002

Retinotopic activation

In the example above (Sensory adaptation) the population of neural units (sensory cortex) responded uniformly to sequentially presented stimuli. Next we present a case where neural units within the population respond differentially across the sequence. We use fMRI data from a visual sequence processing task to show that the response in the primary visual cortex can be used to predict the position of the item in the sequence. However, this is possible not because of any positional code but because of task-selective voxels in the visual cortex.

In [33] participants had to attend a sequence of visually presented images followed by a manual response indicating the order of the items. Importantly, all images were controlled for luminance and cropped to ensure that each image appeared in a similar retinal area: all stimuli subtended a 6° visual angle around the fixation point in order to elicit an approximately foveal retinotopic representation. As a result, all sequence items elicited approximately similar retinotopic response in the foveal area of the primary visual cortex.

The authors observed that the activation of the retinotopically driven voxels was correlated with the relative suppression of the voxels outside of the retinotopically activated areas (Fig 3A). Such suppression has been observed as a function of stimulus location in the visual field [48] and attention [49, 50]. Importantly, the amount of activation and suppression changed across sequence positions. Since the sequence items were presented in immediate succession, the extent of retinotopic suppression and activation varied as a function of item’s position in the sequence. Fig 3B shows data from a single participant’s V1, where voxels are split into two groups: retinotopically activated (red-yellow on Fig 3A) and suppressed (blue-cyan on Fig 3A) represented by red and blue lines. As the activation and suppression of two different sets of voxels changes across positions, a linear classification algorithm can use the difference between activated and suppressed voxels, or the difference between the red and blue lines on Fig 3B, to reliably predict the item’s position.

thumbnail
Fig 3. Interference between task phases: Retinotopic suppression.

(A) Activation and suppression in V1 averaged across all stimuli for a single participant. The activated voxels (yellow, p < 0.001) mark the foveal part of the visual cortex driven by the stimuli (presented at 6° visual angle). (B) Peristimulus time histogram of sequence presentation of two groups of voxels from a single participant’s V1. The black line denotes the average of the voxels activated by the stimuli and the red line denotes the average of the voxels suppressed by the stimuli. Dashed vertical lines indicate the time bins where sequence items were presented.

https://doi.org/10.1371/journal.pone.0176585.g003

This can be further illustrated when linear discriminant analysis (LDA) class boundaries based on item position are plotted with following sets of voxels from V1:

  1. All voxels (including both retinotopically activated and suppressed voxels)
  2. Only activated voxels (p < 0.01)
  3. Only suppressed voxels (p < 0.01)

LDA shows that the linear classifier is only able to reliably predict the position of the item when both activated and suppressed voxels in the brain region are included (Fig 4, top row). The classification is at chance level if only one set of voxels are used (Fig 4, row 2-3).

thumbnail
Fig 4. LDA of item position in V1 using different subsets of voxels.

Top row: all voxels from V1; middle row: only retinotopically activated voxels from V1; bottom row: only retinotopically suppressed voxels from V1. Left column: Bar charts display the average classification accuracy across participants by comparing the known positions (labels) to the predictions made by the classification algorithm. Bars show the proportion of predicted values for each position. Correct classifications are represented with a darker bar. Error bars show the standard error of the mean. The red line depicts the chance level classification accuracy 1/3.Right column: LDA between-class boundaries based on two voxels from the set. Data from [33].

https://doi.org/10.1371/journal.pone.0176585.g004

Next we carry out a simulation of sequentially generated fMRI data to explore whether both uniformly and differentially proceeding collinear processes could be controlled for when trying to extract a positional code.

Simulation of collinear processes

Here we simulate two types of position-collinear processes which can serve as a positional read-out. In the first case the brain area responds uniformly along the sequence (e.g. sensory adaptation) and in the second case units within the population respond differentially. We show that in the first case we can make reasonable a priori assumptions about the nature of the positional code and hence remove a uniform signal. However, when the population responds differentially to sequence positions there are no prior criteria to distinguish positional read-out from a positional code.

The MATLAB/Octave code for the simulated data and plots is freely available at https://github.com/kkalm/poscode.

Uniformly changing signal across sequence positions

Here we model sensory adaptation in a simple sequence processing task as an example of a uniformly changing position-collinear process. We show how human fMRI data obtained with the same task fits the simulation results. We also propose a data pre-processing step—de-meaning of neural responses—as a tool to eliminate univariate signal collinear to the positional code.

Throughout the simulations we use the term ‘brain region’ for a population of neural units and the term ‘voxels’ for units themselves. This makes the terminology compatible with the experimental data presented from human fMRI experiments.

Representation of sequence items in a brain region.

As a baseline condition we simulate the case where the only information stored in a brain region is item information (without any positional code) and where there is no position-collinear information such as decay or interference. We simulate a sensory brain region of n = 20 voxels which encodes identities for three different items i as independent samples from the uniform distribution (Fig 5A): (1) A brain region’s response Y to the item i will be the item pattern ri plus some noise sampled from n-dimensional Gaussian distribution with a zero mean. (2) To model a noisy average of these patterns we simulate an experiment where those three items are presented in different order as sequences for 6 times. The simulated response matrix Y depicts those 6 sequences with item and position values labelled on the x-axis (Fig 5B).

thumbnail
Fig 5. Simulated responses to items.

(A) Item patterns over 20 voxels. (B) Six sequences as permutations of three items. Item codes are displayed on the top of x-axis, position codes at the bottom.

https://doi.org/10.1371/journal.pone.0176585.g005

As a result, the brain region’s response matrix Y contains noisy representations of item identity but no information about position in the sequence. This can be visualised by plotting the scatter of the data Y and LDA class borders according to item and position labels (Fig 6). It is obvious that patterns Y are only linearly separable in terms of item identity but not position.

thumbnail
Fig 6. The scatter of item patterns and LDA between-class boundaries based on the two most informative voxels.

(A) Item information. (B) Position information.

https://doi.org/10.1371/journal.pone.0176585.g006

Sensory adaptation.

So far we have assumed that item representations are completely independent of sequence position. Next we consider the case where there is a degree of sensory adaptation across the sequence. We simulate sensory adaptation for a brain region as a fixed vector across voxels multiplied by a decreasing function of sequence position, plus a Gaussian noise of fixed magnitude. This means that sensory adaptation will influence all voxels in the brain region similarly. In other words, in terms of a neural response of a brain region, sensory adaptation is a univariate signal decreasing monotonically over sequence positions.

We simulate sensory adaptation for all voxels i.e. voxels respond to stimulus positions {1, 2, 3} by a decreasing vector a = [1, 0.7, 0.4]. The average responses of the voxels can be shown as column-wise means of the response matrix (Fig 7B). As a result, the response of the brain region allows us to linearly separate both item identities and their positions in the sequence (Fig 7C). Re-running this simulation 250 times yields a distribution of average LDA accuracy values (Fig 7D).

thumbnail
Fig 7. Simulation of sensory adaptation.

(A) Voxels’ responses with sensory adaptation. (B) Average responses of voxels as column-wise means of the response matrix. (C) LDA between-class boundaries based on the two most informative voxels.(D) Distribution of average LDA accuracy values (based on 250 simulations).

https://doi.org/10.1371/journal.pone.0176585.g007

Eliminating uniform signal by de-meaning.

If we assume that any collinear process to sequence position affects all voxels in the brain region uniformly then simple de-meaning of the response matrix will eliminate any univariate signal from the data.

Here we z-score the response matrix before classification so that column-wise averages equal zero and values of the matrix correspond to z-scores based on the column mean (Fig 8). Carrying out LDA as before shows that the resulting average classification accuracy is at chance level as z-scoring the response matrix removes effects common to all voxels. Similarly, when z-scoring was applied to the fMRI data above (see Sensory adaptation), positional effects were no longer significant.

thumbnail
Fig 8. Simulation of sensory adaptation, z-scoring.

(A) Voxels’ responses with sensory adaptation.; z-scored. (B) Average responses of voxels as column-wise means of the response matrix.

https://doi.org/10.1371/journal.pone.0176585.g008

However, is is also possible that the sensory brain region actually contains a positional code. It is clear that in order to survive a de-meaning process a dedicated positional code must not be uniform across voxels. De-meaning process cannot affect a multivariate positional signal which affects voxels differentially. We can model each voxel’s position preference T as a Gaussian likelihood function over the position values of the stimuli: i.e. each voxel responds most to a single position and less to adjacent positions: , (note that alternative tuning distributions are also feasible, see the simulation code for examples). Next, we add sensory adaptation (Fig 9B), Gaussian noise (Fig 9C), z-score the data (Fig 9D), and carry out LDA, as above. The resulting average classification accuracy will be close to 100%: since z-scoring does not affect voxel pattern similarity, the positional code is used by the linear classifier to successfully distinguish between order positions.

thumbnail
Fig 9. Simulation of sensory adaptation.

(A) Voxel response matrix based on positional preferences. (B) Voxel response matrix: positional preferences + sensory adaptation. (C) Voxel response matrix: positional preferences + sensory adaptation + Gaussian noise. (D) Voxel response matrix z-scored. Column-wise means are zero.

https://doi.org/10.1371/journal.pone.0176585.g009

However, any differential response within the brain region to sequence positions—such as the retinotopic activation example above—will similarly remain unaffected by z-scoring. As a result we can use de-meaning only to remove uniform effects from the brain region’s response.

Summary of position-collinear effects

A number of cognitive processes take place while stimuli are processed in a sequence. Importantly, several of them—time, memory load, sensory adaptation—will be collinear to any signal tracking the position of items in a sequence.

We showed that uniform position-collinear processes—such as sensory adaptation– can be subtracted from neural responses by a de-meaning technique such as z-scoring. Importantly, this relies on an assumption that such processes will influence all units uniformly in a neural population. However, if individual voxels within a brain region respond differentially—such as in the case of retinotopic activation—the neural response becomes indistinguishable from a dedicated positional code.

Positional code from interference

A positional ‘read-out’ without a dedicated positional code can also arise from interference between sequentially presented stimulus representations. Here we use a simulation to show that a model of sequence representation which only includes item codes and no dedicated positional code can elicit positional effects given some interference between item codes. To illustrate this, imagine a brain region where the representations of successive items are overlayed on top of each other. Each successive item elicits a neural pattern that is a mixture of its own representation and a decaying representation of the preceding items. Such superimposed items could be linearly separable in terms of their positions alone without the need of any explicit representation of position.

Here we look at two cases of interference between item codes—additive and proportional interference– and how both can lead to position-like codes. The item representations are modelled exactly as above in Representation of sequence items in a brain region.

Additive interference

Interference between representations can occur when the state of the memory is not completely wiped clean every time a new stimulus arrives. Instead, the new state of the memory might be a mixture of the new stimulus and the previous state of the memory. Here we assume that at sequence position p the response of the brain region Y equals to the item pattern ri plus some residual activity from the previous state of the brain region: (3) where p is the position of the item in the sequence and π is the mixing coefficient which determines the proportion of the residual activity. Here π declines with a constant rate over previous states of Y so that: (4) where β is the rate parameter of the decreasing mixing coefficient π, and the initial value of π0 = 1. Setting the initial value of π to 1 ensures that the current item pattern is always represented in full. To illustrate this mechanism consider two different β values and how they affect interference in a 3-item sequence ‘CBA’: It is clear that the value of the β parameter determines the amount of interference from previous items: when β = 0 there is no interference, and when β > 1 the activity from previous items contributes more to the current activity pattern yp than the current item pattern ri.

Importantly, with each arriving item the overall activity of the brain region, as defined by the vector sum of yp, increases, since some of the previous response is added to the new response. In other words, additive interference as defined above (Eq (3)) guarantees that: Similar increase in brain activity as a function of the number of sequentially presented items has been observed in several neuroimaging studies of short-term memory [46, 47].

Additive interference enables position decoding.

If we simulate additive interference as described above then despite the brain region only encoding item identity information we can linearly separate patterns Y in terms of their position because the total activity increases as a function of position. The effect of additive residual activity on sequence positions can be shown by plotting the positional means before and after interference transform (Fig 10).

thumbnail
Fig 10. The transformation of response values for two voxels as a result of interference (β = 0.5).

Small circular markers depict response patterns, larger circular markers depict pattern means. Empty markers depict the original patterns and means, filled markers depict the data after simulating the interference process. Solid lines depict the movement of class means as a result of interference. (A) Item information. (B) Position information.

https://doi.org/10.1371/journal.pone.0176585.g010

Note that patterns pertaining to the first positions in the sequence (black markers on Fig 11B) have not moved since there is no interference for the first items in the sequence from previous items. The position-wise transformation of the response patterns allows to separate them linearly using both item and position labels (Fig 11).

thumbnail
Fig 11. LDA between-class boundaries for two voxels, interference β = 0.5.

(A) Item information. (B) Position information.

https://doi.org/10.1371/journal.pone.0176585.g011

We can now decode the position of the items significantly above chance because item position correlates with the amount of response in the simulated brain region. Plotting the classification accuracy of both item and position as a function of interference (β parameter value, Eq (4)) we can see that even with relatively small β values positional decoding becomes significantly greater than chance whilst it is always possible to decode item identity above chance (Fig 12).

thumbnail
Fig 12. Linear classification accuracy of item identity (black) and position (red) as a function of additive interference (as represented by the β parameter, Eq (4)).

The red dotted line shows chance level classification accuracy. Error bars depict SEM based on 1,000 simulations of the interference process with fixed parameter values.

https://doi.org/10.1371/journal.pone.0176585.g012

Positional pattern similarity decreases as a function of lag.

Interference between item representations results in a change in pattern similarity across sequence positions. Specifically, between-position pattern similarity decreases as the distance between positions (lag) increases. In other words, pattern similarity is significantly higher across items that shared the same temporal position information than between items that are 1 or more positions apart (Fig 13). For the purposes of creating more positions the following plot (Fig 13) displays data generated exactly as above but with 5-item sequences instead of three.

thumbnail
Fig 13. Positional pattern similarity decreases as a function of lag.

Similarity matrix on the left shows average positional pattern similarity, as measured by Pearson’s ρ, based on additive interference with β = 0.8. Plot on the right visualises this similarity as a function of positional lag. The red line depicts a statistically significant negative slope over positional lag (p < 0.05).

https://doi.org/10.1371/journal.pone.0176585.g013

Such an effect of positional pattern similarity has be observed in a number of animal and human studies [28, 29, 51, 52] and interpreted as a signature of positional code. The size of the lag effect can be measured as the magnitude of the negative slope over lag values as depicted on Fig 13 (right). Since positional effects are here solely caused by the interference mechanism it follows that the size of the lag effect correlates with the β parameter, which determines the extent of residual activity from the previous item. Fig 14 shows how the change in the lag effect as a function of additive interference.

thumbnail
Fig 14. The size of the positional lag effect as a function of additive interference (β).

Error bars depict SEM based on 1,000 simulations of the interference process with fixed parameter values.

https://doi.org/10.1371/journal.pone.0176585.g014

Proportional interference

Pure additive interference is implausible since it presumes unlimited growth of the response in the brain region. We can cap the total response in the brain region (∑yp) by normalising the response pattern every time a new item is presented. The easiest way to do this is to change the role β from the amount of residual activity to the proportion of residual activity. This requires a single change to the interference mechanism (Eq (3)) so that now we also weigh the current item representation ri, but with 1 − π: (5) Although the mixing coefficient π is here calculated exactly as before (Eq (4)): πp = π0βp−1) its meaning has changed. Whereas previously β represented the amount of interference from the previously presented item, now β determines the proportion of the previous item pattern yp−1 in the current item pattern yp. If we set β = 0.2, the representation of a four-item sequence A, B, C, D would evolve as follows: Though the mechanism of interference here is the same as in the previous simulation, we no longer allow the response of the brain region (∑yp) to grow as the sequence proceeds. In other words, we have eliminated any univariate signal correlated with position. Consequently, linear decoding of response patterns based on position is not significantly different from chance any more (Fig 15A, red line). However, the positional lag effect remains since it is based on pattern similarity (as measured by Pearson’s ρ) which is insensitive to class means (Fig 15B).

thumbnail
Fig 15. Classification accuracy and positional similarity as simulated by the proportional interference mechanism.

Error bars depict SEM based on 1,000 simulations of the interference process. Notice that β values on the x-axis have been approximately halved since the parameter now indicated the proportion of residual activity. (A) Linear classification accuracy of item identity (black) and position (red) as a function of proportional interference (β). The red dotted line shows chance level classification accuracy. (B) The size of the positional lag effect as a function of proportional interference (β).

https://doi.org/10.1371/journal.pone.0176585.g015

In sum, de-meaning the neural response patterns only subtracts univariate effects of between-item interference. Pattern similarity effects of interference, such as the positional lag effect, still remain. It follows therefore that the positional lag effect alone is not a sufficient evidence for a neural positional code and additional statistical tests, such as classification analysis, are required.

Summary of item interference effects

Even in the absence of any true positional code, if the encoding of item information is based on overlaying item representations in a non-additive fashion this can potentially masquerade as a positional code. Depending on the magnitude of interference both position decoding and positional lag effects can be successfully simulated. Positional decoding is possible when residual activity from previous items is not capped and the brain region’s mean response grows with sequence position. When the activity patterns are normalised so that the mean response stays the same then only the positional lag effect remains.

Other sources of interference

The mechanism of interference, as described above in the context of item codes, can be similarly applied to other variables of the experimental design. In fact, as outlined in the Introduction, any fixed parameter of the experimental design is collinear with positional effects. Next we briefly discuss how position-like codes emerge as a result of interference between task phases and as a result of temporally convolved measurement.

Interference between task phases

One of the most common tasks used in studying sequence representation is the serial recall task (Fig 16). In the serial recall task presentation of a sequence of items is usually followed by a response phase requiring the participant to recall the sequence. Importantly, the temporal order between task phases themselves is always fixed: recall must necessarily follow presentation, rest always occurs between the trials etc. As a result, the positional structure of the presented sequence in the task is collinear with the structure of the task itself. For example, in the serial recall task the last item in the sequence is always followed by the recall phase. Similarly, the first item in the sequence is always preceded by recall on the previous trial. As a result we can reliably predict the position of an item in the sequence based on its adjacency to different task phases.

We can model interference between task phases by simulating a response of 20 voxels as above, but during a single trial of a serial recall task. The task has two phases occurring in fixed order: presentation and recall (Fig 16). We assume that some voxels in the brain region are selective for the presentation and some for the recall phase. This selectivity can be described as voxels’ likelihood to respond given a task phase. If there is no interference between task phases the response of phase-selective voxels is independent at any stage of the task: the previous phase of the task does not alter the voxels’ activity at current stage (Fig 17A). However, if we implement additive interference as described above then the extent of the response of phase-selective voxels becomes collinear with item position in the sequence (Fig 17B). Importantly, no item codes are necessary here, just sensitivity to task phases suffices. Due to interference we can now linearly separate the response patterns in terms of their sequence position because the total response changes as a function of task phase (Fig 17B). In every other aspect the mechanism is the same as described in Positional code from interference above.

thumbnail
Fig 17. The average simulated activity of two sets of voxels, each sensitive either to the presentation or recall phase of the task.

In this hypothetical task a presentation of three items in a sequence is followed by recall of three items. (A) Without interference. (B) Additive interference.

https://doi.org/10.1371/journal.pone.0176585.g017

Neurons’ or voxels’ sensitivity to a specific task phase is common, since in most experiments perceptual information is impossible to balance across task phases. For example, the presentation phase in serial recall task commonly uses a different stimulus modality (visual or auditory) than the following recall phase (manual or verbal recall, see Table 1). As a result, large patches of the cortex are only engaged during a specific phase of the task creating conditions described above.

Interference from measurement: Functional MRI

So far we have described interference mechanisms arising between neural representations. However, equally importantly, interference between representations can result from noisy measurement. Similarly to representational interference, this can lead to positional effects which are spurious.

Functional MRI measures neural activity by detecting changes in the concentration of oxyhemoglobin and deoxyhemoglobin in neural tissue (BOLD signal). The relationship between a neural event and the corresponding BOLD signal can be described by a haemodynamic response function (HRF). Importantly, the HRF is non-linear and spread out over several seconds (Fig 18A), meaning that the BOLD signal corresponding to temporally adjacent events, such as items in a sequence or task phases, will always contain a response elicited by events preceding the event of interest (Fig 18B). This creates conditions similar to between-item and task phase interference described above—only this time there is no need for cognitive or representational interference. The temporal overlap in the BOLD signal will result in interference between measured item or phase representations even if the neural representations are independent of each other.

thumbnail
Fig 18. Temporal interference in fMRI.

(A) The haemodynamic response function (HRF) with the vertical line representing the corresponding neural event. (B) Temporal interference between two adjacent events: black and red lines.

https://doi.org/10.1371/journal.pone.0176585.g018

Discussion

A major methodological barrier to studying the neural representation of positional code is that in any sequence processing task items in different position necessarily differ on other dimensions too. In this paper we used simulations and experimental data to show how both position-collinear experimental variables, noisy measurement, and interference between sequence items can lead to positional read-out in the absence of a dedicated positional code. This raises two important questions: (1) is it important to distinguish between a positional read-out and a ‘true’ positional code; and (2) what steps can be taken to delineate those in experimental data.

Positional read-out vs. dedicated positional code

Most models of neural sequence representation assume the existence of a dedicated positional code in the brain [1, 5355]. However, since several cognitive processes (e.g. memory load, sensory adaptation) are collinear with any positional signal, a question arises whether those collinear processes could be used as a positional signal not just by the experimenter but also by the brain. We argue that a positional read-out from either simple position-collinear processes or between-item interference is not sufficient to support the storage and recall of a sequence.

Positional read-out from collinear processes is not sufficient for recalling a sequence.

Recalling items in a sequence requires reinstating their order during recall. This problem is solved by positional models of sequence processing by associating each sequence item with its position during encoding and retrieving the order of items during recall by reinstating each positional code, which then cues the associated item (Fig 1B, e.g.: [5356]). However, it is hard to see how processes such as memory load or sensory adaptation could be used to cue associated items. Although experimenters can decode item position in a sequence based on memory load or sensory adaptation it is hard to see how “cortex as receiver” can use those neural signals to represent position and guide behaviour. For example, in case of using memory load (or any monotonically changing signal) as a positional code to cue associated items would require first to reinstate such ‘load’ to cue the corresponding item. However, such interpretation of ‘memory load’, which can be reinstated independently of the amount of items in memory, loses its original meaning and becomes a clumsy re-interpretation of a dedicated positional code. For this reason any effect of sequence processing, such as memory load or sensory adaptation, cannot be inverted into cause that would enable to associate items into sequence.

Positional read-out from between-item interference is not sufficient for recalling a sequence.

We showed that interference between item representations can potentially masquerade as a positional code. This is because noisiness of the item representations changes monotonically over sequence positions as a result of interference. The change in the noise profile can therefore be used by the experimenter to reliably predict the position of the item in the sequence. However, as with simple position-collinear processes, it is hard to see how changes in the signal-to-noise ratio can be used by the brain to guide sequence recall. The main consequence of interference via overlaying item representations is that the later items in the sequence are noisier than the early ones. This contradicts the well-established recency effect in sequence recall, where last items in the sequence are more accurately recalled than the ones in the middle (see [57], for a review). Secondly, using the noisiness of item representations as a positional code to cue associated items conflates the cause and effect relationship in sequence processing, as discussed above. The noisiness of the items would need to be reinstated independently of items themselves, thus necessitating the recoding of the noise levels into a dedicated positional signal.

Methods to dissociate between positional read-out and dedicated positional code

It is not possible to devise a task where the positional signal is orthogonal to other experimental variables since cognitive processes collinear to the positional code will always arise whenever stimuli are presented in sequence. However, the vast majority of previous studies on the positional code (Table 1) do not acknowledge the possibility of the ‘contamination’ of the positional code or take any measures to control for it.

Two assumptions are required to distinguish between a positional read-out and a ‘true’ positional code. First, position-collinear processes like memory load or sensory adaptation will uniformly affect all neural units engaged in encoding the item representations. This assumption is relatively uncontroversial if we presume that such processes are the result (and not source) of sequence processing. Second, we need to assume that a dedicated positional code is reflected in the position-sensitivity within a population of neural units. In other words, units respond selectively to sequence positions based on some tuning function. Under such conditions simple de-meaning (e.g z-scoring) of the neural data with respect to experimental condition (item position) will eliminate any univariate signal from the data including any univariate positional read-outs (see Eliminating uniform signal by de-meaning).

However, we also showed that between-item interference can result in pattern similarity effects which masquerade as positional signal in the form of the lag effect (see Positional pattern similarity decreases as a function of lag and Proportional interference). Effects of pattern similarity are independent of signal amplitude and hence invariant to de-meaning. As a result, the effect of positional lag which has been used in several previous studies of positional code [28, 29, 51, 52] cannot be taken as a proof of neural positional code without ruling out between-item interference first. We show that this can be achieved by using linear classification analyses on the de-meaned neural responses.

Besides cognitive interference– such as based on overlaying item representations—positional read-out can result in noisy measurement, such as the temporal interference inherent in fMRI. In other words, any fMRI signal pertaining to successively presented sequence items will include a positional signal based on measurement error, even if we assume no interference between the neural representations of items themselves. As a result, the positional lag effect alone should never be used in fMRI studies as an indicator of neural positional representation. In fMRI studies sequentially presented stimuli will always be collinearly dependent on each other because of the inherent temporal lag in the BOLD signal. As a solution, whole-sequence data can be used to extract positional information using the representational similarity analysis [32, 58].

Conclusions

In this paper we have explored two types of processes that could enable an experimenter to read out a positional ‘code’ in the absence of a dedicated positional code. First, we show that with any sequence processing task there are experimental variables collinear with the positional signal (e.g. time, memory load, etc.) which can serve as a positional code. Second, we show how interference between item representations, task phases, and measurement modalities can also lead to a similar positional read-outs.

We argue that it is important to distinguish between a positional read-out and a dedicated positional code, since only the latter has been shown to be compatible with experimental data. Furthermore, we argue that such collinear processes which enable positional read-out are the result of sequence representation not cause, and hence would not be able to even theoretically support sequence retrieval. Finally, we suggest practical steps in data analysis to distinguish between a positional read-out and a code. Furthermore, this paper shows that many results from behavioural and neural experiments studying the positional code must be treated with caution.

Author Contributions

  1. Conceptualization: KK DN.
  2. Data curation: KK.
  3. Formal analysis: KK.
  4. Funding acquisition: DN.
  5. Investigation: KK.
  6. Methodology: KK.
  7. Project administration: KK.
  8. Resources: DN.
  9. Software: KK.
  10. Supervision: DN.
  11. Visualization: KK.
  12. Writing – original draft: KK.
  13. Writing – review & editing: KK DN.

References

  1. 1. Henson R, Burgess N. Representations of Serial Order. The Quarterly Journal of Experimental Psychology. 1997;(789284387).
  2. 2. Page M, Norris D. The Primacy Model: A New Model of Immediate Serial Recall. Psychological Review. 1998;105(4):761–781. pmid:9830378
  3. 3. Nakajima T, Hosaka R, Mushiake H, Tanji J. Covert representation of second-next movement in the pre-supplementary motor area of monkeys. Journal of neurophysiology. 2009;101(4):1883–9. pmid:19164110
  4. 4. Averbeck B, Crowe D, Chafee M, Georgopoulos A. Neural activity in prefrontal cortex during copying geometrical shapes. II. Decoding shape segments from neural ensembles. Experimental brain research. 2003;150(2):142–53. pmid:12669171
  5. 5. Inoue M, Mikami A. Prefrontal activity during serial probe reproduction task: encoding, mnemonic, and retrieval processes. Journal of neurophysiology. 2006;95(2):1008–41. pmid:16207786
  6. 6. Naya Y, Suzuki W. Integrating what and when across the primate medial temporal lobe. Science. 2011;333(6043):773–6. pmid:21817056
  7. 7. Berdyyeva T, Olson C. Rank signals in four areas of macaque frontal cortex during selection of actions and objects in serial order. Journal of neurophysiology. 2010;104(1):141–59. pmid:20445037
  8. 8. Mankin E, Sparks F, Slayyeh B, Sutherland R, Leutgeb S, Leutgeb J. Neuronal code for extended time in the hippocampus. Proceedings of the National Academy of Sciences of the United States of America. 2012;109(47):19462–7. pmid:23132944
  9. 9. Manns J, Howard M, Eichenbaum H. Gradual changes in hippocampal activity support remembering the order of events. Neuron. 2007;56(3):530–40. pmid:17988635
  10. 10. Pastalkova E, Itskov V, Amarasingham A, Buzsáki G. Internally generated cell assembly sequences in the rat hippocampus. Science. 2008;321(5894):1322–7. pmid:18772431
  11. 11. MacDonald C, Lepage K, Eden U, Eichenbaum H. Hippocampal “Time Cells” Bridge the Gap in Memory for Discontiguous Events. Neuron. 2011;71(4):737–749. pmid:21867888
  12. 12. MacDonald C, Carrow S, Place R, Eichenbaum H. Distinct Hippocampal Time Cell Sequences Represent Odor Memories in Immobilized Rats. Journal of Neuroscience. 2013;33(36):14607–14616. pmid:24005311
  13. 13. Allen T, Salz D, McKenzie S, Fortin N. Nonspatial Sequence Coding in CA1 Neurons. Journal of Neuroscience. 2016;36(5):1547–1563. pmid:26843637
  14. 14. Amiez C, Petrides M. Selective involvement of the mid-dorsolateral prefrontal cortex in the coding of the serial order of visual stimuli in working memory. Proceedings of the National Academy of Sciences of the United States of America. 2007;104(34):13786–91. pmid:17699624
  15. 15. Averbeck B, Sohn J, Lee D. Activity in prefrontal cortex during dynamic selection of action sequences. Nature neuroscience. 2006;9(2):276–82. pmid:16429134
  16. 16. Averbeck B, Lee D. Prefrontal neural correlates of memory for sequences. The Journal of Neuroscience. 2007;27(9):2204–11. pmid:17329417
  17. 17. Barone P, Joseph J. Prefrontal cortex and spatial sequencing in macaque monkey. Experimental brain research. 1989;78(3):447–64. pmid:2612591
  18. 18. Berdyyeva T, Olson C. Relation of ordinal position signals to the expectation of reward and passage of time in four areas of the macaque frontal cortex. Journal of Neurophysiology. 2011;105(5):2547–2559. pmid:21389312
  19. 19. Carpenter A. Motor Cortical Encoding of Serial Order in a Context-Recall Task. Science. 1999;283(5408):1752–1757. pmid:10073944
  20. 20. Crowe D, Zarco W, Bartolo R, Merchant H. Dynamic Representation of the Temporal and Sequential Structure of Rhythmic Movements in the Primate Medial Premotor Cortex. Journal of Neuroscience. 2014;34(36):11972–11983. pmid:25186744
  21. 21. DuBrow S, Davachi L. Temporal Memory Is Shaped by Encoding Stability and Intervening Item Reactivation. Journal of Neuroscience. 2014;34(42):13998–14005. pmid:25319696
  22. 22. DuBrow S, Davachi L. Temporal binding within and across events. Neurobiology of Learning and Memory. 2016; pmid:27422018
  23. 23. Fujii N, Graybiel A. Time-varying covariance of neural activities recorded in striatum and frontal cortex as monkeys perform sequential-saccade tasks. Proceedings of the National Academy of Sciences. 2005;102(25):9032–9037. pmid:15956185
  24. 24. Gelfand J, Bookheimer S. Dissociating neural mechanisms of temporal sequencing and processing phonemes. Neuron. 2003;38(5):831–42. pmid:12797966
  25. 25. Nieder A, Diester I, Tudusciuc O. Temporal and Spatial Enumeration Processes in the Primate Parietal Cortex. Science. 2006;313(5792):1431–1435. pmid:16960005
  26. 26. Ginther M, Walsh D, Ramus S. Hippocampal Neurons Encode Different Episodes in an Overlapping Sequence of Odors Task. Journal of Neuroscience. 2011;31(7):2706–2711. pmid:21325539
  27. 27. Heusser A, Poeppel D, Ezzyat Y, Davachi L. Episodic sequence memory is supported by a theta-gamma phase code. Nature neuroscience. 2016;19(August):In Revision. pmid:27571010
  28. 28. Hsieh L, Gruber M, Jenkins L, Ranganath C. Hippocampal activity patterns carry information about objects in temporal context. Neuron. 2014;81(5):1165–78. pmid:24607234
  29. 29. Hsieh L, Ranganath C. Cortical and subcortical contributions to sequence retrieval: Schematic coding of temporal context in the neocortical recollection network. NeuroImage. 2015;121:78–90. pmid:26209802
  30. 30. Hyde R, Strowbridge B. Mnemonic representations of transient stimuli and temporal sequences in the rodent hippocampus in vitro. Nature Neuroscience. 2012;15(10):1430–1438. pmid:22960934
  31. 31. Isoda M, Tanji J. Participation of the primate presupplementary motor area in sequencing multiple saccades. Journal of neurophysiology. 2004;92(1):653–9. pmid:14985413
  32. 32. Kalm K, Norris D. The representation of order information in auditory-verbal short-term memory. The Journal of Neuroscience. 2014;34:6879–86. pmid:24828642
  33. 33. Kalm K, Norris D. A shared representation of order between encoding and recognition in visual short-term memory. NeuroImage. 2017;. pmid:28450141
  34. 34. Kraus B, Robinson R, White J, Eichenbaum H, Hasselmo M. Hippocampal“Time Cells”: Time versus Path Integration. Neuron. 2013; p. 1–12. pmid:23707613
  35. 35. Lehn H, Steffenach H, van Strien N, Veltman D, Witter M, Håberg A. A specific role of the human hippocampus in recall of temporal sequences. The Journal of Neuroscience. 2009;29(11):3475–84. pmid:19295153
  36. 36. Manns J, Howard M, Eichenbaum H. Gradual changes in hippocampal activity support remembering the order of events. Neuron. 2007;56(3):530–40. pmid:17988635
  37. 37. Merchant H, Pérez O, Zarco W, Gámez J. Interval tuning in the primate medial premotor cortex as a general timing mechanism. The Journal of neuroscience: the official journal of the Society for Neuroscience. 2013;33(21):9082–96. pmid:23699519
  38. 38. Nieder A. Supramodal numerosity selectivity of neurons in primate prefrontal and posterior parietal cortices. Proceedings of the National Academy of Sciences. 2012;109(29):11860–11865. pmid:22761312
  39. 39. Ninokura Y, Mushiake H, Tanji J. Integration of temporal order and object information in the monkey lateral prefrontal cortex. J Neurophysiol. 2004;91(1):555–560. pmid:12968014
  40. 40. Petrides M. Functional Specialization within the Dorsolateral Frontal Cortex for Serial Order Memory. Proceedings of the Royal Society B: Biological Sciences. 1991;246(1317):299–306. pmid:1686096
  41. 41. Rangel L, Alexander S, Aimone J, Wiles J, Gage F, Chiba A, et al. Temporally selective contextual encoding in the dentate gyrus of the hippocampus. Nature communications. 2014;5:3181. pmid:24518986
  42. 42. De-Wit L, Alexander D, Ekroll V, Wagemans J. Is neuroimaging measuring information in the brain? Psychonomic Bulletin & Review. 2016; pmid:26833316
  43. 43. Henson R, Rugg M. Neural response suppression, haemodynamic repetition effects, and behavioural priming. Neuropsychologia. 2003;41(3):263–270. pmid:12457752
  44. 44. Summerfield C, Trittschuh E, Monti J, Mesulam M, Egner T. Neural repetition suppression reflects fulfilled perceptual expectations. Nature Neuroscience. 2008;11(9):1004–1006. pmid:19160497
  45. 45. Larsson J, Smith A. FMRI repetition suppression: Neuronal adaptation or stimulus expectation? Cerebral Cortex. 2012;22(3):567–576. pmid:21690262
  46. 46. Rottschy C, Langner R, Dogan I, Reetz K, Laird A, Schulz J, et al. Modelling neural correlates of working memory: A coordinate-based meta-analysis. NeuroImage. 2012;60(1):830–846. pmid:22178808
  47. 47. Wager T, Smith E. Neuroimaging studies of working memory: A meta-analysis. Behavioral Neuroscience. 2003;3(4):255–274. pmid:15040547
  48. 48. Maier A, Wilke M, Aura C, Zhu C, Ye F, Leopold D. Divergence of fMRI and neural signals in V1 during perceptual suppression in the awake monkey. Nature Neuroscience. 2008;11(10):1193–1200. pmid:18711393
  49. 49. Gouws D, Alvarez I, Watson D, Uesaki M, Rogers J, Morland B. On the Role of Suppression in Spatial Attention: Evidence from Negative BOLD in Human Subcortical and Cortical Structures. Journal of Neuroscience. 2014;34(31):10347–10360. pmid:25080595
  50. 50. Smith T, Singh K, Greenlee M. Attentional suppression of activity in the human visual cortex. Neuroreport. 2000;11(2):271–277. pmid:10674469
  51. 51. Devito L, Eichenbaum H. Memory for the order of events in specific sequences: contributions of the hippocampus and medial prefrontal cortex. The Journal of Neuroscience. 2011;31(9):3169–75. pmid:21368028
  52. 52. Fortin N, Agster K, Eichenbaum H. Critical role of the hippocampus in memory for sequences of events. Nature Neuroscience. 2002;5(5):458–62. pmid:11976705
  53. 53. Howard M, Kahana M. A Distributed Representation of Temporal Context. Journal of Mathematical Psychology. 2002;46:269–299.
  54. 54. Burgess N, Hitch G. A revised model of short-term memory and long-term learning of verbal sequences. Journal of Memory and Language. 2006;55(4):627–652.
  55. 55. Brown G, Preece T, Hulme C. Oscillator-based memory for serial order. Psychological review. 2000;107(1):127–81. pmid:10687405
  56. 56. Lee C, Estes W. Item and Order Information in Short-Term Memory: Evidence for Multilevel Perturbation Processes. Journal of Experimental Psychology: Human Learning and Memory. 1981;7(3):149–169.
  57. 57. Hurlstone M, Hitch G, Baddeley A. Memory for serial order across domains: An overview of the literature and directions for future research. Psychological bulletin. 2014;140(2):339–73. pmid:24079725
  58. 58. Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis—connecting the branches of systems neuroscience. Frontiers in systems neuroscience. 2008;2(November):4. pmid:19104670