Different states of priority recruit different neural representations in visual working memory

We used functional magnetic resonance imaging (fMRI) to investigate the neural codes for representing stimulus information held in different states of priority in working memory. Human participants (male and female) performed delayed recall for 2 oriented gratings that could appear in any of several locations. Priority status was manipulated by a retrocue, such that one became the prioritized memory item (PMI) and another the unprioritized memory item (UMI). Using inverted encoding models (IEMs), we found that, in early visual cortex, the orientation of the UMI was represented in a neural representation that was rotated relative to the PMI. In intraparietal sulcus (IPS), we observed the analogous effect for the representation of the location of the UMI. Taken together, these results provide evidence for a common remapping mechanism that may be responsible for representing stimulus identity and stimulus context with different levels of priority in working memory.


Introduction
Important for understanding the flexible control of behavior [1,2] is understanding working memory: the mental retention of task-relevant information and the ability to use it to guide contextually appropriate actions [3,4]. State-based theoretical models of working memory posit that information can be held at different levels of priority in working memory, with information at the highest level of priority in the focus of attention (FoA) and the remaining information in a variously named state of "activated long-term memory" [5] or "region of direct access" [6].
Much of the empirical support for these models comes from tasks using a "retrocuing" procedure that allows for the controlled study of the back-and-forth switching of priority between memory items. In the dual serial retrocuing (DSR) task, 2 items are initially presented as memoranda, followed by a retrocue that designates one the "prioritized memory item" (PMI; equivalent to the "attended memory item" in previous publications) that will be interrogated by the impending probe. The uncued item cannot be dropped from working memory, however,

Results
Participants were scanned with fMRI while performing a DSR task in which 2 gratings (9 possible orientations) were presented serially, each at one of 9 possible locations, and after an initial Delay1.1, Cue1 indicated which of the two (first or second) would be tested for recall at the end of the ensuing Delay1.2. Following Recall1, Cue2 indicated the grating to be tested at the end of Delay2, with a 50% of probability for each grating (Stay or Switch; Fig 1). Of primary interest was the neural representation of the sample stimuli during Delay1.2, when Cue1 had given one the status of PMI and the other the status of UMI. The neural representation of the orientation and the location of each item were assessed by training 2 IEMs with leave-onerun-out cross-validation. For "PMI-trained IEMs," the data on each trial were labeled according to the identity of the PMI at the time point of interest, and the converse was true for "UMI-trained" IEMs. Results from the PMI-trained IEMs would be the primary focus of this study because PMI-trained models have been shown to be valid and robust for various brain regions, including occipitotemporal cortex, IPS, and FEF [21][22][23].

General behavioral performance
Behavioral responses were analyzed with a 3-factor mixture model [24] that uses maximum likelihood estimation to generate estimates of 1) the probability of responses to the cued item ("responses to target"; pT), 2) the probability of responses incorrectly made to the uncued item ("responses to nontarget"; pN), and 3) the probability of responses that were guesses, as well as 4) a "concentration" parameter that estimates the precision of nonguess responses. The mixture model estimates for Recall1 showed a concentration of 21.9 ± 3.6, pT of 79.8% ± 1.9%, pN of 5.2% ± 1.1%, and guessing of 15

Reconstructions with PMI-trained IEMs Stimulus orientation: Early visual region of interest (ROI).
For orientation reconstruction in early visual cortex, during portions of Delay1.1, both the reconstructions of the PMI and the UMI were significant (both positive slopes; p-values < 0.020) and were not different from each other (p-values > 0.173). The PMI and the UMI started to differ in their reconstruction strength after the onset of Cue1 (presented at 10 s after trial onset). This difference was significant for 16-18 s after trial onset (p-values = 0.013 and 0.025; Fig 2A). These results were consistent with previous findings [8][9][10] demonstrating a clear priority-based modulation of stimulus representations after retrocuing.
Critically, at the final time point of Delay1.2 (18 s), the reconstructions of the orientation of the PMI and of the UMI went in opposite directions: there was a significant reconstruction with a positive slope for the PMI (p = 0.044) and a significant reconstruction with a negative slope for the UMI (p = 0.048; p-values corrected for multiple comparisons for this and all the subsequent analyses on time 18 s; Fig 3A). The result for the UMI is particularly noteworthy because it indicates that information about the unprioritized orientation was maintained in an active state in early visual cortex but in a format that differed from its representation when prioritized.
Stimulus orientation: IPS ROI. The time course of the reconstructions of orientation in IPS showed no convincing evidence of above-chance reconstruction of either the PMI or the UMI during Delay1.1, although there was a suggestion of an influence of priority during Delay1.2 that was qualitatively similar to what was observed in the early visual ROI, with the reconstructions of the PMI and the UMI moving in opposite directions following Cue1. Reconstruction strengths of the PMI and the UMI differed at 16 s after trial onset (p = 0.022), but not at 18 s (p = 0.288; Fig 2B). Furthermore, the positive slope of the IEM reconstruction of the PMI was not statistically different from 0 at either 16 or 18 s (p-values > 0.128), and that of the UMI was significantly negative at 16 s (p = 0.022), but not at 18 s (p = 0.638; Fig 3B).
Stimulus location: Early visual ROI. The location of the PMI could be reconstructed throughout Delay1.1 and Delay1.2 (positive slopes; p-values < 0.001; Fig 4A). Reconstruction of the location of the UMI was also robust with a positive slope during Delay1.1 (pvalues < 0.001), began to decline after the cue, and was no longer different from 0 during the final time point of Delay1.2 (p = 0.091, Fig 5A).
Stimulus location: IPS ROI. In IPS, the reconstruction time course indicated significant representation of the location of both the PMI and the UMI during portions of Delay1.1 (both positive slopes, p-values < 0.026). Priority-based differentiation then developed during Delay1.2, with the representation of the location of the PMI differing from that of the UMI at 16 s (p = 0.027) and 18 s after trial onset (p < 0.001; Fig 4B). Critically, at 18 s, the slope of the reconstruction of the location of the PMI was significantly positive (p = 0.013), and the slope of the reconstruction of the location of the UMI was significantly negative (p = 0.013; Fig 5B). These results indicate an involvement of IPS in maintaining the location context of both prioritized and unprioritized items in working memory.  All results were from PMI-trained IEMs. Red lines represent the PMI, and blue lines represent the UMI. Gray shaded area indicates display of Cue1 (10-10.75 s). Red, blue, and black dots indicate p < 0.05 for significant reconstruction of PMI, significant reconstruction of the UMI, and a significant difference between the two, respectively. All error bars indicate ± 1 SEM. Data are available at osf.io/G4C3N. a.u., arbitrary unit; IEM, inverted encoding model; IPS, intraparietal sulcus; PMI, prioritized memory item; ROI, region of interest; UMI, unprioritized memory item.

Temporal generalization of reconstructions
Finally, it is worthy of note that temporal generalization analysis demonstrated failed generalization between the encoding and the maintenance periods for both orientation (S5 Fig) and location (S6 Fig), suggesting a dynamic change in neural code from the encoding to different maintenance periods in the DSR task (full statistical results can be found in S1-S8 Tables).  Red, blue, and black dots indicate p < 0.05 for significant reconstruction of PMI, significant reconstruction of the UMI, and a significant difference between the two, respectively. All error bars indicate ± 1 SEM. Data are available at osf.io/G4C3N. a.u., arbitrary unit; IEM, inverted encoding model; IPS, intraparietal sulcus; PMI, prioritized memory item; ROI, region of interest; UMI, unprioritized memory item.

Discussion
What neural mechanism underlies the maintenance of working memory content with different priorities? The present results suggest that, when deprioritized, the neural representations of an item's identity and of its context undergo a transformation that we characterize as priority-based remapping. With fMRI, a neural code refers to the systematic set of mappings between unique stimulus values and unique patterns of neural activity, the property that supports multivariate decoding and encoding. The present results, for example, indicate that 9 values of stimulus orientation map to 9 different high-dimensional patterns of activity within early visual cortex. Furthermore, they suggest that when a stimulus transitions to an unprioritized status, this set of mappings rotates such that the individual mappings between stimulus values and neural patterns are now different, but the distance (in orientation) between neural patterns is preserved. Thus, although the item's neural representation has changed, the code underlying the representation of orientation has not changed. For this reason, we characterize the priority-based transformations reported here as examples of remapping, not of recoding. This remapping account is also consistent with the findings of van Loon and colleagues [19], who observed that a classifier trained on a stimulus category when it was the PMI could also decode that category when it was the UMI, even though the pattern of activity of the UMI projected into an opposite region of multidimensional scaling space relative to the PMI. The characteristics underlying priority-based remapping may account for some of the mixed results in previous work on the effects of retrocuing.

Stimulus identity
In this study, IEM of delay-period activity indicated that early visual cortex represents the orientation of grating stimuli when they are being held in the FoA, an observation that is broadly consistent with previous neuroimaging studies of working memory for stimuli defined by lowlevel visual features (for example, [22,[25][26][27]). When a retrocue designated an item a UMI, IEM indicated that its representation in early visual cortex underwent a transformation that corresponded to a rotation of 90˚relative to how it was represented in the FoA. By extension, the present results also lend support to the interpretation of the priority-based transformation reported by van Loon and colleagues [19]. Because they observed a transformation of the posterior inferior temporal representation of objects in their study (cows, dressers, ice skates) to an "opposite" pattern in multidimensional scaling space, the general principle underlying priority-based remapping may be one of transforming the representation of deprioritized information into a format that is complementary to its representation when in the FoA, effectively maximizing the difference between an item when it is a UMI versus when it is a PMI. Furthermore, although the design of the present study did not allow us to assess stimulus representation during Delay2, results from a different study that used a similar procedure suggest that on "switch" trials, the representation of the previously deprioritized item rotates back to its PMI format [28].

Stimulus context
The binding of information about a stimulus to its trial-unique context is fundamental to what it means to hold information in working memory [29,30]. In the present study, the location at which the items held in working memory had been presented (i.e., their context) could be successfully reconstructed in IPS, and the format of these representations also displayed prioritybased remapping. This effect, as measured by IEM, corresponded to a rotation of the mnemonic representation of an item's location by 180˚when it was a UMI relative to when it was in the FoA.
These results are consistent with the idea that a parietal priority map tracks the location context of all items held in visual working memory and that, similar to the neural representation of stimulus identity, the priority-based representational transformation of location context is also implemented via rotational remapping. Interestingly, in the present study, location context was not needed for task performance because items were cued according to order of presentation. These results are consistent with previous reports of an automatic encoding of location information [31], suggesting that the binding between location and identity is an intrinsic process in working memory for objects.

What brain areas support priority-based remapping?
Noteworthy in our results is the fact that the priority-based remapping of stimulus orientation was observed in early visual cortex, but not in IPS, whereas the priority-based remapping of stimulus location was observed in IPS, but not in early visual cortex. Considering orientation, we cannot rule out the possibility that our study may have simply lacked the power to detect the remapping of orientation in IPS. Already in Delay1.1, when both items were in the FoA, there was only weak evidence for the representation of orientation in IPS, and we know from previous work that successful decoding of low-level visual features is less robust in IPS relative to early visual cortex, particularly in conditions, like high memory load, when stimulus information is weaker (for example, [32]). This low-sensitivity account seems much less plausible, however, for the representation of location, which was robust in early visual cortex for the PMI. Two other factors that may be important for understanding our results are domain specificity and task-specific function.
A domain-specificity account would predict that priority-based remapping is predominantly engaged in regions that are necessary for representing the visual feature in question. From this perspective, our results may have been predictable from the facts that damage to occipital areas, but not IPS, can produce apperceptive agnosia [33], and that damage to IPS, but not to early visual areas, can produce disordered spatial cognition [34]. It is noteworthy, in this regard, that van Loon and colleagues [19] observed priority-based remapping of category representations in fusiform gyrus. A functional account would emphasize the content/context distinction. For tasks for which stimulus location serves as context, the representation of location in IPS might undergo priority-based remapping because of this region's role in context binding (for example, [32]). From this perspective, despite the strong representation of retinotopy in early visual cortex, the absence of a functional role for this brain area in context binding may explain the absence of evidence for remapping of the representation of location. It will be interesting, in future work, to pit these 2 accounts against each other with a task in which stimulus location is the to-be-remembered content and orientation the context that is used for cuing priority.

Comparison with previous results
Why did the present study and that of van Loon and colleagues [19] find evidence for active representation of the UMI in stimulus-representing cortex when many previous attempts with the DSR task have been unsuccessful? One important factor is how the classifier/encoding model is trained for probing the representation of the UMI. In previous studies, researchers either used a separate 1-item task to train the classifier [8][9][10] or used a regression model based on the UMI [11]. In the present study and in van Loon and colleagues [19], it was a decoder/encoder based on the PMI that found evidence for active neural representation of the UMI. Decoders trained on a different task or on the UMI may not be able to discriminate representations that are rotations of the PMI.
The trend in the present study-that results with UMI-trained models were less robust than those with PMI-trained models-is also consistent with previous work. For example, Christophel and colleagues [11] could not decode UMI-related information in an ROI comprising V1-V4 (although decoder performance came closer to statistical significance when the analysis was restricted to V3 and V4). In the study from van Loon and colleagues [19], although decoding of the UMI using a UMI-trained classifier was statistically significant, the decoding accuracy was markedly lower in comparison with using a PMI-trained classifier. Together, the 3 sets of results are consistent with the idea that UMI-trained models/classifiers are not as effective as PMI-trained models/classifiers.
A notable inconsistency between the present results and those from a previous study that used the DSR technique [11] is the absence, in this study, of evidence for an active representation of the UMI in IPS, whether with PMI-or UMI-trained IEMs. One factor, mentioned above, is that in our experience, successful decoding of low-level visual features is less robust in IPS relative to early visual cortex (for example, [32]). Perhaps relatedly, it has been suggested that the representation of stimulus identity in IPS is lower resolution and weaker than in early visual cortex (c.f. [11]), factors that could have influenced our results.

Possible alternative accounts
Because the prioritization-related changes that we have documented here manifest as a "flipping" of IEM reconstructions, there are at least 2 alternatives to priority-based remapping that need to be considered: undershoot and inhibition. Undershoot can occur when the drive on activity in a region is removed, such as for early visual cortex with the offset of visual stimulation. In the present study, an undershoot account would hold that activity in voxels representing the UMI return to baseline when the retrocue triggers the shift of attention to the PMI. Importantly, because undershoot is a passive phenomenon, one would expect it to be present for all conditions. In the present results, however, the effects were specific to domain and to brain region: they were observed in early visual cortex for the representation of orientation, but not of location, and they were observed in IPS for location, but not for orientation. Furthermore, because in early visual cortex, the Delay1.1 representation of location was greater in amplitude than was that of orientation, any undershoot in that region would be expected to be greater for location than for orientation.
A second alternative account of the prioritization-related changes reported here is stimulus-specific inhibition or suppression. By this account, if the signal intensity in voxels driving the orientation channel that the UMI is centered on is suppressed, spontaneous activity in voxels that drive off-orientation channels could cause the inversion in channel responses observed in the IEM results. Further computational modeling may be useful for differentiating between different accounts.
It is also important to note, as we consider alternative accounts, that priority-based remapping of active representations is not incompatible with the previously proposed, activity-silent mechanism of retaining the UMI. Because remapping of the active representations of stimulus information was only evident later in the delay period, it is possible that remapping also occurred at the synaptic level before the representation became reactivated. It is also possible that remapping occurred at the activity level with the activity-silent representation unchanged. Therefore, several different mechanisms may work together to support the representation of priority in working memory.

Priority-based remapping in visual working memory
The proposed mechanism of priority-based remapping can be considered at several levels of description. At the level of its neural implementation, priority-based remapping may be accomplished via a systematic reweighting of the weights that map from neuron space into the lower-dimensional space of population-level representation. This is reminiscent of the ordinal reversal of voxel signal intensities between encoding and retention that has been reported in primary auditory cortex for auditory working memory [35]. It is also reminiscent of the rotational dynamics described in mouse auditory cortex by Libby and Buschman [36], a result of a subset of the neurons representing a memory dynamically inverting their selectivity.
At the level of control, priority-based remapping has been modeled as a consequence of competition between prefrontal pointers that activate their corresponding perceptual representations with different levels of priority [18,37]. Whether the effects that we observed in parietal cortex may also reflect the operation of a source of the priority-based control of information held in working memory, as implemented via the priority-sensitive representation of stimulus context, is an important question for future research. Finally, at a theoretical level, why should the features of a stimulus be represented in rotated formats when the stimulus is a UMI versus when it is a PMI? One possible explanation is that such remapping may be an effective way to keep information in an active, accessible state while also accomplishing the simultaneous goals of protecting it from degradation and minimizing the likelihood that it will interfere with the real-time guidance of behavior. Similar ideas have been proposed, for example, as a basis for retaining remembered information in noisy neural networks [37] or the projection of active representations into a null space [38] as a means of maintaining multiple representations of goal states [39].
To conclude, with multivariate IEMs, we observed and quantified changes in the neural representation of stimulus information as a function of attentional prioritization. In early visual cortex, the representation of the orientation of a sample transformed into an opposite pattern when it transitioned from a PMI to a UMI. In IPS, the same was true for the representation of the location at which that item had been presented. These results suggest a mechanism for a priority-based remapping of information when it is held in working memory but outside the FoA.

Ethics statement
This study was approved by the University of Wisconsin-Madison Health Sciences Institutional Review Board (2017-0344) and was conducted according to the principles of the Declaration of Helsinki. Participants provided written informed consent prior to participation.

Participants
An a priori power analysis based on the effect sizes found in a previous experiment [20] indicated that a target sample size of 13 subjects was needed to detect the smallest of the effects predicted by our hypotheses. A total of 14 individuals participated in the study (3 male, average age 21.1 ± 4.5 years), with one subsequently excluded because of excessive head movement. All were recruited from the University of Wisconsin-Madison community. All had normal or corrected-to-normal vision and were neurologically healthy. All participants were monetarily compensated for their participation.

Stimuli and procedure
All stimuli were created and presented using MATLAB (The MathWorks, Natick, MA, USA) and Psychtoolbox 3 extensions [40,41] on a 60-Hz Avotec Silent Vision 6011 projector (Avotec, Stuart, FL, USA) and viewed through a coil-mounted mirror in the MRI scanner. A trackball response pad (Current Designs, Philadelphia, PA, USA) was employed to record the behavioral responses.
Participants performed a DSR task in the scanner. A white fixation dot was presented at the center of the screen throughout the experiment. On each trial, participants first viewed 2 sample stimuli (sinusoidal gratings: radius = 5˚; contrast = 0.6; spatial frequency = 0.5 cycles/˚; phase angle randomized between 0˚and 180˚) presented sequentially on the screen (0.75 s exposure or each, separated by an ISI of 0.5 s). The orientation of each sample was selected independently from a fixed set of 9 values, spaced by 20˚and with a jitter between 0˚-3˚added to each, and the location of each was selected independently from a fixed set of 9 locations, each centered on an imaginary circle with radius of 8˚from central fixation and each spaced 40˚distant from the nearest locations. Offset of the second sample was followed by a delay period (Delay1.1, of 8 s), then a retrocue (Cue1; central presentation for 0.75 s) specifying whether the sample presented first or second ("1" or "2") would need to be recalled at the end of Delay1.2 (8 s). Recall was prompted by an orientation wheel appearing at the same location as the cued sample, and participants had been trained to use it to reproduce the cued orientation within a 4-s response window (Recall1). 0.5 s after the end of Recall1, a second retrocue (Cue2, central presentation for 0.75 s), indicated which sample ("1" or "2") need to be recalled at the end of Delay2 (2 s). On 50% of trials, Cue2 cued the same item that had been cued by Cue1 (a "Stay" cue), and for the remaining 50%, it cued the previously uncued sample (a "Switch" cue). Recall2 (same procedure as Recall1) was followed by an intertrial interval that varied randomly between 6, 8, and 10 seconds (Fig 1). During Delay1.2, the cued sample was termed the PMI and the uncued sample the UMI.
All participants completed 2 fMRI scanning sessions. For the first 4 participants, the first scanning session comprised 12 blocks of 12 trials each, and a second scanning session comprised thirteen 12-trial blocks for a total of 300 trials. Each scanned block lasted 464 s. The selection of stimulus location and orientation were independent, such that the orientation of the second sample matched that of the first on approximately one-ninth of the trials, and the location of the second sample matched that of the first on approximately one-ninth of the trials. For the remaining participants, we fully counterbalanced the conditions. To fully cross stimulus orientation, stimulus location, Cue1, and Cue2, 324 trials were required. To achieve this, participants 5-13 performed thirteen 12-trial runs during the first scanning session and fourteen 12-trial runs during the second scanning session. Each run consisted of 12 trials, resulting in a run length of 464 s. Before the first session, each participant completed 2 blocks of practice trials (12 trials per block) outside of the scanner and another block of practice within the scanner before the fMRI scanning began. During the scan, eye position was monitored and recorded using the Avotec RE-5700 eye-tracking system (Avotec).
This design was modified from a previous study [42] that found that in early visual cortex, the IEM reconstruction of a stimulus' orientation when a UMI was opposite to its reconstruction when a PMI, and in IPS, the IEM reconstruction of its location when a UMI was opposite to its reconstruction when a PMI. One complication for the representational transformation interpretation of those results, however, was that there existed in both a negative correlation between the identities of the PMI and UMI: both tasks were designed such that the PMI and UMI could never be the same on a single trial. This negative correlation could conceivably produce negative IEM reconstruction for analytic reasons. In the present study, the orientation and location of each trial's samples were selected independently. This aspect of the design, however, meant that retrocuing in terms of an item's location would be ambiguous on trials when the 2 samples appeared at the same location, and so sequential presentation and cuing by order were instituted. Note that although this aspect of the procedure meant that stimulus location was not task-critical (for example, one could ignore or forget where the samples had appeared and still succeed at the task), we expected that participants would nonetheless represent sample location in working memory. For example, previous work has indicated that stimulus location is encoded robustly during working memory for the orientation of individually presented stimuli [23,31].

Behavioral analysis
We analyzed behavioral responses with a 3-factor mixture model [24] that uses maximum likelihood estimation to generate estimates of 1) the probability of responses based on a representation of the probed item ("responses to target"), 2) the probability of responses incorrectly based on a representation of the unprobed item (i.e., "misbinding" or "swap" errors ("responses to nontarget"), and 3) the probability of responses that were guesses not based on either memory item, as well as 4) a "concentration" parameter that estimates the precision of nonguess responses. Conceptually, the concentration parameter is similar to a model-free measure of the precision of responses that is computed as the inverse of the standard deviation of the distribution of responses.

fMRI data acquisition
Whole-brain images were acquired using a 3 Tesla GE MR scanner (Discovery MR750;  A high-resolution T1 image was also acquired for each session with a fast, spoiled gradientrecalled-echo sequence (8.2 ms TR, 3.2 ms TE, 12˚flip angle, 176 axial slices, 256 × 256 inplane, 1.0 mm isotropic).

fMRI data preprocessing
Functional MRI data were preprocessed using AFNI (http://afni.nimh.nih.gov) [43]. The data were first registered to the final volume of each scan and then to anatomical images of the first scan session. Six nuisance regressors were included in GLMs to account for head motion artifacts in 6 different directions. The data were then motion corrected, detrended, and z-score normalized within each run.

fMRI ROI definition
The ROIs were created with a conjunction of anatomically and functionally defined voxels. We first created anatomical ROIs by extracting masks from the probabilistic atlas of Wang and colleagues [44] and warping them to each subject's structural scan in native space to create 2 regional masks, defining an early visual ROI as V1-V2 (merged, both hemispheres) and an IPS ROI as IPS0-5 (merged, both hemispheres). To identify task-related activity, we solved a general linear model (GLM) with AFNI, modeling each epoch of the task with 6 boxcar regressors-Sample (2 s), Delay1.1 (8 s), Delay1.2 (8 s), Recall1 (4 s), Delay2 (2 s), and Recall2 (4 s)convolved with a canonical hemodynamic response function and also including covariates to control for motion. We then created an anatomically constrained functional ROI for bilateral early visual cortex by selecting the 500 voxels inside the early visual anatomical ROI with the strongest loading on the Sample regressor and for bilateral IPS by selecting the 500 voxels inside the IPS anatomical ROI with highest loading on the Delay1.2 regressor.

Multivariate IEM
Mathematics. All IEM analyses were performed using custom functions in MATLAB. The IEM assumes that the responses of each voxel can be characterized by a small number of hypothesized tuning channels. The numbers of orientation and location tuning channels were both 9. Following previous work [21,45], the idealized feature tuning curve of each channel was defined as a half-wave-rectified sinusoid raised to the eighth power for both orientation and location.
For the IEM, we first computed the weight matrix (W) that projects the hypothesized channel responses (C 1 ) to actual measured fMRI signals in the training data set (B 1 ) and then extracted the estimated channel responses (Ĉ 2 ) for the test data set (B 2 ) using this weight matrix. The relationship between the training data set (B 1 , v × n, v: the number of voxels in the ROI; n: the number of repeated measurements) and the channel responses (C 1 , k × n, k: the number of orientations/locations) was characterized by where W was the weight matrix (v × k).
Therefore, the least-squared estimate of the weight matrix (Ŵ) was calculated using linear regression:Ŵ The channel responses (Ĉ 2 ) for the test data set (B 2 ) was then estimated using the weight matrix (Ŵ ):Ĉ

The logic behind inferring neural codes from patterns of IEM training and testing
We understand a neural code to refer to the systematic set of mappings between unique stimulus values and unique patterns of neural activity. For example, 9 values of stimulus orientation might map to 9 different high-dimensional patterns of activity within an ROI. One way in which neural coding in posterior cortex could vary with an item's priority status would be if this set of mappings were to rotate such that the individual mappings between these stimulus values and these neural patterns are now different, but the distance between neural patterns is preserved. In other words, although each item's neural representation has changed, the neural code has not changed. Therefore, we would consider such a transformation to be an example of remapping (of stimulus value to neural pattern), not of recoding. Such remapping would be consistent with the findings of van Loon and colleagues [19], who observed that a classifier trained on a stimulus category when it was the PMI could also decode that category when it was the UMI, even though the pattern of activity of the UMI projected into an opposite region of multidimensional scaling space relative to the PMI. Another way in which neural coding could vary with priority status would be if an altogether different set of mappings between the 9 stimulus values and 9 new patterns of activity were established. This would correspond to recruiting a different neural code. Under such a recoding scenario, cross-condition classification, such as described by van Loon and colleagues [19], would fail. In the present study, neural codes were operationalized by IEMs, and 4 different patterns of results were anticipated as possible outcomes. We illustrate them here with reference to stimulus orientation. The most straightforward scenario would be when an IEM trained on the orientation of stimuli when they are PMIs can successfully reconstruct the orientation of the same stimuli when they are UMIs. This would be interpreted as evidence that stimulus orientation is represented in the same neural code regardless of priority status. A second pattern could be that a PMI-trained IEM can successfully reconstruct the orientation of UMIs but does so in such a way that the reconstructed orientation is systematically shifted by a constant amount. For example, a stimulus with 0˚orientation reconstructs as 90˚when it is a UMI, and a stimulus with 30˚orientation reconstructs as 120˚when it is a UMI. This would be interpreted as evidence for a rotational remapping within the same neural code because it would correspond to the process described in the previous paragraph. A third pattern could be that a PMI-trained IEM fails to reconstruct the UMI, but the UMI can be reconstructed with a different IEM (for example, with a UMI-trained IEM). This would be interpreted as evidence that the same stimulus information is represented in different neural codes depending on priority status. The final possible outcome that we considered would be the failure to reconstruct the orientation of the UMI with any IEM, which would amount to a failure to find evidence for an active representation of the UMI (in the context of all the possible IEMs that have been trained).

Analysis plan
We used a leave-one-run-out cross-validation procedure to train and test IEMs, building weight matrices with signals from a time point of interest from the trials from all but one of the runs, then tested on the signals from that same time point from the trials in the held-out run. This procedure was iterated such that each run was tested using a separate training data set. The estimated channel outputs obtained after each iteration were shifted to a common center, with 0˚corresponding to the tested feature (orientation/location) channel.
Specifically, for "PMI-trained IEMs," the data on each trial were labeled according to the identity of the PMI at the time point of interest, such that the model learned each of the 9 possible values of stimulus orientation. Importantly, because the orientation of the UMI was random relative to the PMI on each of these trials, a PMI-trained IEM could not learn any information about the UMI. For "UMI-trained" IEMs, the data from the same time points were labeled according to the identity of the UMI. For PMI-trained IEMs, the IEMs were tested on data labeled according to the identity of the PMI as well as on data labeled according to the identity of the UMI. When tested with UMI-labeled data, reconstructions from this PMI-trained IEM would index the extent to which the representational format of the UMI was similar to that of the PMI. Results from the PMI-trained IEMs would be the primary focus of this study because PMI-trained models have been shown to be valid and robust for various brain regions, including occipitotemporal cortex, IPS, and FEF [21][22][23]. For UMI-trained IEMs, the IEM was trained on data labeled according to the identity of the UMI and tested on data labeled according to the identity of the both the PMI and the UMI.
For each IEM, we first examined the time course of reconstructions from trial onset to the onset of Recall1 (i.e., from 0 to 18 s after trial onset), which demonstrated how representations of the PMI and of the UMI evolved before and after Cue1 (i.e., during Delay1.1 and Delay1.2). Moreover, because we were interested in the delay period after the retrocue onset (Delay1.2), we focused on 18 s after trial onset (i.e., 7.25 s after Cue1 offset) for statistical comparisons in order to maximize the likelihood that our analyses would capture the effect of the retrocue while taking into account the hemodynamic lag in the BOLD signal. All the IEMs were estimated for orientations and locations separately. Temporal generalization analyses were conducted by training and testing the IEM on every time point to examine whether the neural code at a specific time point could be successfully generalized to another.

Statistical analyses
To characterize the strength of each reconstruction, we collapsed over the channel responses on both sides of the tested channel, averaged them, and calculated the slope of each collapsed reconstruction using linear regression [31,46]. A larger positive slope indicates stronger positive representation, and a larger negative slope indicates stronger negative representation. We used a bootstrapping procedure [21] to characterize the significance of the slopes. For each IEM/ROI, 10 orientation/location reconstructions were randomly sampled with replacement from the reconstruction pool of 13 participants and averaged. This procedure was repeated 10,000 times, resulting in 10,000 average orientation/location reconstructions for each IEM/ ROI and, correspondingly, 10,000 slopes. To obtain a two-tailed measure of the p-values, the probabilities of obtaining a positive (p pos ; reconstruction peaking at the tested channel) or negative (p neg ) slope among the 10,000 slopes were calculated separately, and the p-values of the bootstrapping tests were calculated using the following equation: To characterize the difference between 2 slopes, we first calculated the difference between 2 bootstrapped slopes 10,000 times, which generated 10,000 slope differences. The significance of the slope difference was then calculated using the same two-tailed method as above. All the p-values for the analysis at 18 s were corrected for multiple comparisons across conditions and ROIs; all the p-values for the time course analysis remained uncorrected.