6 Mar 2017: Gurariy G, Killebrew KW, Berryhill ME, Caplovitz GP (2017) Correction: Induced and Evoked Human Electrophysiological Correlates of Visual Working Memory Set-Size Effects at Encoding. PLOS ONE 12(3): e0173650. doi: 10.1371/journal.pone.0173650 View correction
The ability to encode, store, and retrieve visually presented objects is referred to as visual working memory (VWM). Although crucial for many cognitive processes, previous research reveals that VWM strictly capacity limited. This capacity limitation is behaviorally observable in the set size effect: the ability to successfully report items in VWM asymptotes at a small number of items. Research into the neural correlates of set size effects and VWM capacity limits in general largely focus on the maintenance period of VWM. However, we previously reported that neural resources allocated to individual items during VWM encoding correspond to successful VWM performance. Here we expand on those findings by investigating neural correlates of set size during VWM encoding. We hypothesized that neural signatures of encoding-related VWM capacity limitations should be differentiable as a function of set size. We tested our hypothesis using High Density Electroencephalography (HD-EEG) to analyze frequency components evoked by flickering target items in VWM displays of set size 2 or 4. We found that set size modulated the amplitude of the 1st and 2nd harmonic frequencies evoked during successful VWM encoding across frontal and occipital-parietal electrodes. Frontal sites exhibited the most robust effects for the 2nd harmonic (set size 2 > set size 4). Additionally, we found a set-size effect on the induced power of delta-band (1–4 Hz) activity (set size 2 > set size 4). These results are consistent with a capacity limited VWM resource at encoding that is distributed across to-be-remembered items in a VWM display. This resource may work in conjunction with a task-specific selection process that determines which items are to be encoded and which are to be ignored. These neural set size effects support the view that VWM capacity limitations begin with encoding related processes.
Citation: Gurariy G, Killebrew KW, Berryhill ME, Caplovitz GP (2016) Induced and Evoked Human Electrophysiological Correlates of Visual Working Memory Set-Size Effects at Encoding. PLoS ONE 11(11): e0167022. doi:10.1371/journal.pone.0167022
Editor: Andrea Antal, University Medical Center Goettingen, GERMANY
Received: December 16, 2015; Accepted: November 8, 2016; Published: November 30, 2016
Copyright: © 2016 Gurariy et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This material is based upon work supported by the National Science Foundation under grants: NSF 1632848 and NSF 1632849 and the National Institutes of Health under grants: 1P20GM103650 and R15EY022775. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Visual working memory (VWM) refers to the ability to store and manipulate visual information for brief periods of time. Although VWM is an integral aspect of daily cognition [1,2], it is nonetheless capacity limited . Behaviorally, these limitations clearly reveal themselves when set sizes increase. VWM performance decreases as the encoding, maintenance and retrieval demands increase. A recent focus of cognitive neuroscience research is to understand the neural bases of this capacity limitation. Here, we applied high-density electroencephalography (HD-EEG) to investigate both induced and evoked neural set size effects operating during the encoding phase of VWM.
Prior to objects being stored in VWM, stimuli enter iconic memory. This buffer helps explain the richness of visual experience for brief periods of time as well as transsaccadic perception [4,5]. Although brief (<500ms), iconic memory has a nearly limitless capacity that decays quickly [6–8]. In contrast, VWM is limited to ~4 items [9–11], or fewer . Here, we ask: do limitations arise during VWM encoding?
Set size effects are a hallmark of VWM’s capacity limitation. VWM performance accuracy decreases with an increase in the number of items (set size) presented [13,11]. Recent findings aimed at elucidating the neural bases of VWM capacity limitations have sought to link neural correlates of the behavioral set size effect with stages of VWM [14–17]. Functional magnetic resonance imaging (fMRI) data show that during VWM maintenance the BOLD signal in the intraparietal sulcus (IPS) increases according to the number of items a person is maintaining and this signal reaches asymptote at an individual’s maximum VWM capacity [17–19]. Similarly, EEG research reveals an evoked potential localized to occipito-parietal regions during maintenance of items presented to the contralateral side of the visual field. This physiological signal, known as the contralateral delay activity (CDA), increases in amplitude according to set size and also asymptotes at a participant’s VWM capacity limit [20–22]. To bridge these two literatures, we recently measured fMRI while applying a CDA paradigm. We found that the IPS represents contralaterally presented items in VWM, but that VWM capacity limits did not differ when attending to one or both visual hemifields . These findings provide converging neural evidence that VWM maintenance related processes contribute to VWM capacity limitations.
However, it is clear that the retrieval stage also limits VWM capacity. There is a literature devoted to studying how different task demands elicit different behavioral and neural patterns during retrieval. A prevailing view is that we typically scan through the contents of VWM using serial exhaustive search, even after finding the target item [24–26]. Consequently, with more items in VWM, responses slow, subjecting items in VWM to greater decay and worse performance [27,28].
The conversation thus far overlooks the role encoding plays in limiting VWM capacity: encoding necessarily precedes maintenance and retrieval. Since Miller made ‘chunking’ famous , VWM encoding strategy has been considered important [30,31], as is prior knowledge . Behaviorally, encoding duration modulates VWM performance with shorter times leading to less accurate responses during recall . Monkey physiology shows that capacity limitations may emerge during encoding as objects compete for resources that are distributed in a flexible manner within each hemifield . In humans, physiological recordings have shown that the neural activity associated with objects that are physically present is nearly indistinguishable from objects that have been removed from view and are being actively maintained in VWM . These findings confirm the importance of encoding as a contributing factor to capacity limitations.
Recent neural evidence highlighting the importance of encoding comes from studies showing that consistent activations during encoding and maintenance are required for successful retrieval. For example, visual regions (e.g. primary visual cortex) contribute to VWM maintenance in the absence of visual stimulation [36,37]. Furthermore, successful VWM performance is predicted by enhanced functional connectivity between extrastriate cortex and lateral prefrontal cortex , and between frontoparietal regions [39,40]. Furthermore, functional connectivity shifts predictably depending on set size . Set size also predicts activity in regions associated with long-term memory including hippocampal involvement during encoding , and parahippocampal activity during maintenance . It is reasonable to believe that VWM capacity limits arise from each stage of VWM. Yet, questions remain regarding how these limitations emerge.
Recently, we used the classic frequency-tagging EEG technique to examine neural responses to the visual stimuli presented in a VWM task. Frequency-tagging entails rapid and periodic stimulus presentation, inducing corresponding rapid-periodic responses in the simultaneously recorded EEG . We reported that the frequency-tag amplitudes during encoding were significantly larger for successfully remembered items compared to subsequently forgotten items . The frequency-tags of unprobed items were also greater on correct trials.
To summarize this technique more specifically, frequency-tags correspond to transient evoked potentials measured at the scalp using EEG. Transforming the EEG signal into the frequency domain allows for the identification of frequency-tags corresponding to each flickering stimulus. For example, a stimulus flickering at 5 Hz elicits five potentially superimposed visually evoked potentials per second and thus, produces a 5 Hz signal in the frequency domain. This 5 Hz frequency-tag represents the neural correlate of the stimulus response. In addition to frequency-tags corresponding to the rate of stimulus flicker (fundamental frequency), the EEG also contains harmonics of the fundamental frequency. For example, a stimulus flickering at 5 Hz may elicit 2nd and 3rd harmonics at 10 Hz and 15 Hz, respectively. The fundamental frequency and its harmonics are thought to be driven by different neural populations, the former indicative of bottom-up processing and the latter of top-down processing . Although it is conceivable that a flickering stimulus of sort used in our study may lead to the entrainment of neural ensembles underlying cognitive operations such as VWM encoding, we note that this is not goal of the frequency-tagging procedure. Thus, the stimulus frequencies chosen for the experimental procedure (described in Materials and methods section: Experimental procedure) are arbitrary with regards to endogenous neural oscillations involved in cognitive operations. Rather, we use these frequency-tags to identify neural correlates of individual stimuli during VWM encoding.
However, induced oscillations also provide insights regarding cognitive processes, including VWM [47,48]. Unlike evoked oscillations, induced oscillations are not phase-locked to the stimulus and they are associated with cognitive states as well as functional changes in neural processing. These oscillations may reflect changes occurring within as well as between brain structures. Specifically, these changes have been proposed as a mechanism for neural synchronization resulting in short- and long- range communication. Such communication can be cortico-cortical or subcortico-cortical and has been linked to numerous cognitive processes including attention, memory and feature binding . These oscillations can be categorized into the following frequency bands: delta (1–4 Hz), theta (5–8 Hz), alpha (9–12 Hz), beta (12–31 Hz), and gamma (32–100 Hz). For our purposes, frequency-tags are used to investigate the allocation of neural resources to individual items in the VWM paradigm as a function of set size, whereas induced oscillations can be interpreted as a proxy for task-related cognitive processes occurring during the encoding period.
Here, we determined whether neural correlates of set size effects can be observed during VWM encoding exploring both induced and evoked measures. We compared encoding-related neural responses to VWM arrays of set size 2 or 4. We found that during VWM encoding the frequency-tag amplitudes corresponding to successfully retrieved items is modulated by set size. The qualitative nature of this encoding-related neural correlate is distinct from those observed in previous studies examining neural set size effects during maintenance. Unlike fMRI and CDA results showing increases in neural responses with increased set sizes, we find significantly larger neural responses to successfully retrieved items in the 2-item compared to the 4-item set size. In addition, we found an effect of set-size on the amplitude of the induced delta-band activity. Again, increased power was observed in the set size 2 compared to set size 4. These results are consistent with the conclusion that encoding-related VWM processes are capacity limited and attempts to encode multiple items overtaxes this resource and reduces encoding success.
Materials and Methods
Twenty-four, right handed, neurotypical adults with normal or corrected to normal visual acuity participated (13 male, age 20–40) and provided informed written consent. The Institutional Review Board at the University of Nevada, Reno approved all protocols. As described below, three of these participants were excluded due to excessive artifacts detected in the EEG.
Stimuli were displayed on a Mitsubishi Diamond Pro270 CRT monitor (20in, 1024x768) with a 120-Hz refresh rate, running via a 2.6Mhz MacMini and presented using the PsychToolbox [50,51] for MATLAB (MathWorks Inc., Natick, MA). Viewing distance was 57 cm.
The electroencephalogram (EEG) was continuously recorded using a 256 channel HydroCel Geodesic Sensor Net via an EGI Net Amps Bio 300 amplifier (Electrical Geodesics Inc., Eugene, OR) sampling at 1000 Hz. The digital data were recorded using Netstation 5.0(1) software. Impendence values were kept at or below 50 Ω. Frame-accurate timing of stimulus presentation was validated by photodiode.
Participants performed a VWM change detection task with a set size of either 2 or 4 items; see Fig 1. Trials began with fixation (600 ms) consisting of a black central fixation point (0.35° x 0.35°) on a neutral gray background, surrounded by 4 black squares (7° x 7°) that occupied predetermined stimulus positions centered in each of the four quadrants with a random offset of up to 1.5° in any direction. Next, either 2 or 4 square placeholders were replaced by shape stimuli (1000 ms). The shapes were bilaterally symmetrical shapes (7° x 7°) chosen randomly (without replacement) from a set of ten. Stimuli were generated by a previously described algorithm [52,53]. During encoding, each shape reversed contrast (black-white) at a different frequency (3hz, 5hz, 12hz, or 20hz). In the set size 4 condition, each shape flickered at one of the frequencies; in the set size 2 condition, the two shapes flickered at 3hz and 5hz respectively and the two placeholder squares flickered at 12 Hz and 20 Hz. The placeholder squares ensured that any differences between set size 2 and set size 4 were not caused by the flicker of additional stimuli or frequencies in the set size 4 condition. Only the 3Hz and 5 Hz probed trials in the set size 4 condition were analyzed; none of the 12Hz and 20Hz probed items in the set size 4 condition were included in analyses. These specific frequencies were chosen because they allow for an integer number of cycles at the 120hz frame refresh rate during the 1000 ms encoding period, and preserved independence through the third harmonics. The encoding phase was followed by a maintenance period (1000 ms) during which time only the fixation spot was present. Next, a probe stimulus shape reappeared in one quadrant (3000 ms). Participants were instructed to indicate whether this probe item was old (same shape in the same location) or new (different shape or different location) via key press. Thus, correct responses were contingent on accurately maintaining both shape and location. Each frequency (set size 2: 3 Hz, 5 Hz; set size 4: 3 Hz, 5 Hz, 12 Hz & 20 Hz) was probed a total of 76 times (38 were ‘old’, 38 were ‘new’). The entire experiment consisted of 456 trials (set size 2: 152 trials (1/3), set size 4: 304 (2/3)) and 50% of probe items matched what was shown at encoding. Trials were presented in pseudorandom order.
(A) Participants viewed an initial fixation screen (600ms) followed by a memory array (1000ms) in which each item flicked at a distinct frequency (3 Hz, 5 Hz, 12 Hz and 20 Hz). Set size was manipulated to create 2 conditions, set size 2 or set size 4. A delay period (1000ms) followed the stimulus presentation. Finally, a single probed stimulus appeared in one of the four previously presented locations. Participants were instructed to indicate whether the probed stimulus was old (same shape in same location) or new (different shape). Note: black squares were used in the fixation period to control for onset VEP responses that could contaminate the data. (B) Epochs lasting 1000 ms time-locked to the onset of the memory array were extracted from the raw data. (C) In the evoked analysis, once epochs in each condition were averaged together, a Fourier Transform was applied to the data so that the frequency tag amplitudes could be extracted for the fundamental (red) and harmonic (green) frequencies corresponding to probed items in the stimulus array. Conversely, in the induced condition, the Fourier Transform is applied to each trial prior to averaging.
Electrophysiological data preprocessing
The EEG data were processed using Netstation 5.0 and custom scripts written in MATLAB. A high-pass filter of 0.1 Hz removed slow drift noise. Next, the filtered data was segmented into 1000 ms epochs time-locked to stimulus onset and lasting the duration of the encoding period. Segmentation was done using trigger markers sent from the stimulus computer to the acquisition computer at the onset of every encoding period. The time offset between stimulus appearance and trigger registration was measured using a photodiode and compensated for during trial segmentation. Using Netstation, artifact detection was performed to identify trials containing eye blinks, eye movements, and bad channels; this was followed when possible by bad channel replacement. Trials were marked for exclusion if they contained an eye blink, an eye movement, or >10% bad channels (exclusion criteria on the basis of too many bad trials is discussed in Materials and methods section: Artifact detection and subject exclusion). Using custom MATLAB scripts, the correct trials were grouped into the following 4 conditions based on set size and the flicker frequency of the to-be-probed item: correct 3 Hz/set size 2, correct 5 Hz/set size 2, correct 3 Hz/set size 4, correct 5 Hz/set size 4. The trials probing the 12 Hz or 20 Hz item in the set size 4 condition were excluded from analyses, as were incorrect trials.
Artifact detection and subject exclusion
As described in the previous section, a trial was marked bad if it contained an eye blink, an eye movement, or if 10% of the channels were found to be bad during the encoding period. Participants whose number of bad trials exceeded 30% were excluded from the study. Under these criteria, 3 people were excluded from the final analysis leaving a total of 21 participants. Among these remaining participants, the mean number of trials per condition was 50.6 with a standard deviation of 8.46. This number takes into account the trials rejected due to artifacts as well as the good trails that were left out during the permutation analysis (see Materials and methods section: Permutation analysis).
Because only correct trials were analyzed there were unequal numbers of trials per condition per participant. To ensure comparable signal-to-noise rations and fairer comparisons between conditions, we conducted a permutation analysis in which an equal number of trials contributed to each condition average. For each participant, we identified the condition with the fewest number of trials and on each iteration of the permutation analysis, randomly selected that number of trials from the other conditions. This process was repeated over multiple iterations (see Materials and methods sections on frequency-tagging and induced power analysis). The trials from each iteration were averaged together appropriately for each of the four conditions: set size: 2 or 4; frequency of probed item: 3 Hz or 5 Hz. Permutation analyses were performed independently for the frequency-tagging and the induced power analysis; these are described below in more detail.
Frequency-tagging (evoked) analysis
To avoid the random selection of an outlier subset of trials, a total of 50,000 iterations of the permutation analysis were performed and the average across iteration was used as the response to the other conditions. This process resulted in four average time-domain waveforms computed at each electrode for each participant. Each waveform was 1000ms long and corresponded to either the average response for set-size 2 or 4 at the frequency of probed item: 3 Hz or 5 Hz. A Fourier transform was applied to the average of each condition, at each of the 256 channels, so that the frequency-tag amplitudes for the fundamental and second harmonic frequencies could be extracted for the probed items at each set size (2 and 4). Again, the analysis was restricted to the 3Hz and 5Hz trials. For example, if a to-be-probed item flickered at 5 Hz during encoding, then the amplitudes of 5 Hz (1f) and 10 Hz (2f) were used as indices for neural activity related to VWM encoding of the to-be-probed item.
At each electrode site, a set size index was computed for the amplitudes of each probed-frequency, grouped according to whether it was the fundamental: or the 2nd harmonic: These normalizing-indices allow us to combine the 3Hz and 5Hz data while compensating for intrinsic differences in frequency specific amplitudes often observed with EEG data. This was done separately for the fundamental frequencies (SSI1f): 3hz and 5hz index values were averaged together as were the indices for the 2nd harmonics (SSI2f): An index value greater than zero indicates that the neural response to the to-be-probed items are greater in set size 4 conditions than set size 2. Conversely, negative indices indicate greater neural responses in the set size 2 arrays. Index values were then averaged across subjects and subjected to statistical analyses.
At the group level, one sample t-tests (α = 0.05) were performed at each electrode to evaluate whether the indices were significantly non-zero. We applied a False Discovery Rate Correction (q = 0.10) to account for multiple comparisons.
Induced power frequency analysis
To examine induced effects of set-size, we applied the Fourier transform to each individual trial and computed the amplitude of response at each frequency. For each frequency we then averaged the amplitudes across trials (excluding phase). Doing so allows for the extraction of induced oscillatory activity that would otherwise cancel out due to variations in phase . As before, to account for uneven numbers of trials in each condition, we applied a permutation analysis using 25,000 iterations (the smaller number of iterations in the induced power analysis relative to the evoked power analysis was due to the increased computational load of performing a Fourier analysis on individual trials as opposed to the average of those trials). This process resulted in 4 averaged frequency-domain amplitude spectra computed at each electrode for each participant. Each spectra consisted of the induced power at frequencies ranging from 1–500 Hz corresponding to the encoding period. Next, we isolated amplitudes for frequencies between 1–100 Hz. These amplitudes were normalized by creating an index value for each frequency (1–100 Hz) across the 3 Hz and 5 Hz trials and subsequently averaged together (3 Hz & 5 Hz). A sample index value is shown below
These normalizing-indices allow us to again combine data across frequency while compensating for intrinsic differences in frequency specific amplitudes observed in EEG data. Next, the index values were averaged across the following frequency bands: delta (1 Hz—4 Hz), theta (5 Hz—7 Hz), alpha (8 Hz -12 Hz), beta (13 Hz—31 Hz) and gamma (32 Hz—100 Hz).
At the group level, one sample t-tests (α = 0.05) were performed at each electrode to evaluate whether the indices were significantly non-zero. We applied a False Discovery Rate Correction (q = 0.10) to account for multiple comparisons.
First, we examined the behavioral performance in the VWM task to make sure we replicated the set size effect. As expected, accuracy was significantly better in the set size 2 condition (M = 92.32, SD = 5.28) compared to the set size 4 condition (M = 74.62, SD = 6.96; t(20) = 4.83, p <0.0001); see Fig 2. Secondly, we wanted to ensure that behavioral performance was not affected by differences in the flicker frequency of the probed items. In the set size 2 condition, a paired samples t-test yielded no significant difference between the 3 Hz (M = 93.04, SD = 4.31) and 5 Hz trials (M = 91.59, SD = 4.49; t(20) = 1.39, p = 0.18). In the set size 4 condition, a repeated measures ANOVA showed no significant differences between the 3 Hz (M = 71.67, SD = 5.46), 5 Hz (M = 75.68, SD = 6.17), 12 Hz (M = 67.35, SD = 6.07), and 20 Hz trials (M = 73.25, SD = 6.59; F(3, 80) = 0.67, p = 0.57).
Analysis of frequency tagging amplitude and set size effects
In Fig 3A, index values for SSI1f and SSI2f (see Materials and methods section: Frequency-tagging (evoked) analysis) for description) are plotted on topographic maps to illustrate set size effects at encoding for set size 2 > 4 and 4 > 2 . One-sample t-tests were computed for each index across all 256 channels. The subsequent t-statistics are plotted on topographic maps in Fig 3B for both set size 2 > 4 and set size 4 > 2. For these maps, only the t-statistics that were significant at the uncorrected p < 0.05 level are plotted. In Fig 3C, the t-stats are ranked by p-value and plotted for every electrode. Positive values depict greater activity for set size 4 and negative values depict greater activity for set size 2. Green dotted lines represent a threshold equivalent to: α = 0.05. Red-dotted lines represent a FDR corrected threshold of q = 0.1.
(A) Index values (described in Materials and methods section: Frequency-tagging (evoked) analysis) plotted on topographic maps. Green corresponds to set size 4 > set size 2, purple corresponds to set size 2 > set size 4. (B) t-stats plotted on topographic maps. Plotted values correspond to t-stats at or below a p value of 0.05. Red corresponds to set size 4 > set size 2, blue corresponds to set size 2 > set size 4. (C) t-stats at each electrode arranged by p values. Green dotted line represents a threshold of α = 0.05. Red line represents an FDR corrected threshold for q = 0.1 (see results section: Analysis of frequency tagging amplitude and set size effects). Channels significant at or above the FDR threshold are represented by a thicker border.
We highlight the following observations. First, the indices shown in Fig 3A reveal a set of electrodes at central and posterior sites for which set size 4 yields larger responses primarily in the fundamental frequency responses. However, few of these indices are statistically significant, particularly when taking multiple comparisons into consideration (Fig 3B and 3C). Second, another set of electrodes located primarily at frontal electrode sites for which set size 2 yield larger responses in both the fundamental frequency and 2nd harmonic, many of these indices are statistically significant, even when taking multiple comparisons into consideration (Fig 3B and 3C).
Analysis of induced power and set size effects
Index values for five frequency bands (beta, theta, alpha, beta, gamma) are plotted on topographic maps to illustrate induced power set size effects at encoding (Fig 4A). One-sample t-tests were computed for each index across all 256 channels and plotted as a function of set size 2 > 4 and set size 4 > 2 (Fig 4B). Only the t-statistics that were significant at the uncorrected p < 0.05 level are plotted. As before, the t-stats were organized by p-value and plotted for every electrode following the conventions described above (Fig 4C: positive: set size 4 > 2; negative: set size 2 > 4). Green dotted lines represent a threshold equivalent to α = 0.05. For the delta band, the 0.05 alpha value corresponds to an FDR q-value of 0.09. In contrast, few of the t-stats for the theta, alpha and beta frequency bands are significant at α = 0.05 (FDR qs > 0.35, 0.99 & 0.99, respectively). No significant t-stats were observed for the gamma frequency band.
(A) Index values (described Materials and methods section: Induced power frequency analysis) plotted on topographic maps. Green corresponds to set size 4 > set size 2, purple corresponds to set size 2 > set size 4. (B) t-stats plotted on topographic maps. Plotted values correspond t-stats at or below a p value of 0.05. Red corresponds to set size 4 > set size 2, blue corresponds to set size 2 > set size 4. (C) t-stats at each electrode arranged by p values. Green dotted line represents a threshold of α = 0.05. Channels significant at or above the FDR threshold are represented by a thicker border. Each row depicts data from a single frequency band: delta (top), theta, alpha, beta, gamma (bottom).
We highlight the following observations. Increased delta power in the set size 2 > set size 4 was observed in frontal and posterior electrode cites. Although fewer electrodes reached statistical significance, some increased theta activity was observed in the set size 2 condition relative to the set size 4 condition, primarily at frontal locations. However, it should be noted that these electrodes did not survive FDR correction.
To better understand the neural basis of VWM capacity limitations we investigated encoding-related processing differences as a function of set size and retrieval success. To examine potentially stimulus-specific neural correlates of set-size we leveraged the classic EEG frequency-tagging approach. To investigate potentially task-related neural correlates of set-size we examined changes in induced power.
Frequency-tagging involves flickering each stimulus at a unique frequency and measuring the evoked EEG power at the corresponding frequency. Index values were computed allowing a comparison of frequency-tags across the set size 2 and set size 4 conditions. These data revealed that over anterior sites above frontal cortex, the frequency-tags, especially for the 2nd harmonic, were significantly larger in the set size 2 condition compared to the set size 4 condition. This same frontal area and additional sites over posterior regions showed induced power changes in the delta band during the VWM encoding period as a function of set size. These results indicate that VWM set size effects are partially mediated by encoding related processes. One reason why these results were observed primarily over the frontal cortex (as opposed to occipital cortex) may be related to the fact that spatial maps are highly lateralized in the early visual areas but become less lateralized as visual information reaches the frontal lobes. Since items in the set size 2 condition were presented at random locations (either bilaterally or unilaterally) and all correct trials were subsequently averaged together, this may explain the lack of set size effects over posterior electrodes.
These data extend a growing literature honing in on the neural correlates of VWM capacity. Considerable recent interest has focused on VWM maintenance. Previous fMRI studies uncovered a parametric increase in posterior parietal cortex (PPC) as a function of increased VWM set size [52,55]. For each individual, when the number of stored items exceeded working memory capacity, the BOLD signal plateaued . These results suggest the presence of a finite neural resource that is depleted. The EEG literature has produced analogous results using the CDA, which increases with VWM set size up to an individual’s capacity limit [56,22].
The current results of the frequency-tagging (evoked) analysis may appear contradictory to fMRI and EEG literature reporting increased neural measures corresponding with increased set size because we report greater activity for the smaller set size condition. There is a key difference in viewing and interpreting these data comprehensively. Previous work examined the aggregate neural resources afforded all stimulus items. Here, the frequency-tagging approach permits measurement of individual stimuli. Thus, the aggregate response increases with set size, whereas the individual response data shows that the amount of neural resources allocated to any single stimulus item actually decreases. In other words, when more stimuli are presented, more resources are needed to represent them effectively in VWM, yet each item gets a smaller portion of these resources.
There is an essential role of attention as a selection process for further processing and entry into VWM. Indeed, attentional cueing reveals the large storage of iconic memory , and the decay in resolution between iconic and the proposed fragile and robust stages of VWM . Attention facilitates performance on change detection tasks  and in VWM more generally [59,60]. Research on object tracking also shows important parallels between the deployment of attention in visual tasks and VWM capacity limitations. Pylyshyn and Storm  showed that participants track no more than 5 independently moving objects and that there exists an inverse relationship between accuracy and number of targets being tracked. Furthermore, there is evidence that working memory capacity is predictive of an individual’s tracking capacity . A study by Drew & Vogel  showed that the capacity limitations associated with object tracking tasks appear to arise during the initial selection process when subjects are allocating their attentional resources towards the to-be-tracked items.
Additional analogs between object tracking and VWM have come from fMRI studies examining BOLD activation and the effects of attentional load. Specifically, as the number of items being tracked increases there is a corresponding linear increase in PPC activity [64,65] much like the effects observed during the maintenance phase of VWM tasks [17,19]. We note though that attention alone is insufficient to ensure successful VWM encoding. It is likely the case that encoding-specific processes are required, perhaps enabled by attention .
Although the effects observed in the evoked analysis were prominent for the set size 2 > set size 4 condition, it is worth mentioning that some electrodes showed the opposite effect, set size 4 > set size 2. Specifically, this was observed on 4 posterior-central electrodes at the fundamental frequency. Interestingly, these results seem to coincide with the CDA and fMRI VWM data that show increasing neural activity with increased set size [10,21]. Furthermore, enhanced connectivity has been observed between extrastriate regions and frontal and parietal sites [38–40]. Thus, one interpretation of these results can be seen as the shifting of neural resources to posterior regions given increased task demands. However, we reiterate that the majority of electrodes showed the opposite effect and advise caution in making inferences from topography to neural sources given the many known challenges associated with EEG source estimation .
One aspect of our results deserves further comment. The pattern of results is most evident at the 2nd harmonic compared to the fundamental frequency. One possible explanation is that the first and second harmonic underlie differing cognitive processes. One suggestion is that the fundamental frequency and the second harmonic originate from different neural populations with differing topographical distributions . Furthermore, the fundamental frequency is thought to be associated with low-level visual responses whereas the second harmonic is thought to reflect higher-level visual processes . Subsequently, the second harmonics show greater attentional modulation compared to the fundamental frequency . One assumption of the present study is that successful retrieval is contingent upon the distribution of sufficient attentional resources during encoding, thus greater signal would be expected in the second harmonic. Thus, we hypothesize that the observed differences between the fundamental frequency and the 2nd harmonic are consistent with the underlying generators of the frequency-tag signal. Next, we will briefly summarize the advantages of using the frequency-tagging method for further investigating encoding related processes.
One of the major advantages of the frequency-tagging technique is that it allows for the examination of neural responses associated with specific stimuli by extracting the amplitude associated with each “frequency-tag”. However, studying VWM encoding presents a challenge in the form of disambiguating encoding from maintenance phases. This is especially a concern for fMRI research given fMRI’s poor temporal resolution and infrequent use of partial trials . However, even traditional ERPs can present similar challenges because evoked EEG components evolve over tens to hundreds of milliseconds. Thus, frequency-tagging provides an ideal technique to isolate encoding related processes and to characterize the neurophysiological signal associated with individual items.
The induced frequency analysis also showed greater activity in the set size 2 condition in the delta band at frontal and posterior electrode cites as well as some theta activity in frontal electrodes. No significant differences were observed in alpha, beta or gamma bands. The theta band oscillations are consistent with previous observations that they may be driven by cortico-hippocampal interactions . Theta band oscillations are associated with various cognitive domains, including working memory , episodic memory encoding and retrieval , cognitive control , selective attention , and attentional allocation to target stimuli . However, we remind the reader that although some electrodes in the theta band were significant at an alpha of 0.05, none of the electrodes survived FDR correction. Thus, the evidence for the involvement of theta activity in the current task is minimal at best. Less is known regarding the role of delta oscillations in human cognition . Some evidence suggests there are functional similarities between the delta and theta bands specifically as it relates to cognitive control . Given its slow oscillation, the delta band is well suited for long-range communication and may serve as a mechanism by which frontal lobes modulate distant brain regions . Delta oscillations contribute to the P300 oddball response and are modulated by novel distractors as well as task switching cues . Delta band oscillations have also been associated with attentional selection and stimulus expectancy [76, 77]. Finally, as it relates to VWM, studies suggest that delta oscillations increase in regions that relate to inhibition of interference during the task [73,78,79].
On a related note, the set size 2 condition contained two squares flickering at 12 Hz and 20 Hz in addition to the to-be-remembered stimuli. These squares were added so as to avoid any potential artifacts stemming from more flickering stimuli in the set size 4 condition. Participants were instructed to ignore these flickering squares and, the significantly better behavioral performance in the set size 2 condition indicates they did so successfully. Thus, solely the set size 2 condition required the filtering of task-irrelevant distractors whose spatial location differed between trials. Thus, based on the studies reviewed in the previous sections [71–73,78,79], a speculative interpretation of the induced delta and theta power is that these frequencies may be relevant in distractor filtering and attentional selection.
In closing, by examining the neural signals generated by frequency tags for each to-be-encoded stimulus, we build upon our previous findings  that highlighted the role of errors at encoding as a contributing factor of VWM capacity. By focusing on neural set-size effects for correct trials, the current study revealed that like VWM as a whole, encoding-related neural resources are likely capacity-limited. These resources get distributed across to-be-encoded items such that the amount of resources allocated to each item is inversely proportional to the number of items being encoded. We conclude that this encoding-specific capacity limitation can account at least in part for declines in behavioral performance on VWM tasks observed as set sizes increase. These findings further highlight the importance of encoding-stage processes in constraining models of VWM capacity.
- Conceptualization: GG KWK MEB GPC.
- Funding acquisition: MEB GPC.
- Investigation: GG KWK MEB GPC.
- Methodology: GG KWK MEB GPC.
- Resources: GG KWK MEB GPC.
- Supervision: MEB GPC.
- Writing – original draft: GG KWK MEB GPC.
- Writing – review & editing: GG KWK MEB GPC.
- 1. Engle RW, Tuholski SW, Laughlin JE, Conway ARA. Working memory, short-term memory, and general fluid intelligence: A latent-variable approach. J Exp Psychol Gen. 1999;128(3):309–31. pmid:10513398
- 2. Kane MJ, Brown LH, McVay JC, Silvia PJ, Myin-Germeys I, Kwapil TR. For whom the mind wanders, and when—An experience-sampling study of working memory and executive control in daily life. Psychol Sci. 2007;18(7):614–21. doi: 10.1111/j.1467-9280.2007.01948.x. pmid:17614870
- 3. Cowan N. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behav Brain Sci. 2001;24(1):87-+. pmid:11515286
- 4. Irwin DE. Information Integration across Saccadic Eye-Movements. Cognitive Psychol. 1991;23(3):420–56.
- 5. Irwin DE. Memory for Position and Identity across Eye-Movements. J Exp Psychol Learn. 1992;18(2):307–17.
- 6. Sperling G. The information available in brief visual presentations. Psychological Monographs 1960;74:1–29.
- 7. Phillips W. On the distinction between sensory storage and short-term visual memory. Perception and Psychophysics. 1974;16:193–8.
- 8. Averbach E, Sperling G. Short term storage of information in vision. Information Theory. 1961:196–211.
- 9. Cowan N. The magical number 4 in short-term memory: A reconsideration of mental storage capacity (vol 23, pg 87, 2001). Behav Brain Sci. 2001;24(3).
- 10. Luck SJ, Vogel EK. Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn Sci. 2013;17(8):391–400. doi: 10.1016/j.tics.2013.06.006. pmid:23850263
- 11. Vogel EK, Woodman GF, Luck SJ. Storage of features, conjunctions, and objects in visual working memory. J Exp Psychol Human. 2001;27(1):92–114.
- 12. LaRocque JJ, Lewis-Peacock JA, Postle BR. Multiple neural states of representation in short-term memory? It's a matter of attention. Front Hum Neurosci. 2014;8.
- 13. Luck SJ, Vogel EK. The capacity of visual working memory for features and conjunctions. Nature. 1997;390(6657):279–81. doi: 10.1038/36846. pmid:9384378
- 14. Habeck C, Rakitin BC, Moeller J, Scarmeas N, Zarahn E, Brown T, et al. An event-related fMRI study of the neural networks underlying the encoding, maintenance, and retrieval phase in a delayed-match-to-sample task. Cognitive Brain Res. 2005;23(2–3):207–20.
- 15. Linden DEJ. The working memory networks of the human brain. Neuroscientist. 2007;13(3):257–67. doi: 10.1177/1073858406298480. pmid:17519368
- 16. Motes MA, Rypma B. Working memory component processes: Isolating BOLD signal changes. Neuroimage. 2010;49(2):1933–41. doi: 10.1016/j.neuroimage.2009.08.054. pmid:19732840
- 17. Todd JJ, Marois R. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature. 2004;428(6984):751–4. doi: 10.1038/nature02466. pmid:15085133
- 18. Todd JJ, Marois R. Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cogn Affect Behav Ne. 2005;5(2):144–55.
- 19. Xu YD, Chun MM. Dissociable neural mechanisms supporting visual short-term memory for objects. Nature. 2006;440(7080):91–5. doi: 10.1038/nature04262. pmid:16382240
- 20. Luck SJ, Vogel EK. Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn Sci. 2013;17(8):391–400. doi: 10.1016/j.tics.2013.06.006. pmid:23850263
- 21. Vogel EK, Machizawa MG. Neural activity predicts individual differences in visual working memory capacity. Nature. 2004;428(6984):748–51. doi: 10.1038/nature02447. pmid:15085132
- 22. Vogel EK, McCollough AW, Machizawa MG. Neural measures reveal individual differences in controlling access to working memory. Nature. 2005;438(7067):500–3. doi: 10.1038/nature04171. pmid:16306992
- 23. Killebrew K, Mruczek R, Berryhill ME. Intraparietal regions play a material general role in working memory: Evidence supporting an internal attentional role. Neuropsychologia. 2015;73:12–24. doi: 10.1016/j.neuropsychologia.2015.04.032. pmid:25940098
- 24. Sternberg S. High-Speed Scanning in Human Memory. Science. 1966;153(3736):652-+. pmid:5939936
- 25. Sternberg S. Memory-Scanning—Mental Processes Revealed by Reaction-Time Experiments. Am Sci. 1969;57(4):421-+. pmid:5360276
- 26. Townsend JT, Fific M. Parallel versus serial processing and individual differences in high-speed search in human memory. Percept Psychophys. 2004;66(6):953–62. pmid:15675643
- 27. Brown J. Some tests of the decay theory of immediate memory. Quarterly Journal of Experimental Psychology. 1958;10:12–21.
- 28. Peterson LR, Peterson M. S.. Short-term retention of individual verbal items. Journal of Experimental Psychology. 1959;58(3):193–8.
- 29. Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review. 1956;63:81–97. pmid:13310704
- 30. Brady TF, Tenenbaum JB. A Probabilistic Model of Visual Working Memory: Incorporating Higher Order Regularities Into Working Memory Capacity Estimates. Psychol Rev. 2013;120(1):85–109. doi: 10.1037/a0030779. pmid:23230888
- 31. Cusak R, Lehmann M., Veldsman M., & Mitchell D.J. Encoding strategy and not visual working capacity correlates with intelligence. Psychonomic Bulletin & Review. 2009;16(4):641–7.
- 32. Brady TF, Konkle T., & Alvarez G. A.. Compression in visual short-term memory: Using statistical regularities to form more efficient memory representations. Journal of Experimental Psychology. 2009;138:487–502. doi: 10.1037/a0016797. pmid:19883132
- 33. Bays PM, Catalao RFG, Husain M. The precision of visual working memory is set by allocation of a shared resource. J Vision. 2009;9(10).
- 34. Buschman TJ, Siegel M, Roy JE, Miller EK. Neural substrates of cognitive capacity limitations. P Natl Acad Sci USA. 2011;108(27):11252–5.
- 35. Tsubomi H, Fukuda K, Watanabe K, Vogel EK. Neural Limits to Representing Objects Still within View. J Neurosci. 2013;33(19):8257–63. doi: 10.1523/JNEUROSCI.5348-12.2013. pmid:23658165
- 36. Harrison SA, Tong F. Decoding reveals the contents of visual working memory in early visual areas. Nature. 2009;458(7238):632–5. doi: 10.1038/nature07832. pmid:19225460
- 37. Serences JT, Ester EF, Vogel EK, Awh E. Stimulus-Specific Delay Activity in Human Primary Visual Cortex. Psychol Sci. 2009;20(2):207–14. doi: 10.1111/j.1467-9280.2009.02276.x. pmid:19170936
- 38. Cohen JR, Sreenivasan KK, D'Esposito M. Correspondence Between Stimulus Encoding- and Maintenance-Related Neural Processes Underlies Successful Working Memory. Cereb Cortex. 2014;24(3):593–9. doi: 10.1093/cercor/bhs339. pmid:23146963
- 39. Majerus S, Salmon E, Attout L. The Importance of Encoding-Related Neural Dynamics in the Prediction of Inter-Individual Differences in Verbal Working Memory Performance. Plos One. 2013;8(7).
- 40. Wendelken C, Bunge SA, Carter CS. Maintaining structured information: An investigation into functions of parietal and lateral prefrontal cortices. Neuropsychologia. 2008;46(2):665–78. doi: 10.1016/j.neuropsychologia.2007.09.015. pmid:18022652
- 41. Ma LS, Steinberg JL, Hasan KM, Narayana PA, Kramer LA, Moeller FG. Working memory load modulation of parieto-frontal connections: Evidence from dynamic causal modeling. Hum Brain Mapp. 2012;33(8):1850–67. doi: 10.1002/hbm.21329. pmid:21692148
- 42. Schon K, Quiroz YT, Hasselmo ME, Stern CE. Greater Working Memory Load Results in Greater Medial Temporal Activity at Retrieval. Cereb Cortex. 2009;19(11):2561–71. doi: 10.1093/cercor/bhp006. pmid:19224975
- 43. Schon K, Newmark RE, Ross RS, Stern CE. A Working Memory Buffer in Parahippocampal Regions: Evidence from a Load Effect during the Delay Period. Cereb Cortex. 2016;26(5):1965–74. doi: 10.1093/cercor/bhv013. pmid:25662713
- 44. Norcia AM, Appelbaum LG, Ales JM, Cottereau BR, Rossion B. The steady-state visual evoked potential in vision research: A review. J Vision. 2015;15(6).
- 45. Peterson DJ, Gurariy G, Dimotsantos GG, Arciniega H, Berryhill ME, Caplovitz GP. The steady-state visual evoked potential reveals neural correlates of the items encoded into visual working memory. Neuropsychologia. 2014;63:145–53. doi: 10.1016/j.neuropsychologia.2014.08.020. pmid:25173712
- 46. Kim YJ, Grabowecky M, Paller KA, Suzuki S. Differential Roles of Frequency-following and Frequency-doubling Visual Responses Revealed by Evoked Neural Harmonics. J Cognitive Neurosci. 2011;23(8):1875–86.
- 47. Ward LM. Synchronous neural oscillations and cognitive processes. Trends Cogn Sci. 2003;7(12):553–9. pmid:14643372
- 48. Canolty RT, Knight RT. The functional role of cross-frequency coupling. Trends Cogn Sci. 2010;14(11):506–15. doi: 10.1016/j.tics.2010.09.001. pmid:20932795
- 49. Bastiaansen M, Hagoort P. Event-induced theta responses as a window on the dynamics of memory. Cortex. 2003;39(4–5):967–92. pmid:14584562
- 50. Brainard DH. The psychophysics toolbox. Spatial Vision. 1997;10(4):433–6. pmid:9176952
- 51. Pelli DG. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision. 1997;10(4):437–42. pmid:9176953
- 52. Berryhill ME, Olson IR. Is the posterior parietal lobe involved in working memory retrieval? Evidence from patients with bilateral parietal lobe damage. Neuropsychologia. 2008;46(7):1775–86. doi: 10.1016/j.neuropsychologia.2008.03.005. pmid:18439630
- 53. Jiang YH, Olson IR, Chun MM. Organization of visual short-term memory. J Exp Psychol Learn. 2000;26(3):683–702.
- 54. Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM. Brainstorm: A User-Friendly Application for MEG/EEG Analysis. Comput Intel Neurosc. 2011.
- 55. Olson IR, Berryhill M. Some surprising findings on the involvement of the parietal lobe in human memory. Neurobiol Learn Mem. 2009;91(2):155–65. doi: 10.1016/j.nlm.2008.09.006. pmid:18848635
- 56. McCollough AW, Machizawa MG, Vogel EK. Electrophysiological measures of maintaining representations in visual working memory. Cortex. 2007;43(1):77–94. pmid:17334209
- 57. Sligte IG, Vandenbroucke ARE, Scholte HS, Lamme VAF. Detailed sensory memory, sloppy working memory. Front Psychol. 2010;1.
- 58. Rensink RA, O’Regan J. K., & Clark J. J.. To see or not to see: the need for attention to perceive changes in scenes. Psychol Sci. 1997;8:368–73.
- 59. Gazzaley A. Influence of early attentional modulation on working memory. Neuropsychologia. 2011;49(6):1410–24. doi: 10.1016/j.neuropsychologia.2010.12.022. pmid:21184764
- 60. Gazzaley A, Nobre AC. Top-down modulation: bridging selective attention and working memory. Trends Cogn Sci. 2012;16(2):129–35. doi: 10.1016/j.tics.2011.11.014. pmid:22209601
- 61. Pylyshyn Z. Tracking Multiple Independent Targets—Serial and Parallel Stages. B Psychonomic Soc. 1987;25(5):332–.
- 62. Oksama L, Hyona J. Is multiple object tracking carried out automatically by an early vision mechanism independent of higher-order cognition? An individual difference approach. Vis Cogn. 2004;11(5):631–71.
- 63. Drew T, Vogel EK. Neural measures of individual differences in selecting and tracking multiple moving objects. J Neurosci. 2008;28(16):4183–91. doi: 10.1523/JNEUROSCI.0556-08.2008. pmid:18417697
- 64. Jovicich J, Peters RJ, Koch C, Braun J, Chang L, Ernst T. Brain areas specific for attentional load in a motion-tracking task. J Cognitive Neurosci. 2001;13(8):1048–58.
- 65. Culham JC, Cavanagh P, Kanwisher NG. Attention response functions: Characterizing brain areas using fMRI activation during parametric variations of attentional load. Neuron. 2001;32(4):737–45. pmid:11719212
- 66. Caplovitz GP, Fendrich R, Hughes HC. Failures to see: Attentive blank stares revealed by change blindness. Conscious Cogn. 2008;17(3):877–86. doi: 10.1016/j.concog.2007.08.006. pmid:17931887
- 67. Michel CM, Murray MM, Lantz G, Gonzalez S, Spinelli L, de Peralta RG. EEG source imaging. Clin Neurophysiol. 2004;115(10):2195–222. doi: 10.1016/j.clinph.2004.06.001. pmid:15351361
- 68. Miller R. Cortico-Hippocampal Interaction: Theory, Implications and Predictions. New York: Springer-Verlag Berlin Heidelberg; 1991.
- 69. Fell J, Axmacher N. The role of phase synchronization in memory processes. Nat Rev Neurosci. 2011;12(2):105–U1500. doi: 10.1038/nrn2979. pmid:21248789
- 70. Cooper PS, Wong ASW, Fulham WR, Thienel R, Mansfield E, Michie PT, et al. Theta frontoparietal connectivity associated with proactive and reactive cognitive control processes. Neuroimage. 2015;108:354–63. doi: 10.1016/j.neuroimage.2014.12.028. pmid:25528657
- 71. Başar E, Başar-Eroglu C., Parnefjord R., Rahn E., & Schürmann M.. Evoked Potentials: Ensembles of Brain Induced Rhythmicities in the Alpha, Theta and Gamma Ranges. Induced Rhythms in the Brain. 1992:155–81.
- 72. Missonnier P, Deiber MP, Gold G, Millet P, Pun MGF, Fazio-Costa L, et al. Frontal theta event-related synchronization: comparison of directed attention and working memory load effects. J Neural Transm. 2006;113(10):1477–86. doi: 10.1007/s00702-005-0443-9. pmid:16604309
- 73. Harmony T. The functional significance of delta oscillations in cognitive processing. Frontiers in Integrative Neuroscience. 2013;7.
- 74. Basar E, Basar-Eroglu C, Karakas S, Schurmann M. Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int J Psychophysiol. 2001;39(2–3):241–8. pmid:11163901
- 75. Prada L, Barcelo F, Herrmann CS, Escera C. EEG delta oscillations index inhibitory control of contextual novelty to both irrelevant distracters and relevant task-switch cues. Psychophysiology. 2014;51(7):658–72. doi: 10.1111/psyp.12210. pmid:24673586
- 76. Lakatos P, Karmos G, Mehta AD, Ulbert I, Schroeder CE. Entrainment of neuronal oscillations as a mechanism of attentional selection. Science. 2008;320(5872):110–3. doi: 10.1126/science.1154735. pmid:18388295
- 77. Stefanics G, Hangya B, Hernadi I, Winkler I, Lakatos P, Ulbert I. Phase Entrainment of Human Delta Oscillations Can Mediate the Effects of Expectation on Reaction Speed. J Neurosci. 2010;30(41):13578–85. doi: 10.1523/JNEUROSCI.0703-10.2010. pmid:20943899
- 78. Gevins A, Smith ME, McEvoy L, Yu D. High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cereb Cortex. 1997;7(4):374–85. pmid:9177767
- 79. Fernandez T, Harmony, T., Gersenowies, J., Silva-Pereyra, J., Fernández-Bouzas, A., Galán, L., & Díaz-Comas, L. Sources of EEG activity during a verbal working memory task in adults and children. Advances in Clinical Neurophysiology, XV International Congress of Clinical Neurophysiology Supplements to Clinical Neurophysiology. Buenos Aires, Argentina.2002.