Fig 1.
Theoretical background and experimental strategy.
(A) The predictive coding theory of perception describes perception as the inference made about the state of the external world and the causes of the sensory input. In the predictive coding model we appeal to in this study, expectations allow predictive signals to descend from higher to lower levels in the cortical hierarchy, at which they are tested against sensory information. The discrepancy between the two—the prediction error—propagates up the hierarchy, allowing for the higher-level expectations and subsequent predictions to be optimised. Expectations and prediction errors are suggested to be coded within each hierarchical level by distinct neural populations referred to as S and E units, respectively [20,21]. Prediction errors are suggested to be weighted by their estimated (and expected) precision such that high precision estimates lead to enhancement of prediction error signals via synaptic gain mechanisms. (B) Conceptual figure of the three experiments used to comprehensively examine the role of expectation and attention in the integration of top-down and bottom-up signals. In Experiment 1 (red oval), expectation was manipulated while the visual stimuli were kept constant across conditions. In Experiment 2, attention was manipulated while expectation and the visual stimuli were held constant across conditions (green oval). In Experiment 3, we applied a novel analysis method to the data obtained in a previous study (22) in order to simultaneously examine main effects of expectation and attention as well as the interaction between the two (yellow hourglass shape). For consistency, all figures in Results use shades of red for expectation and shades of green for attention. E, error; S, state.
Fig 2.
Stimulus construction and analysis methodologies.
(A) All experiments implemented the HFT method using face and house images. SWIFT sequences are presented at given frequencies, allowing tagging of image-recognition activity (red rectangles). Contrast modulation is applied at a higher frequency, inducing SSVEP (blue sinusoid). When analysing the EEG data in the frequency domain (bottom graph with multiple peaks), peaks in the power spectrum can be seen at the fundamental frequencies and their harmonics (red bars for SWIFT f1 and blue bars for SSVEP f2). Additional peaks at IM components (e.g., purple bars for f2 + f1 and f2 − f1) are suggested to indicate integration of bottom-up SSVEP signals with top-down SWIFT signals. (B) The MSPC [36] quantifies the degree to which an IM frequency component is driven by the phases of the fundamental input frequencies. In other words, the degree to which the IM component reflects an interaction between those input frequencies. Within each epoch, we first calculate the difference between the sum of the (weighted) phases of the fundamental input frequencies and the phase of the IM component. Then, we compute the coherence of this value across multiple epochs applying the same method as in the well-known phase-locking value (see Methods for a detailed description.) Here, we introduced a novel distinction between two measures—MSPCstim and MSPCres—which differ in what we consider the ‘input’ signals to be. Specifically, the MSPCstim ties the IM phase to the phases of the stimulus itself (i.e., the images presented on the screen), while MSPCres ties it to the phases of the tagged neural responses (See Methods). We suggest that these measures distinguish between neural interactions occurring at lower and higher cortical levels, respectively. EEG, electroencephalography; HFT, hierarchical frequency tagging; IM, intermodulation; MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus); SSVEP, steady-state visual evoked potentials; SWIFT, semantic wavelet-induced frequency-tagging.
Fig 3.
Expectation modulates MSPCstim but not MSPCres.
MSPC averaged across the two second-order IM components (f1 + f2 and f1 − f2) in the expected and the unexpected conditions. Results are shown for a posterior ROI (17 electrodes, top) and the scalp topographies (bottom). Error bars represent standard error across subjects (N = 15). The MSPCstim measure (left) quantifies IM responses by examining the degree to which the IM phase is driven by the phases of SWIFT and SSVEP stimulus (image) modulation. Conversely, the MSPCres measure (right) examines the degree to which the IM phase is driven by the tagged SWIFT and SSVEP neural response phases. These measures are therefore suggested to indicate signal integration occurring at earlier and at later stages of cortical processing, respectively. MSPCstim (left) was higher for the PV (expected) trials compared to the IR (unexpected) trials (χ2 = 22.9, P < 0.001), indicating increased neural integration between the SWIFT and SSVEP signals when stimuli are expected. This effect was not evident for the MSPCres measure (χ2 = 1.36, P > 0.05). The data underlying this figure is available in FigShare at DOI: 10.26180/5b9abfe5687e3. IM, intermodulation; MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus); SSVEP, steady-state visual evoked potentials; SWIFT, semantic wavelet-induced frequency-tagging.
Fig 4.
Attention modulates MSPCres but not MSPCstim.
MSPC averaged across the two second-order IM components of the attended and the unattended images. (Note that within each trial, different SWIFT frequencies were used for the attended and the unattended images, each resulting in a different set of second-order IMs.) Results are shown for a posterior ROI (17 electrodes, top) and the scalp topographies (bottom). Error bars represent standard error across subjects (N = 11). The MSPCstim measure (left) quantifies IM responses by examining the degree to which the IM phase is driven by the phases of the SWIFT and SSVEP stimulus (image) modulations. Conversely, the MSPCres measure (right) examines the degree to which the IM phase is driven by the tagged SWIFT and SSVEP neural response phases. MSPCres (right) was higher for the attended compared to the unattended images (χ2 = 41.4, P < 0.001), indicating increased neural integration between the SWIFT and SSVEP signals when stimuli are attended. This effect was not evident for the MSPCstim measure (χ2 = 1.21, P > 0.05). The data underlying this figure is available in FigShare at DOI: 10.26180/5b9abfe5687e3. IM, intermodulation; MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus); ROI, region of interest; SSVEP, steady-state visual evoked potentials; SWIFT, semantic wavelet-induced frequency-tagging.
Fig 5.
The expectation–attention interaction is significant for MSPCres but not MSPCstim.
Predicted MSPCstim (left) and MSPCres (right) values obtained from a full Linear Mixed Effects interaction model, with their standard error indicated by the shaded area. The model included expectation, attention, and an expectation–attention interaction term as the fixed effects, while the random effects included a random intercept for frequency nested within channels nested within participants and random expectation and attention slopes for each participant. Consistent with the colours used in the previous figures, attended images are represented by the dark green lines and unattended images by the light green lines, while the pink–red gradient indicates increasing expectation. The significance of the interaction term was tested using the likelihood ratio test between the full model and the reduced model, which excluded the interaction fixed effect. The expectation–attention interaction was not significant for MSPCstim (χ2 = 3.47, P < 0.05) but was highly significant for MSPCres (χ2 = 19.56, P < 0.001). The data underlying this figure is available in FigShare at DOI: 10.26180/5b9abfe5687e3. MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus).
Fig 6.
Expectation, attention, MSPC, and predictive coding.
Results presented in this study can be accounted for by the predictive coding framework of perception as follows: 1) Expectation (the probability for the appearance of specific stimuli) relates to descending prediction signals. 2) Better predictions (as afforded by the PV trials in Experiment 1) increases the efficiency of top-down and bottom-up signal integration at low-level visual areas, as reflected by the increased MSPCstim with expectation (Figs 2 and 4). 3) Attention reflects a (precision-weighted) control mechanism for the propagation of prediction error signals. 4) Attention effectively increases the influence of prediction error signals on expectations at higher hierarchical levels, as reflected by the increased MSPCres with attention (Figs 3 and 4). 5) The effect of expectation on the integration of top-down and bottom-up information at lower visual areas is less dependent on attention than the integration at higher levels. Hence, while MSPCstim increased with expectation for both attended and unattended stimuli (Fig 5), the influence of expectation on MSPCres was attention dependent (Fig 5). MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus); PV, pattern violation.
Fig 7.
Multispectral phase coherence.
The method for calculating MSPC. (A) Schematic example of stronger (left) and weaker (right) phase coherence. Both the PLV and the MSPC measures examine the consistency of a given phase term across multiple epochs. This can be visualised by first converting the phase term from each epoch into a unit (length = 1) vector pointing at its phase angle. Phase coherence is then obtained by computing the average vector (the sum of all vectors divided by the number of epochs). The result is a vector whose length can vary from 0 (each vector pointing at random directions, no phase consistency across epochs) to 1 (all vectors pointing at the same direction, perfect consistency across epochs). (B) The primary difference between PLV and MSPC measures is the phase term used for each epoch. For the PLV, only the phase of one specific frequency (or frequency band) is extracted for each channel/epoch and is used as the phase term for computing phase locking. When examining consistency between distant channels, the phase term used for the PLV would be the difference (Δφ) between the phases extracted from the different channels. In contrast, the phase term used for the MSPC calculations here is based on the difference (Δφ) between the phase of a specific IM component and the (weighted sum of) phases of the fundamental frequencies within each channel/epoch. In other words, the MSPC can be understood as a measure of the extent to which the IM phases are driven by the phases of the fundamental frequencies. Our distinction between MSPCstim and MSPCres is reflected by the two MSPC formulas shown at the bottom of the figure. Specifically, MSPCstim defines the phases of the stimuli (the on-screen image) as the input (or ‘driving’) fundamental frequencies (left formula, upper case F1 and F2), and MSPCres defines the EEG response phases as the input fundamental frequencies (right, lower case f1 and f2). Note that the weights of the fundamental frequencies in those formulas (n1 and n2) are the coefficients that define the IM frequency (e.g., given F1 = 1.2 Hz and F2 = 15 Hz, the weights for the third-order IM component 2*F1 + F2 = 17.4 Hz would be n1 = 2 and n2 = 1). EEG, electroencephalography; IM, intermodulation; MSPCres, multispectral phase coherency (response); MSPCstim, MSPC (stimulus); PLV, phase-locking value.