High-level visual prediction errors in early visual cortex
Fig 5
Prediction error magnitude scales with high-level visual feature surprise.
(A) Whole-brain results assessing the modulation of surprise responses as a function of high-level (top row) and low-level (bottom row) visual feature dissimilarity. The top row shows that surprise responses to unexpected images were increased if the image was more distant from the expected image in terms of high-level visual features. Color indicates the beta parameter estimate of the parametric modulation, with red and yellow representing increased responses. Black outlines denote statistically significant clusters (GRF cluster corrected). No significant modulation of sensory responses was observed by low-level visual surprise. (B) ROI analysis zooming in on ROIs in early visual (V1), intermediate (LOC), and HVC (encompassing occipito-temporal sulcus and fusiform cortex). Results mirror those of the whole-brain analysis, with significant modulations of the visual responses by high-level visual surprise (red), but not low-level visual surprise (blue). Error bars indicate the 95% within-subject confidence intervals. Gray dots denote individual subjects. P values are FDR corrected. *** p < 0.001, ** p < 0.01, * p < 0.05, = BF10 < 1/3. (C) Prediction errors preferentially scale with high-level visual features (layer 8 and layer 7) throughout most of the visual system, including EVC, LOC and HVC. Color indicates the DNN layer with the largest effect (explained variance) on scaling the neural responses to surprising inputs. Cold colors (purple–blue) represent early layers (i.e., low-level visual features), while warm colors (yellow–red) indicate late layers (i.e., high-level visual features). Analysis was masked to visual cortex and thresholded at a liberal z ≥ 1.96 (i.e., p < 0.05, two-sided) to explore the landscape of prediction error modulations across DNN layers. Results strongly contrast with those observed for prediction-free visual responses during the localizer (Fig 3A). (D, E) ROI analysis regressing BOLD responses (D) or decoded true class probability (E) onto high-level visual dissimilarity. Results indicate a monotonic relationship between high-level surprise and BOLD responses across all 3 ROIs, as well as decoding performance in V1 and HVC. The chance level for decoding the true class probability is 0.125. For display purposes dissimilarities were ranked and averaged across participants, while regression models were fit per participant on the correlation distances. Data and code that support these findings are available at: https://doi.org/10.34973/8e49-2012. DNN, deep neural network; EVC, early visual cortex; HVC, higher visual cortex; LOC, lateral occipital complex; ROI, region of interest; V1, primary visual cortex.