The time course of category representations: Decision localizer

To investigate the time course of category-specific neural representations, we trained classifiers on EEG data recorded during the 2 localizer tasks and applied them to the main perceptual discrimination task data. We will first report the classification when classifiers were trained on the decision localizer (Fig 2A) and then the cross-classification to the main task (Fig 2B). Category-specific neural representations could be decoded based on the decision localizer (10-fold validation scheme, i.e., within-task decoding, Fig 2A). The generalization across time (GAT) matrix, time-locked to stimulus onset, exhibited the expected mixture of transient and stable on-diagonal decoding as well as persistent off-diagonal decoding profiles observed previously [33–37]. Following the analysis approach by Weaver and colleagues [33], who used a similar face/house discrimination task, we used matching time-windows for statistical analysis as used in that earlier study. For these analyses, based on Weaver and colleagues [33], decoding was examined on several stages, first of which a transient peak in an early time-window between 150 and 200 ms (Fig 2A, inset 2) [11,13–15]. The timing (150 to 200 ms; the decoding peak was found at approximately 166 ms) and scalp topography of the early peak is probably related to the N170 ERP component, often related to face processing [36,38–40], although also observed for letters, words, and biological motion [41,42]. Peaks of N170-like decoding have been associated with neural processes in the occipital face area, superior temporal sulcus, and/or the fusiform face area (FFA) in ventral-temporal cortex [43–49]. Previous work has also shown that the strength of this early face-selective component is proportional to the strength (e.g., phase coherence of the stimulus) of the presented stimulus [18,40].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

(A) GAT matrix for the decision localizer (all electrodes) and ROI marked by inset black boxes (numbers 1–4). On the right, covariance/class separability maps for each diagonal ROI, indicating the underlying data contributing to the classification performance. Highlighted channels significantly contributed to the classification performance after cluster-based permutation testing over channels (corrected for multiple comparisons). Covariance/class separability maps 2 and 4 are the same because decoding scores within insets 2 and 4 were obtained using the same training time-window. (B) Classifiers trained on the decision localizer applied to the main discrimination task (cross-task validation procedure, all electrodes), revealing a high degree of generalizability between the 2 tasks. The underlying data and scripts supporting this figure can be found on FigShare (https://doi.org/10.21942/uva.c.6265233.v1). AUC, area under the curve; GAT, generalization across time; ROI, regions of interest.

https://doi.org/10.1371/journal.pbio.3002009.g002

We also focused on a more stable square-shaped decoding profile between 350 and 500 ms (on-diagonal), capturing late category-specific processes (Fig 2A, inset 3), with a central-parietal topography (late-stage topography). This decoding profile has previously been suggested to reflect global ignition, large-scale feedback processes allowing information to be broadcasted throughout the entire brain, making information explicit for report and decision-making [8,34,37,50].

Additionally, we examined decoding accuracy in a late time-window from 350 to 500 ms, based on training on early 150 to 200 ms classifiers (i.e., off-diagonal decoding, (Fig 2A, inset 4)). Given that decoding is in this case based on early classifiers tuned to sensory features, this stage is thought to reflect longer lasting sensory processes of categorical information across time [33,34,51]. This late-latency off-diagonal decoding pattern, referred to as the “perceptual maintenance” stage, has been suggested to reflect reactivation of early sensory stages through feedback processes [33–35].

Finally, although decoding was pronounced during 3 time-windows observed previously by Weaver and colleagues [33] (insets 2, 3, and 4), it was also observed at an even earlier time-window (80 to 130 ms) with a prominent occipital–parietal topography (Fig 2A, inset 1, early-stage topography). Because this early effect was absent in the same task when images where presented vertically [33], this early peak likely reflects differences in the orientations of the images presented, which we varied systematically across trials (note that training and testing was performed on balanced set of left and right oriented images). Although we follow a confirmatory approach by selecting time-windows of interest (TOI) beforehand [33], we will always in addition use across time permutation tests with multiple comparisons corrections to test for (unexpected) effects observed at other time-windows and to further specify the approximate onset and/or duration of our significant effects (see Materials and methods for details).

Cross validation between the decision localizer and the main discrimination task (Fig 2B for all channels; see S2 Fig for occipital–parietal channels) revealed a highly similar decoding profile for the 2 tasks. Classifiers’ performance in differentiating between face and house images were highest on occipital–parietal electrodes, particularly during early latencies (S2 Fig). Since our aim was to examine whether decision errors have a perceptual origin, we report the decoding results based on data from occipital–parietal channels here to maximize sensitivity to neural signals that distinguish between the 2 categories. In the S1 Text, S4 Fig, and S5 Fig, we also report the critical decoding analyses using all channels, which yielded similar results.

Diagonal decoding based on decision localizer

Next, we examined decoding scores as a function of decision correctness (correct/incorrect) and confidence (low confidence (LC): level 1 and 2 versus high confidence (HC): level 3 and 4, as in [33]). This created 4 conditions: HC correct, LC correct, HC incorrect, and LC incorrect trials. GAT matrices and on/off-diagonal time courses for these 4 conditions are shown in Fig 3A–3D. Note that the “testing labels” for classification were based on the decision of the subject in the main task (deciding face or house), not the presented stimulus. For correct trials, the presented stimulus and the decision are by definition the same. For incorrect trials however, the stimulus and the decision are of different categories. This means that if decoding scores are observed that are higher than chance level, the underlying data pattern correlates with the category of the reported stimulus, the decision. We will refer to this situation as positive decoding (with respect to chance level). In contrast, when below chance level decoding is observed, this would reveal that the underlying data pattern correlates with the presented stimulus, not with the reported stimulus (stimulus and response are different on incorrect trials, see Methods for details). The classifier thus finds negative evidence for the decision, and therefore we will refer to this situation as negative decoding (with respect to chance level).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Face/house classification in the main task while training on the decision localizer.

(A) GAT matrices for correct trials and (B) incorrect trials in the main discrimination task (occipital–parietal channels). Trials are separated by confidence—HC and LC—in the decision. (C) On-diagonal and (D) off-diagonal (perceptual maintenance) time courses of correct and incorrect decisions, separated by confidence. Black boxes indicate the time-windows (early, mid, late) used for averaging the AUC scores. Colored horizontal lines indicate periods of significant decoding with respect to chance. Longer-lasting sensory traces are derived by training a classifier on the time-window 150–200 ms and testing it across the entire time-window (panel D). Classification performance was evaluated at each time point using cluster-based permutation testing (two-tailed cluster-permutation, alpha p < 0.05, cluster alpha p < 0.05, N permutations = 1,000). (E) Bar plots show average AUC values for the TOI highlighted in panels (C) and (D). The underlying data and scripts supporting this figure can be found on FigShare (https://doi.org/10.21942/uva.c.6265233.v1). AUC, area under the curve; GAT, generalization across time; HC, high confidence; LC, low confidence; TOI, time-windows of interest.

https://doi.org/10.1371/journal.pbio.3002009.g003

Statistical analyses were performed on the diagonal and off-diagonal time courses (late sensory traces) (Fig 3C and 3D). We first report the analyses for the diagonal time courses using a 3-way repeated measures ANOVA with the factors Latency, Correctness, and Confidence. As anticipated, category decoding was better for correct than incorrect trials (F_1,39 = 135.05, p < 0.001, η_p² = 0.78) as well as for HC versus LC trials (F_1,39 = 27.75, p < 0.001, η_p² = 0.42). Category decoding differed across the 3 (early, mid, late) diagonal decoding time-windows (F_2,78 = 32.16, p < 0.001, η_p² = 0.45), with differences being modulated by the level of confidence (Latency × Confidence: F_2,78 = 6.88, p = 0.002, η_p² = 0.15; no evidence for an interaction between confidence and correctness: F_2,78 = 0.022, p = 0.88, η_p² = 0.001). As hypothesized, the characteristic pattern of decoding across 3 diagonal time-windows differed between correct and incorrect trials (Correctness × Latency interaction: F_2,78 = 36.57, p < 0.001, η_p² = 0.48), independent of confidence (Correctness × Latency × Confidence interaction: F_2,78 = 1.12, p = 0.33, η_p² = 0.03). As shown in Fig 3C, diagonal decoding profiles were very different for correct and incorrect decisions, especially for the early and mid-processing stages. Most prominently, it can be observed that for correct trials, all processing stages show positive decoding, whereas in sharp contrast the 2 early stages for the incorrect trials showed negative decoding. This negative decoding pattern on incorrect decisions illustrates that early EEG data patterns look more similar to the presented stimulus category than to the reported stimulus category (note again that the decisions were used as the classifier labels). The fact that the late stage decoding flips to positive decoding illustrates that later EEG data patterns are more similar to the reported category than the presented category (Fig 3C). We will come back to these effects later in the Results section.

For correct trials, decoding strength differed across the 3 TOI’s (F_1,39 = 47.01, p < 0.001, η² = 0.55) and was overall better for HC trials than for LC trials (F_1,39 = 31.91, p < 0.001, η² = 0.45). Confidence interacted with latency (TOI’s) (F_2,78 = 10.28, p < 0.001, η² = 0.21), indicated by better decoding in the mid/late processing stage for HC versus LC trials (mid-stage: t₃₉ = −6.92, p < 0.001, d = −1.09; late stage: t₃₉ = −3.06, p = 0.004, d = −0.48; Fig 3C), while we did not find evidence for this confidence modulation for the early processing stage (t₃₉ = −1.25, p = 0.22, d = −0.2). These results are summarized in the bar plots of Fig 3E.

On incorrect decisions, decoding scores across 3 diagonal time-windows differed (F_2,78 = 18.84, p < 0.001, η_p² = 0.33) and while decoding was also higher for HC than for LC trials (F_1,39 = 11.64, p = 0.002, η_p² = 0.23, Fig 3C), the interaction between these 2 factors was not robust, although numerically, decoding was higher in HC compared to LC trials, in particular at the later versus early stage (F_2,78 = 2.81, p = 0.07, η_p² = 0.07, BF_excl = 1.96). A series of planned simple comparisons showed that the flip from negative to positive decoding was stronger for high confidence decisions (results are summarized in Fig 3E). The below-chance decoding during the 2 early diagonal decoding time-windows on incorrect decisions indicated that the classifiers were picking up the presented stimulus category during sensory stages of visual information processing. At the earliest time-windows (80 to 130 ms), we did not find evidence that decoding was modulated by confidence (HC versus LC; t₃₉ = −0.28, p = 0.78, d = −0.05, BF₀₁ = 5.64) and indeed, decoding was below chance-level for both confidence levels (LC: t₃₉ = −5.83, p < 0.001, d = −0.92; HC: t₃₉ = −3.38, p = 0.002, d = −0.53). During the following mid-stage (150 to 200 ms), classifiers also mainly decoded the veridical stimulus category, but interestingly in particular when participants reported low confidence in their decision (LC: t₃₉ = −5.05, p < 0.001, d = −0.8; HC: t₃₉ = −0.35, p = 0.73, d = −0.06, BF₀₁ = 5.54; LC versus HC: t₃₉ = −2.95, p = 0.005, d = −0.47). Then finally, during the late processing stage (350 to 500 ms), patterns of neural activity represented the incorrectly reported stimulus category, but in particular when participants where highly confident in their decision, so when the illusion was strongest (HC: t₃₉ = 3.53, p < 0.001, d = 0.56; LC: t₃₉ = 0.44, p = 0.67, d = −0.07, BF₀₁ = 5.36, LC versus HC: t₃₉ = −3.28, p = 0.002, d = −0.52). These results show a reversal from the presented stimulus category to the reported category for incorrect decisions, especially for high confidence decisions.

Off-diagonal decoding based on the decision localizer

Next, we examined decoding of the reported stimulus category off-diagonally, training on the N170 time-window (150 to 200 ms) using the decision localizer task data, separately for correct and incorrect decisions (only 1 stage). First, on correct decisions, decoding of the reported stimulus category was robust for both confidence levels (all p’s < 0.001), although decoding was higher for HC than LC decisions (t₃₉ = −3.98, p = <0.001, d = −0.63; Fig 3D and 3E, right panel), suggesting that the veridical/perceived stimulus category was maintained perceptually. On incorrect decisions, however, what was perceptually represented off-diagonally was not the presented stimulus, but the misreported stimulus category, and this effect was stronger for high confident than low confident incorrect decisions (HC: t₃₉ = 3.86, p < 0.001, d = 0.61; LC: t₃₉ = 1.41, p = 0.17, d = 0.22, BF₀₁ = 2.35; LC versus HC: t₃₉ = −3.06, p = 0.004, d = −0.48; Fig 3E). Thus, based on the transient decoding peak observed at early processing stages (150 to 200 ms in the decision localizer), the erroneous and misreported stimulus category was represented in a persistent perceptual format later in time, despite never being presented on the screen and never being neurally represented at earlier time points (Fig 3D). Statistical tests across time revealed that the incorrect off-diagonal decoding trace emerged relatively early, starting at approximately 213 ms and peaking at 236 ms after stimulus presentation (Fig 3D, horizontal blue colored bars reflect significant time points after cluster-based permutation testing). Although we observed a perceptual representation of an incorrectly reported category, concluding that errors in decision-making in this task have a true perceptual origin (in the case confidence is high), based on the current results may be premature, because the decoded signal could reflect a combination of perceptual and decision-related processes [20,52]. Therefore, we further examined this issue when training our classifiers on the sensory tuned localizer.

The time course of category representations: Sensory localizer

To explore to what extent errors in decision-making have a true perceptual origin, we performed the same set of analysis, but now while training the classifier on the sensory localizer task (Fig 4A). In this task, participants performed a simple change detection task on the color of the fixation cross making the face/house images fully task irrelevant. Because the sensory localizer task does not have a decision component to the face/house images, it is uniquely sensitive to sensory features. Therefore, we expected that the late square-shaped on-diagonal decoding pattern, possibly related to global ignition [34,37,50], would be strongly reduced or even disappear when cross classifying between the sensory localizer and the main perceptual decision task (Fig 4B). Previous studies have shown that this late square-shaped decoding pattern is likely related to decision processes arising after perceptual analysis, which disappear when visual input is task irrelevant [18–20,53]. On the other hand, because decoding is still performed on the face/house images, the sensory stages should be relatively unaffected by this task relevance manipulation [18,49]. In this Results section, we focus on those aspects of decoding that are informative for addressing the question whether decision errors have a perceptual origin, but note that all other effects were similar to the analyses reported in the “decision localizer” section and the details thereof can be found in the Supporting information (including all the ANOVA’s).

For correct decisions, decoding was above chance for both confidence levels (all p’s < 0.001, see Fig 4C) in the early and mid-stage, but we did not find evidence for decoding at the late diagonal stage (LC: t₃₉ = 1.104, p = 0.28, d = 0.18, BF₀₁ = 3.33; HC: t₃₉ = 1.34, p = 0.19, d = 0.21, BF₀₁ = 2.55). Thus, as anticipated, the late on-diagonal 350 to 500 ms stage was not robust when training on the sensory localizer and classifying on the main discrimination task, whereas all other stages of visual processing remained relatively intact (Fig 4C). During the perceptual maintenance stage, the activity patterns reflected the veridical/reported stimulus category (both p’s < 0.001, LC versus HC: t₃₉ = −0.98, p = 0.34, d = −0.15, BF₀₁ = 3.76, Fig 4D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Face/house classification based on the sensory localizer.

(A) GAT matrix for the sensory localizer (all electrodes) and ROI marked by inset black boxes (numbers 1–4). On the right, covariance/class separability maps for each diagonal ROI, indicating the underlying data contributing to the classification performance. Highlighted channels significantly contributed to the classification performance after cluster-based permutation testing over channels (corrected for multiple comparisons). Covariance/class separability maps 2 and 4 are the same because decoding scores within insets 2 and 4 were obtained using the same training time-window. (B) Classifiers trained on the sensory localizer applied to the main discrimination task (cross-task validation procedure, all electrodes). (C) On-diagonal and (D) off-diagonal time courses of correct and incorrect decisions, separated by confidence. Black boxes indicate the time-windows (early, mid, late) used for averaging the AUC scores. Colored horizontal lines indicate periods of significant decoding with respect to chance. Longer-lasting sensory traces are derived by training a classifier on the time-window 150–200 ms and testing it across the entire time-window (panel D). Classification performance was evaluated at each time point using cluster-based permutation testing (two-tailed cluster-permutation, alpha p < 0.05, cluster alpha p < 0.05, N permutations = 1,000). (E) Bar plots show average AUC values for the TOI highlighted in panels (C) and (D). The underlying data and scripts supporting this figure can be found on FigShare (https://doi.org/10.21942/uva.c.6265233.v1). AUC, area under the curve; GAT, generalization across time; HC, high confidence; LC, low confidence; ROI, regions of interest; TOI, time-windows of interest.

https://doi.org/10.1371/journal.pbio.3002009.g004

For incorrect decisions, the early and mid-stage were also similar as compared to when the classifiers were trained on the decision localizer (early stage: LC: t₃₉ = −3.4, p = 0.002, d = −0.54; HC: t₃₉ = −3.91, p < 0.001, d = −0.62; mid-stage: LC: t₃₉ = −3.04, p = 0.004, d = −0.48; HC: t₃₉ = −0.11, p = 0.91, d = −0.02, BF₀₁ = 5.83). Specifically, we observed negative decoding for both early stages (again, except for HC trials in the mid-stage), reflecting the processing of the veridical stimulus input. Crucially, also here, the late-stage decoding disappeared (LC: t₃₉ = 0.16, p = 0.88, d = −0.03, BF₀₁ = 5.8; HC: t₃₉ = 1.27, p = 0.21, d = 0.2, BF₀₁ = 2.8, Fig 4C). However, most importantly, during the off-diagonal perceptual maintenance stage using the classifier trained on the sensory localizer data, we could still decode the incorrectly reported stimulus category, in particular when participants reported high confidence in their decision (LC: t₃₉ = 1.78, p = 0.08, d = 0.28; BF₀₁ = 1.4; HC: t₃₉ = 3.03, p = 0.004, d = 0.48, BF₀₁ = 0.12; LC versus HC: t₃₉ = −1.94, p = 0.06, d = −0.31, BF₀₁ = 1.07, Fig 4D and 4E). Again, here as well, cluster-based testing across the entire time-window revealed that the perceptual representation of the incorrect stimulus spontaneously emerged right after the 150 to 200 ms time-window most strongly representing face/house differentiation (see Fig 4D). This finding shows that misreported decisions may be due to true perceptual illusions.

Stimulus orientation decoding

Our results so far show that initial information processing stages are stimulus related, whereas later off-diagonal decoding captures perceptual aspects of decision errors. To show that these effects are specific for task-relevant features of the images (face/house category), we additionally focused on stimulus orientation decoding as a function of correctness and confidence. Stimulus orientation was a task-irrelevant feature of the stimuli and previous studies have shown that it can be decoded already approximately 100 ms post-stimulus [50]. A single isolated decoding peak within the 30 to 130 ms post-stimulus was observed both for within-task decoding (Fig 5A, 10-fold decoding) as well as cross-task classification (decision classifiers ➔ main discrimination task, Fig 5B and 5C). Decoding strength was similar for correct and incorrect decisions (F_1,39 = 0.06, p = 0.82, η_p² = 0.001, BF₀₁ = 5.7) and more evidence was observed for the absence of an effect. We also did not find evidence that confidence modulated orientation decoding in any way (F_1,39 = 0.54, p = 0.47, η_p² = 0.014, BF₀₁ = 4.77; interaction correctness × confidence: F_1,39 = 0.18, p = 0.67, η_p² = 0.005, BF_excl = 4.22).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Tilt (right/left) classification based on decision localizer.

(A) GAT matrix of the decision localizer data for arbitrating left/right tilted stimuli (irrespective of face/house category, based on occipital–parietal channels). The black square-shaped inset indicates the time-window used for averaging AUC decoding scores (30–130 ms). (B) On-diagonal time courses for correct and incorrect decisions separated by confidence. Colored horizontal lines indicate periods of significant decoding with respect to chance. Classification performance was evaluated at each time point using cluster-based permutation testing (two-tailed cluster-permutation, alpha p < 0.05, cluster alpha p < 0.05, N permutations = 1,000). (C) Bar plots show average AUC values for the time-window of interest highlighted in panels (A) and (B). The underlying data and scripts supporting this figure can be found on FigShare (https://doi.org/10.21942/uva.c.6265233.v1). AUC, area under the curve; GAT, generalization across time; HC, high confidence; LC, low confidence.

https://doi.org/10.1371/journal.pbio.3002009.g005

Ethics statement

Participants provided written informed consent prior to the start of the experiment and were tested following a protocol approved by the ethical committee of the Department of Psychology of the University of Amsterdam (project number: 2019-BC-10724).

Participants

A total of 44 participants (33 female, mean age = 23.34; SD = 5.68) from the University of Amsterdam, all right-handed, with reported normal or corrected-to-normal vision and no history of a psychiatric or neurological disorder, were tested in this study. Participants received research credits or money (10 euros per hour) for compensation. Forty participants, all between the age of 18 and 35, completed 2 experimental sessions of this study and comprised the final sample included in all reported analyses (30 female, mean age = 22.83; SD = 3.31). The remaining 4 participants were identified based on their data from session 1, after which they were excluded from further participation/analyses. One participant was excluded from the final analysis because of surpassing the age criterion (age > 35), while the other 3 participants were excluded due to the around-chance performance in 1 or both localizer tasks in the first session.

Materials

All tasks used in the current study were developed and executed using Matlab 8 and Psychtoolbox-3 software within a Matlab environment (Mathworks, RRID:SCR_001622). Stimuli were presented on 1,920 × 1,080 pixels BenQ XL2420Z LED monitor at a 120-Hz refresh rate on a black (RGB: [0 0 0], ± 3 cd/m²) background and were viewed with a distance of 90 cm from the monitor using a chin rest.

Procedure and stimuli

The experiment consisted of 2 sessions, each approximately 3 h long and scheduled on 2 different days for each participant. In the first session, participants completed a change detection task and a discrimination task while we measured their brain activity using EEG. In the change detection task, participants focused on the central fixation cross and reported its color change whenever it changed from red to light red, while a stream of brief house and face images was shown simultaneously on the screen (S1A Fig). In the discrimination task, in every trial, participants saw a masked image of a face or a house (S1A Fig). At the end of each trial, they were asked to indicate which stimulus category they perceived. The discrimination task administered in the second session was highly similar (see the details of the design below), but in addition to reporting which stimulus category they perceived, participants also provided confidence ratings regarding their decision.

MVPA classifiers were trained on EEG data recorded during the first session, separately for the change detection task (“sensory” localizer) and the discrimination task (“decision” localizer), which were then applied to the discrimination task EEG data recorded in the second session.

Session 1: Change detection task (sensory localizer). In this task, participants focused their attention on a centrally presented fixation cross, which was superimposed on a rapidly changing sequence of house and face images (S1A Fig). The fixation cross remained present on the screen throughout a block of trials. Participants’ task was to monitor the fixation cross and report its brief color change, from red (RBG: [255 0 0]) to a lighter shade of red (RBG: [125 0 0]) for 100 ms on 20% of randomly chosen trials. Participants were instructed to press the spacebar whenever they noticed the color change of the fixation cross. Note that this change detection task adds an orienting response to the fixation cross on some trials, which was however unrelated to, and not predictive of, the occurrence of the task-irrelevant face or house stimuli.

Stimuli set consisted of 180 unique houses and 180 unique faces (90 male and 90 female). Face and house images were gray scale stimuli, obtained from [33]. Face and house stimuli were equated for spatial frequency and luminance (for details, see [33]). All stimuli subtended 16 × 20° visual angle, were presented centrally on a black background, and were tilted to the right or left at a 5° or 355° angle, respectively. Each image was shown for 100 ms with the ISI that was jittered between 1,200 and 1,400 ms. The task-relevant color change of the fixation cross could occur only in the ISI, at a randomly determined moment 15 ms after the start of the ISI and 130 ms before its end. Participants were instructed not to pay attention to images of houses and faces while maintaining their fixation at the center of the screen. Images of houses and faces were thus task irrelevant, although they were concurrently processed visually.

By ensuring that participants’ attention was focused on the centrally presented task, the aim was to minimize the possibility that MVPA classifiers were impacted by systematic eye movements that could be strategically deployed to discriminate between stimuli classes [94]. For example, a consistent eye movement towards the top of the stimulus to detect face-defining features (e.g., eyes) could alone drive multivariate differences between houses and faces stimuli.

In total, 200 images of houses and faces (100 of each category, half tilted to the right and half tilted to the left) were shown in each of 6 experimental blocks. Before the start of the first experimental block, each participant completed 1 practice block of 100 trials in order to get familiar with the task.

Session 1: Discrimination task (decision localizer). The task design was highly similar to the task developed by Weaver and colleagues [33]. An overview of the trial procedure can be found in Fig 1A (and S1B Fig). Each trial started with a fixation dot that remained on the screen throughout the duration of the trial. After a fixation-only interval that was jittered between 600 and 1,000 ms, a scrambled mask stimulus appeared on the screen for 50 ms, followed by a target face or a house stimulus, shown for 80 ms. The target image was followed by another 50 ms-long scrambled mask stimulus. A response screen was presented for 1,000 ms immediately after the offset of the second mask stimulus, during which participants needed to give a speeded response indicating whether they saw a house or a face, using a left-handed (“z”) or right-handed (“m”) keyboard response. Stimulus-response mappings were randomized across blocks of trials to prevent motor response preparation before the response screen was shown. The correct stimulus-response mapping (e.g., a left button press for a face and a right button press for a house stimulus) was presented at the beginning of each block. Furthermore, letters F and H (approximate size 4 × 4°), for faces and houses, respectively, were shown during the entire trial in the right and left upper corner (centered approximately 20 × 24° from fixation), or vice versa, depending on the block, as a reminder of the response mapping to the button press.

Face, house, and mask stimuli were obtained from the study by Weaver and colleagues [33]. Stimuli set consisted of 180 unique houses and 180 unique faces of which 90 faces were male and 90 were female. Face and house stimuli were equated for spatial frequency and luminance (for details see [33]). Visual masks were selected from 900 scrambled face and house images (parsed into 12 × 15 tiles and randomly shuffled) that had been made transparent and superimposed. All stimuli were gray scale, subtended 16 × 20° visual angle, and were presented centrally on a black background. Target stimuli and masks were shown tilted to the right or left, at a 5° or 355° angle, respectively.

Participants completed 14 experimental blocks containing 64 trials each, 896 trials in total. In each block, an equal number of faces and houses were shown, half of which were tilted to the right and half to the left. Before the first experimental block, each participant completed 1 practice block of 64 trials to get familiar with the task.

Session 2: Discrimination task. The task and procedure of the second session discrimination task was highly similar to the discrimination task administered in the first session (cf. [33]). Here, participants again viewed a rapid stream of house and face images and their task was to report at the end of the trial whether they had seen a house or a face. Right (5°) or left (355°) tilted house and face images were preceded and followed by visual masks, which were also tilted in the same direction. The trial procedure was identical to the discrimination task of session 1, except that the target face or house was presented for 20 ms or 30 ms. The duration of target stimuli was determined per participant depending on the discrimination performance during practice blocks, with the aim of achieving approximately 65% discrimination accuracy. Specifically, if a participant scored 60% or higher in correctly discriminating target stimuli when they were shown for 20 ms, this timing was used as the target duration for the remaining experimental blocks. Otherwise, target stimuli were shown for 30 ms.

After a 50 ms-long post-mask, a response screen was presented for 1,300 ms, during which participants needed to give a speeded response indicating whether they saw a house or a face using a left-handed (“z”) or right-handed (“m”) keyboard response. Additionally, following the discrimination response, participants needed to indicate their confidence in the accuracy of their discrimination response using a 4-point scale (1 –unsure, 4 –sure). The next trial began after the response had been given or after a 3-s timeout if no response was recorded. An equal number of right- and left-tilted faces and houses were shown in each block.

The discrimination task started with 4 practice blocks of 84 trials each, administered to familiarize participants with the task and to get an indication of target presentation time for the remainder of the task. This was done to ensure that the stimulus presentation time was sufficiently long, so that accuracy scores during the practice averaged to around 65%. Following the practice, each participant completed 16 experimental blocks of 84 trials each.

EEG measurements and preprocessing

The electroencephalogram (EEG) and electrooculogram (EOG) were recorded using the Biosemi Functional Two system (Biosemi.com). A total of 64 sintered AG/AgCl electrodes were positioned according to the 64 standard international 10/20 system, 6 external electrodes were placed on the earlobes and around the eye. The vertical EOG (VEOG) was recorded from 2 external electrodes located above and below the right eye. The horizontal EOG (HEOG) was recorded from 2 external electrodes located next to the external canthi of the eyes. The VEOG was used to detect eyeblinks and the HEOG was used to detect horizontal eye movements. Electrophysiological signals were digitized at a sampling rate of 512 Hz.

EEG data was preprocessed and cleaned before further analysis using custom scripts, the EEGLAB toolbox (v2019_1), and the Amsterdam Decoding and Modeling toolbox (ADAM; [95]). EEG data was re-referenced to the average of the earlobes, high-pass filtered at 0.5 Hz and low-pass filtered at 40 Hz, in separate steps as recommended by EEGlab. Note that a high-pass filter may distort the temporal estimates of the EEG signal [96,97], which was not the case in our dataset (see S3 Fig illustrating this). The continuous data was then epoched from −500 ms to 2,000 ms around stimulus onset. Trials containing jump artifacts were removed from the data using an adapted version of ft_artifact_zvalue muscle artifact detection function from the FieldTrip toolbox. Eye blink artifacts were removed from the data using a standard regression-based algorithm. A baseline correction in the 200-ms pre-stimulus onset interval was performed. Data were then downsampled to 128 Hz.

Statistical analyses

All analyses were done using custom scripts, the EEGLAB toolbox (v2019_1), the Amsterdam Decoding and Modeling toolbox (ADAM; [95]), and JASP software. We took a frequentist statistical approach and chose an alpha of p < 0.05 as the threshold for significance as is typical, to guard against false positives. Note, however, that this threshold is relatively arbitrary, and that frequentist statistics cannot provide evidence for the null hypothesis (absence of an effect) [98]. Therefore, we also report effect sizes, and in case of nonsignificant effects as indicated by p > 0.05, followed up with a Bayesian equivalent of the same test in order to quantify the strength of evidence for the null hypothesis (H₀). By convention, Bayes factors from 1 to 3 were considered as anecdotal, 3 to 10 as substantial, and those above 10 as strong evidence in favor of H₀.

Behavior

Trials in which participants made a discrimination response <200 ms after the stimulus presentation (3.39%) or gave no response at all (1.86%) were discarded from the analysis. To evaluate participants’ behavioral performance, we computed d-prime (d’, Type-I sensitivity), a measure of perceptual sensitivity to presented stimuli, separately for each confidence level and analyzed using a one-way repeated measures ANOVA with reported confidence ratings [1–4] as within-participant factor.

Decoding analyses

MVPA was applied to EEG data in order to decode patterns of neural activity specific to house and face stimuli. First, to test if we could decode category-specific neural representations in the localizer tasks, we used a 10-fold cross validation scheme. In this procedure, separately per localizer task, after randomizing the order of trials, the dataset was split into 10 equally sized folds. Equal number of house and face stimuli were present in each fold. We then trained a linear discriminant classifier (LDA) [27] to differentiate between house and face images using 9 folds and tested its performance on the remaining fold (using the standard Matlab function fitcdiscr). This was repeated 10 times, until all data were tested exactly once. Classification performance was evaluated for each participant separately by computing the area under the curve (AUC), which indicates the degree of separability between classes of the receiver operating characteristic (ROC) curve. First, we computed the proportion of correct classifications for each stimulus category, after which the scores were averaged across stimulus categories and over the 10 folds. Prior to training and testing procedures, EEG data was downsampled to 128 Hz and epochs were shortened to −100 to 800 ms, centered on the target stimulus, in order to decrease the computational time needed for MVPA.

Decoding analyses were carried out using EEG data recorded at all electrodes and a set of occipital–parietal electrodes [33]. The occipital–parietal set of electrodes, chosen so that it captures early visual “N170-like” response to houses and faces, consisted of Iz, Oz, O1, O2, POz, PO3, PO4, PO7, PO8, Pz, P1, P2, P3, P4, P5, P6, P7, P8, P9, and P10 electrodes.

To test our specific research questions, we used the cross-task validation scheme to evaluate the performance of each classifier (trained on either the sensory or the decision localizer task data) in differentiating between stimulus classes in the main decision task. We applied the same linear discriminant classifier to raw EEG data from the localizer task using voltages at each time sample to train the classifier, which was then applied to the main discrimination task data. This was done for separately for the sensory and decision localizer tasks. Following this procedure, we could examine whether and when house- and face-specific neural representations were associated with participants’ confidence in their perceptual decision, and whether perceptual decisions, in particular when incorrect, rely on the category-specific pattern of activation that is specific for the reported percept, or alternatively, representations are specific for the presented category. Moreover, critically, we could isolate sensory versus decision-related neural processes as the 2 localizers were differently tuned to these processes.

To examine the representational nature of decision errors, and to uncover how category-specific representations evolve across time, and across distinct processing stages, we used the GAT approach in applying the pattern classifiers [61]. Specifically, a classifier trained on a specific time point was tested on that time point as well as on all other time points. The resulting GAT matrix (training time x testing time) thus reveals dynamic changes of neural representations across time. For example, a classifier trained to decode between house and face images at 170 ms can generalize to a wider time-window, e.g., 170 to 350 ms, in which case it would indicate that the early neural representation was maintained in time. This approach is thus informative of how neural representations change across different stages of visual information processing, also permitting us to examine when in time neural representations differ between the perceived stimulus classes and depending on the correctness and confidence therein.

Decoding analyses were performed separately for correct and incorrect trials as a function of 2 confidence levels. Trials in which participants reported confidence ratings 1 and 2 were aggregated into “low” confidence trials, and those on which ratings 3 and 4 were given into “high” confidence trials. To illustrate, when decoding was performed for incorrect high and low confidence trials, classifiers were trained on the presented image category of the localizer task but were tested using trials in which participants reported seeing incorrect stimulus category, followed by high versus low confidence rating. Besides the stimulus category, we also decoded its orientation, i.e., whether a stimulus was tilted to the right or to the left, separately for correct and incorrect trials, as a function of confidence level. This analysis served 2 purposes. First, we wanted to verify that stimulus processing from the very bottom-up input did not differ between high versus low confidence trials. Low-level visual features such as orientation should be encoded in early brain activity in a bottom-up manner [50], but were not expected to be modulated by confidence due to the early timing of that processing stage. Second, the stimulus orientation was a task-irrelevant feature that participants did not need to do anything with and was therefore useful to tests whether task-irrelevant features are processed differently on high versus low confidence trials. To this end, classifiers were trained to distinguish between right versus left tilt of the stimulus using localizer data and then tested on right versus left images orientations using trials in which the correct versus incorrect stimulus category was reported.

Statistical analyses were performed on average classification scores (AUC) in time-windows that we preselected based on previous empirical work. Typically, studies report 2 processing stages using the GAT approach: an early (<250 to 300 ms) cluster of diagonal decoding reflecting initial sensory processes and a late processing stage (>300 ms), which was found to associate with conscious report [33,34,37]. Additionally, some studies have reported a third stage of visual information processing, starting at early latencies and generalizing off-diagonal, presumably reflecting maintenance of early sensory stimulus representations [33,34,53]. Following this body of work, and the study by Weaver and colleagues [33], who used an almost identical task design as in the present study, we focused our statistical analyses on 2 decoding clusters on-diagonal (classifiers were trained and tested on the same time point) and 1 decoding cluster off-diagonal (training and testing were done on different time points). The 2 on-diagonal clusters were 150 to 200 ms and 350 to 550 ms. The off-diagonal time-window spanned from 150 to 200 ms training time to 350 to 550 ms testing time (see below for more details). Finally, although decoding was pronounced during 3 time-windows observed previously by Weaver and colleagues [33], it was also observed at an even earlier time-window (80 to 130 ms) with a prominent occipital–parietal topography (Fig 2A, inset 1, early-stage topography). Because this early effect was absent in the same task when images where presented vertically [33], this early peak likely reflects differences in the orientations of the images presented, which we varied systematically across trials. This early window was also used as an event of interest in this study.

To inspect the pattern of neural activity that drove classification performance, we computed topographic maps for each classifier set (early, mid, late). Weights resulting from backward decoding models are not interpretable as neural sources [99]. For that reason, we plotted topographic maps resulting from multiplying the data covariance matrix with the classifier weights, yielding activity patterns that are interpretable as neural sources underlying decoding results [95]. These covariance/class separability maps were then normalized across electrodes for each participant (mean activity over electrodes was thus zero).

In order to statistically evaluate classifiers’ performance, we extracted diagonal and off-diagonal traces of GAT matrices. Average AUC values within specified clusters were analyzed using repeated measures ANOVA with factors decoding latency (early, late) and confidence (low, high). Specific hypotheses-driven comparisons between conditions were additionally evaluated using paired-sample t tests on AUC values averaged within our TOI. To evaluate decoding off-diagonal, specifically, when the training was done using 150 to 200 ms classifiers that were then applied to the late 350 to 500 ms time-window, we compared average decoding in this time-window between 2 confidence levels (low versus high) using a paired-sample t tests as well. In cases where a specific hypotheses-driven comparison did not indicate a significant result as indicated by an alpha of p < 0.05, we followed up that null-effect by a Bayesian equivalent of the same test in order to quantify the strength of evidence for the null hypothesis (H₀). By convention, Bayes factors from 1 to 3 were considered as anecdotal, 3 to 10 as substantial, and those above 10 as strong evidence in favor of H₀. Besides our hypothesis-driven analyses based on our predefined selection of time-windows, we additionally statistically compared each time point to chance (both for on- and off-diagonal decoding) by applying group-level permutation testing with cluster correction for multiple comparisons [100]. We employed two-tailed cluster-permutation testing with an alpha of p < 0.05 and cluster alpha p < 0.05 (N permutations = 1,000, as implemeted in ADAM toolbox). These latter analyses were mainly performed to evaluate whether our hypothesis-driven analyses may have missed potentially relevant effects, and if so, we recommend future studies to replicate these effects to evaluate the robustness of them [98].