Modeling subjective relevance in schizophrenia and its relation to aberrant salience

In schizophrenia, increased aberrant salience to irrelevant events and reduced learning of relevant information may relate to an underlying deficit in relevance detection. So far, subjective estimates of relevance have not been probed in schizophrenia patients. The mechanisms underlying belief formation about relevance and their translation into decisions are unclear. Using novel computational methods, we investigated relevance detection during implicit learning in 42 schizophrenia patients and 42 healthy individuals. Participants underwent functional magnetic resonance imaging while detecting the outcomes in a learning task. These were preceded by cues differing in color and shape, which were either relevant or irrelevant for outcome prediction. We provided a novel definition of relevance based on Bayesian precision and modeled reaction times as a function of relevance weighted unsigned prediction errors (UPE). For aberrant salience, we assessed responses to subjectively irrelevant cue manifestations. Participants learned the contingencies and slowed down their responses following unexpected events. Model selection revealed that individuals inferred the relevance of cue features and used it for behavioral adaption to the relevant cue feature. Relevance weighted UPEs correlated with dorsal anterior cingulate cortex activation and hippocampus deactivation. In patients, the aberrant salience bias to subjectively task-irrelevant information was increased and correlated with decreased striatal UPE activation and increased negative symptoms. This study shows that relevance estimates based on Bayesian precision can be inferred from observed behavior. This underscores the importance of relevance detection as an underlying mechanism for behavioral adaptation in complex environments and enhances the understanding of aberrant salience in schizophrenia.

In schizophrenia, increased aberrant salience to irrelevant events and reduced learning of relevant information may relate to an underlying deficit in relevance detection. So far, subjective estimates of relevance have not been probed in schizophrenia patients. The mechanisms underlying belief formation about relevance and their translation into decisions are unclear. Using novel computational methods, we investigated relevance detection during implicit learning in 42 schizophrenia patients and 42 healthy individuals. Participants underwent functional magnetic resonance imaging while detecting the outcomes in a learning task. These were preceded by cues differing in color and shape, which were either relevant or irrelevant for outcome prediction. We provided a novel definition of relevance based on Bayesian precision and modeled reaction times as a function of relevance weighted unsigned prediction errors (UPE). For aberrant salience, we assessed responses to subjectively irrelevant cue manifestations. Participants learned the contingencies and slowed down their responses following unexpected events. Model selection revealed that individuals inferred the relevance of cue features and used it for behavioral adaption to the relevant cue feature. Relevance weighted UPEs correlated with dorsal anterior cingulate cortex activation and hippocampus deactivation. In patients, the aberrant salience bias to subjectively task-irrelevant information was increased and correlated with decreased striatal UPE activation and increased negative symptoms. This study shows that relevance estimates based on Bayesian precision can be inferred from observed behavior. This underscores the importance of relevance detection as an underlying mechanism for behavioral adaptation in complex environments and enhances the understanding of aberrant salience in schizophrenia. PLOS

Introduction
Reduced differentiation between relevance and irrelevance, a disruption of salience attribution, is the key component of the aberrant salience hypothesis of psychosis [1][2][3]. According to this theoretical framework, neurobiological noise in terms of increased striatal dopamine turnover may cause the subjective experience of salience or meaningfulness in the absence of relevant contextual events that usually cause dopaminergic saliency signaling. This experience of aberrant salience is then attributed to random, irrelevant events that coincide with it and, thus, these irrelevant events turn subjectively meaningful. At the same time, chaotic aberrant salience signaling was proposed to blur the signal-to-noise ratio leading to decreased processing of contextually relevant events and the formation and maintenance of negative symptoms [4,5]. This salience framework clearly renders the objective experimental measurement of (aberrant) salience challenging and highlights the subjective nature of relevance and salience attribution. This subjectivity aspect can be captured by modeling latent learning processes of individuals, which is a common approach for describing the processing of relevant information, for example, during reinforcement learning [6,7]. However, while computational modeling has already revealed decreased learning from task relevant events in schizophrenia patients [4], this approach has not been applied to learning from relevant compared to irrelevant stimuli, which might shed further light on aberrant salience attribution. Both constructs, relevance and salience, are closely intertwined. In general, cues can be considered salient based on their physical characteristics, or cues are subjectively salient when they have been learned to be relevant in a certain context. For instance, neutral cues that are learned to predict reward can turn subjectively salient. Here, we define salient cues as those that have been learned to reliably predict important outcomes. These associations between cues and outcomes are learned via prediction error (PE) signals that code the surprise and unexpectedness of events; or computationally, the difference between observation and prediction. Thus, an unexpected event elicits a large unsigned prediction error (PE; a directed PE would carry information about the valence/reward) and the corresponding event would turn salient to the individual. Prediction errors are used to update the predictive value (belief) of the preceding stimulus. On the neural level, unexpectedness correlates with brain responses in the so-called salience network [8][9][10][11]; such as the (dorsal) anterior cingulate cortex and the insula [8,[12][13][14][15][16], while some studies also reported unsigned PE signaling in the striatum [for a review see 17].
In multidimensional environments, multiple cues are potentially important and individuals have to adapt to relevant cues that have proven to be reliable or precise predictors. A cue is precise when it announces a specific event with a high probability. Correspondingly, irrelevant cues that are experienced to be noisy and unreliable (= uncertain) should be dismissed. This was investigated by recent learning studies using multisensory cues [18][19][20] where subjects were either instructed to find out the relevant cue feature or were told which information they had to focus on in order to achieve the task of choosing the correct stimulus for reward maximization. While applying computational modeling, these studies provided behavioral and neural evidence for learning from multiple sources of information by integrating these according to their respective subjective relevance. According to the principles of Bayesian learning [21][22][23], as incorporated in the Hierarchical Gaussian filter [HGF; 24], Bayesian precision reflects the computational mechanism capturing the reliability of a stimulus. We use precision as our definition of subjective (in the sense of learned) relevance in multidimensional environments. Furthermore, we probe the influence of this subjective relevance on prediction errors, i.e. when the subject knows that the environment is irrelevant he/she should no longer experience a large prediction error as salient. In line with this approach, a theoretical account proposed that aberrant precision coding underlies psychosis formation [5,[25][26][27].
In schizophrenia and presumably due to chaotic dopaminergic signaling, the detection of task relevant cues seems to be disturbed while irrelevant cues not carrying reliable information can gain high subjective salience [1,2,28]. This latter phenomenon of aberrant salience describes the subjective experience of patients characterized by random stimuli suddenly standing out and turning meaningful. According to the hypothesis, patients make sense of this aberrant salience experience by forming cognitive schemes that on the long run turn into delusional beliefs. However, though the aberrant salience concept offers high descriptive value and plausibility regarding clinical symptoms the behavioral quantification in experimental settings still remains challenging. So far, conclusions about increased aberrant salience attribution has been drawn from two kinds of findings both related to reinforcement learning: (1) heightened responses to cues that predicted (affectively) neutral outcomes and (2) increased responses to irrelevant, i.e. unreliable cues. Regarding the first operationalization, blunted differentiation between cues indicating either reinforcement or neutral outcomes were consistently found in schizophrenia patients. Whereas healthy individuals displayed enhanced responses to the reinforced over neutral cues, patients displayed the opposite pattern; increased responses (reaction times, skin conductance as well as midbrain and striatal BOLD responses) to stimuli that were followed by neutral outcomes [29][30][31][32]. With regard to the second operationalization, aberrant salience may further be reflected in a tendency or bias towards one over another equally irrelevant stimulus, as defined in the Salience Attribution Test [SAT ; 33]. In the SAT, subjects have to speed up their responses to a target to increase their wins. Crucially, the target is preceded by conditioned stimuli with one feature being reliably informative about the following reward (instrumental motivational salience) and another feature being uninformative for predicting reward; being therefore relevant or irrelevant. Here, aberrant salience is reflected by the idiosyncratic bias inside the irrelevant dimension. This quantification of aberrant salience to irrelevant instead of neutral events circumvents instrumental learning deficits reported in schizophrenia [6]. So far, it has revealed mixed results in schizophrenia patients possibly pointing to differential expressions of aberrant salience across the stages of illness. The explicit (= subjective judgment based) aberrant salience measure was increased in first episode patients with delusions and individuals at ultra-high risk for psychosis [33][34][35] and striatal responses to irrelevant events correlated with positive symptoms [34]. The implicit (= reaction time based) aberrant salience measure was increased in a medicated and rather chronic schizophrenia patients sample [36]. However, other studies using the SAT in patients only found deficits in adaptive salience [37,38].
In a previous study [39], we found that this idiosyncratic bias inside the irrelevant cue feature does not interfere with adaptive salience attribution that is needed to successfully solve the task. Hypothetically, this may imply that when schizophrenia patients are confronted with cues that are not associated with task-information and that are therefore imprecise they form a bias towards one of these cues in order to resolve this uncertainty. In other words, cues that are irrelevant within a particular (task) framework and that thereby are unreliable in serving instrumental aims may be inherently prone to capture aberrant salience. Thus, explicit task demands should be low in order to create an atmosphere where aberrant salience attribution can arise. A rather implicit task design would further reduce confounds by motivational, cognitive and stress-related deficits known in schizophrenia patients [40][41][42][43].
The aim of the current study was to test the idea of aberrant salience as an idiosyncratic bias to subjectively unreliable and thus task-irrelevant information. For that, we used computational modeling in order to assess relevance attribution on the subjective level. While we followed the idea of the SAT of having relevant and irrelevant cue dimensions, we used a more dynamic task design including contingency reversals in order to achieve ongoing learning that is better suited for computational modeling. Further, participants were not instructed to explicitly track contingencies between cues and outcomes in order to keep task demands low.
In the current study, 42 schizophrenia patients and 42 healthy individuals performed an implicit salience paradigm during fMRI [ISP ; 39]. In this paradigm, participants had to discriminate between two outcomes (coin/circle) via button press. The outcome could be predicted from preceding graphic cues with dynamically changing contingencies along two distinct features (color and shape). By applying the Bayesian learning framework of the HGF, we used computational modeling to assess individual learning trajectories of these associations. Subjective relevance was formalized as Bayesian precision (as a dynamic reliability measure) and we compared different models, which varied in how subjective relevance affected learning and behavior. We hypothesized that participants would be more surprised by unexpected events and slow down their responses. On the computational level, this was defined via relevance weighted UPEs and we expected their neural correlates to be located in areas previously implicated in salience processing network and/or the striatum. Further, we defined aberrant salience as an idiosyncratic bias towards one unreliable and thus subjectively irrelevant cue feature. We hypothesized that this measure of aberrant salience would be increased in schizophrenia patients.

Ethics statement
All participants gave written informed consent and received monetary compensation as well as the total wins of the task battery. The study was performed in accordance with the Declaration of Helsinki and was approved by the local ethics committee of Charité Universitätsmedizin.

Participants
In total, 42 schizophrenia patients and a matched healthy control group of 42 individuals participated in this study. Healthy individuals reported no past or present psychiatric disorder according to the SKID-I. Patients were diagnosed with schizophrenia according to the DSM-IV and ICD-10. Psychopathology was assessed using the Positive and Negative Syndrome Scale (PANSS) as well as the subscale for delusions and anhedonia of the Scale for the Assessment of Positive (SAPS) and Negative Symptoms (SANS), respectively (for information on demographics and psychopathology, see Table 1). All patients were on antipsychotic medication (for more details, see Table A in the Supplement). They were recruited from the inpatient and outpatient units of the Department of Psychiatry and Psychotherapy, Charité-Universitätsmedizin Berlin and the Psychiatric Department of the Schlossparkklinik Berlin. Aberrant salience raw data scores of a partially overlapping sample (37 healthy controls and 34 schizophrenia patients) were reported in a previous publication [39].

Implicit salience paradigm (ISP)
This paradigm was explicitly instructed like a target-detection task though implicitly being a learning paradigm where features of neutral stimuli predicted certain outcomes. It consisted of 160 trials where subjects were told to discriminate the outcomes (10 Eurocent coin or blue circle) of each trial. Therefore, their only task was to press a respective button when they saw a coin, versus another button when they saw the blue circle. Subjects were told that they would receive the amount of money they had seen during the task irrespective of whether they had pressed a button or not, though they were encouraged by the experimenters not to miss too many trials because this would impede the analysis. The outcomes were preceded by conditioned stimuli that differed in color and shape: gray or colorful squares or triangles (see Fig 1A  and 1B). During the instructions, subjects were told not to pay attention to these stimuli preceding the outcomes. However, prior to scanning, participants were primed with the stimulus features while they were asked to name the color and the shape of each of the four conditioned stimuli. Then, they practiced the outcome detection for 10 trials. In this practice session, all outcomes were preceded by a stimulus that was not presented during the main experiment in the scanner. In the main experiment, the conditioned stimuli predicted the outcome types in a probabilistic manner that reversed during the task. Importantly, only one stimulus feature reliably predicted the outcome (eg, shape). For instance, 80% of all triangles were followed by the coin (20% circle), and 80% of all squares were followed by the blue circle (20% coin). Whether the square or the triangle predicted the coin reversed every 20 trials. In the meantime, the color of the stimuli was irrelevant in predicting the outcome; colorful and gray stimuli were equally followed by coins and circles (50% each). After the first half of the experiment this was reversed, and then the formerly irrelevant feature (here: shape) predicted the outcome, whereas the other feature turned irrelevant (see Fig 2). The relevant dimensions were counterbalanced across participants and coins and neutral outcomes were each displayed in 50% of all trials. In total, the experiment lasted 15 minutes and took place during fMRI scanning. Participants received the amount of coins seen in the experiment (8 Euro).

Raw data analysis
In our raw data analysis of reaction times, we focused on two aspects: (i) learning of the regularly reversing relevant feature and (ii) aberrant salience towards one manifestation of the irrelevant feature. Extreme reaction times (<150 ms and >1.5 s) were excluded. For analyses of potential learning, reaction times (in ms) were log transformed to achieve normal distribution required for variance analyses. First, we tested whether subjects learned the cue-outcome contingencies and slowed down their responses if the outcome could not be predicted based on the preceding cues and thus violated their expectation. For that, log reaction times were compared for expected (i.e. trials when the 0.8 rewarded feature was followed by reward and trials when the 0.8 non-rewarded feature was followed by a circle) versus unexpected events (i.e. trials when the 0.8 rewarded feature was followed by the circle and trials when the 0.8 non-rewarded feature was followed by a coin) of the relevant condition in a repeated-measures ANOVA with group as between-subject factor (HC versus patients) and event type as within factor (expected versus unexpected). Second, a change in reaction times over time following contingency changes inside the relevant cue feature was tested using a repeated-measures ANOVA. For all eight blocks with 20 trials each, log-reaction times of the 16 expected events (ie, spanning those 8 trials when the rewarded feature was followed by reward and the 8 trials when the non-rewarded feature was followed by a circle) were grouped for rewarded versus neutral outcome trials (Condition factor) and combined into 4 time bins each, consisting of 2 consecutive trials (Time factor). For analyses targeting responses with regard to different cue features, please see the Supplement.
For aberrant salience attribution, aberrant salience scores were calculated as in the previous literature [39], and reflected an individual bias towards one of the two irrelevant cue manifestations based on the ground truth contingencies. For that, the mean reaction times (in ms) to each of the four cue features when they were irrelevant to the task were calculated. Then, aberrant salience was calculated as the absolute difference in reaction times between both manifestations of each condition (eg, square over triangle when shape is irrelevant). Then, these two scores were collapsed across test halves. Underlying task contingencies between color and outcome (blue) and shape and outcome (red). Note that the direction of the trajectories reflects a tendency towards an association (eg, 1 for [triangle ! reward; square !circle] and 0 for the opposite associations) and does not reflect a reward value. In this example, during the first half of the ISP, color is relevant for predicting the outcome, whereas the squares and triangles both predict coin and circle with a probability of 0.5 and are therefore irrelevant. The extradimensional relevance reverses after 80 trials, and shape becomes relevant. Inside the relevant features, the 80% rewarded feature manifestation reverses every 20 trial block. That is to say, if colorful cues are rewarded in 80% of trials (gray cues in 20%) in the beginning, gray cues are followed by coins (colorful cues by a circle) for the next 20 trials. Middle plot: One subject's individual belief trajectories for color (blue) and shape (red) associations with outcome based on the best fitting model (2HGF RelPE+IrrelBias). Lower plot: Trajectories that were used to determine individual reaction times in the best response model: Inferred relevance weights (see Eq 5) for color (blue) and shape (red) predictions and the mean relevance weighted prediction error (magenta; see Eq 4.1.2). https://doi.org/10.1371/journal.pcbi.1006319.g002

Computational modeling of relevant and irrelevant predictions
We used detailed computational modeling combined with model selection in order to assess the learning mechanisms driving the observed individual behavior. By that we tested if subjects learned the underlying cue-outcome associations and inferred the relevant cue feature. In keeping with our initial definition of relevance, a cue feature should be perceived as more relevant the more precisely it is believed to predict the outcome. In contrast, if the association between cue feature and outcome is very noisy, because the occurrence of reward and circle are equally probable this feature should be perceived as irrelevant. We set up novel response models that postulated slowing down of responses when the expectations were violated. Computationally, a violation was captured by the unsigned PE of each feature. In different models, we compared if unsigned PEs of the different cue features differentially affected behavior based on their current relevance. By the relevance weighted PE, we implemented increased adaptation to surprising events (unsigned PE) predicted by the most relevant cue feature as well as decreased adaptation to the irrelevant cue feature. Thereby, we tested if individuals adapted their behavior less to events when these had proven to be noisy and uninformative in the past. Instead, they would adapt their behavior to those unexpected events that were held to be informative because these may actually signal a real change in the environment (as the contingency changes in our task).
Our modeling analysis was guided by two aims: 1. Modeling two parallel learning processes of predicting the relevant and the irrelevant cue feature in order to compute distinct prediction error trajectories [for a comparable learning model, please see 44]. Here, it is crucial to determine a formal definition of "relevance" since the participants had to dynamically infer which cue feature was relevant for predicting the outcome (even though this was not necessary to solve the task, that is, discriminating between a coin and a circle).
2. Describing the aberrant salience effect to an irrelevant feature as seen in previous analyses [39] on a computational level by finding interindividual differences in model parameters capturing this effect.
We modeled predictions for each of the two cue features, shape and color, in separate HGF learning models so that a trial-by-trial expectation was computed for each feature (learning model). We did not model relevant and irrelevant conditions separately because of two reasons. First, the generative model was supposed to capture the subjects' learning experience and they were not instructed about the task structure having relevant and irrelevant conditions. Instead they were only primed with the distinction by feature, color and shape, in the practice session. Second, after the first half of the experiment we switched the relevant dimension (e.g. shape to color). Hence, modeling separately for relevance versus irrelevance would have introduced external information on the task structure that was not accessible to subjects and thus not generative. The resulting learning trajectories for shape and color were transformed into trial-by-trial predictors of reaction times (response model).

Hierarchical Gaussian filter [HGF; 24, 45]
According to the 'Bayesian brain' hypothesis [21][22][23]46], an agent forms a generative model of the world in part by increasing the precision of predictions (μ) to successfully adapt one's behavior. The HGF offers a generic framework for Bayesian learning on multiple hierarchical levels. Crucially, the belief update at each level is comprised of a lower-level prediction error d Δm We used a "two branches" version of the HGF for parallel learning of the shape and color associations with the outcome. In our modelling, we focused on the reliability and thus undirected beliefs about associations. Thus, the direction of the learning trajectories (association beliefsm ðkÞ 1 and prediction errors δ) did not reflect the reward value but an arbitrarily determined relationship between cue feature manifestations and outcomes. A more detailed description of the HGF and its levels can be found in the Supplement.

A novel definition of relevance: First-level precision
Since the core aim of this study was the dissociation between learning about relevant versus irrelevant cue features, the term relevance needs to be defined formally. In terms of the ISP, a cue feature should be perceived as more relevant the more precisely it is believed to predict the outcome. In contrast, if the beliefm ðkÞ 1 of an association is 0.5 it should be perceived as irrelevant for outcome prediction (because in this case reward and circle will occur with the same probability and cannot be reliably predicted). In the HGF framework, this interpretation of relevance is reflected in the estimated precision of prediction on the first levelp ðkÞ 1 (see 4).p It is a function only of the first-level association predictionm ðkÞ 1 , which ranges between 0 and 1.p ðkÞ 1 has a minimum of 4 form ðkÞ 1 ¼ 0:5 and increases symmetrically to infinity asm ðkÞ 1 approaches 0 or 1. This relevance could affect how beliefs are updated (for this implementation ('precision feedback'), see the Supplement) and/or how learning signals affect behavior, which we implemented in four response models. Our learning model space contained 2-level and 3-level HGFs with and without precision feedback (see Supplement), leading to four different hierarchical learning models: 2HGF, 2HGFprecfb, 3HGF, and 3HGFprecfb.

Response models
Prediction errors relating to the cue feature that is thought to be more relevant might translate more strongly into reaction times, and the aberrant salience effect found in the raw data may be explained as a bias towards one of the irrelevant features on a trial-by-trial basis. Both aspects were formalized in the following four response models. The baseline response model (see Eq 4) This response model was modified in three ways. In a first modification (4.1.2), the absolute prediction errors of each feature were weighted according to their respective relevance weight (rel; 5). The latter is formalized as the relative amount of each feature's precision given the overall precisions of both features.
In a second modification and in line with the aberrant salience effect, the constant bias was weighted according to its feature's irrelevance (irrel; see Eq 4.2.2). The irrelevance weight of one feature was defined as the relevance weight of the opposing feature (see Eq 5). From the four cue feature parameters (β 2 , . . ., β 5 ) one composite parameter was calculated in two steps. First, the individual absolute differences between the β 2 and β 3 for irrelevance weighted manifestations within color (and between the β 4 and β 5 for shape manifestations) were calculated. Then, they were collapsed across color and shape to achieve one parameter β irrelevance capturing the bias towards one cue feature manifestation that increased with subjective irrelevance. Thus, we compared four different response models: the baseline model (Eq 4), the baseline model with relevance weighted absolute prediction errors (Eq 4 with modification term 4.1.2; RelPE), the baseline model with only the irrelevance bias (Eq 4 with modification term 4.2.2; IrrelBias), and a full model with both modifications (Eq 4 with modification terms 4.1.2 and 4.2.2; RelPE+IrrelBias). This led to a total model space of 16 model combinations (see Figure S1).

Model fitting and Bayesian model comparison
All models were fitted using the HGF toolbox 4.15 [24,45] as part of TNU Algorithms for Psychiatry-Advancing Science (TAPAS, http://www.translationalneuromodeling.org/tapas/). For optimization, a quasi-Newton optimization algorithm was applied. We used random-effects Bayesian Model Selection [BMS, spm_BMS in SPM12, www.fil.ion.ucl.uk/spm; 47] for each subject's and each model's negative free energy (as an approximation to log-model evidence) in order to identify which of the competing models best explained the subjects' response time data. BMS takes into account accuracy of each model and also penalizes for complexity. It accounts for heterogeneity across subjects and treats each model as a random variable in the population. We report protected exceedance probabilities for each model (PXP) and posterior probabilities (PP) as well as exceedance probabilities (XP) for model families (HGF vs. HGFprecfb; 2HGF vs. 3HGF; BL vs. RelPE vs. IrrelBias vs. RelPE+IrrelBias). The XP describes the relative probability that the model would better replicate the data in comparison with the other models. The PXP that governed our model selection protects against the 'null' possibility that there are no differences in the likelihood of models across the population.

Simulation analyses
Based on the estimated individual parameters from the best-fitting model, we simulated trialby-trial reaction time data. In addition to the Bayesian model comparison, we tested the model's credibility by carrying out the same analyses as in the raw data section on the simulated data ((1) Expectedness Ã Group ANOVA, (2) Condition Ã Time Ã Group ANOVA, (3) 'ground truth' aberrant salience group differences). Using this approach, we checked whether the model was capable of reproducing the meaningful effects and group differences that were evident in the data.

Functional magnetic resonance imaging
For single and group statistics, an event-related analysis was applied using the general linear model (GLM) approach as implemented in SPM12 (http://www.fil.ion.ucl.ac.uk/spm/ software/spm12/). On the single subject level, the outcome onsets were convolved with the hemodynamic response function and its temporal derivative. As parametric modulator, the mean of the relevance weighted absolute prediction errors of the best fitting model (2HGF-RelPe+IrrelBias) was introduced, representing how unexpected and salient the observed outcome was based on the subjects' learned expectations about the two cue features. Regressors of no interest were no response trials, trials with reaction times>1.5 seconds and <150 milliseconds, realignment parameters with their first temporal derivative of translational movement, and one regressor for scans with >1mm scan-to-scan movement. For random effects group-level analysis, the individual contrast images for mean relevance weighted PEs were used in a two-sample t-test for between-group comparisons (controls vs. patients). Explorative analysis probed the association between model parameter β irrelevance and neural relevance weighted PE signals in schizophrenia patients. Hence, interindividual β irrelevance scores were introduced as a covariate in a one-sample t-test using the Mean relevance weighted PE contrast. Results are reported using FWE correction at the voxel level across the whole brain. Based on our hypothesis concerning activations in areas previously shown to code salience such as the ACC and insula, namely the so-called salience network [8][9][10][11], and the nucleus accumbens, we applied small volume correction at the voxel level for the respective bilateral anatomical masks derived from the WFU PickAtlas (http://fmri.wfubmc.edu/software/ pickatlas). Thus, three VOIs were used for small volume correction at p FWE <0.05 and we indicate which results survive Bonferroni correction for three tests.

Raw data analysis
Participants displayed increased log reaction times observed for probabilistic (unexpected) events compared to non-probabilistic (expected) events (Main effect of event type: F(1,82) = 5.9, p = 0.018, interaction event type Ã group F(1,82) = 0.02, p>0.8). The Reward Ã Time Ã Group ANOVA revealed that reaction times differed significantly between coin and circle (main effect Condition: F(1, 82) = 22.78, p < .001, see Fig 1C) and showed a trend-wise decrease following a reversal (main effect Time: F(3, 246) = 2.6, p = .053). Participants took around 8 trials to decrease their reaction times after slowing down following a reversal in contingencies (RT difference first time bin (trials 1-4) vs. third time bin (trials 8-12), t = 2.72, p = 0.048, Bonferroni corrected). Groups did not significantly differ in their reaction times (main effect of Group p>.2). The aberrant salience scores calculated based on the ground truth contingencies differed significantly from zero in both groups (mean (SD) for HC = 17.98 (10.

Bayesian model comparison
Across all subjects, the two-level HGF with the full response model was the best fitting model (PP = 0.3755; PXP = 0.5155; see Table 2). Among learning models there was clear evidence against the precision feedback model (PP HGF = .968; PP HGFprecfb = .032; exceedance probability XP HGF = 1), while there was only a very subtle advantage for two-level compared to threelevel models (PP 2HGF = .551; PP 3HGF = .449; XP 2HGF = .551; XP 3HGF = .449). Concerning the response models, the full response model using the mean relevance weighted prediction error and the irrelevance weighted bias clearly explained the data best (PP BL = .015; PP irrelBias = .037; PP relPE = .017; PP relPE+irrelBias = .93; XP relPE+irrelBias = 1). Therefore, we decided to do our fMRI analyses with the 2HGF model and the best response model relPE+irrelBias.

Model checking by data simulation
We repeated the same analyses as for the raw reaction time data for the simulated log RTs based on the best fitting model. The ANOVAs revealed similar behavioral effects: faster responses for expected than for unexpected events (F(1,82)

Response model parameter differences
We tested for group differences on three individual response parameters that weighted the influence of the relevance weighted prediction error (β 1 ), the outcome (β 6 ), and the irrelevance bias (β irrelevance ) on reaction times. Schizophrenia patients showed an increased bias towards one of two equally irrelevant cue features as indicated by increased β irrelevance values (t(74.2) = 2.7, p = 0.036, corrected for multiple comparisons, see Fig 3). Both groups did not differ on β 1 and β 6 (p>0. 7). For all fitted model parameters, see Table 3.

Correlation analyses
In an explorative approach, we investigated how the response model parameter β irrelevance related to psychopathology using Spearman's correlations within groups. In schizophrenia patients, β irrelevance was associated with an increased negative symptoms score from the PANSS

Model based fMRI
Across all participants, the relevance weighted prediction error correlated with increased BOLD response in the anterior cingulate cortex ([12 32 22], t(74) = 4.2, p SVC for ACC VOI = 0.032, p B corr = 0.096, see Fig 4A). A negative correlation with relPE was observed in the left hippocampus ([-32-18-14], t(74) = 5.4, p FWE whole brain = 0.041, see Fig 4C). There was no group difference in relevance weighted PE response in any of the VOIs nor at the whole brain level.
In order to probe the associations between model derived parameters and brain responses in patients, we focused on the model parameter β irrelevance that was increased in schizophrenia patients and related to psychopatholgy. In schizophrenia patients, there was an inverse correlation between β irrelevance values and relPE related bilateral nucleus accumbens response ([-14   5).

Discussion
In the current study, we established a novel definition of subjective relevance based on Bayesian precision of predictions. This computational mechanism was involved in implicit learning about multidimensional and changing environments, as well as in aberrant salience attribution in schizophrenia. To our knowledge, our study stands alone in investigating subjective beliefs during implicit learning in a dynamic appetitive Pavlovian conditioning task. We had three main findings: 1) Both groups learned the underlying associations equally well but patients showed more aberrant salience in terms of a bias towards task-irrelevant features; 2) in all participants unexpected outcomes as indicated by high relevance-weighted unsigned prediction errors were associated with increased dorsal ACC BOLD signal as part of the so-called salience network; and 3) heightened aberrant salience in patients in terms of a bias towards the currently task-irrelevant stimulus feature was associated with a lower neural salience signal in the nucleus accumbens and higher negative symptom severity. On the behavioral level, patients and controls both performed the target detection task with high accuracy and responded faster to the rewarding coin stimulus compared to the neutral circle. These two findings suggested that participants of both groups were engaged in the simple task of pressing one button upon seeing a coin and another for a circle, which had obviously minimal cognitive demands. We found that schizophrenia patients and healthy individuals seemed to use the preceding cues to speed up during the outcome discrimination task.
Subjects were faster for expected than for unexpected events in terms of ground truth probabilistic contingencies. Detailed computational modeling combined with model selection revealed that participants learned and used the underlying cue-outcome associations and could determine the currently relevant feature. Thus, participants discriminated faster if the outcome was predicted based on the preceding cues but slowed down if the observed outcome violated the expectation, which was formalized in the relevance weighted PE. Responses to the most relevant cue feature were increased, whereas reactions to the irrelevant feature were decreased as implemented in the best-fitting response model. Hence, subjects adapted more to those unexpected events that were thought to be reliable and that thus signaled actual changes in the environment. Correspondingly, they downregulated responses to such information that was thought to be noisy and uninformative. To conclude, in our task the relevance weighted unsigned PE can be interpreted as subjective informative surprise that leads to subtle adaptation in behavior even in the absence of instrumental need.
While relevance weights scaled the influence of prediction errors on reaction times, that is, behavioral adaptation, we had no evidence that updating of association beliefs was increased by the subjective relevance of a cue feature beyond the HGF implementation of a cross level precision ratio [24,45]. This may only hold for our implicit and dynamic paradigm and seems to be different when subjects are explicitly instructed to find out the steadily relevant aspect of a multidimensional learning cue [18].
The relevance weighted unsigned PE correlated with BOLD responses in the salience network comprising the dorsal ACC. This is in line with the theory of proximal salience, which proposes that activity of ACC and insula regulates higher order processing of external stimuli [16]. Especially the dorsal ACC/medial prefrontal cortex has been reported to respond to unexpectedness regardless of valence [48,49]. According to the response-outcome theory [50], the dorsal ACC is crucial for detecting discrepancies between expectations and outcomes and thereby drives attentional and behavioral reallocation. Further studies that elaborated how individuals use those unexpectedness signals highlighted the ACC's function in belief updating [51] and in predicting future cognitive load based on previous experience [14]. This can be related to the relevance weighted unsigned PE signal in our study that also comprised estimates of prior reliability of a cue feature. In line with our results, a recent study also used computational modeling of Bayesian conflict learning and reported similar UPE correlates in the dorsal ACC [12]. On the other hand, the hippocampus showed an opposite pattern in our study: high hippocampus activation was associated with low relevance weighted UPE. This stands in contrast to previous findings and theories describing hippocampal activation during mismatching events [for a review see 52]. In our task, low relevance weighted UPE occurred when the observed outcome was not surprising and would therefore be mostly present at the end of each block, after constantly changing contingencies have been learned. In other words, expected events elicit stronger BOLD response in the hippocampus which might indicate higher-order processes related to contextualizing and memorizing these learned contingencies [53,54].
Although participants were not incentivized for target detection, both groups performed well and no group differences emerged for learning contingencies indicating that patients and controls both used the cues for behavioral adaptation. Contrarily, there is sound evidence for learning deficits in schizophrenia patients mostly in more explicit and instrumental tasks [55][56][57][58]. Our study assessed implicit learning which might have led to the subtle behavioral and neural effects. In addition, the reversals inside the relevant condition appeared every 20 trials. Presumably, fewer reversals with longer stable periods for learning the implicit associations might have led to stronger task effects. Switching the relevant condition in the middle of the experiment did not affect learning significantly (please see Supplement), though this seems to be different in tasks where such shifts happen more often and are explicitly instructed as during set-shifting. During these paradigms, schizophrenia patients are impaired [55]. Hence, whereas we focused on the rather implicit and prediction error driven learning about relevance with our paradigm and model space, schizophrenia patients might show more pronounced deficits when explicit reasoning about the structure of the task is required. It has been shown that when healthy individuals were explicitly asked to find out the relevant cue dimension they used explicit strategies reflecting the assumed underlying task structure [59]. Thus, group differences between schizophrenia patients and healthy individuals concerning the detection of relevance (shifting) might be better detectable and more pronounced in such more complex learning paradigms probing deliberative decisions that rely on the use of explicit task knowledge.
In keeping with previous results in a partially overlapping sample [39], schizophrenia patients displayed an increased bias for one of two equally task-irrelevant cue features as formalized using the ground truth contingencies of the task. This is in line with previous studies that reported increased responses in schizophrenia patients to neutral [29][30][31][32]60] or unreliable [33] information. We further elucidated this bias using computational modeling that took into account only subjective and dynamic relevance estimates. The response parameter capturing this irrelevance bias was increased in patients indicating that they attributed more aberrant salience to cues when they were subjectively irrelevant and thus thought to be less informative with regard to the task. This aberrant salience bias was associated with decreased striatal activation during relevance weighted PE signaling. Though this finding has to be treated with caution, since this region did not display a task effect per se, it might show that patients experiencing more idiosyncratic and task-unrelated saliency also showed a reduced striatal processing of relevant information. There is meta-analytic evidence for decreased striatal responses in schizophrenia patients to reward-predicting cues and rewarding feedback [61]. Also, ultra-high risk subjects who decreased their unusual thought content after treatment showed an amelioration of striatal response to relevant and reinforced stimuli in the SAT [35]. These studies focused on striatal reward anticipation, whereas the relevance weighted PE in our study was undirected, i.e. carried no information about reward, only about associability strength and the respective surprise when these associations were violated in both directions, good or bad. This striatal PE coding is in line with a recent study on explicit reasoning that elegantly decorrelated reward PEs from Bayesian surprise and the authors found that the latter was more strongly associated with striatal response [59]. Taken together, the processing of relevant (not only rewarding) and the bias towards irrelevant information seem to be interfering phenomena. We cannot make any claims on causality here and would argue that bidirectional influences are plausible and may appear at the same time. A possible interpretation may be that not being able to figure out the correct (e.g. rewarding) cues for behavioral adaptation may cause a compensatory clinging to random cues as seen in the aberrant salience bias to irrelevant events in our study.
Note that this aberrant salience definition which is based on the SAT literature [33,35] targets increased responses to task-irrelevant and not to neutral events as often used in previous Pavlovian studies [30,31,62]. This different operationalization of aberrant salience could be important for associations with psychopathology. One study reported increased BOLD response towards cues reliably indicating neutral outcomes to be associated with positive symptoms [30]. Our aberrant salience measure to events that were learned to be uninformative of any outcome relates to negative symptoms. In the same vein, orientating behavior to unreliable, and thus irrelevant stimuli was proposed to underlie negative symptom formation [5]. Taken together, a biased focus on uninformative, irrelevant events by possibly limiting attention to relevant events relates to increased negative symptoms. Roiser and colleagues found a similar association between (explicit) aberrant salience to irrelevant events and negative symptoms in patients as well as with anhedonia in healthy individuals [33,63]. They interpreted this to result from 'false negatives' in phasic dopamine signaling to contextually relevant events contributing to reduced processing of reinforcing stimuli [33]. This view is supported by an animal study reporting decreased striatal dopamine transients to relevant, reward-predicting stimuli following amphetamine administration [64]. In the same vein, aberrant salience was related to increased tonic dopamine synthesis capacity and reduced responses to relevant events in the striatum in healthy individuals [65]. Transferred to our findings, processing unreliable information in schizophrenia patients may increase processing of irrelevant as well as decrease processing of relevant events while contributing to both, positive and negative symptoms [4,5]. Our association between the aberrant salience (irrelevance) bias and negative symptoms was found in a chronic patient sample on stable antipsychotic medication showing both negative and positive symptoms.
Taken together, we provide evidence that schizophrenia patients show a bias towards irrelevant stimuli when confronted with an uncertain and changing environment. Future longitudinal studies should examine the time-wise formation of this bias as well as the process of relevance detection and their respective associations with psychopathology.
Several limitations of our study need to be addressed. First, by keeping the contingency structure implicit, it possibly led to a high variance between subjects in how to solve the paradigm. We modeled reaction times and although there are notable exceptions [66][67][68][69], modeling of choice data is more widely used. For reaction time based analysis, unlike choice-based analysis, there is no clear absolute model fit to compare against as in previous studies [56,57]. Second, because learning was not necessary for task performance, the implicit behavioral and neural learning effects were both very subtle and might need larger samples for the detection of group differences. Third, the two different cue features were initially chosen to be easily dissociable resulting in differences regarding their perceptual characteristics and presumably their saliency. As alterations of visual perception have been reported in schizophrenia patients, patients might have processed the cue features differently compared to controls [70]. However, in our supplementary analyses we neither found evidence that the cue features were learned differently nor that cue features were processed differently between groups. Fourth, with our current paradigm we cannot disentangle the saliency of cues from their rewarding valence since we did not include a punishment condition. Finally, future studies of learning about multidimensional information in schizophrenia should include additional methods to detect relevance attribution, such as skin conductance response [30], eye-tracking, or MVPA [18].
In sum, we give a novel computational account of the use of subjective relevance estimates in implicit learning that is based on Bayesian precision. Furthermore, we provide quantitative, model-based evidence of an impairment in the formation and/or use of relevance estimates associated with schizophrenia. In a task probing the implicit learning of multidimensional and dynamic associations, relevance detection and neural learning correlates in the ACC seem to be intact in patients with schizophrenia, but aberrant salience to subjectively irrelevant events was increased in patients and related to negative symptoms and reduced striatal response to salient events. Our findings suggest that individual beliefs about relevance can be inferred from computational models and highlight the importance of relevance detection to complex environmental stimuli.
Supporting information S1 Text. Supplementary material. The supplement contains the following analyses and results: additional information on the ISP and raw data analyses of potential stimulus feature effects, additional information on the HGF and the HGF-precision feedback learning model, overview of the model space, fMRI preprocessing, and tables of the priors of learning and response models. (DOCX)