Ethics

The experimental procedures in this study were approved by the Human Research Ethics Committee of Monash University in Melbourne, Australia (Project ID: 26350). The study was conducted in accordance with the principles of the Declaration of Helsinki. All participants provided written informed consent before commencing the study.

Participants

We recruited healthy adults aged between 18 and 40 years. Exclusion criteria were: females not on hormonal contraception; recent use of psychotropic medication or recreational drugs; personal history of neurological disease or psychiatric illness; and contraindication to the study drug. Four participants were excluded due to inconsistent behavior during calibration of the dynamometers, such that they exerted forces exceeding their nominal maximal voluntary contraction (MVC) by more than 10% during the experimental blocks. The final sample included 42 participants: 19 in the placebo group and 23 in the sulpiride group. Groups did not differ in terms of age, gender, and body mass index (all p > .1; Table 1).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Group characteristics.

https://doi.org/10.1371/journal.pbio.3003765.t001

Study design

We used a double-blind, randomized design to compare a single 800 mg dose of sulpiride to placebo. At an 800 mg dose, sulpiride effectively blocks post-synaptic D2-receptors [48,49]. This study was planned and conducted as a within-subjects cross-over design, in which participants were administered either sulpiride or placebo (microcrystalline cellulose) in identical-looking capsules across two counterbalanced sessions separated by one week to ensure drug washout (T_1/2 = 8 hours). However, post hoc analyses after unblinding at the conclusion of the study revealed strategic changes in learning following the first session, which resulted in significant differences in behavior between the two sessions (S1 Text; S1 Fig). Consequently, in this paper, we analyze the effect of drug versus placebo in the first session only using a between-subjects design [e.g., 13].

Participants were randomized according to a pre-determined schedule prepared by an independent researcher. After ingesting the corresponding capsule in each session, participants were seated for 2 hours to allow for sulpiride to reach peak plasma concentrations before commencing the practice blocks [50–52]. We took baseline and hourly measures of heart rate and blood pressure, and reports of subjective state using a digital version of the Bond and Lader Visual Analogue Scales [53].

Drug reactions and blinding

Sulpiride did not alter heart rate or blood pressure relative to placebo (S1 Text; S2 Fig). There were no effects of sulpiride versus placebo on subjective reports of feeling strong/feeble, well-coordinated/clumsy, or lethargic/energetic, nor on aggregated factors of alertness, contentedness, and calmness. Small differences in feeling alert/drowsy did not survive correction for multiple comparisons (S1 Text; S3 Fig). No participant reported any adverse drug reactions during the testing session.

At the conclusion of testing, participants were asked whether they believed they had ingested the placebo, the active drug, or if they were unsure. These data were then used to calculate the Bang Blinding Index (BBI) for each drug group (1 = complete lack of blinding; 0 = perfect blinding; −1 = complete attribution of assignment to alternate group; [54]). Importantly, the BBI in both groups was close to zero, which reflects effective blinding in our study (placebo, BBI = 0 ± 0.17 (mean ± SEM), 95% CI = [−0.33, 0.33]; sulpiride, BBI = 0.26 ± 0.15, 95% CI = [−0.04, 0.56]; S1 Text). In addition, a supplementary analysis demonstrated that behavior was not significantly related to whether participants were able to correctly guess their assigned drug group (S1 Text).

Behavioral paradigm

To examine the effects of effort on choice and learning, we used a reward learning paradigm involving the exertion of effort [Experiment 2 in 40]. Specifically, participants in both drug groups completed a two-armed bandit task in which they were required to register their responses by applying pre-specified levels of physical force (‘low’ or ‘high’) to a pair of hand-held dynamometers (SS25LA, BIOPAC Systems, USA). Target force levels were standardized for each participant as proportions of their maximum voluntary contraction (MVC; low = 5%; high = 44%). MVCs for each hand were defined at the beginning of the experiment as the maximum force generated from three ballistic contractions with the corresponding dynamometer.

The experimental task consisted of two blocks of 150 trials. On each trial, participants were presented with a pair of abstract stimuli (fractals) on the left and right of the screen, and were required to select which was more rewarding based on probabilistic feedback received on previous trials (Fig 1A). The left/right location of each stimulus was randomized on every trial. On any given trial, both stimuli could have a high probability of being rewarded (P = 0.7); both could have a low probability (P = 0.3); or one could be superior to the other (P = 0.7 versus P = 0.3). Stimulus-reward contingencies changed after every 12 or 24 trials according to a pseudorandomised sequence, and contingency changes were not signaled to participants (Fig 1B). On rewarded trials, a ‘smiley face’ was presented for 0.5 s accompanied by a positively-valenced auditory tone (‘cash register’ sound effect). On nonrewarded trials, a ‘sad face’ was presented for 0.5 s with a negatively-valenced tone (‘wrong answer buzzer’ sound effect). Participants had a maximum of 2 s to register a response on each trial, otherwise a ‘Too slow!’ message was displayed for 0.5 s and then the next trial began. Participants were incentivised by the opportunity to increase their remuneration based on their performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Reward-based learning paradigm with effort manipulation.

(A) Participants made a series of choices between two abstract stimuli by applying physical force to a pair of hand-held dynamometers. One stimulus required a negligible amount of force to select (low effort stimulus; >5% MVC) and the other a greater amount of force (high effort stimulus; >44% MVC). Participants, therefore, had to balance an aversion to the high effort stimulus against their desire to maximize reward. (B) Stimulus-reward contingencies across a single block of 150 trials. The probability of reward upon selecting a given stimulus was either P = 0.7 or P = 0.3. These contingencies reversed every 12 or 24 trials, and reversals were not signaled to participants. Fractal images in panel (A) are by Optoskept, reproduced from https://commons.wikimedia.org under a Creative Commons Attribution 4.0 International license.

https://doi.org/10.1371/journal.pbio.3003765.g001

A critical feature of this task was that one of these two stimuli was always designated the ‘low effort’ stimulus, and could be selected by exerting only a negligible amount of force (>5% MVC). The other was designated the ‘high effort’ stimulus, and required a higher amount of force to be selected (>44% MVC). These stimulus-effort mappings remained constant for the duration of the experiment, and participants were explicitly informed about the identity of the low and high effort stimuli at the start of the testing session. To reinforce these stimulus-effort mappings, participants performed a preliminary block of 50 trials in which they were cued to generate the force corresponding to either the low or high effort stimulus (randomly determined). Participants then received binary feedback (correct versus incorrect) about whether they had generated the correct amount of force (5%–44% of MVC for the low effort stimulus, or >44% MVC for the high effort stimulus). In the subsequent experimental blocks, participants thus had to incorporate a consideration of both the effort required to select each stimulus, which was known in advance, as well as the expected reward, which had to be learned during the task.

The experiment was run in Psychtoolbox [55] implemented in Matlab version 9.4 (2018) [56], and presented on a monitor at a viewing distance of ~60 cm.

Data analysis

Computational models of learning.

We investigated interactions between effort and learning on a trial-by-trial basis by considering a family of three computational models (M1-3) adapted from previous work [40]. All models shared a common structure that combined core features of the classical Rescorla-Wagner model of reinforcement learning [60] with canonical models of effort discounting [25]. The core learning model stated that, on every trial (t), the value (V) of the chosen stimulus (s) is updated according to an RPE (δ) (Eq. 1), which is the difference between the reward obtained (r = 0 or 1) and the reward expected based on the current stimulus value (V) (Eq. 2). The extent to which V is updated by δ is determined by the learning rate (α), which takes a value between 0 and 1.

(1)(2)

To model the effect of effort on decision-making, we also incorporated an effort discounting function [25], which reduces the reward value associated with a stimulus by the amount of effort required to select it. We thus distinguish reward value (V), which participants can learn using trial-by-trial reward feedback, from action value (V’), which also accounts for the effort required to select the stimulus in question (Fig 2B). This effort cost enters the model as E_c = 0.05 for the low effort stimulus and E_c = 0.44 for the high effort stimulus, scaled by a subject-specific effort discounting parameter (k) to capture each individual’s aversion to effort (Eq. 3).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Candidate computational models of effort-based learning.

(A) Schematic diagrams of candidate models (M1-3) depicting learning rates (α) (top row), and their predicted effects on RPEs (bottom row). M1 includes a single learning rate, M2 fits separate learning rates for positive and negative RPEs, and M3 stipulates that learning rates are sensitive to trial-by-trial effort exertion (E_x). (B) The action value (V’) associated with a given stimulus is modeled as its expected reward value (V) discounted by the effort cost required to select it (E_c). Plot depicts V = 0.75, k = 0.5. (C and D) Candidate models were simulated 100 times, yielding model recovery accuracy ≥ 0.84 for all models (C), and parameter reliability ≥ 0.7 for all parameters (D).

https://doi.org/10.1371/journal.pbio.3003765.g002

(3)

Finally, action values are converted into choice probabilities using a softmax function [61], in which the probability (P) of choosing a given stimulus (s₁) depends on its associated action value relative to that of the nonchosen stimulus (s₂; Eq. 4). An inverse temperature parameter (β) accounts for individual differences in choice stochasticity.

(4)

All three candidate models shared this common structure (Eq. 1–4), and differed only in how they estimated the learning rate (α). To allow for the possibility that learning rate varies at the level of single trials, we first defined α as a sigmoidal function of a signal gain term (G) [40].

(5)

We then let the specific contents of G be the defining feature of each candidate model (Fig 2A).

Baseline model (M1).

Model M1 reduces to a standard Rescorla–Wagner model, in which individuals learn equally well from positive and negative reward outcomes. In this model, G is estimated directly via a subject-specific signal gain parameter (γ).

(6)

Dual Learning Rates model (M2).

Model M2 was similar to M1, but accounted for the possibility that learning rates differ following positive and negative reward outcomes. For example, a common finding in reward learning paradigms is that learning rates tend to be higher for positive relative to negative RPEs [8,37–39]. Model M2 accounts for this possibility by estimating the difference between positive and negative learning rates with an additional free parameter (). This model allows for learning to be more efficient from positive than negative RPEs ( > 0), vice versa ( < 0), or equally efficient from both ( = 0).

(7)

Note that, like M1, this model assumes that effort has no effect on learning rates.

Effort Reinforcement model (M3).

Finally, based on our previous work, we included a model that accounts for the possibility that within-subject variations in learning rate are sensitive to the amount of effort invested in each choice [40]. This model is similar to M2 in assuming that learning rates are asymmetrical (i.e., differ following positive and negative reward outcomes). In M3, however, a parameter scales the effect of effort exerted (E_x) on the current trial (), defined as the peak amplitude of force exerted as a proportion of the participant’s MVC. This model allows for the possibility that effort boosts learning from positive compared to negative RPEs ( > 0), vice versa ( < 0), or that effort has no effect on learning rates ( = 0).

(8)

Thus, in both M2 and M3, the parameter captures the degree of learning rate asymmetry. The critical distinction is that, in M2, this parameter is not dependent on effort (). In contrast, in M3, this parameter captures the degree to which effort modulates learning by scaling the effect of effort on learning rate asymmetry ().

Computational model fitting and comparison

Candidate models were fit to the observed choice data using maximum likelihood estimation. The best-fitting parameter values were estimated for each participant separately, using flat priors for all parameters. Model fits were compared based on the Akaike Information Criterion (AIC), which prevents overfitting by penalizing models that have a greater number of free parameters [62]. AIC scores were summed across participants to calculate overall model fits in each drug group. We also calculated Akaike weights (AW) to quantify the relative likelihood that the winning model best accounted for the observed data compared to others in the model space (Eq. 9):

(9)

where is the AIC weight of model ; is the difference in AIC between model and the winning model; and is the number of models in the space.

We confirmed that each candidate model was uniquely identifiable by conducting a model recovery analysis. We ran 100 simulations per model, each of which generated synthetic data from 20 learning agents completing the experiment. We then repeated our model fitting procedure on these synthetic data and calculated the proportion of simulations on which the true generative model was successfully recovered as the winning model (Fig 2C). We also quantified the reliability of the parameter estimates from each model as the median rank-order correlation (Spearman’s ρ) between the true generative values and the recovered values across all 100 simulations (Fig 2D). Computational modeling was performed in Matlab version 9.4 (2018) [56].

Sulpiride did not alter motor performance or effort preference relative to placebo

To test whether sulpiride affected motor capacity, we analyzed participants’ force generation. First, we found no significant difference in MVCs recorded during the calibration phase between the sulpiride and placebo groups (p = .54; Fig 3A). We also confirmed that participants were able to successfully meet the required effort thresholds when registering responses in the experimental blocks (<0.7% of trials missed per participant on average), and found that this did not differ significantly between drug groups (z = −1.46, p = .15). Next, we used a two-way ANOVA to examine the average peak force exerted per response in the experimental blocks (defined as a proportion of each participant’s MVC). Average effort exertion was compared between Drug groups (placebo, sulpiride) as a function of the Stimulus type (low effort, high effort). As expected, the main effect of Stimulus was significant, such that participants exerted significantly more effort when selecting the high versus low effort stimulus (ΔMVC = 0.41 ± 0.01 (mean ± SEM), t(40) = −36.8, p < .001). However, neither the main effect of Drug nor the two-way interaction was significant (both p ≥ .33; Fig 3B). We then ran an equivalent ANOVA testing differences in the standard deviation of peak force across responses (Effort SD). Importantly, the main effect of Drug group was not significant (ΔEffort SD = 0.01 ± 0.01 (mean ± SEM), F(1,40) = 3.36, p = .074; Fig 3C). The effect of Stimulus in this case was not significant (main and interaction both p ≥ .22; Fig 3C). Finally, we compared the overall effort bias of each group based on the tendency of participants to select the less effortful option. We found no significant difference in the proportion of trials on which the low versus high effort stimulus was chosen between the two groups (p = .33; Fig 3D).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Motor and learning performance in the sulpiride and placebo groups.

(A-C) Motor performance did not differ significantly between drug groups with respect to (A) MVCs, (B) mean effort exerted during the experimental blocks, or (C) standard deviation of effort during the experimental blocks. Dotted lines in (B) indicate minimum effort thresholds for the low and high effort stimulus, respectively. (D) Sulpiride did not affect preferences for low vs. high effort actions compared to placebo. (E–G) Learning performance differed between drug groups. Relative to placebo, participants on sulpiride exhibited (E) significantly lower choice accuracy (p = .042), and (F) were less likely to repeat rewarded choices (p = .029), but (G) were no different in their tendency to switch stimulus following non-rewarded choices (p = .4). In addition, across both drug groups, participants were less likely to repeat high effort choices than low effort choices, consistent with effort aversion (p ≤ .009; F and G). Accuracy in panel E is based on trials with reward contingencies of P = 0.7 vs. 0.3. Error bars depict the standard error of the mean. Statistical significance marks are shown for differences between drug groups. *p < .05; **p < .01; PL, placebo (n = 19); SP, sulpiride (n = 23). Data underlying this Figure can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.3003765.g003

In sum, we found no strong evidence that sulpiride affected either the motor capacity or the willingness of participants to produce forceful responses compared to placebo.

Sulpiride impaired reward-based learning relative to placebo

Having established that sulpiride did not affect motor performance relative to placebo, we next tested whether sulpiride affected performance in the reward learning task. To do so, we first compared drug groups on choice accuracy, defined as the proportion of trials on which participants chose the more highly rewarded stimulus (on trials with unequal stimulus-reward contingencies). We used a two-way ANOVA to compare Accuracy between Drug groups (placebo, sulpiride) as a function of the Stimulus type (low effort, high effort). This revealed that, across both stimulus types, sulpiride reduced overall choice accuracy compared to placebo (Drug, ∆Accuracy = 0.04 ± 0.02, t(40) = 2.1, p = .042; Stimulus main effect and interaction, p ≥ .15; Fig 3E).

We next ran analogous ANOVAs to investigate whether sulpiride altered the tendency to make Win-stay and Lose-switch choices, as more specific measures of reward-guided behavior. We found a significant main effect of Drug in the Win-stay analysis, indicating that the sulpiride group was less likely than the placebo group to repeat rewarded choices (∆Win-stay = 0.05 ± 0.02, t(40) = 2.27, p = .029; Fig 3F). This analysis also revealed a main effect of Stimulus type, which reflected that, across both groups, participants were less likely to repeat rewarded high effort choices than rewarded low effort choices (∆Win-stay = −0.02 ± 0.01, t(40) = −2.75, p = .009; Fig 3F). Notably, the Drug × Stimulus interaction was not significant (p = .89).

In the Lose-switch analysis, we found no significant difference between Drug groups (p = .4; Fig 3G). However, we again found a significant main effect of Stimulus type, which in this case indicated that participants were more likely to switch stimulus following nonrewarded high effort choices than nonrewarded low effort choices (∆Lose-switch = 0.03 ± 0.01, t(40) = 2.9, p = .006; Fig 3G). Together with the Win-stay analysis, these effects of Stimulus type are indicative of effort aversion, such that participants were in general less likely to repeat high effort choices than low effort choices. Again, the Drug × Stimulus interaction effect was not significant, providing further evidence that sulpiride did not alter effort aversion in this study (p = .75).

In sum, these analyses reveal that overall reward-related performance was lower on sulpiride than placebo, and that this was associated with changes in win-stay, but not necessarily lose-switch, behavior.

Effort modulated learning rates on placebo, but not sulpiride

We next turned to our central question of how effort modulates learning, and the role of the dopamine D2-receptor in maintaining this relationship. We compared three candidate reinforcement learning models (M1-3) in each drug group (see Methods section for details): a Baseline model (M1) with a single learning rate; a Dual Learning Rates model (M2) capturing different learning rates for positive and negative reward outcomes; and an Effort Reinforcement model (M3) in which learning rates are sensitive to trial-by-trial effort exertion [40].

In the placebo group, we found that choice behavior was best captured by the Effort Reinforcement model (M3). This model yielded a superior fit compared to alternative models (∆ AIC ≥ 16.59; Akaike weight > 0.99; Fig 4A–4C). Analysis of the key parameters in this model indicated that, as expected, the k parameter was significantly greater than zero, indicating that the prospect of exerting effort discounted the value of candidate actions (k = 0.24 ± 0.06, t(18) = 3.97, p < .001; Fig 5A). In addition, the parameter was significantly greater than zero, indicating that the overall effect of effort in this group was to increase learning rate asymmetry, boosting learning rates following positive reward outcomes, and blunting learning rates following negative reward outcomes ( = 0.95 ± 0.37, t(18) = 2.59, p = 0.019; Fig 5A). To confirm that this model was not merely approximating dual learning rates independent of effort, we ran a permutation test in which we randomly shuffled the effort exerted across trials for each participant in the placebo group and re-fit the model. The empirical data provided a better fit than the permuted data on each of 1,000 permutations, confirming that learning rates in this group were sensitive to effort at the level of single trials (permuted p < .001).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Computational modeling results for the placebo and sulpiride groups.

(A–C) In the placebo group (top row, blue, n = 19), the empirical data were best explained by the Effort Reinforcement model (M3), which stipulates that learning rates are modulated by the exertion of effort. This model provided a superior fit to models M1 and M2 based on AIC scores (A) and Akaike weights (B). (C) M3 also provided an accurate account of observed choices on a trial-by-trial basis. The black line shows the proportion of participants on each trial selecting the low effort stimulus (with standard error in gray), and the blue line shows the mean choice probabilities derived from M3. (D–F) In the sulpiride group (bottom row, red, n = 23), the empirical data were best explained by the Dual Learning Rates model (M2), in which effort has no effect on learning rates. This model was superior to the alternative models in this group on the basis of AIC scores (D) and AIC weights (E), and provided an accurate account of observed behavior on sulpiride (F). Data underlying this Figure can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.3003765.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Computational parameter estimates in the placebo and sulpiride groups.

(A) Parameter estimates from the Effort Reinforcement model (M3) in the placebo group (n = 19), including signal gain (γ), inverse temperature (β), effort discounting (k), and effort-sensitive learning rate asymmetry (). k and parameters were both significantly greater than zero, confirming that the prospect of effort was aversive (p < .001), and that the exertion of effort increased positive and decreased negative learning rates (p = .019). (B) k and parameters were positively correlated (p < .001), indicating that effort modulated learning rates to a greater extent in those more averse to exerting it. (C) Parameter estimates from the Dual Learning Rates model (M2) in the sulpiride group (n = 23). The k parameter was significantly greater than zero (p = .021), consistent with effort aversion. Effort-insensitive learning rate asymmetry () was also positive overall (p < .001), indicating higher positive than negative learning rates. (D) k and parameters were not significantly correlated in the sulpiride group (p = .33). Data underlying this Figure can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.3003765.g005

We then repeated the same model comparison in the sulpiride group. Critically, choice behavior in these participants was best captured by the Dual Learning Rates model (M2), in which learning rates are entirely independent of effort (∆ AIC ≥ 34.41; Akaike weight > 0.99; Fig 4D–4F). As in the placebo group, k values were significantly greater than zero, indicating that individuals found effort to be aversive (k = 0.12 ± 0.05, t(22) = 2.48, p = .021; Fig 5C). Consistent with prior work, the parameter was also positive, indicating that learning rates were higher following positive outcomes than negative outcomes ( = 2.06 ± 0.46, t(22) = 4.52, p < .001; Fig 5C).

Together, these results demonstrate that, across both the placebo and sulpiride groups, learning rates were higher for positive than negative reward outcomes. In the absence of dopaminergic blockade, effort reinforced learning by increasing learning rates for positive RPEs, and decreasing learning rates for negative RPEs [40]. Critically, however, this effect of effort on learning was absent in the sulpiride group, thus highlighting the importance of the dopamine D2-receptor in maintaining the effect of effort on reward learning.

On placebo, effort-sensitive learning rates attenuate the detrimental effect of effort aversion on accuracy

These results raise the question of why the exertion of effort should modulate learning at all, and whether dopamine plays an adaptive role in maintaining this relationship. To address these questions, we first considered the placebo group, and asked whether the degree of effort discounting (k) was related to the degree to which effort modulated learning rate asymmetry () in the winning model (M3). Importantly, we found that these parameters were positively correlated, such that those with higher k values also had higher values (r = 0.77, p < .001; Fig 5B). This suggests that individuals whose learning was most sensitive to effort exertion were also those least willing to choose high effort actions in the first place. We used a permutation test to verify that this significant correlation was not merely due to these parameters trading off during model fitting [63]. On each permutation, we generated synthetic data from M3 using randomly sampled, uncorrelated k and values for 20 participants. Across 1,000 permutations, the recovered correlation between these parameters was as or more extreme than our empirical result on just three occasions, confirming that the relationship between and in the placebo group was most likely driven by a true correlation between effort discounting and effort reinforcement (permuted p = .003).

One possible interpretation of this result is that effort plays an adaptive role by facilitating learning in those who are averse to exerting it in the first place. To test this possibility, we first fit a linear regression model predicting mean accuracy in the placebo group as a function of effort aversion (k), learning rate asymmetry (φ), and their interaction. Although none of these parameters was significant (p ≥ .21), the direction of the interaction effect would be consistent with an adaptive role for effort in participants most averse to investing it (S1 Text; S5 Fig). To investigate this more thoroughly, we ran a simulation analysis testing the predictions of model M3 in four groups of n = 100 simulated agents (Fig 6A, 6B). Each group varied according to their degree of effort aversion (‘low’, 0 ≤ k ≤ 0.15; ‘high’, 0.45 ≤ k ≤ 0.6), and the degree to which effort modulated their learning rate asymmetry (‘low’, 0 ≤ ≤ 1; ‘high’, 3 ≤ ≤ 4). Signal gain and temperature parameters were fixed at plausible values close to the observed group means (γ = 2, β = 6).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Simulated learning performance under conditions of intact and disrupted dopamine signaling.

(A) Learning performance simulated using model M3 at positive values of learning asymmetry (; x-axis) and effort aversion (; color shades). Each data point depicts mean choice accuracy (y-axis) based on 1,000 learning agents (total N = 25,000). (B) Choice accuracy (mean ± SEM; y-axis) of M3-simulated learning agents with low vs. high effort aversion (; x-axis), further divided into those with low vs. high learning rate asymmetry (; white vs. black markers). Each group includes 100 simulated agents (total N = 400). The model predicts that, with preserved D2-receptor function, having more effort-sensitive learning rates improves the learning performance of individuals more averse to investing effort (p < .001). (C) Simulation of model M2 (N = 25,000) at positive values of at positive values of learning asymmetry (; x-axis) and effort aversion (; color shades). (D) Choice accuracy (mean ± SEM; y-axis) of M2-simulated individuals with low vs. high effort aversion (; x-axis), further divided into those with low vs. high learning rate asymmetry (; white vs. black markers). Each group includes 100 simulated agents (total N = 400). The model predicts that following D2-receptor blockade, effort aversion is detrimental to performance irrespective of learning rate asymmetry (p < .001). Data underlying this Figure can be found in S1 Data.

https://doi.org/10.1371/journal.pbio.3003765.g006

We formally tested the interaction between and in a two-way, between-subjects ANOVA comparing overall choice accuracy. This revealed a significant k × interaction (F(1,396) = 10.1, p = .002), such that greater effort aversion (high versus low k) led to lower choice accuracy in the low group (∆ Accuracy = −0.042 ± 0.009, t(396) = −4.99, p < .001), but not the high group (p > .99). Accordingly, in simulated agents who were most averse to investing effort (high k), having learning rates that were more sensitive to effort (high versus low ) significantly improved learning performance (∆ Accuracy = 0.038 ± 0.009, t(396) = 4.49, p < .001; low k group, p > .99).

This result indicates that agents who were less motivated to invest effort (high k) had lower accuracy relative to those who were more motivated (low k). Importantly, however, this effect was only seen in those whose learning rates were less sensitive to effort (low ), and not in those whose learning rates were more sensitive to effort (high ). This suggests that effort-sensitive learning rates may confer a learning advantage by significantly improving performance in those who are more averse to effort, while at least maintaining performance in those who are less averse to effort.

On sulpiride, greater effort aversion impairs accuracy regardless of learning rate

To investigate whether dopamine plays a role in maintaining this adaptive relationship between effort and learning rates, we performed the corresponding simulation on the sulpiride group using the M2 model. We began by testing whether effort discounting (k) in this group was significantly related to the asymmetry of learning rates (). Recall that, in this model, is not sensitive to effort, and therefore should not be expected to correlate with effort discounting. Accordingly, this correlation was not significant (p = .33; Fig 5D), confirming that effort discounting did not influence learning rate asymmetry once effort exertion was decoupled from learning.

We examined the predictions of model M2 by simulating learning performance in four groups of n = 100 agents using the same range of parameter values as in the previous analysis. Specifically, groups had fixed signal gain (γ = 2) and inverse temperature parameters (β = 6), but varied according to their degree of effort aversion (‘low’, 0 ≤ k ≤ 0.15; ‘high’, 0.45 ≤ k ≤ 0.6), and learning rate asymmetry (‘low’, 0 ≤ ≤ 1; ‘high’, 3 ≤ ≤ 4; Fig 6C, 6D). A two-way ANOVA showed significant main effects of both k and . In particular, accuracy was significantly lower in agents who were more versus less averse to effort (∆ Accuracy = −0.036 ± 0.006, F(1,396) = 34.03, p < .001). In addition, accuracy was significantly lower in those exhibiting large versus small learning rate asymmetries (∆ Accuracy = −0.045 ± 0.006, F(1,396) = 51.72, p < .001). Importantly, the k × interaction was not significant (p = .27), indicating that learning performance was impaired in agents who were less motivated to invest effort, and this detrimental effect of low motivation was seen regardless of the degree of learning rate asymmetry.

Together with the preceding simulations, these results reveal that D2-receptor blockade removes a mechanism that can compensate for decrements in performance that otherwise result from being less motivated to exert effort.