Theory of performance-gated deliberation

Opportunity costs of deliberation and commitment, and drawbacks of average-reward reinforcement learning.

We consider a class of tasks consisting of a long sequence of trials indexed by k = 1, 2, … (see Fig 1A), each of which provides the opportunity to obtain some reward by choosing correctly. In each trial, a finite sequence of states, S_t, t = 0, …, t_max, is observed that provide evidence for an evolving belief about the correct choice among a fixed set of options. To keep notation simple, we suppress denoting the trial index, k, on quantities such as trial state, S_t, that also depend on trial time, t. The time of decision, , and the chosen option determine both the reward received, R_k, and the trial duration, . Importantly, decision timing can affect performance because earlier decisions typically lead to shorter trials (and thus more trials in a given time window), while later decisions lead to higher accuracy. Effectively balancing such speed-accuracy trade-offs is central to performing well in continuing episodic task settings. For a fixed strategy, the stationary reward rate (see slope of dashed line in Fig 1A(right)) is (1) For a stochastic task environment, the definition of ρ includes an average over different realizations of the task. Free-operant conditioning, foraging, and several perceptual decision-making tasks often fall into this class. Previous work [8, 28] has studied the belief of correct report for binary rewards, b_t = P(R_k = 1|S_t, t^dec = t), which also gives the expected trial reward, [8] (see [29] for more about the relationship between value-based and perceptual decisions). S_t denotes the state sequence observed so far, (S₀, …, S_t). We consider greedy strategies that report the choice with the largest belief at decision time. The decision problem is then about when to decide.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. AR-RL and performance-gated deliberation.

(A) Task setting. Left: Within trial state, S_t evolves over trial time t in successive trials indexed by k. The decision ‘A’ is reported at the decision time (red cross), determining trial reward, R_k, and trial duration, T_k. Right: Sketch of cumulative reward versus cumulative duration. Context-conditioned reward rate (slope of red line), varies with alternating context (labelled 1 and 2) around average reward, ρ (dashed line). (B) Decision rules based on opportunity costs of commitment, , and deliberation, . The AR-RL rule (black ‘x’) finds t that minimizes . The PGD rule (black cross) finds t^dec at which they intersect, . (C) Schematic diagram of each algorithm’s dependency. PGD computes a decision time directly from the two opportunity costs, while AR-RL uses both to first estimate a value function, whose maximum specifies the decision time. (D) Loss (error in performance with respect to the optimal policy, (ρ* − ρ)/ρ*) over learning time in a patch-leaving task (AR-RL: brown, PGD: black). The arrow indicates when the state labels were randomly permuted.

https://doi.org/10.1371/journal.pcbi.1010080.g001

Average-reward reinforcement learning (AR-RL), first proposed in artificial intelligence [30], was later incorporated into reward prediction error theories of dopamine signalling [5] and employed to account for the opportunity cost of time [7]. AR-RL was subsequently used to study reward-based decision-making in neuroscience and psychology [8, 9, 14, 31]. AR-RL centers around the average-adjusted future return, which penalizes the passage of time by using the average reward as a cost rate. Value is defined as this future return averaged over trial statistics. This average of a sum into the future of reward deviations from the average converges without a discount factor on account of the transient effects of conditioning the statistics on the state at which the decision is made. The AR-RL algorithms we consider aim to achieve the highest ρ by also maximizing the average-adjusted value (see Methods for details). We now provide an alternative, but equivalent definition of average-adjusted trial return in terms of opportunity costs incurred by the agent.

We denote the opportunity cost of committing at time t within a trial as , defined as the difference (2) where r_max is the maximum trial reward possible a priori. Within a trial, an agent lowers its commitment cost towards zero by accumulating more evidence, i.e. by waiting. Waiting, however, incurs another opportunity cost: the reward lost by not acting. We denote this opportunity cost of deliberation incurred up to a time t in a trial as . In AR-RL, the constant opportunity cost rate of time is integrated so that for , (3) With these definitions, the average-adjusted trial return for deciding at a time t can be expressed as . It is maximized by jointly minimizing and (Fig 1B), giving the AR-RL optimal solution (see Methods for a formal statement and solution of the AR-RL problem). Expressed in this way, the average-adjusted trial return emphasizes the more general perspective that an agent’s solution to the speed-accuracy trade-off is about how it balances the decaying opportunity cost of commitment and the growing opportunity cost of deliberation.

Despite their utility, value representations such as the average-adjusted trial return can be a liability in real world tasks where task statistics are non-stationary. To illustrate this, we consider the following foraging task. A foraging agent feeds among a fixed set of food (e.g. berry) patches. Total berries consumed in a patch saturates with duration t according to a given saturation profile, shared across patches, as the fewer berries left are harder to find. Patches differ in their richness (e.g. berry density), which is randomly sampled and fixed over the task. Denoting patch identity (serving as context) by s, the food return is directly observed and deterministic given s. To perform well, the agent needs to decide when to move on from depleting the current patch. Further details about the task and its solution are given in the Methods. For a broad class of online AR-RL algorithms, the agent learns the average-adjusted trial return as a function of state and time. For a given patch, it then leaves when this return is at its maximum (c.f. Fig 1B). In Fig 1D, we show how the performance (brown line) approaches that of the optimal policy in time as the estimation of the AR-RL trial return improves with experience (see Methods for implementation details). However, if the agent’s environment undergoes a significant disturbance (e.g. a forest fire due to which the patch locations are effectively re-sampled), the performance of this AR-RL algorithm can drop back to where it started. We implement such a disturbance via random permutation of the state labels at the time indicated by the arrow in Fig 1D. This is true over a range of learning rates and the number of patches (S8 Fig). More generally, any approach that relies on estimating state-value associations shares this drawback, including those approaches that implicitly learn those associations by directly learning a policy instead [32]. Could context-dependent decision times be obtained without having to associate value or action to state? A means to do so is presented in the next section.

Reward filtering for a dynamic opportunity cost of deliberation.

The state disturbance in the toy example above altered task statistics at only a single time point. In general, however, changes in task statistics over time can occur throughout the task experience. A broader notion of deliberation cost beyond the static average reward is thus needed–one that can account for extended timescales over which performance varies. Such a cost serves as a dynamic reference in a relative definition of value based on a non-stationary opportunity cost of time. We first address how performance on various timescales can be estimated.

As a concrete example, we make use of the task that we will present in detail in the following section. This task has a context parameter, α, that can vary in time on characteristic timescales longer than the moment-to-moment and can serve as a source of non-stationarity in performance. Here, the context sequence, α_k, varies on a single timescale, e.g. through periodic switching between two values. The resulting performance (Fig 2A(top)) varies around the stationary average, ρ (purple), with context variation due to the switching (orange), as well as context-conditioned trial-to-trial variation (blue). The decomposition of time-varying performance into these multiple, timescale-specific components can be achieved by passing the reward signal through parallel filters, each designed to retain the signal variation specific to that timescale (Fig 2A(bottom)). There are multiple approaches to this decomposition. We chose a heuristic approach in which the performance over a finite memory timescale can be estimated by filtering the sequence of rewards through a simple low-pass filter [9, 33]. This filter is defined by an integration time, τ, tuned to trade off the bias and variance of the estimate in order to best capture the variation on the desired timescale (e.g. how performance varies over different contexts). We denote such an estimate , and show in the Methods that it approximates the average reward over the last τ time units. We discuss the question of biological implementation in the discussion, but note here that the number and values of τ needed to represent performance variation in a given task could be learned or selected from a more complete set in an online fashion during task learning. In an experimental setting, these learned values can in principle be inferred from observed behaviour and we developed such an approach in the analysis of data that we present in the following section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Non-stationary opportunity cost.

(A) Top: Dynamics of trial performance (; blue) with its distribution as well as dynamics of between context-conditioned averages of performance (; orange), and the effectively stationary average performance (; purple). Bottom: these are decomposed into a hierarchy by filtering reward history on trial, context, and long timescales, respectively. (B) Two hypothetical forms for context-specific trial opportunity cost. Top: Trial-unaware cost in which context varies the slope around ρ. Bottom: Trial-aware cost in which context variation is through a bias (c.f. Eq 5).

https://doi.org/10.1371/journal.pcbi.1010080.g002

Applying this heuristic decomposition here, the stationary reward rate, ρ, can be estimated to high precision by using a long integration time, τ_long, to the reward sequence R_k, producing the estimate . If α_k were a constant sequence, , the stationary opportunity cost of deliberation Eq 3 of AR-RL. However, in this example context varies on a specific timescale, to which the former is insensitive. Thus, a second filtered estimate is needed to estimate performance on this timescale. Unlike , this estimate tracks the effective instantaneous, context-specific performance, . Its estimation error arises from a trade-off, controlled by the integration time, τ_context, between its speed of adaptation and its finite memory.

We consider two distinct hypotheses for how to extend AR-RL to settings where performance varies over context. The first hypothesis, , is the straightforward, trial-unaware extension of Eq 3, shown in Fig 2B(top). Here, performance is tracked only on a timescale sufficient to capture context variation and the corresponding cost estimate, , is incurred moment-to-moment, neglecting the trial-based task structure. However, this incorrectly lumps together two distinct opportunity costs: those incurred by moment-by-moment decisions and those incurred as a result of the effective planning implied by performance that varies over context. In particular, context is defined over trials not moments, and thus the context-specific component of opportunity cost of a trial is a sunken cost paid at the outset of a trial. This inspires a second trial-aware hypothesis (5) Eq 5 is plotted over trial time t in Fig 2B(bottom). Its first term is the AR-RL contribution from the stationary opportunity cost of moment-to-moment decisions using the stationary reward rate, ρ estimated with . The second, novel term in Eq 5 is a context-specific trial cost deviation incurred at the beginning of each trial and computed as the average deviation in opportunity cost accumulated over a trial from that context (T_α is the average duration of a trial in context α). This deviation fills the cost gap made by using the stationary reward rate ρ in the moment-to-moment opportunity cost instead of the context-specific average reward, ρ_α. This baseline cost derived from the orange time series in Fig 2A(bottom) vanishes in expectation, as verified through the mixed-context ensemble average reward (e.g. ρ ≡ ∑_α ρ_αT_α/∑_α T_α when the context is distributed evenly among trials such that ∑_α(ρ_α − ρ)T_α = 0). Thus, this opportunity cost reduces to that used in AR-RL when ignoring context, and suggests a generalization of average-adjusted value functions to account for non-stationary context. We estimate this baseline cost using , where we have used the sample T_k−1 in lieu of the average T_α. See S1 Fig for a signal filtering diagram that produces this estimate of Eq 5 from reward history. A main difference between the cost profiles from the two hypotheses is the cost at early times. Both the behaviour and neural recordings we analyze below seem to favor the second, trial-aware hypothesis Eq 5. We hereon employ that version in the main text, and show the results for the trial-unaware hypothesis in S7 Fig.

Neuroscience application: PGD in the tokens task

In this section, we apply the PGD algorithm to the “tokens task” [22]. We first give a simulated example with periodic context dynamics. We then present an application to a set of non-human primate experiments in which context variation was non-stationary [25]. For the latter, we used the decision time dynamics over trials to fit a model for each of the two subjects. We then validated the models by assessing their ability to explain (1) the concurrently recorded behaviour via their context-specific behavioural strategies and (2) the neural activity in premotor cortex (PMd) via the temporal profile of the underlying neural urgency signals.

In the tokens task, the subject must guess as to which of two peripheral reaching targets will receive the majority of tokens that randomly jump, one by one every 200ms, from a central pool initialized with a fixed number of tokens. Importantly, after the subject reports, the time interval between remaining jumps contracts to once every 150ms (the “slow” condition) or once every 50ms (the “fast” condition), giving the subject the possibility to save time by taking an early guess. The interval contraction factor, 1 − α, for slow (α = 1/4) and fast (α = 3/4) condition is parametrized by α ∈ [0, 1], the incentive strength to decide early, which then serves as the task context. The subject is thus tasked with learning the statistics of the number of tokens at the end of the trial conditioned on an intermediate state in order to balance accuracy and deciding early.

In contrast to the patch leaving task example from Section A, the tokens task has many within-trial states and the state dynamics is stochastic. With the t^th jump labelled S_t ∈ {−1, 1} serving as the state, for the purposes of prediction, the history of states can be compressed into the tokens difference, , between the two peripheral targets with N₀ = 0. The dynamics of N_t is an unbiased random walk (see Fig 3A), with its current value sufficient to determine the belief of a correct report, b_t (computed in Methods). Since for binary rewards, b_t is also the expected reward, N_t is also sufficient for determining the opportunity cost of commitment, (Eq 2). We display this commitment cost dynamics in Fig 3B. It evolves on a lattice (gray), always starting at 0.5 (for p = 1/2) and ending at 0 for all p. We assume the agent has learned to track this commitment cost. The PGD agent uses this commitment cost, along with the estimate of the trial-aware deliberation cost, to determine when to stop deliberating and report its guess.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. PGD agent performs the tokens task for periodic context switching.

(A) A tokens task trial. Left: Tokens jump from a center to a peripheral region (gray circles). Right: The tokens difference, N_t, evolves as a random walk that accelerates according to α (here 3/4) post-decision time, t^dec. The trial duration is T, which includes an inter-trial interval. (B) Decision dynamics in cost space obtained from evidence dynamics in (A). Commitment cost trajectories (gray lattice; thick gray: trial-averaged) start at and end at 0. Trajectory from (A) shown in black. t^dec (black cross) is determined by the crossing of the commitment and deliberation cost. (C) Incentive strength switches between two values every 300 trials. (D) Expected rewards filtered on τ_long (, purple) and τ_context (, green). Black dashed lines from bottom to top are ρ_α=1/4, ρ, and ρ_α=3/4.

https://doi.org/10.1371/journal.pcbi.1010080.g003

Fit to behavioural data from non-human primates and model validation.

Next, we fit a PGD agent to each of the two non-human primates’ behaviour in the tokens task experiments reported in [25] and compare to AR-RL solutions. As with the above example (c.f. Fig 3), trials were structured in alternating blocks of α = 1/4 and α = 3/4. Fig 4A shows context-switching α-sequence from these experiments, which, in contrast to the above example exhibits large, irregular fluctuations in block size (These were primarily as as result of the experimenter adapting to fluctuations in motivation of the subject. D. Thura. Personal communication).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. PGD model fit to NHP behaviour for non-stationary α-dynamics reported in Ref. [25].

(A) Block length sequence used in the experiment. (b,c) decision times (dots) aligned on the context-switching event type (fast-to-slow in gray; slow-to-fast in color) and averaged. Shaded regions are the standard error bounds of the models’ average decision times. (D) Error evaluated on a -plane cut through the parameter space at the best-fitting and (gray area indicates timescales within an order of magnitude of the end of the experiment). Contours show the first 10 contours incrementing by 0.01 error from the minimum (shown as a circle marker). Colors refer to subject, as in (B) and (C). (E) Same for at and . (F) Same for at and .

https://doi.org/10.1371/journal.pcbi.1010080.g004

So far, PGD has only two free parameters: the two filtering time constants, τ_long and τ_context. We anticipated only a weak dependence of the fit on the τ_long, so long as it exceeded the average duration of a handful of trial blocks enabling a sufficiently precise estimate of ρ. In contrast, the context filtering timescale, τ_context, is a crucial parameter as it dictates where the PGD agent lies on a bias-variance trade-off in estimating , the value of which determines the context-specific contribution to the deliberation cost (Eq 2). To facilitate the model’s ability to fit individual differences, we introduce a subjective reward bias factor, K, that scales the rewards fed into the performance filters. We also add a tracking-cost sensitivity parameter, ν, that controls τ_context to avoid wasting adaptation speed (see Methods for details). The latter made it possible to fit the asymmetric switching behaviour observed in the average decision time dynamics. With these four parameters, we quantitatively match the baselines and exponential-like relaxation of the average decision time dynamics around the two context switches (Fig 4B and 4C; see Methods for fitting details).

A comparison of the best-fitting parameter values over the two monkeys (Fig 4D–4F) suggests that the larger the reward bias, K (Fig 4E), the more hasty the context-conditioned performance estimate (the smaller τ_context), and the lower the sensitivity to the tracking cost (Fig 4F). This is consistent with the hypothesis that subjects withhold cognitive effort in contexts of higher perceived reward [9]. Along with the correspondence in temporal statistics of the behaviour (e.g. S6 Fig), the fitted model parameters for the two subjects provides a basis on which to interpret the subject differences in the results of the next section, in particular their separation on a speed-accuracy trade-off, as originating in the distinct reward sensitivity shown here.

To better understand where both the data and the learned PGD agent lie in the space of strategies for the tokens task, we computed reward-rate (AR-RL) optimal solutions for a given fixed context, α (here α ∈ [0, 1]), using the same approach as [8] (we confirmed that the conventional discount-reward value iteration achieved the same solution in the limit of the undiscounted case). In each of average-reward and discount-reward formulations, the dynamic programming approach involves iterating Bellman’s equation to obtain the optimal value functions from which the optimal policy and its reward rate can be obtained (see Methods for details). The optimal reward rate as a function α is shown in Fig 5A. The strategies generating these reward rates interpolate from the wait-for-certainty strategy at low α to the one-and-done strategy [34] at high α. The former decides when the success probability first hits 1 and the latter decides after a single token jump in the direction of that jump. The α-conditioned reward rates achieved by the two primates with their corresponding PGD model, and a reference human (Single subject behavioural data shared by Thomas Thierry.) are also shown in Fig 5A. They clearly fall below the optimal strategy, and, as expected, above the strategy that picks one of the three actions (report left, report right, and wait) at random.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Context-conditioned analysis of PGD and comparison to AR-RL models.

(A) Shown is the reward rate as a function of incentive strength, α. The AR-RL solution with no augmented cost (c = 0) interpolates between the wait-for-certainty strategy (brown) and the one-and-done strategy (red). We also show the slow and fast context-conditioned reward rates for the two primates (blue and orange circles) and the PGD model fitted to them (crosses). For reference, we show the mean+/-std.dev. of a forthcoming dataset of 32 humans. Reward rates for the human and non-human primates are squarely in between the best (black dashed) and uniformily random (gray) strategy. (b,c) The distribution over trials of differences in decision times between model and data, |Δt_dec| = |t_dec,data − t_dec,model|, conditioned on slow and fast block contexts. Solid lines are for PGD. Dotted lines are for the AR-RL solution using the cost rate, c*, with the lowest mean error. The residual sum of squares (RRS) for each model/block combination is displayed. (d-g) Interpolated state-conditioned survival probabilities, P(t^dec = t|N_t, t), over slow (d,f) and fast (e,g) blocks. White dotted lines show the P(t^dec = t|N_t, t) = 0.5 contour. (h,i) State-conditioned decision time frequencies (cross size) from AR-RL optimal decision boundaries across different values of the cost rate, c (colored crosses) for slow (h) and fast (i) conditions. Only samples with N_t < 0 and N_t > 0, respectively, are shown. For comparison, the reflected axes shows as gray crosses the state-conditioned decision time frequencies of the data.

https://doi.org/10.1371/journal.pcbi.1010080.g005

To confirm that this similarity in performance between PGD and the data arises from a better fit to the behaviour than AR-RL, we plotted the distribution of the differences between model and data decision times, |Δt_dec|, conditioned on the context (Fig 5B and 5C). For comparison with previous work [8] and to account for deliberation cost in AR-RL, we added to the AR-RL reward objective a constant auxiliary deliberation cost rate, c, incurred up to the decision time in each trial, and chose the cost rate, c*, that gave the lowest mean difference. In both contexts, PGD exhibits significantly lower error than this c* AR-RL solution (Kolmogorov-Smirnov two-sample test).

To reveal the source of this discrepancy in both performance and behaviour, we turned to analyzing the corresponding policies of PGD and c-based AR-RL agents. A robust and rich representation of the behavioural statistics is the state and time-conditioned survival probability that a decision has not yet occurred. It serves as a summary of the action policy associated with a stationary strategy (see Methods for its calculation from response times). Applied equally to the decision times of both model and data, it can provide a means of comparison even in this non-stationary setting. We give this conditional probability for each of the two contexts for subject 1 and its fitted PGD model in Fig 5D–5G. We left the many possible noise sources underlying the behaviour out of the model in order to more clearly demonstrate the PGD algorithm. However, such noise sources would be necessary to quantitatively match the variability in the data (e.g. added noise in the performance estimates leads to larger variability in the location of the decision boundary and thus also to larger spread in these survival probability functions). In the absence of these noise sources, we see the model underestimates the spread of probability over time and tokens state. Nevertheless, the remarkably smooth average strategy is well captured by the model (white dashed lines in Fig 5D–5G). Specifically, policies approximately decide once either of the peripheral targets receive a certain number of tokens. Comparing results across context, we find that fast block strategies (Fig 5E and 5G) exhibit earlier decision times relative to slow block strategies (Fig 5D and 5F) in both model and data. The strategies for subject 2 are qualitatively similar, but shifted to earlier times relative to subject 1 (S3 Fig). Our model explains this inter-individual difference as resulting from subject 2 having a larger reward bias, K, and faster context integration time, τ_context (c.f. Fig 4E).

The correspondence between the PGD model and data over the many token states in Fig 5D–5G explains their similar performance (c.f. Fig 5A). This similarity in policy is remarkable given that the model has essentially only a single, crucial degree of freedom (τ_context), a priori unrelated to how decision times depend on token state. Note that in both the fitted PGD model and the primate behaviour, residual ambiguity (N_t ≈ 0) is resolved at intermediate trial times (Fig 5B–5E).

The AR-RL strategies are plotted across c in Fig 5G and 5H. In contrast, they give no intermediate decision times at ambiguous (N_t ≈ 0) states, invariably waiting until the ambiguity resolves. This in fact holds over the entire (α, c)-plane (see S9 Fig for the complete dependence), and also under the addition of a movement cost, i.e. a constant cost incurred by either of the reporting actions. Thus, whereas AR-RL policies shift around the edges of the relevant decision space as α or c is varied, the PGD policy lies squarely in the bulk, tightly overlaying the policy extracted from the data. We conclude that the context-conditioned strategies of the non-human primates in this task are well-captured by PGD, while having little resemblance to the behaviour that would maximize reward rate with or without a fixed deliberation cost rate. We address the additional freedom of a time-varying cost rate in the discussion.

Neural urgency and context-dependent opportunity cost.

So far, we have fit and analyzed the PGD model with respect to recorded behaviour. Here, we take a step in the important direction of confronting the above theory of behaviour with the neural dynamics that we propose drive it. The proposal for the tokens task mentioned at the end of the introduction has evidence strength and urgency combining in PMd, whose neural dynamics implements the decision process. In Fig 6A, we restate in a schematic diagram an implementation of this dynamics that includes a collapsing decision boundary. In the one-dimensional belief space for the choice (Fig 6A(top)) [8, 35], the rising belief collides with the collapsing boundary to determine the decision time. In the equivalent commitment and deliberation cost formulation developed here (Fig 6A(middle)), the falling commitment cost collides with the rising deliberation cost. The collapsing boundary in belief space can be parametrized as C − u_t, where C is the initial strength of belief, e.g. some desired confidence, that is lowered by a growing function of trial time u_t > 0. The decision criterion is then b_t > C − u_t, where b_t is the belief, i.e. the probability of a correct report. For AR-RL optimal policies, u_t emerges from value maximization and thus has a complicated dependence on the opportunity cost sequence, . For PGD, in contrast, C is interpreted as the maximum reward r_max and u_t is identically . For a linear neural encoding model in which belief, rather than evidence, is encoded in neural activity, the sum of the encoded belief and the encoded collapsing boundary, , evolve on a one-dimensional choice manifold. According to the proposal, when this sum becomes sufficiently large (e.g. for some threshold ), PMd begins to drive the activity in downstream motor areas towards the associated response.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparing neural urgency and collapsing decision boundaries.

(A) Top: Rising belief (blue) meets collapsing decision boundary (black dashed) in belief space. Middle: Falling commitment cost (blue) meets rising deliberation cost (black-dashed) in cost space. Bottom: Belief/commitment cost is encoded (blue) into a low-dimensional neural manifold, with the addition of an urgency signal (orange) (c.f. Figure 8 in [8]). The decision (red circle) is taken when the sum passes a fixed threshold (black-dashed). (B) Deliberation cost maps onto the urgency signal extracted from zero-evidence conditioned cell-averaged firing rate in PMd (200ms time steps).

https://doi.org/10.1371/journal.pcbi.1010080.g006

Neural urgency was computed from the PMd recordings of [25] in [36]. This computation relies on the assumption that while a single neuron’s contribution to will depend on its selectivity for choice (left or right report), the urgency is a signal arising from a population-level drive to all PMd neurons, irrespective of their selectivity. Thus, can be extracted from neural recordings by conditioning on zero-evidence states () and averaging over cells. In [36], error bars were computed at odd times via bootstrapping; data at even times was obtained by interpolating between N_t = ±1; and data was pooled from both subjects. We have excluded times at which firing rate error bars exceed the range containing predictions from both blocks. To assess the correspondence of the components of the deliberation cost developed here and neural urgency, in Fig 6B we replot their result (c.f. Figure 8B of [36]). We overlay the mean (+/- standard deviation) of the opportunity cost sequence, (shaded area in Fig 4; averaged over all trials produced by applying the two fitted PGD models on the data sequence and conditioning the resulting average within-trial deliberation cost on context). To facilitate our qualitative comparison, we convert cost to spikes/step simply by adjusting the y-axis of the deliberation cost. The observed urgency signals then lie within the uncertainty of the context-conditioned deliberation cost signals computed from the fitted PGD models. There are multiple features of the qualitative correspondence exhibited in Fig 6B: (1) the linear rise in time; (2) the same slope across both fast and slow conditions; and (3) the baseline offset between conditions, where the fast condition is offset to higher values than the slow condition. Such features would remain descriptive in the absence of a theory. With the theory we have presented here, however, each has their respective explanations via the interpretation of urgency as the opportunity cost of deliberation: (1) the subject uses a constant cost per token jump, (2) this cost rate refers to moment-to-moment decisions, irrespective of context, that is reflective of the use of the context-agnostic stationary reward, and (3) trial-aware planning over contexts leads to an opportunity cost baseline offset with a sign given by the reward rate deviation ρ_α − ρ with respect to the stationary average, ρ.

Up to now, the computational and neural basis for urgency has remained largely unexplored in normative approaches, which also typically say little about adaptation effects (see [37] for a notable exception). In summary, we exploited the adaptation across context switches to learn the model and explained earlier responses in high reward rate contexts as the result of a higher opportunity cost of deliberation. While this qualitative effect is expected, we go beyond existing work by quantitatively predicting the average dependence on both time and state (Fig 5B–5E) as well as the qualitative form of urgency signal (Fig 6B). Taken together, the data is thus consistent with our interpretation that neural activity underlying context-conditioned decisions is gated by opportunity costs reflective of a trial-aware timescale hierarchy computed using performance estimation on multiple timescales.

Scientific and clinical implications

In our proposal, we have linked two important and related, but often disconnected fields: the systems neuroscience of the neural dynamics of decision-making and the cognitive neuroscience of opportunity cost and reward sensitivity. The view that tonic dopamine encodes average reward is two decades old [5]. However, the existence of a reward representation decomposed by timescale has received increasing empirical support only in recent years, from cognitive results [41–43] to a recent unified view of how dopamine encodes reward prediction errors using multiple discount factors [44, 45] and of dopamine as encoding both value and uncertainty [46]. Dopamine’s effect on time perception has been proposed [47] and has empirical support [48], but the mechanism by which its putative effect on decision speed is implicated in the neural dynamics of the decision-making areas driving motor responses was unknown. Our theory fills this explanatory gap by considering dynamic evidence tasks and parametrizing urgency using a multiple-timescale representation of performance. One candidate for the latter’s neural implementation is in the complex spatio-temporal filtering of dopamine via release-driven tissue diffusion and integration via DR1 and DR2 binding kinetics [49]. Subsequent neural filtering and computation by striatal network activity could also play a role [50]. The study of spatiotemporal filtering of dopamine is increasingly accessible experimentally [51, 52] and provides an exciting direction for multiscale analysis of behaviour. Our proposal that urgency is the means by which the neural representation of reward ultimately affects neural dynamics in decision-making areas frames a timely research question on which these experimental methods could shed light.

We applied PGD to decisions playing out in PMd, a decision-making area relevant to arm movements. PGD appears to be relevant to other kinds of decisions, however. For instance, a large body of work has studied decisions through recordings in lateral intraparietal cortex in random dot motion tasks whose environment is formally similar to that of the tokens task. One seminal study identified an urgency signal with the same properties as those exhibited by the tokens task: a linear rise at early trial times that is independent of trial evidence and an offset with sign given by the reward rate deviation of the current context, here two and four-choice trials [23]. While decision boundaries obtained using AR-RL are evidence-independent, these models require tailored cost functions that are fit to those experiments in a procedure that assumes optimality a piori [8]. Here, we offer an alternative explanation that behaviour is in fact suboptimal, with the decision boundary determsined directly by the estimated opportunity cost only. PGD decision boundaries are thus independent of evidence by construction. In contrast to the tokens task, however, context in these random dot task experiments was sampled randomly and thus its dynamics lacked temporal correlation [23]. In this case, a natural hypothesis from our approach is that a pair of performance filters, one for each context, tracks the reward history in two parallel streams. In this case, our theory would predict that the ratio of slopes of urgency across the two contexts reflects the ratio of context-conditioned reward rates. An estimation procedure described in the Methods for this data [23] agrees to within 20% error, providing support for the hypothesis that PGD underlies non-human primate behaviour on this widely-studied task. Within the context of the drift-diffusion models typically used to understand neural activity for that task, PGD provides a principled mechanism that implements collapsing decision boundary. PGD is thus easily incorporated into such models and testing the generality of our theory using tailored experiments in this setting is an important next step. We expect that the ideas behind PGD can also provide an explanation to the context dependence observed in response times in non-episodic tasks with varying context (e.g. in bandit tasks [53]) and foraging tasks.

Urgency may play a role in both decision and action processes, potentially providing a transdiagnostic indicator of a wide range of cognitive and motor impairments in Parkinson’s disease and depression [54]. When fit to recorded behaviour from each subject across a set of subjects, our model can be used in clinical contexts to dissect inter-individual differences via the differences in the fitted model parameters. We gave an example here for the two non-human primates, explaining one as more hasty than the other due to higher reward bias and shorter memory. Our theory offers a means to ground these diverse results in neural dynamics by formulating opportunity cost estimation as the underlying causal factor linking vigor impairments (e.g. in Parkinson’s disease) and dysregulated dopamine signalling in the reward system [54–56]. We provide a concrete proposal for a signal filtering system that extracts a context-sensitive opportunity cost from a reward prediction error sequence putatively encoded by dopamine. Neural recordings of basal ganglia provide a means to identify the neural substrate for this system.

Commitment cost estimation

Beyond the estimation of the opportunity cost of deliberation, we assumed that the agent had a precise estimate of the expected reward, which it used to compute the within-trial commitment cost. For the tokens task, a recorded signal in dorsal lateral prefrontal cortex of non-human primates correlates strongly with belief [26], equivalent to the expected reward for binary rewards). How this quantity is computed by neural systems is not currently known. However, for a general class of tasks, a generic, neurally plausible means to learn the expected reward is via distributional value codes [46]. For example, the Laplace code is a distributional value representation that uses an ensemble of units over a range of temporal discount factors and reward sensitivities [57]. The authors show that expected reward is linearly decodeable from this representation.

Experimental predictions

A feature of our decision-making theory is that it is highly vulnerable to falsification. First, with regards to behaviour via the shape of the action policy using our survival probability representation (c.f. Fig 5B–5E, 5G and 5H), PGD varies markedly with reward structure and thus provides a wealth of predictions for how observed behaviour should be altered by it. For example, a salient feature of the standard tokens task is its reflection symmetry in the tokens difference, N_t. We can break this symmetry for which the theory predicts a distinctly asymmetric shape (S10 Fig; for details see Methods). Our theory is also prescriptive for neural activity via the temporal profile of neural urgency. The slope of remained fixed across blocks for relatively short block lengths used in the data analyzed here. In the opposite limit, T_block/τ_long ≫ 1, approaches ρ_α except when undergoing large, transient excursions after context switches. Thus, the deliberation cost is given by the first component in Eq 5 most of the time, with the context specific reward rate as the slope. One simple prediction is that the slope of urgency should exhibit increasing variation as the duration of the blocks increases.

Reinforcement learning theory

We suggest how to generalize average-adjusted value functions to context-varying opportunity cost of time in a way that reduces to AR-RL when context is fixed or not tracked. This adds a continuing task perspective to episodic AR-RL, in line with recent work in machine learning, which is arguably the more appropriate reinforcement learning setting for many decision-making experiments in neuroscience. The epistemic perspective entailed in the estimation of these costs parallels a recent epistemic interpretation of the discount-reward formulation as encoding knowledge about the volatility of the environment [58].

Our work also suggests a new class of reinforcement learning algorithms between model-based and model-free: only parts of the algorithm need adjustment upon task structure variation. This is reminiscent of how the effects of complex state dynamics are decoupled from reward when using a successor representation [59], but tailored for the average-reward rather than the discount-reward formulation. We have left analysis of the algorithmic complexity of PGD to future work, but expect performance improvements, as with successor representations, in settings where decoupling the learning of environment statistics from the learning of reward structure is beneficial.

Comparison with humans

In the space of strategies, PGD lies in a regime between fully exploiting assumed task knowledge (average-case optimal) and assumption-free adaptation (worst-case optimal). Highly incentivized human behaviour is likely to be more structured than PGD because of access to more sophisticated learning. While some humans land on the optimal one-and-done policy in the fast condition when playing the tokens task (Personal communication, Thomas Thierry), most do not. The human brain likely has all the components needed to implement PGD. Nevertheless, the situations in which we actually exploit PGD, if any, are as yet unclear. In particular, how PGD and AR-RL relate to existing behavioural models tailored to explain relative-value, context-dependent decision-making in humans [6], such as scale and shift adaptation [60], is an open question. Whether or not PGD is built into our decision-making, the question remains if PGD is optimal with respect to some bounded rational objective. In spite of the many issues with the latter approach [61], using it to further understand the computational advantages of PGD is an interesting direction for future work.

Despite our putative access to sophisticated computation, humans still exhibit measurable bias in how we incorporate past experience [62]. One simple example is the win-stay/lose-shift strategy, a more rudimentary kind of performance-gated decision-making than PGD, which explains how humans approach the rock-paper-scissors game [63]. In that work, numerical experiments demonstrated that this strategy outperforms at a population level the optimal Nash equilibrium for this game, demonstrating that the use of such seemingly sub-optimal strategies can confer a surprising evolutionary advantage. This example supports the claim that relatively simple and nimble strategies such as PGD make for attractive candidates when acknowledging that a combination of knowledge and resource limitations over task, development, and evolutionary timescales have shaped decision-making in non-stationary environments.

Patch leaving task

We devised a mathematically tractable patch leaving task for which PGD learning is optimal with respect to the average-adjusted value function. Here the value is simply the return from the patch. This value function is related, but not equivalent to the marginal value of optimal foraging, for which the decision rule is [2]). This choice of task allowed us to compare PGD’s convergence properties relative to conventional AR-RL algorithms that make use of value functions. In contrast to PGD, the latter requires exploration. For a comparison generous to the AR-RL algorithm, we allowed it to circumvent exploration by estimating the value function from off-policy decisions obtained from the PGD algorithm using the same learning rate. We then compared them to PGD using their on-policy, patched-averaged reward. This made for a comparison based solely between the parameters of the respective models. If we did not allow for this, the AR-RL algorithms would have to find good learning signals by exploring. In any form, this exploration would lead them converge substantially slower. This setting thus provides a lower bound on the convergence times of the AR-RL algorithm.

In this task, the subject randomly samples (with replacement) d patches, each of a distinct, fixed, and renewable richness defined by the maximum return conferred. These maximum returns are sampled before the task from a richness distribution, p(r_max), with r_max > 0 and are fixed throughout the experiment. The trials of the task are temporally extended periods during which the subject consumes the current patch. After a time t in a patch, the return is defined r(t) = r_max(1 − (λt)⁻¹). This patch return profile, 1 − (λt)⁻¹, is shared across all patches and saturates in time with rate λ, a parameter of the environment that sets the reference timescale. The return diverges negatively for vanishing patch leaving times for mathematical convenience, but also evokes situations where leaving a patch soon after arriving is prohibitively costly (e.g. when transit times are long). A stationary policy is then a leaving time, t_s, for each of d patches, where the s-subscript indexes the patch. Given any policy, the stationary reward rate for uniformly random sampling of patches is then defined as (6) We designed this task to (1) emphasize the speed-return trade-off typical in many deliberation tasks, and (2) have a tractable solution with which to compare convergence properties of PGD and AR-RL value function learning algorithms.

A natural optimal policy is the one that maximizes the average-adjusted trial return, Q(r, t) = r−ρt. Given the return profile we have chosen, the corresponding optimal decision time, , in the sth patch obtained by maximizing r − ρt is , which scales inversely with the reward rate so that decision times are earlier for larger reward rates, because consumption (or more generally deliberation) at larger reward rates costs more. We chose this return profile such that stationary PGD learning gives exactly the same decision times: the condition for patch s here takes the form ρt_s = r_max,s/(λt_s). Thus, they share the same optimal reward rate, ρ*. Using for each patch in Eq 6 gives a self-consistency equation for ρ with solution , where (we have assumed d is large here to remove dependence on s). Described so far in continuous time, the value function was implemented in discrete time such that the action space is a finite set of decision times selected using the greedy policy, , where is the estimated trial return. As a result, there is a finite lower bound on the performance gap, i.e. the relative error, ϵ = (ρ* − ρ)/ρ* > 0 for the AR-RL algorithm. Approaching this bound, convergence time for both PGD and AR-RL learning is limited by the integration time τ of the estimate (c.f. Eq 8) of ρ. We note that PGD learns faster in all parameter combinations tested. To demonstrate the insensitivity of PGD to the state space representation, at 5 × 10⁵ time steps into the experiment we shuffled the labels of the states. PGD is unaffected, while the value function-based AR-RL algorithm is forced to relearn and in fact does so slower than in the initial learning phase, due to the much larger distance between two random samples, than between the initial values (chosen near the mean) and the target sample.

Filtering performance history

For unit steps of discrete time, the step-wise update of the performance estimate, , is (7) with β = 1/(1 + τ) called the learning rate, and τ the characteristic width of the exponential window of the corresponding continuous time filter over which the history is averaged. We add τ as a superscript when denoting the estimate to indicate this. Exceptionally, here t indexes absolute time rather than trial time. Note that a continuous-time formulation of the update is possible via an event-based map given the decision times in which the reward event sequence is given as a sum of delta functions. In either case, to leading order in β, , i.e. the filter sums past rewards. Thus, when , and so when t is large.

The rewards in this task are sparse: R_t = 0 except when a trial ends and the binary trial reward R_k (1 or 0) is received. A cumulative update of Eq 7 that smooths the reward uniformly over the trial duration and is applied once at the end of each trial is thus more compuatationally efficient. Resolving a geometric series leads to the cumulative update [9, 33] (8) where the smoothed reward, , can be interpreted as a trial-specific reward rate. The initial estimate, , is set to 0. Exceptionally, , after which Eq 8 is used. Using the first finite sample as the first finite estimate is both more natural and robust than having to adapt from zero. We will reuse this filter for different τ and denote the filtered estimate from its application with a τ-superscript, . For example, the precision of as an estimate of a stationary reward rate ρ is set by how many samples it averages over, which is determined by the effective length of its memory given by τ_long. Since we assume the subject has learned the expected reward, , we use it instead of R_k when computing .

Tokens task: A random walk formulation

The tokens task is a continuing task of episodes (here trials), which can be formulated using the token difference, N_t. Each trial effectively presents to the agent a realization of a finite-length, unbiased random walk, with N_t = {−t, …, t} and N₀ = 0. We express time in units of these steps. The agent observes the walk and reports its prediction of the sign of the final state, (t_max is odd to exclude the case it has no sign). The time at which the agent reports is called the decision time, t^dec ∈ {0, 1, …, t_max}. For a greedy policy, sign(N_t) can be used as the prediction (and the reporting action selected randomly if N_t^dec = 0). The decision-making task then only involves choosing when to decide. In this case, the subject receives reward at the end of the random walk, i.e. a unit reward for a correct prediction, otherwise nothing (Θ is the Heaviside function: Θ(x) = 1 if x > 0, zero otherwise).

An explicit action space beyond decision time is not necessary for the case of greedy actions. It can nevertheless be specified for illustration in an Markov decision process (MDP) formulation: the agent waits (a_t = 0 for t < t^dec) until it reports its prediction, a_t^dec = ±, after which actions are disabled and the prediction is stored in an auxiliary state variable used to determine the reward at the end of the trial. A MDP formulation for a general class of perceptual decision-making tasks, including the tokens and random dots task, is given in Methods).

Perfect accuracy in this task is possible if the agent reports at t_max since . The task was designed to study reward rate maximizing policies. In particular, the task has additional structure that allows for controlling what this optimal policy is through the incentive to decide early, α, incorporated into the trial duration for deciding at time t in the trial, (9) Here, a dead time between episodes is added via the inter-trial interval, T_ITI, to make suboptimal the strategy of predicting randomly at the trial’s beginning. It was set to 7.5 steps, the value used in [25]. In practise, this time also includes a random wait time (uniformly distributed between 2 and 3 steps) at the beginning of the trial to allow the subject to set-up. We excluded this time from our analysis. Only trials in which the subject completed the trial were analyzed. We emphasize that it is through the trial duration that α serves as a task parameter controlling the strength of the incentive to decide early. When α is fixed, we denote the corresponding optimal stationary reward rate, ρ_α, obtained from the reward rate maximizing policy. This policy shifts from deciding late to deciding early as α is varied from 0 to 1 (c.f. S9(F) and S9(G) Fig).

We consider a version of the task where α is variable across two episode types, a slow (α = 1/4) and fast (α = 3/4) type. The agent is aware that the across-trial α dynamics are responsive (maybe even adversarial), whereas the within-trial random walk dynamics (controlled by the positive jump probability, here p = 1/2) can be assumed fixed (see the next section for how p factors into the expression for the expected reward, .

Expected trial reward for the tokens task

We derived and used an exact expression for the expected reward in a trial of the tokens task. We derive that expression here as well as a simple approximation. The state sequence is formulated as a t_max-length sequence of random binary variables, , S_t = ±1, i = 1, 2, …, t_max. Consider a simple case in which each is an independent and identically distributed Bernoulli sample, , for jump probability p ≥ 1/2. The distribution of is then (10) We will use this distribution to compute expectations of quantities over this space of trajectories, namely the sign of , for some 0 ≤ t ≤ t_max and in particular the sign of the final state, given N_t = n. Note that N_t is even if t is even and same with odd values. We remove the case of no sign in by choosing t_max to be odd.

First, consider predicting sgn(N_t) with no prior information. The token difference, −t ≤ N_t ≤ t, appears directly in . Marginalizing (here just integrating out) the additional degrees of freedom leads to a binomial distribution in the number of S_i for i ≤ t for which S_i = +1, , (11) with n ∈ {0, …, t} and . Thus, the probability that N_t > 0, i.e. , is (12)

Now consider predicting , given the observation N_t = n. Define t′ = t_max − t as the remaining time steps to the predicted time and , i.e. the total count in the remaining part of the realization. Then the probability of ξ = + conditioned on the state N_t = n, denoted p_n,t, is defined in the same way as P(N_t > 0), (13) where is the number of positive jumps in the remaining t′ = t_max − t steps and we have used . The Θ(n′ − (t′ − n)/2) factor effectively changes the lower bound of the sum to max{0, ⌈(t′ − n)/2⌉}, where ⌈⋅⌉ rounds up. If ⌈(t′ − n)/2⌉ ≤ 0 then since the sum is over the domain of the distribution, which is normalized. Otherwise, the lower bound is ⌈(t′ − n)/2⌉, and the probability of ξ = +1 is (14) For odd t_max, the probability that ξ = − is denoted . For the symmetric case, p = 1/2, (15) when ⌈(t′ − n)/2⌉ > 0 and 1 otherwise. This expression is equivalent to equation 5 in [22], which was instead expressed using .

The space of trajectories, i.e. of , maps to a space of trajectories for defined on an evolving lattice in belief space. The expected reward in this case is, (16) (17) (18) where the belief of correct report . The commitment cost , then also evolves on a lattice (see Fig 3B). More generally, for Δr the difference of correct r_correct (here 1) and incorrect r_incorrect (here 0) rewards. Since r_max = r_correct, we have . For p = 1/2 and Δr = 1, .

The shape of is roughly sigmoidal, admitting the approximation, (19) where fitting constants a and b depend on t_max. For t_max = 15, a = 0.03725 and b = 0.3557. We demonstrate the quality of this approximation in S5 Fig. Approximation error is worse at t near t_max. More than 95% of decisions times in the data we analyzed occur before 12 time steps, where the approximation error in probability is less than 0.05. A similar approximation without time dependence was presented in [22]. We nevertheless used the exact expression Eq 15 in all calculations.

PGD implementation and fitting to relaxation after context switches

We identified the times of the context switches in the data and their type (slow-to-fast and fast-to-slow). Taking a fixed number of trials before and after each event, we averaged the decision times over the events to create two sequences of average decision times around context switches (the result is shown in Fig 4A and 4B). We used a uniformly weighted squared-error objective, minimized with the standard (Nelder-Mead) simplex routine in python’s scientific computing library’s optimization package.

Survival probabilities over the action policy

Behavioural analyses typically focus on response time distributions. From the perspective of reinforcement learning, this is insufficient to fully characterize the behaviour of an agent. Instead, the full behaviour is given by the action policy. In this setting, a natural representation of the policy is the probability to report as a function of both the decision time and the environmental state (see Fig 5). These are computed from the histograms of (N_t^dec, t^dec), over trials. However, the histograms themselves do not reflect the preference of the agent to decide at a particular state and time because they are biased by the different frequencies with which the set of trajectories visit each state and time combination. While there are obviously the same number of trajectories at early and late times, they distribute over many more states at later times and so each state at later times is visited less on average than states at earlier times. We can remove this bias by transforming the data ensemble to the ensemble of two random variables: the state conditioned on time (N_t|t), and the event that t = t^dec. Conditioning this ensemble on the state gives P(t = t^dec|N_t, t) = p(N_t, t = t^dec|t)/p(N_t|t). To reduce estimator variance, we focus on the corresponding survival function, P(t < t^dec|N_t, t). So, P(t < t^dec|N_t, t) = 1 when t = 0 and decays to 0 as t and |N_t| increase. Unlike the unconditioned histograms, these survival probabilities vary much more smoothly over state and time. This justifies the use of the interpolated representations displayed in Fig 5B–5E. Note that to simplify the analysis, we have binned decision times by the 200 ms time step between token jumps. This is justified by the small deviations from uniformity of decision times modulo the time step shown in S11 Fig.

Episodic decision-making and dynamic programming solutions of value iteration

We generalize the mathematical notation and description of an existing AR-RL formulation and dynamic programming solution of the random dots task [8], a binary perceptual evidence accumulation task extensively studied in neuroscience. To align notation with convention in reinforcement learning theory, exceptionally here s denotes the belief state variable, ie. a representation of the task state sufficient to make the decision (e.g. the tokens difference, N_t, in the case of the tokens task). We connect this extended formulation to account for a dynamic deliberation cost. We write it in discrete time, though the continuous time version is equally tractable.

The problem is defined by a recursive optimality equation for the value function V(s|t) in which the highest of the action values, Q(s, a|t), is selected. We formalize the non-stationarity within episodes by conditioning on the trial time, t, where t = 0 is the trial start time. Q(s, a|t) is the action-value function of average-reward reinforcement learning [15], i.e. the expected sum of future reward deviations from the average when selecting action a when in state s, at possible decision time t within a trial, and then following a given action policy π thereafter. The action set for these binary decision tasks consists of report left (−), report right (+), and wait. When wait is selected, time increments and beliefs are updated with new evidence. We use a decision-time conditioned, expected trial reward function, r(s, a|t) with a = ±, that denotes the reward expected to be received at the end of the trial after having reported ± in state s at time t during the trial. Note that r(s, a|t) can be defined in terms of a conventional reward function r(s, a) if the reported action, decision time, and current time are stored as an auxiliary state variable so they can be used to determine the non-zero reward entries at the end of the trial.

The average-reward formulation of Q(s, a|t) naturally narrows the problem onto determining decisions within only a single episode of the task. To see this, we pull out the contribution of the current trial, (20) where T is the (possibly stochastic) trial end time and V(s|T + 1) is the state value at the start of the following trial, which does not depend on s_t and a_t for independently sampled trials. Following conventional reinforcement learning notation, the expectation is over all randomness conditioned on following the policy, π, which itself could be stochastic [15]. When trials are identically and independently sampled, the state at the trial start is the same for all trials and denoted s₀ with value V₀. Thus, the value at the start of the trial V(s|t = 0) = V(s|T + 1) = V₀ equals that at the start of the next trial and so, by construction, the expected trial return (total trial rewards minus trial costs) must vanish (we will show this explicitly below). Note that the value shift invariance of E1 20 can be fixed so that V₀ = 0.

The optimality equation for V(s|t) arises from a greedy action policy over Q(s, a|t): it selects the action of the largest of Q(s, −|t), Q(s, +|t), and Q(s, wait|t). The value expression for the wait-action is incremental, and so depends on the value at the next time step. In contrast, expression for the two reporting actions integrates over the remainder of the trial since no further decision is made and so depends on the value at the start of the following trial. The resulting optimality equation for the value function V(s|t) is then (21) Here, t = 0, 1, …, t_max within the current trial and t = T + 1, T + 2… in the following trial, with t_max the latest possible decision time in a trial, and T = T(t) the decision-time dependent trial duration. For inter-trial interval T_ITI, T satisfies T_ITI ≤ T ≤ t_max + T_ITI. c_t is the cost rate at time t. The second term in Q(s, wait|t) uses the notation , i.e. the expectation of z with respect to p(x|y). The last line in Eq 21 is the self-consistency criterion imposed by the AR-RL formulation, which demands that the expected value at the beginning of the trial be the expected value at the beginning of the following trial. The greedy policy then gives a single decision time for each state trajectory as the first time when Q(s, −|t) > Q(s, wait|t) or Q(s, +|t) > Q(s, wait|t), with the reporting action determined by which of Q(s, −|t) and Q(s, +|t) is larger. For given c_t, dynamic programming provides a solution to Eq 21 [8] by recursively solving for V(s|t) by back-iterating in time from the end of the trial. For most relevant tasks, to never report is always sub-optimal, so the value at t = t_max is set by the best of the two reporting (±) actions, which do not have a recursive dependence on the value and so can seed the recursion.

We now interpret this general formulation in terms of opportunity costs. For the choice of a static opportunity cost rate of time, c_t = ρ. This is the AR-RL case. As in [8], a constant auxiliary deliberation cost rate, c, incurred only up to decision time can be added, c_t = ρ + cΘ(t^dec − t). Of course, ρ is unknown a priori. For this solution method, its value can be found by exploiting the self-consistency constraint, V(s|t = 0) = V(s|t = T + 1). This dependence can be seen formally by taking the action value Eq 20, choosing a according to π to obtain the state value, V(s|t), and evaluating it for t = 0, (22) (23) (24) Here, and denotes the expectations over the trial ensemble that, when given the state sequence, transforms to an average over t^dec, the trial decision time, defined as when V(s|t) achieves its maximum on the state sequence, . The expected trial reward function, r(t) ≔ max_a∈{−,+} r(s, a|t) is the expected trial reward for deciding at t. Imposing the self-consistency constraint on Eq 24 recovers the definition .

Asymmetric switching cost model

Here, we present the model component that accounts for the asymmetric relaxation timescales after context switches. The basic assumption is that tracking a signal at a higher temporal resolution should be more cognitively costly. A cost-aware strategy addressing this fact is to adapt the integration time to smaller values when resolution mismatch costs are higher so the system lowers this higher cost quickly. For the tokens task, adapting from faster to slower environments should then happen more quickly than the reverse. We now develop this idea formally (see S4 Fig).

Let T_track and T_sys be the timescale of tracking and of the tracked system, respectively. One way to interpret the mismatch ratio, T_sys/T_track, is via an attentional cost rate, q. This rate should decay with T_track: the slower the timescale of tracking, the lower the cognitive cost. For simplicity, we set q = 1/T_track (S4(A) Fig). Integrating this cost rate over a characteristic time of the system is then the tracking cost, Q = qT_sys = T_sys/T_track, which is also the mismatch ratio. We propose that Q enters the algorithm via a scale factor on the integration time of the reward filter for , τ_context. We redefine τ_context as (25) where ν is a sensitivity parameter that captures the strength of the nonlinear sensitivity of the speed up (for ν > 1) or slow down (for ν < 1) in adaptation with the tracking cost, Q (S4(A) Fig shows how this timescale varies over Q for three values of ν). We note that ν captures the meta-cognitive belief that the subject has about how cognitively effortful is its tracking behaviour. A natural choice for T_sys is T_k, the trial duration. For T_track, we introduce the filtered estimate of the trial duration, (computed using the same simple low-pass filter c.f. Eq 8). Thus, the tracking timescale adapts to the system timescale. As a result of how τ_context is lowered by Q for ν > 1, this adaptation is faster in the fast-to-slow transition relative to the slow-to-fast transition.

Prediction for asymmetric rewards

Given a payoff matrix, R = (r_s,a), where r_s,a is the reward for reporting a ∈ {−, + } in the trial realization leading to s, here the sign of , and the probability that the rightward choice is correct, , the expected reward for the two reporting actions in a trial is given by the matrix equation Here, the corresponding reported choice is . In this paper and in all existing tokens tasks, R was the identity matrix. In this case, and for all cases where R is a symmetric matrix, R = R^⊤, an equivalent decision rule is to decide based on the sign of N_t. When R is not symmetric, however, this is no longer a valid substitute. Asymmetry can be introduced through the actions and the states.

Using an additional parameter γ, we introduce asymmetry via a bias for + actions that leaves the total reward unchanged by replacing the payoff matrix with The result for γ = −0.6, 0, and 0.6 is shown in S10 Fig. For γ > 0 the decision boundary for a = + shifts up proportional to γ. For γ < 0 the decision boundary for a = − shifts down proportional to −γ. The explanation is that the components are set and exchange where the decision is exchanged, N_t = 0 for the symmetric case. This changes to N_t ∝ ±γ for the asymmetric γ ≠ 0 case.

Comparing reward rates and slopes of urgency

Reference [23] parametrize urgency with the saturation value, u_∞, and the half-maximum, τ_1/2. The initial slope is given by their ratio. We used the context-conditioned values published in Table 1 in [23] for the n = 70 (no 90° control) dataset. The context-conditioned reward rates, ρ_α, are computed as the accuracy 〈R〉_|α divided by the average trial time, 〈T〉_|α for choice number α ∈ {2, 4} as context. We computed 〈R〉_|α=2 = 0.71 and 〈R〉_|α=4 = 0.49. The trial time is the sum of the response time, the added time penalty if incorrect, and the inter-trial interval. We computed the response times t_{response,α=2} = 0.527 and t_{response,α=4} = 0.725. While the dataset contains the response times, it does not have the latter two. The time penalty was on the order of 1 second, as was the time penalty (A. Churchland. Personal communication). Under those estimates, the reward rates are ρ_α=2 = 0.40 and ρ_α=4 = 0.22. The ratio between slopes is 1.8 and the ratio of reward rates was 2.3 giving an error of about 20%.