2.1. Stochastic games

Stochastic games, also known as Markov games, are defined by the elements . agents reside in an environment of states . In each state s, each agent has available actions to choose from. is the joint-action set where denotes the Cartesian product over the sets indexed by i. Time advances in discrete steps, and agents choose their actions simultaneously. A joint action is denoted by . With ..., we denote the joint action except agent i’s, and we write the joint action in which agent i chooses and all other agents choose as . We choose an equal number of actions for all states and agents out of notational convenience.

The transition tensor determines the probabilistic state change. is the transition probability from current state s to next state under joint action .

The reward tensor maps the triple of current state s, joint action and next state to an immediate reward scalar for each agent. is the reward agent i receives.

At each time step, actions are chosen from the joint strategy tensor . In state s, agent i chooses action with probability .

Each agent i aims for a strategy to maximize their gain,

(1)

where denotes the reward agent i obtained at time step t + k. The discount factor, , denotes how much agent i cares for future rewards.

2.2. Reinforcement learning

Reinforcement learning is a trial-and-error method of mapping situations to actions to maximize a numerical reward signal [9]. When rewards are a delayed consequence of current actions, so-called temporal-difference or reward-prediction learning has been particularly influential. This type of learning summarizes the difference between value estimates from past and present experiences into a reward-prediction error, which is then used to adapt the current behavior to gain more rewards over time. There also exist remarkable similarities between computational reinforcement learning and the results of neuroscientific experiments [13]. Dopamine conveys reward-prediction errors to brain structures where learning and decision-making occur [12].

At time step t, in state s, agent i evaluates action to be of quality . Learning then means updating these state-action value estimates, , after selecting action and having observed state according to

(2)

where denotes agent i’s learning rate, which regulates how much new information the agent uses for the update, and the current temporal-difference or reward-prediction error. The expression of the temporal-difference error depends on the chosen RL algorithm.

In this work, we focus on two different RL variants: SARSA learning, and Actor-Critic learning. SARSA learning takes into account the five elements of the current State, current Action, Reward, next State, and next Action for one learning update step. In Actor-Critic learning, an approximation of the State-Action value gets “criticized” by (i.e., compared against) an approximation of the state value. Another well-known variant is Q-learning. A comparison of these three variants can be found in [33].

In SARSA learning, the temporal-difference error is expressed as:

(3)

In Actor-Critic learning, by contrast, the temporal-difference error retains the same one-step bootstrap target as in SARSA, but replaces the baseline with the state value ,

(4)

The state value at time t, , quantifies the overall quality of state s at time t, regardless of the chosen action . In Actor-Critic learning, it is separately updated according to:

(5)

Agents select actions based on the current state-action value beliefs , balancing exploitation (i.e., choosing the action of maximum quality) and exploration (i.e., selecting lower-quality actions to learn more about the environment). We employ the widely used Boltzmann policy function. The probability of choosing action under observing state s is

(6)

where the intensity-of-choice parameters, , regulate the exploration-exploitation trade-off. For high , agents exploit their learned knowledge about the environment, leaning toward actions with high estimated state-action values. For low , agents are more likely to deviate from these high-value actions to explore the environment further with the chance of finding actions that eventually lead to even higher values. Our choice of the Boltzmann policy is motivated by the maximum-entropy principle [40]. It is the distribution over actions that maximizes entropy subject to a constraint on the expected Q-value or equivalently, the distribution that maximizes the expected Q-value subject to a constraint on entropy, with the Lagrange multiplier coupling the two objectives [41]. It also resembles the idea of reward-matching behavior in animal and human choice [42,43]. From a modeling standpoint, the Boltzmann policy is particularly convenient for the deterministic dynamics framework of Section 2.4: because the strategy update depends only on differences in Q-values across actions, action-constant quantities cancel, substantially simplifying the dynamics. Alternative exploration policies such as -greedy can also be combined with strategy averaging [38], but at the cost of a larger dynamical state space; more elaborate schemes such as UCB, which track uncertainty explicitly, complicate the dynamics further still, to the point of being analytically intractable unless the exploration process itself is the primary object of study.

2.3. Distributional reinforcement learning

Distributional approaches to RL (DistRL) in artificial intelligence and machine learning research were pioneered by [44,45] and recently gained wider prominence [27]. DistRL diverges from the classical approach insofar as learning agents do not compute expected values, but rather the full distribution of outcomes. Therefore, the goal is not to compute an expected value or quality, but rather a return distribution, which is described by a distributional Bellman equation [27]:

(7)

where is the reward for action a in state s, is the next-state action, and is its random return. The factor reflects the normalized-gain convention used throughout this paper, .

Over the past years, DistRL researchers have used various supports for discretizing the distributions. In categorical DistRL, the discretization support is a set of N categorical “atoms,” which are fixed and evenly spaced (with ) [27]. Later versions of DistRL algorithms rely on the distributions’ statistics as discretization support: the atoms are not fixed anymore, but are updated so as to converge towards quantiles or expectiles of the target distribution [46].

In this work, we will use a categorical discretization support because it maps naturally onto a fixed population of neurons, each tuned to a specific return value. In standard categorical DistRL, atoms span the range of possible returns . Under the normalized gain convention (Eq. 1) used throughout this paper, returns are convex combinations of rewards and therefore lie in . Our atoms, defined at reward values , cover this range exactly. Quantile- and expectile-based representations use adaptive atoms that shift during learning [46,47], which improves approximation quality but would render the deterministic dynamics derivation of this paper incompatible, as the representational state itself would become time-varying.

2.4. Deterministic learning dynamics

Deterministic dynamics of RL remove noise from the learning process. Hence, the agents’ strategies over time can be computed and visualized as deterministic trajectories in phase space. To this end, we use temporal-difference RL in the deterministic limit (DetRL), which was developed by Barfuss et al.[33],. This method uses strategy-averages as a way to remove the noise from the learning dynamics: the prediction error is averaged over all strategies and state transitions. With a formalism analogous to evolutionary game theory [35,36], deterministic RL dynamics yield analytically tractable trajectories in strategy space and enable the exact characterization of fixed points, their stability, and bifurcations — properties we exploit to identify bistable choice regimes in Section 3.4. More precisely, SARSA dynamics with a Boltzmann strategy function are analogous to the replicator-mutator equation [33] — a mutation term prevents strategies from collapsing to a pure strategy — whereas Actor-Critic dynamics are analogous to the pure replicator equation [35], which drives strategies toward deterministic fixed points. Thus, deterministic RL dynamics can be viewed as distinct level of analysis of multi-agent learning [39]. Stochastic RL algorithms form their noisy counterparts: the stationary distribution of the stochastic system concentrates around the stable fixed points of the deterministic part and spends negligible time near unstable ones, provided the noise is sufficiently weak and the deterministic dynamics have isolated stable fixed points rather than more complex attractors [38,39]. Under these conditions, the deterministic approximation captures average learning behavior while abstracting away intrinsic fluctuations. A related but conceptually distinct line of work is the ODE method for stochastic approximation [48,49], which links stochastic learning algorithms to a limiting ordinary differential equation in the vanishing step-size regime. There, the ODE serves as an asymptotic tool to establish convergence of the stochastic recursion to its invariant sets. These are precisely the conditions under which the link between deterministic dynamics and stochastic algorithms holds. In contrast, however, we do not take a vanishing learning-rate limit, but derive the induced deterministic update dynamics directly at finite step sizes (Section 3.1), treating their discrete-time behavior, including fixed points, stability, and bifurcations, as objects of intrinsic interest rather than as auxiliary constructs for convergence analysis.

First, we compute the different quantities we need to calculate the average temporal-difference error. As explained in Section 2.1, denotes the probability that agent i chooses action in state s; denotes the joint strategy tensor.

We call the average reward given the joint policy that agent i obtains by taking action in state s. More precisely, the average is taken over state transitions from s to , and over other agents’ actions :

(8)

Similarly, we call the average next-state Q-value given the joint policy for agent i taking action in state s:

(9)

where denotes the average Q-value of action taken by agent i in state , described by the recursive Bellman equation:

(10)

Throughout this section, a subscript denotes the strategy-averaged quantity evaluated at the joint strategy at time t. In the deterministic approximation, these are the true on-strategy expectations under , not estimates, so satisfies the above Bellman recursion exactly. The time index t is therefore inherited entirely through , and substitutions that combine into are identities by construction. Unlike Eqs. 3 and 4, where t indexes the single state and action sampled at that step and only is updated, here range over every state-action pair. All get updated at each step t together as a consequence of the joint strategy update (Eq. 13). The joint strategy is the sole dynamic variable. Only it carries a time index t.

Therefore, for a SARSA agent, the average prediction error for agent i taking action in state s at time t is

(11)

where is the choice intensity, or inverse temperature, of agent i.

For an Actor-Critic agent, by contrast, the average Q-value is compared with the average state value (the “critic”). Since this term is constant in action, we can omit it from the formula [33]. Thus, for an Actor-Critic agent, the average prediction error for agent i taking action in state s at time t is:

(12)

The combination of the Q-value update (Eq. 2) and the Boltzmann formula (Eq. 6) yields the deterministic update of strategy , applied to every pair at once:

(13)

3.1. Theoretical framework

In this section, we derive deterministic distributional reinforcement learning dynamics. In order to model the optimism-pessimism heuristic, we apply deterministic learning dynamics to distributional RL. By doing so, we combine the best of both worlds: on the one hand, DetRL provides a transparent, reproducible, and computationally fast approach to RL; on the other hand, DistRL allows us to bias the return distribution of the learning dynamics and devise a simple model at the neuronal level.

Our approach is presented in Fig 1. The agent-environment interface is formalized as a Markov game comprised of N agents, a set of different states, a set of available actions, a transition tensor T, and a reward tensor . In DDRL, rewards and qualities are not computed as expected values, but as full distributions. These distributions are discretized according to a categorical support and a smoothing procedure. Then, the Q-return distribution is distorted according to a weighting algorithm, implementing the optimism heuristic.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The agent-environment interface.

N agents play in an environment formalized as a set of Z states . In each state, agent i can choose between M actions from the action set . The probability that a transition from state s to state occurs when agents play the joint action is . The right panel represents reward discretization and Q-return distribution estimation for one agent i. 1) An environmental reward quantity, , is detected with smoothing width by several neurons (in light blue). Each neuron k represents the probability that the environmental reward equals a quantity ; this probability is the area, taken between and , under the Gaussian curve centered around with standard deviation . 2) This yields the discretized reward distribution . In DDRL, all quantities are no longer calculated as average estimates, but rather as discretized distributions. 3) Example Q-return distribution, which is calculated according to our DDRL dynamics (Section 3.1). 4) The optimistic heuristic is applied to the Q-return distribution (in blue). It transforms the original distribution into a distorted one (outlined in magenta). The mean of this new distribution (magenta bar) is an optimistic estimate of the average Q-value (black bar).

https://doi.org/10.1371/journal.pcbi.1014723.g001

In the following subsections, we explain how we discretize and compute the relevant distributions.

3.1.1. Discretization and reward distribution.

In contrast to standard distributional RL, we already discretize the rewards and not just the returns. We do so to be able to compute the return distribution in closed form (Sect. 3.1.4), which is necessary to formulate the deterministic dynamics. We assume an agent has reward “neurons,” each neuron being sensitive to a given range of amount of reward. Throughout this work we use n = 29, balancing accuracy and computational cost across environments (see S1 Text, Part IV). We define (resp. ) the minimal (resp. maximal) amount of reward that can be detected. Similarly to categorical DistRL [46], we assume a regular grid such that , where denotes the width separating two consecutive grid points.

Fig 1 describes the process of discretization by reward neurons. In particular, it shows how an arbitrary reward amount gets represented as a discretized distribution by defining a Gaussian kernel centered around . To smooth out the effects of discretization, we assume that each neuron k is maximally sensitive to the reward amount , quantified by a smoothing width, . Each reward neuron “fires” with intensity , which quantifies the probability that . More precisely, is calculated as the area between and under the Gaussian bell curve centered around and with standard deviation ,

(14)

Therefore, this smoothing procedure captures some uncertainty about the categorization into different bins. This uncertainty is modulated by the smoothing width, . This smoothing is a soft projection: each atom or neuron receives probability mass rather than all mass going to the nearest neuron. It is analogous to label smoothing in supervised learning [50] and has been applied in distributional RL by Farebrother et al. [51]. Biologically, it reflects the overlapping tuning curves of reward-sensitive neurons, which respond to nearby stimuli [52,53]. Ultimately, this discretization method allows us to transform any stochastic game’s reward tensor into a reward distribution. For all , we define the reward distribution for agent i, current state s, joint action , and next state , as:

(15)

Note that each is an entry of the original reward tensor . In other terms, is the area between and under the bell curve centered around .

3.1.2. Reward distributions and .

The previous reward distribution can be averaged over joint actions and next states . We call the reward distribution for agent i taking action in state s, averaged over transitions to the next state , and over other agents’ actions :

(16)

The reward distribution denotes the probability that agent i receives reward r in state s. It is averaged over joint actions (including agent i’s) and next states :

(17)

3.1.3. Bellman equation and smoothed Bellman Kronecker .

In traditional, expected-value RL, the value of a state s for agent i is computed via the Bellman equation:

where is the value of the next state, , and is the effective state transition matrix given joint policy .

In distributional form, we want to compute the probability that (discretized) values r, v and satisfy the Bellman equation . To this end, we introduce a smoothed Bellman Kronecker delta,

(18)

using the same discretization method as above. The idea is that peaks when the Bellman condition is fulfilled for agent i, and its reward bins v, r, and , allowing for some uncertainty, modulated by the smoothing width, , around the discretization into different bins. We use this smoothed Bellman Kronecker delta to compute the V-return distribution in closed form.

3.1.4. V-return distribution .

The V-return distribution represents the probability that agent i in state s receives a long-term return v. It results from four instances that must occur together. For each short-term reward bin , for all possible next states , and each long-term return bin , i) v, r and must fulfill the Bellman equation according to probability , ii) agent i must receive short-term reward r in state s, according to probability , iii) state s must transition to state , according to probability , and finally iv) in state , agent i must receive a long-term return . More concisely,

(19)

We now summarize the smoothed Bellman-Kronecker delta, reward, and transition probabilities into a tensor Doing so, we can rewrite Eq. 19 as . We now combine the indices and , which transforms Eq. 19 into,

(20)

In other terms, for each agent i and joint policy X, the V-return distribution fulfills . Thus, is the eigenvector of the matrix corresponding to eigenvalue 1, normalized in such a way that for each state s, the probabilities to receive values sum up to 1, .

3.1.5. Q-return distribution .

The probability for agent i to receive long-term return v in state s under action is the joint probability that from all possible short-term reward values r, all possible next states , and all possible next long-term returns , i) v, r and fulfill the Bellman equation according to probability , ii) agent i receives short-term reward r in state s under action , according to probability , iii) state s transitions to state under agent i’s action , according to probability (the effective transition given joint strategy , marginalized over the other agents’ actions), and iv) in state , agent i receives a long-term return , according to probability . More concisely,

(21)

3.1.6. Temporal-difference error.

The SARSA and Actor-Critic temporal-difference errors differ slightly (see Section 2.4). Denoting the mean of a distribution on dimension d, the deterministic approximation of the temporal-difference error at time t for a SARSA agent i taking action in state s, , is:

(22)

For an Actor-Critic agent, the last term is not needed, as the “critic” (i.e., ) is constant in action (see Section 2.4). The remaining in Eq. 23 is therefore not the full TD error in the classical sense; it is the action-dependent component, which yields the same deterministic strategy update as the full TD error because the omitted critic is constant across actions and cancels in the Boltzmann update. Therefore, the deterministic approximation of the temporal-difference error at time t for an Actor-Critic agent i taking action in state s, , is:

(23)

3.1.7. Biasing the Q-return distribution.

Optimism (resp. pessimism) causes agents to overestimate (resp. underestimate) Q-values. There are different ways to obtain an optimistic (resp. pessimistic) estimate of a distribution. Here, we adopt a weights method: return distribution mass is rescaled by above the threshold and by unity below (reversed for pessimism). The primary motivation is a formal equivalence with asymmetric RL at a two-armed bandit choice task: a weights-DDRL agent with upper weight reaches the same steady-state Q-values as an asymmetric-RL agent with bias ratio , where and are the learning rates applied to positive and negative temporal-difference errors, respectively, (proven in S1 Text, Section 12.2), directly connecting this probability distortion to the established cognitive science literature [54]. In S1 Text, Section 12.1 we compare this against quantile- and expectile-based distortions of the return distribution [29]. While the qualitative outcomes are similar, these other distortions exhibit a less smooth parameter dependence as a result of our categorical discretization. In contrast, our weights method yields smooth, monotonic dependence on .

The weights method involves applying a weights tensor, , to the discretized distribution so as to distort it. Although there are different ways to distort a distribution, we use a stepwise function for simplicity, and because it is equivalent with asymmetric RL on a two-armed bandit task (see S1 Text, Section 12.2). For agent i in state s, the step is located at the scalar defined below: the mean of the action-averaged return distribution. When the agent is optimistic (resp. pessimistic), a higher weight is put on the part of the distribution which is above (resp. below) that step.

We define the threshold as the expected value of the action-averaged return distribution at state s under joint strategy . Marginalizing over agent i’s strategy yields the state-value distribution (consistent with Eq. 19); taking its mean over v gives the scalar threshold,

(24)

We define the weights tensor as follows. For all agent i, state s, action , and neuron ,

• if ;
• if ,

where if the agent is pessimistic, and if it is optimistic.

We obtain the weighted estimate of distribution by computing the product of the distribution with the weights vector, normalizing this product by a normalization factor , and computing the mean of the resulting distribution:

(25)

The normalization factor, , helps us make sure the integral of the weighted distribution over is 1.

To validate our framework and highlight its usefulness for uncovering new insights, we test our model across three different choice environments: an individual bandit choice task under resource variability, a social coordination challenge, and an intertemporal risky choice problem.

3.2. Example application: Bandit choice under resource variability

Here, we use our model to confirm previous findings showing that the influence of optimism on performance in a two-armed bandit task is modulated by resource scarcity. The two-armed bandit is a standard task in cognitive psychology to study asymptotic strategy formation between options with different reward probabilities. An agent must choose between two options, a₁ and a₂. a₁ gives a reward +1 with probability p₁, and a punishment with probability ; a₂ gives a reward +1 with probability p₂, and a punishment with probability . By repeatedly choosing between the two options, the agent must learn which is most rewarding on average (Fig 2A). We use SARSA rather than Actor-Critic dynamics here because Actor-Critic dynamics (analogous to the pure replicator equation), drive agents toward deterministic strategies, whereas SARSA (analogous to the replicator-mutator equation) admits mixed asymptotic strategies, as observed in previous research on this task [54].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Influence of optimism/pessimism on a two-armed bandit environment with varying resource availability.

A The two-armed bandit environment. B–C Comparison between final strategies obtained via DDRL (yellow curve) vs. asymmetric RL (blue curve), B on a “poor” two-armed bandit task (p₁ = 0.3, p₂ = 0.1); C on a “rich” two-armed bandit task (p₁ = 0.9, p₂ = 0.7). These final strategies vary as a function of bias strength . For DDRL agents, the final strategies were obtained from a trajectory of 150 time steps, after checking that this number of time steps was sufficient for convergence (see Fig 2D and 2E). For asymmetric agents, the final strategies were obtained analytically, using the formula in [54]. D–E Strategy plots of a DDRL agent performing a two-armed bandit task with various levels of optimism/pessimism, D in a poor environment (p₁ = 0.3, p₂ = 0.1); E in a rich environment (p₁ = 0.9, p₂ = 0.7). These plots represent the evolution of an agent’s probability to choose the best arm over time. Blue lines indicate pessimistic agents (), while red lines indicate optimistic agents (). Neutral agents () are represented by a black line. In all panels, all DDRL agents had n = 29 neurons, and a smoothing width to minimize errors (see S1 Text, Part IV).

https://doi.org/10.1371/journal.pcbi.1014723.g002

In previous research using two-armed bandit tasks, it has been established via computational modeling that optimism (implemented as an asymmetric pair of learning rates) increases or decreases an agent’s performance depending on resource scarcity. Optimism is beneficial for performance in scarce environments, and detrimental in abundant environments [54]. Empirical research on human decision-making also revealed an asymmetric sensitivity to positive and negative prediction errors. Individual differences in the ratio of positive to negative learning rates predict risk-seeking behavior in bandit tasks [55] and reflect variation in dopaminergic signaling [56,57]. Humans tend toward a stable positivity bias across environments rather than adaptively modulating this asymmetry with resource scarcity [58,59], consistent with the non-adaptive optimism weights in DDRL. Therefore, we tested our framework on a “poor” two-armed bandit, in which the arms’ reward probabilities are p₁ = 0.3 and p₂ = 0.1, and on a “rich” version of the same task, in which the arms’ reward probabilities are p₁ = 0.9 and p₂ = 0.7.

First, we find that DDRL closely approximates the behavior of an asymmetric-RL agent on a two-armed bandit task (Fig 2B and 2C). To do so, we checked that both algorithms converged towards the same final strategy –i.e., an agent’s probability to choose the best arm after convergence. We computed final strategies for different bias strengths. In asymmetric RL, bias strength is defined as (where updates positive temporal-difference errors and , negative temporal-difference errors). An analytical calculation of as a function of b is given in [54]. In DDRL, bias strength is modulated by the upper weight, . Here, we simulated the deterministic trajectory of an agent’s strategy X over time until convergence for different values of .

Second, we replicate the aforementioned findings on the influence of optimism/ pessimism in poor/rich environments (Fig 2B and 2C). In both scarcity conditions, DDRL closely approximates asymmetric RL. More precisely, the final strategy of a DDRL agent with weight matches the final strategy of an asymmetric-RL agent with bias strength . We give a formal proof of this equivalence in S1 Text, Section 12.2. Fig 2B and 2C shows how the influence of optimism/pessimism on performance is modulated by resource scarcity. In a poor environment (Fig 2B), optimistic agents () converge to a higher probability of choosing the best arm than neutral () and pessimistic () agents. In a rich environment (Fig 2C), by contrast, pessimistic agents converge to a higher probability of choosing the best arm than neutral () and optimistic () agents. Furthermore, both curves indicate the presence of an optimal bias strength (or upper-weight value), for which the final strategy is maximal. In our poor environment, this optimal upper-weight value lies close to ; whereas, in our rich environment, it is close to .

This interaction between optimism/pessimism and resource scarcity, previously established in Ref [54]., is further visualized in Fig 2D and 2E. These strategy plots depict the evolution of a DDRL agent’s strategy–i.e., its probability of choosing the best arm–over time for different levels of optimism/pessimism. In a poor environment (Fig 2D), starting from a uniform random strategy (X = 0.5), optimistic DDRL agents (, red lines) converge towards a higher probability of choosing the best arm than the neutral agent (, black line); whereas pessimistic DDRL agents (, blue lines) converge towards lower performance. In line with previous findings [54] and the results mentioned above, this pattern is reversed in a rich environment (Fig 2E).

Overall, our results confirm that DDRL closely approximates asymmetric RL on a two-armed bandit task with varying resource scarcity. In particular, we replicate previous findings indicating that optimism is beneficial in poor environments, while pessimism enhances performance on the rich version of the task. This means DDRL is a suitable model for optimism/pessimism in this basic choice task. Next, we turn to an environment which is widely used in biology, economics, and sustainability science: the Stag-Hunt game.

3.3. Example application: Social coordination task

Here, we use our model to confirm previous findings that show optimism improves coordination in the Stag-Hunt game. The Stag-Hunt game is an iconic coordination problem in the space of social dilemmas [60]. A social dilemma is a situation in which agents individually have an incentive to deviate from outcomes that would be best for the group. Each agent must choose between two actions: cooperate (c) or defect (d). In the two-agent case, if both cooperate, they get a payoff R (called “reward”). If both defect, they get a payoff P (called “punishment”). If one cooperates and the other defects, the cooperator receives a payoff of S (called “sucker”) and the defector gets a payoff of T (called “temptation”) (Fig 3A). In the Stag-Hunt game, the payoff ordering is as follows: R > T > P > S. Therefore, both mutual defection and mutual cooperation are Nash equilibria. However, if both agents defect, they receive the punishment payoff P, which is lower than the maximum reward payoff R obtainable by mutual cooperation. This means individual interests align with collective interests, but the best outcome can only be achieved through coordination among agents.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Influence of optimism/pessimism on a Stag-Hunt game.

A Structure of a social dilemma. In the Stag-Hunt game, R > T > P > S. For our analyses, we set R = 3, T = 1, P = 0 and as illustrative values satisfying this ordering; the qualitative structure of fixed points and their stability depends only on the ordering, not the specific magnitudes. B Phase plot and C strategy plot of an agent playing the Stag-Hunt game for various levels of optimism/pessimism. Since the Stag-Hunt game is symmetric among agents, we assumed the second agent had the same strategy as the first. On the phase plot (B), X = 1 indicates maximal cooperation, and X = 0, maximal defection. Intersections of the curves with the x-axis indicate stable and unstable fixed points, depending on the sign of the derivative dX. The strategy plot (C) represents the evolution of an agent’s probability to cooperate over time, given an initial cooperation probability of 0.5. Blue lines represent pessimistic agents (), while red lines represent optimistic agents (). Neutral agents () are represented by a black line. D–F Flow plots of two agents playing the Stag-Hunt game D with a pessimistic heuristic (); E with no heuristic (“neutral,” ); F with an optimistic heuristic (). Arrows represent the direction and intensity of the strategies’ derivative dX. Colored lines are sample trajectories starting from various initial conditions (+), while round dots indicate fixed points. Dashed gray lines delineate the flow plots’ separatrices–i.e., the threshold of initial cooperation probability above which agents learn towards maximal cooperation, and below which they learn towards maximal defection. In all panels, all DDRL agents had n = 29 neurons, and a smoothing width to minimize errors (see S1 Text, Part IV).

https://doi.org/10.1371/journal.pcbi.1014723.g003

Previous research in multi-agent reinforcement learning suggests that variants of optimism enhance coordination on various tasks. For instance, Lenient Learning, an algorithm that initially has agents dismiss negative outcomes, has led to performance improvements across multiple coordination games, such as the climb and penalty games [23]. Likewise, Hysteretic Q-learning (a multi-agent equivalent of asymmetric RL) has been shown to improve coordination on several tasks: the aforementioned climb and penalty games, a collaborative ball balancing task, and a pursuit game (where predators need to coordinate to capture prey) [22]. In this section, we investigate whether optimism enhances coordination in the Stag-Hunt game. The climb and penalty games are two three-action versions of the Stag-Hunt problem, which received its name from a story in which hunters need to coordinate to capture a stag, analogous to predators coordinating to capture prey. Thus, the Stag-Hunt game is well-suited to capture the essence of these coordination problems. Human experimental evidence corroborates this connection. When participants reason under time pressure, favoring intuitive, optimistic social expectations, they cooperate more in Stag-Hunt tasks than when reasoning deliberatively [61]. Shared optimistic beliefs about a partner’s trustworthiness increase cooperation by up to 40 percentage points [62], and optimism about partner behavior predicts payoff-dominant choices in experimental coordination tasks [63].

In line with previous findings, we find that optimistic DDRL agents converge to mutual cooperation when starting from a uniformly random strategy. In contrast, pessimistic agents converge to the suboptimal equilibrium of mutual defection. A neutral agent remains at its mixed strategy for the payoff values of our Stag-Hunt game (Fig 3C). Fig 3B shows that the uniform random strategy (i.e., a cooperation and defection probability of 0.5) is an unstable fixed point of the learning dynamics for neutral agents. Therefore, if a neutral agent’s initial strategy is higher than 0.5 (and provided the second agent’s initial strategy is the same), it converges toward cooperation. If, by contrast, both agents’ initial strategies are below 0.5, they converge toward defection. Optimism (red lines in Fig 3B) lowers this threshold–i.e., two optimistic DDRL agents may converge towards cooperation even if they started with a strategy X < 0.5–while pessimism heightens it. This is why, in Fig 3C, when all agents start with a uniformly random strategy, optimistic agents end up mutually cooperating, whereas pessimistic agents converge towards mutual defection. Thus, optimism facilitates cooperation, while pessimism hinders it.

The flow plots in Fig 3D, 3E and 3F give an overview of the learning dynamics in the two-dimensional phase space spanned by each agent’s individual strategy. Thus, the phase plots in Fig 3B are equivalent to the diagonal in the flow plots in Fig 3D–3F, for the relevant optimism levels. The strategy changes dX are represented by the flow arrows. The separatrix curves separating the two basins of attraction of the two equilibria, mutual cooperation and mutual defection, are shown with dashed gray lines. Optimism distorts these separatrices. When agents are pessimistic, it takes higher initial levels of cooperation to eventually learn to fully cooperate (Fig 3D). When they are optimistic, learning to cooperate fully is possible even from initially lower levels of cooperation (Fig 3F).

Additionally, we tested our model on a prisoner’s dilemma with T > R > P > S. Here, agents have the incentive to defect, regardless of what the other agent does. Thus, mutual cooperation is no longer an equilibrium despite being the optimal solution for the collective. We find that optimism/pessimism does not affect the learning dynamics. In other terms, whatever the upper-weight value , DDRL agents always converge towards mutual defection in this environment (see S1 Text, Section 13.3). In this dilemma, indeed, the reward R is lower than the temptation T. Therefore, if an agent is optimistic that the other will cooperate, it is still incentivized to defect, since it will then obtain T instead of R. The mutual-defection equilibrium thus cannot be changed through the optimism/pessimism heuristic. In a Stag-Hunt game, by contrast, if an agent is optimistic that the other agent will cooperate, it increases its incentive to cooperate as well (see S1 Text, Section 13.4 for an analytical explanation of this difference between the two games).

In short, our results confirm that optimism enhances coordination in DDRL agents, while pessimism hinders it. These results echo previous findings across several multi-agent reinforcement learning algorithms in coordination games. Additionally, our flow plot visualization provides further insights into the collective learning dynamics in coordination games and into how the optimistic/pessimistic heuristic distorts this landscape. Next, we consider a lesser-known intertemporal risky choice problem.

3.4. Example application: Intertemporal risky choice problem

Here, we use our model to confirm previous findings that show optimism increases risk-taking. We also uncover new findings in this lesser-known choice environment [39,64], as it incorporates a novel component of decision-making into our analysis: intertemporal choice. In the choice environment, an agent must trade off a low but safe reward with a high reward with potentially negative future consequences. This is implemented via two environmental states: a prosperous and a degraded one. The agent must choose between a risky action and a safe action. In the prosperous state, taking the risky action leads to a reward , whereas the safe action leads to a reward . However, choosing the risky action in the prosperous state results in a transition to the degraded state with probability . There, the agent gets no reward , and can only transition back to the prosperous state with probability if it chooses the safe action (Fig 4A). To simplify the analysis, we use an Actor-Critic agent here, since it has no explicit exploration term (see Section 2.2). We presume that a SARSA agent would exhibit additional, exploration-related effects and interactions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Influence of optimism/pessimism in a risk-reward dilemma.

A Structure of a risk-reward dilemma. In our analyses, we set the collapse probability and the recovery probability ; we fixed the risky reward , the safe reward , and the degraded reward . B–C Phase plots of a DDRL agent performing a risk-reward dilemma with various levels of optimism/pessimism, B with a low discount factor (); C with a high discount factor (). Blue lines indicate pessimistic agents (), while red lines indicate optimistic agents (). Neutral agents () are represented by a black line. D–E Phase plots of an agent performing a risk-reward dilemma with various discount factors, D using a regular deterministic Actor-Critic algorithm, E using a neutral () DDRL Actor-Critic algorithm. On all phase plots (B–E), X = 1 corresponds to maximal safety, and X = 0, to maximal risk. Intersections with the x-axis indicate stable and unstable fixed points (depending on the sign of the derivative dX). F Stability landscape of a DDRL on a risk-reward dilemma for various combinations of weights and discount factors . Red patches correspond to regions in the parameter space where the risky policy () is stable, while blue patches indicate that the safe policy () is stable. Purple patches correspond to regions in the parameter space where both policies are stable. Stars mark regions of interest, where the safe or risky policy has highest value but is unstable. In all panels, the agent had n = 29 neurons, and a smoothing width to minimize errors (see S1 Text, Part IV).

https://doi.org/10.1371/journal.pcbi.1014723.g004

To our knowledge, no behavioral experiment has studied an environment combining these features: repeated stochastic choice with state-dependent consequences and temporal recovery dynamics. Risk-taking and intertemporal discounting are each extensively studied in isolation; their combination has been explored in simplified choice tasks [65], but not in a multi-state repeated structure of this form. Previous research on risk-sensitivity in RL agents uses asymmetric learning rates to implement different attitudes towards risk, where optimism corresponds to risk-seeking and pessimism to risk-aversion [66,67]. Thus, we hypothesize that in our intertemporal risky choice problem, an optimistic DDRL agent is more likely to choose the risky action than a neutral agent. We also hypothesize that this pattern is reversed for pessimistic DDRL agents.

We confirm both hypotheses. Optimism increases risky behavior while pessimism favors safe behavior (Fig 4B and 4C). Fig 4B and 4C show the strategy change versus the strategy to choose the safe action in the prosperous environmental state for different levels of optimism and pessimism and two discount factors. In the degraded state, only the safe action leads to a recovery to the rewarding prosperous state. Thus, we assumed for Fig 4B and 4C. Flow plots of the entire strategy space for given discount factors and weight values can be found in S1 Text, Fig 14.6 and S1 Text, Fig 14.7.

When the agent cares little about future rewards (low discount factor , Fig 4B), only intense pessimism can push an agent to act safely. Only the most pessimistic agents ( and ) have a chance to converge towards complete safety in the prosperous state (). All the other agents, including the moderately pessimistic and neutral ones, converge towards maximal risk-taking (). This means that, in such situations, to act safely, intense pessimism can compensate for an agent’s lack of concern for future rewards.

When the agent cares more about future rewards (high discount factor , Fig 4C), the more optimistic an agent is, the more likely it is to converge toward the risky strategy. In fact, for a moderately-to-strongly optimistic agent (), the risky strategy is the only stable one. A mildly optimistic agent () could converge to a maximally safe approach, but only if its initial safety levels are very high. A reversed pattern is found for the different levels of pessimism. Because is high, risk-seeking is overall lower than with a low discount factor.

In both cases, we observe instances of bistability in the strategy space. Which strategy the agent converges to depends on the initial strategy. This is particularly interesting as non-distributional agents (using expected values from a continuous reward spectrum) do not exhibit any bistable regimes (Fig 4D). This observation aligns with the finding that an optimal strategy in a Markov Decision Process is always deterministic [68]. Fig 4E compares our distributional agent dynamics without any bias ( = 1) for different discount factors. We find certain discount factors (e.g., , ) at which the learning dynamics are bistable. The neutral weight rules out optimism or pessimism as a cause; the only remaining structural difference between Fig 4D and 4E is therefore the discretization of the return distribution. Thus, bistability does not solely emerge from optimism or pessimism, but from an intrinsic feature of our DDRL framework: the discretization of the return distribution. Fig 4F visualizes how the stability of the safe and risky strategies depends on the interaction of bias weight values and discount factors . The red zone denotes the stability of the risky strategy (); the blue zone, that of the safe one (); whereas, in the purple zone, both strategies are stable. Thus, bistability emerges for a broad range of weights and discount factors, including for neutral agents (). Overall, we find that these bistable regimes depend on the interplay among the discount factor, the optimism weight, and return discretization, whose granularity is set by the smoothing width .

We also find a kind of incoherent choice pattern, which we coin “individual dilemma.” Just as in a social dilemma, an agent would be better off playing the strategy that yields the highest value, but it cannot realize this, as it is pulled towards the other, suboptimal strategy, which is stable (Stars in Fig 4F). To see this, we highlighted the region in parameter space where the safe policy yields a higher value to the agent than the risky policy () and likewise, . We define and . For most parameter values, when a strategy yields a higher overall value, it is also stable. Nevertheless, Fig 4F exhibits some deviations from this pattern. For example, at and , the safe strategy is stable while the risky strategy yields a higher value (Star 1 in Fig 4F). At and , the risky strategy is stable, while the safe strategy yields a higher value (Star 2 in Fig 4F). Although the state value of the safe strategy is higher than that of the risky one, the state-action value of the risky action is higher than that of the safe action. In S1 Text, Fig 14.10, we show the corresponding agent’s V-return and Q-return distributions.

Overall, our results confirm previous findings regarding the relationship between optimism, pessimism, and risky behavior. Optimism enhances risky behavior, whereas pessimism favors risk-averse behavior. Additionally, our framework generates two novel predictions: bistable regimes and individual dilemmas. Both have no counterpart in non-distributional dynamics and await empirical test.