The policy compression framework.

The human mind must navigate numerous constraints stemming from the cognitive resources at its disposal [10,26], and these resources have been conceptualized at both physiological [27] and computational levels [28–31]. Here, we specifically focus on the influence of channel capacity, which is the maximum information that can be transmitted across a noisy channel [32,33], on decision-making processes (Fig 1B).

The framework we propose is an application of rate-distortion theory to action selection. Rate-distortion theory describes how to construct an optimal channel that minimizes some notion of error (the distortion)—or, in our case, maximizes reward—subject to a constraint on the information transmission rate [34]. The utility of rate-distortion theory lies in its generality: beyond action selection [23–25], it has been applied to various cognitive processes, including visual working memory [35–37], perception [38], intertemporal decision-making [22], economic behavior under imperfect information [39], cognitive abstraction formation [40], and task-switching costs [41]. Another reason for using rate-distortion theory comes from its information-theoretic nature, which has been classically applied to explain the influence of the number of available actions on response times via the Hick-Hyman Law [42,43]. Given this alignment, rate-distortion theory serves as a fitting framework for our exploration of action consideration sets, offering both theoretical insight and empirical grounding.

For a resource-rational agent, we formalize the cognitive cost as the mutual information between states and actions , which we call the policy complexity:

(1)

where P(s) is the state distribution, is the policy, a probabilistic mapping from states to actions, and is the marginal probability of choosing action a. Intuitively, high-complexity policies preserve state information (e.g., deterministic mappings from states to actions) whereas low-complexity policies discard state information (e.g., random actions).

If the agent has infinite cognitive resources, it would be optimal to map each state to the most rewarding action under it. However, we assume that policies are subject to a capacity limit C, which acts as an upper bound on policy complexity. Consequently, for a given task with state probability distribution P(s) and state-specific action rewards , agents would maximize trial-averaged reward under their constraint . The above constrained optimization problem prescribes the following optimal policy, which can be empirically found via the Blahut-Arimoto algorithm [44–46]:

(2)

where is the optimal marginal action distribution, and β is a Lagrangian multiplier whose value depends on C. Intuitively, at high policy complexity (corresponding to large C), the value of β is large and the optimal policy is dominated by Q-values, which renders it state-dependent. At low policy complexity (small C), the value of β is close to 0 and Q-values have minimal impact on the optimal policy. Moreover, low-complexity policies are dominated by the term, a form of perseveration (state-independent actions; Fig 1C). In general, high-complexity policies enable more trial-averaged reward than low-complexity policies due to their state-dependence. By varying β and calculating the optimal policy, we can trace out the reward-complexity frontier (also known as the rate-distortion frontier; Fig 1D), which delimits the maximal trial-averaged reward obtainable for a given policy complexity.

A noteworthy observation is the optimal policy’s dependence on the marginal action distribution P^*(a), which grows stronger as policy complexity decreases. In tasks where P^*(a) is uniform over actions, the optimal policy reduces to the traditional softmax choice rule [47]. However, in tasks where P^*(a) is non-uniform and biased towards specific actions, then the influence of perseveration is non-trivial. This prediction differs from a traditional softmax model, where the policy approaches a uniform distribution as β approaches 0. In recent works, we identified behavioral signatures unique to policy compression—and not predicted by the traditional softmax choice rule—in human data [24,25], which we will replicate in this manuscript.

Incorporation of action consideration sets.

The above formulation assumes that the agent represents all task information: the marginal state distribution P(s) and the reward structure . However, when the action space is either large or infinite, it becomes infeasible for an agent to consider all possible actions exhaustively and represent them in a policy. Instead, agents may simplify the problem either deliberately or by necessity, considering only a subset of all available actions. This simplification enables agents to address a more manageable subproblem—optimizing and implementing a policy over this action consideration set.

It is straightforward to operationalize action consideration sets in the policy compression framework. Since Eq 2 depends only on the task’s P(s) and , one can simply remove the excluded actions from the matrix, and recompute the optimal policy via the Blahut-Arimoto algorithm. The resultant reward-complexity frontier corresponds to the action consideration set, and one can compare it with the true task’s reward-complexity frontier to assess the extent of suboptimality caused by not considering all actions.

We illustrate the proposed analysis on an example task, in which P(s) assigned equal probabilities to all 6 states and assigns each state a unique optimal action (Fig 2A). We can obtain the reward-complexity frontier, assuming access to the full action space. Given the symmetric nature of P(s) and in this example task, we can study the influence of partial action consideration sets by removing any number of actions, and similarly deriving reward-complexity frontiers for the remaining action consideration set without loss of generality (Fig 2B). We can hence evaluate the loss in reward associated with using each action consideration set compared to the full-action-space reward-complexity frontier, by subtracting the latter out at each policy complexity level (Fig 2C). This provides a notion of suboptimality along the orthogonal dimensions of policy complexity and action consideration set size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The influence of partial action consideration sets on a symmetric reward structure task.

(A) The reward matrix of the example task. Each state has a unique optimal action. We assume that the task has flat P(s) (all states being equiprobable). (B) Reward-complexity frontiers associated with different numbers of actions (N_a; colors) retained in the action consideration set. The frontier corresponding to the full action space is colored black. Note that the maximum policy complexity allowed by N_a is bits, such that different colored curves end at different policy complexity levels. (C) The deviation of partial-action-space reward-complexity frontiers to the full-action-space reward-complexity frontier, at different policy complexity levels.

https://doi.org/10.1371/journal.pcbi.1013444.g002

Simulation-specific model components.

In simple tasks—such as the one presented in Fig 2 and our later human experiment—it is feasible to derive reward-complexity frontiers for every possible action consideration set. However, this exhaustive approach quickly becomes impractical in more realistic settings with asymmetric reward structures and large action spaces. While an optimal action consideration set exists for any given task and policy complexity level, systematically identifying it is computationally infeasible for agents and difficult to generalize across tasks. This challenge highlights the need for heuristics that guide action consideration set formation and subsequent policy optimization—heuristics that agents could learn over longer timescales, amortize in memory, and apply flexibly to new tasks [48].

Motivated by the challenge above, we explore the advantages of biased action sampling and the interplay between policy complexity and action consideration sets by employing numerical simulations in more complex task setups. Such simulations require modeling action proposal distributions and incorporating bias-correction algorithms, both of which serve as tractable heuristics for constructing effective action consideration sets and approximating optimal policies.

Action proposal distributions.

Following the two-stage decision-making architecture described earlier (Fig 1A) [19], we first consider the proposal distribution P₀(a) through which actions are sampled and enter the consideration set. The first two candidates are the flat distribution and the general-value-based distribution, as specified in past works [7]. Given the policy compression framework, a third proposal distribution of interest is the optimal policy’s marginal distribution P^*(a) at whichever policy complexity level the agent commits to, assuming the full action space. This proposal distribution is an oracle, because one needs to know the optimal policy to find it. However, it serves as a benchmark for the performance of the previous two candidates. To summarize, we consider three candidate proposal distributions:

1. Flat distribution:
2. General-value-based distribution:
3. Oracle distribution:

Another model parameter worth exploring is whether actions are sampled with or without replacement from these proposal distributions. This would impact the bias-correction algorithms we introduce below.

Bias-correction algorithms.

After sampling actions from the proposal distribution P₀(a), the simplest solution is to perform the Blahut-Arimoto (BA) algorithm on the action sample. The BA algorithm would enable us to find the optimal policy at multiple policy complexity levels, assuming that the action consideration set equals all unique actions sampled. Notably, BA is indifferent to how often each action is sampled—it considers only the number of distinct actions in the sample, without accounting for their frequencies. Furthermore, BA does not correct for any bias in the P₀(a) that may make some actions more likely to be sampled than others.

In contrast to BA, a truly bias-correcting algorithm is self-normalized importance sampling (SNIS), which assumes sampling actions with replacement and is informed by the repetitive sampling of the same action [49,50]. We consider SNIS a compelling alternative algorithm to examine for two main reasons. First, SNIS has previously been shown to account for the human tendency to oversample extreme outcomes when estimating the expected utilities of actions [51]. Second, similar importance sampling algorithms have successfully explained a wide range of human behaviors under resource constraints, including multiple object tracking [52], concept learning [53], reinforcement learning [54], and sentence parsing [55]. While sampling outcomes and actions are technically different domains, we postulated that SNIS may be an equally effective algorithm for action selection.

According to SNIS, we assume that independent action samples are drawn with replacement from the proposal distribution P₀(a). We can then construct the following estimator for under the full action space:

(3)

where is the indicator function comparing to a. Intuitively, any bias in the action proposal distribution would reflect in both the counts and the denominator P₀(a), such that they would cancel out in the asymptotic limit of infinite samples and thus achieve bias correction. Through SNIS, is an asymptotically unbiased estimator of at the same β value, and can be used for action selection.

The BA and SNIS algorithms differ in their underlying objectives. BA is designed to maximize trial-averaged reward given a specific action consideration set and varying levels of policy complexity [44,45]. In contrast, SNIS does not aim to optimize rewards within the current consideration set but instead prioritizes unbiasedness—seeking to approximate the full-action-space optimal policy at the same β without introducing bias. Later in the paper, we assess how SNIS’s objective influences reward attainment. Using an approach similar to BA in Fig 2C, we evaluate the average deviation of (computed for each action sample ) from the true task’s reward-complexity frontier, across different sample sizes n. This deviation quantifies the potential loss in achievable reward.

Note that the SNIS assumption of sampling actions with replacement requires a different resource formulation—not the action consideration set size (i.e. the number of distinct actions considered, N_a), but the number of action samples (n). There is limited evidence on whether humans sample actions with or without replacement, and hence we will explore both cases. Luckily, the simpler BA algorithm is compatible with both sampling methods—when sampling actions with replacement, it simply does not make use of how often each action is sampled, relying only on the set of unique actions. Hence, we can fairly compare BA and SNIS in the regime of sampling with replacement. To summarize, we consider two bias-correcting algorithms in the simulations:

1. Running Blahut-Arimoto on the retained actions (BA): no bias correction, applicable to sampling actions both with and without replacement.
2. Self-normalized importance sampling (SNIS): has bias correction, applicable only to sampling actions with replacement.

Experimental task.

We manipulated response time (RT) deadlines consistent with previous works on policy compression and option generation [20,24], while leaving the state distribution P(s) and reward structure fixed. Participants completed three blocks of trials with RT deadlines 2s, 1s, and 0.5s respectively in random order. On each trial, participants see one of six possible images (state) and must press one of seven keys (actions) within the RT deadline. Reward delivery was deterministic: each state corresponded to a unique reward-maximizing (optimal) action, and the remaining action is a “safety” action that yielded a small but positive reward regardless of the current state (Fig 4A and 4B). Failure to respond within the RT deadline resulted in a −1 reward penalty. After making a response, participants receive feedback on their reward yielded, before being redirected to the next trial (Fig 4C). Due to the simplicity of the task, we can exhaustively enumerate reward-complexity frontiers associated with different action consideration sets, depending on their size (N_a) and whether the safety action is included (Fig 4D and 4E). We can then find the higher of the two frontiers associated with the same consideration set size, and visualize it as the N_a-specific reward-complexity frontier (Fig 4F). Inspection of these frontiers reveal that the normative interplay between policy compression and action consideration set size, as disclosed by previous simulations, holds in this experimental task, thus enabling our predictions below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Human experiment task setup.

(A) The six possible states (images) and the corresponding optimal actions (key presses). The optimal mapping between images and keys was randomized across participants. The example stimuli images visualized are adapted from [59] and differed slightly from those used in the actual experiment. (B) The reward structure of the task. There is a safety action that yields deterministically a low but positive reward for all states. Within each block, we counterbalance the number of trials where each state is used, such that the empirical P(s) is flat. (C) On every trial, the participant observes an image (state) and responds by pressing a key (action). Then, reward feedback is provided as a border around the image where the border color denotes reward, and then as a numeric reward value, before the next trial starts. Participants can track cumulative reward (number above image) for the block. After the block ends, participants receive feedback on the total reward gained in the block. Participants are informed of the block’s RT deadline before starting. (D) Reward-complexity frontiers under different action consideration set sizes, where the safety action is retained in the set. (E) Reward-complexity frontiers under different action consideration set sizes, where the safety action is not retained in the set. (F) The theoretical reward-complexity frontier under different action consideration set sizes, constructed by finding the maximum between (D) and (E) for each policy complexity level and action consideration set size.

https://doi.org/10.1371/journal.pcbi.1013444.g004

Experiment predictions.

We make the following predictions regarding policy compression alone, which help assess the behavioral relevance of this normative framework: shorter RT deadlines should be associated with 1) lower policy complexity; 2) lower trial-averaged reward; 3) higher probability of choosing the safety action. The first prediction arises from previous works showing that policy complexity incurs time costs [24,25]; the latter two predictions arise from the task’s reward-complexity frontier, which takes into consideration the safety action having higher general value than other actions, and that it should always be chosen at policy complexity 0 bits. Given the time cost implication of policy complexity, we also predict that 4) higher policy complexity incurs longer RTs.

Regarding the interplay of policy complexity and number of distinct actions chosen, we make the following predictions: 5) shorter RT deadlines should be associated with fewer numbers of distinct actions chosen, and 6) higher numbers of distinct actions chosen incur longer RTs, because both predictions are consistent with previous accounts of set size effects on RT as well as sampling costs [30,42,43]. Based on the simulation results, we also predict that 7) In terms of achieving higher trial-averaged reward, policy complexity has a positive effect, while increasing the number of actions chosen exacerbates the above positive effect; 8) In terms of achieving lower suboptimality (loss in trial-averaged reward compared to the full-action-space reward-complexity frontier at the same policy complexity level), policy complexity has a negative effect, while increasing the number of actions chosen mitigates the above negative effect. A normative corollary of the predictions above is that 9) policy complexity and the number of actions chosen should be positively correlated, to mitigate any suboptimalties. Note that while the number of actions chosen places an upper bound on policy complexity, participants still have freedom to empirically adjust one independent from the other across task conditions (e.g., incorporate more actions while maintaining low policy complexity), such that nonsignificant correlations are in principle possible. Hence, any correlations identified in 9) may still be interpreted qualitatively with theoretical significance.

Orthogonal to the policy compression-action consideration set framework, we also made predictions regarding how a human participant decides on their policy complexity level and action consideration set size. We postulate that higher training block accuracy rates (averaged over training blocks) should be associated with 10) higher mean policy complexity and 11) higher mean number of actions chosen across test blocks.

The predictions above are validated using a combination of linear mixed-effects (LME) regression models and Pearson correlation coefficients. The statistical analysis details are elaborated in Methods.

Experimental results.

Consistent with previous studies [21,24,25], most participants achieved trial-averaged reward levels that were not far from the full-action-space reward-complexity frontier (Fig 5A). However, given the task’s large action space, visible suboptimalities did exist. Most participants lay closer to the ranges of their N_a-specific reward-complexity curves, indicating that the number of actions chosen could qualitatively explain some portion of participant suboptimality. However, there also existed a cluster of extremely suboptimal participants with low policy complexity and N_a = 6 or 7, who were earning much lower trial-averaged reward compared to that yielded by always choosing the safety action (y intercept of all reward-complexity curves at y = 0.2; Fig 5B and 5C). In the Supporting Information, we re-performed all analyses after removing this cluster of N = 13 participants (post-hoc; identified based on an empirical trial-averaged reward cutoff to accommodate occasional key-pressing errors), thus concluding that our results reported below were not solely the result of significant participant suboptimality (Table A in S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Human experiment behavioral results.

(A) Trial-averaged reward and policy complexity of participants across RT deadline conditions (colored points), and the full-action-space reward-complexity frontier (black line) at each policy complexity level. Semitransparent gray lines between points connect the same participant’s data points in different RT deadline conditions. (B) The same figure as (A), but visualizing the number of distinct actions taken (color) by each participant in each condition, and the reward-complexity frontier associated with each number of distinct actions taken (colored lines). (C) The same figure as (B), but visualizing each data point’s deviation from the full-action-space reward-complexity frontier. (D-H) Mean±SEM of participant trial-averaged reward (D), probability of choosing the safe action (E), policy complexity (F), number of distinct actions chosen (G), and mean RT (H) across RT deadline conditions. All SEM errorbars are within-participant [60]. (I) Correlating policy complexity and number of distinct actions chosen. The color scheme is identical to (A).

https://doi.org/10.1371/journal.pcbi.1013444.g005

In line with typical predictions of policy compression, during long RT deadline blocks participants used policies with higher complexity (fixed effects , , p < 10⁻¹⁸, random effects SD = 0.401; Fig 5D), earned higher trial-averaged reward (fixed effects , , p < 10⁻¹⁷, random effects SD = 0.144; Fig 5E), chose the safety action less frequently (fixed effects , , p < 10⁻¹⁷, random effects SD = 0.0910; Fig 5F), and consequently incurred longer RTs (fixed effects , , p < 10⁻²³, random effects SD = 0.112; Fig 5G). More importantly, individual participant RT related positively to their policy complexity levels (fixed effects , , p < 10⁻⁶⁹, random effects SD = 0.0468; S6B Fig). These results suggest that the policy compression framework offers a good first-hand description of human behavior in our task, thus motivating further analyses.

We next tested predictions related to the number of actions chosen. As predicted, participants increased their number of actions chosen for longer RT deadline conditions (fixed effects , , p < 10⁻¹³, random effects SD = 0.0488; Fig 5H), featuring adequate variability needed for condition-specific analyses. Also, Individual RT related positively to their number of actions chosen (fixed effects , , p < 10⁻²⁷, random effects SD = 0.0282; S6C Fig). To assess whether the number of actions chosen contributes to RT in a way independent from its influences on policy complexity, we ran a post-hoc LME analysis and found significant positive effects for both policy complexity (fixed effects , , p < 10⁻¹¹, random effects SD = 0.0568) and the number of actions chosen (fixed effects , , p = 0.0200, random effects SD = 0.0177). However, the statistical significance of the number of actions chosen was lost when we changed the threshold for action counting, reducing it to a trend (see Tables B and C in S1 Appendix). Given these ambivalent results, we postulate that classical empirical results connecting the number of actions to RT may be largely explained by policy complexity, a direction that warrants future testing [42,43].

We next tested the most important framework predictions regarding the interplay between policy complexity and the number of actions chosen. As predicted, in terms of trial-averaged reward, policy complexity contributed positively (fixed effects , , p < 10⁻²⁴, random effects SD = 0.0768), while the interaction term supported our hypothesis that increasing the number of actions chosen would exacerbate the above positive relationship (fixed effects , , p < 10⁻¹⁰, random effects SD = 0.00659). Similarly, in terms of reducing loss in trial-averaged reward compared to optimal, policy complexity contributed negatively (i.e. enlarging the suboptimality; fixed effects , , p < 10⁻²⁶, random effects SD = 0.0864), while the interaction term supported our hypothesis that increasing the number of actions chosen would mitigate the above negative relationship (fixed effects , , p < 10⁻⁴⁰, random effects SD = 0.0103). These results suggest that the normative interaction between policy complexity and the number of actions considered, as revealed by the simulations, reflects strongly in the reward and suboptimality patterns of human behavior in our task.

We then assessed the correlation between policy complexity and the number of actions chosen. As predicted, the Pearson correlation was positive and significant (R = 0.671, p<10⁻³⁰; Fig 5I). This correlation, however, was not strong enough to introduce multicollinearity problems in previous LME regresions (variance inflation factor (VIF)  = 1.82), such that their coefficient estimates are still trustworthy. One potential concern is that the number of actions chosen (N_a) places an upper bound on policy complexity ( bits), which raises the alternative explanation that the observed correlation simply reflects a uniform sampling of policy complexity within its allowable range. To address this, we performed post hoc Kolmogorov-Smirnov tests on the empirical distribution of policy complexity at each N_a. These tests revealed significant deviations from uniformity for N_a = 2,3,6,7 (p < 10⁻⁴). Moreover, visual inspection of the policy complexity distributions revealed a shift from right-skewness to left-skewness as N_a increased from 2 to 6 (S7 Fig). This pattern may explain the inability to reject uniformity for intermediate N_a values (N_a = 4,5), and supports the prediction that humans tend to increase both N_a and policy complexity across RT deadline conditions in an effort to reduce suboptimality.

We assessed our final hypothesis that participants set their policy complexity and number of actions chosen in a way related to their training block accuracy. As predicted, training accuracy correlated positively with mean test block policy complexity (R = 0.792, p < 10⁻¹⁶; S6E Fig) and mean test block number of actions chosen (R = 0.327, p = 0.00416; S6F Fig). We wondered if the ability to adjust policy complexity and number of actions chosen based on training accuracy is related to suboptimality, and hence performed additional post-hoc analyses on the N = 13 cluster of suboptimal subjects identified earlier in Fig 5A–5C. Subgroup analyses revealed that these subjects did not significantly adjust their number of actions chosen based on training accuracy (which is mostly low; R = 0.0740, p = 0.810), whereas the remaining N = 62 subjects did (R = 0.907, p < 10⁻⁴; Table A in S1 Appendix).

Connections to other frameworks.

The optimal policy derived from policy compression shares similarities with the policy prescribed by Kullback-Leibler (KL) regularized control [69], but there are key differences in how each framework handles action distributions and optimization constraints.

In KL-regularized control, a predefined default action distribution—typically uniform over actions—is assumed. Deviations from this distribution are penalized as a cost (negative reward), with β in Eq 2 interpreted as a cost sensitivity parameter. In contrast, our framework replaces this fixed distribution with a flexible marginal action distribution, P^*(a), and formulates the problem as a constrained optimization with , where β acts as a Lagrange multiplier.

A distinguishing feature of policy compression is the dependence of P^*(a) on the policy ). This property is central to its normative implications and behavioral predictions. Specifically, it predicts a tendency toward perseveration, where agents favor actions they have frequently chosen in the past—especially when policy complexity is low (β is small), making P^*(a) the dominant term in Eq 2. In contrast, KL-regularized control enforces policy regularization toward a fixed default distribution that is independent of the policy, not optimized for individual tasks, and unable to account for perseverative behavior that exploits task regularities.

Participants.

One-hundred-and-one participants (75 women, 25 men, 1 prefer not to say) were recruited. We selected the sample size based on the reaching of statistical significance in all planned analyses in a separate group of N = 30 pilot participants (data excluded from final analysis). All analyses were preregistered at https://aspredicted.org/s44z-xf4p.pdf unless otherwise noted. We excluded 26 participants for not responding within the response time (RT) deadline for more than or equal to 20 trials across all test blocks, leaving data from 75 participants (52 women, 23 men) for subsequent analyses. Participants were paid a base pay of $6 and a performance bonus of up to $2 for completing the task. Participants took, on average, 27 minutes to complete the entire experiment, and their average payout was $7.43.

Procedures.

Each participant completed three test blocks containing 96 trials each. On each trial, participants see an image and must press a key (action) before the RT deadline. There were six possible images (states) and seven available actions, which are shared across blocks. Each state was assigned a unique optimal action (one of the six number keys; the mapping is randomized across participants), while the remaining action (the letter key “E”) was a “safety” action that guaranteed a smaller reward across all stimuli (Fig 4A–4B). The three blocks featured RT deadlines of 0.5s, 1s, and 2s in randomized order, which are revealed to participants before starting each block. Participants were informed that the mapping from state to action was held fixed across all blocks, and that they would receive a bonus proportional to their summed performance across blocks.

Reward delivery was deterministic. The safety action that always yields +0.2 reward regardless of the state. The remaining six “unsafe” actions are uniquely optimal for each of the six states, yielding +1 or −0.18 reward according to the state identity. Failure to respond within the RT deadline resulted in a −1 reward penalty and automatic progression to the next trial. After making a response (or when the RT deadline arrived), participants were given immediate feedback for 0.5 s—a border around the image whose color matched the reward value (dark green +1, light green +0.2, pink −0.18, red −1). Then, the numeric reward value was displayed for another 0.5 s, before the next trial’s stimulus appeared. Participants could track the total reward earned during the block, displayed as a number above the image. At the end of each block, they were provided with feedback on the total reward they earned in that block (Fig 4C).

Before the three test blocks mentioned above, participants completed four training blocks, with each block lasting 60 trials. These training blocks are not analyzed. The first shared training block had RT deadline 2 s. In this shared training block, participants were provided feedback regarding the optimal action for the preceding stimulus, after their response and before the next trial began. The next three condition-specific training blocks had RT deadlines 2 s, 1 s, and 0.5 s, and no longer contained feedback on optimal actions just like the test blocks. These condition-specific training blocks help participants prepare themselves for the test blocks and develop their key-pressing strategy for each RT deadline. The optimal state-to-action mappings were revealed to participants before the shared training block, the first condition-specific training block, and the first test block.

Reward-complexity frontiers for different numbers of actions chosen.

The simplicity of the task structure allows us to exhaustively enumerate reward-complexity frontiers of different action consideration sets N_a, depending on their size and whether the safety action is included (Fig 4D–4E). The frontiers may end before the maximum policy complexity allowed by the action consideration set size at bits, as under the frontier’s corresponding action consideration set, further increasing policy complexity towards the maximum bound would only confer less trial-averaged reward. We consider the higher of the two reward-complexity frontiers associated with either including or excluding the safety action at each policy complexity level, and visualize it as the N_a-specific reward-complexity frontier (Fig 4F). The downward kink for N_a = 4,5 reflects the fact that the reward-complexity frontier associated with including the safety action extends farther towards higher policy complexity, but yields lower trial-averaged reward compared to excluding the safety action. In other words, the frontier after the downward kink reflects a suboptimal action consideration set choice at the same N_a level, but it still has a normative basis in being optimal under that suboptimal consideration set choice.

Estimating policy complexity and number of actions considered.

We defined policy complexity as the mutual information between the observed states and chosen actions. Following prior work [21,24,25], we estimated the policy complexity of each participant in each ITI condition using the Hutter estimator [70]. Specifically for each state, we assume a symmetric Dirichlet prior with for all actions chosen, and use the empirical action counts to reach a posterior Dirichlet distribution over action probabilities. We then estimate policy complexity as the mutual information of the posterior mean policy. The above procedure is informed by previous literature, reporting that the resulting estimates exhibit reasonably good performance when the joint distribution is sparse [71]. The choice of is informed by rate-distortion theory, stating that empirical trial-averaged reward values cannot be above the reward-complexity frontier. We have chosen empirically to obey this property.

To estimate the number of actions considered in a test block by a participant, we use the number of distinct actions chosen by that participant in that block (N_a). The estimator was chosen to best compare participant performance to the reward-complexity frontiers of corresponding action consideration set size—if a participant has chosen N_a = 4 distinct actions within a block, their performance is upper-bounded by the reward-complexity frontier corresponding to N_a = 4. However, to accommodate the possibility that participants brushed over unintended keys, we re-performed all analyses based on stricter thresholds, counting a chosen action into the action consideration set size N_a if and only if the participant has chosen the action for at least two or three times within the block (see Tables B and C in S1 Appendix). None of the conclusions change as a result of thresholding, which highlights the robustness of our findings.

Statistical analysis.

We fit linear mixed-effects (LME) models to study the relationship among behavioral variables of interest: policy complexity, the number of actions chosen, RT, trial-averaged reward, and loss in trial-averaged reward. We obtained parameter estimates using maximum likelihood estimation with the “fitlme” function in MATLAB R2023a. In the main text, we report the fitted coefficients and p-values for fixed effects of interest, as well as the standard deviation of their corresponding random effects.

We also computed Pearson correlation coefficients between pairs of behavioral variables: policy complexity, number of actions chosen, and training block accuracy rate. In the main text, we report the coefficients and their p-values.

Based on all our predictions, we have performed the following analyses:

1) PolicyComplexity ∼ RTDeadlineCond + (RTDeadlineCond|Participant);

2*) Reward ∼ RTDeadlineCond + (RTDeadlineCond|Participant);

3) P(a_safety) ∼ RTDeadlineCond + (RTDeadlineCond|Participant);

4) RT ∼ PolicyComplexity + (PolicyComplexity|Participant);

5) Na ∼ RTDeadlineCond + (RTDeadlineCond|Participant);

6) RT ∼ Na + (Na|Participant);

7) Reward ∼ PolicyComplexity*Na + (PolicyComplexity*Na|Participant);

8) RewardLoss ∼ PolicyComplexity*Na + (PolicyComplexity*Na|Participant);

9) Correlation: PolicyComplexity and Na;

10) Correlation: TrainAccuracy and PolicyComplexity;

11) Correlation: TrainAccuracy and Na;

12*) RT ∼ PolicyComplexity*Na +(PolicyComplexity*Na|Participant);

where the asterik () denotes post-hoc analyses. The above numeric indexing of statistical analyses is also used in S1 Appendix.