Feedback learning in CBGT networks efficiently increases reward rate

Suppose that an agent encounters a new environment for which it has no relevant prior experience or bias, so that the selection of all options is equally likely at first. In a simple two-choice bandit task, with one rewarded and one unrewarded option, this unbiased starting point would correspond to a 50% error rate. With learning it should be possible to make fewer errors over time, leading to increased rewards, but exactly how this is achieved in practice depends on the decision policy that the agent adopts. For example, if the agent prioritizes speed over all else in its action selection, then its error rate will likely remain high, leading to fewer rewards over time. Conversely, by making sufficiently slow decisions, the agent may be able to achieve an extremely low error rate, leading to greater likelihood of reward on individual trials, but if the response speed is too slow then the rate of reward return over time may take a significant hit. The overall reward rate achieved by the agent thus depends on both decision speed and accuracy. Intuitively, this may be optimized for a fixed level of experience via some compromise between these two dimensions [13,32].

To understand how optimized speed and accuracy emerge from CBGT circuits, we simulated 300 instances of a spiking computational model of CBGT pathways. To generate these instances, we started from parameter ranges defined in our past work [29] (with slight updates - see S2 Table) and used genetic algorithms (see Materials and Methods – Genetic algorithms) to derive a collection of networks that operate in a cortico-basal ganglia-thalamic driven regime (i.e., rather than bypassing the basal ganglia to have thalamic activity dictated by its cortical input), that have average firing rates of all relevant cell types within known biological ranges (S1 FigA), and that exhibit a heterogeneous range of response times (S1 FigB). The networks performed a two-armed bandit task with deterministic reward feedback (i.e., the reward probability was 100% for the optimal choice and 0% for the suboptimal one). Using a deterministic reward task, as opposed to a task where rewards are delivered probabilistically, explicitly ties accuracy to reward return and also makes the optimal learning strategy a simple “win-stay/lose-switch” policy [33]. Learning in the network was implemented with dopamine-dependent plasticity at the cortico-striatal synapses, where the magnitude of the phasic dopamine response following each decision was based on reward prediction error (for details see Section: Dopamine-dependent plasticity of corticostriatal synaptic weights and [34]). It should be noted that despite this being a deterministic task, there is ample variability in performance due to noisy background input provided to all populations in the network. Even after 15 trials of learning, by which point learning saturates (S4 Fig), the networks do not reach perfect performance, averaging ≈ 90% accuracy (S3 Fig).

Following a set of simulated trials in this task, we fit the reaction time (RT) and choice probabilities of each network, together reflecting the form of its decision policy, with a hierarchical version of the DDM [35,36]. The DDM provides an intuitive framework for mapping behavioral responses to an evidence-accumulation representation of the decision policy that can be described by only a few parameters [8]. Although there are many possible variants of the DDM that we could use, including versions with collapsing bounds [37–39] or trial-wise evolution of specific parameters [40], we did not include such extensions because our task does not involve factors like urgency or non-stationarity of task states. To try to capture changes in DDM parameters with learning, we fit the behavioral responses across the 15 learning trials with both the standard static DDM and the reinforcement learning (RL)-DDM variant – specifically the HDDMrl implementation from the HDDM package – which allows for drift rate to change during the learning process based on variations in relevant task or performance parameters. Both approaches yielded poor fits (S6 Fig), likely due to the small number of trials available before learning saturated. Thus, we instead opted for an alternative approach in which, after each predetermined step in learning (2, 4, 6, and 15 trials with plasticity on), we froze the network by turning off plasticity, simulated 300 trials to generate an RT distribution and choice probabilities from the current state of the network, and fit the static DDM and the RL-DDM to these behavioral measures; because performance of the static DDM was substantially better than that of RL-DDM (S6 FigA, S6 FigB), we henceforth focus on the static DDM results. After these probe trials, learning was turned back on and the task progressed. This process captured the behavioral data with high fidelity (S6 Fig) and allowed us to plot each network’s performance as a trajectory in the DDM parameter space.

Because of the mapping from network behavioral responses to DDM parameters, we will refer to the 2-dimensional plane of drift rates (v) and boundary heights (a) as a decision policy manifold. Fig 2 shows the average trajectories of three groups of networks on the (v,a) decision policy manifold. For each v and a we also estimated the average RT (Fig 2A), accuracy (Fig 2B) and reward rate (Fig 2C; see also [14]). The three network groups shown in this figure represent a tertile-based split of the full set of simulated networks into fast (short RT, orange), intermediate (medium RT, brown), and slow (long RT, red) groups, based on their initial RT values (S1 FigB). We implemented this split to determine whether decision policy adjustments due to learning were influenced by initial biases in the networks. Despite their initial speed differences, all three network groups showed chance level performance before plasticity (S1 FigC) and converged to similar regions of the (v,a) space with learning (Fig 2, shaded ellipses). A comparison of behavioral measures and DDM parameters before and after plasticity is presented in S3 Fig.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Dopamine-dependent cortico-striatal plasticity drives CBGT networks in the direction of reward rate maximization.

(A) The evolution of RTs achieved by a DDM fit to CBGT network behavior, projected to (v,a)-space. The average starting position for the fast (orange), intermediate (brown) and slow (red) networks are shown as stars. The squares indicate the evolution of each network group over the plasticity stages, which converge after 15 trials (shaded elliptical regions). The yellow (purple) colors represent high (low) RTs. The network trajectories do not evolve in the direction that would be expected to minimize the RTs (e.g., optimal direction shown in blue from the initial position of all three speed groups). (B) The yellow (purple) colors represent high (low) accuracy. The networks evolve towards increasing expected accuracy but not in an optimal fashion (trajectories vs. blue arrows). (C) The yellow (purple) colors represent high (low) reward rate. The network evolution aligns closely with the direction that maximizes the reward rate (blue arrows). (D) The cosine distances calculated for every network at each plasticity stage for RT, accuracy and reward rate are were pooled together and shown as distributions.

https://doi.org/10.1371/journal.pcbi.1013712.g002

These trajectories clearly demonstrate that our CBGT network can learn from simple dopaminergic feedback at the cortico-striatal synapses. But what exactly is the objective being optimized by the network? To address this question, we compared the change at each step of learning to the predicted direction that the network would take if it were maximizing one of the three possible behavioral objectives: speed, accuracy, or reward rate. Note that we can plot contours for each of these quantities (using RT as a gauge of speed) over the (v,a) domain. Although the mapping between (v,a) and either speed, accuracy, or reward rate is not bijective, we will nonetheless refer to the (v,a) plane as the speed manifold, accuracy manifold, or reward rate manifold when it is shown along with the contours of the corresponding quantity. The predicted directions of objective maximization are illustrated as blue vectors in Fig 2A–2C, reflecting steps from each initial point that are in the direction of the gradient of each objective (i.e., the direction of maximal change, which lies orthogonal to the contours, shown with the same length as the vector representing the actual network evolution at the first step of learning in each case). Analysis of the trajectories in Fig 2A reveals that while plasticity decreases RTs with learning, the angles of the learning trajectories do not align with the optimal directions for maximally reducing RT. Similarly, the network trajectories do not align with the vectors that would be expected if they were maximizing accuracy alone (Fig 2B). In contrast, the average trajectories along the reward rate manifold were closest to the gradient and hence to the optimal trajectories attainable for that manifold (Fig 2C). Moreover, the rate of increase in reward rate was similar regardless of the network’s initial speed bias.

To quantify the alignment of observed network trajectories to the expected directions of maximal change, we calculated the cosine distance between the observed vector and the optimal vector, normalized to the observed vector’s length, at each learning step. While there is substantial variability across networks (Fig 2D), there was a consistent effect of objective type on the network fits (F[3597, 2]=47.2, p < 0.0001). Fits to the reward rate trajectories were consistently better than to either speed (t(299)=13.59, p < 0.0001) or accuracy (t(299)=8.35, p < 0.0001) trajectories. This effect held regardless of a network’s initial speed (S4 Fig). This effect also held when the network was trained on non-deterministic reward probabilities (S5 Fig), except for high reward uncertainty such as 75%, where reward rate and accuracy fits approximately equalized (see Discussion).

Overall, our DDM results show that our biologically detailed model of the CBGT circuit can effectively learn to increase reward rate by managing the accuracy and speed of the evidence accumulation process via dopaminergic plasticity at the cortico-striatal synapses. Moreover, this increase is near-maximal when reward probabilities are deterministic.

Low-dimensional control ensembles that map to general decision policies

The CBGT network and DDM are, respectively, implementation-level and algorithmic-level descriptions of the evidence accumulation process that guides goal-directed behavior. We have previously shown that there is a low-dimensional, multivariate mapping between these two levels of analysis in the absence of learning [29]. Here we set out to replicate this observation with the CBGT parameter sets used in the current study, with the aim of analyzing their contributions to the dopaminergic learning process. For this step, we considered two aspects of activity within each CBGT population: global activation across the two action representations (sum of the activity in that region, across both channels; Σ) and bias towards one action representation (difference in activity within each region, across the action channels; Δ). Using canonical correlation analysis (CCA), we captured the low-dimensional components that maximally correlate variation in CBGT activity with variation in DDM parameters. This analysis identified three such components (Fig 3). We refer to these low-dimensional components as control ensembles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Canonical correlation analysis (CCA) identifies control ensembles (cf. [29]).

Given matrices of average firing rates, F (both summed rates across channels, Σ, and between-channel differences, Δ), and fit DDM parameters, D, derived from a set of networks at baseline (left panels), CCA finds the low-dimensional projections, for firing rates and for DDM parameters (right panels), which maximize the correlation, ρ, between the projections and of F and D. Blue lines in the F plot show left channel activity, orange show right channel activity, and green shows populations that go across both channels.

https://doi.org/10.1371/journal.pcbi.1013712.g003

The three control ensembles identified by our analysis nearly perfectly replicate our prior work [29], where they are described in more detail (see also Section Upward mapping). Thus we kept the labels responsiveness, pliancy, and choice ensembles for the first, second, and third components, respectively. The recovered components are shown in both CBGT and DDM parameter spaces in Fig 3 (right panels). The responsiveness component describes the agent’s sensitivity to evidence, both in terms of the delay before the agent starts to accumulate evidence (t) and how significantly the presence of evidence contributes to achieving the decision threshold (a). The dominant features of CBGT activity that vary along the responsiveness control ensemble loadings are a global inhibitory signal, including fast-spiking interneuron (FSI) and overall internal globus pallidus (GPi(Σ)) activity, as well as overall excitatory and inhibitory cortical activity (Cx(Σ), CxI). Because the dominant CBGT and DDM loadings for the responsiveness control ensemble have the same sign (all negative), they imply that a decrease in the weighted activity of the loaded populations corresponds to an decrease in onset time, t, and a and, hence, to an increase in overall responsiveness.

The pliancy component refers to the level of evidence that must be accumulated before committing to a decision. As with responsiveness, pliancy loads mostly on a and t, but now with opposing signs for these two loadings, corresponding to the idea that even though an agent is attentive to evidence (small t), it requires a substantial accumulation of evidence to reach its threshold (large a). The CBGT activity features that characterize pliancy are the overall engagement of the BG input nodes (i.e., global dSPN and iSPN activity, with a smaller STN contribution), as well as total GPi and thalamic activity, with oppositely signed loadings to each other. For the pliancy component, a change in the activity consistent with the cell type loadings (e.g., increase in SPN activity) corresponds to a decrease in overall pliancy (e.g., increase in a).

Lastly, the choice component represents the intensity of the choice preference and is reflected largely in v and the neural correlates of competing choice representations in the CBGT (i.e., differences in activity across the two action channels within each BG region). A change in activity consistent with the cell type loadings (e.g., greater difference in dSPN activity between the two channels) corresponds to a stronger commitment towards the more rewarded option (i.e., larger v).

In summary, each CBGT control ensemble can be interpreted as specifying a coordinated collection of changes in CBGT neural activity that modulates specific aspects of the decision policy, as represented by the DDM parameters needed to fit that behavior. Now that we have delineated the control ensembles embedded within the CBGT network (cf. [29]), we are ready to consider how dopamine-dependent plasticity regulates their influence in a way that collectively drives decision policies to maximally increase reward rate.

Cortico-striatal plasticity drives control ensembles during learning

Our analysis of the CBGT network behavior (Fig 2) shows that dopamine signaling at the cortico-striatal synapses is enough to elicit changes in the evidence accumulation process that increases reward rate. This observation suggests that there are emergent driver mechanisms, originating from cortico-striatal synaptic changes, that tune the control ensembles in a way that achieves this outcome. That is, if each control ensemble represents a knob to tune an aspect of the decision policy, then a driver mechanism selects a set of adjustments of the knobs that yields an overall decision policy selection. We next set out to identify these emergent drivers.

As a first step toward quantifying the modulation of CBGT activity after 15 learning trials, we calculated the principal components of the overall change in firing rates of all 300 networks. The first 5 of these components collectively explain more than 90% of the observed variance (S7 FigA, thick blue line marked “All"). The loading weights (Fig 4A) show that the first and third components reflect the global activity of subsets of CBGT nuclei. The second, fourth and fifth components relate more strongly to the bias towards one option, with predominant loadings on differences in rates across action channels. Together, these components represent the collection of changes in firing rates that result from learning-related changes at the cortico-striatal synapses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Plasticity-induced changes of control ensemble influence.

(A) The loading weights of the first 5 PCs of firing rate changes from before to after plasticity, pooled for all networks. (B) The drivers (columns of S), which quantify the modulations of control ensembles (responsiveness, pliancy, choice) that capture each PC (pooled for all network classes). (C): The variance-weighted drivers for the three control ensembles, computed separately for the three network classes (fast, intermediate and slow).

https://doi.org/10.1371/journal.pcbi.1013712.g004

We next calculated the matrix S of weighting factors (drivers) for the firing rate components, which describe what combination of adjustments to the control ensembles best account for the associated firing rate changes (Fig 4B; for full description of this approach, see Methods subsection Modulation of control ensembles by plasticity). To interpret the drivers of control ensemble influence (Fig 4B), it is important to note that positive (negative) coefficients correspond to changes in control ensemble activity in the same (opposite) direction as indicated by the loadings in Fig 3. The first driver corresponds to a large amplification of the responsiveness control ensemble, and hence a decrease in various forms of global inhibition in the CBGT network. The first driver is also associated with a boost to the choice control ensemble, increasing the bias towards the rewarded choice. The second driver has a strong negative weight on the choice control ensemble and a positive weight on the pliancy control ensemble. The third, fourth and fifth drivers feature weaker effects, with small modulations of all three control ensembles. Based on this analysis across all of the networks, the overall modulation of the control ensembles due to plasticity, calculated as the weighted sum over all drivers, adjusted by the % of variance explained by each PC, is shown in S7 FigB. All three control ensembles end up being boosted. This means that, to varying extents, the activity measures that comprise these ensembles change in the directions indicated by their loadings in Fig 3. In this way the general trend is for the CBGT networks to become more responsive, yet less pliant, which together amount to an earlier onset of evidence accumulation without much change in boundary height. Coincident with this, we also see that the CBGT networks exhibit more of an emergent choice bias with learning.

Because of the difference in decision policies across the fast, intermediate, and slow networks, we recomputed the drivers separately for each network type. This was done by considering the firing rate differences () and calculating the S loadings for fast, intermediate, and slow networks separately (see Methods - section Modulation of control ensembles by plasticity). The explained variance for each of the three network types is shown in S7 FigA, and their corresponding PCs and goodness of fits are shown in S8 Fig. As expected, the drivers show different changes across the network types (Fig 4C). The driving factor corresponding to responsiveness is negative for fast networks, while remaining positive for the others. The pliancy and choice factors were positive for all three networks, but pliancy was by far the largest for fast networks and quite small for the other two network types. Referring to the DDM parameter changes associated with changes in control ensemble loadings (Fig 3), we see that the decrease in loading of responsiveness and strong increase in loading of pliancy for fast networks would both promote an increase in boundary height, a. This aligns with the fact that, of the three network types, only fast networks show an increase in a over the course of learning (Figs 2 and S9 Fig). The DDM parameters exhibit saturation after approximately 15 learning trials, as shown in S9 FigA–S9 FigC, as a result of the maximum bound imposed on the corticostriatal weights, shown in S3 Table. The upper limit on the corticostriatal weights is the result of the multiplicative formulation of weight update (Eq 2 in section Dopamine dependent plasticity of corticostriatal weights). This ensures that the weight updates are smaller as the weights approach certain maximum or minimum bounds. Multiplicative update rules are widely used in plasticity models and have been shown to produce unimodal weight distributions and stable learning dynamics [41,42]. Overall, we see that the specific way that plasticity adjusts the weighting of the control ensembles to drive changes in decision policies depends on the initial tuning of the network. Since plasticity results from the sequence of decisions and rewards that occur during learning, we next investigate more directly how specific decision outcomes lead to this dependency.

The influence of feedback sequences on driving of control ensembles

In the previous section we described the overall effects of cortico-striatal plasticity on control ensemble tuning. We now turn to analyzing the early temporal evolution of these effects by focusing on the initial two learning trials. Specifically, we examined the modulation of the control ensembles for different combinations of successes (i.e., rewarded trials; R) and failures (i.e., unrewarded trials; U) achieved by the first two consecutive choices. For this analysis, we implemented our usual DDM fitting process followed by CCA for networks that were frozen (i.e., with plasticity switched off) after two trials, and we grouped the results based on the sequence of choice outcomes. The drivers for each sequence of outcomes, U-U, U-R, R-U and R-R, are the linear combinations of columns of S that express the corresponding changes in firing rates in terms of adjustments to the control ensembles (Fig 5A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Suboptimal and optimal choices modulate control ensembles in opposite directions.

(A) The modulation of control ensembles associated with various reward sequences encountered in two initial trials with cortico-striatal plasticity. U represents “Unrewarded" and R represents “Rewarded" trials. (B) The reward rate changes obtained by simulation of networks with synaptic weights frozen after various reward sequences occurred on two initial trials.

https://doi.org/10.1371/journal.pcbi.1013712.g005

First, consider the case of networks that receive no rewards (U-U). Here we infer that the boundary height increases, due to a simultaneous decrease in driving of the responsiveness ensemble and increase in driving of the pliancy ensemble, both of which result in a boost of the boundary height. In addition, driving of the choice ensemble is reduced. Thus, two consecutive unsuccessful trials yield an overall increase in the degree of evidence needed to make a subsequent decision by simultaneously increasing the boundary height and decreasing the drift rate. Moreover, slow networks encounter U-U outcomes more often than other network classes in the first two trials (S1 Table), which presumably constrains the increase in responsiveness and choice seen in these networks during learning (Fig 4C). On average, however, fast networks make more mistakes than the other networks. This result, which we can display graphically in terms of the proportion of unrewarded trials, or mistakes, encountered after the first two plasticity trials (S9 FigD), likely explains the negative loading for responsiveness and high positive loading for pliancy for fast networks shown in Fig 4C.

In contrast, two consecutive successful trials (R-R, far right of Fig 5A) produce largely the opposite effect. The influences of the responsiveness and choice ensembles increase, resulting in lower onset time and boundary height, along with an increase in the drift rate. This coincides with a weak change in pliancy. As a result, in the R-R case, the decision policy is tuned to include a decreased degree of evidence needed to make subsequent decisions.

Not surprisingly, the two mixed combinations of outcomes (U-R, R-U) have largely similar effects on the responsiveness and pliancy ensembles, regardless of the order of outcomes. In both cases the loading of responsiveness increases and that of pliancy decreases, resulting in less overall evidence needed to trigger a decision (by shrinking the boundary height, without much change in the onset time). However, when the first trial is unsuccessful (U-R) the influence of the choice ensemble decreases, while it increases when the first trial is successful (R-U). Indeed, looking at the progressive change in the choice ensemble across the four unique sequences of trials, it appears that early success (i.e., reward in the first trial) boosts the choice ensemble influence while early failure (i.e., unrewarded first trial) does the opposite. When these combined drivers are recomputed separately for each network class, the learning-induced modulations of the ensembles follow the same general trend (S10 Fig), with quantitative details depending on the network class.

The preceding analysis shows how the relative contributions of the control ensembles to the evidence accumulation process depend on trial outcomes. What are the results of these changes on the performance of the network? To illustrate these effects, we plot the distribution of changes in reward rates associated with each set of outcomes and separate by network types in Fig 5B. Although all distributions are generally positive, there is significant variation in reward rate changes across the different feedback sequences (F(586 , 3) = 254.4, p < 0.0001). The reward rate also varies significantly with the network type (F(586, 2) = 46.8, p < 0.0001), and the interaction term between network types and feedback sequences is significant as well (F(586, 6)=3.8, p = 0.001). Compared to all other conditions, the networks that made two consecutive unsuccessful choices (U-U) yielded the smallest changes in reward rates (values of all network types pooled together, all two-sample t(319) < -18.27, all p < 0.0001). The two mixed feedback conditions (U-R, R-U) had higher growth in reward rates than the condition with two rewarded trials (R-R; all t(384) > 8.1, all p < 0.001), because mixed conditions not only lead to strengthening of the correct choice but also weakening of the incorrect choice, unlike R-R which only leads to the former. In all cases, the trend was for faster networks to achieve greater increases in reward rate than slower networks. As expected, the impact of feedback sequences on reward rate is associated with underlying changes in both accuracy (S11 FigA) and decision speed (S11 FigB). Like reward rate, the increase in accuracy was highest for the mixed feedback conditions (U-R, R-U) due to the combined strengthening of the correct choice and weakening of the incorrect choice. Two consecutive unsuccessful choices (U-U) represents the only condition that leads to an increase in decision times, expressed as negative ΔRTs. This outcome is consistent with the increase in boundary height that occurs in this case, whereas all other feedback conditions lead to a decrease in decision times.

CBGT network

The CBGT model used in this work is a biologically constrained spiking network including neuronal populations from the striatum (dSPNs, iSPNs and FSIs), globus pallidus external segment (GPe), subthalamic nucleus (STN), globus pallidus internal segment (GPi), thalamus and cortex (excitatory and inhibitory components). For a two-choice task, each choice representation is implemented as a “channel” [23,28,29,34,75], so the model includes two populations of each neuron type except FSIs and inhibitory cortical neurons, which are shared across channels. On each trial, the two excitatory cortical populations receive excitatory synaptic inputs, representing evidence related to the available options, from a stochastic spike generator. This process has a baseline rate sampled from a normal distribution with a mean and standard deviation of 2.5 Hz and 0.06, respectively. To this baseline, we add a ramping component representing the presence of some stimulus or internal process that drives the consideration of possible choices. This component rises linearly until it reaches a maximum value (), which was kept constant for all simulations in order to appropriately compare decision times. Specifically, we take

where dt is the integrator time step, such that the total frequency of inputs to the cortical populations evolves according to

In all of our simulations, evidence for the two options, as represented by this frequency of inputs to the two cortical populations, was equally strong, such that changes in outcomes across conditions resulted entirely from learning downstream from the cortex.

All populations have an external current I_ext to tune their baseline firing rate, given by

where S_ext,x for is a mean-reverting random walk derived from the stochastic differential equation

Here, W_t is a Wiener process, is the time decay of the external current, and and are computed as

The parameter f_ext,x is the external input frequency, E_ext,x is the mean efficacy of the external connections, N_ext,x is the number of connections, and is the time decay constant. Values of all of these parameters are specified in S4 Table.

Specifically, the cortico-striatal projections to both dSPNs and iSPNs in the model were plastic and were modulated by a dopamine-dependent spike timing dependent plasticity rule [31,76,77]. On a trial, a choice was selected if the firing rate in the thalamic population within its action channel reached 30 Hz before the rate of the other thalamic population hit that level. The complete details of this network can be found in our methods paper [34].

Characterization of networks before plasticity

In our previous work, we identified control ensembles based on extensive simulation of the CBGT network with each of 300 parameter sets selected using Latin hypercube sampling from among the ranges of synaptic weights that maintained biologically realistic firing rates across all populations [29]. In that work, in which no learning occurred, however, the cortico-striatal projections to the action channels were considered to be independent, and variability in decisions was introduced by directly and independently biasing the corticostriatal weights associated with each option, resulting in networks that were inherently biased towards one of the choices.

Hence, some sampled network configurations were biased towards one of the choices. Because we studied the evolution of the control ensembles under plasticity in this work, to ensure that any effects observed during learning were not confounded by pre-existing biases, we started with completely unbiased networks. Hence we resampled the networks from the joint synaptic weight distribution using genetic algorithms (see below) and isolated 300 networks that produced firing rates of all CBGT populations within experimentally observed ranges. The comparison of the ranges for free parameters in [29] and the present work are listed in S2 Table. The actual firing rate distributions are shown in S1 FigA. The networks before plasticity showed a diversity of reaction times (RTs, S1 FigB). The RT distribution was divided into 3 equal tertiles and used to define “fast" (orange), “intermediate" (brown) and “slow" (red) networks. All of the networks before plasticity showed chance levels of accuracy (S1 FigC).

Dopamine-dependent plasticity of corticostriatal synaptic weights

Synaptic plasticity in CBGTPy is implemented using a dopamine-dependent plasticity rule, where the synaptic updates are governed solely by local factors, without requiring individual neurons to access information about the global system state. This rule is an adaptation of the plasticity mechanism presented in [31,34].

At each corticostriatal AMPA synapse, the model tracks three key values: eligibility E(t), weight w(t), and conductance g_x(t). The conductance is associated with the synaptic current. How much the conductance grows with each pre-synaptic spike is determined by the weight. The weight is the plastic element in the system, which changes over time depending on the time courses of eligibility and dopamine release.

At a computational level, E(t), representing a synapse’s eligibility to undergo weight modification, depends on the relative spike times of the pre- and post-synaptic neurons involved in the synapse. To compute this quantity, we first define the variables A_PRE(t) and A_POST(t), which serve as instantaneous estimates of the recent levels of pre- and post-synaptic spiking, respectively. Each time a spike occurs in the pre- or post-synaptic cell, these values are increased by a fixed amount ( and , respectively), and between spikes, they decay exponentially with a time decay constant and , respectively. That is,

where X_PRE(t) and X_POST(t) are sums of Dirac delta functions representing the spike trains of the two neurons. These sums take the form

where t_s is the spike onset time, is the set of times of all spikes of the cortical neuron involved in the synapse, and refers to the set of spike times of the target postsynaptic neuron within the striatum. Eligibility (E(t)) changes over time according to

(1)

where is a time constant. The corticostriatal synaptic conductance g_x takes the value of the synaptic weight, w(t), at each pre-synaptic spike time and decays exponentially in-between these spikes:

where x labels the specific connection, is the decay time constant associated with AMPA synapses, and w itself changes over time based on dopamine release and the post-synaptic neuron’s eligibility. The evolution of w is given by

(2)

where () represents a function whose output is the value inside the brackets if it is positive (negative) and 0 otherwise. The learning rate is denoted in Eq (2) by , for , depending on to which of the two populations the post-synaptic neuron belongs. This rate has a positive sign for dSPN neurons and a negative one for iSPN neurons to reproduce the observation that positive feedback signals lead to a strengthening of the eligible direct pathway connections and a weakening of the eligible indirect pathway connections. Furthermore, and are upper and lower bounds for the weight w, respectively, for .

In Eq (2), the variable K_DA represents the level of available dopamine in the network, which is computed from the amount of dopamine released via the differential equation

where the increment of dopamine, relative to a baseline level, that is delivered at time t_j. That is, after a specific decision i is made at time t_j, a reward value associated to action i is received, which induces a dopamine increment based on the reward prediction error

where is the expected reward for action i at time t_j. This expected reward obeys the update rule

where is the learning rate for action values and . The update of DA_inc in turn impacts the evolution of K_DA. Finally, the function f(K_DA) in Eq (2) represents the impact that the available dopamine K_DA has on plasticity, such that, if the target neuron lies in the dSPN population, then

while if the target neuron lies in the iSPN population, then

for fixed, positive scaling parameters . Parameters values used for the plasticity implementation can be found in S3 Table

Genetic algorithms

We use genetic algorithms to sample a high-dimensional (14-d) space of parameters and choose a collection of networks that operate in a cortico-basal ganglia-thalamic driven regime and exhibit a heterogeneous range of response times. To this end, we used the DEAP library [78] to run a genetic algorithm (GA) designed to sample CBGT networks with parameters from the ranges used in our previous work [29]. Two additional criteria were used for the optimization function of the GA, namely (a) the network should produce trial timeouts (when no action was selected within 1000 ms) on fewer than 1% of trials, and (b) the network should be cortico-basal-ganglia driven; that is, the correlation between cortical activity and striatal activity should be positive. The first criterion ensured that we had ample decision trials included in the data, as needed to appropriately fit the DDM parameters (timeouts are dropped before fitting the DDM parameters). The second criterion ensured that the networks did not operate in a cortico-thalamic driven regime, in which cortical inputs alone directly pushed thalamic firing over the decision threshold.

The range for each parameter specified in past work [29] was divided into 30 bins and this grid was sampled to create populations. The indices of each bin served as a pointer to the actual values of the parameters in the ranges considered. The GA uses these indices to create, mate and mutate the populations. This ensures that the values of parameters remain within their specified ranges. For example, suppose that parameter A has range (–2.0,2.0) and parameter B has range (–0.3,1.0) and these ranges are each divided into 5 bins. The grids for parameters A and B would then be:

If individual population members have indices and for (A,B), then they have and , respectively. Supposed that the individuals mate by crossing over the 1st and 2nd elements. Then with parameter values (2,0.025) and with parameter values (–2,–0.3). The individuals ind₃ and ind₄ are included in the next iteration of evolution.

New individuals created from mating were used to overwrite the original individuals that were mated together (cxSimulatedBinary). The individuals could also mutate by shuffling of the indices of the attributes (mutShuffleIndexes) with a probability of 0.2. After a round of mating and mutation, tuples of two values for each individual, namely the % of timeouts and the Pearson’s correlation coefficient between cortical and striatal activity, were compared to select the individuals for the next round of evolution. The selection algorithm that was used was tournament selection (selTournament) of size 3, which picked the best individual among 3 randomly chosen individuals, 10 times, where 10 is the size of the population of networks in every iteration of the GA. During every iteration, any network configuration that met the criteria (a) and (b) above was saved as a correct solution. The GA was run for 2000 iterations or until 300 solutions were found, whichever was sooner. Post hoc, we confirmed that the firing rates of the members of the final, selected populations remained within the originally targeted ranges (S1 Fig).

DDM fits

A drift diffusion model (DDM) as shown in Fig 3 assumes decision to be a noisy random walk with a drift rate (v) towards one of the boundaries that represent the two choices. The difference between the two boundary heights (a) represents the decision threshold required to commit to a decision. The amount of time required for the decision variable to reach one of the boundaries constitute the reaction times (RT). The DDM also has additional parameters, non decision time (t), that may correspond to the sensory processing time or other processes not related to the decision time and the inherent bias (z) towards one of the choices.

where x(t) represents the noisy decision variable, v represents drift rate, σ represents the level of noise in the process, W(t) represents a Weiner process and was drawn from a normal distribution with mean zero and variance dt. The process x(t) terminated at the minimum time t, when x(t) reaches one of the boundaries (i.e or ).

The DDM parameters were fit to each of the 300 selected networks independently using the HDDM package [35]. In order to ensure that the HDDM fits describe the choice and RT distributions well, we compared the post-predictive distributions of the network simulations with those generated by the corresponding DDM parameters, both before (S2 FigA) and after (S2 FigC) plasticity. The quantile-quantile plots for percentiles 5, 10, ,90, 95 show a significant and very high correlation between the network-generated and DDM-generated data (S2 FigB and S2 FigD).

Accuracy, RT and reward rate manifolds

The manifolds shown in Fig 2A–2C were generated by simulating the DDM with all combinations of drift rate (v) and boundary height (a) values that the naive CBGT networks can show before and after plasticity. The values of RTs, accuracy and reward rates were averaged over 15 seeds of 200 trials each.

Upward mapping

The DDM parameters and activity of the CBGT nuclei for our 300 network configurations, before plasticity, were used to identify CBGT control ensembles through canonical correlation analysis (CCA), as was also done in our previous work [29] and is illustrated in Fig 3. The CCA identifies independent components in two sets of variables such that the correlation between the projections of the data onto these components is maximized. It is used to uncover linear combinations of features that have the strongest correlations with each other. In our previous work [29], we applied CCA to link behavior (captured by DDM parameters) and CBGT network activity (described by the sums and differences of activity levels in corresponding populations across channels). This analysis identified “control ensembles": linear combinations of network features that modulate decisions in a two-choice task. Both our previous and current studies consistently reveal three key control ensembles, corresponding to linear combinations that we term responsiveness, pliancy, and choice. These are described in more detail in the section Low-dimensional control ensembles that map to general decision policies.

The CCA scores were calculated using k-fold validation (k=4), where the 300 networks were divided into groups of 4 (75 networks each) and a CCA score was calculated for each of the groups. The CCA scores for actual data were compared with a shuffled version of data (firing rates and DDM components for 300 networks) and the set of components giving rise to the maximum CCA score, which we found to include three elements as in our previous work [29], were selected.

Modulation of control ensembles by plasticity

We used a single approach to compute a set of effective drivers of the control ensembles either from the full collection of CBGT networks or from one of the network subtypes (fast, intermediate, or slow) that we considered. Let denote the class of networks being used. From the set of vectors of changes in CBGT firing rates computed by subtracting firing rates before plasticity from those after plasticity (), we extracted 5 principal components (PCs) that together explain at least about 90% of the variance (Fig 4A and S7 FigA). was then projected onto these 5 PCs to form the target matrix P_X. Specifically, we computed

(3)

where the 5 PCs comprise the columns of . Note that P_X is an n by 5 matrix, where n is the number of firing rate data vectors used. was also projected onto the three control ensemble components obtained from the full collection of baseline networks before plasticity, via the mapping

(4)

where the components of the 3 control ensembles form the columns of U, such that C_X is an n by 3 matrix. Finally, we found the least squares solution S_X, representing the element in the range of C_X that is closest to P_X, from the normal equation

(5)

The least squares solution S_X is a matrix independent of n. The columns of are displayed in Fig 4B. The sums of the columns of the appropriate S_X, each weighted by the percent of variance explained, comprise Figs 5C and S10 Fig (, , and ), as well as Figs 5A and S7 FigB ().

Reward rates

The reward rate was calculated as:

where p(err) denotes the error rate and where the reaction time, RT, is the sum of the decision time, DT, and the additional non-decision time that arises within each trial, T₀, which in our analysis is ascribed to the onset delay represented by the DDM parameter t.

Plasticity stages

The effect of plasticity on the network was studied at four stages: (a) after 2 trials of plasticity, (b) after 2 additional trials (total 4) of plasticity, (c) after 2 more additional trials (total 6) of plasticity, (d) after 9 additional trials (total 15) of plasticity. The state of the network was frozen at each of these stages by suspending the plasticity, so that we could use the frozen network to perform probe trials. The choices and reaction times from the probe trials were used to calculate DDM parameters and reward rate distributions for each stage of plasticity, based on upward mapping and CCA, and thus to generate the trajectories in Fig 2, the time courses in S9 Fig, and the 2-trial results in Figs 5, S10 Fig, and S11 Fig.