Neuron-astrocyte interactions constitute a hypernetwork with multi-scale dynamics

We proceed to develop a reduced model of neuron-astrocyte interaction that captures key aspects of neurobiology while enabling fundamental analysis regarding dynamical expressiveness and links to function.

Neuron-astrocyte structure as a hypernetwork.

Classically, biological interactions between neurons, astrocytes, and synapses have been conceptualized in terms of the tripartite synapse structure [11, 50, 51] (as shown in Fig 1A). Within this framework, astrocytes interact with neurons at synapses, modulating synaptic efficacy [52] and controlling synaptic plasticity [53]. Such interactions may occur in a higher-order and ‘closed-loop’ fashion, wherein astrocytes respond to neurotransmitters released during pre- and post-synaptic neuronal activity (see Section A in S1 Appendix for detailed description) and this has been the mainstream assumption in past work attempting to model astrocytes. While this description may capture an important dimension of neuron-astrocyte interaction, it is increasingly clear that astrocytic modulation of neuronal activity is more general and multifaceted. The contextual guidance hypothesis [8] espouses that astrocytes regulate synaptic activity not only in response to synaptic activity itself, but also as an adaptative response to external drives, such as vigilance state, sensory salience, metabolic load, or underlying pathology (see Fig 1A). As such, astrocytes may actively ‘control’ neural dynamics in a state-dependent manner [34, 35]. While we focus here on effects at the synapse via the release of astrocyte-derived neuroactive transmitters, in principle astrocyte modulation can also occur at cell bodies via the alteration of ionic conditions, notably potassium levels [6, 19, 54–56].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

A. In a tripartite synapse, the presynaptic axon and postsynaptic dendrite are surrounded by an astrocyte [11, 50, 51], enabling multifaceted effects of neurotransmitters and (astrocyte-derived) gasotransmitters. B. A graphical illustration of the neuron-astrocyte hypernetwork: the circles and stars represent neurons and astrocytes respectively; the colored triangles denote the hyperedges and represent the multiplexed intralayer interactions. C. Schematic representation of the feedback interconnections between subsystems in the multi-scale neuron-astrocyte network model.

https://doi.org/10.1371/journal.pcbi.1012186.g001

The above schema of neuron-astrocyte interactions is difficult to capture as a traditional graphical network representation. As a result, we introduce the framework of a hypernetwork to describe the neuron-astrocyte architecture (see Fig 1B for the illustration and detailed description in Section B in S1 Appendix). We distinguish neurons and astrocytes by representing them on two different layers of the network. The interlayer relationships are all hyperedges, which embody the ability of astrocytes to modulate neuronal synaptic activity and therefore neuronal activity indirectly.

Astrocytic modulation acts as a pseudo-bifurcation parameter that changes meta-plasticity and neural circuit dynamics.

To analyze the dynamics of (1), we reduce it to its simplest motif, i.e., the interaction of two neurons and a single astrocyte. Here, we assume that the neurons form a reciprocal excitatory-inhibitory loop, itself a common canonical motif for cortical interactions between pyramidal and inter-neurons. From (1), the neuron-astrocyte motif amounts to a set of 5 ODEs: (2) The dynamics of this system are asymptotically bounded (see Section D in S1 Appendix). Within this bounded set, the motif may exhibit a unique fixed point, or multiple fixed points, depending on parameterization. Fig 2B and 2C show the case of three fixed points under the assumption that astrocytes evolve at a time-scale two orders of magnitude slower than neurons and synapses (i.e., τ₃ = 100τ₁, τ₂). Fig 2D illustrates the time evolution of a specific trajectory within this landscape. As expected, z evolves much slower than the other variables. Notably, this slowly-changing astrocytic activity variable seems to drive neural variables to transit between nearly stationary regimes corresponding to phases 1 and 3 of astrocytic states in the lower plot of Fig 2D, suggesting that astrocytes can systematically ‘control’ stationary neural activity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Neuron-astrocyte network motifs and dynamic properties.

A. Graphical representation of the network motif; u₁, u₂, v include inputs from other nodes of the hypernetwork as well as those from external sources. B, C. Several examples of trajectories in the state space (x₂, w₂, z) of the network motif system, where the boxes show the starting points. The parameter conditions are a₁ = 0.7, a₂ = 0.6, b₁ = 1.6, b₂ = 1.7, c₁ = 12, c₂ = −10, d₁ = −4, d₂ = 5, ϵ = 0.6, h = 6, and τ₁ = τ₂ = 0.01, τ₃ = 1. The system has three fixed point points, of which two are stable (red dots) and one is unstable (blue dot). The system dynamics converge to these two stable fixed points. D. Trajectory associated with the thick phase curve from B, C. illustrating two stationary regimes (indicated by phases 1 and 3 in the lower plot). E. depicts the bifurcation diagram of the neural dynamics with respect to the astrocyte output ψ(z), where the red curve shows that one branch of fixed point always exists, while the blue curve shows how the other branch of fixed points changes via the saddle-node bifurcation. F, G. Vector fields of the neuron-synaptic dynamics to either side of the saddle-node bifurcation.

https://doi.org/10.1371/journal.pcbi.1012186.g002

In order to understand this phenomenon in more detail, we performed a singular perturbation analysis (see Section E in S1 Appendix) to better clarify the mechanisms by which astrocyte signals may be modulating neural dynamics. This analysis treats the astrocyte state as a fixed parameter, premised on its relatively slow evolution relative to the neural dynamics. We can then study how this parameter affects the vector field and attractor landscape of the neural subsystem. Fig 2E provides the pseudo-bifurcation diagram of the above motif by showing the position of the fixed points in the x₁-dimension as a function of the ψ(z). When ψ(z) is small, there is only one fixed point (the red line, also see Fig 2F). When ψ(z) is large, the neural subsystem manifests three fixed points by means of a saddle-node bifurcation (see Fig 2G). In other words, at the bifurcation point, there is a fundamental change in the shape of the neuronal-synaptic vector field and hence dynamics. Thus, astrocytic modulation can drastically alter the flow of neuronal and synaptic activity as a function of time. We hypothesize this mechanism may be particularly powerful for the contextual guidance premise as it may enable astrocytes to reshape the dynamics of synaptic adaptation and hence neural computation, based on exogenous contextual signals, e.g., via v(t). Thus, astrocytes form, in essence, a pathway for context-guided meta-plasticity and targeted neuromodulation. Below, we probe this hypothesis within the reinforcement learning task paradigm.

Neuron-astrocyte networks are able to learn context-dependent decision-making problems

We apply the proposed multi-scale neuron-astrocyte network model to context-dependent decision-making problems. We focus specifically on multi-armed bandits (MABs), a well-known class of reinforcement learning problems, wherein an agent aims to maximize its cumulative reward over time by selecting actions (arms) from a set of available options [61]. MABs find applications in various domains, including recommendation systems, clinical trials, and cognitive tasks in neuroscience, as they provide a powerful framework for decision-making under uncertainty [62]. While well-studied, this class of problems nonetheless poses persistent challenges when environments are non-stationary. Our prevailing hypothesis is that the disparate time-scales of signaling emanating from astrocytes can enable learning in such settings.

A standard MAB assumes a constant environment, in which the probabilities of reward associated with different arms are stationary. Our goal, however, is to study the capacity of our proposed neuron-astrocyte networks, by virtue of their time-scale separation, to learn in more challenging non-stationary and/or context-dependent settings. Thus, we designed both stationary and non-stationary Bernoulli bandit environments (see Fig 3 and Multi-armed bandit tasks in Methods) within which to evaluate learning efficacy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3.

A. The architecture of the learning algorithm. The three plots on the right represent a stationary Bernoulli bandit scenario where the arm means remain (0.4, 0.8, 0.1) constantly over time, a flip-flop non-stationary Bernoulli bandit where Arm 2’s mean alternates between 0.92 and 0.042, and a smooth-change non-stationary Bernoulli bandit where all arm means change according to a smooth periodic function, respectively. The left figure shows the architecture of the learning algorithm. B, C, and D show the learning performance of the neuron-astrocyte (abbreviated as “Neuro-astro” in plots) method relative to other learning methods for the stationary bandit task, where B is for a single run, C is the average result for 10 runs and D is the mean and standard deviation of the asymptotic regrets (the UCB method is not compared in C and D, as it performs much worse than other methods).

https://doi.org/10.1371/journal.pcbi.1012186.g003

Performance comparison.

We conducted a comprehensive learning performance analysis of the proposed neuron-astrocyte network in comparison to other neural network architectures (vanilla RNN, LSTM, GRU), all trained the same way using the above method. In addition, we also deployed traditional algorithms for solving bandit problems, the Upper Confidence Bound (UCB) and Thompson Sampling (TS) methods. The specific learning procedures for all neural network-based methods are similar, as described in the above section.

Stationary case. Fig 3B–3D illustrates the comparison of the learning performance of different methods (neuron-astrocyte, LSTM, TS, vRNN, GRU, UCB) in a stationary bandit task with arm probability settings of (0.4, 0.8, 0.1). Each method requires exploration of the environment, resulting in high regret during the initial time steps. However, all methods eventually converge with comparable rates and cumulative regret of the same order of magnitude. In particular, the neuron-astrocyte architecture performs similarly to the other network-based implementations in this case. Single-run simulation results show that the neuron-astrocyte method uses less time to converge (see Section F.1 in S1 Appendix). In addition, this method tends to be robust as the tasks become more challenging due to the small distance between arm probabilities (see Section F.2 in S1 Appendix).

Non-stationary case. However, in the presence of non-stationarity, the neuron-astrocyte architecture displays significant gains in capability. Indeed, these networks can achieve almost stationary regrets over time as shown in Fig 4A and 4B, with the former depicting results for the flip-flop bandit and the latter for the smooth changing bandit. In contrast, other methods consistently result in escalating regrets. It is important to emphasize again that the setup for learning here is identical across all networks. These results are consistent across different non-stationary scenarios, evident in both individual and multiple runs (see Section F.3 in S1 Appendix). In addition, similar learning performance is observed in scenarios with a different number of actions (see Section F.5 in S1 Appendix), which suggests the generality of the neuron-astrocyte method. These observations indicate that the neuron-astrocyte network is able to leverage contextual signals and adapt its actions to the changing environment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Learning performance.

Performance comparison of the neuron-astrocyte method relative to other learning methods for non-stationary bandits: A. flip-flop switching and B. smooth changing. C, D. neuron-astrocyte learning performance for different time-scale separation. E. Single learning traces for τ = 1 and τ = 0.01, highlighting the role of time-scale separation in enabling RL over contexts. F. Astrocyte and synaptic activity projections for both contexts (indicated as a and b) in early, middle, and late phases of learning, highlighting the formation of distinct synaptic weight trajectories.

https://doi.org/10.1371/journal.pcbi.1012186.g004

Toward a fuller accounting of brain circuit dynamics

In this paper, we have examined the potential role of neuron-astrocyte interactions in context-dependent learning, with a specific focus on reinforcement-based bandit problems. We began by forming a simplified model of such interactions in the form of a dynamical system, leveraging canonical descriptions of neural firing rate activity and several abstractions of astrocytic activity and modulation that are based on extant neurobiological theory. In particular, we simplified the dynamical description of astrocytes and focused on key aspects: (i) their orders-of-magnitude time-scale separation from neurons, (ii) their modulation of synaptic processes, (iii) their indirect modulation of neuronal firing rates, and (iv) their ability to engage in response to contextual information. Our goal was to understand whether these aspects of neuron-astrocyte interaction, which are known to exist in the brain, matter for network computation.

Contextually-guided meta-plasticity and slow modulatory dynamics

From this perspective, our analysis indicates the potential for astrocytes to reshape neural and synaptic vector fields in significant ways, such as in the formation of multiple stationary regimes of activation and changing the geometry of synaptic weight evolution. Perhaps most notably, astrocytes can modify the dynamics of synaptic plasticity, effectively switching the network between slow and fast weight adaption regimes (see Fig 4F). This forms a powerful mechanism by which astrocytes can use external and internal contextual information [8] to shift networks between different modes of learning, which we view as a form of meta-plasticity in the wide sense of that term.

One new and central premise to our work is the use of a contextual signal that is accessible by astrocytes and neurons, with the premise that such a signal may embed task-relevant information and/or other circuit contexts, which is highly consistent with the body or work showing astrocytes ability to detect and respond to functionally salient physiological covariates such as neuromodulators (e.g., dopamine), hormones (e.g., glucocorticoids), or local cytokines. Our abstraction of this signal may be viewed as overly strong, insofar as it presents ‘clean’ context information to the network. From this perspective, we emphasize that all our alternative architectures, and especially the neuron-astrocyte model without time-scale separation, had access to this information. Thus, it is not merely the presence of contextual signaling that augments learning performance in our model, but the specific astrocyte-dependent dynamical mechanisms by which this information alters neurons and synapses.

Following the above, we emphasize that our goal was not simply to add slow modulation to neuronal networks, since this could be done in a myriad of ways. Rather we chose a specific, biologically motivated and hypothesized pathway (involving astrocytes). While we cannot exclude the role of all other slow processes, we do believe that the astrocyte meta-plasticity pathway has unique advantages. To illustrate this, we pose a simple null model where we eliminated astrocytes but added a slow, passive potassium gating to neurons (see Section F.7 in S1 Appendix). This null model enacts a slow time-scale, but without the nested feedback loop structure that we believe is key to neural-astrocyte interaction. After the same training procedure as in our primary results, we find that this null model can only learn the stationary (but not the non-stationary) version of the task (see Section F.7 in S1 Appendix). This lends credence to the idea that the unique interactivity properties of astrocytes with neurons are important to the hypothesized functional benefits.

Consistency with biology of spatiotemporal neuron-astrocyte interactions

In this paper, we have focused most of our attention on the temporal separation of neuronal and astrocytic activity and have discussed the consistency of our parameters with known biophysics in this regard. However, equally interesting are spatial features such as the ratio of astrocytes to neurons. In biology, the ratio of astrocytes to neurons is believed to be between 1:1.5 to 1:2 and we used the latter end of this range in our results. However, we also performed a sensitivity analysis to examine whether dynamics change appreciably as a function of this ratio. In the stationary case, an increased ratio positively influences learning performance, leading to a reduction in asymptotic cumulative regret as more astrocytes are introduced. However, the dynamic nature of the flip-flop bandit introduces a subtle impact on the ratio: optimal cumulative regret occurs at intermediate ratios, whereas both excessively low and high ratios detrimentally affect learning by increasing regret (see Section F.6 in S1 Appendix). It turns out that the optimal ratio is around 7/10, hence our simulation observation is consistent with the biology and indeed predicts a functional optimum within this range.

Astrocytic activity as a stabilizer of catastrophic forgetting

Catastrophic forgetting is a phenomenon in artificial neural networks that arises when networks are tasked with learning multiple tasks sequentially [63]. In this scenario, it often is the case that previously encountered tasks are ‘overwritten’ when the algorithmic optimization (i.e., learning) strategies are deployed to update the network parameters/weights to meet new task demands. Our results indicate that astrocytic modulation of neuronal and synaptic dynamics mitigates catastrophic forgetting. Here, we believe that the slow time-scale of astrocytes is instrumental in protecting previously learned network outputs upon encountering of a new context. As described above, the slow activation of astrocytes makes them generally less sensitive to parametric adjustment relative to neurons and synapses. Thus, their effects are more stable context-to-context. Furthermore, as we have seen, astrocytes have the effect of controlling neuronal and synaptic dynamics, such that those faster processes can occupy distinct regions of state space depending on astrocytic modulation. The combination of these two phenomena means that astrocytes can effectively insulate the learned trajectories/dynamics of one context preventing overwriting when learning is engaged for a subsequent context. These findings underscore the importance of dynamical heterogeneity in the brain and support the functional advantages that astrocytes may confer.

Clearly, an important next step for these models will be to validate them with appropriate experiments. As mentioned in the Introductions, tools for in vivo study of astrocyte function have lagged relative to those for neurons. For those tools that do exist, e.g., to disrupt astrocyte function [64], studies such as ours can identify salient behavioral paradigms within which experiments may be conducted. Specifically, our model suggests that astrocytes contribute to learning in context-dependent or, potentially, multi-task settings, more than they might in simpler behavioral paradigms. This can be tested using adequate history-dependent learning tasks and popular astrocyte silencing tools such as CalEx [65]. Hence, available tools could be deployed to test formally the role of astrocytes in such scenarios. For example, by examining the learning efficacy of rodents engaging multi-arm bandit paradigms [66].

Insights into algorithmic learning systems

While our goal in this paper has been to explore new theories regarding the potential significance of neuron-astrocyte interactions in the brain, it is nonetheless interesting to consider the implications of these results in the domain of algorithmic systems. We have already commented on the fact that traditional algorithmic methods of learning bandit tasks have difficulty in context-dependent settings, even in the presence of informative signaling. This begs the question of whether neuron-astrocyte type architectures may have utility beyond the bandit/reinforcement learning settings.

In this regard, there certainly exist recurrent neural networks designed to deal with multiple time-scale features, notably LSTMs [67] and hierarchical RNNs [68]. The LSTM has an internal memory cell state that enables it to deal with tasks that involve long-term dependencies. In hierarchical RNNs, multiple layers of RNNs are stacked on top of each other, where each layer captures information at a different level of temporal abstraction. The lower layers focus on short-term dependencies, while the higher layers focus on longer-term dependencies. The multi-scale neuron-astrocyte network considered here is in the form of a feedback-connected multi-layered network with different embedded time-scales, and hence may blend the different features of these extant machine learning architectures. It is thus possible that this framework may be extendable to other machine learning domains, especially ones involving disparate time scale requirements such as meta-learning [69].

Limitations and features not explained

Our model suggests a key role of slow astrocytic modulation of synaptic plasticity in enabling learning over long time-scales. This model effect was premised on prior theory and empirical findings, including [14]. However, astrocyte interactions with synapses are heterogeneous across and within brain regions (see [70] for instance) both in extent (number of synapses impinged upon) and nature (pre-, postsynaptic, or both). Hence the effects we show in this paper should be interpreted as a demonstration of sufficiency rather than necessity, and certainly not monolithically across brain areas. Importantly, we do not imply that all slow brain dynamics are enacted by astrocytes. Indeed, recent evidence indicates that slow oscillatory activity in the entorhinal cortex may enable information processing across minutes or longer [71]. Our findings demonstrate that: (i) astrocytes are particularly apt to convey active, calcium-mediated modulation of synaptic dynamics, and (ii) this modulation is particularly potent in learning scenarios, relative to more diffuse and passive slow dynamics (e.g., potassium, as discussed previously above). As also pointed out above, these results set up clear potential for future empirical work aimed at the disruption of astrocytic calcium signaling in complex function.

Other important facets of astrocytes are their gap junction coupling, which is believed to be a basis for a form of functional inter-connectivity, as well as their structured arrangement in non-overlapping domains throughout the entire brain, something referred to as tiling [72]. While our model described nested feedback loops of astrocyte-neuron modulation (i.e., connectivity), we did not explicitly explore the role of tiling and gap-junction couplings, leaving this question as future work.

Multi-scale modeling of neuron-astrocyte network dynamics

In general, neural dynamics can be described by recurrent neural network models. Here, we consider the biology-inspired continuous-time RNN (CTRNN) [57, 73]. Consider a group of n neurons where each neuron is connected to some other neurons via synapses. Let be the state of the unit i, which denotes the mean membrane potential of the neuron. Then, the model of CTRNN is defined by ODEs (4) where τ_n > 0 and a_i > 0 are the time constant and decaying parameter respectively, and u_i is the external input to unit i. ϕ(x_j) is the activation function. It is noted that each unit i collects the outputs ϕ(x_j) (i.e., short-term average firing frequency) from all the connected neural units in the network, weighted with the synaptic connection coefficients , where the positive or negative w_ij indicates an excitatory or inhibitory synapse respectively.

Synapses are capable of modifying their strength via synaptic plasticity, which is usually formulated as a learning rule where the change of a synaptic strength w_ij depends on the correlation between the firing rate of a presynaptic neuron j and the firing rate of the postsynaptic neuron i. We consider the Hebbian learning rule: the weight between two neurons strengthens when they are correlated and weakens otherwise. This rule is defined mathematically by the equation [74] (5) where b_ij > 0 is the decaying parameter; τ_w > 0 is the time constant; is a parameter which indicates an existing synaptic connection when it is non-zero. When c_ij takes a positive value, (5) is called the Hebbian learning, and the case with c_ij < 0 is anti-Hebbian learning.

In principle, neurons influence astrocytes by releasing neurotransmitters that induce calcium ion elevations within astrocytes. Biophysically, the increase in calcium levels within individual astrocytes can propagate to neighboring astrocytes over long distances, forming calcium waves [75]. The mechanisms of this propagation may involve astrocyte to astrocyte gap junctions, which are well validated biologically [59, 76] and are believed to form spatially contiguous groups of astrocytes referred to as a network [77] (but also see astrocyte syncytium [78]). Current biophysical mathematical models for astrocytes, including gap junction connectivity, are excessively complex and not easily translatable for analytical and computational purposes. In the development that follows, we propose a simplified model to describe astrocyte dynamics based on [79], which abstracts the mathematical description of astrocyte-to-astrocyte connectivity within a network formulation.

Consider a group of m astrocytes. Let be the state of astrocyte k which denotes the activity of calcium wave. For the glial node z_k, we assume the output of astrocyte calcium wave is similarly defined by an activation function. To distinguish it from the neuron, we use a different function, for instance, the hyperbolic tangent function ψ(z_k) = tanh(z_k). Then, in the absence of neuron-astrocyte interactions, the dynamics of z_k is described by (6) where τ_a is a constant time parameter; f_kl denotes the weight of the (network) connection from astrocyte l to k; v_k captures other external inputs. The usage of this phenomenological model can be justified with analogous arguments in [34], where a neuronal leaky integrate-and-fire model is used for astrocytes. Such a model is easy to modify to incorporate the neuro-synapse-astrocyte interactions and greatly facilitates the numerical and analytical investigation as shown in the first subsection of Results.

Stacking all the equations of neurons, synapses, and astrocytes together, we will arrive at the mathematical model for the neuron-astrocyte network as a whole. (7a) (7b) (7c) where the additional terms d_ijψ(z_k) and h_kϕ(x_i)ϕ(x_j) with are present to capture the high-order interaction between neurons, astrocytes and synapses according to the description in tripartite synapse structure. In system (7), there are n and m equations for x and z respectively. The number of synaptic connections is flexible and denoted by o with m ≤ o ≤ n(n − 1). Therefore, the dimension of system (7) is actually (m + n + o).

It is known that the activities of neurons, synapses, and astrocytes evolve on different time-scales. Neural firing occurs in milliseconds, synapse plasticity changes at a slower speed, and astrocyte processes take even longer, ranging from seconds to minutes. These varying time-scales significantly impact information processing in neuron-astrocyte interactions. To investigate the effects of these differences, we need to set the time-scale parameters, denoted as τ_n, τ_w, and τ_a, to different values. To make the speeds of the evolution of these variables distinguishable, we have the assumption: 0 < τ_n ≪ τ_w ≪ τ_a, with ≪ indicating the former entity is much smaller than the latter. As the main goal of this work is to study neuron and astrocyte computation, we set τ_n = τ_w for simplicity when applying the neuron-astrocyte model to solving the tasks.

Dynamic context-dependent multi-armed bandit tasks

In the setting of a stochastic MAB, there is a set of actions (arms) to choose from, and the bandit lasts T rounds in total. In each round t, an agent (decision-maker) chooses one action and obtains a reward r_t. The goal of the agent is to optimize the accumulated reward, i.e., . We consider the Bernoulli bandits which belong to stochastic MABs. In the context of Bernoulli bandits, the reward of each action is binary, either 1 or 0 depending the outcome is a success or failure. The reward r_i of the i-th action is drawn from a Bernoulli distribution, i.e., where μ_i ∈ [0, 1] is a constant denoting the mean of the distribution. Different actions have different μ_i where a larger value represents a higher probability of the successful outcome and thus a higher expectation of the reward. The reward sequence up to time T is a random process (8) In the Bernoulli bandit, the goal of optimizing the accumulated reward is equivalent to minimizing the cumulative regret (3). The standard Bernoulli bandit is stationary where all μ_i are fixed over time. In addition to the stationary case, we further consider non-stationary variants by making the means changeable and time-dependent. Two subcases are considered in this work:

1. Flip-flop switching: the means μ_i of actions remain constant for a certain period of time, and then abruptly transit to different values at certain time instants.
2. Smooth changing: the means change according to a continuous function of time. Here, we use the periodic function (9) where is a fixed value in [0, 1]; S(⋅) is the sigmoid function; P is used to control the period of this function and the term makes that the action with the highest expected reward can change between the available actions over time. When Q is large, this type of function is dominated by an approximately constant value, such that it looks like a smooth square wave. We set P and Q to 10000 and 100 respectively.

In dynamic bandits, when the arm means change over time and the action with the highest mean switches, contextual information can be revealed to the agent. This contextual information represents the changes in underlying contexts. Therefore, the tasks we considered become context-dependent. We define the contextual signals as a scalar in all the simulations presented in this work. However, it is important to note that these signals can also be expanded into a multi-dimensional vector to accommodate more general settings.

Discrete-time neuron-astrocyte network

For simplification, we assume that the self-decay parameters are all one and the time-scales of neurons and synapses are the same. Then, the neuron-astrocyte network model without inputs can be rewritten in the compact form (10) where x = [x₁, …, x_n]^⊤ and z = [z₁, …, z_m]^⊤ are state vectors for neurons and astrocytes; W = [w_ij] is the matrix for synapse weights and denotes the element-wise derivative of W; ϕ(x) = [ϕ(x₁), …, ϕ(x_n)]^⊤ and ψ(z) = [ψ(z₁), …, ψ(z_m)]^⊤ are vectors of activation functions while Φ(x) is the flatten vector of the matrix [ϕ(x_i)ϕ(x_j)]; C, D, F, and H are the parameter matrices with corresponding entries.

In (10), we have set the time constant for astrocytes to the unit, while time constants for neurons and synapses are both τ ≪ 1. In this way, τ is dimensionless and represents the time-scale difference rate between neurons and astrocytes. Note that (10) can be rewritten equivalently by a change of time, so that τ appears on the right hand side of .

By using the first-order Euler discretization method [80], we can transfer the continuous-time neuron-astrocyte model to the discrete-time approximated version (11) where γ is the discretization step size. In the following simulations, γ and τ are set to be 0.1 and 0.01 respectively. We use the sigmoid function ϕ(x) = 1/(1 + e^−x) and the hyperbolic tangent function ψ(z) = tanh(z) for neural and astrocyte layer in the simulations.

We incorporate this discrete-time neuron-astrocyte model as the hidden layer within the entire learning network, where a pair of linear input and output layers are placed before and after the hidden layer according to the machine learning convention. The input and the output are feed into and read from neuron-astrocyte network after multiplied by matrices and W_out. Therefore, the network as a whole is represented by (12) where b_out the bias vector with the corresponding dimension.

Reinforcement learning procedure

A key step of our study is the implementation of our model to reinforcement-learning paradigms. Within this functional setting, at each trial, the agent (i.e., the neuron-astrocyte network) is presented with a new reward. This reward is used to algorithmically optimize (i.e., train) the parameters of the model in a trial-wise fashion. In other words, at the conclusion of the trial, the current parameters and outputs of the model, along with the current reward, are used to evaluate a loss function (see below) that determines future parameter adjustments. The research question at hand is whether the neuron-astrocyte architecture and dynamics enable this form of learning to be efficacious. Table 1 summarizes all parameters, both fixed and trainable, and their values in the model and training process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Parameters in the neuron-astrocyte model and model training.

https://doi.org/10.1371/journal.pcbi.1012186.t001

The neuron-astrocyte network architecture comprises 128 neurons and 64 astrocytes (except for simulations where we vary the neuron-astrocyte ratio), with randomly initialized connections within each layer and interlayer hyperedges. The complete learning framework is depicted in Fig 3A. We first initialize the matrices C, D, F, H with entries drawn randomly from normal distributions with the zero mean, i.e., where N_M is the dimension of the focal matrix M. The elements of input and output matrices , , W_out and bias vector b_out are initialized from a uniform distribution , where N_M is again the dimension.

The dimension of the output y_t is the same as the number of actions in the bandits, i.e., 3 in most simulations. After multiplied by the readout matrix and plus the bias, the output is fed to a softmax function, and it produces a probability distribution over the available actions . The probability of selecting the action is (13) An action a_t is then sampled from this probability distribution and subsequently executed by the agent. The bandit environment will provide the agent with a reward, represented as . And according to [81], we use the loss function where is the average of rewards up to t and is the logarithm of the probability.

We adopted the traditional policy-based RL algorithm REINFORCE for the network training [81]. The gradient of the loss function L is calculated and used to update the network’s free parameters via the backpropagation (BP). During BP, we use the Adam method to optimize the aforementioned matrices and vectors with the default learning rate of 0.001.

In the case of other RNN-based methods as described in the comparison section below, we simply replace the neuron-astrocyte network with the alternative network models. To ensure a fair comparison of a comparable magnitude of training parameters, all these conventional RNNs are configured with 2 stacked layers, each consisting of 128 units. In the 2 stacked layers structure, the first layer is forward-connected to the second layer: the external input (contextual cues) is fed to the first layer, which has default trainable intra-connection weights; the output of the first layer is fed to the second layer as the input associated with a trainable matrix; and then the output of the second layer is further used to generate actions. The weights are initialized using a default method, and the training procedure remains consistent.

The network models and training procedures are implemented using PyTorch in Python.

Learning performance comparison

Numerous machine learning algorithms have been developed to tackle MABs. Among them, Upper Confidence Bound (UCB) and Thompson Sampling (TS) are widely recognized as the most prominent approaches for standard MABs. Discounted UCB (DUCB) and switching-window UCB (SWUCB) have been devised to handle changing environments in non-stationary scenarios. In addition to these canonical bandit algorithms, some neuro-bandit algorithms that utilize feedforward or recurrent neural networks to model the agent’s policy have been developed in recent years.

To perform a thorough yet not overly exhaustive assessment of learning performance, we analyze the asymptotic cumulative regret of our approach in comparison to selective algorithms across various scenarios. For stationary MABs, we evaluate our method against the UCB and TS algorithms, as well as RNN-based models including LSTM, vRNN, and GRU. In the context of non-stationary MABs, our method is compared to DUCB, SWUCB, and other RNN-based algorithms. It’s worth noting that the training procedures for all RNN-based models remain consistent with the previously described methodology.