Neural activity in awake behaving animals exhibits a vast range of timescales that can be several fold larger than the membrane time constant of individual neurons. Two types of mechanisms have been proposed to explain this conundrum. One possibility is that large timescales are generated by a network mechanism based on positive feedback, but this hypothesis requires fine-tuning of the strength or structure of the synaptic connections. A second possibility is that large timescales in the neural dynamics are inherited from large timescales of underlying biophysical processes, two prominent candidates being intrinsic adaptive ionic currents and synaptic transmission. How the timescales of adaptation or synaptic transmission influence the timescale of the network dynamics has however not been fully explored. To address this question, here we analyze large networks of randomly connected excitatory and inhibitory units with additional degrees of freedom that correspond to adaptation or synaptic filtering. We determine the fixed points of the systems, their stability to perturbations and the corresponding dynamical timescales. Furthermore, we apply dynamical mean field theory to study the temporal statistics of the activity in the fluctuating regime, and examine how the adaptation and synaptic timescales transfer from individual units to the whole population. Our overarching finding is that synaptic filtering and adaptation in single neurons have very different effects at the network level. Unexpectedly, the macroscopic network dynamics do not inherit the large timescale present in adaptive currents. In contrast, the timescales of network activity increase proportionally to the time constant of the synaptic filter. Altogether, our study demonstrates that the timescales of different biophysical processes have different effects on the network level, so that the slow processes within individual neurons do not necessarily induce slow activity in large recurrent neural networks.
Brain activity spans a wide range of timescales, as it is required to interact in complex time-varying environments. However, individual neurons are primarily fast devices: their membrane time constant is of the order of a few tens of milliseconds. Yet, neurons are also subject to additional biophysical processes, such as adaptive currents or synaptic filtering, that introduce slower dynamics in the activity of individual neurons. In this study, we explore the possibility that slow network dynamics arise from such slow biophysical processes. To do so, we determine the different dynamical properties of large networks of randomly connected excitatory and inhibitory units which include an internal degree of freedom that corresponds to either adaptation or synaptic filtering. We show that the network dynamics do not inherit the slow timescale present in adaptive currents, while synaptic filtering is an efficient mechanism to scale down the timescale of the network activity.
Citation: Beiran M, Ostojic S (2019) Contrasting the effects of adaptation and synaptic filtering on the timescales of dynamics in recurrent networks. PLoS Comput Biol 15(3): e1006893. https://doi.org/10.1371/journal.pcbi.1006893
Editor: Peter E. Latham, UCL, UNITED KINGDOM
Received: December 19, 2018; Accepted: February 19, 2019; Published: March 21, 2019
Copyright: © 2019 Beiran, Ostojic. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files. The codes that generate the figures are available at https://github.com/emebeiran/adapt_synapt.
Funding: This work was funded by the Ecole des Neurosciences de Paris Ile-de-France, the Programme Emergences of City of Paris, Agence Nationale de la Recherche grants ANR-16-CE37-0016-01 and ANR-17-ERC2-0005-01, and the program “Investissements d’Avenir” launched by the French Government and implemented by the ANR, with the references ANR-10-LABX-0087 IEC and ANR11-IDEX-0001-02 PSL* Research University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Adaptive behavior requires processing information over a vast span of timescales , ranging from micro-seconds for acoustic localisation , milliseconds for detecting changes in the visual field , seconds for evidence integration  and working memory , to hours, days or years in the case of long-term memory. Neural activity in the brain is matched to the computational requirements imposed by behavior, and consequently displays dynamics over a similarly vast range of timescales [6–8]. Since the membrane time constant of an isolated neuron is of the order of tens of milliseconds, the origin of the long timescales observed in the neural activity has been an outstanding puzzle.
Two broad classes of mechanisms have been proposed to account for the existence of long timescales in the neural activity. The first class relies on non-linear collective dynamics that emerge from synaptic interactions between neurons in the local network. Such mechanisms have been proposed to model a variety of phenomena that include working memory , decision-making  and slow variability in the cortex . In those models, long timescales emerge close to bifurcations between different types of dynamical states, and therefore typically rely on the fine tuning of some parameter . An alternative class of mechanisms posits that long timescales are directly inherited from long time constants that exist within individual neurons, at the level of hidden internal states . Indeed biophysical processes at the cellular and synaptic level display a rich repertoire of timescales. These include short-term plasticity that functions at the range of hundreds of milliseconds [14, 15], a variety of synaptic channels with timescales from tens to hundreds of milliseconds [16–19], ion channel kinetics implementing adaptive phenomena , calcium dynamics  or shifts in ionic reversal potentials . How the timescales of these internal processes affect the timescales of activity at the network level has however not been fully explored.
In this study, we focus on adaptative ion-channel currents, which are known to exhibit timescales over several orders of magnitude [23–25]. We contrast their effects on recurrent network dynamics with the effect of the temporal filtering of inputs through synaptic currents, which also expands over a large range of timescales . To this end, we extend classical rate models [27–30] of randomly connected recurrent networks by including for each individual unit a hidden variable that corresponds to either the adapting of the synaptic current. We systematically determine the types of collective activity that emerge in such networks. We then compare the timescales on the level of individual units with the activity within the network.
We consider N coupled inhibitory and excitatory units whose dynamics are given by two variables: the input current xi and a slow variable si or wi that accounts for the synaptic filtering or adaptation current respectively. The instantaneous firing rate of each neuron is obtained by applying a static non-linearity ϕ(x) to the input current at every point in time. For simplicity, we use a positive and bounded threshold-linear transfer function (1) where [⋅]+ indicates the positive part, γ is the activation threshold and ϕmax the maximum firing rate.
Single neuron adaptation is described by the variable w(t) that low-pass filters the linearized firing rate with a timescale τw, slower than the membrane time constant τm, and feeds it back with opposite sign into the input current dynamics (see Methods). The dynamics of the i-th adaptive neuron are given by (2) where Ii(t) is the external input current to neuron i.
Synaptic filtering consists in low-pass filtering the synaptic input received by a cell with time constant τs, before it contributes to the input current. The dynamics of the i-th neuron in a network with synaptic filtering are (3)
The matrix element Jij corresponds to the synaptic coupling strength from neuron j onto neuron i. In this study we focus on neuronal populations of inhibitory and excitatory units, whose connectivity is sparse, random, with constant in-degree: all neurons receive exactly the same number of excitatory and inhibitory connections, CE and CI, as in [31–33]. All excitatory synapses have equal strength J and all inhibitory neurons −gJ. Furthermore, we consider the large network limit where the number of synaptic neurons N is large while keeping the excitatory and inhibitory inputs CE and CI fixed.
Single unit: Timescales of dynamics
In the models studied here the input current of individual neurons is described by a linear system. Thus, their activity is fully characterized by the response h(t) to a brief impulse signal, i.e. the linear filter. When such neurons are stimulated with a time-varying input I(t), the response is the convolution of the filter with the input, x(t) = (h * I)(t). These filters can be determined analytically for both neurons with adaptation or synaptic filtering and directly depend on the parameters of these processes. Analyzing the differences that these two slow processes produce in the linear filters is useful for studying the differences in the response of adaptive and synaptic filtering neurons to temporal stimuli (Fig 1A), and will serve as a reference for comparison to the effects that emerge at the network level.
A: Firing rate response of two different neurons with adaptation (red curves) and two different neurons with synaptic filtering (grey curves) to the same time-varying input (black curve). B: Normalized linear filters for the neurons shown in A. C: Timescales of the linear filter for neurons with adaptation (red lines) and for neurons with synaptic filtering (grey lines) as a function of the timescale τw or τs, respectively. The dashed lines indicate the effective timescale of the evoked activity obtained by weighing each individual timescale with its amplitude in the linear filter. The effective timescale for neurons with adaptation saturates for large adaptation time constants, while it grows proportionally to the synaptic time constant for neurons with synaptic filtering. Note that for the adaptive neuron, if the two eigenvalues are complex conjugate, there is only one decay timescale. The triangles on the temporal axis indicate the time constants used in A and B. Adaptation coupling gw = 5. D: Variance of the input current as a function of the slow time constant when the adaptive and synaptic neurons are stimulated with Gaussian white noise of unit variance. In the case of neurons with adaptation, two different values of the adaptation coupling gw are shown. Time in units of the membrane time constant τm.
In particular, the filter of a neuron with synaptic filtering, hs(t), is the sum of two exponentially decaying filters of opposite signs and equal amplitude, with time constants τs and τm: (4) where Θ(t) is the Heaviside function (see Methods). Thus, the current response of a neuron to an input pulse received from an excitatory presynaptic neuron is positive and determined by two different timescales. The response first grows with timescale τm, so that the neuron cannot respond to any abrupt changes in the synaptic input faster than this timescale, and then decreases back to zero with timescale τs (grey curves, Fig 1B).
The adaptation filter is given as well by the linear combination of two exponential functions. In contrast to the synaptic filter, since the input in the adaptive neuron model affects directly the current variable xi(t), there is an instantaneous change in the firing rate to an input delta-function (red curves, Fig 1B). The timescales of the two exponentials can be calculated as (5) When the argument of the square root in Eq (5) is negative, the two timescales correspond to a pair of complex conjugate numbers, so that the filter is an oscillatory function whose amplitude decreases monotonically to zero at a single timescale. If the argument of the square root is positive, for slow enough adaptation, the two timescales are real numbers and correspond to exponential functions of opposing signs of decaying amplitude. However, the amplitudes of these two exponentials are different (see Methods). To illustrate this, we focus on the limit of large adaptation time constants with respect to the membrane time constant, where the two exponential functions evolve with timescales that decouple the contribution of the membrane time constant and the adaptation current. In that limit, the adaptive filter reads (6) The amplitude of the slow exponential is inversely related to its timescale so that the integral of this mode is fixed, and independent of the adaptation time constant. This implies that a severalfold increase of the adaptation time constant does not lead to strong changes in the single neuron activity for time-varying signals (Fig 1A).
Furthermore, we can characterize the timescale of the single neuron response as the sum of the exponential decay timescales weighed by their relative amplitude, and study how this characteristic timescale evolves as a function of the time constants of either the synaptic or the adaptive current (Fig 1C). For adaptive neurons, the activity timescale is bounded as a consequence of the decreasing amplitude of the slow mode, i.e. increasing the adaptation time constant beyond a certain value will not lead to a slower response. In contrast, the activity of an individual neuron with synaptic filtering scales proportionally to the synaptic filter time, since the relative amplitudes of the two decaying exponentials are independent of the time constants.
When any of the two neuron types are stimulated with white Gaussian noise, the variance in the response is always smaller than the input variance, due to the low pass filtering properties of the neurons. However, this gain in the variance of the input currents is modulated by the different neuron parameters (Fig 1D). For a neuron with synaptic filtering, the gain is inversely proportional to the time constant τs. In contrast, for a neuron with adaptation, increasing the adaptation time constant has the opposite effect of increasing the variance of the current response. This is because when the adaptation time constant increases, the amplitude of the slow exponential decreases accordingly, and the low-pass filtering produced by this slow component is weaker. Following the same reasoning, increasing the adaptation coupling corresponds to strengthening the low-pass filtering performed by adaptation, so that the variance decreases (Fig 1D, dashed vs full red curves).
In the absence of any external input, a non-trivial equilibrium for the population averaged activity emerges due to the recurrent connectivity of the network. The equilibrium firing rate is identical across network units, since all units are statistically equivalent. We can write the input current x0 at the fixed point as the solution to the transcendental equation (7) for the network with adaptation, and to (8) for synaptic filtering (see Methods). Based on Eq (7), we find that the adaptation coupling gw reduces the mean firing rate of the network, independently of whether the network is dominated by inhibition or excitation (Fig 2A). Synaptic filtering instead does not play any role in determining the equilibrium activity of the neurons, since Eq (8) is independent of the synaptic filtering parameter τs.
A: Firing rate of the network with adaptation at the equilibrium ϕ(x0) for increasing adaptive couplings and three different values of the effective recurrent coupling Jeff = J(CE − gCI). Stronger adaptation leads to lower firing rates at equilibrium. B: Phase diagram of the population-averaged activity for the network with adaptation. C: Phase diagram for the network with synaptic filtering.
We next study the stability and dynamics of the equilibrium firing rate in response to a small perturbation uniform across the network, xi(t) = x0 + δx(t). Because of the fixed in-degree of the connectivity matrix, the linearized dynamics of each neuron are identical, so that the analysis of the homogeneous perturbation on the network reduces to the study of a two-dimensional deterministic system of differential equations which corresponds to the dynamics of the population-averaged response (see Methods). The stability and timescales around equilibrium depend on the two eigenvalues of this linear 2D-system. More specifically, the fixed point is stable to a homogeneous perturbation if the two eigenvalues of the dynamic system have negative real part, in which case the inverse of the unsigned real part of the eigenvalues determines the timescales of the response. For both the network with synaptic filtering and the network with adaptive neurons, the order parameter of the connectivity that determines the stability of the fixed point is the effective recurrent coupling J(CE − gCI) each neuron receives, resulting from the sum of all input synaptic connections. A positive (negative) effective coupling corresponds to a network where recurrent excitation (inhibition) dominates and the recurrent input provides positive (negative) feedback [32, 33].
For networks with synaptic filtering, we find that the synaptic time constant does not alter the stability of the equilibrium state, so that the effective coupling alone determines the stability of the population-averaged activity. As the effective input coupling strength is increased, the system undergoes a saddle-node bifurcation when the effective input is J(CE − gCI) = 1 (Fig 2C). In other words, the strong positive feedback loop generated by the excitatory recurrent connections destabilizes the system.
To analyze the timescales elicited by homogeneous perturbations, we calculate the eigenvalues and eigenvectors of the linearized dynamic system (see Methods). We find that for inhibition-dominated networks (J(CE − gCI) < 0), the network shows population-averaged activity at timescales that interpolate between the membrane time constant and the synaptic time constant. As the effective coupling is increased, the slow timescale at the network level can be made arbitrarily slow by tuning the effective synaptic coupling close to the bifurcation value, a well-known network mechanism to achieve slow neural activity .
In the limit of very slow synaptic timescale, the two timescales of the population-averaged activity are (9) (10) so that the timescale τ− is proportional to the membrane time constant and τ+ is proportional to the slow synaptic time constant, effectively decoupling the two timescales. The relative contribution of these two timescales is the same, independently of the time constant τs, as we found in the single neuron analysis.
The network with adaptation shows different effects on the population-averaged activity. First, the presence of adaptation modifies the region of stability: the system is stable when the effective recurrent input J(CE − gCI) is less than the minimum of 1 + gw and (see Methods). Therefore, the stability region is larger than for the network with synaptic filtering (Fig 2B vs Fig 2C). In other words, the effective excitatory feedback required to destabilize the network is larger due to the counterbalance provided by adaptation. Moreover, adaptation allows the network to undergo two different types of bifurcations as the effective input strength increases, depending on the adaptation parameters. One possibility is a saddle-node bifurcation, as in the synaptic case, which takes place when J(CE − gCI) = 1 + gw. Beyond that instability all neurons in the network saturate. The other possible bifurcation, which happens if , at an effective coupling strength , is a Hopf bifurcation: the fixed point of network becomes unstable, leading in general to oscillating dynamics of the population-averaged response. Note that in the limit of very slow adaptation, the system can only undergo a Hopf bifurcation (Fig 2B).
The two timescales of the population-averaged activity in the stable regime for the adaptive network decouple the two single neuron time constants when adaptation is much slower than the membrane time constant. In this limit, up to first order of the adaptive time ratio , the two activity timescales are (11) (12) Similar to the single neuron dynamics, the amplitude of the slow mode, corresponding to τ−, decreases as τw is increased, so that the contribution of the slow timescale is effectively reduced when τw is very large. On the contrary, the mode corresponding to τ+, proportional to the membrane time constant can be tuned to reach arbitrarily large values. This network mechanism to obtain slow dynamics does not depend on the adaptation properties.
Linear stability analysis.
Previous studies have shown that random connectivity can lead to heterogeneous dynamics where the activity of each unit fluctuates strongly in time [29, 33–35]. To assess the effects of additional hidden degrees of freedom on the emergence and timescales of such fluctuating activity, we examine the dynamics when each unit is perturbed independently away from the equilibrium, xi(t) = x0 + δxi(t). By linearizing the full 2N-dimensional dynamics around the fixed point, we can study the stability and timescales of the activity characterized by the set of eigenvalues of the linearized system, λs and λw for the network with synaptic filtering neurons and adaptation, respectively. These sets of eigenvalues are determined by a direct mapping to the eigenvalues of the connectivity matrix, λJ (see Methods). The eigenvalues λJ of the connectivity matrices considered are known in the limit of large networks [33, 36]: they are enclosed in a circle of radius , except for an outlier that corresponds to the population-averaged dynamics, studied in the previous section. Therefore, we can map the circle that encloses the eigenspectrum λJ into a different shape in the space of eigenvalues λs/w (insets Fig 3). In order to determine the stability of the response to the perturbation, we assess whether the real part of the eigenspectrum λs/w is negative at all possible points. Furthermore, the type of bifurcation is determined by whether the curve enclosing the eigenvalues λs,w crosses the imaginary axis at zero frequency or at a finite frequency when the synaptic coupling strength is increased, leading respectively to a zero-frequency or to a Hopf bifurcation .
Numerical integration of the dynamics for the network with adaptive neurons (row A) and the network with synaptic filtering (row B), as the coupling standard deviation is increased. Colored lines correspond to the firing rates of individual neurons, the black line indicates the population average activity. Insets: complex eigenspectrum λw/s of the linearized dynamical matrix around the fixed point. Dots: eigenvalues of the connectivity matrix used in the network simulation. Solid line: theoretical prediction for the envelope of the eigenspectrum. The imaginary axis, Re(λ) = 0, is the stability boundary. i. Both the network with adaptation and synaptic transmission are stable. ii. The network with synaptic filtering crosses the stability boundary and shows fluctuations in time and across neurons, while the network with adaptation remains stable. iii. The network with synaptic filtering displays stronger fluctuations. The network with adaptive neurons undergoes a Hopf bifurcation leading to strong oscillations at a single frequency with uncorrelated phases across units. Note in the inset that for this connectivity matrix there is only one pair of complex conjugate unstable eigenvalues in the finite network. iv. The network with synaptic filtering shows strong fluctuations. The network with adaptation displays fluctuating activity with an oscillatory component. Parameters: in A, gw = 0.5, and τw = 5, in B, τs = 5.
The order parameter of the connectivity that affects the stability and dynamics of the network is now the radius of the circle of eigenvalues λJ, i.e. . This parameter is the standard deviation of the synaptic input weights of a neuron (see Methods), which contrasts with the order parameter of the population-averaged response, that depends on the mean of the synaptic input weights. The mean and standard deviation of the synaptic connectivity can be chosen independently, so that while the population-averaged activity remains stable, the individual neurons might not display stable dynamics. To analyze solely the heterogeneous response of the network to the perturbation, we focus in the following on network connectivities whose population-averaged activity is stable, i.e. the effective synaptic coupling is inhibitory or weakly excitatory.
We find that in the network with synaptic filtering, the eigenspectrum λs always crosses the stability bound through the real axis, which takes place when the spectral radius of the connectivity is one, . Thus the system undergoes a zero-frequency bifurcation similar to randomly connected networks without hidden variables [29, 33, 35, 38], leading to strong fluctuations at the single neuron level that are self-sustained by the network connectivity (Fig 3Bii–3Biv). The critical coupling at which the equilibrium firing rate loses stability is independent of the synaptic time constant, i.e. synaptic filtering does not affect the stability of heterogeneous responses (Fig 4A). However, the synaptic time constant τs affects the timescales at which the system returns to equilibrium after a perturbation, because the eigenvalues λs (see Eq (69) in Methods) depend explicitly on τs.
A: Phase diagram for the network with synaptic transmission. The only relevant parameter to assess the dynamical regime is the connectivity strength. The circles indicate the parameters used in Figs 3 and 6. Triangles correspond to the parameter combinations used in Fig 5. B: Frequency at which the eigenspectrum loses stability for the network with adaptive neurons as a function of the ratio between membrane and adaptation time constant, τm/τw, for three different adaptive couplings. The dots indicate the fastest adaptive time constant for which the system undergoes a Hopf bifurcation (Eq 84). C: Phase diagrams for the two adaptation parameters, (i) the coupling gw and (ii) the adaptive time constant τw vs the coupling standard deviation.
For a network with adaptive neurons, we calculate the eigenspectrum λw and find that the transition to instability Re(λw) = 0 can happen either at zero frequency or at a finite frequency (see Methods), leading to a Hopf bifurcation (as in inset Fig 3Aiii). In particular, the network dynamics undergo a Hopf bifurcation when (13) so that strong adaptation coupling and slow adaptation time constants lead to a finite frequency bifurcation. In particular, if the coupling gw is larger than , only the Hopf bifurcation is possible, since by construction . We can also calculate the frequency of oscillations at the Hopf bifurcation. We find that, for slow adaptive currents, the Hopf frequency is inversely related to the adaptation time constant (Fig 4B), so that slower adaptation currents produce slower oscillations at the bifurcation.
Adaptation also increases the stability of the equilibrium firing rate to a heterogeneous perturbation, in comparison to a network with synaptic filtering (Fig 4C). This can be intuitively explained in geometrical terms by analyzing how adaptation modifies the shape of the eigenspectrum λw with respect to the circular eigenspectrum of the connectivity matrix λJ.
The Hopf bifurcation leads to the emergence of a new dynamical regime in the network (Fig 3Aiv), which is studied in the following section. Right at the Hopf bifurcation, the system shows marginal oscillations at a single frequency that can be reproduced in finite-size simulations whenever only one pair of complex conjugate eigenvalues is unstable (Fig 3Aiii).
Fluctuating activity: Dynamical mean field theory.
The classical tools of linear stability theory applied so far can only describe the dynamics of the system up to the bifurcation. To study the fluctuating regime, we take a different approach and focus on the temporal statistics of the activity, averaged over different connectivity matrices: we determine the mean and autocorrelation function of the single neuron firing rate, and characterize the timescale of the fluctuating dynamics [29, 33–35, 38–40]. For large networks, the dynamics can be statistically described by applying dynamical mean field theory (DMFT), which approximates the deterministic input to each unit by an independent Gaussian noise process. The full network is then reduced to a two-dimensional stochastic differential equation, where the first and second moments of the noise must be calculated self-consistently. We solve the self-consistent equations using a numerical iterative procedure, similar to the schemes followed in [34, 41–44] (see Methods for an explanation of the iterative algorithm and its practical limitations).
For the network with synaptic filtering, we find that the autocorrelation function of the firing rates in the fluctuating regime corresponds to a monotonically decreasing function (Fig 5A), qualitatively similar to the correlation obtained in absence of synaptic filtering . This fluctuating state has often been referred to as rate chaos and shows non-periodical heterogeneous activity which is intrinsically generated by the network connectivity. The main effect of synaptic filtering is on the timescale of these fluctuations. When the synaptic time constant is much larger than the membrane time constant, the timescale of the network activity is proportional to the synaptic time constant τs, as indicated by the linear dependence between the half-width of the autocorrelation function and the synaptic timescale τs, when all other network parameters are fixed (Fig 5B).
A: Autocorrelation function of the firing rates in the network with synaptic filtering; dynamical mean field results (solid lines) with their corresponding envelopes (dashed lines), and results from simulations (empty dots). Connectivity strength . B: Effective timescale of the network activity as a function of the synaptic time constant for the network with synaptic filtering. The network coupling does not have a strong effect on the effective timescale. C: Autocorrelation function of the firing rates, as in A, for the system with adaptive neurons. Jcs = 1.3. D: Effective timescale of the firing rates, as in B, for the system with adaptive currents.
For the network with adaptation, we focus on large adaptation time constant τw, where the network dynamics always undergo a Hopf bifurcation. The autocorrelation function in such a case displays damped oscillations (Fig 5C). The decay in the envelope of the autocorrelation function is due to the chaotic-like fluctuations of the firing rate activity.
We define the time lag at which the envelope of the autocorrelation function decreases as the timescale of the network dynamics (see Methods). The timescale of the activity increases as the adaptation timescale is increased, when all the other parameters are fixed (Fig 5D). However, this activity timescale saturates for large values of the adaptation timescale: the presence of very slow adaptive currents, beyond a certain value, will not slow down strongly the network activity. This saturation value depends on the connectivity strength.
Effects of noise.
The networks studied so far, for a fixed connectivity matrix, are completely deterministic. We next study the effects of additional white noise inputs to each neuron, as a proxy towards understanding recurrent networks of spiking neurons with adaptation and synaptic filtering. On the mean-field level, such noise is equivalent to studying a recurrent network whose neurons fire action potentials as a Poisson process with instantaneous firing rate ϕ(xi(t)) [35, 45].
Numerical simulations show that in the stable regime the additive external noise generates weak, fast stationary dynamics around the fixed point (Fig 6Ai and 6Bi). The timescale of these fluctuations and their amplitude depend on the distance of the eigenspectrum to the stability line, so that the stable fluctuations for weak synaptic coupling standard deviation (Fig 6Ai) are smaller in amplitude than those for larger coupling standard deviation (Fig 6Aii), whose eigenspectrum is closer to the stability boundary. For adaptation, in the fluctuating regime beyond the Hopf bifurcation, the network activity shows again a combination of fluctuating activity and oscillations.
Numerical integration of the dynamics with units receiving additive external white noise, as a proxy for spiking noise. A: Network with adaptive neurons. B: Network with synaptic filtering. Colored lines correspond to the firing rate of individual neurons, the black line indicates the population average activity. Insets: complex eigenspectrum λw/s of the dynamic matrix at the fixed point. Dots: eigenvalues of the connectivity matrix used in the network simulation. Solid line: theoretical prediction for the envelope of the eigenspectrum. i. Both the network with adaptation and synaptic transmission are stable, the external noise generates stationary fluctuations around the fixed point. ii. The network with synaptic filtering undergoes a zero-frequency bifurcation. Noise adds fast temporal variability in the firing rates. The network with adaptation remains stable, and the fluctuations are larger in amplitude. iii. The network with adaptation undergoes a Hopf bifurcation. The firing rate activity combines the fast fluctuations produced by white noise and the chaotic activity with an oscillatory component. iv. The network with adaptation shows highly irregular activity, and strong effects due to the activation and saturation bounds of the transfer function. Parameters as in Fig 4, external noise ση = 0.06.
We further extend the DMFT analysis to account for the additional variance of the external white noise sources (see Methods). The autocorrelation function of the firing rates, as predicted by DMFT, does not vary drastically when weak noise is added to the network, except for very short time lags, at which white noise introduces fast fluctuations (see Fig 7). For the network with adaptation, the autocorrelation function of the firing rates still shows damped oscillations (Fig 7A), while for the network with synaptic filtering, similarly, weak noise does not affect much the decay of the autocorrelation function (Fig 7D). Very strong external noise on the other hand will reduce the effect of the underlying recurrent dynamics of the rate network, since the signal to noise ratio in the synaptic input of all neurons is low.
A: Autocorrelation function of the firing rates for the network with adaptive neurons for three different noise intensities. Adaptation time constant τw = 1.25. B: Variance of the firing rate as a function of the adaptation time constant for two different adaption couplings gw. Increasing the adaptation time constant or decreasing the adaptation coupling increases the variance. ση = 0.15. C: Timescale of the firing rate as a function of the adaptation time constant, and three different noise levels. Parameters: gw = 0.5, and . D: Autocorrelation function of the firing rate for the network with synaptic transmission for three different noise levels. Synaptic time constant τs = 1.25. E: Variance of the firing rate as a function of the synaptic time constant, for three different external noise levels. Synaptic filtering reduces the variance. F: Timescale of the activity for the network with synaptic filtering and external noise.
For a fixed external noise intensity, reducing the adaptation coupling or increasing the adaptation time constant increases the variance of the firing rate (Fig 7B), which resembles the dependence of the variance gain for individual neurons (Fig 1D). Conversely, slower synaptic filtering reduces the variance of the neuron’s firing rates. This is because in the network with synaptic filtering the noise is also filtered at the synapses –in the limit of very large τs, the whole white noise is filtered out– whereas in the network with adaptation the noise affects directly the input current, without being first processed by the adaptation variable.
However, the timescale of the activity is nonetheless drastically affected by strong noise. External noise adds fast fluctuations on top of the intrinsically generated dynamics of the heterogeneous network with adaptation or synaptic filtering. If the noise is too strong, the effective timescale of the activity takes into account mostly this fast component. In that limit, the timescale of the activity is almost independent of the synaptic or adaptive time constants (Fig 7C and 7F, largest noise intensity).
We examined dynamics of excitatory-inhibitory networks in which each unit had a hidden degree of freedom that represented either firing-rate adaptation or synaptic filtering. The core difference between adaptation and synaptic filtering was how external inputs reached the single-unit activation variable that represents the membrane potential. In the case of adaptation, the inputs directly entered the activation variable, which was then filtered by the hidden, adaptive variable through a negative feedback loop. In the case of synaptic filtering, the external inputs instead reached first the hidden, synaptic variable and were therefore low-pass filtered before being propagated in a feed-forward fashion to the activation variable. While both mechanisms introduce a second timescale in addition to the membrane time constant, our main finding is that the interplay between those two timescales is very different in the two situations. Surprisingly, in presence of adaptation, the membrane timescale remains the dominant one in the dynamics, while the contribution of the adaptation timescale appears to be weak. In contrast, in a network with synaptic filtering, the dominant timescale of the dynamics is directly set by the synaptic variable, and the overall dynamics are essentially equivalent to a network in which the membrane time-constant is replaced with the synaptic one.
We used a highly abstracted model, in which each neuron is represented by membrane current that is directly transformed into a firing-rate through a non-linear transfer function. This class of models has been popular for dissecting dynamics in excitatory-inhibitory [27, 28, 46–48] or randomly-connected networks [29, 30, 33], and for implementing computations [49, 50]. Effects of adaptation in this framework have to our knowledge not been examined so far, but see  for a simultaneously and independently developed study of adaptation in networks of multidimensional rate units with random Gaussian connectivity. We therefore extended the standard rate networks by introducing adaptation in an equally abstract fashion , as a hidden variable specified solely by a time constant and a coupling strength. Different values of those parameters can be interpreted as corresponding to different specific membrane conductances that implement adaptation, e.g. the calcium dependent potassium Iahp current or the slow voltage-dependent potassium current Im, which are known to exhibit timescales over several orders of magnitude [52, 53]. To cover the large range of adaptation timescales observed in experiments , it would be straightforward to superpose several hidden variables with different time constants. Our approach could also be easily extended to include simultaneously adaptation and synaptic filtering.
A number of previous works have studied the effects of adaptation within more biologically constrained, integrate-and-fire models. These works have in particular examined the effects of adaptation on the spiking statistics [54–56], firing-rate response [57, 58], synchronisation [25, 56, 59–61], perceptual bistability  or single-neuron coding [63, 64]. In contrast, we have focused here on the relation between the timescales of adaptation and those of network dynamics. While our results rely on a simplified firing-rate model, we expect that they can be directly related to networks of spiking neurons by exploiting quantitative techniques for mapping adaptive integrate-and-fire models to effective firing rate descriptions .
A side result of our analysis is the finding that strong coupling in random recurrent networks with adaptation generically leads to a novel dynamical state, in which individual units exhibit a mixture of oscillatory and strong temporal fluctuations. The characteristic signature of this dynamical state is a damped oscillation found in the auto-correlation function of single-unit activity. In contrast, classical randomly connected networks lead to a fluctuating, chaotic state in which the auto-correlation function decays monotonically [29, 33–35]. Note that the oscillatory activity of different units is totally out of phase, so that no oscillation is seen at the level of population activity. This dynamical phenomenon is analogous to heterogeneous oscillations in anti-symmetrically connected networks with delays . In both cases, the oscillatory dynamics emerge through a bifurcation in which a continuum of eigenvalues crosses the instability line at a finite-frequency. Similar dynamics can be also found in networks in which the connectivity is a superposition of a random and a rank two structured part . In that situation, the heterogeneous oscillations however originate from a Hopf bifurcation due to an isolated pair of eigenvalues that correspond to the structured part of the connectivity.
Our main aim here was to determine how hidden variables could induce long timescales in randomly-connected networks. Long timescales could alternatively emerge from non-random connectivity structure. As extensively investigated in earlier works, one general class of mechanism relies on setting the connectivity parameters close to a bifurcation that induces arbitrarily long timescales [12, 29]. Another possibility is that non-random features of the connectivity, such as the over-representation of reciprocal connections [66, 67] slow down the dynamics away from any bifurcation. A recent study  has indeed found such a slowing-down. Weak connectivity structures of low-rank type provide yet another mechanism for the emergence of long timescales. Indeed, rank-two networks can generate slow manifolds corresponding to ring attractors provided a weak amount of symmetry is present .
Ultimately, the main reason for looking for long timescales in the dynamics is their potential role in computations performed by recurrent networks [70, 71]. Recent works have proposed that adaptive currents may help implement computations in spiking networks by either introducing slow timescales or reducing the amount of noise due to spiking [72, 73]. Our results suggest that synaptic filtering is a much more efficient mechanism to this end than adaptation. Identifying a clear computational role for adaptation in recurrent networks therefore remains an open and puzzling question.
We compare the dynamics of two different models: a recurrent network with adaptive neurons, and a recurrent network with synaptic filtering. Each model is defined as a set of 2N coupled differential equations. The state of the i-th neuron is determined by two different variables, the input current xi (t) and the adaptation (synaptic) variable wi (t) (si (t)).
The dynamics of the recurrent network with adaptive neurons are given by (14) where ϕ(x) is a monotonically increasing non-linear function that transforms the input current into firing rate. In this study, we use a threshold-linear transfer function with saturation: (15)
In Eq (14) adaptation in single neuron rate models is defined as a low-pass filtered version with timescale τw of the neuron’s firing rate ϕ(xi (t)), and is fed back negatively into the input current, with a strength that we call the adaptation coupling gw. For the sake of mathematical tractability, we linearize the dynamics of the adaptation variable by linearizing the transfer function (Eq 15), ϕ(xi (t)) ≈ xi (t) − γ. Therefore, the dynamics of the network model with adaptation studied here read (16)
Note that this approximation allows for adaptation to increase the input current of a neuron, when the neuron’s current is below the activation threshold γ.
For the recurrent network with synaptic filtering, the dynamics are (17) In Eqs (14), (16) and (17), I(t) represents the total external input received by the neuron. In general, we are interested in the internally generated dynamical regimes of the network, so that the input is given by the synaptic inputs (18) The matrix element Jij indicates the coupling strength of the j-th neuron onto the i-th neuron. The connectivity matrix is sparse and random, with constant in-degree [32, 33, 74]: all neurons receive the same number of input connections C, from which CE are excitatory and CI inhibitory. All excitatory synapses have coupling strength J while the strength of all inhibitory synapses is −gJ. Moreover, each neuron can only either excite or inhibit the rest of the units in the network, following Dale’s principle. Therefore, the total effective input coupling strength, which is the same for all neurons, is (19)
We used the parameters in Table 1 for all figures unless otherwise specified.
Single neuron dynamics
The dynamics of each individual neuron are described by a two-dimensional linear system, which implies that the input current response x(t) to a time-dependent input I(t) is the convolution of the input with a linear filter h(τ) that depends on the parameters of the linear system: (20)
In general, for any linear dynamic system , where A is a square matrix in and b(t) is a N-dimensional vector, the dynamics are given by (21) where Θ(t) is the Heaviside function. Thus, comparing Eqs (21) and (20), the linear filter is determined by the elements of the so-called propagator matrix P(t) = eAtΘ(t).
For a single neuron wit synaptic filtering, the dynamics are given by Eq (17), where the input Ii(t) represents the external current. We write the response in its vector form (x(t), s(t))T and the input as (0, I(t))T. The dynamic matrix is (22)
The linear filter, hs(t′), is given by the entries of the propagator matrix that links the input I(t) to the output element x(t), which are in this case only the entry in row one and column two: hs(t′) = [P(t′)]12. To compute the required entry of the propagator, we diagonalize the dynamic matrix A = VDV−1. The matrix D is a diagonal matrix with the eigenvalues of matrix A in the diagonal entries, and V is a matrix whose columns are the corresponding eigenvectors. Applying the identity and the definition of propagator we obtain that (23) The two timescales of the activity are defined by the inverse of the eigenvalues of the system, which coincide with τm and τs. Every time a pulse is given to the neuron, both modes get activated with equal amplitude and opposing signs, as indicated by Eq 23. This means that there is a fast ascending phase after a pulse, at a temporal scale τm, and a decay towards zero with timescale τs.
The dynamics of a single adaptive neuron are determined by Eq (16), where Ii(t) is the external input to the neuron. We apply the same procedure to determine the timescales of the response of an adaptive neuron to time-dependent perturbations. The dynamic matrix for an adaptive neuron reads (24) Its eigenvalues are (25) and the eigenvectors (26) The eigenvalues are complex if and only if gw > (4τmτw)−1 (τw − τm)2, and in that case their real part is . As the adaptive time constant becomes slower, at a certain critical adaptation time constant both eigenvalues become real. We are interested in the behavior when the adaptation time constant is large. The absolute value of the inverse of the eigenvalues determines the time constants of the dynamics. Therefore, for large τw we can calculate the two real eigenvalues to first order of (27) (28)
In this limit of slow adaptation, the time constant of one eigenmode is proportional to τw, whereas the second mode scales with τm. We are interested in the amplitude of each mode with respect to the other.
By explicitly calculating the first entry of the propagator matrix we obtain the adaptive filter in terms of the eigenvectors and eigenvalues, (29) where we use the notation to indicate the first component of the eigenvector associated to the eigenvalue λ+. Approximating to leading order of the eigenvectors in Eq (26), we obtain the eigenvectors (30) (31) Then, using Eqs (29), (30) and (31), we determine the linear filter: (32)
Interestingly, in contrast with synaptic filtering, the amplitude of the two modes are not equal. The amplitude of the slow mode (first term in Eq 32), whose timescale is proportional to τw, decays proportionally to with respect to the fast mode, when τw ≫ τm(2gw + 1). Therefore, the area under the linear filter corresponding to this mode is independent of τw for very large adaptation time constants: (33) It follows that, if the adaptation timescale is increased, its relative contribution to the activity will decrease by the same factor, so that very slow adaptive currents will effectively be masked by the fast mode.
The two systems possess a non-trivial equilibrium state at which the input current of all units stays constant. Since all units are statistically equivalent, the equilibrium activity is the same for all units. For synaptic filtering, the input current at equilibrium is given by a transcendental equation, that is obtained by setting to zero the left hand side of Eq (17): (34) This equilibrium coincides with the fixed point of the system without synaptic filtering.
For adaption, instead, from Eq (16) we obtain that the equilibrium is determined by (35) We further assume unless otherwise specified that the fixed point of the system is in the linear regime of the transfer function, so that ϕ(x) = x − γ. In that case x0 = (J(CE − gCI) − gw) (x0 − γ), so that larger adaptation coupling corresponds to weaker input currents, i.e. decreasing stationary firing rate. The adaptation time constant does not affect the fixed point.
Dynamics of homogeneous perturbations
Linearizing Eq 17 we obtain (37) where we use the notation . Because the perturbation δx in Eq (37) is independent of j, using Eq (19) the dynamics for all units are equivalent to the population-averaged dynamics and are given by (38) From Eq (38) we can define the dynamic matrix (39) The only difference in the linearized dynamics of the population-averaged current with respect to the single neuron dynamics (Eq 22) is the non-diagonal entry . When either the derivative at the fixed point cancels, or when the total effective input is zero, the population dynamics equals the dynamics of a single neuron. The eigenvalues of the population-averaged dynamics are (40) and the eigenvectors (41) For very large synaptic time constants, the eigenvalues are approximated to leading order as (42) (43) Approximating as well the eigenvectors to leading order, we obtain (44) (45) the filter of the linear response to weak homogeneous perturbations reads: (46) (47) Note that the amplitude of the two exponential terms is the same, independently of the effective coupling and time constants.
For the system with adaptive neurons, the linearized system reads (48) As for the network with synaptic filtering, the dynamics of the perturbation are equivalent for each unit, so that we can write down the dynamic matrix for the population-averaged response to homogeneous perturbations (49) The difference with respect to the linear single neuron dynamics (Eq 48) is that the effective recurrent coupling appears now in the first diagonal entry of the dynamic matrix.
When the fixed point is located within the linear range of the transfer function, the derivative is one, so that we do not further specify the factor in the following equations. Consequently, the dynamics of the system to small perturbations do not depend on the exact value of the fixed point, which does not hold for more general transfer functions.
In the limit of very slow adaptation, given that the two eigenvalues are real, they can be approximated to leading order as (52) (53) and the corresponding eigenvectors read (54) (55) Therefore, if the perturbation is stable (see next section) we can write down the corresponding linear filter as (56)
Stability of homogeneous perturbations
The equilibrium point is stable when the real part of all eigenvalues is negative. Equivalently, in a two dimensional system –as it is the case for the population-averaged dynamics–, the dynamics are stable when the trace of the dynamic matrix is negative and the determinant positive.
In the system with synaptic filtering, the trace and determinant are (58) (59) The trace is therefore always negative. The determinant is positive, and therefore the population-averaged dynamics are stable, when the effective coupling J(CE − gCI) is smaller than unity. In contrast, if the effective coupling is larger than unity, i.e. if positive feedback is too strong, the equilibrium firing rate is unstable, so that any small perturbation to the equilibrium firing rate will lead the system to a different state. Right at the critical effective coupling, one eigenvalues is zero and the other one equals Trs, implying that the population-averaged dynamics undergo a saddle-node bifurcation. Beyond the bifurcation, the network reaches a state where the firing rates of all neurons saturate.
In the adaptive population dynamics, the recurrent connectivity has a different effect on the stability of the adaptive population dynamics. The trace and determinant of the dynamic matrix are (60) (61) Both the timescale τw and the strength gw of adaptation affect the trace and determinant of the dynamic matrix, and therefore the stability. The system is unstable if the determinant is negative (one positive and one negative real eigenvalue) or if the determinant is positive and the trace is positive. The determinant is negative, and therefore the system becomes unstable through a saddle-node bifurcation, when J(CE − gCI) > 1 + gw. Note that the adaptation strength increases the stability of the system: a stronger positive feedback loop is required to destabilize the fixed point, in comparison to the network with synaptic filtering. The determinant and trace are positive if J(CE − gCI) < 1 + gw but , respectively, leading to a Hopf bifurcation: the system produces sustained marginal oscillations at the bifurcation in response to small perturbations around the fixed point. Beyond the Hopf bifurcation, the oscillations are maintained in time, unless the system shows a fixed point when all neurons saturate (). This fixed point exists if x0 > ϕmax + γ.
We next study the network dynamics beyond the population-averaged activity, along modes where different units have different amplitudes. We study perturbations of the type (62) We define the 2N-dimensional vector . Since the dynamics of each unit is now different, the dynamic matrix of the linearized system, A, is described by a squared matrix of dimensionality 2N. Therefore, the perturbations generate dynamics along 2N different modes whose timescales are determined by the eigenvalues of the matrix A. The eigenvalues are determined by the characteristic equation |A − λI| = 0. In order to calculate these eigenvalues, we make use of the following identity which holds for any block matrix Z = A − λI, that is composed by the four square matrices P,Q, R, and S and the block S is invertible: (63) Consequently, if we set Eq (63) to zero, since we assumed that |S| ≠ 0, we obtain (64) The identity in Eq (63) can be shown by using the decomposition (65) together with the fact that when a non-diagonal block is zero. The determinant of such a matrix is the product of determinants of the diagonal blocks.
The dynamical matrix for the network with synaptic filtering, obtained by linearizing Eq (17), is (66) The matrix J is the connectivity matrix. Again, we assume in the following that the fixed point is located in the linear range of the transfer function, so that .
The characteristic equation, obtained by combining Eqs (64) and (66), reads (67) where λJ are the eigenvalues of the connectivity matrix. Solving for λJ we obtain the equation which maps the eigenvalues of the synaptic filtering network dynamics λs onto the eigenvalues of the connectivity matrix λJ, (68) In contrast, solving for the eigenvalues of the dynamic matrix λs we obtain the inverse mapping (69) In other words, Eqs 69 and 68 constitute two different approaches to assessing the stability of the system . One approach is to examine whether the domain of eigenvalues λs resulting from Eq (69) intersect the line Re (λs) = 0 (Fig 3, insets in B). The eigenvalues λJ of the connectivity matrix are distributed within a circle in the complex plane, whose radius is proportional to the synaptic strength, plus an outlier real eigenvalue at J(CE − gCI) that corresponds to the homogeneous perturbations studied above (see ). We focus in this section on the bulk of eigenvalues that corresponds to modes of activity with different amplitudes for different units. We can therefore parametrize the eigenvalues λJ as (70) and introduce the parametrization into Eq (69) to obtain an explicit expression for the curve that encloses the eigenspectrum λs. Note that in an abuse of notation, we denote the limits of the eigenspectrum as λ and not the eigenvalues themselves that constitute the eigenspectrum.
The alternative approach is to use the inverse mapping from the eigenvalues λs to the eigenvalues of the connectivity λJ, by mapping the line Re (λs) = 0 into the space of eigenvalues λJ (see S1 Fig). More specifically, the line Re (λs) = 0 can be parametrized as (71) and introduced into Eq (68). In this case, the stability is assessed by whether the eigenspectrum of the connectivity matrix J crosses the stability boundary or not (insets in Fig 3). This alternative approach is useful for some calculations due to the simple geometry of the connectivity eigenspectrum λJ.
The first point of the stability curve intersecting with a circle of increasing radius centered at the origin is the closest point of the curve to the origin, i.e. the minimum of with respect to ω. The squared distance to the origin is (73) whose minimum happens trivially at ω = 0, λJ = 1 (see S1 Fig). In conclusion, the system is unstable if (74) Note that this is the same condition as in the case without synaptic filtering, The synaptic filtering system approaches the no-filtering system when τs → 0. Although we are considering in this work synaptic timescales that are larger than the membrane time constant, the analysis is valid for arbitrarily fast synaptic time constants. In that limit, the stability curve in Eq (72) approaches the curve , retrieving the stability boundary found in .
To study the limit of slow synaptic time constant, τs ≫ τm, we analyze the direct approach, i.e. study how the parameters of adaptation modify the eigenspectrum of the dynamic matrix As in the complex plane of eigenvalues λs. To this end, we introduce the parametrized connectivity eigenspectrum (Eq 70) into Eq (69), and approximate it to leading order of . We obtain that the eigenspectrum of eigenvalues λs are enclosed by the curves (75) (76) The equations above approximate the full eigenspectrum by two disjoint circles of radius , the one corresponding to the eigenvalues centered at , and the other circle λ− centered at . The circle centered closer to the instability bound, sets the slow timescales of the network, and its associated timescale is proportional to τs. This gives an intuitive explanation to why the network timescale scales linearly with the synaptic time constant (Fig 5).
Using Eqs (64) and (77) we can obtain the characteristic equation. Solving for λJ we obtain the mapping between the λw eigenvalues and the connectivity eigenvalues (78) while solving for λw we obtain the expression for the inverse mapping: (79)
The bifurcation parameters can then be found by determining the closest point of the stability boundary to the origin. The corresponding value of ω determines the oscillatory frequency of the first unstable mode. This value can be zero, corresponding to a zero-frequency bifurcation, which generally leads to slowly fluctuating activity referred to as rate chaos (, , , ). Alternatively, when the parameter ω that minimizes the norm of is non-zero, the system undergoes a Hopf bifurcation.
It is useful to consider the different geometries of the stability curve in Eq (80) in order to identify the closest point of the curve to the origin. Note that the curve shows symmetry with respect to the real axis, .
The curve might cross the real axis Im (λJ) = 0 either in one or two different values of |ω|. Solving , we find that the curve crosses twice the real axis, when τm < τwgw (see S1Bi Fig). In that case, one crossing is the point and the other τmλJ = 1 + gw. This second intersection corresponds to ω = 0. Therefore, it is clear that, since the first crossing of the real axis is closer to the origin than the point at ω = 0, the bifurcation necessarily occurs at non-zero frequency for τm < τwgw.
When the curve crosses only once the zero axis, the point λJ = 1 + gw, corresponding to a zero-frequency, is not necessarily the closest one to the origin (S1Bii Fig). One approach to determine analytically whether the system undergoes a Hopf or a zero-frequency bifurcation is to look at the curvature at the point ω = 0 and compare it to the curvature of a circle with radius 1 + gw. To do so, we approximate both the stability line and the circle by a parabola, and compare their curvatures (dashed curve, S1Bii and S1Biii Fig). First, we write the stability boundary in its implicit form, , as (81) Then, we consider small deviations of the coordinates xsb = 1 + gw + ϵx and ysb = ϵy. If we approximate up to first order of ϵx and second order of ϵy we obtain the parabola (82)
Repeating the same procedure for the circle of eigenvalues, with radius r = 1 + gw we obtain . By requiring the circle of eigenvalues to be interior to the boundary curve (for the same ϵx, ), we obtain that the instability parabola is exterior to the circle, therefore the system undergoes a zero-frequency bifurcation (S1C Fig), when (83) which simplifies to (84)
In the limit of the adaptation timescale approaching the membrane time constant, the left side of the inequality above approaches one. Introducing this value in Eq 84, we find that for adaptive couplings stronger than only a Hopf bifurcation is possible.
Dynamical mean field theory
The linearization of the dynamical system from the previous section is only valid up to the instability boundary. A commonly used method to study the dynamics that arise beyond the bifurcation is dynamical mean field theory (DMFT) [29, 33–35, 39–41]. DMFT approximates the deterministic input to each element of the system by a Gaussian stochastic process, whose first and second moment are determined self-consistently.
The dynamics of the i-th neuron in the synaptic and adaptive network are approximated as (85) (86) where ξi(t) is a Gaussian variable. In the thermodynamic limit, the noise sources are independent between neurons, so that for i ≠ j [ξi(t)ξj(t′)] = 0.
The next step is to determine the self-consistent equations, that links the distribution of ξi to the statistics of the original system in Eqs (16) and (17). First, we relate the statistics of the noise, currents xi and rates ϕ(xi) based on the dynamics. Then, we close the equations by explicitly assuring that the transfer function relates the currents and the rates.
To determine the first moment of the noise, we apply that ξi(t) = ∑j Jij ϕ(xj(t)) and average over the population, as in . The first moment of the noise then obeys (87)
We calculate next the relation for the second moment of the noise, which again is the same as in : (88) where C(τ) = 〈ϕi(t)ϕi(t + τ)〉.
These equations show that the first and second moment of the Gaussian sources do not depend on the identity of neuron i, so that all neurons are statistically equivalent. Thus, we can reduce the full 2N-deterministic system to a two-variable stochastic system, describing a prototypical neuron in the network.
The Eqs (87) and (88) describe how the noise is related to the properties of the connectivity and the statistics of the rates ϕ(x). The next step is to calculate how the first and second moment of the noise are related to the statistics of the input current, which we write as μ ≔ [xi] for the first moment and Δ(τ) ≔ [xi (t) xi (t + τ)] − μ2 for the second moment.
By differentiating twice Δ(τ) with respect to the lag τ and using Eqs (85) and (88), as in [29, 33] we obtain: (91) where is the autocorrelation function of the single neuron filter hs (Eq 23). Equivalently, for the adaptive system, using Eqs (86) and (88) we obtain (92) where we define in relation to Eq (6) , and the autocorrelation function of the adaptive filter .
Secondly, in order to close the self-consistent description, we can link the statistics of the rates ϕi(t) with the statistics of the currents xi(t) by writing the input currents explicitly as Gaussian variables. We can write down the input current at time t and t + τ explicitly as (see ): (93) (94) This explicit construction in terms of Gaussian variables z1, z2 and z3 realizes the constraints [x2 (t)] − μ2 = Δ(0), [x2 (t + τ)] − μ2 = Δ(0) and [x(t) x (t + τ)] − μ2 = Δ(τ). Now, explicitly calculating the first moment of the rates by replacing the average for a Gaussian integral and using Eq (93) we obtain (95) where we use the short-hand notation .
Therefore, in order to determine the self-consistent solution, we need to find a mean and autocorrelation function for the currents that satisfy both Eqs (95) and (96) and Eqs (89) and (91) (for the synaptic system) and Eqs (90) and (92) (for the adaptive system). Once the statistics of the currents and rates are known, it is straight-forward to obtain the statistics of the noise, using Eqs (87) and (88).
In previous works [29, 33, 35, 38, 40] it was possible to further simplify the self-consistent equations because the resulting analogous equation to Eqs (91) and (92) was a conservative system. However, in the networks studied here, synaptic filtering and adaptation add the convolutional terms in Eqs (91) and (92) that make the system non-conservative. Therefore, we followed an alternative approach and found the solutions to the self-consistent equations using an iterative scheme, that circumvents solving directly the integral equations.
- First, we simulate the dynamics in Eqs (85) and (86) assuming white Gaussian noise with a certain mean [ξ](0) and autocorrelation function .
- We calculate the autocorrelation functions of the firing rate and input currents empirically, μ(0), Δ(0), 〈ϕ〉(0) and C(0)(τ).
- We simulate in the new iteration k + 1 the noise following the self-consistent statistics obtained in the previous iteration, as indicated by Eqs (87) and (88) (97) (98) In order to numerically generate a Gaussian variable with autocorrelation function G(τ), we first generate the noise in the Fourier domain, where each frequency component of the noise is given by (99) where denotes the Fourier transform of the target autocorrelation function, and ψ is a random variable with uniform probability density in the range [−π, π].
- We repeat the previous step until the values μ(k), Δ(k), 〈ϕ〉(k) and C(k)(τ) do not vary beyond a certain tolerance for new iterations.
We find that such an iterative method applied to the systems studied here converges to a solution for the parameters of the noise after a few iterations, independently of the noise properties used in the initial step.
The drawbacks of this iterative scheme are that the two-dimensional system needs to be simulated several times at each iteration in order to determine the first and second order statistics of the input current and the firing rate, which is in general a computationally costly operation. We also find that the method converges more robustly to the solution (given the fact that both the trial length in the simulation and the number of trials are finite), at the expense of initial speed convergence, when the first and second moments of the noise are only partially updated at each iteration, so that (100) and similarly for the second-moment equation, where α is a parameter between zero and one. In this work, we used α = 0.6.
This method is inefficient for very large adaptation and synaptic time constants, since it requires simulating with both a fine temporal resolution (faster than the membrane time constant) over very large intervals (much larger than the slow adaptive/synaptic timescale). Another drawback of the iterative method is that its convergence is based on the assumption that smooth changes in the noise statistics lead to smooth changes in the statistics of the firing rates. In general, close to a bifurcation, this requirement may not hold.
Dynamics with intrinsic noise.
We next study how white Gaussian noise, independent between neurons and intrinsic to each unit in the network, affects the dynamics of the system. On the mean-field level, this is equivalent to studying a network where each neuron spikes at a Poisson process whose rate varies in time as ϕ(xi (t)) . The additional input to each neuron, whose dynamics are given in Eqs (2) and (3), is now (101) where [ηi] = 0, and , and Gaussian distributed. The DMF equations are derived following the same steps as in the absence of intrinsic noise. The stochastic variable ξ(t) is the sum of the recurrent input and the intrinsic noise. Its first moment remains unchanged: (102) (103) which is the same result as Eq (87). The second moment of the stochastic process is the sum of the variance generated by the recurrent connections and the variance of the intrinsic noise (104) (105)
Accordingly, the iterative scheme now takes into account the equation above, so that the equation for the second moment of the self-consistent relation (Eq 98) reads when there is intrinsic noise (106)
Adding white noise produces a discontinuity in the derivative of the autocorrelation function of the firing rates at zero lag (Fig 7A and 7D). This can be shown by integrating explicitly both sides of the Eqs (91) and (92) around zero when the external noise is added. It results in the condition (107)
Since the autocorrelation function is a symmetric function, , leading to (108) Thus, the autocorrelation function of the input current decays linearly at zero time lag with a slope proportional to the external noise intensity, which also extends to the autocorrelation function of the firing rate.
Definition of the timescale of the activity
The activity of multivariable dynamical systems ranges over several timescales. In particular, for stable linear systems, the timescales of the activity are given by the inverse of the absolute values of the real part of the eigenvalues. As we showed before, for single adaptive or synaptic neurons, the activity consists of two modes that evolve at two different timescales. However, the relative contribution of each of the excited modes can make one timescale more predominant than the other, as it happens for slow adaptation time constant, which becomes effectively undetectable in the single neuron dynamics.
In this work, we calculate the timescale of the activity for linear systems as the average of the timescales of the activated input current modes, weighed by their contribution (Fig 1). For a linear system with filter , the correlation time is (109)
For large networks, which are high-dimensional non-linear systems, we define the main timescale of the activity as the time lag at which the autocorrelation function has decayed to a fraction of its maximum (Figs 5 and 7): (110) where E[C(τ)] is the envelope of the autocorrelation function, calculated as the norm of its analytic signal, computed using the Hilbert transform. This corresponds to the width of the envelope at which the autocorrelation decays to e−0.5 of its value. For an exponentially decaying correlation function, this measure corresponds to the decay time constant. For a Gaussian envelope, this measure would correspond to two times its standard deviation, 2σ.
S1 Fig. Geometrical description of the bifurcation of the heterogeneous activity.
A: Instability bound for the system with synaptic filtering (grey line, Eq 72) and eigenspectrum for the weakest unstable synaptic coupling J. For any parameter combination, the instability bound, a parabola, is first touched by the growing circle of eigenvalues at ω = 1 and value B: Three different configurations of the instability bound for the system with adaptation in the complex plane of eigenvalues of the connectivity matrix, λJ. The black dots indicate the intersection between the instability boundary (full red line) and the eigenspectrum of λJ (dashed black line) with weakest coupling that is unstable. (i) The instability boundary intersects the real axis twice, leading to a Hopf bifurcation. (ii) It intersects the real axis just once and still leads to a Hopf bifurcation, because the intersection with the real axis is not the closest point of the curve to the origin. (iii) It intersects the real axis once and leads to a zero-frequency bifurcation, because the crossing of the real axis is the closest point to the origin. In (ii) and (iii) we draw the parabolic approximation of the instability bound (red dashed line, Eq 82). If the curvature of this parabola is exterior to the λJ eigenspectrum, as in (iii), the system undergoes a zero-frequency bifurcation. C: Oscillatory frequency at which the network with adaptation undergoes a bifurcation. To the right of the white line (Eq 84), the network displays a Hopf bifurcation, whereas to the left, the bifurcation happens at zero-frequency. The triangles indicate the parameter combinations used in B.
- 1. Fairhall AL, Lewen GD, Bialek W, De Ruyter van Steveninck RR. Efficiency and ambiguity in an adaptive neural code. Nature. 2001;412(6849):787–792. pmid:11518957
- 2. Grothe B, Pecka M, McAlpine D. Mechanisms of Sound Localization in Mammals. Physiological Reviews. 2010;90(3):983–1012. pmid:20664077
- 3. Tchumatchenko T, Malyshev A, Wolf F, Volgushev M. Ultrafast Population Encoding by Cortical Neurons. Journal of Neuroscience. 2011;31(34):12171–12179. pmid:21865460
- 4. Smith PL, Ratcliff R. Psychology and neurobiology of simple decisions. Trends in Neurosciences. 2004;27(3):161–168 pmid:15036882
- 5. Miyashita Y, Chang HS. Neuronal correlate of pictorial short-term memory in the primate temporal cortex. Nature. 1988;331(6151):68–70. pmid:3340148
- 6. Bair W, Movshon JA. Adaptive Temporal Integration of Motion in Direction-Selective Neurons in Macaque Visual Cortex. Journal of Neuroscience. 2004;24(33):7305–7323. pmid:15317857
- 7. Bernacchia A, Seo H, Lee D, Wang XJ. A reservoir of time constants for memory traces in cortical neurons. Nature Neuroscience. 2011;14(3):366–372. pmid:21317906
- 8. Murray JD, Bernacchia A, Freedman DJ, Romo R, Wallis JD, Cai X, et al. A hierarchy of intrinsic timescales across primate cortex. Nature Neuroscience. 2014;17(12):1661–1663. pmid:25383900
- 9. Wang XJ. Synaptic reverberation underlying mnemonic persistent activity; Trends in Neurosciences. 2001;24(8):455–643. pmid:11476885
- 10. Wang XJ. Decision Making in Recurrent Neuronal Circuits; Neuron. 2008;60(2):215–234. pmid:18957215
- 11. Litwin-Kumar A, Doiron B. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nature Neuroscience. 2012;15(11):1498–1505. pmid:23001062
- 12. Huang C, Doiron B. Once upon a (slow) time in the land of recurrent neuronal networks… Current Opinion in Neurobiology. 2017;46:31–38. pmid:28756341
- 13. Buonomano DV, Maass W. State-dependent computations: Spatiotemporal processing in cortical networks. Nature Reviews Neuroscience. 2009;10(2):113–125. pmid:19145235
- 14. Zucker RS, Regehr WG. Short-Term Synaptic Plasticity. Annual Review of Physiology. 2002;64(1):355–405. pmid:11826273
- 15. Markram H, Wang Y, Tsodyks M. Differential signaling via the same axon of neocortical pyramidal neurons. Proceedings of the National Academy of Sciences. 1998.
- 16. Newberry NR, Nicoll RA. Direct hyperpolarizing action of baclofen on hippocampal pyramidal cells. Nature;308(5958):450–2.
- 17. Batchelor AM, Madge DJ, Garthwaite J. Synaptic activation of metabotropic glutamate receptors in the parallel Fibre-Purkinje cell pathway in rat cerebellar slices. Neuroscience. 1994;63(4):911–915. pmid:7535396
- 18. Garthwaite J. Glutamate, nitric oxide and cell-cell signalling in the nervous system. Trends in neurosciences. 1991;14(2):60–7. pmid:1708538
- 19. Lester RAJ, Clements JD, Westbrook GL, Jahr CE. Channel kinetics determine the time course of NMDA receptor-mediated synaptic currents. Nature. 1990;346(6284):565–567. pmid:1974037
- 20. Johnston D, Wu SMs. Foundations of cellular neurophysiology. MIT Press; 1995.
- 21. Berridge MJ, Bootman MD, Roderick HL. Calcium: Calcium signalling: Dynamics, homeostasis and remodelling; 2003. Nature Reviews Molecular cell biology. 2003;4(7):517. pmid:12838335
- 22. Gal A, Eytan D, Wallach A, Sandler M, Schiller J, Marom S. Dynamics of Excitability over Extended Timescales in Cultured Cortical Neurons. Journal of Neuroscience. 2010;30(48):16332–16342. pmid:21123579
- 23. La Camera G, Rauch A, Thurbon D, Lüscher HR, Senn W, Fusi S. Multiple Time Scales of Temporal Response in Pyramidal and Fast Spiking Cortical Neurons. Journal of Neurophysiology. 2006;96(6):3448–3464. pmid:16807345
- 24. Benda J, Herz AVM. A Universal Model for Spike-Frequency Adaptation. Neural Computation. 2003;15(11):2523–2564. pmid:14577853
- 25. Ermentrout B, Pascal M, Gutkin B. The effects of spike frequency adaptation and negative feedback on the synchronization of neural oscillators. Neural Computation. 2001;13(6):1285–1310. pmid:11387047
- 26. Hennig MH. Theoretical models of synaptic short term plasticity. Frontiers in Computational Neuroscience. 2013;7.
- 27. Wilson HR, Cowan JD. Excitatory and Inhibitory Interactions in Localized Populations of Model Neurons. Biophysical Journal. 1972;12(1):1–24. pmid:4332108
- 28. Wilson HR, Cowan JD. A mathematical theory of the functional dynamics of cortical and thalamic nervous tissue. Kybernetik. 1973;13(2):55–80. pmid:4767470
- 29. Sompolinsky H, Crisanti A, Sommers HJ. Chaos in random neural networks. Physical Review Letters. 1988;61(3):259–262. pmid:10039285
- 30. Abbott LF. Decoding neuronal firing and modelling neural networks. Quarterly Reviews of Biophysics. 1994;27(3):291–331. pmid:7899551
- 31. Amit D, Brunel N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cerebral Cortex. 1997;7(3):237–252. pmid:9143444
- 32. Brunel N. Dynamics of networks of randomly connected excitatory and inhibitory spiking neurons. Journal of Physiology Paris. 2000;94(5-6):445–463.
- 33. Mastrogiuseppe F, Ostojic S. Intrinsically-generated fluctuating activity in excitatory-inhibitory networks. PLoS Computational Biology. 2017;13(4):1–40.
- 34. Rajan K, Abbott LF, Sompolinsky H. Stimulus-dependent suppression of chaos in recurrent neural networks. Physical Review E. 2010;82(1):1–5.
- 35. Kadmon J, Sompolinsky H. Transition to chaos in random neuronal networks. Physical Review X. 2015;5(4).
- 36. Rajan K, Abbott LF. Eigenvalue spectra of random matrices for neural networks. Physical Review Letters. 2006;97(18):2–5.
- 37. Bimbard C, Ledoux E, Ostojic S. Instability to a heterogeneous oscillatory state in randomly connected recurrent networks with delayed interactions. Physical Review E. 2016;94(6):3–8.
- 38. Schuecker J, Goedeke S, Helias M. Optimal sequence memory in driven random networks; Physical Review X. 2018;8(4):041029.
- 39. Aljadeff J, Stern M, Sharpee T. Transition to Chaos in Random Networks with Cell-Type-Specific Connectivity; Physical Review Letters. 2015;114(8). pmid:25768781
- 40. Harish O, Hansel D. Asynchronous Rate Chaos in Spiking Neuronal Circuits. PLoS Computational Biology. 2015;11(7):e1004266. pmid:26230679
- 41. Stern M, Sompolinsky H, Abbott LF. Dynamics of random neural networks with bistable units. Physical Review E. 2014;90(6):1–7.
- 42. Lerchner A, Sterner G, Hertz J, Ahmadi M. Mean field theory for a balanced hypercolumn model of orientation selectivity in primary visual cortex. Network: Computation in Neural Systems. 2006;17(2):131–150.
- 43. Dummer B, Wieland S, Lindner B. Self-consistent determination of the spike-train power spectrum in a neural network with sparse connectivity. Frontiers in Computational Neuroscience. 2014;8(September):1–12.
- 44. Wieland S, Bernardi D, Schwalger T, Lindner B. Slow fluctuations in recurrent networks of spiking neurons. Phys Rev E. 2015;92:040901(R).
- 45. Ostojic S, Brunel N. From Spiking Neuron Models to Linear-Nonlinear Models. PLoS Computational Biology. 2011;7(1):e1001056. pmid:21283777
- 46. Troyer TW, Miller KD. Physiological Gain Leads to High ISI Variability in a Simple Model of a Cortical Regular Spiking Cell. Neural Computation. 1997;9(5):971–983. pmid:9188190
- 47. Murphy BK, Miller KD. Balanced amplification: a new mechanism of selective amplification of neural activity patterns. Neuron. 2009;61(4):635–48. pmid:19249282
- 48. Ahmadian Y, Rubin DB, Miller KD. Analysis of the stabilized supralinear network. Neural computation. 2013;25(8):1994–2037. pmid:23663149
- 49. Jaeger H. The “echo state” approach to analysing and training recurrent neural networks. In: GMD-Forschungszentrum Informationstechnik Report 148; 2001.
- 50. Sussillo D, Abbott LF. Generating Coherent Patterns of Activity from Chaotic Neural Networks. Neuron. 2009;63(4):544–557. pmid:19709635
- 51. Muscinelli SP, Gerstner W, Schwalger T Single neuron properties shape chaotic dynamics in random neural networks. arXiv preprint. 2018;arXiv:1812.06925
- 52. Brown DA. M-Current: From Discovery to Single Channel Currents. In: Slow Synaptic Responses and Modulation. Tokyo: Springer Japan; 2000. p. 15–26.
- 53. Stanley DA, Bardakjian BL, Spano ML, Ditto WL. Stochastic amplification of calcium-activated potassium currents in Ca2+ microdomains. Journal of Computational Neuroscience. 2011;31(3):647–666. pmid:21538141
- 54. Naud R, Marcille N, Clopath C, Gerstner W. Firing patterns in the adaptive exponential integrate-and-fire model. Biological Cybernetics. 2008;99(4-5):335–347. pmid:19011922
- 55. Schwalger T, Fisch K, Benda J, Lindner B. How noisy adaptation of neurons shapes interspike interval histograms and correlations. PLoS Computational Biology. 2010;6(12):e1001026. pmid:21187900
- 56. Ladenbauer J, Augustin M, Obermayer K. How adaptation currents change threshold, gain and variability of neuronal spiking. Journal of Neurophysiology. 2013;111(5):939–953. pmid:24174646
- 57. Richardson MJE, Brunel N, Hakim V. From Subthreshold to Firing-Rate Resonance. Journal of Neurophysiology. 2003;89(5):2538–2554. pmid:12611957
- 58. Brunel N, Hakim V, Richardson MJE. Firing-rate resonance in a generalized integrate-and-fire neuron with subthreshold resonance. Physical Review E. 2003;67(5):051916.
- 59. Ladenbauer J, Augustin M, Shiau LJ, Obermayer K. Impact of adaptation currents on synchronization of coupled exponential integrate-and-fire neurons. PLoS Computational Biology. 2012;8(4). pmid:22511861
- 60. Augustin M, Ladenbauer J, Obermayer K. How adaptation shapes spike rate oscillations in recurrent neuronal networks. Frontiers in Computational Neuroscience. 2013;7:9. pmid:23450654
- 61. Schwalger T, Lindner B. Patterns of interval correlations in neural oscillators with adaptation. Frontiers in Computational Neuroscience. 2013;7:164. pmid:24348372
- 62. Laing CR, Chow CC. A Spiking Neuron Model for Binocular Rivalry. Journal of Computational Neuroscience. 2002;12(1):39–53. pmid:11932559
- 63. Naud R, Gerstner W. Coding and Decoding with Adapting Neurons: A Population Approach to the Peri-Stimulus Time Histogram. PLoS Computational Biology. 2012;8(10):e1002711. pmid:23055914
- 64. Pozzorini C, Naud R, Mensi S, Gerstner W. Temporal whitening by power-law adaptation in neocortical neurons. Nature Neuroscience. 2013;16(7):942–948. pmid:23749146
- 65. Augustin M, Ladenbauer J, Baumann F, Obermayer K. Low-dimensional spike rate models derived from networks of adaptive integrate-and-fire neurons: Comparison and implementation. PLoS Computational Biology. 2017;13(6):1–46.
- 66. Sjöström PJ, Turrigiano GG, Nelson SB. Rate, Timing, and Cooperativity Jointly Determine Cortical Synaptic Plasticity. Neuron. 2001;32(6):1149–1164. pmid:11754844
- 67. Ko H, Hofer SB, Pichler B, Buchanan KA, Sjöström PJ, Mrsic-Flogel TD. Functional specificity of local synaptic connections in neocortical networks. Nature. 2011;473(7345):87–91. pmid:21478872
- 68. Martí D, Brunel N, Ostojic S. Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks. Physical Review E. 2018;97(6):062314. pmid:30011528
- 69. Mastrogiuseppe F, Ostojic S. Linking Connectivity, Dynamics, and Computations in Low-Rank Recurrent Neural Networks. Neuron. 2018;99(3):609–623.e29. pmid:30057201
- 70. Sussillo D. Neural circuits as computational dynamical systems. Current Opinion in Neurobiology. 2014;25:156–163. pmid:24509098
- 71. Barak O. Recurrent neural networks as versatile tools of neuroscience research. Current Opinion in Neurobiology. 2017;46:1–6. pmid:28668365
- 72. Nicola W, Clopath C. Supervised learning in spiking neural networks with FORCE training. Nature Communications. 2017;8(1):2208. pmid:29263361
- 73. Bellec G, Salaj D, Subramoney A, Legenstein R, Maass W. Long short-term memory and learning-to-learn in networks of spiking neurons. arXiv preprint. 2018;arXiv:1803.09574.
- 74. Ostojic S. Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons. Nature Neuroscience. 2014;17(4):594–600. pmid:24561997