Replica-mean-field framework for spiking networks.

In principle, the RMF approach can apply to any network dynamics whereby elementary network constituents—here neurons—interact via spikes. By spikes, we mean that interactions come in the form of precisely time-stamped, stereotypical, transient impulses. Fig 1a shows an example of networks evolving in response to spiking interactions. In such finite-size networks, spiking interactions generally introduce complex dependencies between neurons, with non-trivial correlation structure. Because of these complex dependencies, almost all finite-size spiking networks are analytically intractable. To circumvent this limitation, classical modeling approaches approximate the dynamics of interest via simplifying mean-field limits with trivial correlation structure, i.e., with independently firing neurons [19–22]. However, these limits are obtained by considering approximating networks of infinite size, whereby each neuron interacts with an infinite number of neurons, but via vanishingly small interactions. Thus, classical mean-field approaches erase the neural variability originating from the finite size of interactions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. RMF models.

Panel (a) shows the original networks of K = 3 neurons. Panel (b) represents the finite-replica model with r = 4 replicas. When a neuron spikes (e.g., green neuron), it interacts with downstream neurons sampled uniformly at random across replicas. Panel (c) shows schematically that the RMF models are obtained in the limit of an infinite number of replicas and represent infinite-size physical models supporting RMF dynamics. This figure is reproduced from [52] with permission.

https://doi.org/10.1371/journal.pcbi.1010215.g001

The purpose of the RMF approach is to better capture neural variability by performing a modified mean-field approximation that preserves finite-size effects. Specifically, we design the RMF approach in [23, 28] so that neurons still emit spikes in response to a variable but discrete number of inputs, each with finite size, as opposed to being subjected to a deterministic average drive. In short, these RMF dynamics are obtained by assuming that each neuron receives inputs from other neurons via independent Poisson processes. This amounts to assuming that interactions occur as randomly as possible, i.e., with trivial dependencies, given that these interactions still take the form of variable, discrete, spiking events. A priori, it is unclear why such a simplifying assumption, commonly referred to as the Poisson hypothesis in network theory [48, 49], would lead to the well-posed dynamics of some limit physical system. Indeed, even when driven by Poissonian independent inputs, spiking neuronal models generally have non-Poissonian outputs. In other words, the Poissonian regime does not naturally stabilize in finite-size dynamics.

Fortunately, it turns out that a simple replication process allows one to build physical limit networks supporting RMF network dynamics. This replication process produces enlarged networks made of R copies of the original network, each with K neurons indexed by i, 1 ≤ i ≤ K. In such R-replica networks illustrated in Fig 1b, when a neuron (i, r) of type i spikes from a replica r, 1 ≤ r ≤ R, it interacts with neurons (j, q_j), j ≠ i, according to the same rule as in the original dynamics, except that each target replica q_j is drawn uniformly and independently across replicas. Thus, the construction of RMF networks relies crucially on the possibility to precisely time interactions in spiking networks as it allows for the randomized routing of interactions across replicas. RMF networks are obtained by taking the limit of an infinite number of replicas R → ∞ as depicted in Fig 1c. In this limit, neurons become independent as suggested by the fact that their probability to interact over a finite period of time vanishes as 1/R. Accordingly, the law of rare (interaction) events [50] suggests that when collected across replicas, inputs to a given neuron (i, r) from neurons of type j are received with the same interaction size μ_ij according to a Poisson process. Thus, in RMF networks, neurons are allowed to individually deviate from being Poisson spike generators because randomizing interactions across replicas guarantees aggregated Poissonian spiking deliveries. These intuitive arguments can be checked numerically and shall hold for all spiking networks. In this work, we focus on rate models called intensity-based spiking networks, which comprise a large class of models for which we rigorously established the Poisson hypothesis in [51].

Mean-field limits for a simple intensity-based spiking network.

In general, the activity of a K-neuron spiking network can be modeled as a K-dimensional stochastic point process {N_i(t)}_1≤i≤K, where t denotes time and i is the neuron index [53, 54]. For each neuron i, the component N_i(t) is specified as the counting process registering the spiking occurrences of neuron i up to time t. In other words, , where denotes the full sequence of spiking times of neuron i and we label spikes so that t_i,0 ≤ 0 < t_i,1 for all 1 ≤ i ≤ K by convention. denotes the indicator function of set A, where if x is in A and if x is not in A. The simplest instance of such counting processes N_i(t) is the Poisson process for which spike generation is governed by a deterministic, instantaneous, firing rate function λ_i(t). Informally, the rate λ_i(t) determines the probability of finding a spike in the infinitesimal time interval (t, t + dt] as . A shortcoming of Poisson processes is that they do not allow for stochastic spiking inputs to individually shape the instantaneous firing rate, as should be the case in realistic stochastic spiking models. To address this point, the instantaneous firing rate λ_i(t) must be modeled as a stochastic process, whose probability law depends on the past states of the network. Formally, this corresponds to defining λ_i(t) as the stochastic intensity of the point process N_i(t) [55, 56]. Then, specifying the dynamics of the corresponding intensity-based networks consists in mechanistically relating the joint law of {λ_i(t)}_1≤i≤K to the past spiking history of the network {N_i(s)}_{1≤i≤K,s≤t}, possibly given some initial condition {λ_i(0)}_1≤i≤K.

In the absence of a reset mechanism, the simplest mechanistic intensity-based models are perhaps the celebrated self-excited linear Hawkes processes [57, 58]. These processes are specified by the linear integral equations (1) where we only consider excitatory interactions: μ_ij ≥ 0. As they only involve exponential integrands with neuron-specific time constants τ_i, the above integral equations are equivalent to the perhaps more familiar stochastic differential equation (2) which clearly separates the relaxation term and the interaction terms. According to this equation, in the absence of interactions, the rate λ_i(t) relaxes to its base value h_i with time constant τ_i, whereas λ_i(t) instantaneously increases by an amount μ_ij whenever neuron j ≠ i spikes. Given certain stability conditions precluding rate explosion, it is known that linear Hawkes models admit stationary dynamics. For such dynamics, the stationary rates satisfy a system of linear equations (3) This system can be derived by taking the stationary expectation of (1), using the fact that we have at stationarity. Then, the stability condition amounts to imposing that the above system admits a set of nonnegative rates {β_i}_1≤i≤K as solution.

Albeit linear, Hawkes stationary processes have nontrivial correlation structures due to finite-size interactions. In view of this, let us use Hawkes processes as a concrete example to compare approximations via the TMF limit and the RMF limit. In the TMF limit, each neuron is subjected to a deterministic drive obtained by averaging over an infinite-size network via the law of large numbers. Concretely, this means that one should substitute β_jt for the original Hawkes processes N_j(t) in (1). This implies that the rates λ_i are also deterministic, and consistently setting their constant value to be β_i directly yields the exact system of equations (3). However, all variability of the rates λ_i is erased in the process. By contrast, the RMF limit provides us with a Poissonian approximation obtained by substituting independent Poisson processes P_j(t) with rate β_j for the original Hawkes processes N_j(t) in (1). Concretely, this means that (4) where by stationarity, there is no lack of generality to choose λ_i = λ_i(0). Thus, by contrast with the TMF limit, the RMF limit provides us with a stochastic approximation for the rates. Taking the stationary expectation of (4) also yields the exact system of equations (3) for the mean firing rates, whereas the rate variability can be estimated as (5) by virtue of Campbell’s formula for Poisson processes [55, 56]. The above simple treatment illustrates the distinction between TMF and RMF limits for linear Hawkes dynamics. Both TMF and RMF limits recover the exact system of equations (3). This is a special feature of linear Hawkes dynamics, and for more general intensity-based network dynamics, we shall expect TMF or RMF limits to only yield approximate mean-field relations. However, only the RMF limit provides us with consistent variability estimates, obtained at the cost of imposing a trivial correlation structure via the Poisson hypothesis. Here our goal is to quantify neural variability in the RMF limits of certain class of nonlinear intensity-based spiking networks, referred to as linear-exponential reset (LER) networks.

Finite-size linear-exponential reset neuronal dynamics.

The LER neuron model is based on the class of linear-nonlinear Poisson (LNP) neurons [59–61], with the addition of individual post-spiking reset rules [62, 63]. Such models of LNP neurons with reset were formally introduced as the Galves-Löcherbach models [64]. In the LER network, the stochastic intensity λ_i(t) is exponentially related to a neuron-specific internal variable x_i(t), which integrates past neuronal interactions. Specifically, we have (6) where h_i and a_i are positive constants. The dynamics of the internal variable x_i(t) is as follows. Whenever neuron i spikes, x_i(t) instantaneously resets to zero, erasing all memory effects at the individual neuron’s level. At the same time, neurons j ≠ i register the instantaneous deliveries of Dirac-delta impulses in their internal variables x_j(t) via synaptic strengths μ_ij. This registration proceeds in a leaky fashion so that x_i(t) obeys the linear stochastic differential equation (7) where τ_i denotes the relaxation time of the neuronal internal variable to zero. Observe that the last integral term in Eq (7) implements the post-spiking reset rule to zero. Moreover, note that in view of Eq (7), h_i appears as the (nonzero) base spiking rate of neuron i, whereas a_i models its excitability.

Given (6), Eq (7) fully specifies the dynamics of LER networks in the same fashion as (2) does for linear Hawkes processes. It is also instructive to specify LER dynamics via a stochastic integral equation akin to (1), which directly relates the stochastic intensities {λ_i(t)}_1≤i≤K to the past spiking history of the network {N_i(s)}_{1≤i≤K,s≤t}. Such an integral equation reads (8) where l_i(t) is the last time neuron i spikes before t: l_i(t) = sup_k{t_i,k ≥ 0|t_i,k ≤ t}. A merit of the above equation is to exhibit the finite memory of individual neuronal dynamics with reset. To see this, observe that λ_i(t) only depends on the past spiking history of the network starting from l_i(t), i.e., on . Another merit of Eq (8) is to reveal that the reset mechanism implements a stochastic feedback that effectively precludes rate explosions. This follows from the fact that the last spiking time l_i(t) is a stochastic variable, even in mean-field limits, and from the fact that the larger the rate, the shorter the memory length over which spiking inputs are integrated. This is because the memory length is bounded by the interspike intervals t_i,k+1 − t_t,k, which on average, scales in inverse proportion to the mean firing rate: . One final merit of Eq (8) is that it can generalize to spiking interactions of any shapes and to arbitrary rate dependency. However, in view of obtaining tractable RMF limits, we restrict ourselves to exponential nonlinearities, for which the specification via stochastic differential Eq (7) is possible.

To sum up in more concrete terms, we model a neural network as a directed weighted graph whose nodes are neurons and whose directed edges are synaptic weights μ_ij from neuron j to i. Within the network, the state of a neuron i is given by its internal variable x_i, which determines the neuronal instantaneous firing rate λ_i via the exponential relation Eq (6). The individual neuronal dynamics consists of three components: (i) interaction, (ii) reset, and (iii) relaxation. (i) When neuron i fires, x_j, j ≠ i, updates to x_j + μ_ji. (ii) At the same time, x_i resets to zero. (iii) In between spikes, each x_i(t) relaxes toward zero with time constant τ_i. Thus specified, the dynamics of LER networks defines a continuous-time Markov chain. Considerations from the regenerative theory of Markov chains show that in the presence of relaxation, i.e., whenever max_i τ_i < ∞, LER network dynamics are ergodic [23, 65]. This means that collecting the typical states of the networks defines a unique stationary distribution, independent of initial conditions.

Replica mean-field limit for networks of exponential neurons.

A benefit of considering LER models is that they naturally accommodate inhibition by allowing for negative synaptic weight μ_ij < 0. This is by contrast with linear models considered in [23] for which including inhibition conflicts with the required nonnegativity of the rate functions λ_i(t). Unfortunately, just as for their linear counterparts, an exact analytical treatment of the finite-size LER models hinders on the complex structure of the activity correlations. This limitation motivates considering LER networks in the RMF limit, which comprises an infinite number of replicas of the original systems [52]. By original system, we mean the K-neuron network with LER dynamics described by Eq (7). The R-replica model is obtained by considering that a neuron i within each replica r, 1 ≤ r ≤ R, follows the same autonomous dynamics as that of neuron i in the original system. However, the difference with the original system is that upon spiking, a neuron i from replica r interacts with neurons (j, q), j ≠ i, via the original weights μ_ji but in replicas q, chosen uniformly at random. Formally, this corresponds to the following set of stochastic equations (9) where for all s ≥ 0, 1 ≤ r, q ≤ R, 1 ≤ i ≠ j ≤ K, v_r,ij(s) are independent random variables uniformly distributed over {1, …, R}. These random variables are routing addresses specifying that when neuron (j, r) spikes at time s, it targets a neuron of type i in replica v_r,ij(s).

Intuitively, the randomization of interactions present in Eq (9) degrades dependences between neurons and across replicas. For large number of replicas, i.e., in the RMF limit, the neurons become asymptotically independent. Moreover, each neuron of type i asymptotically receives inputs from neurons of type j ≠ i with Poissonian statistics across replicas. For often being only conjectured, the emergence of this simplified limit dynamics is referred to as the “Poisson Hypothesis” in network theory [48]. The Poisson Hypothesis was recently established rigorously for generic RMF limits, including LER models [51]. Under the Poisson Hypothesis, in RMF limits of LER networks, a neuron i from a representative replica admits the effective dynamics given by where {P_j(t)}_j≠i denote independent Poisson processes with stationary firing rate β_j. Thus, at fixed network structure, the dynamics x_i(t) only depends on the network background activity via the rates {β_j}_j≠i. To be consistent under the assumption of stationarity, these rates must collectively satisfy the system of equations (10) where the notation emphasizes that β_i is evaluated as a function of the rates {β_j}_j≠i. In the following, we will refer to as a rate-transfer function. The fact that firing rates characterize alone the stationary distribution of the neural network is a feature of RMF limits. In fact, specifying RMF limits for LER networks consists in solving the self-consistent system Eq (10). This is one of the main goals of this work.

Functional characterization via delay differential equations.

Heretofore, we have only considered RMF limits and their associated self-consistent system Eq (10) formally. To exploit RMF limits computationally, one must find explicit forms for the rate-transfer functions featured in Eq (10). Determining these explicit forms is the main technical challenge of the RMF approach and is the topic of the next section. Here, as a first step toward this goal, we establish functional relations between the stationary rates β_i via probabilistic arguments. Specifically, we exploit the rate-conservation principle for point processes [66, 67] to exhibit a system of delay differential equations (DDEs) featuring the rates β_i.

In a nutshell, the rate-conservation principle states that in the stationary regime, any real-valued function of the network states reaches an equilibrium where there is a balance between its rate of increase and its rate of decrease. By virtue of the Poisson Hypothesis for RMF limits, we can apply the rate-conservation principle to any function of the internal states x_i(t) independently. A natural choice is to consider exponential function , whose stationary expectation defines the moment-generating function (MGF) . The MGF L_i fully specifies the stationary distribution of x_i(t) and is a well-behaved analytical function when it exists on an open interval containing zero. In the following, we assume that the MGF L_i always exists in the RMF limit. By this, we mean that the MGF L_i remains finite on an open interval containing zero, which implies analyticity in zero so that x_i(t) admits moments of all order.

As a process, satisfies a stochastic equation which can be deduced from Eq (7) as (11) The three consecutive integral terms in the RHS correspond to continuous relaxation, independent Poisson bombardments, and post-spiking reset, respectively. At stationarity, we have , so that the rate-conservation principle implies that the RHS has zero stationary expectation. In turn, we can interpret the stationary expectation of the three integrals in term of the MGF L_i (see S1 Appendix for derivation). Ultimately, the rate-conservation principle takes the form of the following linear DDE (12) where we have introduced for brevity. In the above equation, the nonlocal term L_i(u + a_i) is due to the exponential form of Eq (6) for the rate λ_i(t) as interpreting the expected reset term in Eq (11) involves evaluating (13) In the framework of perturbation theory, Eq (12) can be viewed as a singularly perturbed delay differential equation, whose perturbation parameter h_i, unlike many other more common cases, is on the delay term. To see this, note that , so that Eq (12) reads (14) where h_i appears as the coefficient of a forward discrete derivative. In the limit of h_i → 0, Eq (14) becomes an analytically solvable, first-order homogeneous ODE. When h > 0, the presence of a delay term drastically changes the nature of the problem at stake, at least mathematically. Because of this drastic change, one needs to resort to techniques from singular perturbation theory to numerically solve Eq (14). In this regard, observe that Eq (14) also exhibits the relaxation rate 1/τ_i as a singular perturbation parameter. Indeed, in the limit of τ_i → ∞, Eq (14) becomes a pure delay equation, with no relaxation component. However, by contrast with the ODE recovered when h_i → 0, the delay equation obtained when τ_i → ∞ resists closed-form resolution. For this reason, h_i will play the central part as a perturbation parameter.

The main caveat to solving Eq (14) for generic LER models (1/τ_i, h_i ≠ 0) is that the nonlocality of the DDEs Eq (12) precludes a direct analytical treatment, while posing problems with respect to solution uniqueness. To see this, observe that if one interprets the variable u as a time parameter, the nonlocal term L_i(u + a_i) formally introduces a negative delay −a_i. Due to the negativity of this delay, one can only solve the DDE Eq (14) of the form x′(u) = f(x(u), x(u + a_i)) backward, i.e., for decreasing value of u. Moreover, such solutions are only unique given the knowledge of some initial conditions on an interval of duration a_i. However, there is no natural notion of initial conditions at our disposal. Luckily, we will be able to address this caveat in a probabilistic setting. Specifically, we will see that there is a natural representation for a probabilistically interpretable solution to Eq (14), with no need to specify any initial conditions.

Self-consistent equations via resolvent formalism.

Our goal is to characterize the unique MGF solution to the DDE Eq (14). Achieving this goal will allow us to give an explicit form to the rate-transfer functions formally defined by Eq (10). For simplicity, we omit all the neuronal indices whenever possible in the following. For instance, we will denote the output stationary firing rate of neuron i by β when unambiguous. Our strategy is to adapt the resolvent formalism to our delayed framework [68] in three steps.

First, we consider the forward discrete derivative term (L(u + a) − L(a))/u as a known inhomogeneous term so that we can view Eq (14) as a linear ODE about L. The method of the variation of parameters yields integral forms for L involving its shifted version L(⋅ + a), but also some undetermined initial condition. We resolve the latter indeterminacy by selecting the only solution taking value L(a) = β/h, as required by the definition of the stationary rate: . This yields the following integral equation (15) where q denotes the homogeneous solution to Eq (12) excluding the forward discrete derivative term: .

Second, in view of Eq (15), we define the auxiliary function H(u) = (L(u + a) − L(a))/u. Substituting H(u) into Eq (15), we obtain the following integral equation (16) where the delayed nature of the problem appears via the nonlocal integration upper bound. Observe that the inhomogeneous term in Eq (16) is analytic, whereas the integral term is analytic whenever H is analytic. Eq (16) is the basis for adapting the resolvent formalism to our delayed framework. To see this, let us define the sequence of iterated kernels (17) with Q₀(u) = (q(u + a) − 1)/u. Then, we can formally write a solution to Eq (16) as the series (18) where h appears as a perturbation parameter via the dimensionless quantity hτ. Note that the kernels Q_m are independent of h but also of β, the yet-to-be-determined output firing rate. Note also that as the Q_m are analytic in u, H is an analytic function around zero as soon as the series converges uniformly on an open disk containing zero. We conjecture that this is always the case. This is supported by simulations which show that the probability density function of the internal variable x is decaying superexponentially. Such superexponential decay implies that the moment generating function of x, as well as the function H, should be analytical (see S1 Fig).

Third, we specify β as a function of the input parameters by exploiting the normalization constraint of the MGF: L(0) = 1. From the definition of H above, we have that −aH(−a) = L(0) − L(a) = 1 − β/h. Thus, together with Eq (18), we must have (19) where the dependencies on the input rates β_j, j ≠ i, are mediated by the kernels Q_m. Observe that in the absence of inputs, i.e., for V(u) = 0, we have q(u) = 1 so that all the terms Q_m(−a) = 0. Thus, in the absence of inputs, we recover consistently that β = h, the spontaneous spiking rate. In the presence of inputs, Eq (19) determines the rate-transfer function of a neuron in a feedforward network. When considering a recurrent network, these rate-transfer functions define the sought-after self-consistency equations Eq (10), which must be jointly satisfied. These equations take the explicit forms: (20) where we have indexed all neuron-specific quantities and highlighted the dependencies on the input rates β_j, j ≠ i. In the following, we will refer to the quantities computed by Eq (20) as “RMF calculation”. We discuss the numerical methods used to perform the RMF calculation in the next section.

Nonlinear corrections and reset mechanism.

The derivation of Eq (14) shows that its delayed nature is due to the presence of a post-spiking reset mechanism. This post-spiking reset mechanism becomes irrelevant when h = 0 as Eq (14) simplifies to the solvable homogeneous ODE: uL′(u) = V(u)L(u). Then, the normalization condition L(0) = 1 imposes that L(u) = q(u)/q(0), leading to β = hL(a) = h/q(0). The abrupt loss of the delayed terms in Eq (12) for h = 0 reveals the latter solution as a distinguished limit for L when h → 0. In that respect, one can check that h/q(0) is precisely the first-order term in Eq (20) and reads explicitly (22) where w_j is the effective synaptic weight of upstream neuron j. The nontrivial dependence of the effective weights w_j on the original synaptic strengths μ_j follows from including finite-size effects in the RMF framework [28, 52]. For small synaptic weights μ_j ≪ 1/a, we recover the classical mean-field regime for which the linear scaling w_j ≃ τμ_j a holds. We will later see that even for moderate values of μ_j, finite-size effects can have a substantial impact on evaluating the stationary moments of the dynamics.

Eq (22) reveals that any nonlinear correction to the first-order estimate is due to including the reset mechanism. For Eq (20) being a singular expansion, these higher-order corrections do not always yield a convergent series. However, we expect the first-order term to be a valid approximation as long as the reset timescale, i.e., the mean interspike interval 1/β, remains large compared with the relaxation time τ: τ ≪ 1/β. This is because in this regime, fast relaxation erases the slower influence of the reset mechanism in shaping the distribution of the internal variable x. By contrast, in the regime of moderately large excitation 1/β ≃ τ, the reset mechanism starts to significantly impact the dynamics of x and to dampen the first-order exponential dependence of β on the input rates β_j. In principle, neurons can fire at a rate up to 200Hz for a leak time constant τ larger than 10ms. Thus, if one interprets the internal variable x as a proxy for the membrane voltage, the biophysically relevant range of neural activity includes values in the intermediary regime βτ ≃ 1, for which nonlinear corrections are necessary. Exploring this regime requires using the Padé approximants associated to the divergent series.

We confirm numerically the above discussion by considering the simplest excitatory neuronal model, whereby a neuron is subjected to an input of rate β_e via the positive synaptic weight μ_e. Fig 5a and 5b quantify nonlinear corrections to the first-order prediction by plotting β/h as a function of h at fixed input conditions. As expected, the no-reset limit β = h/q(0) holds for h → 0. By contrast, in the opposite limit h → ∞, the frequent resets due to spontaneous spiking erase the impacts of the relaxation as well as the inputs in between spikes, so that the spontaneous spiking emission dominates the dynamics: β → h. By comparison with Fig 5a and 5b shows that the stronger excitatory drive, the larger the correction to the first-order terms. We mark the transition between the two asymptotic regimes by the mid-point h_1/2 such that β (h_1/2) = h(1 + 1/q(0))/2. To explore the domain of validity of the first-order expansion, we then check that at the mid-point value h_1/2, we have βτ ≃ 1, even for eventually large nonlinear correction. To this end, we perform a systematic RMF calculation to represent βτ for h_1/2 as a function of the input rate β_e and the synaptic weight μ_e. Fig 5c shows that the stronger the input, the smaller value of βτ. However, this dependence is weak and βτ ≃ 1 holds throughout. This confirms our analysis about the domain of validity of the first-order approximation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Nonlinearity of firing rates with post-spiking reset mechanism.

In panel (a) and (b), the log-linear plots of β/h v.s. h show the nonlinearity resulted from the post-spiking reset mechanism. In panel (a), the neuron is subjected to a single excitatory input with β_e = 1 kHz and μ_e = 1, while in panel (b), the input is of β_e = 1.5 kHz and μ_e = 2.5. Panel (c) is the contour plot of the value of βτ at mid-point h_1/2, where excitatory input rate β_e and strength μ_e are varied. The contours are equidistant with respect to the value of βτ. For the simulated data, we run the event-driven simulation until 400 spiking events for the single neuron are accumulated. This process is repeated 32 times. During each repetition, we record the total time T_k(k = 1, ⋯, 32). The mean firing rates of the neuron are then computed by . The standard deviations of the rates (indicated by error bars) over these 32 repetitions are computed by taking the square root of . Parameters: a = 0.1, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g005

Moment analysis of the neuronal response.

The crux of the RMF approach is to capture the stationary dynamics of a neuron via a parametric probabilistic model. Moreover, this model admits the output rate β as a sufficient statistics, assuming the biophysical parameters and the input rates known. This means that given these assumptions, all the moments of the stationary neuronal dynamics can be deduced from knowing β. For LER neurons, the moments of interest are those of the stochastic intensity, , n > 1, and those of the internal variable x, , n > 0. With knowledge of β, M_n(x) can also be computed efficiently for all n > 0 via expansions akin to Eq (20) (see S1 Appendix). In turn, this allows one to estimate M_n(λ), n > 1, by truncation of the series (23)

Here, we demonstrate that the above RMF computational framework accurately quantifies the response of LER neurons subjected to Poissonian bombardments. We proceed by comparison with exact, event-driven, Monte-Carlo simulations under various driving conditions and for biophysically-relevant parameter values [33, 34]. Bear in mind that in all cases, our computational results are obtained incomparably faster than those estimated via Monte-Carlo simulations. We will see that such a computational advantage leverages to neural networks in the next section. In demonstrating our point, we will discuss the general features of the LER neuronal response in light of the competition existing between the two timescales at play: the relaxation time τ, and the mean interspike interval 1/β. We will also consider closed-form approximations for various regimes of activity.

For a feedforward neuron, the input-rate-dependence of β is encoded via the rate-transfer function Eq (20). In Fig 6, we explore this rate-transfer function numerically to reveal that—perhaps surprisingly—LER neurons essentially behave as stochastic rectifier linear units (ReLUs). In addition to our RMF calculations, we consider three types of approximations for comparison. At a low spiking rate, the relaxation timescale dominates, e.g., τ ≪ 1/β < 1/h, and we utilize the first-order, no-reset approximation. At a high spiking rate, e.g., τ ≃ 1/β, we consider two heuristically-derived approximations in the TMF limit and with or without relaxation. Both approximations are obtained via a simple probabilistic argument (see S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Rate-transfer functions: Mean firing rates β and mean internal variables .

For panel (a) and (b), the neuron is subjected to an excitatory input with varying rates β_e from 0 to 10 kHz and fixed strength μ_e = 1.0. In panel (a), the top-left inset shows the ratio β/β_TMF in a wider range of input rates. For panel (c) and (d), the neuron is subjected to an inhibitory input with varying rates β_i from 0 to 5 kHz and fixed strength μ_i = −1.0. The inset in panel (c) is in logarithmic scale to show its exponential dependence. For the simulated data, we run the event-driven simulation until 400 spiking events for the single neuron are accumulated. This process is repeated 32 times. During each repetition, we record the total time T_k(k = 1, ⋯, 32). The mean firing rates of the neuron are then computed by . The standard deviations of the rates (indicated by error bars) over these 32 repetitions are computed by taking the square root of . Meanwhile, we record the entire time series of the internal variable x, with which we compute the mean of x. The standard deviations of the internal variable x (indicated by error bars) are computed over these 32 repetitions. Parameters: h = 1 Hz, a = 0.1, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g006

In Fig 6a, when excitation dominates, the zero-order exponential approximation breaks down when β exceeds 10Hz, which happens at about 2kHz for the considered synaptic strength. Note that the input rates quantify the frequencies of synaptic activations so that 2kHz corresponds to, e.g., 20 upstream neurons firing at 100 Hz or 200 upstream neurons firing at 10Hz. By contrast, the RMF calculations accurately predict the quasilinear input-rate dependence of β (up to convergence failure). At high drive β_e, the output rate adjusts so that on average, the β_e/β received inputs are canceled by a single reset over the timescale 1/β. This balance generically leads to a weakly sublinear rate dependence in the RMF limit as well as in the TMF limit. In the TMF limit, we heuristically establish that at large β_e, β_TMF ≃ aβ_e μ_e/ln(1 + aβ_e μ_e/h). This TMF approximation produces the right scaling but only becomes accurate for exceedingly high rates, when finite-size effects become negligible (see inset for the ratio β/β_TMF). In Fig 6b, we show that the mean internal variable mirrors the behavior of β after logarithmic compression. A low-rate linear growth is followed by a logarithmic behavior at high drive. In Fig 6c and 6d, when inhibition dominates, the neuron seldom spikes so that τ ≪ 1/β, and the reset becomes irrelevant. Then, long interspike intervals allow for the integration of many inputs so that finite-size effect can be neglected as in the heuristic TMF limit. As a result, all approximations perform accurately.

Fig 7 compares the second-moment predictions deduced from our various approximations. Comparing Fig 7a and 7b with Fig 7c and 7d consistently shows that predicting the variability in both λ and x requires RMF calculation when excitation dominates whereas the zero-order, no-reset approximation suffices when inhibition dominates (see S1 Appendix). Aside from this core observation, two remarks are worth making: First, we remark in Fig 7a and 7b that the RMF calculations interpolate well between the low-rate and high-rate variability regimes. When the input rate is low, the variability mainly comes from the relaxation as in the no-reset limit. When the neuron spikes more frequently in response to strong drive, the reset mechanism becomes the dominant source of variability. The latter can be captured asymptotically by the TMF limit. Second, we remark in Fig 7c and 7d that the TMF approximations fail to capture neural variability for both quantities, even though the mean response is accurately predicted in the inhibitory case. This is because TMF limits inherently neglect finite-size effects by assuming that the stochastic intensity varies deterministically in between spikes. Such an assumption leads to drastic underestimation of the variability in the inhibition-dominated regime, at least before the neuron becomes virtually silent.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Rate-transfer functions: Standard deviation of the neuron firing rates Std(λ) and of the internal variables Std(x).

For panel (a) and (b), the neuron is subjected to an excitatory input with varying rates β_e from 0 to 10 kHz and fixed strength μ_e = 1.0. For panel (c) and (d), the neuron is subjected to an inhibitory input with varying rates β_i from 0 to 5 kHz and fixed strength μ_i = −1.0. For the simulated data, we run the event-driven simulation until 400 spiking events for the single neuron are accumulated. This process is repeated 32 times. During each repetition, we record the entire time series of the internal variable x, with which we can compute the moment of x for any given order. Std(λ) is computed from Eq (23) with cut-off order 15. Std(x) is computed by taking the square root of . The standard deviations of the quantities (indicated by error bars) are computed over these 32 repetitions. Parameters: h = 1 Hz, a = 0.1, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g007

Finite-size effects and balanced regime.

As mentioned above, a benefit of the RMF framework is that it enables the study of finite-size effects. In TMF approximations, finite-size effects are erased in the process of scaling interactions in the limit of infinite-size networks. As a result of this process, synaptic weights only appear as multipliers of the input rates in the rate-transfer functions [23]. This is not the case in RMF limits as shown by the nontrivial dependence of the effective weights w_j, which act as rate multipliers, on the original weight μ_j in Eq (22).

We quantify these finite-size effects by elucidating the dependence of the mean and variance of the LER neuronal response on the synaptic weights at fixed overall mean drive. By this, we mean that we jointly vary the numbers K_e, K_i and the weights μ_e, μ_i of the synapses so as to maintain the overall levels of excitation and inhibition, denoted by E = K_e μ_e and I = K_i μ_i. Specifically, we compare the three following RMF approximations with their TMF counterparts: (1) K_e excitatory inputs alone; (2) K_i inhibitory inputs alone; (3) K_e excitatory inputs balanced by K_i inhibitory inputs with K_e = K_i and E = I. The latter balance condition is a core assumption to a broad class of models accounting for the maintenance of neural variability in the limit of infinite-size networks, the so-called balanced network models [72].

Fig 8 shows the RMF calculations and the simulated values for the mean and standard deviation of the internal variable with biologically relevant parameters. In all cases, our RMF predictions coincide with simulated results to numerical error. As expected for excitatory inputs (Fig 8a and 8a’), the input variability tends to average out for large number of inputs K_e. Accordingly, we observe that the TMF limit is accurate in this regime. By contrast, when K_e is small and finite-size effects are no longer negligible, the TMF limit marginally overestimates the mean, while underestimating the standard deviation by about twofold when K_e = 7. This is consistent with the TMF limit erasing variability, even in the presence of a stochastic reset. When driven by purely inhibitory inputs (Fig 8b and 8b’), neurons remain silent for long period over which the variability in the inputs averages out. As a result, the TMF limit performs well for all input number K_i. Nevertheless, the neural variability is still largely underestimated. This is again because TMF generally erases input variability, with a more drastic effect with inhibitory inputs. For instance, the TMF limit yields a seventeenfold reduction of the variability for K_i = 7. The overperformance of RMF over TMF is magnified in the balanced case (Fig 8c and 8c’). Under this condition, the TMF limit gives zero mean and variance for the internal variable as there is no net drive: ∑_j β_jμ_j = 0. However, the corresponding finite-size system is dominated by the next order fluctuation, yielding nonzero mean and variability. Actually, one can check that for balanced conditions, the mean firing rate increases in keeping with the standard deviation of x, and against the mean of x, as K_e = K_i decreases. This confirms that the neural response is dominated by input variability in this regime. The slight negative trend of the mean of x is due to the nonlinearity of the exponential function, which biases against upward excursions from zero.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Finite-size effect.

This figure shows the means and standard deviations of the internal variable of a neuron subjected to different configurations of input, varying the number of input channels. For all panels, E = 20 and I = −20, μ_e = E/K_e, μ_i = I/K_i. Panel (a) and (a’): excitation only, β_e = 50 Hz. Panel (b) and (b’): inhibition only, β_i = 50 Hz. Panel (c) and (c’): K_e = K_i = K, β_e = β_i = 50 Hz. For the simulated data, we use the same methods described in the captions of Figs 6 and 7 to compute the values (points) and standard deviations (error bars). Parameters: a = ln(100)/20 ≈ 0.23, h = 1 Hz, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g008

Network activity via fixed-point resolution scheme.

We are now in a position to characterize the dynamics of a recurrent LER network in the RMF limits. Recall that such limit dynamics are approximate versions of the original K-neuron dynamics, which are obtained via randomization of interactions across an infinite number of replicas. As a result of this randomization, within each replica, each neuron i experiences inputs as if delivered according to independent Poisson processes with rate β_j, j ≠ i. The parametrization of the inputs via rates alone is the root cause for the RMF frameworks being computationally tractable. Within a recurrent network, these rates need to be determined by solving the system Eq (20) for self-consistent stationary rates β = {β₁, …, β_K}. By virtue of its interpretation in terms of a physical limit, the RMF system Eq (20) must admit some solutions β^⋆. These solutions can be computed efficiently for LER networks.

Interpreting Eq (20) as a fixed-point problem naturally leads to estimate possible solutions β^⋆ via the naive iterative scheme: . Aside from issues of Padé convergence, we expect this naive scheme to be locally convergent because the rate-transfer functions are weakly sublinear for large rates, which precludes runaway iterations. At the same time, these rate-transfer functions are also strongly supralinear (exponential) at low rate, so that solution uniqueness is not necessarily guaranteed. In practice, we find that whenever the Padé approximants summation converges, the naive scheme always locally converges toward a solution β_n → β^⋆, which may depend on the initial condition β₀. A good choice for the initial condition is of the form β₀ = h + ϵ where ϵ represents a possibly random perturbation. Such initial conditions can be made to fall into the region of Padé convergence, while it is possible to achieve distinct solutions by tuning ϵ (if several solutions exist). We compare the RMF and simulated rates for two network structures with strong connections: In Fig 9a and 9b, the considered network has a dominant sparse feedforward structure, which promotes input independence as in the RMF limit. Accordingly, RMF rate approximations yield accurate predictions. In Fig 9c and 9d, the considered network has a dense recurrent structure, which promotes input correlations in excitatory networks. However, recurrent inhibition appears to maintain input independence and RMF calculation is accurate as well.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of RMF and simulated rates in inhomogeneous networks.

Two networks of 80 excitatory and 20 inhibitory neurons are considered. In panel (a), the network is feedforward with 4 excitatory layers and one last inhibitory layer inhibiting the first layer. The neurons between layers are connected with probability 75% and the synaptic strengths are uniformly random between 3 and 6, as shown in (b). Panel (c) and (d) are for a network with no particular structure. Each pair of neurons is connected with probability 50% and the synaptic strengths are uniformly random between 0 and 4. For the simulated data, we run the event-driven simulation until 40000 spiking events for the entire network are accumulated. This process is repeated 32 times. During each repetition, we record the total time T_k(k = 1, ⋯, 32) and spiking event count S_i,k for each neuron i(i = 1, ⋯, 100). The mean firing rates of neurons are computed by . The standard deviation of the rates (indicated by error bars) over these 32 trials are computed by . Parameters: a = 0.1, h = 5 Hz, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g009

The two networks considered above have a single fixed-point RMF solution β^⋆. Intuitively, this is because the original system has a single “equilibrium” state, so that the corresponding ergodic dynamics is dominated by a single relaxation time. By contrast, metastable systems admit several local “pseudo-equilibria”, leading to multi-timescale dynamics: At small timescales, the dynamics is dominated by the relaxation time to the pseudo-equilibrium it occupies. At large timescales, the dynamics is dominated by sharp transitions between distinct pseudo-equilibria. In the TMF approach, the rate of transitions between pseudo-equilibria vanishes exponentially fast with the size of the system, and the system becomes multistable in the infinite-size limit [35]. We expect a similar picture in the RMF approach. To validate this expectation, we consider the RMF limit for the simplest metastable dynamics, that of a neural-network model alternating between only two pseudo-equilibria. We expect to detect multistability via a transition whereby the RMF system Eq (20) starts admitting several stable solutions. In practice, the stability of a solution can be checked by creating an ensemble of iterations starting from randomized initial values.

Multistable limit of metastable networks.

In principle, elementary computations can unfold in neural circuits by allowing some inputs to gate transitions between distinct output states of the circuit. In noisy neural networks, such gating processes give rise to metastable dynamics [18, 73–75]. Metastability has been studied for neural networks in the TMF limits [35, 76]. Here, we extend the analysis of metastability to the RMF framework with of focus on neural variability. Specifically, we demonstrate that our computational approach can capture the bistable response of a network model used to emulate perceptual rivalry [36].

The structure of the network is shown schematically in Fig 10a. The network comprises two symmetric groups of neurons Group₁ and Group₂, with identical features and symmetric connections. Each group comprises a cluster of excitatory neurons (Exc_1,2) and a cluster of inhibitory neurons (Inh_1,2). The excitatory neurons in Exc_1,2 are fully connected within their own clusters, represented by the self-pointing arrows. The inter-cluster arrows represent full connection between corresponding clusters. For simplicity, we set all the excitatory and inhibitory synaptic strengths to be equal to μ_e and μ_i, respectively. Aside from their within-network interactions, all neurons are subjected to TMF-type inputs of fixed rates and strength with (βμ)_TMF = 1.5 kHz. This corresponds, e.g., to each neuron having 1000 weak synapses of strength μ_TMF = 0.1 activating at a rate of β_TMF = 15 Hz. Accounting for these TMF-type inputs amounts to adding a linear term in the function V(u) → V(u) + (βμ)_TMF u. This hybrid picture is biologically relevant considering recent evidence that only a restricted set of inputs is responsible for neuronal tuning, with most synapses only engaging in background activity [9, 37]. The inclusion of strong mutual inhibition across symmetric groups of neuron shall turn the network into a stochastic bistable switch. In the following, we study such a switch for a small total number of neurons (N = 40 with 10 neurons in each cluster), so that neural variability will be a key determinant of the overall dynamics. The ability to deal with such small systems is one of the core benefits of the RMF framework.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Network structure and simulated time series of internal variable x.

Panel (a) shows that the system has two identical groups (Group_1,2) of neurons with symmetric connections. In each group, there is one cluster of excitatory neurons and one cluster of inhibitory neurons. Each cluster consists of an equal number of K/4 neurons. The connections between clusters are as shown. The synaptic strength is μ_e from any excitatory neuron and μ_i from any inhibitory one. In panel (b), (c) and (d), we plot the time series of x for a representative neuron in Exc₁, with fixed inhibitory strength μ_i = −4.0. The system exhibits: in panel (b), monostability with μ_e = 0.7; in panel (c), fluctuation with μ_e = 1.2; in panel (d), bistability with μ_e = 1.7. Parameters: K = 40, h = 1 Hz, a = ln(100)/20 ≈ 0.23, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g010

We first confirm through simulations that this system can indeed exhibit bistability. Fig 10b, 10c and 10d show the time series of the internal variable x of one representative neuron in Exc₁. In Fig 10b, we simulate with μ_e = 0.7 and μ_i = −4.0. In this case, all activities are strongly suppressed by the inhibition. The system exhibits no bistability. Upon increasing the excitatory strength to μ_e = 1.2 while leaving μ_i unchanged, the system becomes marginally bistable as shown by the enhanced fluctuations observed in Fig 10c. This is an indication that the corresponding deterministic system is on the edge of a dynamical transition or bifurcation. However, such bifurcations are notoriously difficult to detect in a noisy setting [38]. Further increasing μ_e reveals clear alternations between two relatively stable states as shown in Fig 10d for μ_e = 1.7. We refer to these states as the “up” state or the “down” state depending on the mean value of x during these states. Fitting the simulated dynamics via a hidden Markov model allows us to parse out the various dominance periods during which either Group₁ or Group₂ is up [77, 78]. This approach leverages the fact that conditionally to be up or down, the distribution of the internal variable x remains approximately Gaussian.

Next, we check that our RMF framework can capture the bistable switch behavior, and perhaps even detect the bifurcation. To this end, we numerically solve the RMF system Eq (20) obtained for the considered network. As expected, we find two distinct stable solutions for high enough cross-inhibition μ_i. When the RMF limit is multistable, different choices of initial values can lead to distinct solutions. Not surprisingly, if we choose larger initial values for neurons in Group₁ (e.g., β_0,i = h + ϵ, i ∈ Group₁, ϵ > 0), the iteration will converge to a state where Group₁ is in the up state. The thus obtained two sets of rate solutions parametrize two probabilistic models for two up and down metastable states.

In Fig 11, we compare exact simulation results in the finite-size network (data points) to our theoretical calculations in the infinite-size RMF limit (solid curves). The branching behavior of , and clearly indicates a transition from a monostable regime to a bistable regime. Our calculation is in good agreement with the simulated values outside of the shaded region. In the shaded region, where the bifurcation between monostability and bistability occurs, it is numerically ill-posed to distinguish between up and down states. The system does not persist in either states long enough for us to accurately compute the desired quantities. Fortunately, and as in the TMF limit, our theoretical calculation offers to precisely pinpoint the bifurcation via the forking behavior of the solution rates. However, contrary to the TMF limit, the RMF limit allows us to retain the neural variability due to stochastic inputs. In particular, at the cost of neglecting correlations, the RMF approach provides us with estimates about the higher moments of the dynamics. In Fig 11c, we show that the simulated and theoretically calculated second moments are well matched.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Mean firing rates β, mean internal variables and the second moments of the internal variables for the bistable networks from RMF calculations and simulations.

These figures shows in panel (a), the output rates β; in panel (b), the mean internal variables and in panel (c), the second moments of x, with varying excitation strength μ_e and fixed inhibitory strength μ_i = −4.0. Gray-shaded region is where the transition from monostable system to bistable system happened and we are not able to compute the simulated rates accurately, however, we can still perform RMF calculation within this region. For the simulated data, we run the event-driven simulation until 16000 spiking events for the entire network are accumulated. This process is repeated 32 times. During each repetition, we record the total time T_k(k = 1, ⋯, 32) and total spiking event count S_i,k for each neuron cluster i(i = 1, ⋯, 4). The mean firing rates of neurons are computed by . The standard deviation of the rates (indicated by error bars) over these 32 trials are computed by . Meanwhile, we record the entire time series of the internal variable x, with which we can compute the moment of x for any given order. The standard deviations of the quantities (indicated by error bars) are computed over these 32 repetitions. Parameters: K_total = 40, h = 1 Hz, a = ln(100)/20 ≈ 0.23, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g011

The absence of correlation in our RMF model appears not to affect our prediction, whereas Fig 12a shows that the original network exhibits a non-trivial correlation structure. This is because the non-trivial correlation structure of the original network is largely due to stochastic switching and simplifies to a tractable feedforward correlation structure when conditioning on the group of neurons that is up or down, as shown in Fig 12b and 12c. During a dominance period of the original dynamics, up-state neurons constitute the main source of variability. Fig 12b and 12c show that the activity of up-state neurons is essentially uncorrelated, in part owing to their frequent but irregular spiking resets. Such an activity is well-approximated in the RMF limit. By contrast, the activity of down-state neurons is dominated by strong, shared feedforward inhibitory inputs. Fig 12b and 12c show that as a result, the activity of down-state is strongly correlated but in response to an approximatively Poissonian drive. Such an activity is also well-approximated in the RMF limit. More generically, we expect the RMF approximation to perform well for metastable systems with more than two quasi-equilibria, as long as a distinct quasi-equilibrium corresponds to a distinct population of neurons silencing all other populations of neurons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Correlation structure of the original bistable network.

Panel (a) reveals that neurons are positively correlated within the same group, but negatively correlated across groups. This shows that the correlation structure of the metastable dynamics is primarily shaped by stochastic switches between up and down states. Panels (b) and (c) show the correlation structure of the same dynamics but conditioned on (b) Group₁ being up or (c) Group₁ being down. This shows that up-state neurons are weakly correlated, whereas down-state neurons are strongly correlated for receiving strong, shared inhibitory inputs. The correlations are estimated from simulated data, where we run the event-driven simulation until 400000 spiking events for the entire network are accumulated. We record the time series of the internal variables x_i. The unconditioned correlations are computed by Cor(x_i, x_j) = Cov(x_i, x_j)/(Std(x_i)Std(x_j)). The conditioned correlations are computed using the corresponding time series components (up/down state) detected and cropped from the whole time series. Parameters: μ_e = 1.5, μ_i = 4.0, a = ln(100)/20 ≈ 0.23, h = 1 Hz, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g012

Finite-size effects and metastability.

Synaptic distributions in real neural networks follow a highly skewed distribution: there is a large number of weak connections but only a few strong ones, whose strengths vary over two orders of magnitudes [11]. Studies have also shown that meaningful neural activity is triggered by strong, correlated synaptic inputs rather than uncorrelated, weak ones [9, 37]. Our RMF modeling approach allows us to quantify how the size of the synaptic inputs impacts bistability, at the cost of neglecting activity correlations. To do so, we consider the same hybrid model where neurons are all subjected to TMF-type background inputs, but where the circuit connections implementing cross-inhibition are treated in the RMF limits.

In this setting, we can quantify the emergence of bistability when varying the synaptic strength μ_e and μ_i involved in the cross-inhibitory circuit, thereby performing a phase space analysis of the network. Importantly, this phase space analysis only requires solving the corresponding self-consistent RMF equations rather than prohibitively costly simulations. We utilize the RMF solution rates to quantify bistability via the following bifurcation observable Δ: (24) where the subscripts Exc_u, Inh_u and Exc_d, Inh_d refer to neuronal groups in the up and down states, respectively. By definition, as both excitation and inhibition are larger in the up state, the observable Δ ranges between 0 and 1. In the monostable regime, Δ ≈ 0 is minimum since . By contrast, in the bistable regime, when the neurons are silenced in the down state, i.e. , Δ ≈ 1 is maximum. Fig 13a shows the parametric density plot of Δ obtained by varying μ_e and μ_i.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Phase transition.

Panel (a): In this density plot of bifurcation observable Δ, μ_e and μ_i are varied. The white dashed line represents the parameters (μ_e, −μ_i) used in Fig 11. The black circles correspond to the parameters for the simulations shown in Fig 10b, 10c and 10d. Panel (b) is the log-log plot of the value of μ_e at the onset of bifurcation varying the size of the network, determined by thresholding Δ = 0.01 in the RMF calculation (black circles). The red dashed line is the linear fitting in the log-log scale, which has a slope of −0.48. Parameters: h = 1 Hz, a = ln(100)/20 ≈ 0.23, τ = 10 ms.

https://doi.org/10.1371/journal.pcbi.1010215.g013

As expected, the regime of activity of the system is controlled by the overall level of cross-inhibition, which appears loosely linear in μ_e and μ_i. This is because cross-inhibition obviously requires the excitation of the mediating inhibitory neurons. For weak cross-inhibition, the network response is dominated by the uniform TMF background and fluctuates around its only equilibrium state. Increasing μ_e and μ_i leads to a sharp transition to a bistable regime. For small circuits, with about 10 neurons per subnetworks, this transition occurs for large synaptic weights requiring an RMF treatment as the TMF approximation yields wrong rate estimates. The sharpness of the transition indicates a strong silencing of neurons in the down state, in line with the strong nonlinearity of the network dynamics. In that respect, we find that the TMF approximation predicts that bistability emerges for smaller synaptic weights than observed in small networks, while also underestimating the firing rates.

We conclude by utilizing our RMF computational approach to exhibit behaviors that are otherwise challenging to obtain via simulations. Specifically, we exhibit in Fig 13b the scaling of the critical weights at the bifurcation μ_e with respect to the subnetwork size K, assuming a fixed ratio μ_e/μ_i = 0.2 and μ_e/μ_TMF = 10. Interestingly, we found that the critical weight scales approximately as , similar to the scaling formally defining the balanced thermodynamic limit. In this limit, inhibition and excitation cancel one another on average. As a result, neuronal activity is driven by the fluctuations in the inputs. The scaling ensures that neural variability is preserved in the large K limits for independent inputs. We observe a similar phenomenon for critically tuned networks in the RMF limit, for which inputs are always independent. This is because neural variability is an invariant of the dynamics at the bifurcation, where networks start alternating between the newly individualized up and down states. Preserving this variability throughout network sizes in the RMF limit naturally requires a scaling, which remarkably persists down to very small system sizes.