Introduction

Recurrent network models are known to produce different types of dynamics, ranging from regular to irregular, and from transient to persistent activity [1–6]. Moulding network dynamics to resemble experimental observations usually involves changes in the network architecture, i.e., the existence of synapses and their efficacies [7–9]. With this approach, the eigenspectrum and the non-normality of the connectivity matrix are indirectly affected, and the relationship between changes in those qualities of the weight matrix and the network dynamics remain unclear. This is challenging because both the eigenspectrum and non-normality carry important information about the dynamics. The eigenspectrum, i.e., the distribution of eigenvalues in the complex plane, carries information about the stability of the network (e.g., asymptotic behaviour) [6, 8, 10], and timescale of the dynamics [4, 11]. However, the eigenspectrum alone is not sufficient to describe the transient dynamical behaviour of a network [10, 12]. The transient dynamics, and more specifically, the phenomenon of transient amplification, depends on the alignment between the eigenvectors of the connectivity matrix [10]. Importantly, the more aligned the eigenvectors are, the more non-normal a matrix is. The non-normality of a matrix can be assessed through the Schur decomposition, an orthogonal similarity transformation that results in an upper triangular matrix on which the eigenvalues appear along the diagonal. The Schur decomposition of a matrix W can be formally written as W = U(Λ + T)U^†, where U is an unitary matrix (its columns are the orthogonal Schur modes), Λ is a diagonal matrix containing the eigenvalues of W (complex eigenvalues are represented as 2-by-2 block diagonals with the real and imaginary parts), and T is a strictly upper triangular matrix. The strictly upper triangular part contains information related to the interactions between the corresponding Schur modes [12]—it’s useful to note that a normal matrix has this strictly upper triangular part equal to zero because all eigenvectors are orthogonal. As such, the strength (measured by the norm [13]) of the strictly triangular part of the Schur decomposition plays an important role for the dynamics.

Therefore here, we consider upper triangular matrices and manipulate their spectrum and non-normality, such that their characteristics can be directly translated into dynamical properties (Fig 1A). These matrices no longer represent the neuronal connectivity, but modes of activation that are arranged in a feedforward manner [10, 12, 14] (Fig 1B). We are particularly interested in the different forms of transient amplification, a phenomenon that can resemble motor cortex activity during reaching [15–17] and also emulate long-lasting working memory dynamics [18–20]. For a network to be able to transiently amplify an input signal (i.e., initial condition), not only the eigenvectors need not be orthogonal (allowing for non-normal amplification), but also eigenvalues cannot have any real part larger than unity (allowing for the transient amplification to be followed by a decay back to baseline) [6, 10, 12]. This constrains the possible structure of the eigenspectrum. For example, when the connectivity matrix is random, the bulk of the eigenvalues is uniformly distributed on a disc centred at zero, which together with the stability constraint imposes that the radius can’t be larger than 1 [2, 6, 12]. Taking into account this effect, random weight matrices are not particularly flexible for generating distinct eigenspectrum distributions, not allowing for a rich plethora of network dynamics. Specific eigenspectrum distributions (given that the matrix is non-normal), can indeed elicit richer dynamics through either optimisation of weights taking into consideration their effect on the eigenspectrum [8] or the combination of matrices with different statistics [6, 21, 22]. However, it is still unclear how modifications in the distribution of only real or imaginary parts of the eigenspectrum change the dynamical regimes of a non-normal network which is able to transiently amplify inputs. Moreover, it’s not entirely known how these changes can be imposed to biologically realistic networks in which neurons are either excitatory or inhibitory, and neurons might not connect to themselves, i.e., without self connections.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Elements explored in the manuscript.

A, Schematic of the elements explored in this manuscript. Top left and clockwise: The connectivity matrix W^B; its corresponding Schur upper triangular decomposition ; the eigenspectrum; and the induced dynamics. In W^B and , red dots indicate positive (excitatory) connections while blue dots indicate negative (inhibitory) connections. B, The upper triangular matrix with the quantities that we alter in this manuscript in pink.

https://doi.org/10.1371/journal.pcbi.1010365.g001

To better understand how the manipulations in eigenspectrum and non-normality of a connectivity matrix are translated to biologically plausible networks, in this work we manipulate them directly in upper triangular matrices and then translate our main findings to biologically plausible networks. We start by defining the relationship between the network’s activity and the eigenmodes given by the Schur decomposition, showing how simple manipulations of the eigenspectrum’s imaginary diameter and real distribution can drastically affect the neuronal dynamics. We then systematically explore how the eigenspectrum’s imaginary diameter and the feedforward norm control the different regimes of transient amplification, showing that either shrinking the eigenspectrum’s imaginary diameter or increasing the feedforward norm increase amplification levels. Analysing the neuronal dynamics of these networks via the effective rank of the eigenvectors of the connectivity matrix, we find a trade-off between amplification and the underlying dimensionality of the possible dynamics of the network: networks with high (respectively low) levels of amplification produce dynamics in low (respectively high) dimensional subspaces. After a dissection of the underlying mechanisms of transient amplification using general upper triangular matrices, we consider biological constraints on the spectral distributions, and consequently, on the dynamics. Finally, we show how we can implement our findings in a biological plausible connectivity matrix with excitatory and inhibitory neurons, i.e., a matrix satisfying Dale’s law. We explore three different manipulations: lack of autapses, global inhibition, and ratio between excitatory-to-inhibitory and excitatory-to-excitatory connections. We show that increased global inhibition can lead to more amplifying dynamics due to its connections with the eigenspectrum, and that changing excitatory-to-inhibitory connections affect the eigenspectrum’s imaginary diameter.

Results

Throughout the paper we use the following notation for the connectivity matrix: W for a generic connectivity matrix, for a matrix given in upper triangular form, and W^B for a matrix following biological constraints (Fig 1A). The dynamics of the recurrent network are defined by (1) where x(t) is the internal state of the network at time t, and x_i(t) can be understood as the membrane potential of the i^th neuron of the network (with i = 1, …, N; N is the number of neurons in the network). This internal state of the neurons evolves with a characteristic time constant τ (fixed at τ = 200 ms throughout the paper) and is affected by the activity of other neurons of the network through the recurrent connections determined by W. Finally, the activation function, f(x(t)) = r(t), represents the input-output relation between the internal state, x(t), and the firing rate deviation, r(t), from the baseline activity, r₀. We assume linear dynamics, r = f(x) = x, for the mathematical analysis and compare to networks with richer dynamics using a known non-linear function given by (2) where r_min = 1 Hz and r_max = 4 Hz are the bounds of the sigmoid function f(x). Note that in this case, and generally used in previous works [4, 8], negative values of r_i(t) means less than baseline activity (see S1 Text for a non-negative version). In the linear case, the network dynamics can be described using the eigenvalues, λ_i, and eigenvectors, v_i, of the weight matrix W. To quantify whether and by how much the network can amplify specific inputs, we calculate the norm of the rate vector, ‖r(t)‖, by decomposing it along the directions of the eigenvectors of W, (3) where is the complex conjugate of r_k(t) and 〈v_k, v_j〉 is the inner product of the complex vectors v_k and v_j (see Methods). Here, is the solution of the system along the direction of the eigenvector v_k, which is associated with the eigenvalue λ_k ( is a constant, uniquely determined by the initial condition). In a stable regime, Re(λ_k) < 1, ∀k, the system exhibits a single fixed point that represents the baseline activity. An increase of the response norm, ‖r(t)‖, with respect to the norm of the initial condition, ‖r(t₀)‖ (here always normalised to 1), defines the phenomenon of transient amplification. A necessary condition for this to happen is the non-normality of W, i.e., the eigenvectors do not form an orthogonal basis [10]: 〈v_k, v_j〉 ≠ 0, for some j, k.

To explore regimes of transient amplification, we thus focus on matrices of the form (Fig 1B), with the diagonal, Λ, containing the eigenvalues [10, 12, 14], and the strictly upper triangular part, T, representing the feedforward structure between patterns of activation (see S1 Text). Note that Λ contains 2 × 2-blocks around the diagonal to accommodate for complex eigenvalues in real-valued matrices. The real parts of the eigenvalues are on the diagonal and the imaginary parts lie on the off-diagonal entries of the 2 × 2 blocks (Fig 1B; see Methods). We create Λ by sampling the real and imaginary parts of the eigenvalues from different distributions, but keeping the number of complex versus real eigenvalues constant (see Methods). The imaginary distribution needs to be symmetric with respect to zero (a condition imposed by the conjugacy of the complex eigenvalues), while the real distribution must be below 1 (and is here always set to have 0.5 as a supremum: max_λRe(λ) < 0.5) to ensure stability. For our analysis, we define the spectrum norm as the Frobenius norm of the matrix Λ, (note that the spectrum norm defined here is different than the commonly used spectral norm [23]). We create T in two different ways: from the Schur decomposition of a Stability-Optimised Circuit (SOC) [8] or sampled from a uniform distribution. A SOC is created from an initially unstable network (i.e., the initial weight matrix has some of the eigenvalues with real part greater than unity, Re(λ_k) > 1 for some k), whose inhibitory connections are modified (optimised) so that no eigenvalue has real part greater than unity (see Methods). After optimisation, a SOC produces strong non-normal transient amplification [8]. We thus use the SOC’s corresponding feedforward structure because it is tuned to create transient amplification. We linearly scale all the elements of T after its structure has been fixed to vary its norm.

The subsequent sections are organised as follows. First, we explore the effects of changing the distribution of the eigenspectrum’s imaginary and real parts. We then examine the influence of the spectrum and feedforward norms, ||Λ|| and ||T||, respectively, for different regimes of transient amplification, analysing the dimensionality of the dynamics in such regimes. Finally, we link some of the findings from abstract to biological networks through manipulations in a biologically realistic weight matrix that satisfies Dale’s law.

Shrinking the imaginary distribution increases amplification

We start our investigation of how the eigenspectrum affects the dynamics by drawing both real and imaginary parts from uniform distributions with diameters d_im and d_re, respectively (Fig 2A–2D, top left). To quantify the dynamical response of the network, we find an orthogonal basis of initial conditions of the linear network that elicit maximum amplification, , ordered according to their evoked energy [8], : if x(t = 0) = a, ||a|| = 1, then . We first calculate the initial condition a₁ that maximises , following an iterative calculation of the subsequent orthogonal initial condition, a_i, that maximise in the subspace orthogonal to the previously calculated initial conditions, {a₁, …, a_i−1}. To make sure that the evoked energy is due to an amplified response rather than merely a slower exponential decay, we compute the maximum value of the norm of the firing rate vector, for all vectors in (Fig 2A–2D, bottom left).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Eigenspectra and dynamics of the corresponding networks.

A–D, Four cases of eigenspectra and dynamics of the corresponding network of size N = 200. In each panel, clockwise: The spectrum; linear dynamics; non-linear dynamics; the logarithm of the maximum norm of the firing rate per initial condition. The same initial condition that elicits the maximum norm is used for both linear and non-linear dynamics. Pink dotted line indicates the percentage of conditions whose norm is amplified by at least 50%. The feedforward structure is taken from a stability-optimised circuit [8] and its Frobenius norm is fixed to 75. Real and imaginary parts follow an uniform distribution with diameters d_im and d_re, respectively. A, When d_im = d_re = 10, only 1% (2 out of 200) of the conditions are slightly amplified. B, When d_im = 10 and d_re = 1, the system is capable of more amplification. C, Here, d_im = 1 and d_re = 10, surprisingly creating more amplification compared to the case shown in panel A. D, When d_im = d_re = 1, the system amplifies almost half of the initial conditions. The dynamics, given an initial condition of norm 1, reach the value of ∼ 10⁵ Hz in the linear case and consequently long-lasting dynamics in the non-linear case.

https://doi.org/10.1371/journal.pcbi.1010365.g002

With broad distributions, the system can slightly amplify a few conditions (Fig 2A). When the range of the real-part distribution is decreased and the real parts are pushed towards 0.5, the resulting network produces stronger amplification (Fig 2B). This can mainly be attributed to the fact that the eigenvalues have now larger real parts and hence longer decay envelopes. A longer decay envelope allows for the network dynamics to undergo a larger proportion of a full period of the underlying oscillation without damping of the maximum amplitude, revealing the hidden feedforward structure, and thus increasing the maximum norm. Indeed, clustering away from 0.5 leads to less amplification (explored in the next section).

More surprisingly, shrinking, instead, the imaginary distribution also leads to more amplification (Fig 2C), and shrinking both distributions produces very large amplification that in the non-linear case lasts for a long time (longer than 10 seconds), approximating timescales of working memory dynamics (Fig 2D), previously known to arise through spectral abscissas near the stability line. Additionally, the percentage of conditions that are amplified is considerably increased, i.e., the ability of such a network to amplify orthogonal initial conditions is enhanced. Note that splitting and clustering the (positive and negative) imaginary parts away from zero gives rise to slightly different amplification regimes that also depend on the linearity of the system (S1 Fig). From these examples (Fig 2 and S1 Fig) it is clear that varying the diameter and position of the centre of the distribution of both imaginary and real parts of the eigenspectrum play distinct roles in the levels of amplification of a network undergoing transient amplification.

Manipulating the spectrum and the feedforward norms

When we study the effects of the imaginary and real distributions more systematically, we find that the shape of the real distributions affects the levels of amplification, but has minimum effect on how amplification changes with the imaginary distribution (Fig 3Ai and 3Aii and S2(A) Fig). Amplification emerges from the non-normality of W, which can be partly quantified by the angles between the eigenvectors (Eq 3); if more pairs have overlaps, the matrix will be more non-normal. The imaginary distribution changes the geometry of the eigenvectors (Fig 3Aiii), providing a mechanism for its drastic effect on the amplification in these networks (Fig 3Ai and 3Aii). This is a surprising effect given that we do not alter the feedforward norm, ‖T‖, i.e., the Frobenius norm of the strictly upper triangular part of , nor the decay envelopes at all (see S2(B) Fig and S1 Text for variations of feedforward structures).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Effects of manipulating the spectrum and the feedforward norms.

A, Maximum response norm for the preferred initial condition (i), percentage of directions whose norm is amplified more than 50% (ii), and the percentage of angles (between pairs of eigenvectors) that are less than 45° (iii). Every line is a function of the imaginary diameter. We plot four real distributions. Light green: a uniform distribution in which all real parts are distributed uniformly in the interval (−0.5, 0.5). Dark green: a uniform distribution in which all real parts are distributed uniformly in the interval (−9.5, 0.5). Light pink: a single valued real distribution in which all real parts are equal to zero. Dark pink: a single valued real distribution in which all real parts are equal to −0.5. In all cases the network size is N = 200 and the feedforward Frobenius norm is fixed at 75. d_re indicates the diameter of the uniform distribution of the eigenspectrum’s real part. B, Same as panel A, but plotted as a function of the feedforward Frobenius norm. Different colours correspond to 5 different spectra; all spectra have fixed single-valued real distributions (equal to zero) and different imaginary diameters. d_im indicates the diameter of the uniform distribution of the eigenspectrum’s imaginary part. C, Normalised inner product between vectors from a simplified 3-by-3 upper triangular matrix (Eq 4) as a function of the imaginary diameter (β in Eq 4) for three conditions: “strong feedforward” (ϕ_norm = 30 and α − γ = −0.3); “weak feedforward” (ϕ_norm = 3 and α − γ = −0.3); and “large d_re” (ϕ_norm = 3 and α − γ = −3). D, Same as panel C, but plotted as a function of the feedforward norm for three different conditions: “small d_im” (β = 100 and α − γ = 0); “intermediate d_im” (β = 1000 and α − γ = 0); and “large d_im” (β = 10000 and α − γ = 0).

https://doi.org/10.1371/journal.pcbi.1010365.g003

The feedforward norm is more directly linked to the non-normality [10], and as expected, it increases both the norm of the maximum response (Fig 3Bi), and the percentage of amplified conditions (Fig 3Bii), for larger values. The percentage of eigenvector pairs with small angles also grows with increasing feedforward norm strength (Fig 3Biii). Interestingly, there is a saturating point of eigenvector pairs aligned with angles smaller than 45° that depends on the imaginary distribution. Once the number of pairs saturates, increased amplification may be associated with an increase in the matrix norm, ||W||, or further alignment of these eigenvector pairs (even smaller angles).

To get an intuition of the mechanisms behind the changes in amplification levels we analysed a 3-by-3 upper triangular matrix with one purely real eigenvalue, λ₁ = γ, and two complex eigenvalues, . Two out of the three eigenvectors of this matrix are orthogonal, 〈v₂, v₃〉 = 0. The inner product of the non-orthogonal eigenvectors is given by (see Methods for details) (4) where ϕ_norm is the feedforward norm. The result from the simplified 3-by-3 matrix follows the same trends as the simulations (Fig 3C and 3D). The overlap between the eigenvectors increases as the imaginary part of the eigenvalues, β, decreases—equivalent to shrinking the imaginary diameter in the large upper triangular connectivity matrix, . The eigenspectrum’s real elements, α and γ, have a similar effect: the larger their difference the more the eigenvectors are aligned. Interestingly, when α = γ, the real distribution has no influence over the overlap between the eigenvectors, such as seen when we decrease the single real-valued distribution (S2(A) Fig, bottom)—the decrease in amplification levels is mostly due to the faster decay times when decreasing the single real-value distribution (S2(A) Fig, top). Because of the square, the larger the absolute value of α − γ (negative or positive) the less the eigenvectors are aligned, which we confirm with a large upper triangular matrix, , varying the diameter of the eigespectrum’s real distribution while keeping its maximum value at max_λRe(λ) = 0.5 (S2(C) Fig). The eigenspectrum’s real distribution may affect the level of amplification in two ways: it changes the asymptotic behaviour (decay times) and the eigenvector alignment. These results give a broad intuition for the distinct contribution of the eigenspectrum’s imaginary diameter and feedforward norm, but we still do not exactly know how the dynamics of such networks evolve. Thus, we next study the relative directions of the eigenvectors in state-space.

The geometry of output trajectories

If most eigenvectors are pointing in similar directions, the dynamics will be biased towards these directions too (Eq 3). This does not mean that W or the eigenvector matrix V are not full rank—on the contrary, they almost always are. What it means is that, in order to quantify the global eigenvector geometry, we have to use the effective rank of V. The effective rank of V measures the average number of significant dimensions in its range, and is formally defined as the exponential of the spectral entropy of its normalised singular values [24]. Specifically, if σ₁, σ₂, ⋯, σ_N are the singular values of V, and , with , then (5) where H(p₁, ⋯, p_N) is the Shannon entropy, i.e., .

The effective rank of V is indeed small in the highly amplifying regimes (Fig 4A), revealing an underlying duality between amplification and output dimensionality. The consequence for the dynamics is that, even though the system may amplify many initial conditions, they nevertheless evolve in the same low dimensional subspace [22]. To identify the dimensionality of this subspace we compute the effective rank of the matrix which is constructed as follows: the j^th column of is the first principal vector of the dynamics, given the j^th amplified initial condition of the basis. We find that there is a discrepancy between the number of amplified directions and effective rank when the system produces large amplifications (Fig 4B; see S3 Fig for results with larger number of principal components). This suggests that the dynamical responses evoked by orthogonal initial conditions evolve in the same subspace, indicating that any added noise will be amplified in the same subspace and that different initial conditions could potentially lead to similar linear readouts (see ref. [6] for capacity estimates of transient amplifying networks). There is thus a trade-off between the number of orthogonal amplified conditions and the noise robustness of the system. The effective rank of V is preserved in a recurrent network that is a rotated version of an upper triangular one (S4(A) Fig) while the effective rank of the connectivity matrices differ (S4(B) Fig). Similarly, the discrepancy between the number of amplified directions and effective rank is preserved for the rotated (recurrent) weight matrix (S4(C) Fig; compare to Fig 4B), which highlights the robustness of our method to describe the geometry of the network dynamics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Geometry of output trajectories.

A, The effective rank of the eigenvector matrix V of as a function of the imaginary diameter (left) and the feedforward norm (right). B, Amplified directions and effective rank of the matrix (see text) in the linear and nonlinear cases as a function of the imaginary diameter (left) and the feedforward norm (right). The feedforward structure is random from a uniform distribution, and the real distribution is uniform on (−0.5, 0.5). In all cases the network size is N = 200. The feedforward Frobenius norm is fixed at 75 for the plots with varying imaginary diameter. The imaginary diameter is fixed at 20 for the plots with varying feedforward norm.

https://doi.org/10.1371/journal.pcbi.1010365.g004

To further describe the system, we use the timescale of the transient amplification, i.e., period, Δt, for which ‖r(t)‖ ≥ 1 for the nonlinear network (Fig 5A and S5 Fig). This timescale varies continuously as a function of the norms of the eigenspectrum and the feedforward structure (Fig 5B and S6(A)–S6(F) Fig). Importantly, the dynamics evoked in each of these regimes—defined here by the transient amplification period—lie in different subspaces. For very short periods (e.g., Δt ≤ 500 ms; Fig 5A “weak”), the eigenvectors are effectively orthogonal to each other but span the entire output space equally. For short transient periods (e.g., 500 < Δt < 2000 ms; Fig 5A “short transient”), there is a good balance between amplification of orthogonal inputs and diversity in the responses. For long transient periods (e.g., Δt ≥ 2000 ms; Fig 5A “long transient”), many initial conditions are amplified but the responses lie in the same low-dimensional subspace. This result is well explained by the alignment of the eigenvectors of the simplified 3-by-3 upper triangular matrix (Eq 4): larger feedforward norm or smaller eigenspectrum norm result in eigenvectors being more aligned (S6(G)–S6(I) Fig). Indeed, when we fix the norm of W, and distribute a—continuously decreasing—percentage of this norm on the diagonal and the rest on the feedforward structure, the network transitions from weakly to strongly amplifying (Fig 5C). Thus, it’s the relation between the diagonal (representing the spectrum) and feedforward parts of the matrix that shapes the dynamics of the network.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Amplification regimes.

A, Representative examples of non-normal amplification defined by the timescale of the transient response of the nonlinear network—period, Δt, for which ‖r(t)‖ ≥ 1: “weak” (Δt ≤ 500 ms); “short transient” (500 < Δt < 2000 ms); and “long transient” (Δt ≥ 2000 ms). Grey dotted line indicates ‖r‖ = 1. B, Timescale of the response in the nonlinear network (as in panel A), parametrised by the norms of the spectrum and feedforward structure. Yellow indicates timescale longer than 10 seconds. Boxes correspond to the values used for the plots in panel A (colour coded): (feedforward norm, spectrum norm) = (100, 700), (500, 500), and (700, 100) for weak, short transient, and long transient, respectively. C, Maximum norm of the dynamical response per initial condition for different percentages of the norm assigned to the spectrum, ranging from a matrix whose entire norm is assigned to the spectrum (yellow; 100% case, normal matrix) to a matrix whose entire norm is assigned to the feedforward part (dark red; 0% case, nilpotent matrix). The network size is N = 200 in all panels. Both eigenspectrum and feedforward structures are random uniform.

https://doi.org/10.1371/journal.pcbi.1010365.g005

Towards biologically realistic networks that satisfy Dale’s law

Up to here we explored different regimes of transient amplification in networks defined by an upper triangular connectivity matrix. This allowed us to have precise control over both eigenspectrum and feedforward structure of the connectivity matrix. However, these abstract networks do not represent biologically realistic neural networks, especially because of Dale’s law, i.e., neurons are either excitatory or inhibitory. In an upper triangular connectivity matrix, , the feedforward structure (and consequently feedforward norm) is given by the elements of its strictly upper triangular part, while the eigenspectrum (and consequently spectrum norm) is given by the elements of its diagonal. However, both the eigenspectrum and the feedforward structures are not trivially manipulated in a biological connectivity matrix, W^B. As a last application, we thus explore how to navigate the regimes of transient amplification in biological networks (i.e., satisfying Dale’s law) based on our results from upper triangular matrices. First, we consider the effect of the absence of self loops in the connectivity matrix. We then focus on the indirect manipulation of the spectrum norm and the distribution of eigenvalues via modifications of global inhibition and the indirect manipulation of the eigenspectrum’s imaginary diameter via modifications of the ratio between excitatory-to-inhibitory and excitatory-to-excitatory connections, respectively.

In the simulations of networks satisfying Dale’s law, half of the neurons are chosen to be excitatory (only positive output weights) and the other half to be inhibitory (only negative output weights) [2, 8, 12]. For additional biological plausibility, the connections are sparse, i.e., elements are set to zero following an uniform distribution. We thus have a set of constraints to fulfil: (i) stable asymptotic dynamics (lim_t→∞ ‖r(t)‖ = 0, i.e., max_λRe(λ) < 1); (ii) transient amplification (‖r(t)‖ > ‖r(0)‖ for t₀ < t < t₀ + Δt, with t₀ ⩾ 0 and Δt > 0); and (iii) neurons are excitatory or inhibitory. To do so we build biological weight matrices, W^B, with the algorithm from Stability-Optimised Circuits (SOC) [8] (see Methods), which fulfils all constraints mentioned above. The SOC algorithm optimises inhibitory connections to ensure that no eigenvalue has real part greater than α_max, α_max < 1 (max_λRe(λ) < α_max). Moreover, it allows for additional constraints to be implemented, such as the level of global inhibition, and whether neurons have autapses, which we use to translate the results from upper triangular matrices, , to biological matrices, W^B.

The absence of neuronal self loops shrinks the real distribution.

Experimental evidence supports the existence of autapses [25, 26], yet it is common for modelling work to impose no self connections [27, 28]. As a starting point, we thus analysed the consequence of the lack of self connections in networks, i.e., networks without autapses. When neurons are not structurally connected to themselves, the trace of the weight matrix of such a network is equal to zero. This unfolds as follows: (6) Given that due the conjugacy of the eigenvalues, the weight matrix W^B without self loops has . This, together with the stability constraint, max_λRe(λ) = α_max < 1, bounds the real distribution from below and above, restricting it to a limited diameter that is less than α_maxN. The maximum range for the real distribution is bounded by −α_max(N − 1) and α_max, when all real parts but one—defined here as the outlier—are equal to α_max and the outlier is equal to −α_max(N − 1). This observation explains why the spectrum of a stability-optimised circuit [8], which doesn’t have self loops, has an elongated shape along the imaginary axis after optimisation. Not only the positive but also the negative real parts of the eigenspectrum are pushed towards the stability line after optimisation, with the exception of the outlier, which has a large negative value due to the non-self loops constraint and inhibition dominating over excitation [8]. The zero trace condition (particularly ) is necessary but not a sufficient condition for the absence of self loops, yet it provides an intuition for its effect on networks defined by upper triangular matrix that can be translated to biological networks.

The existence of the negative outlier together with the zero trace condition has an interesting effect in upper triangular matrices. The larger the value of the outlier (in absolute value), the bigger the amplification (Fig 6A) and the number of amplified directions (Fig 6B). On one hand, this can be explained by the fact that more real parts are pushed to the right, creating longer decay envelopes, hence prolonging the time for the hidden feedforward structure to be amplified. On the other hand, this is not the sole source of the increased amplification; the combination of a large negative outlier with the zero trace condition has an additional non-intuitive effect on the geometry of the eigenvectors, i.e., it gives rise to larger eigenvector overlaps (Fig 6C). Without the zero trace condition, increasing the (negative) value of the outlier does not give rise to the same levels of amplification (S7 Fig), confirming that large amplification is an effect from the combination of the negative outlier and the zero trace condition. Interestingly, the negative outlier, γ, reflects the level of global inhibition [2, 8, 12, 29], such that (see Methods) (7) where E and I are the sum of excitatory and inhibitory weights per postsynaptic neuron, respectively (Fig 6D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. A large negative outlier increases amplification in upper triangular matrices without self loops and is proportional to I/E ratio in networks satisfying Dale’s law.

A, Maximum response norm for the preferred initial condition as a function of the imaginary diameter using upper triangular connectivity matrices with the zero trace condition and different outliers (coloured coded). The network size is N = 200 and the feedforward Frobenius norm is set to 75 in all cases. B, Percentage of directions whose norm is amplified more than 50% as function of the imaginary diameter as in panel A. C, The percentage of angles, between pairs of eigenvectors, that are less than 45°, as a function of the imaginary diameter as in panel A. D, Position of the outlier as a function of the I/E ratio for a network with 100 excitatory and 100 inhibitory neurons sparsely connected with no self loops. An initially random network is optimised with the Stability-Optimised Circuit (SOC) algorithm [8] with I/E = 40 (see Methods). The additional outliers are calculated by linearly scaling all inhibitory weights to I/E = 3, 5, 10, 20, 40.

https://doi.org/10.1371/journal.pcbi.1010365.g006

The eigenspectrum norm is influenced by global inhibition.

Taking into account that a large outlier reflects a connectivity matrix with inhibition dominating over excitation [12], i.e., the mean of inhibitory weights is larger than the mean of excitatory weights (Fig 6D), we tested whether strengthening the inhibitory weights would have a similar effect as increasing the (absolute) value of the outlier. For that, we built networks with excitatory and inhibitory neurons with an initial spectral radius R_outer = 10 and various ratios of inhibition and excitation, . Non-zero excitatory and inhibitory weights are defined as and , respectively, with (8) where p is the probability of a connection being non-zero, taken from a uniform distribution. As a result, the eigenspectrum is mostly distributed inside a circle in the complex plane of radius R_outer [2] with one negative outlier due to inhibition dominating over excitation [12], given by (see Methods) (9) Importantly, an inhibition dominated network is characterised by a non-uniform distribution of eigenvalues inside the circle of radius R_outer, with a denser region near the origin. The denser region is limited by an inner circle [2] with radius given by (see Methods) (10) Thus, the eigenspectrum of a inhibition dominated network is characterised by the presence of a negative outlier and a denser region of eigenvalues near the origin (Fig 7A); properties that are preserved after stabilisation of the weight matrix via the SOC algorithm (Fig 7B). The outlier’s value is not drastically changed by the SOC algorithm and is well captured by the analytical expression from Eq 9 (Fig 7C). Considering only the imaginary distribution, the outer circle shrinks while the inner circle remains mostly constant (Fig 7D), the latter being well described by Eq 10.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The effect of inhibitory dominance in Dalean matrices.

A, Eigenspectra of connectivity matrices satisfying Dale’s law: 100 excitatory and 100 inhibitory neurons sparsely connected (probability of connection, p = 0.1) without self loops, constructed with spectrum of radius 10 and global inhibitory dominance of strength I/E (indicated on top of each panel). Outlier, outer radius (R_outer), and inner circle (R_inner) are highlighted. See Methods for details. B, Eigenspectra of connectivity matrices from panel A after optimising inhibitory weights with the SOC algorithm (see Methods for details). C, Value of the purely real outlier before (open circle) and after (closed circle) optimisation. Dashed line represents the analytical expression (Eq 9). Circles correspond to average over 1000 realisations. D, Imaginary diameter of the outer (left) and inner (right) circles. Open and closed circles represent average values before and after SOC optimisation algorithm, respectively, for 1000 random realisations. The outer and inner radii are calculated as the radius for which the density of imaginary elements drops below 0.005 and below half the maximum density, respectively. Dashed grey line (left) indicates d_im = 20, and purple dashed line (right) represents the analytical expression (Eq 10). E, Maximum norm per initial condition for different I/E ratios. Grey dotted line corresponds to a response norm that is 50% larger than the norm of the initial condition. Pink dashed lines indicate the percentage of initial conditions that elicit transients with maximum norm larger than 50% for I/E = 3 (lower percentage) and I/E = 40 (higher percentage). We linearly scale all weights to keep the same Frobenius norm (equal to 100) for comparison. F, The spectrum and the feedforward norms for different values of I/E in the corresponding real Schur transformation. G, Percentage of amplified conditions and effective rank of the corresponding matrix (defined in the text) in the linear case.

https://doi.org/10.1371/journal.pcbi.1010365.g007

The existence of a denser region of imaginary elements near the origin suggests an effect similar to shrinking the imaginary distribution. This effect may be augmented by the lack of autapses, which moves the real elements towards the stability maximum to accommodate for the large negative outlier that emerges from inhibition dominating over excitation. We find that, indeed, larger global inhibitory strength leads to more amplified conditions and also to slightly larger amplification per condition when the Frobenius norm of the weight matrix is kept fixed (Fig 7E). By assigning larger values to the inhibitory weights, the spectrum norm decreases and the feedforward norm increases (Fig 7F). To highlight this finding, we note that when the inhibitory to excitatory ratio is large, I/E = 40, the strength of every nonzero excitatory-to-excitatory connection is 0.08, and yet the network is capable of stronger amplification compared to when I/E = 3 in which the nonzero excitatory-to-excitatory weights are set to 1.05 (see Eq 8).

Finally, the new amplified conditions induced by the strongest inhibition do not share their first principal component directions in their dynamical responses (Fig 7C), i.e., the noise robustness of the system is not compromised in this case [6]. This is possible because we are still in the short transient regime; the long transient regime cannot be reached by solely increasing the global inhibitory strength, i.e., large feedforward norms are always accompanied by small spectrum norms in stability-optimised circuits (S8 Fig), which restricts the accessibility to the different dynamics regimes of transient amplification (S6(D)–S6(F) Fig). Since the overall norm of the matrix stays the same (for comparison reasons), further increasing the inhibitory dominance would unavoidably decrease the excitatory weights even further. Therefore, the amplification power of the network through this mechanism eventually saturates before reaching the long transient regime. Notice that the Frobenius norm of the weight matrix was constrained by our choice of the initial spectral radius, R_outer, that, consequently, constrains the possible spectrum and feedforward norms.

These results indicate that the strength of the global inhibition, if modulated by an external signal, could thus serve as a dynamical switch between amplifying and non-amplifying regimes, and counter-intuitively, weakening of inhibitory synaptic weights would decrease the level of amplification of an originally amplifying network. To test this hypothesis, we built a network with strong inhibitory connections (I/E = 40) that enables the network to amplify certain inputs in the short transient regime (Fig 8A–8C). This network transitions into a network that is unable to amplify any inputs by scaling its inhibitory weights down (Fig 8D–8F), confirming that global inhibition can indeed serve as an external switch that controls whether the network is able or not to transiently amplify inputs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. From short transient to non-amplifying with weakened inhibition.

A, Schematic of a network with strong excitatory and inhibitory connections and I/E ratio of 40. B, The network’s dynamics given the preferred initialisation. The resulting network is in the short transient regime; the preferred initialisation yields amplifying dynamics. C, Maximum response norm for all orthogonal conditions, in decreasing order. Grey dotted line corresponds to a response norm that is 50% larger than the norm of the initial condition. Pink dashed line indicates the percentage of initial conditions with maximum norm larger than 50% of the initial condition. D, Schematic of the same network from panel A, but the inhibitory weights are scaled down by a factor of 40 (yielding an I/E ratio of 1), which could be interpreted as the resulting effect of modulation of the inhibitory neurons (or synapses). E, The dynamical response given the preferred initialisation; inset depicts the same dynamics on a different scale. F, Maximum response norm per condition. The network is unable to amplify any inputs. The maximum norm of the dynamics is equal to the norm of the initial condition (set to be 1) for all initialisations. The network is composed by 100 excitatory and 100 inhibitory neurons sparsely connected and without self loops. The schematics (panels A and D) is adapted from ref. [8].

https://doi.org/10.1371/journal.pcbi.1010365.g008

The diameter of the imaginary distribution is influenced by the relationship between excitatory-to-inhibitory and excitatory-to-excitatory connections.

In the upper-triangular version of the weight matrix, , the eigenspectrum’s imaginary distribution is given by the off-diagonal terms of (see Fig 1B). However, for a biologically plausible weight matrix, W^B, the manipulation of the imaginary distribution is not as trivial [2, 12]. To get an intuition of the role of each connection type, we use a mean-field approach in which a large weight matrix is simplified as a 2-by-2 matrix [10]: rows and columns correspond to the excitatory and inhibitory populations (Fig 9A), and the elements represent their mean connections (Fig 9B). We define W_EI, W_IE, W_EE, and W_II (all positive) as the mean connectivity strength of inhibitory-to-excitatory (I-to-E), excitatory-to-inhibitory (E-to-I), excitatory-to-excitatory (E-to-E), and inhibitory-to-inhibitory (I-to-I) groups, respectively. The eigenvalues are complex when the condition 4W_EIW_IE > (W_EE + W_II)² is satisfied (see Methods for detailed calculation). Real and imaginary parts of the two eigenvalues corresponding to the simplified matrix are thus (11) (12) When inhibitory connections, W_EI and W_II, are optimised for stability [8], we are left with excitatory connections to manipulate the imaginary diameter of the eigenspectrum. Weakening E-to-I, W_IE, or strengthening E-to-E, W_EE, should shrink the imaginary distribution according to Eq 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The relationship between E-to-I and E-to-E connectivity strengths alters the imaginary distribution of eigenspectra of Dalean networks.

A, Schematics of the mean-field analysis of a network with a group of excitatory (E) and a group of inhibitory (I) neurons. The mean weight from E-to-E, E-to-I, I-to-E, and I-to-I are represented by W_EE, W_IE, W_EI, and W_II, respectively. B, Weight matrix of a simplified network from panel A [10]. Inhibitory connections are optimised by the SOC algorithm [8]. C, Imaginary diameter of a network with 100 excitatory and 100 inhibitory neurons as a function of the ratio E-to-I to E-to-E weights. D, Feedforward (orange) and spectrum (red) norm as a function of the ratio E-to-I to E-to-E weights for the same networks from panel C. E, Maximum norm per initial condition for different ratios E-to-I to E-to-E weights. Grey dotted line indicates response norm that is 50% larger than the norm of the initial condition. Pink dashed lines indicate the maximum percentage of orthogonal initial conditions that evoke response norm 50% larger than initial condition for W_IE/W_EE = 1.8 (lower percentage) and W_IE/W_EE = 0.4 (higher percentage).

https://doi.org/10.1371/journal.pcbi.1010365.g009

To test the intuition from the mean-field analysis, we built networks with connectivity matrices obeying Dale’s law and without self loops according to the Stability-Optimised Circuit (SOC) algorithm [8] with varying ratios W_IE/W_EE. For each network initialisation, we systematically increased the strength of E-to-I connections, W_IE, while keeping the same strength of E-to-E connections, W_EE. We then optimised inhibitory connections (both W_EI and W_II) with the SOC algorithm, and linearly scaled all weights to maintain the same Frobenius norm of the connectivity matrix, ‖W^B‖, for all realisations. We found that, in line with Eq 12, the imaginary diameter of the Dalean weight matrix enlarges with the increase of the ratio W_IE/W_EE (Fig 9C). As a consequence, the spectrum norm increases as the feedforward norm decreases with the strengthening of the E-to-I compared to E-to-E connections (Fig 9D). This indicates that in a stabilised network with connectivity following Dale’s law, i.e., when inhibitory connections are optimised for stability, changing the relationship between E-to-I and E-to-E connectivity can influence the diameter of the imaginary distribution of eigenvalues. Due to a decrease in the imaginary diameter, the max norm of dynamics evoked by the orthogonal initial conditions are slightly larger when E-to-I are weaker than E-to-E weights (Fig 9E). These results indicate that excitatory-to-inhibitory connections may play a role in shaping the network’s dynamics through their effect on the imaginary distribution of the connectivity matrix.

Discussion

In this article we used upper triangular matrices as abstract representations of the dynamical properties of a connectivity matrix to control the quantities that are relevant for the neural dynamics in the transient amplification regime. By by-passing, temporarily, the connectivity matrix and focusing on a hypothetical Schur transformation, we found new dynamical regimes of large amplification that translated into long transients in non-linear networks. We showed that the amount of transient amplification that a network can produce may be controlled by the ratio between the norms of the spectrum and hidden feedforward structure. Increasing the feedforward norm or decreasing the eigenspectrum’s imaginary diameter resulted in larger amplifying dynamics.

Different combinations of the eigenspectrum’s and feedforward norms resulted in regimes of transient amplification that lay in subspaces with different dimensionalities and evolved with distinct timescales for non-linear networks. Very short transient periods in non-linear networks spanned the entire output space but only a few orthogonal conditions elicited large amplification. Short transient periods in non-linear networks lay in a lower subspace, but more orthogonal conditions elicited large amplification. Finally, long transient periods in non-linear networks lay in the same low-dimensional subspace, but many orthogonal initial conditions evoked large amplification. The real distribution could considerably change the levels of amplification, but did not affect the relationship between amplification and spectrum or feedforward norms. When the majority of real parts were close to the upper bound of stability, a zero trace condition had an interesting effect: the larger the negative outlier, the closer the real parts were to the stability limit, thus creating larger amplification levels. Moreover, we found a trade-off between the number of orthogonal initial conditions that elicit large amplifications (larger than 50% of initial condition) and the dimensionality of the underlying neuronal dynamics. The source of amplification, i.e., the overlaps of the eigenvectors, inevitably restricted the subspace in which the dynamical outputs evolve, indicating that any noise added to the system is also amplified in the same subspace, giving rise to low robustness to noise [6].

In biologically plausible networks that satisfy Dale’s law (i.e., with excitatory and inhibitory neurons), we found that stronger global inhibitory dominance controlled the spectrum norm due to its relationship with the eigenspectrum outlier and a non-uniform distribution of the remaining eigenvalues. Additionally to global inhibition affecting the spectrum norm, we also showed that excitatory-to-inhibitory connections, more specifically its relationship to excitatory-to-excitatory connections, can alter the eigenspectrum’s imaginary diameter. We could therefore link the results from upper triangular matrices and biologically plausible networks via two properties related to specific connectivity patterns: global inhibition and excitatory-to-inhibitory connectivity.

Our work complements previous studies relating manipulations of eigenspectrum and their consequence to neuronal dynamics (e.g., [2, 6, 8, 10, 12, 14, 21, 30]), more specifically on the topic of transient amplification (e.g., [6, 8, 10, 12, 21]). Transient amplification is a phenomenon tightly linked to eigenspectrum and non-normality of a connectivity matrix. For the dynamics to be transient, the network’s activity must eventually decay back to baseline, which constrains the eigenspectrum’s real part to be less than unity—a stability constraint. For the dynamics to amplify certain inputs, the connectivity matrix must be non-normal [6, 8, 10]. Neuronal networks with excitatory and inhibitory neurons are always non-normal, because of the separation of positive and negative columns [10], but non-normality of a connectivity matrix is not sufficient for a network to transiently amplify inputs [6]. A necessary condition in, e.g., random matrices is that the symmetric (real) part of the connectivity matrix has eigenvalues greater than unity [6]. In networks with sparsely connected excitatory and inhibitory neurons, amplification levels increase with the eigenspectrum radius and the connectivity sparsity, which reflects the global strength of excitatory and inhibitory connections [12]. If the radius is greater than the unity, the network’s dynamics are unstable, which can be stabilised by iteratively adjusting inhibitory connections to decrease the spectral abscissa until its values is less than one (and therefore stable) [8]. This algorithm, referred to as Stability-Optimised Circuits (SOC), and used here to generate Dalean matrices, is able to optimise networks to elicit largely amplified transients. Interestingly, after optimisation, the eigenspectrum retains a similar imaginary diameter but the eigenspectrum’s real part accumulates near the stability limit apart from the outlier [8]. We confirmed that in Dalean networks, the existence of a large negative outlier imposes that the remaining of the eigenspectrum’s real elements be pushed near the spectral abscissa in networks without autapses.

Any structure imposed to a connectivity matrix affects its eigenspectrum in a particular way. Excitatory-inhibitory balance constrains the eigenspectrum to a well defined circle for finite-sized networks, with a non-uniform distribution of eigenvalues inside the circle when the distribution of excitatory and inhibitory weights differ [2]. Cyclic connectivity is reflected by eigenvalues lying on a circle (as oppose to inside for random matrices) [14], and the combination of random matrix and a single feedforward chain with purely imaginary eigenvalues of alternating sign results in an eigenspectrum with two centres at ±i, results in large amplification of oscillatory inputs [21]. Complementary to these previous studies, our work explored simple changes in the eigenspectrum of a connectivity matrix, such as the diameter of imaginary distribution, real negative outlier, and zero trace condition. More importantly, we could directly link these simple changes to properties of Dalean matrices, shining a light on how such changes might be implemented in biologically plausible networks.

Our results in matrices following Dale’s law allowed us to find a potential role for changes in global inhibition in neuronal networks. Quick changes in the efficacy of inhibitory synapses, both inhibitory-to-inhibitory and inhibitory-to-excitatory, could act as a switch in the dynamical state of the network, changing from amplifying to non amplifying by reducing all inhibitory efficacies. This switch may be controlled by neuromodulators such as the neuropeptide proctolin [31], acetylcholine [32], dopamine [33], or serotonin [34], as well as stereotypical connectivity motifs [35, 36] that could influence inhibitory activity in a similar way to weakening or strengthening all inhibitory weights. A modulatory increase in the global levels of inhibition could be thus allowing the system to amplify inputs, facilitating signal processing in the brain. This effect resembles the paradoxical decrease in inhibitory activity as a result of an increase of external (excitatory) input onto inhibitory neurons [37]. We did not implement any external input, and thus the counter-intuitive (or paradoxical) effect originates from scaling all inhibitory weights up or down. Intuitively, one would expect that by increasing inhibitory weights, the amplitude of the network activity should decrease, and vice-versa. However, we counter-intuitively found the opposite to be true.

The dynamics of real biological neuronal networks, assessed from large-scale recordings of neural activity, has been shown to typically lie in a low-dimensional space [38], i.e., the number of dimensions necessary to explain the majority of the data variance is much smaller than the number of neurons recorded. The low-dimensional dynamics seen in experiments are well described by, e.g., non-liner rate networks (such as the one used in our work), when connections are optimised for either the experiment’s task [7] or recorded neuronal activity [39]. Complementary to large-scale recordings of neuronal dynamics, recent technological improvements have allowed researchers to build connectivity maps of large-scale brain regions [40]. Finding the relationship between connectivity and dynamics is thus crucial to understand the function of brain circuitry. The rank of the weight matrix carries information about the dimensionality of the underlying dynamics generated by the network, but it does not necessarily indicate the dimensionality of the subspace in which the dynamics lie. When, e.g., a weight matrix is constructed as a sum of a unit-rank (generated from two independent vectors) and a random (zero mean) matrix, the underlying dynamics evolves along the direction of the vector used to generate the unit-rank matrix [22]. These vectors are the eigenvectors of the unit-rank matrix [22], and because of the construction of the weight matrix, both weight and eigenvector matrices will share the same effective rank. We have shown a more general case in which the effective rank of the eigenvector matrix can carry information about the dimensionality of the dynamics, even when the weight matrix itself has a high effective rank.

We limited our exploration in upper triangular matrices to the effects of simple changes in the spectrum and feedforward structures, by controlling their range and norm. In networks that satisfy Dale’s law, excitatory weights were sparsely connected with the same connection strength for all non-zero entries, changing only the ratio between excitatory-to-inhibitory (E-to-I) and excitatory-to-excitatory (E-to-E) connections. In reality, synapses are constantly changing, following stereotypical synaptic plasticity rules that allow learning of memories [5, 41–43] or additional types of computation such as dynamical switch [44] and input categorisation [45]. Most computational models typically explore learning as a process involving changes in excitatory-to-excitatory (E-to-E) connections accompanied by modifications in inhibitory-to-excitatory (I-to-E) connections for stability [41, 42]. Typically, excitatory-to-inhibitory (E-to-I) connections are randomly initialised and kept fixed throughout the simulations (but see ref. [46]). Our results linking E-to-I connections to the eigenspectrum’s imaginary distribution, and consequently, amplification levels, suggest a role for such connection type. Future theoretical and experimental work would be thus necessary to design and measure potential plasticity rules for E-to-I connections that, in synergy with E-to-E [47] and I-to-E plasticity [44], may, e.g., generate new classes of activity patterns.

Our work opens the door for the exploration of new questions related to neuronal dynamics, such as how the structure—besides the norm—of the feedforward part as well as how non-uniform imaginary distributions affect the dynamics of biologically plausible networks.

Methods

Supporting information

Acknowledgments

We thank Friedemann Zenke for his comments, especially on the effect of the self loops on the spectrum. We also thank Ken Miller and Bill Podlaski for helpful comments.