1.1 Local vs. global representations of random recurrent connectivity

We study networks of N rate units with random recurrent connectivity given by the connectivity matrix J, where the entry J_ij corresponds to the strength of the synapse from neuron j to neuron i. A full statistical description of the random connectivity would require specifying the joint distribution P({J_ij}) of the N² synaptic weights. Determining the dynamics from this high-dimensional distribution is however in general intractable. We therefore focus on connectivity models that make simplifying assumptions on the underlying statistics.

Our specific goal is to relate two different classes of such models, which we refer to as the local and the global representations of recurrent connectivity. Both representations assume that the network consists of P populations, and the statistics of connectivity depend only on the pre- and post-synaptic populations. The two representations are however based on different statistical features of the connectivity.

The local representation defines the connectivity statistics by starting from the marginal distributions Prob(J_ij = J) of individual synaptic weights, and by including progressively higher-order correlations referred to as connectivity motifs [1, 10, 14]. In this work, we will consider only the first two orders, i. e. the distribution of individual weights and the pairwise correlations η_ij between reciprocal connections J_ij and J_ji that quantify pairwise motifs (Fig 1A). Our key assumption is that both the marginal distributions of J_ij and the correlations η_ij depend only on the populations p and q that the post- and pre-synaptic neurons belong to: (1)

All synapses connecting the same two populations therefore have identical statistics, leading to a block-like statistical structure for the connectivity matrix J (Fig 1B left panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Local vs global representations of recurrent connectivity.

(A) The local representation defines the statistics of synaptic weights J_ij by starting from the marginal probability distribution of individual synaptic weights (left) and then specifying reciprocal motifs in terms of correlations η_ij between reciprocal weights J_ij and J_ji connecting neurons i and j (right). Both the marginal distribution and the reciprocal correlations are assumed to depend only on the populations p and q that the neurons i and j belong to. (B) The resulting connectivity matrix J has block-structured statistics, where different blocks correspond to connections between the P different populations (P = 2 in this illustration). It can be decomposed into a superposition of a mean component , and a remaining zero-mean random connectivity component Z that has block-structured variances. (C) The global, low-rank representation defines the connectivity matrix J as the sum of R outer products between connectivity vectors m^(r), n^(r) for r = 1…R. The statistics of connectivity are defined in terms of the joint probability distribution over neurons i of their entries on connectivity vectors. We specifically consider the class of Gaussian-mixture low-rank models, where each neuron is first assigned to a population p, and within each population the entries on connectivity vectors are generated from a multivariate Gaussian distribution with fixed statistics. Here we illustrate this distribution for one pair of connectivity vectors (R = 1) and P = 2 populations. Each dot represents the connectivity parameters of one neuron i, the red and blue colours denote the two populations, white dots and the rotations of the dot clouds indicate the mean and covariance of the distribution for each population. (D) Relating the local and global representations of recurrent connectivity for a simplified excitatory-inhibitory network. In this model, the mean connectivity depends only on the presynaptic population (indicated by red and blue colours). The mean connectivity is in this case rank-one, and can be written as an outer product of vectors and . We approximate the full connectivity by a rank-one matrix, with connectivity vectors m and n obtained from and using perturbation theory.

https://doi.org/10.1371/journal.pcbi.1010855.g001

The global representation of connectivity instead refers to the situation where J is defined as a low-rank matrix [30, 31, 33]: (2)

Here and for r = 1…R are referred to as connectivity vectors, where R is the rank of J. In this representation, the statistics of connectivity are defined by the distribution of vector elements, rather than directly by the distribution of synaptic weights as in the local representation. Specifically, each neuron i is characterized by its set of entries over the connectivity vectors. For each neuron, these 2R entries are generated from a joint distribution, independently of the other neurons, and the parameters of this joint distribution depend on the population p the neuron belongs to. Here we focus on the broad class of Gaussian-mixture low-rank networks, in which for population p, the joint distribution of elements is a multi-variate Gaussian defined by the means and covariances of the 2R entries [31, 33] (Fig 1C).

To relate the local and the global representations of connectivity, a key observation is that any matrix J generated from the local statistics defined in Eq (1) can be expressed as (3) where contains the mean values of the connections, and Z contains the remaining, zero-mean random part [40]. Because of the underlying population structure (Eq (1)), consists of P × P blocks with identical values within each block (Fig 1B middle panel), and is therefore at most of rank P. The random part Z is instead in general of rank N, but obeys block-like statistics, with variance and normalized covariance parameters defined by P × P matrices (Methods Secs. 2.1.1–2.1.2, Eqs (27) and (32)).

For the sake of simplicity, in this study, we focus on a simplified excitatory-inhibitory model [41, 43]. This network consists of one excitatory and one inhibitory population, so P = 2 and in the following we use the population indices p, q = E, I. A central simplifying assumption in this model is that the mean synaptic weights depend only on the pre-synaptic population, so that and . The mean connectivity matrix therefore consists of only two blocks and is unit rank (Fig 1D). The statistics of the random part Z instead depend on both pre- and post-synaptic populations, and are therefore described by 2 × 2 matrices of variance and normalized covariance parameters (see Methods Sec. 2.1.3, Eq (37)).

1.2 Approximating locally-defined connectivity with low-rank connectivity

To relate the local and global representations of connectivity, we start from a connectivity matrix J generated from the local statistics (Eq (1)) and approximate it by a rank-R matrix of the form given in Eq (2). As the locally-defined connectivity matrix J is of rank N, this is equivalent to the classical low-rank approximation problem, for which a variety of methods exist [30, 31, 33]. Here we use simple truncated eigen-decomposition as it preserves the dominant eigenvalues that determine nonlinear dynamics.

Applying the standard eigenmode decomposition, J can be in general factored as (4) where m^(r) and n^(r) are rescaled versions of the r-th right and left eigenvectors (Methods Sec. 2.3, Eqs (44)–(49)), ordered by the absolute value of their eigenvalue λ_r for r = 1…N. A rank-R approximation that preserves the top R eigenvalues can then be obtained by simply keeping the first R terms in the sum in Eq (4). In this study, we focus on R = 1, corresponding to the dominant eigenvalue. Higher rank approximations will be described elsewhere.

Eigenvalues and eigenvectors are in general complex nonlinear functions of the entries of the matrix J. To determine the dominant eigenvalues and the corresponding vectors of J, we capitalize on the observation in Eq (3) that a locally-defined connectivity matrix can in general be expressed as a sum of a low-rank matrix of mean values and the remaining random part Z. Previous studies have found that the eigenspectra of matrices with such structure typically consist of two components in the complex plane: a continuously-distributed bulk determined by the random part, and discrete outliers controlled by the low-rank structure [30, 32, 39, 44–46]. In this study, we extend previous approaches to determine the influence of the block-like statistics of Z on the outliers that correspond to dominant eigenvalues. We then use perturbation theory to determine the corresponding left and right eigenvectors and their statistical structure. Here we summarize the main steps of this analysis (full details are provided in Methods), and then apply it to specific cases in the following sections.

We focus on the simplified E-I network for which the mean part of the connectivity is unit rank and can therefore be written as , so that the full connectivity matrix is (5)

The mean part of the connectivity has a unique non-trivial eigenvalue which can give rise to one or several outliers λ in the eigenspectrum of J. To determine how the random part of the connectivity influences λ, we start from the characteristic equation for the eigenvalues of J and exploit Eq (5) to apply the matrix determinant lemma (Eq (59)) and get (6)

As long as (I − Z/λ) is invertible, this equation determines the eigenvalues of .

We next assume that the maximal eigenvalue of Z is smaller than the outlying eigenvalues corresponding to the low-rank approximation we aim to determine (Eq (4), Methods Sec. 2.2 Eqs (40) and (41)). This condition holds as long as the variance amplitudes of the random part Z of the connectivity are not too large [15, 18](S4 Text). This assumption allows us to do series expansion which leads to a nonlinear equation for λ [32]: (7)

Although this nonlinear equation is a polynomial with infinite terms, there are at most finite N solutions for the eigenvalue outliers [32]. More specifically, in this work, we are only focusing on the second-order reciprocal motifs in the random component. The second order coefficient θ₂ in Eq (7) is the first non-trivial term for the reciprocal case, so we truncate the series summation at k = 2 (included) to provide a more straightforward and understandable comprehension of how reciprocal motifs modify the eigenvalue outlier. In addition, we give the computation of eigenvalue outliers without truncation, and more elaborate explanations in S6 Text.

Truncating the sum to second order yields an approximate third order polynomial for λ: (8)

The statistics of the outlying eigenvalue can then be obtained by averaging over the random part of the connectivity Z.

An approximate expression for the right and left connectivity vectors m and n of J corresponding to the outliers λ can be determined using first order perturbation theory [47], where we treat the variance amplitudes of the random part Z of the connectivity as small parameters. We first note that and are the right- and left-eigenvectors of corresponding to the non-trivial eigenvalue λ₀. Interpreting the full connectivity matrix J as perturbed by a random matrix Z, at first order m and n can be expressed as (9) with (10)

A key observation is that each element of Δm and Δn is a sum of N random variables. If the elements {z_ij} in Z are random samples drawn from a distribution with overall mean and finite variance, the central limit theorem holds and therefore predicts that, in the limit of large N, the statistics of Δm_i and Δn_i, and therefore m_i and n_i, follow a Gaussian distribution. In general, the mean and variance of m_i and n_i and their correlation are determined by the mean, variance and correlation of the elements of J, but not the specific form of the probability distribution. Since the matrix Z has block-like statistics determined by the population structure, the statistics of the resulting m_i and n_i depend on the population p the neuron i belongs to. Overall, the distribution of elements of m and n obtained from perturbation theory therefore follow a Gaussian-mixture distribution, so that our approach effectively approximates a locally-defined J by a Gaussian-mixture low-rank model specified by the means , the variances and the covariances of the entries on the connectivity vectors for p = E, I.

We next apply the perturbative approach described here to networks with independent random components, and then to networks with reciprocal motifs.

1.3 Low-rank structure induced by independently generated synaptic connections

We first apply our approach for a low-rank approximation to the simplest version of the locally-defined excitatory-inhibitory network where each J_ij is generated independently from a Gaussian distribution with a mean that depends only on the pre-synaptic population, i. e. with p, q ∈ E, I. The entries of the eigenvectors and of the mean connectivity matrix are then given by: (11) (12) (13)

In the large network limit, averaging over Z in Eq (7) yields [θ_k] = 0 for all k > 0 [32], so that the outlier is on average given by [λ] = θ₀ = λ₀ (Fig 2A). Our approach moreover gives an expression for the standard deviation of the outlier in the finite-size network, which grows linearly with g (Fig 2B, Eq (80)). Examining the entries of the left and right eigenvectors n and m of J corresponding to the outlier, as expected, the distribution of (m_i, n_i) is well described by a mixture of two Gaussians centred at (Fig 2C). We further find that perturbation theory accurately accounts for the individual entries of the eigenvectors as long as g is sufficiently below unity (Fig 2D). Perturbation theory provides a lower bound for the values of the corresponding variances. For large values of g, the distributions remain Gaussian, but their variances increase above the predictions of perturbation theory (Fig 2E). The reason for this deviation from the theory is that perturbation theory assumes a small value of the scaling factor g in the variance of J_ij; as a result, the deviation also reflects the systematic error resulting from the first-order approximation. Importantly, the entries of the left and right eigenvectors are uncorrelated, and only their means, but not their variances, differ between the two populations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Eigenvalues and dominant eigenvectors for locally-defined Gaussian connectivity with independent synaptic weights.

(A) Eigenvalue spectra of excitatory-inhibitory connectivity matrices J with elements generated from Gaussian distributions with identical variances g²/N over neurons. The coloured dots in the circular bulk shows 600 eigenvalues for one realization of the random connectivity for each value of g. Different colours correspond to different values of g. Dashed envelopes indicate the theoretical predictions for the radius r_g of the circular bulk computed according to Eqs. (147), (148). Outlying eigenvalues are shown for 30 realizations of the random connectivity, and for different g their location on the y-axis is shifted to help visualization, the dispersion reflects finite-size effects. The red arrow points to the eigenvalue λ₀ of the mean connectivity matrix . (B) Statistics of outlying eigenvalues over realizations of random connectivity. Empirical distribution (the red area shows mean ± standard deviation and reflects finite-size effects), compared with the theoretical predictions for the mean (black dashed line) and standard deviation (gray dashed line) obtained using Eq (80). (C) Scatter plot showing for each neuron i its entry n_i on the left eigenvector against its entry m_i on the right eigenvector. Red and blue colours represent respectively excitatory and inhibitory neurons. The white dots and the dashed lines respectively indicate the means and covariances for each population obtained from simulations. (D) Comparison between eigenvector entries obtained from direct eigen-decomposition of J with projections obtained using perturbation theory (Eqs (9) and (10)) in a given realization of Z. (E) Comparison between simulations (dashed lines) and theory (full lines) for the variances , of eigenvector entries corresponding to different populations (Eq (90)). To help visualization, the curves for the excitatory population are thicker than the curves for the inhibitory population. (F-J) Identical quantities for connectivity matrices in which the variance parameters are heterogeneous: g_EE : g_EI : g_IE : g_II = 1.0 : 0.5 : 0.2 : 0.8, g_EE increases from 0 to 1. Other network parameters N_E = 4N_I = 1200 and J_E = 2.0, J_I = 0.6 in all simulations.

https://doi.org/10.1371/journal.pcbi.1010855.g002

We next turn to the case where the variances of synaptic weights depend on the pre- and post-synaptic populations q, p, and are given by . In that case, the entries of the random part of the connectivity Z are independent, but not identically distributed Gaussians. Previous studies [11, 44] have shown that the spectrum of Z remains circularly symmetric, but its radius r_g is determined by a combination of variance parameters g_pq (S4 Text, Eqs. (147), (148)). Examining the resulting connectivity matrix J, we found that the results for the uniform case directly extend to this heterogeneous situation. The eigenspectrum of J still consists of an independent superposition of the spectra of Z and (Fig 2F). In particular, the random part of the connectivity does not modify the average value of the outlier, but only impacts its variance, which now depends on a combination of the variances g_pq (Fig 2G, Eq (90)). Similarly to the uniform case, the distribution of the entries of the left and right eigenvectors is well described by a mixture of two Gaussians, with variances predicted by perturbation theory. The entries of the left and right eigenvectors are uncorrelated, but now both their means and variances depend on the population the neuron belongs to (Fig 2H–2J).

In summary, when synaptic connections J_ij are generated independently across pairs of neurons, the equivalent global representation is a Gaussian-mixture low-rank model where the entries of the structure vectors are independent with mean values determined by the low-rank structure of the mean connectivity matrix . Although the mean connectivity establishes the network’s basic structure, the Gaussian-mixture low-rank approximation improves on it in terms of preserving connectivity fluctuations and reflecting the cell-type-dependent structural variances in the original locally-defined random connectivity part Z. Importantly, in that situation, the dominant outlying eigenvalues of J are on average identical to those of , that is, [λ] = λ₀.

1.4 Low-rank structure induced by reciprocal motifs

We next turn to locally-defined excitatory-inhibitory networks with reciprocal connectivity motifs quantified by the correlation η_ij between reciprocal synaptic weights J_ij and J_ji. We assumed that these reciprocal correlations are identical for any pair of neurons i and j belonging to a given pair of populations p and q, and used the corresponding parameters η_pq to generate the connectivity matrix J (Methods Sec. 2.1.2). Within the decomposition of J in a mean and random part Z (Eq (24)), the additional reciprocal correlations affect only the statistics of Z.

We first consider the homogeneous case where the reciprocal correlation is identical across all populations, i. e. η_pq = η (Fig 3). Previous studies have shown that a random matrix Z with zero mean and reciprocal correlations η has a continuous spectrum that is deformed from a circle into an ellipse as η is increased [18, 48]. Superpositions between correlated random matrices, and low-rank structure such as have, to our knowledge, not been previously studied. Inspecting the eigenspectrum of , we find that it still consists of a continuous bulk and discrete outliers (Fig 3A). The continuous bulk is contained in an ellipse in the complex plane identical to the spectrum of Z, as in the uncorrelated case. In contrast, we find that the outliers deviated from the eigenvalues of as η is increased (Fig 3B). These deviations are well captured by our analytic approach summarized in Eq (8). Indeed, when averaging Eq (8) over Z, reciprocal correlations generate a non-zero due to Z². This term leads to a cubic equation in Eq (8) and therefore has two effects. First, the non-zero θ₂ induces deviations of the outliers from the eigenvalue λ₀ of . The direction of these deviations is positive if excitation dominates (λ₀ = J_E − J_I > 0) and negative if inhibition dominates (λ₀ = J_E − J_I < 0, Fig 3G–3I). Second, the cubic equation can have up to three solutions and therefore potentially generates additional outliers, and in particular complex conjugate ones. Whether these additional outliers are observed depends on the accuracy of the third-order approximation (Eq (8)) to the determinant lemma (Eq (59)), and on the norm of these outliers compared to the spectral radius, in both scenarios where the network has a homogenous variance g²/N (Fig 3A and 3G) and when g_pq differ between populations (Fig 3J). Please refer to S6 Text for a more thorough discussion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Eigenvalues and dominant eigenvectors for locally-defined Gaussian connectivity with homogeneous reciprocal correlations.

(A) Eigenvalue spectra of excitatory-inhibitory connectivity matrices J, with homogeneous reciprocal correlations η. Different colours from top to bottom correspond to networks with different values of η. The dots in the elliptical bulk show 600 eigenvalues for one realization of the random connectivity. Outlying eigenvalues are shown for 30 realizations of the random connectivity, the dispersion reflects finite-size effects. The red arrow on the top points to the eigenvalue λ₀ of the mean connectivity . Coloured circles are the eigenvalues predicted using determinant lemma and truncated series expansion (Eqs (7) and (8), truncating at k = 2), coloured triangles are the eigenvalues predicted using determinant lemma without finite truncation (see S6 Text, Eqs. (153)-(157)). (B) Comparison of the eigenvalues from the finite-size simulation with the predictions of the determinant lemma as the reciprocal correlation η is increased. The coloured solid lines show the roots of the third-order polynomial in Eq (8) (truncated series expansion). The light purple area indicates the empirical distribution of the dominant outlier for 30 realizations, reflecting finite-size effects, while the black dashed line is the unperturbed eigenvalue λ₀. The grey areas represent the areas covered by the eigenvalue bulk. (C) Scatter plot showing for each neuron i its entry n_i on the left eigenvector against its entry m_i on the right eigenvector. Red and blue colours represent respectively excitatory and inhibitory neurons. The white dots and the dashed lines respectively indicate the means and covariances for each population. (D) Comparison between eigenvector entries obtained from direct eigen-decomposition of J with projections obtained using perturbation theory (Eqs (9) and (10)) in a given realization of Z. (E) Comparison between simulations (coloured areas, finite-size effects) and theoretical predictions (coloured lines, Eq (97)) for the population covariance of the entries on the left and right connectivity eigenvectors to different populations. (F) Comparison of the overall covariance σ_nm (Eq (72)) with the deviation Δλ of the dominant outlying eigenvalue from the unperturbed value λ₀. Empirical covariance (gradient blue area reflects finite-size effects, where the colour depth represents η) compared with the theoretical prediction (black line) obtained using Eqs (97) and (92). The x-axis uses the theoretical prediction of the deviation of the eigenvalue λ from λ₀. Other network parameters: J_E = 2.0, J_I = 1.2, N_E = 4N_I = 1200 and homogeneous variance parameters g_pq = g = 0.3. (G-I) Same as (A, B, F) for an inhibition dominates connectivity matrix where J_I = 2.0, J_E = 1.2, with homogeneous reciprocity η and variance parameters g = 0.3. (J-M) Same as (A-C) and (E) for excitatory-and-inhibitory connectivity matrices with homogeneous reciprocal correlations η but cell-type-dependent variance parameters g_EE : g_EI : g_IE : g_II = 1.0 : 0.5 : 0.2 : 0.8 and g_EE = 0.3. In (B, E, F, H, I, K, M), the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the first-order perturbation approximation of eigenvalues and eigenvectors.

https://doi.org/10.1371/journal.pcbi.1010855.g003

We next examine the right- and left-eigenvectors m and n corresponding to the dominant outlier. Analogous to the uncorrelated case in the networks with independent connections, perturbation theory accounts for the individual entries of these vectors from a specific realization of the locally-defined random connectivity Z, and these individual entries m_i and n_i exhibit Gaussian-mixture statistics (Fig 3C and 3D, Eq (10), and Methods Sec. 2.3 Eqs (69) and (70)). Unlike in the uncorrelated case, reciprocal correlations now induce correlations between Δm_i and Δn_i (Methods Sec. 2.5.2). Indeed, perturbation theory predicts that the first-order effects Δm and Δn of the random connectivity on m and n are respectively determined by Z and its transpose Z^⊤ (Eq (10)). Reciprocal correlations between z_ij and z_ji directly lead to correlations between Z and Z^⊤ and therefore a non-zero covariance σ_nm between elements of m and n, that can be predicted by mean field theory (Fig 3E, Eq (72)). The strength of the covariance between eigenvector entries reflects the strength of the additional feedback loop due to reciprocal correlations, and is therefore directly related to the deviations of the outlying eigenvalue from the uncorrelated value λ₀ (Eqs (42) and (96), Fig 3F). When the network has both homogeneous variance parameters and correlation parameters, the excitatory and inhibitory populations have the same covariance (Eq (97)). If the synaptic variances g_pq differ across populations, the covariances σ_nm are different for excitatory and inhibitory populations even if the reciprocal correlations are uniform (Fig 3L and 3M).

These results directly extend to networks with heterogeneous reciprocal correlations η_pq, p, q = E, I (Fig 4). Finite-size simulations in this circumstance show the existence of additional, complex conjugate outliers accurately predicted by the cubic term in Eq (8) (Fig 4G, coloured circles and triangles are overlapping, containing outlier scatters at conjugate positions), and this is particularly true in the case where the impact of higher-order structures is marginal compared to that of structures up to the second-order. Moreover, the covariances between the entries of low-rank connectivity vectors in this case differ between the excitatory and inhibitory population.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Eigenvalues and dominant eigenvectors for network connectivity matrix with heterogeneous reciprocal correlations.

(A) Eigenvalue spectra of excitatory-inhibitory connectivity matrices J, with homogeneous variance parameter g = 0.3 but cell-type-dependent reciprocal correlations η_EE = η_EI = −η_II > 0, and η_EE increasing from 0 to 1. Different colours from top to bottom correspond to networks with different values of η_EE. The dots in the elliptical bulk show 600 eigenvalues for one realization of the random connectivity. Outlying eigenvalues are shown for 30 realizations of the random connectivity, the dispersion reflects finite-size effects. The red arrow on the top points to the eigenvalue λ₀ of the mean connectivity . Coloured circles are the eigenvalues predicted using determinant lemma and truncated series expansion (see Eqs (7) and (8), truncating at k = 2), coloured triangles are the eigenvalues predicted using determinant lemma without finite truncation (see S6 Text, Eqs. (153)-(157)). (B) Comparison of the eigenvalues from the finite-size simulation with the predictions of the determinant lemma as the reciprocal correlation η_EE (−η_II) is increased. The coloured solid lines show the roots of the third-order polynomial in Eq (8) (truncated series expansion). The light purple area indicates the empirical distribution of the dominant outlier for 30 realizations, reflecting finite-size effects; while the black dashed line is the unperturbed eigenvalue λ₀. The grey areas represent the areas covered by the eigenvalue bulk. (C) Scatter plot showing for each neuron i its entry n_i on the left eigenvector against its entry m_i on the right eigenvector. Red and blue colours represent respectively excitatory and inhibitory neurons. The white dots and the dashed lines respectively indicate the means and covariances for each population. (D) Comparison between eigenvector entries obtained from direct eigen-decomposition of J with projections obtained using perturbation theory (Eqs (9) and (10)) in a given random connectivity Z. (E) Comparison between simulations (coloured areas, finite-size effects) and theoretical predictions (coloured lines, Eq (97)) for the population covariance of the entries on the left and right connectivity eigenvectors to different populations. (F) Comparison of the overall covariance σ_nm (Eq (72)) with the deviation Δλ of the dominant outlying eigenvalue from the unperturbed value λ₀. Empirical covariance (gradient blue area reflects finite-size effects, where the colour depth stands for η) compared with the theoretical prediction (black line) obtained using Eqs (97) and (92). The x-axis uses the theoretical prediction of the deviation of the eigenvalue λ from λ₀. (G-L) Same as (A-F) for a connectivity matrix with heterogeneous reciprocal correlations: η_EE = −η_EI = −η_II > 0, and η_EE increasing from 0 to 1. In (B, E, F, H, K. L), the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the first-order perturbation approximation of eigenvalues and eigenvectors. Other network parameters: N_E = 4N_I = 1200 and homogeneous variance parameters g_pq = g = 0.3 in all simulations, J_E = 2.0, J_I = 1.3 for networks in (A-F), J_E = 2.0, J_I = 1.4 for networks in (G-L).

https://doi.org/10.1371/journal.pcbi.1010855.g004

1.5 Approximating low-dimensional dynamics for locally-defined connectivity

In previous sections, we developed a rank-one approximation of locally-defined excitatory-inhibitory connectivity. Here we use this approximation to describe the resulting low-dimensional dynamics. We consider networks of rate units, where the activation x_i of unit i obeys (14)

Here ϕ(x) = 1 + tanh(x − θ) is a positive transfer function, and for simplicity, we focus on autonomous dynamics without external inputs. We start from a locally-defined excitatory-inhibitory connectivity matrix, and compare the resulting activity with the theoretical predictions of our rank-one approximation, for which the dynamics are low-dimensional and analytically tractable. We first summarize the theoretical predictions for those dynamics, and then examine the specific cases of independent and reciprocally-correlated connectivity.

Recent works have showed that in networks with a rank R connectivity matrix, the trajectories x(t) = {x_i(t)}_i=1…N are confined to a low-dimensional subspace of the N − dimensional space describing the activity of all units [30–33]. In absence of external inputs, this subspace is R-dimensional and spanned by the set of connectivity eigenvectors m^(r) for r = 1…R, so that the trajectories can be parametrized as where κ_r is a collective latent variable representing activity along m^(r). For a rank-one (R = 1) connectivity corresponding to an approximation of our locally-defined E-I network, the dynamics can therefore be represented by a single latent variable κ, so that the activation of unit i is given by (15) (16) where we inserted the expression for m_i obtained from a first-order perturbation (Eqs (9) and (10)). Note that since , the first term in the r. h. s. of Eq (16) corresponds to population-averaged mean activity , while the second term is the deviation of the activity of unit i from the population average, which statistically leads to the population-averaged variance of neuronal activations (Methods Sec. 2.6.1, Eq (111)) (17)

By inserting the values for Δm_i obtained using specific realizations of random connectivity in Eq (10) into Eq (16), the rank-one approximation accounts for the heterogeneous firing activity of single units coming from the synaptic weights in specific instances of locally-defined networks Z. Moreover, the rank-one theory predicts that both the population-averaged mean and the standard deviation of activations in the network are proportional to κ.

The values taken by the latent variable κ can be determined by projecting Eq (14) onto m and inserting Eq (16) (Methods Secs. 2.6.1, 2.6.2). This leads to a closed equation for the dynamics of κ(t): (18) where the brackets denote a Gaussian average (see Eq (112)), α_p is the fraction of neurons belonging to population p, is the population covariance between elements m_i and n_i with i belonging to population p (Table 1). The steady state then obeys (19) where (20)

The two terms in the r. h. s. of Eq (19) show that the contributions of recurrent synaptic inputs to the latent dynamics κ come from two sources: (i) the population means of the left and right connectivity eigenvectors and that contribute to F_mean(κ) (Eqs (54) and (56)); (ii) the covariance between the left and right connectivity eigenvectors that contributes only to F_cov(κ). In the low-rank approximation (Eqs (2) and (41)) of the locally-defined E-I connectivity (Eq (5)), these two terms have distinct origins: the mean comes from the independent components of the connectivity (Eqs (55) and (56)); while the covariance comes from reciprocal correlations between connections (Eqs (96) and (97)). We next examine separately the effects on dynamics of these two connectivity components.

1.5.1 Independently generated local connectivity.

When synaptic connections are generated independently from a Gaussian distributions based on the identities of pre- and post-synaptic populations, the rank-one approximation of connectivity leads to uncorrelated left and right connectivity vectors n and m, so that for p = E, I. In consequence, only the first term is present in the r. h. s. of Eq (19), and the fixed point of the latent dynamics is given by a difference between excitatory and an inhibitory feedback (Eq (116)): (21)

As long as the mean inhibition J_I is strong enough to balance the mean excitation J_E, Eq (21) predicts a single fixed point. As J_E is increased, positive feedback begins to dominate and leads to a bifurcation to a bistable regime for the latent dynamic variable κ (Fig 5A–5C, S2 Text).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Predicting low-dimensional dynamics using a rank-one approximation of networks with independent Gaussian connectivity.

(A) Fixed points of the latent variable κ in the rank-one approximation. The lines show the dynamics as function of κ, predicted by Eq (18) (solid line: J_E = 2.4; dashed line: J_E = 1.5). The intersections with y = 0 correspond to fixed points (filled dots: stable; unfilled dot: unstable). (B) Contribution of mean connectivity to the latent dynamics, F_mean(κ) in Eq (20), for two values of J_E. (C) Bifurcation diagram for increasing J_E: analytical predictions of Eq (21) compared with simulations of the full network with locally-defined connectivity. orange line: analytical prediction including only the mean part of the connectivity ( in Eq (21)); purple line: analytical prediction including the first-order perturbation term in the rank-one approximation; gray: projection of simulated activity x onto the connectivity vector m computed by perturbation theory for 30 realizations of random connectivity Z, the shaded area reflects finite-size effects (Eq (104)). (D) Comparison between simulations and mappings obtained from the low-rank approximation (Eqs (10) and (16)) for the activity of individual units in a given realization of the random connectivity Z. For each unit i, a dot shows the deviation Δx_i of its steady-state activity from the population average, against its value Δm_i of the perturbed part of the connectivity vector m (Eq (16)). The low-rank theory predicts Δx_i = κΔm_i. Orange, cyan and gray scatters show excitatory or inhibitory populations, each for several values of J_E. Lines represent y = κx, where κ is obtained from Eq (21). Upper panels show the result in a realization with a high fixed point, bottom panels show the result in a realization with a low fixed point. (E) Comparison between the predictions (solid lines) and simulations (shaded areas) for the population-averaged variances of Δx_i. Shaded areas show mean±std and reflect finite-size effects. In C, E, the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the approximation of the latent dynamical variable κ as well as the low-rank connectivity statistics. Other network parameters: N_E = 4N_I = 1200, J_I = 0.6, g_EE: g_EI: g_IE: g_II = 1.0: 0.5: 0.2: 0.8 and g_EE = 0.8. The transfer function ϕ has parameter θ = 1.5.

https://doi.org/10.1371/journal.pcbi.1010855.g005

This bistability due to positive feedback is expected on the basis of mean connectivity alone. Indeed, replacing the connectivity matrix by its mean is equivalent to a rank-one approximation with and which lead to Eq (115) with for p = E, I, this reduced model actually corresponds to the classic Amari-Grossberg rate model [49–52]. The additional first-order perturbation term in the rank-one approximation (Eqs (9) and (10)) additionally takes into account fluctuations in the connectivity, which leads to a non-zero , and modifies the fixed points predicted by Eq (21). In consequence, the bifurcation to bistability takes place at higher values of J_E than predicted from mean connectivity alone (purple lines compared to orange lines in Fig 5C).

More importantly, we find that the first-order perturbation in the rank-one approximation accurately accounts for the heterogeneous firing rates of individual neurons for specific instances of the random, locally-defined connectivity Z (Eqs (16) and (69), Fig 5D), and therefore predicts well the variance across neurons of the steady state of population dynamics . In particular, cell-type dependent variances in the synaptic connectivity, lead to distinct variances and for excitatory and inhibitory populations (Eqs (90) and (130), Fig 5E).

Note that the independently generated local connectivity can be treated analytically without resorting to a rank-one approximation, by using a different variant of mean-field theory originally developed for randomly connected networks. [30, 53]. That theory is not perturbative, and takes into account an additional term in the variance (see Methods Sec. 2.6.3 and S3 Text for more details). However, in contrast to the rank-one approximation, it does not predict the activity of individual neurons, and is challenging to extend beyond independent random connectivity.

1.5.2 Reciprocal motifs.

We next turn to the predictions of the rank-one approximation for dynamics resulting from locally-defined connectivity with reciprocal motifs. In this case, the additional reciprocal correlations in the random part of the connectivity lead to a non-zero covariance σ_nm between the connectivity vectors n and m in the rank-one approximation (Eqs (92) and (97)). This covariance in turn generates an additional feedback component in the dynamics of the latent variable, the second term in the r. h. s. of Eq (19).

Specifically, the excitation-dominated dynamical regime is largely determined by the noiseless mean connectivity , but beyond this mean approximation, positive reciprocal correlations combined with the excitation-dominated connectivity enhance positive feedback with respect to mean connectivity alone (Methods Sec. 2.5.2, Eq (97)). As a result, progressively increasing the reciprocal correlations can therefore induce a bifurcation to bistability, even if the mean excitation provided by the mean approximation is not sufficient by itself to support two stable states (Fig 6A–6C). This is a major novel effect of reciprocal motifs on collective dynamics. As in the case of independent connectivity, we moreover find that the perturbative term in the rank-one approximation accounts for the heterogeneous activity of individual neurons in specific realizations of the connectivity Z (Fig 6D, Eqs (16) and (69)), and therefore also predicts the cell-type dependent variances of activity (Fig 6E).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Predicting low-dimensional dynamics using a rank-one approximation of networks with homogeneous reciprocal motifs.

(A) Influence of reciprocal correlations on fixed points of the latent variable κ in the rank-one approximation. The lines show the dynamics as function of κ, predicted by Eq (18) (solid line: η = 1; dashed line: η = 0). The intersections with y = 0 correspond to fixed points (filled dots: stable; unfilled dot: unstable). (B) Comparison of the contributions of mean connectivity F_mean(κ) and covariance F_cov(κ) to the latent dynamics of κ (Eq (20)) for η = 1. (C) Bifurcation diagram for increasing η at fixed J_E. Solid purple lines: analytical predictions of Eqs (19) and (20); gray areas: projection of simulated activity x onto the connectivity vector m computed by perturbation theory (Eq (104)) for 30 realizations of random connectivity Z, the shaded area reflects finite-size effects. (D) Comparison between simulations and mappings obtained from the low-rank approximation for the activity of individual units in a given realization of the random connectivity Z. For each unit i, a dot shows the deviation Δx_i of its steady-state activity from the population average, against its value Δm_i of the perturbed part of the connectivity vector m (Eq (16)). The low-rank theory maps Δx_i = κΔm_i. Orange, cyan and gray scatters show excitatory or inhibitory populations, each for two values of η. Lines represent y = κx, where κ is obtained from Eqs (19) and (20). Upper panels show the result in a realization with a high fixed point, bottom panels show the result in a realization with a low fixed point. (E) Comparison between the predictions (solid lines, Eq. (134)) and simulations (shaded areas) for the population-averaged variances of Δx_i. Shaded areas show mean±std and reflect finite-size effects. In C, E, the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the approximation of the latent dynamical variable κ as well as the low-rank connectivity statistics. Other network parameters: N_E = 4N_I = 1200, J_E = 1.9, J_I = 0.6, g_EE: g_EI: g_IE: g_II = 1.0: 0.5: 0.2: 0.8 and g_EE = 0.8. The transfer function ϕ has parameter θ = 1.5.

https://doi.org/10.1371/journal.pcbi.1010855.g006

More generally, our rank-one approximation allows us to describe the latent dynamics when the degree of reciprocal correlation depends across the pre- and post-synaptic populations (Results Sec. 1.4). Such heterogeneity in reciprocal correlations can enhance different types of feedback. For example, antisymmetric connectivity within inhibitory populations (η_II < 0) disinhibits excitatory population and thus facilitates bistable transitions (Fig 7A–7C) compared to networks with homogeneous reciprocal correlations. In contrast, excitation-dominated connectivity with homogeneous negative reciprocity (η < 0) generate negative feedback and therefore suppress the global dynamics from bistable state to quiescent (Fig 7D–7F).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Predictions for low-dimensional dynamics using a rank-one approximation of networks with non-homogeneous and anti-symmetric reciprocal motifs.

(A-D) Fixed points of latent dynamics in networks with heterogeneous, cell-type-dependent reciprocal correlations: η_EE = η_EI = −η_II. Network parameters: N_E = 4N_I = 1200, g_EE: g_EI: g_IE: g_II = 1.0: 0.5: 0.2: 0.8, g_EE = 0.8, J_E = 1.9 and J_I = 0.6. (E-H) Fixed points of latent dynamics in networks with homogeneously anti-symmetric motifs, η_pq = η ∈ [−1, 0]. Network parameters: N_E = 4N_I = 1200, g_EE: g_EI: g_IE: g_II = 1.0: 0.5: 0.2: 0.8, g_EE = 0.8, J_E = 2.2 and J_I = 0.6. In C and F, gray areas show projections of simulated activity x onto the connectivity vector m computed by perturbation theory (Eq (104)), shaded areas show mean±std and reflect finite-size effects. In C, F the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the approximation of the latent dynamical variable κ as well as the low-rank connectivity statistics.

https://doi.org/10.1371/journal.pcbi.1010855.g007

Importantly, describing the role of reciprocal correlations on latent dynamics relies on our global low-rank approximation of locally-defined connectivity. In particular, the effects of such correlations cannot be captured by considering only mean connectivity and population-averaged activity (first term in the r. h. s. of Eq (19)). Moreover, including reciprocal correlations in classical mean-field approaches to randomly connected networks is technically challenging [18].

1.6 Extension: E-I networks with sparse connectivity

In previous sections, we examined locally-defined connectivity generated using Gaussian distributions of individual synaptic weights (function f^pq in Eq (1)). Our results for the low-rank approximation of locally-defined connectivity are however independent of the precise form of the distribution f^pq. In particular, our finding that the resulting low-rank structure obeys Gaussian-mixture statistics is universal, in the sense that it is valid for any distribution f^pq for which the central limit theorem holds (see Discussion). To illustrate this universality, here we turn to networks with sparse connectivity, generated from Bernoulli distributions f^pq taking values 0 and A_q (where A_q = A_E, A_I refer to strengths of excitatory and inhibitory connections), with a uniform fraction c of non-zero connections (see Eq (28)). In this case, the variance of the synaptic strengths J_ij scales as 1/N, as we assumed for Gaussian connectivity. The statistics of the corresponding rank-one approximation are fully determined by the mean, variance and covariance of synaptic weights in the excitatory and inhibitory populations. Here we compare the predictions of our perturbative approximation with direct simulations of full-rank networks with locally-defined connectivity. We first consider independently generated connectivity, and then turn to reciprocal motifs.

1.6.1 Independently generated sparse connectivity.

For networks with independently generated sparse connectivity, the mean and variance of individual synaptic weights are given by cA_q and , where q = E, I refers to the population of the presynaptic neuron. Previous works have shown that for such sparse networks where the variances of synaptic weights scale as 1/N, the bulk of the eigenvalues are distributed within a circle in the complex plane with a radius r_g determined by Eq. (149) (Fig 8A) [54, 55], as expected from the universality theorem for random matrices [56]. The overall mean of the synaptic weights instead determines the mean outlying eigenvalue (Eq (78), Fig 8A). In sparse networks, the main novelty with respect to the Gaussian case is that the mean and variance of synaptic weights are not independent parameters, but are instead both set by the synaptic strengths A_E and A_I as well as the network’s sparsity c (Eqs (38) and (39)). In consequence, varying these couplings changes both the radius of the bulk and the outlying eigenvalue, and can lead to intersections where the outliers dip into the bulk (see S4 Text for details).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Rank-one approximation and predicted low-dimensional dynamics for sparse excitatory-inhibitory networks.

(A) Left: Comparison of the predicted eigenvalue outlier λ₀ = c(N_EA_E + N_IA_I) (black line) with finite-size simulations (red area shows mean±std for 30 realizations and reflects finite-size effects). The gray area represents the area covered by the eigenvalue bulk. Right: example spectrum of one realization of connectivity matrix with E/I ratio A_E/A_I = 0.33, and the radius r_g of the eigenvalue bulk computed from the statistically equivalent Gaussian connectivity (see S4 Text). (B) Scatter plot showing for each neuron i its entry n_i on the left eigenvector against its entry m_i on the right eigenvector. Red and blue colours represent respectively excitatory and inhibitory neurons. The white dots and the dash lines respectively indicate the means and covariances for each population obtained from simulations. For visualization purposes, the x− and y-axis are scaled unequally. (C) Comparison between eigenvector entries obtained from direct eigen-decomposition of J with projections obtained using perturbation theory (Eqs (9) and (10)) in a given realization of the sparse connectivity J. (D) Bifurcation diagram for increasing the ratio A_E/A_I: analytical predictions of Eq (117) compared with simulations of the full network with locally generated sparse connectivity. Purple line: analytical prediction including the first-order perturbation term in the rank-one approximation; gray: projection of simulated activity onto the connectivity vector m computed by perturbation theory Eq (104) for 30 realizations of sparse connectivty J, the shaded area reflects finite-size effects. (E) Comparison between the predictions (solid lines) and simulations (shaded areas) for the population-averaged variances of Δx_i. Shaded areas show mean±std and reflect finite-size effects. (F) Comparison between simulations and mappings obtained from the low-rank approximation for the activity of individual units in a given realization of the sparse connectivity J. For each unit i, a dot shows the deviation Δx_i of the steady-state activity from the population average, against the corresponding value Δm_i of the perturbed part of the connectivity vector m (Eq (16)). The low-rank theory maps Δx_i = κΔm_i. Orange, cyan and gray scatters show excitatory or inhibitory populations, each for two values of the ratio A_E/A_I. Lines represent y = κx, where κ is obtained from Eqs (19) and (20). Upper panels show the result in a realization with a high fixed point, bottom panels show the result in a realization with a low fixed point. The gray vertical dashed line in A left, D, E correspond to the critical point c at which the absolute value of the outlier is equal to the radius of the eigenvalue bulk. In A, D, E the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the approximation of the low-rank connectivity statistics as well as the latent dynamical variable κ. Network parameters: N_E = 4N_I = 800, c = 0.3, A_E = 0.025. The transfer function ϕ has parameter θ = 1.5.

https://doi.org/10.1371/journal.pcbi.1010855.g008

As expected, as long as the outlier lies outside of the eigenvalue bulk, the statistics of entries on the resulting low-rank approximating eigenvectors are well described by a Gaussian-mixture distribution with parameters fully determined by the mean and variance of the synaptic weights (Eq (91), Fig 8B). Our perturbative approximation Eq (69) as well, accounts for the individual entries of the right- and left-eigenvectors corresponding to the outlier from the specific realization of the fluctuation matrix Z (Fig 8C).

Our predictions for the low-dimensional dynamics based on the rank-one approximation therefore directly extend to sparse networks. Comparing with direct simulations, we found that Eq (117) predicts well the global latent variable κ obtained by projecting the activity x onto the approximated rank-one eigenvector m (Eq (104)). As the E/I ratio is increased, positive feedback increases, and the latent variable κ undergoes a transition from a single fixed point to two bistable states (Fig 8D).

From the statistics of the right eigenvector m (Eq (91)), our analysis predicts the heterogeneity of activity in terms of population-averaged variances (Fig 8E). This heterogeneity is identical in excitatory and inhibitory populations, as their right eigenvectors m^E, m^I have identical fluctuations (Fig 8B and 8C). For individual realizations of the sparse connectivity, the rank-one approximation x = κm moreover captures the activation x_i of individual neurons in the specific simulation instances (Fig 8F).

1.6.2 Sparse EI networks with reciprocal motifs.

For sparse E-I networks, we generate reciprocal motifs by introducing a fraction ρ_pq of reciprocally connected pairs of neurons. Together with the sparsity c and the synaptic strengths, the parameter ρ_pq determines the cell-type dependent reciprocal correlation η_pq (Methods Sec. 2.1.2 Eqs (34) and (39), Fig 9A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Characterizing connectivity statistical properties and low-dimensional dynamics for the sparse network with reciprocal motifs.

(A) Schematics of a sparse EI network with four forms of paired connections. White and black rectangles represent the non-zero excitatory and inhibitory sparse connections. (B) Eigenvalue spectrum of the sparse connectivity (upper panel) and from the equivalent Gaussian connectivity (bottom panel) with reciprocal motifs. Cell-type-dependent reciprocal correlations are η_EE = 0.71, η_EI = −0.71, η_II = −0.43 in both connectivity matrices, continuous eigenvalue bulks show eigenvalues for one realization of the network connectivity. Red arrows point to the unperturbed eigenvalue λ₀. Outlying eigenvalues are shown for 30 realizations of the network connectivity. Coloured circles are the eigenvalues predicted using determinant lemma and truncated series expansion (see Eqs (7) and (8), truncating at k = 2), coloured triangles are the eigenvalues predicted using determinant lemma without finite truncation (see S6 Text, Eqs. (153)-(157)). (C) Comparison of the eigenvalues from the finite-size simulation of the sparse connectivity, with the predictions of the determinant lemma as progressively increasing the reciprocal correlation η_EE (−η_EI). The coloured solid lines show the roots of the third-order polynomial in Eq (8). The purple area indicates the empirical distribution (mean±std, finite-size effects) of the dominant outlier for 30 realizations of sparse connectivity J, while the black dashed line is the eigenvalue λ₀ of the corresponding independent sparse connectivity matrix (Eq (78)). The gray areas correspond to the areas covered by the eigenvalue bulk. (D) Scatter plot showing for each neuron i its entry n_i on the left eigenvector against its entry m_i on the right eigenvector. Red and blue colours represent respectively excitatory and inhibitory neurons. The white dots indicate the means for each population obtained from simulations. For visualization purposes, the x- and y-axis are scaled unequally. (E) Comparison between eigenvector entries obtained from direct eigen-decomposition of J with projections obtained using perturbation theory (Eqs (9) and (10)) in a given realization of the sparse connectivity J. (F) Comparison between the population covariance of the entries on the left and right connectivity eigenvectors to different populations (coloured areas, finite-size effects) and the predictions of perturbation theory (coloured lines, Eq (99)). (G) Bifurcation diagram for increasing the reciprocal correlation η_EE (−η_EI): analytical predictions of Eq (119) compared with simulations of the full network with locally generated sparse connectivity and reciprocal motifs. Purple line: analytical prediction including the first-order perturbation term in the rank-one approximation; gray: projection of simulated activity onto the connectivity vector m computed by perturbation theory Eq (104) for 30 realizations of sparse connectivity J, the shaded area reflects finite-size effects. (H) Comparison between simulations and mappings obtained from the low-rank approximation for the activity of individual units in a given realization of the sparse connectivity. For each unit i, a dot shows the deviation Δx_i of the steady-state activity from the population average, against its value Δm_i of the perturbed part of the connectivity vector m (Eq (16)). The low-rank theory maps Δx_i = κΔm_i. Orange, cyan and gray scatters show excitatory or inhibitory populations, each for two values of the reciprocal correlation η_EE. Lines represent y = κx, where κ is obtained from Eqs (19) and (20). Upper panels show the result in a realization with a high fixed point, bottom panels show the result in a realization with a low fixed point. (I) Comparison between the predictions (solid lines) and simulations (shaded areas) from the population-averaged variances of Δx_i, shaded areas show mean±std and reflect finite-size effects. In C, F, G, I, the reciprocal correlations η_EE = −η_EI progressively increase from −0.43 to 1.0 while keeping η_II = −0.43 constant (ρ_EE = ρ_EI increase from 0 to 1 and ρ_II = 0 is fixed). In C, F, G, I, the departure of the centres of the numerical simulation results from the theoretical predictions reflect the systematic errors due to the approximation of the low-rank connectivity statistics as well as the latent dynamical variable κ. Network parameters: N_E = 4N_I = 800, c = 0.3, A_E = 0.023, A_E/A_I = 0.3. The transfer function ϕ has parameter θ = 1.5.

https://doi.org/10.1371/journal.pcbi.1010855.g009

We first examine the effect of the reciprocal motifs on the statistical properties of the eigenvalues and eigenvectors. The spectrum still consists of continuous eigenvalues and a discrete outlier. Since in the independent case the outlier depends only on A_E/I and sparsity c, here we fix these two variables but increase η_EE = −η_EI while keeping η_II constant. As in the case of dense, Gaussian networks, the outlier increases with the increasing reciprocal correlation and deviates from the outlier of the corresponding independent sparse connectivity matrix (Fig 9B and 9C). Moreover, in the large network limit, we find that if means, variances, and reciprocal correlations are identical, dense Gaussian connectivity leads to the same eigenvalue spectrum as the sparse connectivity (Methods Secs. 2.1.2, 2.1.3, Eqs (38) and (39)). We furthermore mathematically predict two additional conjugate eigenvalue outliers generated by the reciprocal connections in the sparse case (Eqs (82), (88) and (89), Fig 9B).

As for uncorrelated connectivity, perturbation theory accounts for the individual left and right eigenvector entries for specific instances of the sparse connectivity, which altogether follow Gaussian statistics as expected (Fig 9D and 9E, Eqs (69) and (70)). Importantly, reciprocal correlations induce a non-zero covariance σ_nm between the entries m_i and n_i of the right and left eigenvectors (Eq (99), Fig 9F).

Finally, we examine the population dynamics in the sparse network with reciprocal motifs using the low-rank approximation derived above. The reciprocal motifs in the example network generate an overall positive feedback. Therefore, gradually increasing the reciprocal correlation η_EE (−η_EI) in the example network induces a bifurcation into bistability (Eq (119), Fig 9G). Analogous to the projections depicting the individual eigenvector entries obtained using perturbation theory (Methods Sec. 2.3 Eq (69)), the low-rank approximation analytically accounts for the activity of individual neurons in specific connectivity realizations (Eq (16), Fig 9H), and hence the cell-type dependent variances of neuronal activation (Fig 9I) obtained from finite-size simulations of the original sparse networks.

2.1 Locally-defined multi-population connectivity

In this section, we introduce a first class of connectivity models, in which the synaptic couplings are generated based on local statistics determined by the identity of pre- and post-synaptic neurons. The N neurons in the network are organized in P populations, where population p has N_p neurons. Denoting by p and q the populations neurons i and j belong to, the value of the synaptic coupling J_ij is drawn randomly from a distribution in which statistics depend on the pre- and postsynaptic population q and p. The full connectivity matrix J therefore has a block structure, in the sense that all connections within the same block share identical statistics.

We examine two variants of this model class: (1) independent random connectivity [53]; (2) connectivity with reciprocal motifs [10, 82]. In each case, we examine two specific examples of distributions of synaptic strengths, Gaussian, and Sparse distributions.

2.2 Globally-defined connectivity: Gaussian-mixture low-rank networks

In this section, we introduce a second broad class of connectivity models, Gaussian-mixture low-rank networks [31, 33], in which the connectivity matrix is generated from a global statistics of vectors over neurons.

Low-rank networks are a class of recurrent neural networks in which the connectivity matrix J is restricted to be of rank R, assumed to be much smaller than the number of neurons N. Such a connectivity matrix can be expressed as a sum of R unit rank terms (40)

We refer to and as the r-th left and right connectivity vectors. The 2R connectivity vectors together fully specify the connectivity matrix. Each neuron i is then characterized by its set of 2R entries on these vectors. For unit-rank networks, the main focus of this study, the connectivity matrix is simply given by the outer product of a pair of connectivity vectors m and n: (41)

Gaussian-mixture low-rank networks are a subset of the class of low-rank networks, for which the entries of the connectivity vectors are drawn independently for each neuron from a mixture of Gaussians distribution [31]. Specifically, a fraction α_p of neurons is assigned to a population p, and within each population, the entries on the connectivity vectors are generated from a given 2R-dimensional Gaussian distribution. For a unit-rank network, for a neuron i in the population p, the connectivity parameters (m_i, n_i) are generated from a bi-variate Gaussian distribution with mean , variance and covariance .

For any unit-rank matrix of the form in Eq (41), the only potentially non-zero eigenvalue is given by , and the corresponding right (resp. left) eigenvector is m (resp. n). For a Gaussian-mixture model, in the large N limit this eigenvalue becomes (42)

Starting from a Gaussian-mixture low-rank model in which the connectivity is globally defined, and the assumptions that m_i, n_i are drawn independently between neurons from a multi-variate Gaussian distribution is satisfied, it is straightforward to compute the resulting local statistics of the connectivity, i. e. the mean , variance (Methods Sec. 2.1.2) and reciprocal correlation η_pq (Methods Sec. 2.1.1) as: (43) in the large network limit. The expression for the local statistics of network connectivity using rank-R connectivity is in S5 Text.

2.3 Approximating locally-defined connectivity with Gaussian-mixture low-rank models

In this section, we describe our general approach for approximating an arbitrary connectivity matrix J with a rank-R matrix J_R. We then show that for J corresponding to locally-defined multi-population connectivity (Methods Sec. 2.1), the resulting approximation J_R in general obeys Gaussian-mixture low-rank statistics as defined in Methods Sec. 2.2.

The connectivity matrices J that we studied have randomly generated entries. Such matrices can be diagonalized almost surely (with probability 1) on complex numbers, as the set of non-diagonalizable matrices is of measure 0. The matrices we consider are however not symmetric and therefore non-normal [41], so that the left and right eigenvectors are in general not identical. Instead, they form a bi-orthogonal set [84]. Specifically, to approximate a full rank matrix J with a rank-R matrix J_R, we use truncated eigen-decomposition, which preserves the dominant eigenvalues. We start from the full eigen-decomposition of J: (44) where λ_r is the r-th eigenvalue of J (ordered by decreasing absolute value), while R_r and L_r are the corresponding right- and left-eigenvectors that obey (45) (46) (47)

In the following, we constrain the right eigenvectors R_r to be of unit norm, while the normalization of the left eigenvector is determined by Eq (47).

We obtain a rank-R approximation J_R of J by keeping the first R terms in Eq (44): (48)

The R non-trivial eigenvalues and eigenvectors of J_R therefore correspond to the first R eigenvalues and eigenvectors of J. We then set (49) (50) to have the same normalization for J_R as in Eq (40).

To obtain a low-rank approximation for a connectivity matrix J generated from locally-defined statistics defined in Methods Sec. 2.1, we first determine its dominant eigenvalues and eigenvectors. Starting from Eq (24), this problem becomes equivalent to finding the dominant eigenvalues and eigenvectors of a low-rank matrix perturbed by a random matrix Z with block-like statistics. We compute the statistics of these eigenvalues and eigenvectors by combining the Matrix’s Determinant Lemma, the Matrix Perturbation Theory and the Central Limit Theorem. Below we summarize this general approach before applying it to different specific cases in Methods Secs. 2.4, 2.5.

We focus on the case where is unit rank as in the simplified excitatory-inhibitory network introduced in Methods Sec. 2.1.3. In that case, the unique non-zero eigenvalue of is (51) and the corresponding left and right eigenvectors are (52)

can then be rewritten as (53) where the structure vectors and are uniquely defined by rescaling the left and right eigenvectors of (Eq (52)) as in Eq (49), so that (54) (55) (56)

The full connectivity matrix J can be then expressed as (57)

Eigenvalues. For a random matrix Z with independently distributed elements, the eigenvalues are distributed on a disk of radius r_g centred at the origin in the complex plane [11, 44, 85]. Correlations between elements in general modify the shape of this continuous spectrum [18, 86]. In contrast, adding a low-rank component typically induces isolated eigenvalues outside the continuous part of the spectrum [44, 45]. To obtain a low-rank approximation of the full matrix, we focus on determining these outliers when they exist.

All eigenvalues λ of J satisfy the characteristic equation (58)

To determine the outlying eigenvalues of a random connectivity with low-rank structure, we apply the matrix determinant lemma to the l. h. s. of the characteristic equation [32]: (59)

As outliers by definition cannot be eigenvalues of Z, they correspond to zeros of the first term in the r. h. s., and therefore satisfy: (60) where I is the identity matrix. We assume that the maximal eigenvalue of Z is smaller than the outlying eigenvalues λ we aim to determine, the upper bound on this maximal eigenvalue is provided by the spectral norm of the Z matrix [87]. As a result, we can expand (I − Z/λ)⁻¹ in series, and further get [32] (61) with (62)

Here θ₀ corresponds to the eigenvalue λ₀ of (Eq (51)), and the higher order terms specify how this eigenvalue is modified by the random part of the connectivity. Truncating Eq (61) at a given order, and averaging over Z yields a polynomial equation for the mean eigenvalues of J. In Methods Sec. 2.4, we exploit this equation to determine the effects of different cell-type specific random connectivity Z on the outlying eigenvalues.

Note that within first-order perturbation theory, the eigenvalues are given by λ = λ₀ + Δλ with (63)

Eigenvectors. To determine the eigenvectors corresponding to the outlying eigenvalue of J, we treat it as perturbed by Z (Eq (24)). Matrix perturbation theory then states that, at first order, the right- and left-eigenvectors R and L of J corresponding to the outlying eigenvalue λ are given by [47]: (64) (65) where and are the right- and left-eigenvectors of defined in Eq (52), and (66)

Using the normalization in Eq (49), we then get (67) with constant entries on and defined in Eqs (54) and (56) and (68) where we approximated λ at first order by λ₀.

Statistics of Eigenvector entries. While and are deterministic vectors, the perturbations and are random variables obtained by multiplying and with the random matrix Z (Eq (68)). We therefore next consider the statistics of the elements m_i and n_i of m and n defined in Eq (67).

Since the elements of Z have zero mean, the mean values of m_i and n_i are given by and defined in Eqs (54) and (56). The mean value of n_i, but not m_i, therefore depends on whether the neuron i belongs to the excitatory or inhibitory population. Taking into account that Z has block-like statistics, we split the matrix product in Eq (68) into the sum of items corresponding to excitatory and inhibitory pre-synaptic neurons. Using Eqs (54) and (56), Δm_i and Δn_i can be written as (69)

We next take the limit N_E, N_I → ∞, and apply the central limit theorem, which states that each sum converges to a Gaussian random variable, so that we have (70) where p ∈ E, I is the population the neuron i belongs to, and are the variance of z_ij, z_ji respectively, for i, j in populations p, q. The perturbations Δm_i and Δn_i. therefore converge to Gaussian random variables of zero mean and variances and given by: (71)

In order to guarantee stability in the large network limit, we assume that the variance of the local random synaptic weights is satisfying . Consequently, we have and (Eqs (70) and (71)).

The population covariance between elements m_i and n_i with i belonging to population p can furthermore be written as (72) while the overall covariance σ_nm between all m_i and n_i reads (73)

Altogether, m_i and n_i determined by perturbation theory therefore follow Gaussian-mixture statistics, where the mean and variance depend on whether the neuron i belongs to the excitatory or inhibitory population.

Comparison with simulations. The theoretical predictions for eigenvalues obtained from Eqs (61) and (62) can be verified by comparing them with the eigenvalue outliers computed by direct eigen-decomposition of the full matrix J. We compute the average and standard deviation of eigenvalue outliers over 30 realizations of J.

The predictions of perturbation theory for eigenvectors given by Eq (66) can also be verified by direct eigen-decomposition, but to compare individual entries, an appropriate normalization is required [47]. Indeed, perturbation theory assumes that is normalized and satisfies (Eq (52)), but the perturbed eigenvectors in Eq (64) do not obey the same normalization. We therefore first use numerical eigen-decomposition to get the right- and left-eigenvector and of J. We then normalize to 1, and so that . To compare with perturbation theory, we then normalize as (74) the eigenvectors L, R after normalization have the same statistics as , (Eqs (62) and (64)).

2.4 Eigenvalues

Here, we apply Eqs (61) and (62) to determine the mean and variance of outlying eigenvalues for different forms of local connectivity statistics.

2.5 Eigenvectors

Here we apply Eqs (71) and (72) to determine the variances and covariances of eigenvector entries obtained from perturbation theory for different forms of local connectivity statistics.

2.6 Dynamics

In this section, we show how approximating locally-defined connectivity by a global low-rank structure allows us to analyse the emerging low-dimensional dynamics. We first summarize the mean-field theory (MFT) for Gaussian-mixture low-rank networks [31, 33]. We then apply it to unit-rank connectivity obtained as an approximation of locally-defined connectivity. We finally compare the resulting description of the dynamics with an alternate mean-field approach for random connectivity consisting of a superposition of low-rank and full-rank random parts as in Eq (24) [30, 32].

Throughout this study, we consider recurrent networks of rate units with recurrent interactions defined by a connectivity matrix J. The dynamical activity of unit i is represented by a variable x_i(t), which we interpret as the total synaptic input current. The firing rate of unit i is given by r_i(t) = ϕ(x_i(t)) where ϕ(x) = 1 + tanh (x − θ) is a positive transfer function. We focus on networks without external inputs, so that the dynamics of synaptic input to neuron i is given by (100)

In Figs 5–7, we compare the dynamics determined by direct simulations for a locally-defined connectivity matrix with a mean-field description obtained for a unit-rank approximation.