Asynchronous firing in heterogeneous networks

We performed Monte Carlo simulations of recurrent, randomly connected E/I networks, as described in Methods: Neuron model and network setup. To connect to previous literature on asynchronous spiking, we compared networks with and without single-cell variability—referred to as heterogeneous and homogeneous respectively. Heterogeneity was introduced by allowing cell threshold to vary, which induced a corresponding range of firing rates (see Methods: Neuron model and network setup for details). We first chose parameters so that the networks exhibited the classical asynchronous irregular (Asyn) regime, in which each neuron has irregular Poisson-like spiking, correlations are low, and the population power spectra are flat [42]. In Fig 1A we show raster plots from both the heterogeneous and homogeneous networks, in this regime. The heterogeneous network shows a gradient in its raster plot, because cells are ordered by decreasing firing rate. The population power spectra were flat, for both E and I cells and in both homogeneous and heterogeneous networks (Fig 1C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Two firing regimes in heterogeneous networks.

Monte Carlo simulations illustrating two firing regimes we consider in this paper. (A) Raster plots from the asynchronous (Asyn) regime. (B) Raster plots from the strong asynchronous (SA) regime, showing occasional bursts of activity. (C) Power spectra in the asynchronous regime. (D) Power spectra in the strong asynchronous regime. (E) Firing rates in the asynchronous (top panel) and SA (bottom panel) regimes. In (A-B), cells are ordered by increasing threshold value. Power spectra (C-D) are normalized to their maximum value and expressed in decibels/Hz.

https://doi.org/10.1371/journal.pcbi.1005506.g001

When we increased excitation (by increasing both W_EE and W_IE, where W_XY is the conductance strength from type Y to X; see Table 1 for parameter values), we observed occasional bursts of activity. However, the bursts do not occur at regular intervals and do not involve the entire population (we found excitatory bursts involved at most 25% of the population). The network is still moderately inhibition-dominated and neurons are spiking irregularly; example raster plots are shown in Fig 1B. The population power spectra (Fig 1D) are no longer flat (compare to the asynchronous regime, Fig 1C); they show local maxima around 8 Hz, but it is not a pronounced peak. We will refer to this as the strong asynchronous (SA) regime [43].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Excitatory connection strengths mediate between different firing regimes.

https://doi.org/10.1371/journal.pcbi.1005506.t001

In both Fig 1C and 1D, we note that—despite the apparent differences in the distribution of spikes across the network, evident in the raster plots—both the autocorrelation functions (Fig 1C and 1D, insets) and the power spectra from the heterogeneous and homogeneous networks are very similar. Thus, we have a fair comparison to examine the role of heterogeneity, independent of other characteristics of the network.

The distribution of both excitatory and inhibitory firing rates are extremely narrow in the homogeneous network, but broad in the heterogeneous network (Fig 1E). This is expected, as each excitatory (inhibitory) cell in the homogenous network has the same uncoupled firing rate; because the number of synaptic inputs is likewise fixed, population variability in synaptic input is limited. The heterogeneous networks have a range of firing rates, which allows us to investigate the possibility of a relationship between (variable) firing rate and pairwise correlations. Population-averaged firing rates were very similar between the heterogeneous and homogeneous networks: in the asynchronous regime ⟪ν_E⟫ = 10.6 Hz (heterogeneous) and ⟪ν_E⟫ = 10.1 Hz (homogeneous), while ⟪ν_I⟫ = 44.3 Hz (heterogeneous) and ⟪ν_I⟫ = 43.5 Hz (homogeneous). In both regimes Fano factors ranged between 0.9 and 1.1, consistent with Poisson-like spiking (more statistics are given in S1 and S3 Tables).

Correlation increases with firing rate in the strong asynchronous regime

We next sought a possible relationship between pairwise correlations—quantified via the Pearson’s correlation coefficient for spike counts, —and single-cell firing rates. Such relationships have been found in feed-forward networks [22, 30, 31], and impact information transfer when considered in concert with stimulus selectivity (i.e. signal correlations) [7, 8, 15, 19]. In heterogeneous networks, the large range of firing rates—equivalently the large range of operating points—admits the possibility that cells at different operating points may differ in their ability to transfer correlations.

To investigate this we plotted pairwise correlations for each distinct excitatory pair ρ_ij, versus the geometric mean of the firing rates , in both regimes (asynchronous and strong asynchronous), for a range of time scales (blue stars in Fig 2). We focus here on excitatory-excitatory (E-E) pairs, because excitatory synaptic connections provide the predominant means of propagating cortical sensory information to higher layers. Our results show a striking difference between the two spiking regimes; while there is no clear relationship with firing rate in the asynchronous regime (Fig 2, top row), the strong asynchronous regime shows a distinct positive trend with firing rate (Fig 2, bottom row). We can quantify a hypothesized relationship between ν and ρ with linear regression, and indeed find that geometric mean firing rate explains a substantial part of the variability of correlations in the strong asynchronous regime obtained from the Monte Carlo simulations, with R² values (i.e. percentage of variability explained) of 0.41, 0.37, and 0.34 for time windows of T = 5, 50, and 100 ms respectively (in contrast, R² values for the asynchronous network are below 0.005).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Correlation increases with firing rate in the strong asynchronous regime.

E-E correlation ρ_ij vs. geometric mean firing rate , cell-by-cell comparison of Monte Carlo simulations (blue stars) and linear response (magenta circles), in a heterogeneous network. Left to right: time window T = 5 ms and 100 ms. Top row: asynchronous regime. Bottom row: strong asynchrony.

https://doi.org/10.1371/journal.pcbi.1005506.g002

In recurrent networks, the response of each cell is shaped by both direct and indirect connections through the network. We used the linear response theory described in Methods: Linear Response Theory and Methods: Computing statistics from linear response theory to predict the full correlation matrix C_T at various time scales, including the limit of long time scales: . We found that this theory successfully captured E-E correlations, both the full distribution of values and coefficients of individual cell pairs (details, including figures, can be found in: Supporting Information: S1 Text).

We then plotted the predicted correlation, , vs. geometric mean firing rate (magenta circles in Fig 2). The predicted correlations captured the same positive relationship observed in Monte Carlo results, with R² values of 0.47, 0.4, and 0.36.

Decomposition of correlation by graph motifs shows strong role for second-order motifs

Why does a correlation/firing rate relationship emerge in one spiking regime, but not the other? In feed-forward networks, a positive correlation/firing rate relationship results from transferring common input through fluctuation-driven, asynchronously-firing cells [22, 30]. In contrast, the amount of shared input into two cells in a recurrent network is determined by both direct and indirect connections through the network. To separate the impact of different network pathways, we decomposed the linear response-predicted correlations at long time scales (i.e. ) into normalized contributions from n-th order motifs, as described in Methods: Quantifying the role of motifs in networks. Common input from a divergent connection, for example, results from the 2nd-order motif K*C⁰K. In Fig 3, we plot the summed contributions up to sixth order—i.e. , for k = 1, 2, …6—versus geometric mean firing rate, . The total normalized correlation, , is shown as well. In all cases, we plot long time scale correlations ω = 0; each distinct E-E pair is represented.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Pairwise correlations are built from graph motifs.

Contributions of different orders to prediction of E-E correlations with linear response theory. (A) Normalized contributions to pairwise correlation () vs. geometric mean firing rate () for heterogeneous networks in the asynchronous (top panel) and strong asynchronous (bottom panel) regimes. (B) As in (A), but plotted vs. total predicted normalized correlation (). See main text for further discussion. (C) To quantify the relative importance of different motifs, we report the fraction of variance explained (R²) from linear regressions, in which we regressed total correlation () against the contributions at each specific order (). As suggested by (B), second-order contributions (red) overwhelmingly determine total correlations in the asynchronous network (R² values for first- and third-order terms are shown, but barely visually distinguishable; R² values for higher orders are also small, within 0.08 up to k = 6). (D) Fraction of total correlation from each order, strong asynchronous regime.

https://doi.org/10.1371/journal.pcbi.1005506.g003

In the asynchronous regime (top panel of Fig 3A), first-order contributions () separate into three distinct “curves”, reflecting a 1-1 relationship with firing rate conditioned on first-order connectivity (no connection between i and j; one connection between i and j; bidirectional connection between i and j). Second-order contributions are overall positive while third-order contributions are overall negative (consistent with [28]); neither appear to have a relationship with firing rate. Second-order contributions are conspicuously dominant; fifth and sixth order terms are near zero.

This qualitative picture changes when we consider the strong asynchronous regime (bottom panel of Fig 3A). First-order contributions follow a similar pattern as in the asynchronous regime, and second-order contributions are likewise positive. However, third-order contributions are positive, and in the heterogeneous network they have a distinctly positive relationship with firing rate (top panel). Thus, in the asynchronous regime, negative third-order contributions partially cancel with positive second-order contributions; in the strong asynchronous regime, first, second, and third-order motifs reinforce each other, contributing to an overall positive relationship with firing rate (black dots).

Despite these differences, second-order contributions are the major determinant of total correlation in both regimes. In Fig 3B we plot the same data () vs. total correlation, rather than geometric mean firing rate. In the asynchronous regime, second-order contributions cluster near the unity line, suggesting they are strongly predictive of total correlation. To quantify this intuition we computed the fraction of variance explained (R²) by performing a linear regression of total normalized correlation () against contributions of each order (Fig 3C); in the asynchronous regime R² values for , , and were 0.004, 0.969, and 0.0002, respectively. R² values for higher orders were likewise small: for , , and they were 0.047, 0.034, and 0.074.

This statistic was more ambiguous in the strong asynchronous regime, where R² values for , , and were 0.595, 0.474, and 0.509 respectively. However, note that and were positive for all cell pairs; and total correlation were negative for less than 0.3% of cell pairs. Thus, we considered how each motif contributed to the total correlation by taking the ratio of each contribution to the total, averaged over all cell pairs (Fig 3D). By this measure, second-order contributions were largest; fraction explained for , , and were 0.239, 0.601, and 0.420, respectively. Note that this measure cannot be used for the asynchronous (Asyn) regime because of the negative values of .

Taken together, this evidence points to a distinguished role for second-order motifs () in determining total correlation. In the asynchronous regime in particular, is a near-perfect predictor of total correlation.

Inhibitory common input is the most important second-order motif

We next analyzed contributions from specific second-order motifs in Fig 4. There are four distinct second-order motifs that can correlate two E cells. There are two types of chains, from K²C⁰ and C⁰ (K*)². An E → E → E chain tends to positively correlate, while an E → I → E chain will negatively correlate; these are shown as blue and green respectively. There are two types of common input, from KC⁰(K*); they correspond to common input from E and I cells, i.e. E ← E → E and E ← I → E. They both lead to positive correlations and are shown as red and magenta respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Inhibitory common input is the dominant second-order motif in both asynchronous and strong asynchronous networks.

(A) Contributions of different 2nd-order motifs to prediction of E-E correlations in a heterogeneous network, in the asynchronous (top) and strong asynchronous (bottom) regimes. (B) As in (A), but plotted vs. total contribution from second-order motifs . In both panels, inhibitory common input (magenta) clusters near the unity line. (C) To quantify the relative importance of different motifs, we report the fraction of variance explained (R²) from linear regressions, in which we regressed the total contribution from second-order motifs () against the contribution from specific motifs types.

https://doi.org/10.1371/journal.pcbi.1005506.g004

In the asynchronous regime (left panel of Fig 4A), the dominant contributions are I common input (magenta) and negative (E → I → E) chains (green); correlating chains (blue) and excitatory common input (red) are barely visible, as they are clustered near zero. In the strong asynchronous case (right panel), blue and red dots are now visible and show a clear 1-1 trend with firing rate. In both regimes, inhibitory common input appears to be the dominant second-order motif. In Fig 4B we plot the contribution from different second-order motifs vs. the total contribution from second-order motifs, (rather than geometric mean firing rate, ). In both panels, the inhibitory common input (magenta) clusters around the unity line, showing it is the best predictor of the total second-order contribution. In Fig 4C we quantify this observation by reporting fraction of variance explained (R²) from linear regressions: the R² value for inhibitory common input exceeds 0.8 in both networks, while the R² values for all other motifs types are less than 0.1.

In conclusion, decomposition of pairwise correlations into graph motifs has shown us two important things: first, while third-order motifs probably contribute to the positive correlation/firing rate relationship observed in the SA regime, second-order motifs still dominate in both regimes. Second, inhibitory common input is the most important second-order motif in both regimes (Fig 4B).

Susceptibility to inhibition can either increase or decrease with firing rate

In feedforward networks—i.e., in the absence of a path between two cells—correlations in outputs (i.e. spike trains) must arise from correlations in inputs; for example, through shared or common inputs. We have found that inhibitory common input is the dominant contributor to pairwise correlations in both the asynchronous and strong asynchronous regimes; we now turn our attention to modeling this term (inhibitory common input) specifically.

Previous work that analyzed the relationship between the long-time correlation and firing rate in feedforward networks [22, 30] quantified a susceptibility function that measures the ratio between output and input correlations: (1) If both cells receive a large (but equal) number of uncorrelated inputs, c would be the fraction of inputs that are common to both i and j.

In the networks examined here, each cell had a fixed in-degree for both excitatory and inhibitory cells; however, for any given pair of cells i and j, the number of E and I inputs that synapsed onto both cells will vary from pair to pair. Thus, we next considered the possibility that our (negative) finding in the asynchronous network could be explained by accounting for variable c_ij.

We focus on inhibitory common input, which is the dominant second-order contribution in the asynchronous network (Fig 4). We segregated pairs by whether they had 0, 1, 2, etc.. common inhibitory inputs; we then use this number as a proxy for c (recall that each excitatory cell had exactly 7 inhibitory inputs, so that this number divided by 7 approximates the common input fraction; two common inputs imply c ≈ 0.28 for example). We plot the results for the asynchronous network in Fig 5A, top panel (data for each distinct value of c is presented by color). As we might expect, correlation increases as c increases. However, for a fixed c, there is not an apparent relationship between firing rate and correlation; if anything, there appears to be a slight decrease. Correlation also increases with c in the strong asynchronous network (Fig 5A, bottom panel); however, here we also see a modest increase with geometric mean firing rate .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Susceptibility to conductance fluctuations can explain correlation-firing rate relationships.

In (A-C): heterogeneous asynchronous (top) and heterogeneous strong asynchronous (bottom). (A) Correlation (ρ) from I common inputs vs. firing rate, segregated based on the number of common inhibitory inputs. (B) Estimated correlation susceptibility to fluctuations in inhibitory conductances vs. firing rate (). (C) Correlation susceptibility to fluctuations in inhibitory currents vs. firing rate ().

https://doi.org/10.1371/journal.pcbi.1005506.g005

Previous theoretical work [22, 30] identified an increase in susceptibility with firing rates in current-driven neurons; we next considered the possibility that this fails to hold for conductance-driven neurons. As described in Methods: Quantifying correlation susceptibility, we estimated correlation susceptibility for each pair of neurons, by using the susceptibility function for each neuron to conductance fluctuations (computed as part of the linear response theory), divided by a measure of the long-timescale spike count variance: (2) We plotted the results for both networks in Fig 5B; while susceptibility increases with firing rate in the strong asynchronous network (except for the largest firing rates), it actually decreases with firing rate in the asynchronous network.

We can contrast with the estimated susceptibility to current fluctuations (i.e. A_μ,i, with μ_i, τ_eff,i, and σ_eff,i as in Eq 29) which we also computed for the same set of cell pairs, shown in Fig 5C. (3) Here, we see that increases with firing rates, in both networks.

Understanding the susceptibility function

We next sought to understand how susceptibility depends on neural parameters; that is, we define the single-cell susceptibility (4) where (5) (6) (“” indicates that derivative is taken with respect to the first argument, 〈g_{I, i}〉). We have also used the asynchronous spiking assumption, that (compare with Eq 3).

This quantity is shown in Fig 6, where it is plotted vs. firing rate ν_i (blue stars). Note that this is a negative quantity; since the susceptibility for a neuron pair is the product (and therefore positive), an increase in will result in a decrease in and vice versa.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. How firing rate diversity is achieved in a heterogeneous network will affect susceptibility.

Single-cell susceptibility function(s) for a conductance-based LIF neuron, as a function of firing rate ν. Successive approximations shown are: original single-cell susceptibility, (Eq 4, blue stars); most parameters set to average value, (Eq 7, red triangles); all parameters but θ_i set to average value, (Eq 9, gold squares); and θ fixed, (Eq 10, purple diamonds). (A) Asynchronous regime. (B) Strong asynchronous regime.

https://doi.org/10.1371/journal.pcbi.1005506.g006

In principle, the firing rate function (Eq 5)—and therefore susceptibility—can depend on all six parameters defining the cell: our next step was to reduce the dimensionality of the problem. We first looked for any possible relationship between single-cell firing rates and cell parameters (see S1 Text: Approximating single-cell susceptibility in a heterogeneous network, S6 and S7 Figs): in both networks, only threshold θ_i had an obvious relationship with firing rate. Among the remaining parameters, the mean inhibitory conductance 〈g_I〉 had the greatest relative range of values in the asynchronous network (S6A Fig). Therefore, we hypothesized that we could accurately capture , by approximating it as a function of the two parameters θ_i and 〈g_I〉.

We reevaluated the firing rate function, where σ_I,i, 〈g_E,i〉, σ_E,i and σ_i have been replaced by their average values: i.e. (7) where (8) and 〈 ⋅ 〉_p denotes the population average. The results are also illustrated in Fig 6 (red triangles). In the asynchronous regime (Fig 6A), the results are remarkably close to the original quantities, indicating that using average parameter values has little effect; in the strong asynchronous regime (Fig 6B) the difference is larger, but the points appear to occupy the same “cloud”. However, we can now visualize the susceptibility as a function of only two parameters, and we do so in Fig 7 by evaluating on a (θ, 〈g_I〉) grid; the points corresponding to the actual excitatory cells in our network are illustrated in red. In both the asynchronous and strong asynchronous regimes, the red stars form a scattered cloud around the average value 〈〈g_I,i〉〉_p, with no obvious relationship with θ_i.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Susceptibility as a function of inhibitory conductance and threshold.

Single-cell susceptibility function for a conductance-based LIF neuron, as a function of mean inhibitory conductance 〈g_I〉 and threshold θ: (defined in Eq 7). Other parameters are set to the population average. Overlays show (〈g_I,i〉, θ_i) values of the actual cells in the network (red stars) and an alternative curve through the plane, (〈g_I〉, 1), along which comparable firing rate diversity can be observed (black squares). (A) Asynchronous regime. (B) Strong asynchronous regime.

https://doi.org/10.1371/journal.pcbi.1005506.g007

This fact motivated a further simplification, (9) i.e., we replaced 〈g_I,i〉 with its population average, 〈〈g_I,i〉〉_p, in essence approximating a one-dimensional “path” that the cells take through parameter space. The results are shown in Fig 6 (gold squares) and, as we should expect, allow us to discern a functional relationship with firing rate ν_i; importantly, it appears to capture the average behavior of the actual susceptibility values . Here, we can see clearly that in the asynchronous regime, correlations should actually decrease with firing rate, for ν_i > 5 Hz. In the strong asynchronous regime, correlations will increase with firing rate, saturating around 10–15 Hz.

Susceptibility depends on the mechanism underlying firing rate diversity

Finally, recall that our actual network sampled a relatively small part of the (θ, 〈g_I〉) plane. This may be attributed to the fact that we generated firing rate diversity (and therefore heterogeneity), by modulating cell excitability through the cell threshold θ_i. How might our results have changed, if we had generated firing rate diversity through some other mechanism? In both regimes, we can increase firing rates by either decreasing 〈g_I,i〉, or by decreasing θ (see S7 Fig). To explore this, we computed susceptibility values along another curve in the (θ, 〈g_I〉) plane; specifically, we held θ fixed and instead varied 〈g_I〉 (illustrated with black squares on Fig 7); i.e. (10) where Results are shown in Fig 6 (purple diamonds) and show a strikingly different relationship with firing rate; in the asynchronous regime, correlations should increase with firing rate for ν < 15 Hz; in the strong asynchronous regime correlations will increase with firing rate, saturating near 20 Hz.

To summarize the previous two subsections, we first defined a single-cell susceptibility function (Eq 4), which captures a linear approximation to the cell’s response to input. This quantity relies on an underlying firing rate, which is a function of all parameters that define single-cell dynamics; in this case, six. Each cell occupies a point in this six-dimensional parameter space. We found that in each network studied here, the occupied points approximately lie along a one-dimensional path through this parameter space, along which we could visualize the susceptibility. Finally, we considered the consequences of taking other paths through this parameter space: these paths can be interpreted as generating firing rate heterogeneity using other network mechanisms.

Low-rank structure in neural correlations is mediated by correlation-firing rate relationship

Previous work has identified low-dimensional structure in neural correlation matrices [25, 38–41]; its origin is not always known [3]. We next hypothesized that the positive correlation-firing rate relationship we observed in the strong asynchronous regime, might be reflected in low-dimensional structure in the correlation matrix. For simplicity, suppose that correlations were really represented by a function of firing rate (as in [22]): i.e. ρ_ij = cS(ν_i)S(ν_j). Then we could represent the off-diagonal part of the correlation matrix as C_T = cSS^T, where S is a length N vector such that S_i = S(ν_i); that is, C_T would be a rank-one matrix.

We followed the procedure outlined in Methods: Low-rank approximation to the correlation matrix to approximate each correlation matrix, C_T, as the sum of a diagonal matrix and low-rank matrix: (11) where λ is given in closed form by the eigenvalues of C_T: (12) and σ₁, u₁ are the first singular value and singular vector of C_T.

In Fig 8, we show the results from heterogeneous networks in both the asynchronous (top panel in each subfigure) and strong asynchronous (bottom panel in each subfigure) regimes. We first show C_T − λI, where λ is given by Eq 12, in Fig 8A. Cells are ordered by (decreasing) firing rate. While no pattern is visible in the asynchronous state (top panel), the strong asynchronous state (bottom panel) shows larger values in the upper left corner, suggesting that correlation increases with firing rate. This is even more visible in the rank one approximation, , shown in Fig 8B.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Low-rank structure in correlation matrices.

Approximating correlation matrices for the heterogeneous networks as a diagonal plus rank-one. Neurons are ordered by firing rate (highest to lowest). In each column of (A-D), the asynchronous (top) and strong asynchronous (bottom) regimes are shown; T = 100 ms.(A) The shifted E-E correlation matrix, C_T − λI, for an appropriately chosen λ. (B) A rank-one approximation to C_T − λI. (C) True correlation coefficients vs. rank-one approximation, cell-by-cell. (D) Weight in the first singular vector, u₁ vs. geometric mean firing rate .

https://doi.org/10.1371/journal.pcbi.1005506.g008

We now use to approximate C_T, and compare the results, cell pair-by-cell pair (Fig 8C). In the asynchronous network, the approximated correlations take on a narrow range (between 0 and 0.01, compared to between −0.015 and 0.03 for the measured coefficients) and do not show an obvious positive relationship. In the strong asynchronous regime, the range is more accurate (between 0.02 and 0.1, vs. 0.01 and 0.15 for the measured coefficients) and the points cluster around the unity line.

In Fig 8D, we plot the weight of each cell in the first singular vector, (u₁)_j vs. the firing rate ν_j. We can clearly see a positive relationship in the strong asynchronous regime (bottom panel), suggesting that the positive relationship between correlation and firing rate is related to the success of the low-rank approximation.

Impact on stimulus coding

The networks studied here were not encoding a stimulus; correlations were generated by recurrent activity, given that each neuron had a baseline firing rate in the absence of recurrent input. However, we can readily connect this network to a stimulus coding task, in order to understand how the correlation-firing rate relationship can impact coding.

Consider a population of cells that is responsible for encoding a single scalar stimulus θ, such as movement direction or orientation of a visual stimulus, and that each cell has roughly a bell-shaped tuning curve. Furthermore, we model an incoming stimulus by modulating a stimulus-dependent background current I_i,(θ); i.e., cells which prefer the current stimulus have a higher level of current, and thus a higher firing rate, than cells which prefer an orthogonal or opposite stimulus. The network we studied here would model the response to a single stimulus θ₀; that is, the firing rate diversity we observe is present because some cells are strongly tuned to the current stimulus, while others are not.

We could extend this model, by resetting background currents to model a complete set of stimuli {θ₁, θ₂, …θ_{n − 1}}. For each stimulus θ_j, correlations would show the rough firing rate dependence displayed in the strong asynchronous network, resulting in a stimulus-dependent correlation structure in which pairwise correlations vary like geometric mean firing rate. This is the structure analyzed in [21, 27]: the authors found that such a stimulus-dependent correlation code enhances information, when compared to a stimulus-independent code with the same average correlation level. Intuitively, the mean population response lives on the surface of a (hyper-)sphere in neural response space; the population encodes location on this surface. Positive correlations between similarly tuned cells produce response distributions that are stretched in the radial direction, “orthogonal” to this sphere, and thus have a minimal impact on the encoded variable.

Moreover, the mechanism that produced stimulus-dependent correlations in [21, 27] was similar to that shown here (see also [25]); common input modulated by stimulus-dependent gain factors. Here, we demonstrated how these stimulus-dependent gain factors might arise (or not) in a recurrent network. If excitation is tuned to put the network in the strong asynchronous regime, then the (stimulus-dependent) correlation structure that results will be favorable to coding. If excitation is tuned to put the network in the asynchronous regime, then correlations are overall low and not stimulus-dependent (although, given that average correlations are not matched, we do not here compare information contained within the two networks).

Future work

This work has, necessarily, focused only on a subset of network attributes that might affect firing statistics. One important feature is the frequency of higher-order graph motifs; experiments have shown that specific motifs will occur more frequently, than would be expected in an Erdős-Rényi network with fixed single-cell connection probability [49]. Theoretical work has found that in networks of integrate-and-fire neurons, an overabundance of divergent and chain motifs will lead to enhanced correlation [33] (this finding does depend on the dynamical regime; different motifs impact correlations in networks of coupled oscillators [32]). In [33], the authors use the assumption of homogeneous single-cell characteristics to find parsimonious and instructive formulae for the average correlation, and give a roadmap for how this might be generalized to heterogeneous networks. We look forward to considering the combined effect of single-cell and network heterogeneity in future work.

Another source of cell-to-cell heterogeneity is how cells respond to stimuli, as emphasized in the previous discussion [17, 20, 21, 27, 50] (see [19] for a review). Here, we did not consider a specific sensory system with tuning but rather focus on the general question of how the distribution of correlation values arise in recurrent networks. Given the previous discussion, one next step will be to investigate how correlations covary with firing rates, when cell-to-cell heterogeneity is produced by stimulus tuning in a structured network responding to a single variable (such as direction or orientation).

Finally, for numerical tractability our simulations here were performed in relatively small networks. While high average correlations have been measured in experiments [51], theoretical models of asynchronous networks have found that correlations must go to zero as the system becomes large (N → ∞) [2]. However, recent work has found that this does not have to be true, as long as spatial structure is introduced into the network [52]. We anticipate that this may carry over to other forms of heterogeneity, such as single-cell variability, and that therefore the effect we observe here persists for larger networks. We look forward to reporting on this in future work.

Neuron model and network setup

We considered randomly connected networks of excitatory and inhibitory neurons. Each cell was a linear integrate-and-fire model with second-order alpha-conductances, i.e. membrane voltage ν_i was modeled with a stochastic differential equation, as long as it remained beneath a threshold θ_i: (13) When ν_i reaches θ_i, it is reset to 0 following a refractory period: (14) Each neuron was driven by a Gaussian, white background noise, with magnitude σ_i depending only on the cell type; that is, 〈ξ_i(t)〉 = 0 and 〈ξ_i(t)ξ_i(t + s)〉 = δ(s). The membrane time constant, τ_m, and excitatory and inhibitory synaptic reversal potentials, and , are the same for every cell in the network.

Each cell responded to synaptic input through conductance terms, g_E,i and g_I,i, which are each governed by a pair of differential equations: (15) (16) where Y = {E, I} denotes the type of cell i and X = {E, I} denotes the type of the source neuron j. Each spike is modeled as a delta-function that impacts the auxiliary variable ; here t_j,k is the k-th spike of cell j. The rise and decay time constants τ_r,X and τ_d,X and pulse amplitude α_X depend only on the type of the source neuron; i.e. they are otherwise the same across the population. The parameter W_YX denotes the strength of X → Y synaptic connections, which are (once given the type of source and target neurons) identical across the population. The “raw” synaptic weight (listed in Table 1) is divided by N_YX, the total number of X → Y connections received by each Y-type cell.

We chose connections to be homogeneous and relatively dense, consistent with the local architecture of cortex. Connection probabilities ranged from 20%–40%, consistent with experimentally measured values [53–55]. For our baseline network state, we then chose synaptic weights so the network is moderately inhibition-dominated (α_EW_IE < α_IW_II and α_EW_EE < α_IW_EI); that is both E and I cells receive more inhibition than excitation) and shows noisy spiking consistent with the classical asynchronous state. Each neuron receives a fixed number of incoming connections, the identities of which are chosen randomly. (The specific cell ID numbers differ in the different simulations shown below.) For most of the networks we discuss here N = 100 with the 80/20 ratio typical of cortex (i.e n_E = 80, n_I = 20). Each excitatory cell receives N_EE = 32 (40%) excitatory and N_EI = 7 (35%) inhibitory connections; each inhibitory cell receives N_IE = 16 (20%) and N_II = 8 (40%) inhibitory projections.

In heterogeneous networks, the threshold θ_i varied across the population. For both excitatory and inhibitory neurons, the thresholds θ_i were chosen from a log-normal distribution between 0.7 and 1.4 (where the rest potential, V_r = 0). To be precise, log θ_i was chosen from a (truncated) normal distribution with mean and standard deviation s_θ. With this choice, θ_i has mean 1 and variance: . Thus we can view s_θ as a measure of the level of threshold heterogeneity.

Throughout this paper, we set s_θ = 0.2, which results in a wide range of firing rates compared to the homogeneous case. This was the only source of cell-to-cell heterogeneity; all other parameters were identical across the population, conditioned on neuron type (values listed in Table 2). In homogeneous networks, the threshold was the same across the population: θ_i = 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Other parameters used in network simulations.

https://doi.org/10.1371/journal.pcbi.1005506.t002

Monte Carlo simulations were performed using the stochastic forward- Euler method (Euler-Maruyama), with a time step much smaller than any time scale in the system (Δt = 0.01 ms). Each network was simulated for one second of simulation time, after an equilibration time. Then, a large number of realizations of this interval (n_R = 10⁵) were simulated. Spike counts were retained in each 1 ms window (for a total of 1000 windows) within a realization. With this large number of realizations/trials, the error bars on the resulting time-dependent firing rates were small. Therefore we emphasize that the firing rate pattern is largely driven by network connectivity; while firing is driven by random fluctuations in the background noise, any cell-to-cell variability in the trial-averaged firing rates are not an artifact of the finite number of trials.

Linear response theory

In general, computing the response of even a single neuron to an input requires solving a complicated, nonlinear stochastic process. However, it often happens that the presence of background noise linearizes the response of the neuron, so that we can describe this response as a perturbation from a background state. This response is furthermore linear in the perturbing input and thus referred to as linear response theory [56]. The approach can be generalized to yield the dominant terms in the coupled network response, as well; we will use the theory to predict the covariance matrix of activity.

We first consider the case of a single cell: an LIF neuron responding to a mean zero current ϵX_i(t) (otherwise, the mean of X_i can simply be absorbed into E_i).

For a fixed input ϵX_i(t), the output spike train y_i(t) will be slightly different for each realization of the noise ξ_i(t) and initial condition ν_i(0). Therefore we try to work with the time-dependent firing rate, ν_i(t) ≡ 〈y_i(t)〉, which is obtained by averaging over all realizations and initial conditions. Linear response theory proposes the ansatz that the firing rate can be described as a perturbation from a baseline rate proportional to the input ϵX_i: (17) ν_i,0 is the baseline rate (when X = 0) and A_i(t) is a susceptibility function that characterizes this firing rate response up to order ϵ [22, 29, 57].

We now consider the theory for networks; here cell i responds to the spike train of cell j, y_j(t), via the synaptic weight matrix W, after convolution with a synaptic filter F_j(t): In order to consider joint statistics, we need the trial-by-trial response of the cell. We first propose to approximate the response of each neuron as: (18) that is, each input X_i has been replaced by the synaptic input, and J_ij = W_ijF_j(t) includes both the i ← j synaptic weight W_ij and synaptic kernel F_j (normalized to have area 1); A_i(t) is the susceptibility function from Eq 17. In the frequency domain this becomes (19) where is the Fourier transform of the mean-shifted process (ν_i is the average firing rate of cell i) and for all other quantities. In matrix form, this yields a self-consistent equation for in terms of : (20) where is the interaction matrix, in the frequency domain. The cross-spectrum is then computed (21) To implement this calculation, we first solve for a self-consistent set of firing rates: that is, ν_i is the average firing rate of (22) where E[f_i] = ∑_j W_ijν_j.

We must then compute the power spectrum and the susceptibility A_i(ω), which is the (first order in ϵ) response in the firing rate in response to an input current perturbation X(t) = ϵ exp(ıωt) (here ı is used for , while i denotes an index). Both can be expressed as the solution to (different) first-order boundary value problems and solved via Richardson’s threshold integration method [47, 58].

In our simulations, we used conductance-based neurons; this requires modification, compared with the simpler current-based models. We first approximate each conductance-based neuron as an effective current-based neuron with reduced time constant, following the discussion in [59]. First, separate each conductance into mean and fluctuating parts; e.g. g_E,i → 〈g_E,i〉 + (g_E,i − 〈g_E,i〉). Then we identify an effective conductance g_0,i and potential μ_i, and treat the fluctuating part of the conductances as noise, i.e. g_E,i − 〈g_E,i〉→σ_E,i ξ_E,i(t): (23) where (24) (25) (26) (27) We next simplify the noise terms by writing (28) and assume that the fluctuating part of the voltage, ν_i − μ_i, is mean-zero and uncorrelated with the noise terms ξ_E,i(t) [59]. That allows us to define an effective equation (29) where (30) (31) and the fluctuating voltage, ν_i(t) − μ_i, now makes no contribution to the effective noise variance.

Finally, we consider how to model the conductance mean and variance, e.g. 〈g_E,i〉 and . In our simulations, we used second order α-functions: each conductance g_X,i is modeled by two equations that take the form (32) (33) where X = E, I and the summation is over all type-X spikes incoming to cell i. (For notation purposes, includes all factors that contribute to the pulse size in Eq 16, including synapse strength and pulse amplitude.) The time constants τ_r,X, τ_d,X may depend on synapse type; the spike jumps may depend on synapse type and target cell identity. We assume that each spike train is Poisson, with a constant firing rate: i.e. each spike train is modeled as a stochastic process S(t) with Then a straightforward but lengthy calculation shows that (34) (35) where ν_X,i is the total rate of type-X spikes incoming to cell i.

We now describe how these considerations modify the linear response calculation. First, for the self-consistent firing rate calculation, Eq 22 is replaced by an equation with a modified time constant, conductance, and noise (Eq 29).

We next compute the susceptibility in response to parameters associated with the conductance, i.e. 〈g_E,i〉 and . This differs from the current-based case in two ways: first, there is voltage-dependence in the diffusion terms, which results in a different Fokker-Planck equation (and thus a different boundary value problem to be solved for the power spectrum ). Second, modulating the rate of an incoming spike train will impact both the mean and variance of the input to the effective equation, Eq 23 (via μ_i and σ_X,i). Furthermore, this impact may differ for excitatory and inhibitory neurons, giving us a total of four parameters that can be varied in the effective equation. However, neither consideration presents any essential difficulty [47].

Therefore we apply Richardson’s threshold integration method directly to Eq 23: (36) When we compute susceptibilities, the parameter to be varied is either a mean conductance—〈g_E,i〉 → 〈g_E,i〉₀ + 〈g_E,i〉₁ exp(ıωt) or 〈g_I,i〉 → 〈g_I,i〉₀ + 〈g_I,i〉₁ exp(ıωt)—or a variance— or . Thus we have a total of four susceptibility functions , , , and . Since the Fokker-Planck equation to be solved is linear, we can compute both susceptibilities separately and then add their effects. We now have the interaction matrix: (37) where plays a similar role as , but for the effect of incoming spikes on the variance of conductance. Its relationship to (either in the frequency or time domain) is given by the same simple scaling shown in Eq 35: i.e., for j excitatory, (38) where the first factor comes from the effective spike amplitude (and is the scale factor proposed in [47], Eq (64)), and the second arises from using second-order (vs. first-order) alpha-functions.

We use a modified version of the implementation given by [29] for Richardson’s threshold integration algorithm [47, 58] to compute rate ν_i, power , and the various susceptibilities (, , , and ) for an LIF neuron. We validated our code using exact formulas known for the LIF [60], and qualitative results from the literature [61].

Computing statistics from linear response theory

Linear response theory yields the cross spectrum of the spike train, , for each distinct pair of neurons i and j (see Eq 21). To recover a representative set of statistics, we rely on several standard formulae relating this function to other statistical quantities.

The cross correlation function, C_ij(τ), measures the similarity between two processes at time lag τ, while the cross spectrum measures the similarity between two processes at frequency ω: (39) (40) The Weiner-Khinchin theorem [56] implies that are a Fourier transform pair: that is, (41)

In principle, the crosscorrelation C(t) and cross-spectrum matrices are functions on the real line, reflecting the fact that correlation can be measured at different time scales. In particular, for a stationary point process the covariance of spike counts over a window of length T, n_i and n_j, can be related to the crosscorrelation function C_ij by the following formula [4]: (42) The variance of spike counts over a time window of length T, n_i, is likewise given by integrating the autocorrelation function C_ii: (43)

It can be helpful to normalize by the time window, i.e. (44) we can now see that for an integrable cross correlation function (and bearing in mind that the cross-spectrum is the Fourier transform of the cross correlation), that (45) while (46) Thus, we can use and C_ij(0) as measures of long and short time correlations respectively.

Finally, the Pearson’s correlation coefficient of the spike count defined as: (47) is a common normalized measure of noise correlation, with ρ ∈ [−1, 1]. While Cov_T and Var_T grow linearly with T (for a Poisson process, for example), ρ_T,ij in general will not (although it may increase with T). In general, ρ_T,ij depends on the time window T; however for readability we will often suppress the T-dependence in the notation (and use ρ_ij instead).

Quantifying the role of motifs in networks

We next explain how we can use the results of linear response theory to give insight into the role of different paths in the network. We begin with our predicted cross-spectrum (Eqs 21 and 40) and apply a standard series expansion for the matrix inverse: (48) (49) (50) where is a diagonal matrix containing the power spectra of the unperturbed processes; i.e. . This double sum will converge as long as the spectral radius of is less than 1 [29].

By truncating this double sum to contain terms such that k + l ≤ n, we define the nth approximation to the cross-spectrum: (51) (52) Each distinct term in the inner sum can be attributed to a particular undirected path of length k. Terms of the form and account for unidirectional paths from j → i and i → j respectively; the term captures the contribution from a cell that has a length l path onto cell j and a length k − l path onto cell i. Thus, we can use Eq 52 to decompose the correlation into contributions from different motifs ([28], see also [31, 62]).

We can also consider the contribution from all length-n paths; that is, If the sum in Eq 50 converges, we should expect the magnitude of contributions to decrease as n increases.

We will also show the normalized contribution from length-n paths, which we define as follows: let Λ(ω) be the diagonal matrix with . Then we define the matrix of contributions from length-n paths as follows: (53) Equivalently, . This effectively normalizes the cross correlation by the autocorrelation; in particular, we can use this to decompose the correlation coefficient (Eq 47) for long time windows, because .

In general, we will show long-timescale correlation (e.g. or ) (Eq 45); results were qualitatively similar for other timescales.

Quantifying correlation susceptibility

We next consider how to quantify the (linear) susceptibility of correlation to a change in parameter. Returning to Eq 17, but written in terms of the single-cell response: (54) (55) Here, X_μ(t) is a (possibly) time-dependent change in a parameter, such as input current or mean inhibitory conductance; y_i,0 is the baseline spike train (when X = 0). A_μ,i(t) is a susceptibility function that characterizes the cell’s response (to the parameter variation) as long as X_μ(t) is small [22, 29, 57]. Following [22], the cross-spectrum of y can now be approximated as: (56) (57) where is the spectrum of the parameter variation. The susceptibility has an appealing interpretation in the limit ω → 0, as the derivative of the classical f-I curve: (58) where ν_i is the steady-state firing rate of cell i, assuming we can measure it for specific values of the parameter μ. (59) (60) This motivates the definition of a correlation susceptibility, which approximates the change in pairwise correlation induced by a parameter change experienced by both cells i and j: (61) If this increases with firing rate—that is, if —then correlations will also increase with firing rate.

We can further analyze this quantity by making an assumption for asynchronous spiking, that spike count variance is equal to spike count mean; i.e. . Then (62) which motivates the definition of the single-cell quantity In general, the firing rate depends on all single cell parameters included in Eqn.; i.e. there exists some function f such that (63) (64) (recall that the susceptibility for ω = 0 is the derivative of the firing rate with respect to the appropriate parameter (here, mean inhibitory conductance 〈g_I〉).

Low-rank approximation to the correlation matrix

We consider the correlation matrix of spike counts, as measured from Monte Carlo simulations; while these are in principle related to the cross-correlation functions C(t) defined in Methods: Computing statistics from linear response theory we will use C_T to denote the matrix of correlation coefficients measured for time window T; i.e. (65) Furthermore, we will restrict to the E-E correlations; i.e. C_T will be a n_E × n_E matrix, with ones on the diagonal (as ρ_T,ii = 1).

When we examined the singular values of the E-E correlation matrices obtained from Monte Carlo simulations, we noticed a consistent trend: there was usually one large cluster with one positive outlier. This motivates the following simple idea: by subtracting off a multiple of the identity matrix, λI, we shift the cluster towards zero; consequently C_T − λI is close to a rank-1 matrix. We then propose to use the sum of the two as an approximation to C_T: (66)

We seek the value λ which maximizes the fraction of the Frobenius norm explained by the first singular vector: i.e. in terms of the singular values, (67) (68) Since C_T is symmetric semi-positive definite, the singular values σ_j are equal to the eigenvalues λ_j: here σ₁ ≥ σ₂ ≥ ⋯ ≥ σ_r ≥ 0 and r is the rank of C_T. This has an exact solution: (69) Because we have subtracted a multiple of the identity matrix, none of the singular vectors will have changed. We then have (70) (71) By truncating this sum, we approximate C with a shifted low-rank matrix: (72) This procedure is similar to factor analysis, in which one seeks to explain a data vector as the sum of a random vector (u) and the linear combination of some number of latent factors (z) [48]: the entries of x would then have the correlation matrix Ψ + ΛΛ^T, where Ψ is a diagonal matrix containing the variances of u.