Overview

Our goal is to derive a relationship between the effective synaptic interactions between recorded neurons and the true synaptic interactions that would be obtained if the network were fully observed. This makes explicit how the synaptic interactions between neurons are modified by unobserved neurons in the network, and under what conditions these modifications are—or are not—significant. We derive this result first, using a probabilistic model of network activity in which all properties are known. We then build intuition by applying our result to two simple networks: a 3-neuron feedforward-inhibition circuit in which we are able to qualitatively reproduce measurements by Pouille and Scanziani [43], and a 4-neuron circuit that demonstrates how degeneracies in hidden networks are handled within our framework.

To extend our intuition to larger networks, we then study the effective interactions that would be observed in sparse random networks with N cells and strong synaptic weights that scale as [44–47], as has been recently observed experimentally [48]. We show how unobserved neurons significantly reshape the effective synaptic interactions away from the ground-truth interactions. This is not the case with “classical” synaptic scaling, in which synaptic strengths are inversely proportional to the number of inputs they receive (assumed ), as we will also show. (The case of classical scaling has also been studied previously using a different approach in [49–52]).

Model

We model the full network of N neurons as a nonlinear Hawkes process [53], often referred to as a “Generalized linear (point process) model” in neuroscience, and broadly used to fit neural activity data [16–23, 54]. Here we use it as a generative model for network activity, as it approximates common spiking models such as leaky integrate and fire systems driven by noisy inputs [55, 56], and is equivalent to current-based leaky integrate-and-fire models with soft-threshold (stochastic) spiking dynamics (see Methods).

To derive an approximate model for an observed subset of the network, we partition the network into recorded neurons (labeled by indices r) and hidden neurons (labeled by indices h). Each recorded neuron has an instantaneous firing rate λ_r(t) such that the probability that the neuron fires within a small time window [t, t + dt] is λ_r(t)dt, when conditioned on the inputs to the neuron. The instantaneous firing rate in our model is (1) where λ₀ is a characteristic firing rate, ϕ(x) is a non-negative, continuous function, μ_r is a tonic drive that sets the baseline firing rate of the neuron, and is the convolution of the synaptic interaction (or “spike filter”) J_i,j(t) with spike train from pre-synaptic neuron j to post-synaptic neuron i, for neural indices i and j that may be either recorded or hidden. In this work we take the tonic drive to be constant in time, and focus on the steady-state network activity in response to this drive. We consider interactions of the form , where the temporal waveforms g_j(t) are normalized such that for all neurons j. Because of this normalization, the weight carries units of time. We include self-couplings J_i,i(t) not to represent autapses, but to account for intrinsic neural properties such as refractory periods () or burstiness (). The firing rates for the hidden neurons follow the same expression with indices h and r interchanged.

We seek to describe the dynamics of the recorded neurons entirely in terms of their own set of spiking histories, eliminating the dependence on the activity of the hidden neurons. This demands calculating the effective membrane response of the recorded neurons by averaging over the activity of the hidden neurons conditioned on the activity of the recorded neurons. In practice this is intractable to perform exactly [57–60]. Here, we use a mean field approximation to calculate the mean input from the hidden neurons (again, conditioned on the activity of the recorded neurons). The value of deriving such a relationship analytically, as opposed to simply numerically determining the effective interactions, is that the resulting expression will give us insight into how the effective interactions decompose into contributions of different network features, how tuning particular features shapes the effective interactions, and conditions under which we expect hidden units to skew our measurements of connectivity in large partially observed networks.

As shown in detail in the Methods, the instantaneous firing rates of the recorded neurons can then be approximated as The effective baselines , are simply modulated by the net tonic input to the neuron, so we do not focus on them here. The ξ_r(t) are effective noise sources arising from fluctuation input from the hidden network. At the level of our mean field approximation these fluctuations vanish; corrections to the mean field approximation are straightforward and yield non-zero noise correlations, but will not impact our calculation of the effective interactions (see the Methods and SI), so as with the effective baselines we will not focus on the effective noise here.

The effective coupling filters are given in the frequency domain by (2) These results hold for any pair of recorded neurons r′ and r, and any choice of network parameters for which the mean field steady state of the hidden network exists. Here, the ν_h are the steady-state mean firing rates of the hidden neurons and is the linear response function of the hidden network to perturbations in the input. That is, Γ_h,h′(t − t′) is the linear response of hidden neuron h at time t due to a perturbation to the input of neuron h′ at time t′, and incorporates the effects of h′ propagating its signal to h through other hidden neurons, as demonstrated graphically in Fig 2. Both ν_h and are calculated in the absence of the recorded neurons. In deriving these results, we have neglected both fluctuations around the mean input from the hidden neurons, as well as higher order filtering of the recorded neuron spikes. For details on the derivations and justification of approximations, see the Methods and Supporting Information (SI).

The effective coupling filters are what we would—in principle—measure experimentally if we observe only a subset of a network, for example by pairwise recordings shown schematically in Fig 1. For larger sets of recorded neurons, interactions between neurons are typically inferred using statistical methods, an extremely nontrivial task [16–23, 28, 40, 41], and details of the fitting procedure could potentially further skew the inferred interactions away from what would be measured by controlled pairwise recordings. We will put aside these complications here, and assume we have access to an inference procedure that allows us to measure without error, so that we may focus on their properties and relationship to the ground-truth coupling filters.

Structure of effective coupling filters

The ground-truth coupling filters (as defined in Eq (1)) are modified by a correction term . The linear response function admits a series representation in terms of paths through the network through which neuron r′ is able to send a signal to neuron r via hidden neurons only.

We may write down a set of “Feynmanesque” graphical rules for explicitly calculating terms in this series [53]. First, we define the input-output gain of a hidden neuron h, , calculated in the absence of recorded neurons. The contribution of each term can then be written down using the following rules, shown graphically in Fig 2: i) for the edge connecting recorded neuron r′ to a hidden neuron h_i, assign a factor ; ii) for each node corresponding to a hidden neuron h_i, assign a factor ; iii) for each edge connecting hidden neurons h_i ≠ h_j, assign a factor ; and iv) for the edge connecting hidden neuron h_j to recorded neuron r, assign a factor . All factors for each path are multiplied together, and all paths are then summed over.

The graphical expansion is reminiscent of recent works expanding correlation functions of linear models of network spiking in terms of network “motifs” [61–63]. Computationally, this expression is practical for calculating the effective interactions in small networks involving only a few hidden neurons (as in the next section), but is generally unwieldy for large networks. In practice, for moderately large networks the linear response matrix can be calculated directly by numerical matrix inversion and an inverse Fourier transform back into the time domain. The utility of the path-length series is the intuitive understanding of the origin of contributions to the effective coupling filters and our ability to analytically analyze the strength of contributions from each path. For example, one immediate insight the path decomposition offers is that neurons only develop effective interactions between one another if there is a path by which one neuron can send a signal to the other.

Feedforward inhibition and degeneracy of hidden networks in small circuits

Strongly coupled large networks

Constructing networks that produce particular effective interactions is tractable for small circuits, but much more difficult for larger circuits composed of many circuit motifs. Not only can combinations of different circuit motifs interact in unexpected ways, one must also take care to ensure the resulting network is both active and stable—i.e., that firing will neither die out nor skyrocket to the maximum rate. Stability in networks is often implemented by either building networks with classical (or “weak”) synapses whose strength scales inversely with the number of inputs they receive, assumed here to be proportional to network size, and hence , or by building balanced networks in which excitatory and inhibitory synaptic strengths balance out, on average, and scale as [44, 48] (but note the distinction that we use a “soft threshold” firing model with nonlinearity that is fixed as N varies, whereas previous work has typically used hard threshold models). In both cases the synapses tend to be small in value in large networks, but are compensated for by large numbers of incoming connections. In the case of 1/N scaling, neurons are driven primarily by the mean of their inputs, while in “strong” balanced networks neurons are driven primarily by fluctuations in their inputs.

Our goal is to understand how the interplay between the presence of hidden neurons and different synaptic scaling or network architectures shapes effective interactions. Previous work has studied the hidden-neuron problem in the weak coupling limit [49–52] using a different approach to relate inferred synaptic parameters to true parameters; here we use our approach to study the strong coupling limit, theoretically predicted to be an important feature that supports computations in networks in a balanced regime [44–47]. Moreover, experiments in cultured neural tissue have been found to be more consistent with the scaling than 1/N [48], indicating that it may have intrinsic physiological importance.

We analytically determine how significantly effective interaction strengths are skewed away from the true interaction strengths as a function of both the number of observed neurons and typical synaptic strength. We consider several simple networks ubiquitous in neural modeling: first, an Erdős-Réyni (ER) network with “mixed synapses” (i.e., a neuron may have both positive and negative synaptic weights), a balanced ER network with Dale’s law imposed (a neuron’s synapses are all the same sign), and a Watts-Strogatz (WS) small world network with mixed synapses. Each network has N neurons and connection sparsity p (only 100p% of connections are non-zero). Connections in ER networks are chosen randomly and independently, while connections in the WS network are determined by randomly rewiring a fraction β of the connections in a (pN)^th-nearest-neighbor ring network. such that the overall network has a backbone of local synaptic connections with a web of sparse long-range connections. In each network N_rec neurons are recorded randomly.

For simplicity we take the baselines of all neurons to be equal, μ_i = μ₀ (such that in the absence of synaptic input the probability that a neuron fires in a short time window Δt is λ₀Δt exp(μ₀)). We choose the rate nonlinearity to be exponential, ϕ(x) = e^x; this is the “canonical” choice of nonlinearity often used when fitting this model to data [16–18, 20, 67]. We will further assume exp(μ₀) ≪ 1, so that we may use this as a small control parameter. For i ≠ j, the non-zero synaptic weights between neurons are independently drawn from a normal distribution with zero mean and standard deviation J₀/(pN)^a, where J₀ controls the overall strength of the weights and a = 1 or 1/2, corresponding to “weak” and “strong” coupling. For simplicity, we do not consider intrinsic self-coupling effects in this part of the analysis, i.e., we take for all neurons i. For the Dale’s law network, the overall distribution of synaptic weights follows the same normal distribution as the mixed synapse networks, but the signs of the weights correspond to whether the pre-synaptic neuron is excitatory or inhibitory. Neurons are randomly chosen to be excitatory and inhibitory, the average number of each type being equal so that the network is balanced. Numerical values of all parameters are given in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Network connectivity parameter values for Figs 5–8 and S1–S3.

See individual captions for other figures.

https://doi.org/10.1371/journal.pcbi.1006490.t001

We seek to assess how the presence of hidden neurons can shape measured network interactions. We first focus on the typical strength of the effective interactions as a function of both the fraction of neurons recorded, f = N_rec/N, and the strength of the synaptic weights J₀. We quantify the strength of the effective interactions by defining the effective synaptic weights ; c.f. for the true synaptic weights. We then study the sample statistics of the difference, , averaged across both subsets of recorded neurons and network instantiations, to estimate the typical contribution of hidden neurons to the measured interactions. The mean of the synaptic weights is near zero (because the weights are normally distributed with zero mean in the mixed synapse networks and due to balance of excitatory and inhibitory neurons in the Dale’s law network), so we focus on the root-mean-square of . This measure is a conservative estimate of changes in strength, as may have both positive and negative components that partially cancel when integrated over time, unlike J_r,r′(τ). An alternative measure we could have chosen that avoids potential cancellations is , i.e., the integrated absolute difference between effective and true interactions. However, this will depend on our specific choices of waveform g(τ) in our definition , whereas does not due to our normalization . As |∫ dτ f(τ)| ≤ ∫ dτ |f(τ)|, for any f(τ), we can consider our choice of as a lower bound on the strength that would be quantified by .

Numerical evaluations of the population statistics for all three network types are shown as solid curves in Fig (5), for both strong coupling and weak coupling. All three networks yield qualitatively similar results. The vertical axes measure the root-mean-square deviations between the statistically expected true synaptic and the corresponding effective synaptic weight , normalized by the true root mean square of . Thus, a ratio of 0.5 corresponds to a 50% root-mean-square difference in effective versus true synaptic strength. We measure these ratios as a function of both the fraction of neurons recorded (horizontal axis) and the parameter J₀ (labeled curves).

There are two striking effects. First, deviations are nearly negligible () for 1/N scaling of connections (gray traces in Fig 5). Thus, for large networks with synapses that scale with the system size, vast numbers of hidden neurons combine to have negligible effect on effective couplings. This is in marked contrast to the case when coupling is strong ( scaling), when hidden neurons have a pronounced impact (purple traces in Fig 5). This is particularly the case when f ≪ 1—the typical experimental case in which the hidden neurons outnumber observed ones by orders of magnitude—or when J₀ ≲ 1.0, when typical deviations become half the magnitude of the true couplings themselves (upper blue line). For J₀ ≳ 1.0, the network activity is unstable for an exponential nonlinearity.

To gain analytical insight into these numerical results, we calculate the standard deviation , normalized by , for contributions from paths up to length-3, focusing on the case of the ER network with mixed synapses (the Dale’s law and WS networks are more complicated, as the moments of the synaptic weights depend on the identity of the neurons). For strong coupling we find (4) corresponding to the black dashed curves in Fig 5 left. Eq (4) is a truncation of a series in powers of , where f = N_rec/N is the fraction of recorded neurons. The most important feature of this series is the fact that it only depends on the fraction of recorded neurons f, not the absolute number, N. Contributions from long paths remain finite, even as N → ∞. In contrast, the corresponding expression for in the case of weak 1/N coupling is a series in powers of , so that contributions from long paths are negligible in large networks N ≫ 1. (See [67] for derivation and results for N = 100.) Deviations of Eq (4) from the numerical solutions in Fig 5 indicate that contributions from truncated terms are not negligible when f ≪ 1. As these terms correspond to paths of length-4 or more, this shows that long chains through the network contribute significantly to shaping effective interactions.

The above analysis demonstrates that the strength of the effective interactions can deviate from that of the true direct interactions by as much as 50%. However, changes in strength do not give us the full picture—we must also investigate how the temporal dynamics of the effective interactions change. To illustrate how hidden units can skew temporal dynamics, in Fig 6 we plot the effective vs. true interactions between N_rec = 3 neurons in an N = 1000 neuron network. Because the three network types considered in Fig 5 yield qualitatively similar results, we focus on the Erdős-Réyni network with mixed synapses.

Four of the true interactions between neurons shown in Fig 6 are non-zero (, , , and ). Of these, three exhibit only slight differences between the true and effective interactions: and have slightly longer decay timescales than their true counterparts, while has a slightly shorter timescale, indicating the contribution of the hidden network to these interactions was either small or cancelled out. However, the interaction differs significantly from the true interaction, becoming initially excitatory but switching to inhibitory after a short time, as in our earlier example case of feedforward inhibition. This indicates that neuron 2 must drive a cascade of neurons that ultimately provide inhibitory input to neuron 3.

Contrasting the true and effective interactions shown in Fig 6 highlights many of the ways in which hidden neurons skew the temporal properties of measured interactions. An immediately obvious difference is that although the true synaptic connections in the network are sparse, the effective interactions are not. This is a generic feature of the effective interaction matrix, as in order for an effective interaction from a neuron r′ to r to be identically zero there cannot be any paths through the network by which r′ can send a signal to r.¹ In a random network the probability that there are no paths connecting two nodes tends to zero as the network size N grows large. Note that this includes paths by which each neuron can send a signal back to itself, hence the neurons developed effective self-interactions, even though the true self-interactions are zero in these particular simulations.

Dealing with degeneracy

The issue of degeneracy in complex biological systems and networks has been discussed at length in the literature, in the context of both inherent degeneracies—multiple different network architectures can produce the same qualitative behaviors [34, 37–39], as well as degeneracies in our model descriptions—many models may reproduce experimental observations, demanding sometimes arbitrary criteria for selecting one model over another. All have implications for how successfully one can infer unobserved network properties. One kind of model degeneracy, “sloppiness” [35, 65], describes models in which the behavior of the model is sensitive to changes in only a relatively small number of directions in parameter space. Many models of biological systems have been shown to be sloppy [35]; this could account for experimentally observed networks that are quite different in composition but produce remarkably similar behaviors. Sloppiness suggests that rather than trying to infer all properties of a hidden network, there may be certain parameter combinations that are much more important to the overall network operation, and could potentially be inferred from subsampled observations.

Another perspective on model degeneracy comes from the concepts of “universality” that occur in random matrix theory [68, 69] and Renormalization Group methods of statistical physics [64]. Many bulk properties of matrices (e.g., the distribution of eigenvalues) whose entrees are combinations random variables, such as our , are universal in that they depend on only a few key details of the distribution that individual elements are drawn from [70]. Similarly, one of the central results of the Renormalization Group shows that models with drastically disparate features may yield the same coarse-grained model structure when many degrees of freedom are averaged out, as in our case of approximately averaging out hidden neurons. Different distributions (in the case of random matrix theory) or different models (in the case of the Renormalization group) that yield the same bulk properties or coarse-grained models are said to be in the same “universality class.” Measuring particular quantities under a range of experimental conditions (e.g., different stimuli) may be able to reveal which universality class an experimental system belongs to and eliminate models belonging to other universality classes as candidate generating models of the data, but these measurements cannot distinguish between models within a universality class.

Purely feedforward networks and recurrent networks are simple examples of broad universality classes in this context. In any randomly sampled feedforward network, only the feedforward interactions are modified or generated; no lateral or feedback connections develop because there is no path through hidden neurons that a recorded neuron can send signals to recorded neurons in the same or previous layers. Thus, the feedforward structure—a topological property of the network—is preserved. However, adding even a single feedback connection can destroy this topological structure if it joins two neurons connected by a feedforward path—i.e., such a link creates a cycle within the network, and it is no longer feedforward. If this network is heavily subsampled (f ≪ 1) the resulting effective interactions can even be fully recurrent. The majority of the effective interactions may be very weak, but nonetheless from a topological perspective the network has been fundamentally altered. Accordingly, any interactions that represent a purely feedforward network could not have come from a network with recurrent interactions. In practice, we expect few, if any, cortical networks to be purely feedforward, so most networks will be recurrent if we consider only the network connections and not the connection strengths. Thus, a more interesting question is how the statistics and dynamics of synaptic weights further partition topologically-defined universality classes; for example, whether the distribution of synaptic weights can split the sets of that arise from recurrent networks and predominantly feedforward networks with sparse feedback and lateral interactions into different universality classes. A thorough investigation of such phenomena will be the focus of future work.

Inference of hidden network features

Despite the many possible confounds network degeneracy produces, much of the work on inference of hidden network properties has focused on inferring the individual interactions between neurons, with varying degrees of success. Both Dunn and Roudi [40] as well as Tyrcha and Hertz [41] studied inference of hidden activity in kinetic Ising models, sometimes used as simple minimal models of neuronal network activity. They found that the synaptic weights between pairs of observed neurons and observed-hidden pairs could be recovered to within reasonably small mean-squared-error when the number of hidden neurons was less than the number of observed neurons. However, both methods also found it difficult to infer connections between pairs of hidden neurons, resorting to setting such connections to zero in order to stabilize their algorithms. Tyrcha and Hertz also note their method recovers only an equivalence class of connections due to degeneracy in the possible assignment of signs of the synaptic weights and hidden neuron labels. This suggests inferring hidden network structure will be nearly impossible in the realistic limit N_rec ≪ N_hid.

A series of papers by Bravi and Sollich perform theoretical analyses of hidden dynamics inference in chemical reaction networks, modeled by a system of Langevin equations [57–60]. Although the applications the authors have in mind are signaling pathways such as epidermal growth factor reaction networks, one could imagine re-interpreting or adapting these equations to describe rate models. The authors develop a variety of approaches, including Plefka expansions [57–59] and variational Gaussian approximations [60], to study how observations constrain the inferred hidden dynamics, assuming particlar properties of the network structure. Ref. [60] in particular takes an approach most similar to ours, deriving an effective system of Langevin equations for the subsampled dynamics of the chemical reaction network. The effective system of equations contains a memory kernel that plays a role analogous to the correction to the interactions between neurons in our work (second term in our Eq (2). However, the structure of the memory kernel in [60] has a rather different form, being exponentially dependent on the integral of the hidden-hidden interactions, in contrast to our Γ_h,h′(t − t′), which depends on the inverse of δ_h,h′δ(t − t′) − γ_hJ_h,h′(t − t′) (see Methods). Though Bravi and Sollich do not expand their memory kernel in a series as we do, it would admit a similar series and interpretation in terms of paths through the hidden network, as in our Fig 2A. However, due to the exponential dependence on the hidden-hidden interactions, long paths of length ℓ through hidden networks are suppressed by factors ℓ!, suggesting the hidden network may have less influence in such networks compared to the network dynamics we study here.

Closest to our choice of model, Pillow and Latham [28] and Soudry et al[25] both use modifications of nonlinear Hawkes models to fit neural data with unobserved neurons. Pillow and Latham outline a statistical approach for inferring not just interactions with and between hidden neurons, but also the spike trains of hidden neurons, testing the method on a network of two neurons (one hidden). To properly infer the spike train of the hidden neurons, the model must allow for acausal synaptic interactions. This is acceptable if the goal is inferring hidden spike trains: for example, if the hidden neuron were to make a strong excitatory synapse onto the observed neuron, then a spike from the observed neuron increases the probability that the hidden neuron fired a spike in the recent past. An acausal synaptic interaction captures this effect, but is of course an unphysical feature in a mechanistic model, precluding physiological interpretation of such an interaction.

Soudry et al are concerned with the fact that common input from hidden neurons will skew estimates of network connectivity. To get around this issue, they present a different take on the hidden unit problem: rather than attempt to infer connectivity in a fixed subsample of a network, they propose a shotgun sampling method, in which a sequence of overlapping random subsets of the network are sampled over a long experiment. Under this procedure, a large fraction of the network can be sampled, just not contiguously in time, and reconstruction of the entire network could in principle be accomplished. Soudry et al show this strategy works in their simulated networks (even when the generative model is a hard-threshold leaky-integrate-and-fire rather than the nonlinear Hawkes model, which can be interpreted as a soft-threshold leaky-integrate-and-fire model; see SI). However, sampling the entire network may only be feasible in vitro; sampling of neurons in vivo, such as in wide-field calcium imaging studies, will still necessarily miss neurons not in the field of view or too deep in the tissue; in such cases our work provides the means to properly interpret the inferred effective interactions obtained with such a method.

Although a thorough treatment of statistical inference of hidden network properties is beyond the scope of our present work, we may make some general remarks on future work in these directions. The nonlinear Hawkes model we use here is commonly used to fit neural population activity data, and one could infer the effective baselines , interactions , and noise ξ_r(t) using existing techniques. In particular, Vidne et al. [22] explicitly fit the noise, which is likely important for proper inference, as otherwise effects of the noise could be artificially inherited by the effective interactions. Once such estimates are obtained, one could then in principle infer certain hidden network properties by combining a statistical model for these properties with the relationships between effective and true interactions derived in this work, such as Eq (2). (Detailed physiological measurements of ground-truth synaptic interactions in small volumes of neural tissue can be used to refine estimates). As we have stressed throughout this paper, inferring the exact connections between hidden neurons may be impossible due to a large number of degenerate solutions consistent with observations. However, one may be able to infer bulk properties of the network, such as the parameters governing the distribution of hidden-network connections, or even more exotic properties such as the eigenvalue distribution of the hidden network connection weight matrix. We leave these ideas as interesting directions for future work.

Hidden neurons and dimensionality reduction

Given the challenges that hidden network inference poses, one might wonder if there are network properties that can be reliably measured even with subsampled neural activity. Collective, low-dimensional dynamics have emerged as a possible candidate: recent work has investigated the effect that subsampled measurements have on estimating collective low-dimensional dynamics of trial-averaged network activity (using, e.g., principal components analysis). During a task, the effective dimensionality of a network’s dynamics is constrained [71, 72], opening the possibility that the subsampled population may be sufficient to accurately represent these task-constrained low-dimensional dynamics. Indeed, under certain assumptions—in particular that the collective dynamical modes are approximately random superpositions of neural activity and that sampled neurons are statistically representative of the hidden population—Gao et al. [72] calculate a conservative upper bound on the number of sampled neurons necessary to reconstruct the collective dynamics, finding it is often less than the effective dimensionality of the network.

The assumption that the collective dynamics are random superpositions of neural activity is crucial, because it means that each neuron’s trial-averaged dynamics are in turn a superposition of the collective modes. Hence, every neuron’s activity contains some information about the collective modes, and if only a few of these modes are important, then they can be extracted from any sufficiently large subset of neurons.

While modes of collective activity alone may be sufficient for answering certain questions, such as decoding task parameters or elucidating circuit function, explaining the structure of these modes—and in particular how the dynamical patterns that emerge under different task conditions or sensory environments are related—will ultimately require an understanding of the distribution of possible underlying network properties, which remain difficult to estimate from subsampled populations. We may be able to establish such structure-function relationships using our theory of effective interactions presented in this work: if we can relate the collective dynamics extracted from subsampled neurons to the properties of the effective interactions , then we can link them to the true interactions through our Eq (2). With an understanding of how network properties shape such collective dynamics, we can begin to understand what network manipulations achieve desired patterns of activity, and therefore circuit function.

Implications beyond experimental limitations

The fact that many different hidden networks may yield the same set of effective interactions or low-dimensional dynamics suggests that the effective interactions themselves may yield direct insight into a circuit’s functions. For instance, many circuits consist of principal neurons that transmit the results of circuit computation to downstream circuitry, but often do not make direct connections with one another, instead interacting through (predominantly inhibitory) intermediaries called interneurons. From the point of view of a downstream circuit, the principal neurons are “recorded” and the interneurons are “hidden.” A potential reason for this general arrangement is that direct synaptic interactions alone are insufficient to produce the membrane responses required to perform the circuit’s computations, and the network of interneurons reshapes the membrane responses of projection neurons into effective interactions that can perform the desired computations—it may thus be that the effective interactions should be of primary interest, not necessarily the (possibly degenerate choices of) physiological synaptic interactions. For example, in the feedforward inhibitory circuits of Figs 3 and 4, the roles of the hidden inhibitory neurons may simply be to act as interneurons that reshape the interaction between the excitatory projection neurons 1 and 2, and the choice of which particular circuit motif is implemented in a real network is determined by other physiological constraints, not only computational requirements.

One of the greatest achievements in systems neuroscience would be the ability to perform targeted modifications to a large neural circuit and selectively alter its suite of computations. This would have powerful applications for both studying a circuit’s native computations, but also repurposing circuits or repairing damaged circuitry (due to, e.g., disease). If the computational roles of circuits are indeed most sensitive to the effective interactions between principal neurons, this suggests we can exploit potential degeneracies in the interneuron architecture and intrinsic properties to find some circuit that achieves a desired computation, even if it is not a physiologically natural circuit. Our main result relating effective and true interactions, Eq (2), provides a foundation for future work investigating how to identify sets of circuits that perform a desired set of computations. We have shown in this work that it can be done for small circuits (Figs 3 and 4), and that the effective interactions in large random networks can be significantly skewed away from the true interactions when synaptic weights scale as , as observed in experiments [48]. This holds promise for identifying principled approaches to tuning or controlling neural interactions, such as by using neuromodulators to adjust interneuron properties or inserting artificial or synthetic circuit implants into neural tissue to act as “hidden” neurons. If successful, this could contribute to the long term goal of using such interventions to aid in reshaping the effective synaptic interactions between diseased neurons, and thereby restore healthy circuit behaviors.

Model definition and details

The firing rate of a neuron i in the full network is given by (5) where λ₀ is a characteristic rate, ϕ(x) ≥ 0 is a nonlinear function, μ_i (potentially a function of some external stimulus θ) is a time-independent tonic drive that sets the baseline firing rate of the neuron in the absence of input from other neurons, is an external input current, and J_ij(t − t′) is a coupling filter that filters spikes fired by presynaptic neuron j at time t′, incident on post-synaptic neuron i. We will take for simplicity in this work, focusing on the activity of the network due to the tonic drives μ_i (which could be still be interpreted as external tonic inputs, so the activity of the network need not be interpreted as spontaneous activity).

While we need not attach a mechanistic interpretation to these filters, a convenient interpretation is that the nonlinear Hawkes model approximates the stochastic dynamics of a leaky integrate-and-fire network model driven by noisy inputs [55, 56]. In fact, the nonlinear Hawkes model is equivalent to a current-based integrate-and-fire model in which the deterministic spiking rule (a spike fires when a neuron’s membrane potential reaches a threshold value V_th) is replaced by a stochastic spiking rule (the higher a neuron’s membrane potential, the higher the probability a neuron will fire a spike). (It can also be mapped directly to a conductance-based in special cases [73]). For completeness, we present the mapping from a leaky integrate-and-fire model with stochastic spiking to Eq (5) in the Supporting Information (SI).

Derivation of effective baselines and coupling filters

To study how hidden neurons affect the inferred properties of recorded neurons, we partition the network into “recorded” neurons, labeled by indices r (with sub- or superscripts to differentiate different recorded neurons, e.g., r and r′ or r₁ and r₂) and “hidden” neurons labeled by indices h (with sub- or superscripts). The rates of these two groups are thus To simplify notation, we write . If we seek to describe the firing of the recorded neurons only in terms of their own spiking history, input from hidden neurons effectively acts like noise with some mean amount of input. We thus begin by splitting the hidden input to the recorded neurons up into two terms, the mean plus fluctuations around the mean: where denotes the mean activity of the hidden neurons conditioned on the activity of the recorded units, and ξ_r(t) are the fluctuations, i.e., . Note that ξ_r(t) is also conditional on the activity of the recorded units.

By construction, the mean of the fluctuations is identically zero, while the cross-correlations can be expressed as where is the cross-covariance between hidden neurons h₁ and h₂ (conditioned on the spiking of recorded neurons). If the autocorrelation of the fluctuations (r = r′) is small compared to the mean input to the recorded neurons (), or if J_r,h is small, then we may neglect these fluctuations and focus only on the effects that the mean input has on the recorded subnetwork. At the level of the mean field theory approximation we make in this work, the spike-train correlations are zero. One can calculate corrections to mean field theory (see SI) to estimate the size of this noise. Even when this noise is not strictly negligible, it can simply be treated as a separate input to the recorded neurons, as shown in the main text, and hence will not alter the form of the effective couplings between neurons. Averaging out the effective noise, however, would generate new interactions between neurons; we leave investigation of this issue for future work.

In order to calculate how hidden input shapes the activity of recorded neurons, we need to calculate the mean . This mean input is difficult to calculate in general, especially when conditioned on the activity of the recorded neurons. In principle, the mean can be calculated as This is not a tractable calculation. Taylor series expanding the nonlinearity ϕ(x) reveals that the mean will depend on all higher cumulants of the hidden unit spike trains, which cannot in general be calculated explicitly. Instead, we again appeal to the fact that in a large, sufficiently connected network, we expect fluctuations to be small, as long as the network is not near a critical point. In this case, we may make a mean field approximation, which amounts to solving the self-consistent equation (6) In general, this equation must be solved numerically. Unfortunately, the conditional dependence on the activity of the recorded neurons presents a problem, as in principle we must solve this equation for all possible patterns of recorded unit activity. Instead, we note that the mean hidden neuron firing rate is a functional of the filtered recorded input , so we can expand it as a functional Taylor series around some reference filtered activity , Within our mean field approximation, the Taylor coefficients are simply the response functions of the network—i.e., the zeroth order coefficient is the mean firing rate of the neurons in the reference state , the first order coefficient is the linear response function of the network, the second order coefficient is a nonlinear response function, and so on.

There are two natural choices for the reference state . The first is simply the state of zero recorded unit activity, while the second is the mean activity of the recorded neurons. The zero-activity case conforms to the choice of nonlinear Hawkes models used in practice. Choosing the mean activity as the reference state may be more appropriate if the recorded neurons have high firing rates, but requires adjusting the form of the nonlinear Hawkes model so that firing rates are modulated by filtering the deviations of spikes from the mean firing rate, rather than filtering the spikes themselves. Here, we focus on the zero-activity reference state. We present the formulation for the mean field reference state in the SI.

For the zero-activity reference state , the conditional mean is The mean inputs are the mean field approximations to the firing rates of the hidden neurons in the absence of the recorded neurons. Defining , these firing rates are given by in writing this equation we have assumed that the steady-state mean field firing rates will be time-independent, and hence the convolution , where . This assumption will generally be valid for at least some parameter regime of the network, but there can be cases where it breaks down, such as if the nonlinearity ϕ(x) is bounded, in which case a transition to chaotic firing rates ν_h(t) may exist (c.f. [74]). The mean field equations for the ν_h are a system of transcendental equations that in general cannot be solved exactly. In practice we will solve the equations numerically, but we can develop a series expansion for the solutions (see below).

The next term in the series expansion is the linear response function of the hidden unit network, given by the solution to the integral equation The “gain” γ_h is defined by where ϕ′(x) is the derivative of the nonlinearity with respect to its argument.

For time-independent drives μ_r and steady states ν_h (and hence γ_h), we may solve for Γ_h,h′(t − t′) by first converting to the frequency domain and then performing a matrix inverse: where .

If the zero and first order Taylor series coefficients in our expansion of are the dominant terms—i.e., if we may neglect higher order terms in this expansion—then we may approximate the instantaneous firing rates of the recorded neurons by where are the effective baselines of the recorded neurons and are the effective coupling filters in the frequency domain, as given in the main text. In addition to neglecting the higher order spike filtering terms here, we have also neglected fluctuations around the mean input from the hidden network. These fluctuations are zero within our mean field approximation, but we could in principle calculate corrections to the mean field predictions using the techniques of [53]; we do so to estimate the size of the effective noise correlations in the SI.

In the main text, we decompose our expression for into contributions from all paths that a signal can travel from neuron r′ to r. To arrive at this interpretation, we note that we can expand in a series over paths through the hidden network. To start, we note that if for some matrix norm ||⋅||, then the matrix admits a convergent series expansion [75] where is a matrix product and . We can write an element of the matrix product out as inserting yields This expression can be interpreted in terms of summing over paths through network of hidden neurons that join two observed neurons: the are represented by edges from neuron h_j to h_i, and the are represented by the nodes. In this expansion, we allow edges from one neuron back to itself, meaning we include paths in which signals loop back around to the same neuron arbitrarily many times before the signal is propagated further. However, such loops can be easily factored, contributing a factor . We thus remove the need to consider self-loops in our rules for calculating effective coupling filters by assigning a factor γ_h/(1 − γ_h J_{h, h}(ω)) to each node, as discussed in the main text and depicted in Fig 2. (The contribution of the self-feedback loops can be derived rigorously; see the SI for the full derivation).

Although we have worked here in the frequency domain, our formalism does adapt straightforwardly to handle time-dependent inputs; however, among the consequences of this explicit time-dependence are that the mean field rates ν_h(t) are not only time-dependent, but solutions of a system of nonlinear integral equations, and hence more challenging to solve. Furthermore, quantities like the linear response of the hidden network, Γ_h,h′(t, t′), will depend on both absolute times t and t′, rather than just their difference, t − t′, and hence we must also (numerically) solve for Γ_h,h′(t, t′) directly in the time domain. We leave these challenges for future work.

Model network architectures

Our main result, Eq (2), is valid for general network architectures with arbitrary weighted synaptic connections, so long as the hidden subset of the network has stable dynamics when the recorded neurons are removed. An example for which our method must be modified would be a network in which all or the majority of the hidden neurons are excitatory, as the hidden network is unlikely to be stable when the inhibitory recorded neurons are disconnected. Similarly, we find that synaptic weight distributions with undefined moments will generally cause the network activity to be unstable. For example, drawn from a Cauchy distribution generally yield unstable network dynamics unless the weights are scaled inversely with a large power of the network size N.

Choice of nonlinearity ϕ(x)

The nonlinear function ϕ(x) sets the instantaneous firing rate for the neurons in our model. Our main analytical results (e.g., Eq (2) hold for arbitrary choice of ϕ(x). Where specific choices are required in order to perform simulations, we used ϕ(x) = max(x, 0) for the results presented in Figs 3 and 4 and ϕ(x) = exp(x) otherwise. The rectified linear choice is convenient for small networks, as high-order derivatives are zero, which eliminates corresponding high-order “loop corrections” to mean field theory [53]. The exponential function is the “canonical” choice of nonlinearity for the nonlinear Hawkes process [16–18, 20]. The exponential has particularly nice theoretical properties, but is also convenient for fitting the nonlinear Hawkes model to data, as the log-likelihood function of the model simplifies considerably and is convex (though some similar families of nonlinearities also yield convex log-likelihood functions).

An important property that both choices of nonlinearity possess is that they are unbounded. This property is necessary to guarantee that a neuron spikes given enough input. A bounded nonlinearity imposes a maximum firing rate, and neurons cannot be forced to spike reliably by providing a large bolus of input. The downside of an unbounded nonlinearity is that it is possible for the average firing rates to diverge, and the network never reaches a steady state. For example, in a purely excitatory network (all ) with an exponential nonlinearity, neural firing will run away without a sufficiently strong self-refractory coupling to suppress the firing rate. This will not occur with a bounded nonlinearity, as excitation can only drive neurons to fire at some maximum but finite rate.

This can be a problem in simulations of networks obeying Dale’s law. For unbounded nonlinearities, the mean field theory for the hidden network occasionally does not exist due to an imbalance of excitatory and inhibitory neurons caused by our random selection of recorded of neurons. However, the Dale’s law network is stable if the nonlinearity is bounded. We demonstrate this below in Figs 7 and 8, comparing simulations of the effective interaction statistics in Erdős-Réyni networks with and without Dale’s law for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Same as Fig 5A in main text, but for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

https://doi.org/10.1371/journal.pcbi.1006490.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Same as Fig 5B in the main text, but for a sigmoidal nonlinearity ϕ(x) = 2/(1 + e^−x).

Because the sigmoid is bounded the mean field solution cannot diverge, yielding better results.

https://doi.org/10.1371/journal.pcbi.1006490.g008

Another consequence of unbounded nonlinearities is that the mean firing rates are either finite or they diverge. Bounded nonlinearities, on the other hand, may allow for the possibility of a transition to chaotic dynamics in the mean-field firing rate dynamics (cf. the results of the [74]).

Specific choices of network properties used to generate figures

Feedforward-inhibitiory circuit model details.

3 neuron circuit (Fig 3).

Using our graphical rules (Fig 2), we calculated the effective interaction from neuron 1 to 2 for the circuit shown in Fig 3A, giving Eq 3. In principle, our mean field approximation would not be expected to hold for such a small circuit; in particular, loop corrections [53] to our calculation of the rate ν₃ and associated gain γ₃ might be significant. However, as loop corrections depend on derivatives of the nonlinearity ϕ(x), we can minimize these errors by choosing ϕ(x) = max(x, 0), for which ϕ′(x) = Θ(x), the Heaviside step function. Accordingly, we can solve for and γ₃ = λ₀ for this particular network.

To generate the plots shown in Fig 3C, we take the inter-neuron couplings to have the form and the self-history couplings to have the form .

Using Mathematica to perform the inverse Fourier transform, we obtain an explicit expression for the effective interaction, In order for the inverse Fourier transform to converge and result in a causal function, we require that .

Parameter values used to generate the plots in Fig 3C are given in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Parameter values for Fig 3C.

Setting λ₀ = 1.0 simply sets the units of frequency and time to be measured relative to λ₀ (e.g., the value α₃₁ = 1.8 really means α₃₁ = 1.8λ₀ and really means ).

https://doi.org/10.1371/journal.pcbi.1006490.t002

4 neuron circuit (Fig 4).

Like for the 3-neuron circuit, we can use our graphical rules (Fig 2) to calculate the effective interaction for our 4-neuron circuit (Fig 4A) in the frequency domain: in going to the last equality we have separated the terms out into contributions from each of the paths, in order, shown in Fig 4B.

To generate the plots in Fig 4C, we choose ϕ(x) = max(x, 0), which gives γ_i = λ₀, as in Fig 3C, and interaction filters for the direct interaction and for all other interactions shown—i.e., all other interactions have the same decay time α⁻¹ for simplicity.

Inverting the Fourier transform using Mathematica yields In order for this result to converge, we require . Splitting this result up into the contributions to each plot in Fig 4C, using the specific parameter choices λ₀ = 1 and , gives Parameter values used to generate the plots in Fig 4C are given in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Parameter values for Fig 4C.

Setting λ₀ = 1.0 simply sets the units of frequency and time to be measured relative to λ₀ (e.g., the value α = 1.294 really means α = 1.294λ₀ and really means ).

https://doi.org/10.1371/journal.pcbi.1006490.t003

Series approximation for the mean field firing rates for the case of exponential nonlinearity ϕ(x) = e^x

The mean field firing rates for the hidden neurons are given by where we focus specifically on the case of exponential nonlinearity ϕ(x) = exp(x). For this choice of nonlinearity, γ_h = ν_h, so we do not need to calculate a separate series for the gains.

This system of transcendental equations generally cannot be solved analytically. However, for small exp(μ_h) ≪ 1 we can derive, recursively, a series expansion for the firing rates. We first consider the case of μ_h = μ₀ for all hidden neurons h. Let ϵ = exp(μ₀). We may then write Plugging this into the mean field equation, Thus, matching powers of λ₀ϵ on the left and right hand sides, we find and for ℓ > 0.

For ℓ = 1, the sum in m truncates at m = 1 (as is zero for m > ℓ, as all indices are positive). Thus, With this we have calculated the firing rates to .

The analysis can be straightforwardly extended to the case of heterogeneous μ_h, though it becomes more tedious to compute terms in the (now multivariate) series. Assuming ϵ_h ≡ exp(μ_h) ≪ 1 for all h, to we find

Variance of the effective coupling to second order in N_rec/N & fourth order in (exponential nonlinearity)

To estimate the strength of the hidden paths, we would like to calculate the variance of the effective coupling and compare its strength to the variance of the direct couplings , where and , as in the main text.

We assume that the synaptic weights are independently and identically distributed with zero mean and variance for i ≠ j, where a = 1 corresponds to weak coupling and a = 1/2 corresponds to strong coupling. We assume no self-couplings, for all neurons i. The overall factor of p in comes from the sparsity of the network. For example, for normally distributed non-zero weights with variance , the total probability for every connection in the network is

Because the are i.i.d., the mean of : where we used the fact that depends only on the hidden neuron couplings , which are independent of the couplings to the recorded neurons, and . This holds for any pair of neurons (r, r′), including r = r′ because of the assumption of no self-coupling.

The variance of is thus equal to the mean of its square, for r ≠ r′, In this derivation, we used the fact that due to the fact that the synaptic weights are uncorrelated. We now need to compute . This is intractable in general, so we will resort to calculating this in a series expansion in powers of ϵ ≡ exp(μ₀) for the exponential nonlinearity model. Our result will also turn out to be an expansion in powers of J₀ and 1 − f ≡ N_hid/N.

The lowest order approximation is obtained by the approximation ν_h ≈ λ₀ϵ and Γ_h,h′ ≈ ν_hδ_h,h′, yielding (7) This result varies linearly with f, while numerical evaluation of the variance shows obvious curvature for f ≪ 1 and J₀ ≲ 1, so we need to go to higher order. This becomes tedious very quickly, so we will only work to (it turns out corrections vanish).

We calculate using a recursive strategy, though we could also use the path-length series expression for , keeping terms up to fourth order in ϵ. We begin with the expression and plug it into itself until we obtain an expression to a desired order in ϵ. In doing so, we note that , so we will first work to fourth order in ν_h, and then plug in the series for ν_h in powers of ϵ.

We begin with The third order term vanished because we assume no self-couplings. We have obtained to fourth order in ν_h; now we need to plug in the series expression for ν_h to obtain the series in powers of λ₀ϵ. We will do this order by order in ν_h. The easiest terms are the fourth order terms, as The second order term is where . We need the average . The third-order term will vanish upon averaging, and using the fact that synaptic weights are independent (giving the factor) and self-couplings are zero (giving the factor). We thus obtain

The first fourth order term in , , will vanish upon averaging because matching indices requires h^′′ = h = h′ and we assume no self-couplings. The second fourth order term is , which averages to , where the factor of (1 − δ_h,h′) again accounts for the fact that this term does not contribute when h = h′ due to no self-couplings. We thus arrive at Putting everything together, For weak coupling, this tends to 1 in the N ≫ 1 limit, as , for fixed fraction of observed neurons f = N_rec/N. For strong coupling, , which is constant as N → ∞, and hence (8) where we have used little-o notation to denote that there are higher order terms dominated by (λ₀ J₀ϵ)⁴(1 − f)². With this expression, we have improved on our approximation of the relative variance of the effective coupling to the true coupling; however, the neglected higher order terms still become significant as f → 0 and J₀ → 1, indicating that hidden paths have a significant impact when synaptic strengths are moderately strong and only a small fraction of the neurons have been observed.

Because the synaptic weights are independent, we may rewrite Eq (8) as or, in terms of the ratio of standard deviations, where we used the approximation for x small.

In the main text, we plotted results for N = 1000 total neurons (Fig 5A). For strongly coupled networks, the results should only depend on the fraction of observed neurons, f = N_rec/N, while for weak coupling the results do depend on the absolute number N. To demonstrate this, in Fig 9 we remake Fig 5 for N = 100 neurons. We see that the strongly coupled results have not been significantly altered, whereas the weakly coupled results yield stronger deviations (as the deviations are ).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Same as Fig 5, but for N = 100 neurons and N_rec = {1, 11, 21, 31, 41, 51, 61, 71, 81, 91, 99} recorded neurons.

Because we plot the relative deviations of the coupling strength against the fraction of observed neurons, the curves for the strongly coupled case are the same as for N = 1000, as expected, while the weakly coupled case yields stronger deviations.

https://doi.org/10.1371/journal.pcbi.1006490.g009