Modeling Inhibitory Interneurons in Efficient Sensory Coding Models

There is still much unknown regarding the computational role of inhibitory cells in the sensory cortex. While modeling studies could potentially shed light on the critical role played by inhibition in cortical computation, there is a gap between the simplicity of many models of sensory coding and the biological complexity of the inhibitory subpopulation. In particular, many models do not respect that inhibition must be implemented in a separate subpopulation, with those inhibitory interneurons having a diversity of tuning properties and characteristic E/I cell ratios. In this study we demonstrate a computational framework for implementing inhibition in dynamical systems models that better respects these biophysical observations about inhibitory interneurons. The main approach leverages recent work related to decomposing matrices into low-rank and sparse components via convex optimization, and explicitly exploits the fact that models and input statistics often have low-dimensional structure that can be exploited for efficient implementations. While this approach is applicable to a wide range of sensory coding models (including a family of models based on Bayesian inference in a linear generative model), for concreteness we demonstrate the approach on a network implementing sparse coding. We show that the resulting implementation stays faithful to the original coding goals while using inhibitory interneurons that are much more biophysically plausible.


RFs of excitatory cells
Here we derive the RFs of principal cells in a linear generative model. The stimuli array of sparse dots used for mapping RF is represented by an identity matrix I = [s 1 , ..., s M ], and the rows of the response matrix A are the RFs. 1 In a linear generative model: (S1) First consider the case when Φ (and thus Φ T Φ) is full rank. The receptive field mapped with sparse dots is the Moore-Penrose inverse of the dictionary matrix: Specifically when Φ is orthogonal, indicating that the RFs of the excitatory cells are approximately the same as the dictionary elements.
For an arbitrary overcomplete Φ, there is no unique solution to Eq. (S1). If one regularizes the problem with a sparsity constraint on the columns of A, A can be inferred using a linear dynamical system coupled with a nonlinearity (Eq. (4)) [2]. The solution at steady state under sparse dots stimuli, ignoring the thresholding (a ≈ u) can be written as: similar to Eq. (S2). When the dictionary elements are sufficiently different from one another (i.e. Φ T Φ is close to identity), again we have Eq. (S3). Empirically in simulations we observe that the tuning property of the RFs are indeed similar to that of the dictionary when using an overcomplete dictionary learned from natural scenes. A similar observation was also made previously in [1].

RFs of inhibitory cells in the direct implementation
When the Gramian matrix is decomposed directly (Eq. (8)), the RFs of the inhibitory cells are (Φ T Φ) + A. According to Eq. (S5), this is approximately the same as the dictionary elements (orientation tuned).

RFs of inhibitory cells in the Gramian decomposition
When the Gramian matrix Φ T Φ is used to represent the recurrent inhibition directly (Fig. 2B), the RFs of the interneurons are columns of ΦA, where columns of A are the responses of principal cells to sparse dots stimuli. It is straightforward to infer from Eq. (S1) that the RFs of the interneurons are dots (simulation in Fig. 2B) since ΦA is the identity matrix.

RFs of inhibitory cells in low-rank decomposition
Denote the SVD of Φ as to scaling constants. The RFs of the inhibitory neurons implementing the low-rank part of the connectivity matrix can thus be approximated up to a scaling constant by: In words, this means that the RFs of these interneurons can be approximated by the (weighted) singular vectors of the dictionary Φ. Since the dictionary is learned from natural scenes, the singular vectors resemble the PCA components learned from natural scenes, which are known to be sinusoidal [3]. The simulation confirms this (Fig. 5A).

Feedforward inhibition
While the main text addressed the role of inhibition in the recurrent cortical connections, similar considerations arise when modeling the feedforward inhibition prevalent in the thalamocortical system [4]. The classical push-pull feedforward model [5] proposed the existence of tuned feedforward inhibitory neurons with RFs that "mirror" those of the excitatory neurons and supply disynaptic hyper-polarization to the principal cells in layer 4. To account for the discrepancy in the numbers of inhibitory and excitatory cells, it was suggested that these inhibitory neurons are multiplexed (i.e. a single inhibitory neuron supplies inputs to multiple excitatory neurons [6]), though there is uncertainty in the potential implementations of this multiplexing. Assuming that the interneurons mediating the feedforward inhibition is a separate population from the recurrent inhibition population, we show that multiplexing can be implemented naturally when the representation is overcomplete (Fig. S1). Specifically we rewrite the feedforward transformation (Φ T ) in Eq. (16) as follows: We can interpret Ω as the connectivity matrix from the inputs to the feedforward inhibitory interneurons and Ψ as the connectivity matrix from the interneurons to the principal cells. Note that the dimension of Ω can be arbitrarily assigned as long as the decomposition holds. A simple choice is to make Ω a full-rank dictionary matrix (N × N ) with Gabor-like elements. As a result when the principal cell representation is overcomplete (Ψ : M × N with M > N ), fewer feedforward interneurons than principal cells are needed. Furthermore, the RFs of the feedforward interneurons are Gabor-like, similar to those in the classical push-pull model [6]. Note that Ψ is uniquely defined given Φ and Ω, because in this case Ω −1 exists .

Global inhibition
While many implementations of inhibitory interneurons are discussed in the main text, an alternative strategy to improve the E/I ratio is to use a global inhibition approach. In the network of Eq. (4), we can offset the baseline excitatory synaptic weights by a constant c > 1 to make all of the individual synaptic weights positive: (c − φ j , φ i ) > 0, ∀j. With this offset to the individual weights, the equivalent dynamics model becomes: where we now have a single non-orientation tuned inhibitory cell (Fig. S2) that sums over the activity of all principal cells. In biological reality global inhibition is unlikely to manifest in this manner, if at all. A more likely scenario is that this type of interneuron inhibits only a local population of principal cells. This may in fact contribute to diverse orientation tunings of the inhibitory neurons, depending on the local distribution of principal cell orientation tuning properties (e.g. location on the pinwheel). We note several additional features of this global inhibition implementation of sparse coding. First, the global inhibition rises and falls with the overall excitatory activity in the network. This implies that the model inhibitory neurons have inherently higher firing rates, reminiscent of one class of inhibitory neurons in the cortex that are fast-spiking [7]. Second, the network equation indicates that excitatory cells only connect to those sharing overlapping RFs, reminiscent of the connectivity pattern seen in layer 2/3 where cells with overlapping inputs share connections [8,9], while inhibitory interneurons form dense connections with all neighboring excitatory neurons, similar to the pattern seen in physiology [10]. Third, the excitatory synaptic weights may result from combined Hebbian and anti-Hebbian learning with rates that depend on the overlap of the RFs. Functionally, the connection pattern in Eq. (S8) can be interpreted as a way for excitatory connections to share responsibilities for representing inhibition by shifting the baseline activity to a non-zero value. The cost of such an implementation is that the excitatory population has a higher gain, which may potentially induce a higher sensitivity to noise.