Generative model (decoder)

We assume that the brain holds an internal generative model of the environment (Fig 1, left). This model is specified by the joint probability of the neural activity of N neurons, r = {r₁, …, r_N}, and a scalar sensory stimulus, x, p_ψ(r, x), where ψ denotes a set of parameters. The neural activity is viewed as a latent variable, sampled from a prior distribution, p_ψ(r). Given the neural activity, or latent state, a ‘decoder’ maps it to a distribution over stimuli, p_ψ(x|r). This process specifies likely sensory experiences corresponding to a neural activity pattern, according to the generative model. We assume that the same generative process, which outputs a simulated quantity, x, is used by the brain when, e.g., producing an estimate in the context of a behavioral task.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Model architecture.

Left: generative model. In the generative model (purple), a neural activity pattern (white and black circles), sampled from the prior distribution (Ising model), p_ψ(r), is mapped by the decoder to a probability distribution over stimuli, p_ψ(x|r) (dashed purple curve), parametrized by functions of the activity patterns (here, mean and variance of a Gaussian distribution). The sum of the decoder output distributions, weighted by the prior over neural activity, defines the generative distribution (solid purple curve). The objective of the generative model is to produce stimuli according to the stimulus distribution in the environment, π(x) (green curve). In order to do so, the posterior distribution over neural activity, given external stimuli, is approximated by an encoder, q_θ(r|x) (right, blue). Neurons emit spikes according to bell-shaped tuning curves (grey curves) in response to a stimulus, x (green dot), drawn from the stimulus distribution π(x) (green curve). The population response consists in a binary neural activity pattern, r (white and black circles). The two models are trained such as to match the generative and stimulus distributions; this objective is approximated by minimizing two loss functions. The distortion, D, maximizes the likelihood of the observed stimulus after the encoding-decoding process (upper dashed black arrow). The rate, R, minimizes the Kullback-Leibler divergence between the conditional encoding distribution and the marginal (prior) distribution of neural activity of the generative model (bottom dashed black arrow).

https://doi.org/10.1371/journal.pcbi.1012240.g001

We consider neural activity over short time intervals, such that each neuron either emits one spike or is silent; this assumption is valid in scenarios when sensory coding occurs on a short time and have been largely studied in the literature (see, e.g., work on maximum entropy models [36, 37] or generalized linear models [38].) The set of possible activity patterns is then the set of binary vectors, r = (r₁, r₂, …, r_N), where r_i ∈ {0, 1}; in what follows, the sum ∑_r ⋅ denotes the sum over these 2^N binary patterns. We model the prior distribution over neural activity as the maximum-entropy distribution constrained by the first- and second-order statistics, a model which has been proposed as a model of the distribution of activity in neural systems, e.g., in retina and in cortex [36]. In the case of binary patterns, this maximum-entropy distribution takes the form of an Ising model, or Boltzmann machine, (2) where h is the vector of biases, J is the matrix of couplings (with our choice of parametrization, the diagonal elements of J vanish), and Z = ∑_r exp(h^Tr + r^TJ r) is a normalization constant (also called partition function). Throughout the paper, we employ Eq (2) as the prior distribution of neural activity, which allows for correlations among neurons. The only exception is in S3 Fig, where we illustrate the consequences of choosing a less flexible distribution, i.e., a product of N independent Bernoulli distributions, obtained by setting to 0 all entries of the coupling matrix, J.

On the basis of experimental findings, it has been suggested that the brain encodes a probability distribution of the stimulus, rather than a simple point estimate [20, 39]. We follow this view by assuming that the generative model encodes a probability distribution over stimuli in a parametric form, with parameters obtained as flexible functions of the neural activity pattern. We consider two forms of generative distributions. First, we model the generative distribution as a simple Gaussian, by assuming that the decoder maps activity patterns to a mean and a variance, (3) we parameterize these functions as two-layer neural networks, and we denote by ϕ the set of weights and biases in the network. Second, to account for heavy-tail behavior of some natural stimulus distributions, we posit that the brain represents mean and variance in a logarithmic scale. Thus, we also consider a log-normal distribution whose mean and variance of the logarithm of the stimulus values are parametrized as two-layer neural networks, (4) The parameters of the generative distribution and of the prior, ψ = {ϕ, h, J}, constitute the full set of parameters of the generative model.

With this parametrization choices, for N neurons the distribution p_ψ(x) = ∑_r p_ψ(x|r)p_ψ(r) is a mixture of 2^N normal or log-normal distributions, and is thus highly flexible. (In the Gaussian case, it has been shown that a well-chosen Gaussian mixture can be used to approximate any smooth density function [40, 41].) We will denote the decoder as ‘Gaussian’ or ‘log-normal,’ depending on whether we use Eqs (3) or (4) as the decoding distribution, respectively. We note that the generative distribution is a member of the exponential family, whose natural parameters are non-linear functions of the latent variables. This represents a generalization of the classic Helmoltz machine, that takes into account higher-order sufficient statistics [14, 19].

Recognition model (encoder)

The generative model assumes a prior distribution over neural activity, p_ψ(r): typically neural activity changes as a function of external stimuli to encode information about the sensory world. The recognition model prescribes the way in which stimuli evoke (are mapped to) neural activity, through a conditional distribution q_θ(r|x). In the next section, we relate these two models by optimizing a loss function which takes both account.

We consider a population of N neurons which responds to a continuous scalar sensory stimulus, x, distributed according to a prior distribution, π(x) (Fig 1, right). In order to avoid confusion with the prior distribution over neural activity patterns, p_ψ(r), defined above, we will refer to π(x) as the data, or stimulus, distribution. The encoding distribution is the conditional probability distribution over neural activity patterns given the stimulus, q_θ(r|x), where θ denotes the set of parameters. For the sake of simplicity, we assume that neurons spike independently conditional on the stimulus, such that . (We focus on conditionally independent neural activity because modeling correlations would require additional parameters and further assumptions about their structure (e.g., related to the connectivity that gives rise to the neural activity). As it has been extensively documented in the literature, the same coding performance can be achieved with different configurations of signal and noise correlations [42, 43]. Allowing for noise correlations would complicate the interpretation of our results.)

We consider a Bernoulli distribution, with the probability of spiking of a neuron obtained as (5) where f_i(x) is referred to as the neuron’s tuning curve. (This model can also be viewed as the limit of a Poisson model for spiking neurons with low rates [44, 45].) We parametrize tuning curves as Gaussian functions, a shape widely observed in early sensory areas, as (6) with c_i the preferred stimulus of neuron i, w_i the tuning width, and A_i the amplitude. Thus, the probability of spiking of a neuron can be written as , with and , the logistic function. (We note the analogy between this encoder and the classic recognition model of the Helmoltz machine, with the difference that here we employ a quadratic function of the stimulus as the argument of the sigmoid, while a linear function is employed in the classic Helmoltz machine.) In the canonical form of the exponential family, the resulting multivariate Bernoulli distribution can be written as (7) with η(x) = (η₁(x), …, η_N(x)) the vector of natural parameters and the set of parameters of the encoder.

Training objective

We introduced two models of joint distributions of neural activity and sensory stimulus, the generative model, p_ψ(x|r)p_ψ(r), and the recognition model, q_θ(r|x)π(x). Optimality requires that their joint distributions be matched. To approach this condition, we set the parameters of the generative and recognition model so as to minimize the Kullback-Leibler divergence between the two joint distributions, (8) This quantity, is an upper bound to the Kullback-Leibler divergence between the true stimulus distribution and the marginal generative distribution over stimuli: we have (9) where the second term on the right-hand side is non-negative by definition of the Kullback-Leibler divergence. Eq (9) implies that, by optimizing Eq (8), we minimize an upper bound on the divergence between the true distribution of stimuli and the generative distribution (first term on the right-hand side, purple and green curves in the top-left panel in Fig 1). Furthermore, this quantity, up to a constant term, is equal to the negative log-likelihood of the observed data under the generative model, 〈log p_ψ(x)〉_π(x), a quantity called ‘evidence’ and often used as a maximization objective in machine learning frameworks [23, 24] (see Discussion below and Eq (14)).

The quantity in Eq (8) can be rewritten as (10) where we recognize in the last two terms of the right-hand side the opposite of the so-called ‘evidence lower bound’ (ELBO). Thus, by ignoring the stimulus entropy that does not depend on the parameters, the problem in Eq (8) can be formulated as (11) Borrowing the nomenclature from rate-distortion theory, we call distortion the quantity (12) equal to the second term on the right-hand-side of Eq (10), and rate the quantity (13) equal to the third term. This framework goes by the name of variational autoencoder (VAE) [23, 24]. The encoder maps a stimulus sample, x, to a neural activity pattern, r, according to q_θ(r|x). The activity pattern corresponds to a realization of the latent variable in the generative model, and is mapped back (‘decoded’) to a distribution over stimuli according to p_ψ(x|r). As one typically does not have access to the true data distribution, but only to a set of samples, the average over π(x) is approximated by an empirical average over a set of N_trn samples, .

In machine learning, the derivation of the VAE is often presented from a different starting point. The generative model parameters are required to maximize the evidence, i.e., the log-likelihood of data samples under the generative process, 〈log p_ψ(x)〉_π(x), which is equivalent, up to a constant term corresponding the data entropy, to the first term on the right-hand side of Eq (9), (14) This, however, requires the inversion of the generative model, to obtain p_ψ(r|x), a typically intractable task. A common approach is to introduce a variational approximation of this posterior distribution, q_θ(r|x), and, by combining the identities in Eqs (14), (10) and (9), to maximize the quantity −(D + R) which constitutes a (variational) lower bound to the evidence. Yet an alternative derivation is obtained by applying Jensen’s inequality to the evidence rewritten in such a way to feature the variational approximation, 〈log p_ψ(x)〉_π(x) = 〈log ∫drq_θ(r|x)p_ψ(x, r)/q_θ(r|x)〉_π(x) [23–25].

Mean-squared error

We note that, due to the fact that the variance of the generative distribution depends on the neural responses, the distortion differs from the more usual mean-squared error (MSE) loss function of classical autoencoders, also commonly employed to measure the performance of neural codes. Indeed, in the case of a Gaussian decoder, the distortion function is written as (15) while the MSE, if we assume that the brain has access to the full (approximation) of the posterior distribution over stimuli, p_ψ(x|r), is obtained as (16) where we have used the maximum a posteriori (MAP) estimate, . In the case of a log-normal decoder, the distortion is obtained as (17) where the MAP estimate is given by . (Note that we quote instead the square root of the MSE (RMSE) in comparing the predictions of our model to the data on acoustic frequency-difference limens, in the section “Case study: neural encoding of acoustic frequencies.”).

If we assume that the brain makes use of the same generative model to produce behavioral responses (i.e., does not have access to an auxiliary (ideal) decoder), a straightforward operation is to draw samples, rather than computing statistics such as the mode. Thus, in the Results section, we also consider the MSE obtained when the stimulus estimate, , is sampled from the posterior distribution, , as (18)

We note that since our decoder is non-ideal and might be biased, the Fisher information of the encoder distribution, often used to quantify the encoding properties of neural populations as it provides a lower bound to the variance of any unbiased estimator [31], here yields a poor prediction of the coding performance. In S6 Fig, we compare the MSE with the inverse of the Fisher information, which in the case of independent Bernoulli neurons can be obtained as [10] (19) with η_j(x) defined as in Eq (7).

Constrained optimization and connection with efficient coding

It is a known issue in the VAE literature that, when the generative distribution is flexible given the data distribution (meaning that p_ψ(x|r) has enough degrees of freedom to approximate π(x)), the ELBO optimization problem exhibits multiple solutions. Optimization algorithms based on stochastic gradient descent are biased towards solutions with low rate and high distortion, a phenomenon which goes by the name of posterior collapse [28, 46]. In the extreme case, the model relies entirely on the power of the decoder and ignores the latent variables altogether: all realizations of the latent variables are mapped to the data distribution, p_ψ(x|r) ≈ π(x), and, consequently, all stimuli are mapped to the same representation, q_θ(r|x) ≈ p_ψ(r).

We overcome this issue by addressing a related, constrained optimization problem. We minimize the distortion subject to a maximum, or ‘target,’ value of the rate, : (20) The set of parameters {θ, ψ} that satisfy the constraint is called feasible set. By writing the associated Lagrangian function with multiplier β ≥ 0, we have that (21) Solutions of Eq (20) can thus be found as solutions to the ‘minimax’ problem, (22) The Lagrangian has a form similar to that of the negative ELBO, with an additional β factor multiplying the rate; this framework was presented as an extension of the classical VAE, with the aim of obtaining disentangled latent representations, in Refs. [28, 47].

Before addressing the optimization problem, we note that the two terms contributing to the ELBO are related to the mutual information of stimuli and neural responses of the encoder, (23) through the bounds (24) where H(π) is the entropy of the stimulus distribution [28]. The two inequalities arise because in the variational approximation the posterior over stimuli, p_ψ(x|r), replaces q_θ(x|r), and the prior over activity patterns, p_ψ(r), replaces q_θ(r), respectively. Since we are considering continuous stimuli, H is a differential entropy, and is thus defined up to a constant, and D can take negative values. Below, we will illustrate the properties of the generative model also through the D_KL divergence between the stimulus and the marginal generative distribution, i.e., the first term on the right-hand side of Eq (9), which is non-negative.

Eq (24) has two important consequences. First, it allows us to interpret the problem in Eq (20) as an efficient coding problem, where the objective is to maximize a lower bound to the mutual information, H − D, subject to a bound on the neural resources, . Contrary to the classical efficient coding literature, in which a metabolic constraint is imposed by hand, here it results from the original formulation of the problem as optimization of the ELBO, and it is affected by the assumptions made on the generative model (more specifically, on the prior distribution). The rate is minimized when the stimulus has little impact on the distribution of neural responses. A metabolic constraint is similarly bound to disfavor large changes in neural activity induced by stimuli. For this reason, the rate can interpreted as an abstract form of metabolic constraint.

Second, it prescribes limiting values to the solutions of Eq (20). If the variational distributions, p_ψ(r) and p_ψ(x|r), are flexible enough to approximate q_θ(r) and q_θ(x|r), we can achieve both inequalities, and we have . Along this line in the rate-distortion plane, the negative ELBO achieves its minimum value, equal to the stimulus entropy. In what follows, we call ‘optimal’ the solutions of Eq (20) and ‘lowest limiting values’ the values on the line of ideal solutions, .

We address the minimax problem of Eq (22) numerically through a two-timescales, alternated stochastic gradient descent-ascent, Alg. 1. We denote by {θ*, ψ*, β*} the optimal parameters. It is possible to show under some assumptions that (i) the parameter set {θ*, ψ*} is a solution of the problem defined in Eq (20); (ii) if β* > 0, the constraint on the rate is satisfied as an equality, , and β* is the negative slope of the curve of minimum distortion as a function of the target rate, ; (iii) if β* = 1, then the parameters {θ*, ψ*} maximize the ELBO. We report the conditions under which these statements are true, and their proofs, in the section S1 Appendix.

Algorithm 1 Two-timescale optimization algorithm.

1: Inputs: target rate dataset

2: Initialize: β = 1, encoder/decoder parameters= {θ_i, ψ_i}

3: while convergence do

4:  Define β-ELBO: L_β = D + βR

5:  for batch in do

6:   Update parameters: (θ, ψ) ← Adam (∇_θL_β(batch), ∇_ψL_β(batch))

7:  end for

8:  

9: end while

10: return

Numerical optimization and related computations

Numerical simulations are carried out using PyTorch. We solve the optimization problem in Eq (22) through stochastic gradient descent on the loss on a training dataset with N_trn samples from π(x). Except for the results shown in the section “Generalization and role of the rate as regularizer”, we used N_trn = 2000, divided in minibatches of size 128, with the Adam optimizer [48] with learning rate equal to 10⁻⁴ and otherwise standard hyperparameters. The learning rate for β, η_β, is set to 0.1. The training is iterated over multiple passes over the data (epochs) with a maximum of 5000 epochs and it is stopped when the training loss running average remains unchanged (with a tolerance of 10⁻⁵) for 100 consecutive epochs. The properties of the model are then quantified and illustrated using another dataset of N_tst = 2000 samples from π(x). Except for the results shown in in the section “Generalization and role of the rate as regularizer”, due to the volume of the training set, the metrics (e.g., distortion, rate, MSE) recorded using the training and the test set are indistinguishable, and we refer to them without making the distinction explicit. In the section “Generalization and role of the rate as regularizer” we explore the issue of generalization, and we use different values of N_trn; we adapt the size of the minibatch to keep it approximately in proportion to the training set size. Here, the metrics recorded on the test set deviate from those obtained from the training set, and we differentiate them by using the ‘trn’ and ‘tst’ suffixes.

The parameters are initialized as follows. The preferred positions, c_i, are initialized as the centroids obtained by applying a k-means clustering algorithm (with k = N) to the set of stimuli in the dataset. Tuning widths are initialized by setting w_i = |c_i − c_j|, with c_j the closest preferred position to c_i, and the amplitude is set equal to 1, corresponding to a maximum probability of spiking of 0.5. Random noise of small variance is applied to the initial value of the parameters. The biases of the prior, h, are set equal to 1. The coupling matrix, J, is sampled from a Wishart distribution with n = p = N degrees of freedom, with variance equal to 1/N and diagonal elements set to 0. The weights of the decoder neural network are initialized with the standard Kaiming initialization, corresponding to a uniform distribution between a and −a, with , and n_l−1 the size of the input layer; the biases are set to 0. The results are averaged over 16 network initializations. An example of the evolution of D, R, and β during training is illustrated in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example of training.

, N = 12, (A) Evolution of negative ELBO, and the two terms, D and R, with training epochs. Plot in log-log scale. (B) Joint evolution of R and D in the rate-distortion plane, colored according to the epoch (increasing from blue to yellow, colors in logarithmic scale). (C) Evolution of β during training.

https://doi.org/10.1371/journal.pcbi.1012240.g002

We illustrate results for N small enough so that it be possible to compute explicitly the sums over activity patterns appearing in the loss function. This also allows us to explore regimes in which the information is compressed in the activity of a finite population of neurons. In S2 Appendix, we discuss the numerical issues encountered when the population size is large, and we mention possible solutions.

For the results shown in the sections “Optimal allocation of neural resources and coding performance” and “Case study: neural encoding of acoustic frequencies”, we use the function curve_fit from the scipy package to fit the MSE, the neural density, and the tuning width as functions of the stimulus probability with power laws. Concretely, we log-transform the data, and we fit a linear model, log f(x) = A − γ log π(x), with A and γ as parameters. The MSE as a function of x is obtained by averaging across 16 initializations of the model parameters. We computed the neural density by applying a Kernel density estimate to the centers of the tuning curves. In order to match the scale of the predicted error, ε(f), to the scale of the experimental data on frequency-difference limens, y(f), we found the parameter a which minimized the squared distance between the logarithm of the data and the logarithm of the mean error, across all data points, .

Degeneracy of optimal solutions

We begin by illustrating two alternative solutions of the ELBO optimization problem, Eq (11), characterized by different contributions of the two terms, D and R. We first consider the simple, but instructive, case of a Gaussian distribution over stimuli, , together with a Gaussian decoder. In order to minimize the rate, a possible solution is to set the parameters of the encoder so as to map all stimuli to the same distribution over neural activity patterns which in turn approximates the prior distribution, q_θ(r|x) ≈ p_ψ(r). This is achieved by neurons with low selectivity, i.e., with broad and overlapping tuning curves (Fig 3A, top). Despite the uninformative neural representation, a perfectly accurate generative model is obtained (in this special, Gaussian case) by mapping all activity patterns to the parameters of the data distribution, μ_ψ(r) = μ_p and for all r; in this way, the generative distribution becomes independent from the neural activity, p_ψ(x|r) ≈ π(x) (Fig 3B, top). The rate term becomes negligible and the distortion equal to the stimulus entropy, thereby satisfying the leftward inequality in Eq (24). The sum of the two terms is equal to the stimulus entropy, the lower bound of the negative ELBO; the neural representation, however, retains no information about the stimulus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Qualitatively different optimal configurations.

In all simulations, N = 10 and . Top row: high-distortion, low-rate solution. Bottom row: low-distortion, high-rate solution. (A) Bell-shaped tuning curves of the encoder (probability of neuron i to emit a spike, as a function of x). (B) Comparison between the stimulus distribution, π(x) (green curve), and the generative distribution, p_ψ(x) = ∑_r p_ψ(x|r)p_ψ(r) (purple curve). (C) Numerical values of the ELBO, and the distortion and rate terms.

https://doi.org/10.1371/journal.pcbi.1012240.g003

At the opposite extreme, it is possible to minimize the distortion by learning an injective encoding map that associates distinct stimuli to distinct activity patterns. In our framework, this is achieved by narrow and non-overlapping tuning curves that tile the stimulus space (Fig 3A, bottom). For a given encoding distribution, the optimal prior distribution which minimizes the rate, Eq (13), is equal to the marginal encoding distribution, (26) (See Ref. [49] for an application of this optimal prior in the context of VAEs.) If the encoding distribution is different for each stimulus, the rate term does not vanish, but, numerically, we find that the parameters of the prior can still be set so as to approximate Eq (26), achieving the rightward inequality in Eq (24). Furthermore, the decoder then maps each activity pattern to a narrow Gaussian distribution over stimuli, so as to suppress the distortion to negligible values. As a consequence, the negative ELBO again achieves its lower bound, and it is possible to obtain a generative model that approximates closely the stimulus distribution, though less smoothly (Fig 3B, bottom).

Although these two solutions yield comparable values of the ELBO (Fig 3C) and equally accurate generative models, the corresponding neural representations are utterly different. This case is special and contrived, because the conditional generative distribution has the same functional form as the stimulus distribution, and thus a perfect generative model is obtained even when it ignores the latent variables. However, the reasoning extends to more complex cases, and the choice of the forms of the decoding distribution and the prior determines the ability of the system to optimize the ELBO in different ways [28]. In order to achieve a small distortion at low rates, the generative distribution must be flexible enough to approximate the data distribution even when the latent variables carry little information about the stimulus. Conversely, prior distributions which can fit marginal encoding distributions in which each data point is mapped precisely to a realization of the latent variables, achieve a low value of the rate for small distortion. Indeed, as we show next, we observe the existence of multiple solutions of the ELBO optimization problem for more complex stimulus distributions.

Analysis of the family of optimal solutions

We explore systematically the space of solutions which optimize the ELBO, by minimizing the distortion subject to a constraint on the maximum (‘target’) value of the rate, , a formulation which yields a generalized objective function (Eq (22)) with a factor β that weighs the rate term (see Materials and methods). The value of is an upper bound to the mutual information between stimulus and neural response; it thereby imposes a degree of ‘compression’ of the information in the encoding process. We illustrate results for the simple, yet non-trivial, choice of a log-normal stimulus distribution and Gaussian decoder, which exhibits a similar degeneracy as the simple case described above, in spite of the discrepancy between the stimulus and generative distributions (S1 Fig). Furthermore, the degree of degeneracy of solutions is comparable to the case in which a log-normal decoder is used instead of Gaussian one (S2 Fig). Similar observations hold for other distributions as well: in S4 Fig we illustrate the case of a multimodal distribution.

Each solution is associated with a point in the rate-distortion plane. By varying the value of , we trace the curve of the optimal distortion as a function of the target rate (Fig 4A). We focus on the range of values of resulting in β* = 1, for which and the corresponding solutions also yield an optimal value of the ELBO (shaded grey area). These solutions fall on the line of lowest limiting values D = H(π) − R, with H(π) the stimulus entropy, such that both inequalities in Eq (24) are achieved; as a result, the mutual information is equal to (Fig 4A, inset). Deviations from this line appear for extreme values of the target rate. On the one hand, as the stimulus and the generative distributions do not belong to the same parametric family, it is not possible to achieve the limiting value of the distortion with (this can be achieve by using a log-normal decoder, S2 Fig). On the other hand, for sufficiently large , the distortion stops decreasing and saturates; this occurs when the tuning curves are as narrow as possible while still tiling the stimulus space (Fig 4B, bottom). The distortion can be further decreased by increasing the number of available activity patterns, which depends on the population size (Fig 5A). By increasing the number of neurons, we also increase the number of tuning-curve arrangements which correspond to an optimal model, i.e., the number of degenerate solutions. This degeneracy is stronger at lower values of the rate, implying that the lowest limiting value of distortion can be achieved even with small population sizes. This is because, once the critical size needed to achieve the lowest limiting value of the distortion is reached, adding neurons does not raise the informational content of the population activity. When the rate is large, instead, the lowest limiting value can be achieved only in large enough populations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Characterization of the optimal solutions as functions of the target rate.

In all simulations, N = 12, . Solid curves illustrate the mean across different initializations and shaded regions correspond to one standard deviation. (A) Solutions of the ELBO optimization problem as a function of target rate, (blue curve), and theoretical optimum, (black curve), in the rate-distortion plane. Values of where the solutions coincide with the theoretical optimum (grey region). Solutions depart from the optimal line when the rate is very low (poor generative model) or very high (saturated distortion). Inset: mutual information between stimuli and neural responses as a function of . (B) Kullback-Leibler divergence between the stimulus and the generative distributions, as a function of . (C) Optimal tuning curves for different values of . Each dot represents a neuron: the position on the y-axis corresponds to its preferred stimulus, the size of the dot is proportional to the tuning width, and the color refers to the amplitude (see legend). Tuning curve parameters are averaged across 16 initializations, ordering the neurons as a function of their preferred position. The curve on the right illustrates the data distribution, π(x). (D) Entropy of the prior distribution over neural activity, p_ψ(r), as a function of . Insets show two configurations of the coupling matrices, with rows ordered according to the neurons’ preferred stimuli, and coupling strengths colored according to the legend. (E) MSE in the stimulus estimate, obtained as the MAP (blue curve, scale on the left y-axis), or from samples (orange curve, scale on the right y-axis), as a function of . Inset: MSE (MAP) as a function of the average tuning width.

https://doi.org/10.1371/journal.pcbi.1012240.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Dependence of the results on the population size.

Solid curves illustrate the mean across different initializations and shaded regions correspond to one standard deviation. Same simulations as in Fig 4, with different values of the population size. (A) Optimal solutions (blue curves), , for different population sizes, N, (legend in panel B) and theoretical bound (black curve), , in the rate-distortion plane. (B) Kullback-Leibler divergence between the stimulus and the generative distributions, as a function of . (C) MSE in the stimulus estimate, obtained as the MAP, as a function of .

https://doi.org/10.1371/journal.pcbi.1012240.g005

The quality of the generative model is quantified by the D_KL divergence between the generative distribution, p_ψ(x), and the stimulus distribution, π(x); it is negligible for all values of in the region of interest (Fig 4B). (We recall that the ELBO, up to a constant, is a lower bound to this quantity, and the gap is the D_KL divergence between the true and the approximate posterior distribution over neural activity, Eqs (9) and (10).) The U-shape is due to the jaggedness of the generative model at high values of (see next section), which is attenuated as the population sizes increases (Fig 5B).

Different values of also result in different encoders, corresponding to different arrangements of the tuning curves (Fig 4C). For small values of , tuning curves are broad and the spacing between preferred positions is small, causing large overlaps: different stimuli are mapped to similar distributions over neural activity patterns. Moreover, they are characterized by low amplitudes and, thus, higher stochasticity; indeed, stochastic neurons yield compressed representations [50]. Increasing causes noise to be suppressed through an increase in the amplitude, and narrower and more distributed tuning curves.

The solutions also differ in the structure of the prior over neural activity, p_ψ(r) (Fig 4D, insets). In the regime in which the decoder ignores the latent variables, i.e., p_ψ(x|r) ≈ p_ψ(x), the prior, p_ψ(r), is unstructured, and the couplings, J, are weak. By contrast, when is large, the structure of the stimulus distribution affects the coupling matrix in the prior, inducing coupling strengths that depend on the distances between the neurons’ preferred positions. (The capacity of the model to reach the lowest limiting value of the distortion for a large range of target rates is precisely due to the prior being very flexible and able to capture correlations between neural activity. The flexibility of the model is reduced when the couplings in the prior are set to 0 (S3 Fig).) As the coupling strengths increase, the entropy of the prior distribution decreases (Fig 4D).

Finally, we characterize the decoding properties in terms of a quantity commonly used in perceptual experiments and theoretical analyses: the mean-squared error (MSE) in the stimulus estimate. We approximate the maximum a posteriori (MAP) estimate as the mode of the decoding distribution (see Materials and methods). In the absence of an auxiliary, ideal estimator, the brain can produce estimate by sampling from the decoder; we thus also consider the MSE in the limiting case in which the stimulus estimate is evaluated as a single sample from the posterior distribution, . In Materials and methods, we compute the two corresponding functional forms, (see Eqs (16)–(18)), which differ by a term equal to the posterior variance in the case of Gaussian decoder. We note that our decoder is not ideal, especially at low rates, and might be biased; it does not saturate the Cramer-Rao bound. As a consequence, as in the cases of extremely noisy neurons and complex encoding schemes [51–53], the Fisher information of the encoding distribution yields a poor estimate of the decoding accuracy (see Materials and methods, S6 Fig).

As expected from the behavior of the mutual information between stimuli and neural responses, the decoding performance of the system increases as a function of , with a similar qualitative behavior of the error in the two estimation schemes (Fig 4E). But it is worth examining the behavior quantitatively. In both schemes, the MSE does not decrease linearly with , but rather exhibits a rapid decrease followed by a slower one; the quantitative value at high rates depends on the population size (Fig 5C). In particular, the system achieves comparable decoding performances for a broad range of values of the tuning width (Fig 4E, inset).

Generalization and the role of the rate as regularizer

The result illustrated in Fig 4B implies that intermediate representations are preferred to representations with extremely narrow tuning curves, in that they yield a smoother approximation of the stimulus distribution. This observation suggests that, in the case of a limited volume of data, the rate might serve as regularizer which prevents the model from overfitting the data. We thus expect that limiting the rate benefits generalization properties, i.e., suppresses errors due to the stochasticity of the training dataset.

We test this hypothesis by training the model on a limited volume of data, and comparing the resulting performance to the asymptotic limit of a large training set explored in the previous section (Fig 6). We find that, at intermediate values of target rate (grey region in Fig 4), the distortion in the training set is always smaller than the distortion in the asymptotic limit, and the more so the smaller the training set, suggesting that the model overfits (Fig 6A, top). Moreoever, this trend is enhanced for larger values of the rate. This is further confirmed by an analysis of the distortion in a test set (a large number of stimulus samples), which is larger than in the asymptotic limit, yielding a ‘generalization gap’ that depends on the value of the target rate (Fig 6A, bottom). The rate, instead, exhibits no difference between the training and the test set (Fig 6A, insets).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Characterization of optimal solutions as functions of training set size.

In all simulations, N = 12, and . Solid curves represent the mean across different initializations, and shaded regions correspond to one standard deviation. The legend in panel A serves as a legend for all panels. (A) Solutions of the ELBO optimization problem as functions of the target rate, for the training set (top) and for the test set (bottom). Top: distortion, , and rate, (inset), for the training set as a function of the target rate, for different sizes of the training set, colored according to the legend. For smaller training sets, at higher rates the model tends to overfit the data, resulting in a lower training distortion than optimal (red line, large training set, same data as in Fig 4). Bottom: distortion, , and rate, (inset), for the test set as functions of the target rate, for different sizes of the training set. For smaller training sets, at higher rates the model does not generalize to unseen samples, resulting in a large distortion. (B) Left: Kullback-Leibler divergence between the stimulus and the generative distributions, as a function of , for different sizes of the training set. At higher rates, the generative model fits poorly the stimulus distribution. Right: examples of comparisons between stimulus (green line) and generative distribution (red and orange line) at low (top) and high (bottom) rates, for different sizes of the training set, N_trn = 100 and N_trn = 2000, colored according to the legend as in panel A. (C) Tuning width, w_i, as a function of the location of a preferred stimulus, c_i (dots), at low (left) and high (right) rates, for different sizes of the training set, N_trn = 100 and N_trn = 1000. The grey curve represents the stimulus distribution, π(x). (D) MSE in the stimulus estimate, obtained as the MAP, as a function of , for different sizes of the training set.

https://doi.org/10.1371/journal.pcbi.1012240.g006

The generalization gap results in a poorer fit of the stimulus distribution by the generative model (Fig 6B). When the number of training samples is small, at high rates the location of the peaks in the generative distribution (see also Fig 3B, bottom) strongly depend on the stimuli present in the training set; this leads to a poor approximation of the stimulus distribution (Fig 6B, bottom). Low rates, instead, induce a smoother generative distribution. They also reduce the dependence of the quality of fit on the training set size, and enhance generalization (Fig 6B, left). Thus, the target rate effectively acts as regularizer, that prevents overfitting and favors generalization. A limitation in the volume of training data also affects the encoder and decoder properties: tuning curves become narrower at higher rates (Fig 6C), and the MSE in the stimulus estimate is much more sensitive to the volume of the training set at high rates (Fig 6D). We verify the consistency of these observations across other stimulus distributions by examining the case of a multimodal distribution (S5 Fig). Jointly, the results illustrated in Figs 4 and 6 point to the benefit of intermediate values of the rate that regularize the model. In this regime, the encoder is characterized by broad tuning curves, and the decoder achieves low coding error and generalizes to unseen data (Figs 4E and 6D).

Optimal allocation of neural resources and coding performance

The classical efficient coding hypothesis prescribes an allocation of neural resources as a function of the stimulus distribution: more frequent stimuli are represented with higher precision. This has been proposed as an explanation of a number of measurements of perceptual accuracy and behavioral bias [10, 54, 55]. We investigate, in our model, the relations between stimulus distribution, the use of neural resources (tuning curves), and the coding performance, and how each these vary with . We emphasize that the functional form of the stimulus distribution affects these relations, through its interplay with the functional form not only of the encoder (as in the classical efficient coding framework), but also of the generative distribution. In order to make statements about the typical behavior of the system, we average our results over different random initializations of the parameters; single solutions might deviate from the average behavior due to the small number of neurons and the high dimensionality of the parameter space. We illustrate results for the non-trivial case of a log-normal distribution of stimuli and a Gaussian decoder; in S7 and S8 Figs, we report results obtained when the stimulus and decoding distributions belong to the same parametric family (Gaussian and log-normal, respectively). Our conclusions can be compared with results from previous studies. In particular, we invoke the analytical results derived in Ref. [10] for a similar population coding model; in Sec. S3 Appendix, we provide an alternative derivation of these results and we comment on the main differences with our model. Here, we note that our results are obtained by considering a regime of strong compression of the information (small population sizes), while previous studies focused on the asymptotic regime with N → ∞.

As illustrated in Fig 4C, the target rate affects the neural density, i.e., the number of neurons with preferred stimuli within a given stimulus window. In previous work, maximizing the mutual information required that the neural density be proportional to the stimulus density, d(x) ∝ π(x) [10, 31, 56]. In our case, the range of possible behaviors is richer, especially when the stimulus distribution is non-trivial (i.e., it does not have the same functional form as that of the generative distribution). At low rates, the location of maximum density might be different from the mode of the stimulus distribution, depending on the interplay between the generative and the stimulus distributions (Fig 7A, S7(A) and S8(A) Figs). The neural density becomes more sensitive to the stimulus distribution for large values of : a power law functional form, , yields a good agreement with our numerical results, with an exponent, γ_d, close to 1/2 (Fig 7A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Optimal allocation of neural resources.

In all simulations, N = 12 and . Results are illustrated for regions of the stimulus space where the coding performance is sufficiently high, defined as the region where the MSE is lower than the variance of the stimulus distribution. Below, we mention exponents of the power law fit when the variance explained is larger then a threshold, R² ≥ 0.7. (A) Neural density as a function x (dashed curves) and power-law fits (solid curves, R² = (0.21, 0.83, 0.95), γ_d = (−, 0.43, 0.62)), for three values of (low, intermediate, and high); the grey curve illustrates the stimulus distribution. The density is computed by applying kernel density estimation to the set of the preferred positions of the neurons. (B) Tuning width, w_i, as a function of preferred stimuli, c_i (dots), and power-law fits (solid curves, R² = (0.78, 0.42, 0.82), γ_w = (1.15, −, 0.7)) for three values of ; the grey curve illustrates the stimulus distribution. (C) Tuning width, w_i, as a function of the neural density, d(c_i), for three values of ; Pearson correlation coefficient ρ = (−0.74, −0.66, −0.79).

https://doi.org/10.1371/journal.pcbi.1012240.g007

In Ref. [10, 57], analytical results were obtained by constraining the neural density and the tuning width relative to each other. This is equivalent to fixing the overlap between tuning curves, by imposing w(x) ∝ d⁻¹(x) ∝ π(x)⁻¹ (see Sec. S3 Appendix). In our case, the tuning width and neural density vary independently of each other, and the distribution of widths exhibits an intricate behavior at small values of (Fig 7B, S7(B) and S8(B) Figs). At large values of , the tuning width decreases for large values of the stimulus distribution, and its behavior is well described by a power law, . As a result, as increases, the inverse correlation between the neural density and the tuning width becomes sharper (Fig 7C and S7(C) Fig).

A consequence of the heterogeneous allocation of neural resources is a non-uniform coding performance across stimuli. Fig 8A shows that the MSE exhibits an inverse relation as a function of the stimulus distribution, with more frequent stimuli encoded more precisely. This is broadly consistent with previous studies [10, 58], which maximized the mutual information to obtain the expression (27) (Similar power-law behaviors, with different exponents, arise from different loss functions [10, 12, 59, 60].) Concretely, these power-law expressions were derived using the Fisher information, whose inverse is a lower bound to the variance of an unbiased estimator, and which can be related to the mutual information in some limits. Here, for all values of , the error is well described by a power law, , where the exponent changes as a function of , and depends on the choice of the decoder (Fig 8A, S7(D) and S8(D) Figs). In particular, we find that when the decoder and the generative distributions belong to the same parametric family, the dependence of the error on the stimulus distribution is stronger, characterized by larger values of γ_e. Finally, we illustrate how the configuration of the tuning curves affects the coding performance, by plotting the MSE as a function of the neural density and tuning width. We observe a correlation between high coding performance and regions of high neural density and narrow tuning widths (Fig 8B and 8C, S7(E), S7(F), S8(E) and S8(F) Figs).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Optimal allocation of coding performance.

Same numerical simulations as in Fig 4. (A) MSE (MAP estimate) as a function of x (dashed curves), and power-law fits (solid curves, R² = (0.98, 0.98, 0.76), γ_e = (0.87, 0.74, 0.59)), for three values of . (B),(C) MSE as a function of the neural density (B) and tuning width (C), for three values of ; Pearson correlation coefficient ρ_density = (−0.66, −0.96, −0.81), ρ_width = (0.36, 0.59, 0.70).

https://doi.org/10.1371/journal.pcbi.1012240.g008

To summarize, given our choice of the loss function, which constrains the encoding stage as a function of the decoding stage, we obtain a family of optimal neural representations. In weakly constrained systems (large values of ), we qualitatively recover previously derived relationships between tuning curves, stimulus distribution, and coding performance. (The difference in the numerical values of the exponents in the power laws can be explained by the differences between the models (see Sec. S3 Appendix), and depends on the similarity between the functional forms of the generative and the stimulus distributions. We note that, in previous work [10, 12, 59] the numerical value of the exponents also change as a function of the form of the loss function.) In systems with stringent information compression (small values of ), the optimal resource allocation exhibits a more intricate behavior that depends on the functional form of the stimulus distribution and the properties of the generative model.

Case study: Neural encoding of acoustic frequencies

We validate our theory on existing data by considering the empirical distribution of acoustic frequencies in the environment and relating it to behavioral performance. This distribution was obtained in Ref. [57] by fitting the power spectrum of recordings data, S(f), with a power-law, (28) with A = 2.4 × 10⁶, f₀ = 1.52 × 10³ and p = 2.61 (Fig 9A, inset). Since the stimulus distribution exhibits a heavy-tail, we test our model with both a Gaussian and a log-normal decoder. Despite exhibiting similar degeneracy in the space of solutions (Fig 4 and S2 Fig), the two decoders yield different quantitative predictions on the optimal allocation of neural resources (Figs 7 and 8, and S8 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Generative models for the distribution of acoustic frequencies.

In all simulations, N = 12. The decoder is either Gaussian (top row) or log-normal (bottom row). (A) Solutions of the optimization problem as a function of the target rate, (blue curve), in the rate-distortion plane. Inset: environmental distribution of acoustic frequencies, π(f), and generative model fit for two different values of the target rate, colored according to the legend. (B) Optimal tuning curves for different values of . Each dot represents a neuron: the position on the y-axis corresponds to its preferred stimulus, the size of the dot is proportional to the tuning width, and the color refers to the amplitude (see legend in Fig 4). The curve on the right illustrates the stimulus distribution, p(f). Insets show two examples. (C) Frequency discrimination as a function of acoustic frequency. Red markers are data points from three different subjects, data from Ref. [61]. Solid curves are the RMSE for three values of , scaled by a factor of , with variance explained R² = (0.42, 0.41, 0.66). (D)-(F) Same as panels (A)-(C) in the case of a log-normal decoder. In panel F, , with variance explained R² = (0.92, 0.81, 0.96). Solid curves illustrate the mean across different initializations and shaded regions correspond to one standard deviation.

https://doi.org/10.1371/journal.pcbi.1012240.g009

We observe that a broad range of values of results in comparable values of the ELBO and comparable generative model performances (Fig 9A and 9D). A log-normal decoder yields a better fit of the stimulus distribution as it is better suited than a Gaussian to capture the heavy tail in the stimulus distribution (Gaussian decoder D_KL ≈ 0.1, log-normal decoder, D_KL ≈ 0.04). As before, for increasing , solutions are characterized by an encoder with increasingly narrow tuning curves and the locations of preferred stimuli sensitive to stimulus probability (Fig 9B and 9E).

Finally, we test the prediction of our model regarding the dependence of the error on the stimulus value by comparing it to experimental data. We borrow experimental measurements of the so-called frequency-difference limens, the minimum detectable changes in the frequency of a sinusoidal sound wave, from Ref. [61]. We employ the square root of the MSE (RMSE) of the stimulus estimate as a proxy for perceptual resolution. Since the small number of neurons imposes a fundamental bound to the coding performance, we scale the RMSE by a constant factor, , which can be thought of as a population size gain, to allow for a comparison (see Materials and methods).

In the case of a Gaussian decoder, the functional form of the RMSE does not capture the behavior of the frequency-difference limens for any value of the target rate (Fig 9C). By using a log-normal decoder, instead, we obtain a faithful description of the frequency-difference limens for a broad range of values of (Fig 9F). These results show that, despite a large variability in the parameters of the encoder, as is commonly observed in biological systems, robust predictions in the perceptual domain can be obtained and are consistent with experimental data. In future work, it would be interesting to conduct a more systematic study on the aspects of the generative model, e.g., properties of the functional form of the decoder, that can be extracted from data.

VAE, efficient coding, and learning from examples

The VAE can be viewed as a generalized efficient coding framework. The encoder is jointly optimized with the decoder to maximize a variational approximation of the mutual information under a constraint on neural resources. However, unlike classical efficient coding frameworks in which the functional forms of the encoding accuracy and the resource constraint can be chosen with a certain degree of arbitrariness [10–12, 30, 58, 59], here they are constrained by a unique objective function that links them together. Thus encoding and decoding are captured in a unified approach [62]. In previous models, the form of the predictions, such as power laws governing the dependence of the coding performance on the stimulus probability density depended on the choice of the two terms in the loss function. Different combinations of alternative choices of these two terms can yield the same prediction [12, 59]. Here, instead, flexibility is achieved through the choice of the functional form of the generative model and the bound on the resource constraint. Furthermore, our model exhibits broad tuning curves for intermediate values of the target rate without further constraints (see section “Optimal tuning width” and S3 Appendix).

Another difference with the classic efficient coding framework is that, here, the decoder does not make use of the knowledge of the distribution of stimuli to perform: no ‘Bayesian inversion’ is needed. The benefit of an approximate Bayesian inversion (through a variational approximation of the ‘inverted encoder’), is that solutions can be learned on the basis of data samples [26, 27, 63]. This is an advantage of the VAE framework, as in typical natural situations the stimulus distribution is accessible only through observations.

VAEs in neuroscience: Related studies

VAEs are among the state of the art approaches to unsupervised learning, and in recent years they have been applied in different contexts in neuroscience to model neural responses. Several studies have considered neuroscience-inspired VAEs, in which the generative model was based on a decomposition of natural images into sparse combinations of linear features [64]. It was then paired with a powerful encoder, which models the sensory encoding process, and specific assumptions on the prior distribution of the latent variables, to obtain representations similar to the ones observed in the early visual pathway (in V1 and V2) [18, 26, 65]. In these models, the simplicity of the generative distribution prevented posterior collapse. We note that, in our case, we reverse this approach, by assuming a specific a simple and biologically motivated form of the encoder (a set of tuning curves), while we allow for a flexible decoder.

In the context of higher visual areas instead, more complex generative models were needed to capture neural representations [66]; to overcome the issue of posterior collapse, the authors used a loss function akin to the one in Eq (22), but the value of β was chosen by hand. In doing so, they obtained an empirical advantage in the semantic interpretability of the latent variables, at the cost of abandoning the requirement that the loss function be a bound to the log-likelihood. This, so-called, β-VAE approach was also employed in Ref. [67] to study optimal tuning curves in a population coding model of spiking neurons similar to ours. In this study, however, the population as a whole was constrained to emit one spike only, limiting the number of available activity patterns to N (the number of neurons). Moreover, the encoder and the decoder are not optimized independently; this choice prevented the emergence of multiple alternative neural representations in the β = 1 case. By varying β, the authors obtained neural representations which differed in the shape of the optimal tuning curves, but, since for β ≠ 1 the ELBO was not optimized, the resulting generative model was not accurate.

What causes the degeneracy of the optimal solutions?

Degeneracy in the space of solutions results from the flexibility of the generative model. In our case, the generative model is a mixture of distributions: we restrict our choice to two examples of parametric families, Gaussian and log-normal. Despite their simplicity, the generative model is equipped with high approximation capabilities. Indeed, the marginal distribution, p_ψ(x) = ∑_r p_ψ(x|r)p_ψ(r), is a mixture of distributions, and, in the case of Gaussian decoder, is a universal approximator of densities (i.e., a well-chosen Gaussian mixture can be used to approximate any smooth density function [40, 41]).

With a population of N binary neurons the generative model consists, in principle, of a mixture of 2^N different distributions. The association between each activity pattern and the mean and variance of mixture components is determined by the decoder, while the prior on the neural activity constrains how many distributions are effectively employed, as it governs the mixing factors. At low rates, the prior distribution is not constrained by the encoder and can exhibit high entropy. At low distortion, instead, latent variables are mapped to distinct output distributions. The prior distribution over latent variables is constrained to capture the correlations in neural activity imposed by narrow tuning curves; its entropy is thus suppressed, resulting in a limited number of non-vanishing mixing factors (and the performance of the model degrades if the prior is not flexible enough to capture these correlations, S3 Fig). Intermediate rates balance these two limits, yielding a generative model with a sufficient number of mixing factors and distinct output distributions, which reproduces the stimulus distribution and generalizes to unseen data [68–70].

Here, we focused on relatively simple, one-dimensional stimulus distributions. As the statistics of many natural features are dominated by low-frequency components (e.g., spatial frequencies in natural images), and if powerful decoders are to represent deep brain areas [18, 66], we expect degeneracy in the space of solutions even in the case of multi-dimensional stimuli. In future work, it would be interesting to pair powerful decoders with biologically inspired high-dimensional encoders, e.g., multidimensional tuning curves, and to characterize the degeneracy of the solutions. In the case of more complex stimuli, the rate will represent a more stringent condition in controlling the generalization properties of the model [69, 70].

Internal models and perception as inference

Our choice on the form of the decoder stems from the assumption that organisms interact with their environment with the use of internal models. These allow them to perform inference and make predictions. But what form do internal models take and what is their neural substrate? In previous studies [17, 21, 22, 71], internal models were defined by conditioning the probability of a stimulus, x, on the realization of a latent variable, z, through their joint distribution, p(z, x) = p(x|z)p(z). The latent variables were chosen so as to allow for a semantic interpretation, such as the presence or absence of a given image feature (e.g., Gabor patches or texture features [18]). Sensory areas were then assumed to compute a posterior distribution over the latent variable, q(z|x), and the neural activity was invoked as a way to represent this posterior distribution, either approximately from samples [17, 22, 72], or as parameters of a parametric distribution [19, 20].

We chose to define the generative model directly as a joint distribution of two random variables, p(r, x); r is the neural activity, while x is defined on the space of stimuli. Although we preserve the mathematical structure of generative models proposed in neuroscience (e.g., the Helmoltz machine and its more recents extensions [14, 19]), our interpretation is different. The neural activity itself plays the role of a latent representation of the stimulus, but it is not set, a priori, to some interpretable feature, such as the presence or the intensity of a Gabor filter in models involving natural images (as in Refs. [17, 22]). In order to constrain sensory areas, we assume the generative model to be implemented in downstream areas and we model its output with a flexible function, a neural network, which outputs a point estimate and an uncertainty about the value of the stimulus [39, 73]. This output corresponds to a perceptual representation of the stimulus in the brain, and can be related to behavioral measurements (as in Fig 9C).

Mathematically, the encoding distribution, q_θ(r|x), is obtained as a variational approximation of the posterior distribution of the generative model, p_ψ(r|x), as in previous work. This distribution, however, is defined on the space of neural activity patterns, and not on a set of abstract features. This choice has the drawback of the absence of a simple semantic interpretation of the latent features, but presents the advantage of a natural connection with an encoder based on properties of a neural system, e.g., a set of tuning curves and a model of neural noise. In the case of flexible generative models, different statistics of the latent variables turn out to be optimal. In this sense, the choice of the encoder, as well as the prior of the generative model, is useful to impose a structure on the characteristics of the neural representations.

Optimal tuning width

Our choice of encoding model allows us to compare our results with those of earlier studies that considered the optimal arrangement of neurons with bell-shaped tuning curves in the presence of non-uniform stimulus distributions [10, 31]. While for higher values of the target rate we recover the previously derived allocation of neural resources as a function of the stimulus distribution, the behavior for lower values of the target rate is more intricate, and depends on the specifics of the stimulus distribution. Thus, in our case, the constraint on neural resources has a stronger impact on their optimal allocation than, for example, in Ref. [10], where the bound on the mean activity in the population acts merely as a scaling factor, and the behavior of the tuning curves is more constrained. In particular, in Ref. [10] the tuning width was fixed a priori to be inversely proportional to the neural density, to enforce a fixed amount of overlap between tuning curves: it was not optimized. This choice was made to avoid a common issue in this type of calculations: in the case of a one-dimensional stimulus and in the asymptotic limit of infinitely many neurons, the maximization of the mutual information yields the pathological solution of infinitely narrow tuning curves [74, 75]. Metabolic constraints on the neural activity do not solve the issue, as narrow tuning curves can exhibit a moderate activity (as long as their amplitude is not too large).

In our framework, instead, the optimal tuning width and the amount of overlap between tuning curves are both optimized and vary as a function of . Moreover, a regime with intermediate values of the constraint, in which tuning curves are broad, exhibits both a smooth generative model (low D_KL divergence) and a low MSE (Fig 4B and 4E). Broad tuning curves are beneficial to obtain smooth generative models, while still allowing high for coding performance.

Interpretation of the resource constraint

The constraint in Eq (20) involves the divergence between the evoked neural activity and its marginal distribution according to the generative model. This formulation is different from usual metabolic constraints which are designed to account for the energetic cost of neural activity [8]. In general, the cost should increase with the magnitude of the perturbation induced by a stimulus. Our more abstract formulation of the cost satisfies this property.

In our case, the prior distribution is parametrized by the biases and couplings of an Ising model. As we have shown, there are multiple ways to achieve a statistically optimal internal model and to minimize the D_KL divergence between the two sides of Eq (26), which differ in the value of the rate. At low rates, Eq (26) is approximated by relying on the optimization of the encoder parameters which are set so as to make q_θ(r|x) similar to the prior for all stimuli; this then results in an unstructured coupling matrix in the prior distribution (Fig 4E, top). Conversely, at high rates, the encoder has a well defined structure which achieves a low distortion, and Eq (26) is approximated by optimizing the parameters of the prior and embedding the structure of the average posterior distribution in the connectivity matrix (Fig 4E, bottom). The value of the target rate can therefore be thought of as a cost of imposing structure in prior (spontaneous activity), through circuit properties. Thus, our model suggest an alternative normative principle to govern neural couplings as compared to information maximization, as proposed in Ref. [37].

A more concrete biological interpretation of the rate can be made by referring to the results obtained in Ref. [71], in which the prior distribution over latent variables of an internal model is related to the spontaneous neural activity. The authors start from the observation that in a well-calibrated internal model the prior equals the mean posterior, Eq (26) [15]. By comparing the average evoked activity to the spontaneous activity according to the D_KL divergence, the authors show that the two quantities become closer during development, and that this phenomenology is specific to naturalistic stimuli. This finding is then proposed as evidence of an internal model in primary visual cortex optimized for natural images, acquired gradually during development. In this picture, the prior distribution of the generative model is identified with the spontaneous neural activity; we note, however, that there is no a priori reason to expect this relationship.

We can also relate information-theoretic quantities to biophysical processes by invoking results from statistical physics. A recent study has shown that the magnitude of the response of a system to an external perturbation is bounded above by the Kullback-Leibler divergence between the probability distributions describing the perturbed and unperturbed system [76]. Thus, if we view the marginal distribution as describing the unperturbed state of the neural population, the rate term provides an upper bound on the magnitude of the response, i.e., the change in firing rate, of the neural population upon a stimulus presentation. In turn, this quantity is proportional to the spiking metabolic cost. Ultimately, one would like to derive formulations of the abstract, information-theoretic costs that govern joint encoder-decoder models from known microscopic biophysical processes.