2.1 Redundant information is necessary to solve the assignment problem

Here, we investigate the assignment problem that arises when each of N stimuli have representations that are split across two brain regions, R_X and R_Y—in combinatorial optimization, this is referred to as the balanced assignment problem [9]. We want to correctly integrate the distributed representations of these N stimuli, which we formalize as finding the correct one-to-one mapping between the representations in R_X and the representations in R_Y. To give an example: a dog and a cat will be represented in at least two distinct regions of the brain, an auditory region R_X and a visual region R_Y (Fig 1a). The set of representations in R_X, referred to as , will consist of representations of each of the animals’ vocalizations—similarly, in R_Y consists of representations of each animals’ visual features (see Methods in Definition of the representations for more details). Many behaviors rely on integrated information about both the visual features and vocalizations of the animals (for instance, when deciding which animal to approach, it is useful to know whether they are barking or meowing).

So, we want to choose the correct one-to-one map M between and (Fig 1b). We show that if and are independent, then all maps are equally likely (see Methods in Definition of assignment errors). In this case, assignment errors will occur with probability 1 − 1/N!. Thus, redundancy between the representations in R_X and R_Y is necessary to reliably solve the assignment problem. More concretely, the case where and are independent would be like the case where, in a working memory study similar to the ones discussed in the Introduction, we ask the participant to remember the color and orientation of a collection of bars, then cue them with a spatial frequency. Without additional instruction or extensive learning, the participant could do no better than chance—just as our system here could solve the assignment problem no more reliably than by guessing.

The required redundancy can take many forms. We will focus on the case where the redundancy manifests as a linear dependence between some of the features represented in R_X and some of the features represented in R_Y. In the working memory task example, this is similar to remembering both the color and orientation of each stimulus in an array. Then, being asked to report the color of the stimulus that had a particular orientation. Our analysis can also be extended to some kinds of nonlinear mappings between between the features in R_X and R_Y (see Fig B in S1 Text).

One specific kind of nonlinear redundancy between the two regions is prior information about which stimuli are more or less likely. For example, someone familiar with cats and dogs could infer that a meowing noise likely belongs to the cat they see in front of them, while the barking likely belongs to the dog. While such inferences are likely extremely important for behavior, they require the prior information to be learned over experience with these particular stimuli. We focus our analyses here on solving the assignment problem without this learning process (as for novel stimuli) and where such differences in prior likelihood are not available (e.g., assigning two different barks to two different novel dogs).

Finally, throughout this work we organize our discussion using the concept of distinct brain regions, by which we mean distinct populations of neurons that may share information with each other through anatomical connections. We primarily use this concept for simplicity of exposition. In general, all of the same principles apply when different features (or combinations of features) are represented in distinct subspaces of the same neural population activity. That is, the assignment problem will also need to be solved to integrate across these distinct subspaces, even though the subspaces do not correspond to distinct populations of neurons.

2.2 The assignment error rate depends on the precision of the common feature representation

Here, we study redundancy between R_X and R_Y that results from the representation of one or more of the same stimulus features (e.g., azimuthal position in both the visual and auditory systems, Fig 1c, left). These C common, or linearly related, features define a C-dimensional common feature space. Importantly, we do not yet make any assumptions about the representational format of these features—and while the common features are assumed to be the same (or, at least, correlated) across the two regions, the format of their representation need not be.

Then, following the combinatorial assignment problem, we show that the most likely mapping is the one that minimizes the sum squared distance between integrated pairs in this common feature space (Fig 1c, right: top matrix is the the correct mapping, M_C, bottom matrix is the incorrect mapping M_AE, the sum of distances between integrated pairs in the two mappings are shown below). If the estimates of the common features in both regions are exactly correct, then this sum will be zero—an exact matching (Fig 1c). However, in general, we assume that each feature represented in R_X is decoded with Gaussian-distributed local variance D_X (Fig 1d). Further, we show that assignment errors occur precisely when the representation of one object crosses over the representation of another object in one region but not the other (Fig 1d, red lines). For two objects at distance δ from each other in the common feature space space, the probability of an assignment error between those two stimuli is approximated by (1) where Q(x) is the cumulative density function for the standard normal distribution (Fig 1e, left)—for the full expression and derivation, see Methods in Probability of assignment errors.

The overall probability of an assignment error also relies on the probability that two objects are at distance δ from each other in the common space P_C(δ) (Fig 1e, right for C = 1) and on the number of stimuli N. Incorporating these, we show that the overall probability of an assignment error is upper bounded by, (2) For one overlapping stimulus feature (C = 1) and for stimuli that are uniformly distributed in feature space, we show that, (3) where s is the size of the feature space for the commonly represented feature. This approximation (dashed lines, Fig 1f) closely matches the empirical assignment error rate (solid lines) across different numbers of stimuli (different colors; see Methods in Assignment error rate approximation for C = 1 for a full derivation of this expression). Now, using this formalization, we show that the assignment error rate decreases with additional commonly represented features C > 1 in the next section. We also show that the assignment error rate increases if the features are represented with asymmetric variance across the two brain regions—i.e., if D_X ≠ D_Y (see Fig A in S1 Text).

2.3 Additional commonly represented features decrease the assignment error rate

While distinct sensory systems tend to have fixed amounts of overlapping information, such as a common estimate of azimuthal position across the auditory and visual systems (which has been shown to be crucial for single stimulus integration [21]), within a single sensory system information is distributed across multiple brain regions, and those brain regions can represent variable amounts of overlapping information about the stimuli. In our framework, we show that increasing the number of commonly represented stimulus features across the two brain regions C decreases the assignment error rate.

In particular, increasing the number of commonly represented features C increases the average distance between pairs of points in the common feature space (Fig 2a). This change is captured by changes in the p_C(δ) term in Eq 2. Increasing the number of commonly represented features moves the probability mass of this distribution away from zero (Fig 2b). The increase in the number of overlapping features from C to C + 1 leads to an orders of magnitude decrease of the assignment error rate at the mean distance (Fig 2c). As a result of this shift to the right in p_C(δ) and the fact that Eq 1 is strictly (and exponentially) decreasing in δ, the integral in Eq 2 will also be strictly decreasing as C is increased. Thus, increasing C decreases the overall probability of assignment errors when holding local variance constant (Fig 2d).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Increasing the number of commonly represented features decreases the assignment error rate, but increases the level of redundancy between the representations.

The color legend is the same throughout the plot, and provided in D. A Schematic showing that the distance between two points increases with the dimensionality of the space that they are in. B The distribution of distances between two uniformly distributed points in a space of one, two, and three dimensions. The distribution shifts to the right (toward larger distances) as the dimensionality of the space increases. This is true for points with any distribution that has non-zero variance. C At the average distance between two points (dashed lines), the probability of an assignment error decreases by orders of magnitude as the dimensionality of the commonly represented space increases, without changing the estimator variance for the representations D_XY = 10. D The overall assignment error rate also decreases by orders of magnitude as the dimensionality of the commonly represented feature space increases, while holding D_XY constant. The difference becomes even larger as the precision ratio increases. E The redundancy between representations and also increases as the dimensionality of the commonly represented feature space increases—again, the difference is increased at larger precision ratios. Thus, the assignment error rate is driven down at the cost of additional redundancy.

https://doi.org/10.1371/journal.pcbi.1011327.g002

However, increasing C while holding local variance constant comes with a significant cost. In particular, increasing the number of commonly represented stimulus features also increases the redundancy in information represented by the two brain regions. As discussed above, it is precisely this redundancy that allows the assignment problem to be solved (and solved more reliably for more overlapping features). Yet, in neural systems with limited metabolic resources, this redundancy reduces the efficiency of neural representations. We show that the redundancy between the two regions depends on both the number of overlapping features and the local variance (Fig 2e), where I(.; .) is the mutual information and H(.) is the Shannon entropy, and s is the extent of the overlapping features (see Methods in Calculating the redundancy between representations for more details). Thus, as the assignment error rate is reduced by increased C, the redundancy of the representation is increased—we make the consequences of this explicit below.

Further, we show that a similar tradeoff holds for asymmetric feature representations (Fig A in S1 Text). On one hand, the common features can be represented with the same precision in both regions. This is the maximally redundant representation for a given C, and therefore leads to the lowest assignment error rate. On the other hand, the common feature can be represented asymmetrically (with high variance in one region and low variance in the other). This is a less redundant representation, and is associated with a higher assignment error rate.

2.4 Redundancy reduces the precision of the neural code

Now, we make our theory more concrete by making an assumption about the format of the neural representations in R_X and R_Y. This assumption allows us to quantify how the assignment error rate and local variance both depend on the metabolic energy available to the code, in the form of spikes and neurons. To begin, we analyze the errors made by a population of neurons with randomly positioned Gaussian receptive field (RF) responses (Fig 3a, top). This kind of neural code is thought to be used for a variety of different sensory features [24–28], including spatial position in the visual system (but, interestingly, not in the mammalian auditory system [39–42]). While we assume that R_X and R_Y both use the same representational format, this is not required by our theory of assignment errors—and we explore the case in which R_X and R_Y have different representational formats below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The efficiency of random receptive field codes.

A A random Gaussian receptive field population. The width of the RFs are chosen to minimize the total MSE of the population. B A histogram of errors made by an example RF code. While the majority of errors are local (left, grey), a small proportion of errors are non-local threshold errors (right, red). C The two kinds of errors made by RF codes. Threshold errors dominate at small RF sizes, while local errors dominate elsewhere. We choose the RF size for our codes based on the combined error rate. D The dependence of total MSE on population SNR for fixed population size. The solid line is from simulations, the dashed line is the prediction of our theory, and the dotted line is local MSE only. E The dependence of total MSE on population size for fixed population SNR. The different line types are the same as D. F (above) Schematic of the situation being investigated. A K = 4 dimensional stimulus is represented across two distinct populations of neurons R_X and R_Y. Each population of neurons has 10, 000 units and an SNR varying between 5 and 10. (left) The average local MSE across the two codes and the redundancy across the two regions of the code schematized above, shown for different numbers of overlapping features C. The grey lines connect points with the same SNR across the different overlap levels. (right) The same as on the left, but for the assignment error rate.

https://doi.org/10.1371/journal.pcbi.1011327.g003

Recent work has shown that RF codes make two kinds of errors [43]. The first kind of error is local, and is equivalent to the local variance we have discussed so far. In this case, the decoder maps the noisy response to a nearby position on the stimulus representation manifold (Fig 3a, bottom, “local error”). This local variance is captured by the Fisher information of the code [43–49]. The second kind of error is non-local, and is referred to as a threshold (or catastrophic) error [43, 50]. Catastrophic errors occur when the noisy response is mapped to a distant region on the stimulus manifold, due to large noise that points toward one of these regions of response space (Fig 3a, bottom, “threshold error”). In a noisy system, every response will give rise to some local variance (Fig 3b, “local”); threshold errors only occur for a subset of responses, but their magnitude has the same order as the size of the stimulus space (Fig 3b, “threshold”).

These two kinds of error are in tension with each other. To see this, we begin by deriving the average Fisher information of a random RF code with a particular number of neurons and a particular average level of activity across the whole population (i.e., a budget of metabolic energy to be used for spiking), (4) where V is the sum squared activity across the population (i.e., the average response energy), σ_N is the standard deviation of the Gaussian noise added to the responses, is the population signal-to-noise ratio, and w is the width of the receptive fields, which is assumed to be small (see Methods in Random gaussian receptive field codes for the full derivation). We can see that the Fisher information increases when V increases or when w decreases (Fig 3c, local). However, the total error of the code is given by, where p_thr is the probability of a threshold error, and this probability increases as w decreases (Fig 3c, threshold). All else held equal, decreasing the width of the RFs in the code will reduce the local variance of the code (Fig 3c, local), while simultaneously increasing the probability of threshold errors (Fig 3c, threshold). In the following, we choose the RF width that minimizes the total error of the code (Fig 3c, combined; see Methods in Random gaussian receptive field codes for more details).

Using this framework, we show how this total error scales with both population signal-to-noise ratio () and population size. In particular, we show that, for high SNRs, the total error is explained primarily by the local variance (Fig 3d, solid vs dotted line). This results from the fact that the threshold error rate decreases exponentially with SNR (Methods in Random gaussian receptive field codes). Second, we show that increasing the population size while holding the SNR constant leads to a smaller decrease in total error (Fig 3e). That is, while increasing the population size pushes smaller RFs to be optimal, thereby decreasing the local variance, the tension between local and threshold errors causes the decrease in optimal RF size to be small. We also show that, as expected, requiring the code to represent K = 2 instead of K = 1 features causes higher error rates (Fig 3d and 3E, different lines).

Next, we link these RF codes to our theory of assignment errors. The local variance of the RF code is equivalent to the local variance in our framework. However, the threshold errors have no clear analogue in our current theory. Since the inferred stimulus produced from a threshold error is uniformly distributed across the whole stimulus space, they are extremely disruptive to assignment: The optimal strategy is not to integrate the threshold error representation during the assignment process at all. This unbalanced version of the assignment problem has been studied in combinatorics [9], but is beyond the scope of the current work. However, threshold errors are typically unlikely once the optimal RF width has been chosen. In particular, we find that codes with non-negligible threshold error rates also tend to have high total error. To proceed, we exclude codes with high total error, both due to the issues described for threshold errors and because our analytic calculation for the assignment error rate assumes local variance that is small relative to the size of the stimulus space.

Finally, we analyze the case of K = 4 total features represented across two different brain regions, with C = 1, 2, and 3 overlapping features. We show that the increase in overlapping features leads to an increase in both the redundancy between the two representations and the local variance of the code (Fig 3f, left, purple lines show constant SNR). Then, we show that, as anticipated, the increase in the number of overlapping features also leads to a decrease in the assignment error rate at these higher redundancy levels (Fig 3f, right). Thus, this modeling approach allows us to quantify the consequences of different levels of redundancy in a biologically plausible neural code.

2.5 Modularity has benefits for encoding high-dimensional stimuli

So far, to motivate the importance of the assignment problem, we have focused on the case of stimulus features that are encoded by separate brain regions as a consequence of being received by distinct sensory systems—such as the pitch of a sound and the spatial frequency of an image. In this case, modularity is unavoidable and the assignment problem must be solved to integrate the disparate percepts. However, there is extensive evidence for modularity in the brain even within single sensory systems (e.g., [30], though these modules are at least partially redundant [51]). In our framework, this comes with a clear drawback when decisions have to be made based on combinations of features represented in different regions: Making those decisions reliably requires the assignment problem to be solved (e.g., identifying which of several cats just hissed or the car in a line of traffic that just honked), and this means that those brain regions must have some redundancy with each other.

Here, we contrast receptive field (RF) codes for relatively high-dimensional stimuli (K = 8) that are distributed across one, two, or three brain regions (Fig 4a). For example, a two-region code for the K = 8-dimensional stimulus could encode K_X,Y = 4 features in each region. However, such a code would make assignment errors at chance levels when presented with multiple stimuli. Thus, we consider codes which represent at least one redundant feature for every two brain regions (e.g., one region could represent four features and the other five, as in Fig 4a, middle). In every case where we compare multi-region codes to each other, the total code (that is, summing across brain regions) is constrained to have the same number of neurons and the same total population power.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Multiple-region representations minimize error in many conditions.

A Schematic of three different possibilities for representing a K = 8-dimensional stimulus. The features can all be represented in a single population (left); they can be distributed across two populations (middle); they can be distributed across three populations (right). The two and three population codes must represent some features redundantly to avoid assignment errors. B (top) The scaling of local MSE for the three codes above with population SNR (left) and the total number of units (right). The colors are the same as in A and C. (middle) The same as top but showing the scaling of the assignment error rate. (bottom) The same as top but showing the scaling of the total MSE. C The number of regions that minimizes the total distortion of the representation of K = 8 features for different numbers of neural units and population SNR. The white parts of the plot are when no representational scheme achieved total MSE < .5.

https://doi.org/10.1371/journal.pcbi.1011327.g004

Within each region, we use the random RF codes described in the previous section. We use the local variance of each region code (Fig 4b, top) to compute the assignment error rate when integrating across regions (Fig 4b, middle), as developed earlier in the paper. Together, these lead to a total error that accounts for local variance and catastrophic assignment errors (Fig 4b, bottom). Codes with total error that exceeds half the size of the stimulus space are excluded from this analysis for the reasons discussed above.

We show that representing the K-dimensional stimuli in multiple brain regions can produce representations with lower total error, even when accounting for assignment errors. As a result, this work provides a justification for the modularity observed in the brain from the perspective of reliable and efficient neural codes. For stimuli with K = 8 features and smaller neural population sizes or population SNRs, then the lowest total error is achieved by a single region code (Fig 4c, bottom and left). In this case, avoiding catastrophic assignment errors is crucial. However, for larger neural population sizes and population SNRs, codes that distribute the representation over two or even three regions can provide lower total error than the single region code (Fig 4f, top right).

This results from a combination of two factors: First, the multi-region codes must represent redundant information to reliably solve the assignment problem. Thus, they have a disadvantage compared to the single region code, which need not represent any redundant information. However, second, lower-dimensional RF codes tend to produce smaller errors than higher-dimensional codes, especially very high-dimensional codes. In particular, the local variance of an RF code depends on RF width, and not directly on dimensionality (Eq 4). Thus, if low- and high-dimensional codes have the same RF width, then they could, in principle, have the same local variance. In practice, though, the high-dimensional code will require exponentially more neurons than the low-dimensional code to achieve full coverage of the stimulus space at that width (and to control threshold errors). In our setting, the single region, higher-dimensional codes do have more neurons, but only by a factor of two or three—not enough to achieve equivalent local variance to the lower-dimensional RF codes. Thus, multi-region codes begin to outperform single-region codes when the gains from their inherent efficiency become larger than the losses from their required redundancy.

We also observe that the difference in total error between these different coding strategies is relatively small (Fig 4b, bottom). While our current theory does not provide a guarantee of this in all cases, it points to an interesting potential invariance: The increase in redundancy required to represent a set of stimuli across multiple regions appears to be largely compensated for by the increase in efficiency of the constituent codes. Thus, many different coding strategies appear to provide similar performance.

Finally, while we assume that the features are interchangeable with each other, this is often not the case. In general, we believe that the assignment problem is solved to make decisions that require the linking of two or more stimulus features (e.g., asking what the value of feature X is for stimuli with a particular value of feature Y, as in the human working memory studies discussed above). So, one structural way to reduce the likelihood of assignment errors is to instead represent features X and Y in the same region, while representing the features that are relevant to a different decision in another region. The primate dorsal and ventral visual streams have been argued to follow this form: where the dorsal stream is thought to underly the planning of visually guided actions and the ventral stream is thought to underly perceptual decisions [31, 52]. However, a single division is unlikely to work for all decisions—and, in some cases, this simply shifts the need of solving the assignment problem to an earlier stage of processing. Thus, we believe that the simplified scenario we have analyzed here still provides a useful intuition.

2.6 Feedforward neural networks can solve the assignment problem

In the previous section, we analyzed the best-case assignment error rate for populations of neurons with randomly positioned RFs. However, there is no guarantee that a readout from the neural population will be able to achieve this best-case error rate without extensive and potentially metabolically expensive computations. In particular, solutions to the assignment problem used in combinatorial optimization are often framed as linear programming problems and require operations that have no clear neural analog [9].

Here, we focus on the case with two stimuli that are described by three total features (K = 3). The stimuli are initially represented by two distinct populations R_X and R_Y—each of which encode two features (Fig 5a, left). For both populations, one feature is unique to that population and the other is common across both populations (C = 1). We train a feedforward neural network model (Fig 5b) to take noisy input representations (Fig 5a, right) from R_X and R_Y and produce a representation of the two unique features from R_X and R_Y in an output population, which is also structured as a random RF code (Fig 5a, right). To do this reliably, the network must solve the assignment problem.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The assignment problem can be solved with one feedforward layer.

A Example of the input, first without (left) and then with (right) additive noise. The visualization shows the activity level of units in the random RF population. Each point is one unit, and the point is positioned at the unit’s RF location. The color shows the level of activity. The two inputs share one common feature (x-axis, C = 1) and each have one unique feature (y-axis). B Schematic of the feedforward integration, then marginalization network. In the full network, the input layers are followed by two hidden layers, then an integration layer, then the output layer. The output layer constructs a representation of only the two originally unique features of the stimuli, marginalizing out the common feature. C Example of target activity in the integration layer. The three features of the stimuli are represented together. The visualization is the same as A. D Example of a correct output (top) and an output with an assignment error (bottom). The output produced by an example network is shown on the left; the right shows the target output (top) or the prototypical assignment error (bottom). E (left) Both R_X and R_Y use the random Gaussian receptive field input developed in the previous sections. (right) The assignment error rates for stimuli with different distances in the common feature, using several variations on the network described in B. The end-to-end learning models are trained to reconstruct the marginalized representation from the inputs, either with or without intervening hidden and integration layers. The integration learning models are trained to construct both the integrated representation and the output target, with or without intervening hidden layers. F The same as E except R_Y uses an auditory-like input format, where the common feature (putatively, position) is encoded through a linear ramp.

https://doi.org/10.1371/journal.pcbi.1011327.g005

First, we train networks to perform this integration (of the two inputs) and marginalization (of the common feature) directly, without a hidden layer (Fig 5b, only R_X, R_Y, and “output”). Then, we quantify whether the activity in the output network is more similar to the correctly or incorrectly assigned representation (Fig 5d). The network without any hidden layers does not learn to assign the stimuli correctly (Fig 5d, end-to-end learning with no integration layer).

This implies that representations of features from multiple sensory modalities (e.g., the expression of a speaker and their words) may need to be derived from higher-dimensional representations that also incorporate the linking, common feature (e.g., estimates of spatial positions in the previous case). However, this does not imply that all of the features of an object must be represented at once: only those that need to combined, along with however many common features are required to achieve reliable assignment. This two-stage computation (a dimensionality expansion followed by a dimensionality reduction) has been shown to useful for other computations, such as representation frame-of-reference recoding [20].

Second, we show that networks trained to first reconstruct the three dimensional stimulus representations (Fig 5c) and then marginalize out the common feature achieve the minimal assignment error rate, while networks trained directly to construct the marginalized representations achieve low, but not minimal assignment error rates (Fig 5e, end-to-end learning compared to integration learning). While full understanding of this performance difference would require extensive investigation of the representations in the integration layer that are outside of the scope of this paper, this result emphasizes the importance of the integration step. Further, it demonstrates that random RF codes can be used to solve the assignment problem in an optimally reliable way and that they outperform representations learned through backpropagation.

Finally, while assignment between two representations that both use Gaussian receptive fields is analogous to assignment between parallel regions in visual cortex, there is evidence that neurons in the auditory system primarily represent spatial position using ramp-like rather than Gaussian-like tuning functions [39–42] (and see Methods in Random ramp codes). While our theory does not depend on the assumption of a specific representational format, it is possible that solving the assignment problem may be more difficult for a simple neural network when the formats of the involved representations are different. To test this, we studied the case where R_X had the same receptive field tuning as before, while R_Y was composed of neurons that had a linear response to the common feature and Gaussian tuning to the unique feature (Fig 5f, left R_X). This mirrors that ramp-like position and restricted pitch tuning observed in auditory cortex. Despite this difference in format between the two inputs, the results were qualitatively similar to before, with the best networks nearly saturating the bound on performance provided by our theory (Fig 5f, right). However, average performance was further from the bound in this case, indicating that the difference in format does increase the difficulty of the task.

4.1 Definition of the objects

An object set S of size N is N independent samples from the multidimensional probability distribution p(s)—thus, . Each s_i is a vector of length K, so p(s) is a probability distribution over a K-dimensional space. Each of these dimensions represents a feature of the object, such as color, spatial frequency, pitch, or orientation. In the majority of the paper, we assume that the objects are uniformly distributed in the space—that is, each point in the K-dimensional volume is equally likely to occur.

4.2 Definition of the representations

We focus on the representation of our K-dimensional objects in two brain regions, R_X and R_Y. We assume that both regions have some common and some unique information. That is, neither region encodes all K object features. This is guaranteed to be the case when R_X and R_Y represent early sensory areas from different sensory modalities—and there is evidence that a balance of unique and common information is preserved across hierarchies of sensory brain regions even within single modalities, such as in the primate visual system [29–32].

Formally, R_X encodes a subset of the K total object features, denoted as F_X—and similarly for R_Y. Each of the features can be identified by their index from 1 to K, and region R_X is said to encode feature i ∈ {1, 2, …, K} with local distortion , which is the variance of an optimal estimator for the value of feature i from the neural activity in region R_X—and, again, similarly for R_Y.

Thus, the subset of features represented in R_X that are not represented in R_Y are the unique information from R_X—that is, F_X \ F_Y. When R_X is an auditory region and R_Y is a visual region, then these unique features might include representations of pitch and timbre. Further, the subset of features represented in both R_X and R_Y are the common information, which is essential for solutions to the assignment problem. The size of this intersection |F_X ∩ F_Y| = C_XY = C has important consequences for the assignment error rate, and the achievable local distortion when the representation capacity is constrained.

Here, we study the reliability of inferences about the original set of N objects S, each of which are described by K feature dimensions, from the neural activity in two distinct regions, R_X and R_Y, which both encode some common and some unique information about the objects. Inferring the original object set from two distinct representations requires the two sets of representations to be combined with each other, which we refer to as assignment.

4.3 Definition of assignment errors

As mentioned above, to make inferences about the whole object set from two distinct sources of information (i.e., R_X and R_Y), the two sets of representations must be integrated with each other. When there is only one object (N = 1), then this integration is trivial as there is only one possible one-to-one mapping between the two sets of representations, and this mapping is correct. However, when there is more than one object (N > 1), then assignment errors become possible. In particular, for N objects, there are N! possible one-to-one mappings (i.e., assignments) between the two sets of representations, and . Thus, if there is no information about which mapping to select, assignment errors have the probability 1 − 1/N!, which is near 1 for even relatively small N.

Formally, we can frame the mapping selection problem as an inference about which of the possible maps M is most likely to account for the observed representations and in R_X and R_Y, respectively. This can be written as, (M1) where we proceed by first assuming that the prior probability of each map is the same (i.e., p(M) is uniform) and that the representations in one region do not depend on the map (i.e., , and similarly for ). Thus, we are left with a single term, , that gives the likelihood of the representations observed in one region (here, , but X and Y are interchangeable) conditioned on a particular map M and the representations observed in the other region (). If and are independent of each other, then , as above, and all maps are equally likely. So, it is dependence between the representations in R_X and R_Y that enables the correct assignment to be selected at a rate better than chance. Thus, dependence between those representations is necessary for a reliable solution to the assignment problem. In general, this observation already indicates to us that we should expect pairs of brain regions to encode some common information—so that the assignment problem can be solved—and some unique information—due to both distinct sensory systems, but also due to considerations related to efficient coding, which we make explicit below.

4.4 Random Gaussian receptive field codes

We consider codes for a uniformly distributed K-dimensional stimulus. Without loss of generality, we take the features of the stimulus to be between 0 and 1. The code consists of N units with Gaussian receptive fields. We assume that all units have the same field widths w and height P, but random centers μⁱ. In particular, the response of a neuron i is given by where and, here, we assume that w = w_j∀j.

We model the response to multiple stimuli as the sum of responses to each individual stimulus, so that where X is a set of stimuli. Note that the noise ν is only added once.

4.5 The Fisher information of the code

We will use the Fisher information as a measure of the magnitude of local errors via the Cramer-Rao bound. In our framework, the Fisher information along a particular dimension j is given by

We want to use the Fisher information to understand the average code performance across the whole stimulus space, rather than only for single points in the stimulus space. There is a complication here: the Fisher information is related to the MSE of an optimal, unbiased estimator by the Cramer-Rao bound, which states: so, we would really like to evaluate However, we were not able to evaluate this analytically. Instead, we evaluate which provides a good approximation of the former quantity so long as the variance of I_j (x) over x is small relative to the mean.

So, following a similar sequence as above for V. For small relatively small w (roughly, w < .5), we can approximate this expression as

4.6 Random ramp codes

To explore a situation more similar to representations in the mammalian auditory system, we also use neural representations with the following form: where the stimulus x is now divided into dimensions that are encoded with ramp-like tuning x_R and receptive field-like tuning x_G. The neuron is also assumed to have a preferred axis u_R in the space of features encoded with ramp-like tuning. In our simulations, u_R is chosen to be a random unit vector.

In the case of the auditory system, single neurons are thought to respond to a specific frequency band (i.e., a receptive field-like representation) while having the intensity of their response modulated by the position of the sound (i.e., a ramp-like representation) [39–42].

4.7 Calculating the redundancy between representations

We quantify the redundancy between R_X and R_Y as the mutual information between and , where R is the redundancy between the representations in region X and Y (in nats). As discussed throughout this work, this redundancy is crucial for solving the assignment problem. The redundancy is proportional to the number of commonly represented features C—so, increasing C will produce relatively large increases in redundancy, and, as we have seen, can be expected to effectively reduce assignment errors. Further, increasing the asymmetry of feature representations ΔD reduces the level of redundancy between the R_X and R_Y—so, as anticipated, increasing this asymmetry will increase the assignment error rate.

This redundancy represents the cost of our solution to the assignment problem. Thus, we desire a solution to the assignment problem that achieves a particular assignment error rate and local distortion magnitude while using as few bits—and, in particular, as few redundant bits—as possible.

4.8 The random RF integration model

The stimuli for the integration model are described by K = 3 dimensions. On each “trial,” we sample two uniformly distributed, random stimuli. Two features are represented in each of the two input populations. The two populations each represent C = 1 overlapping and one unique feature, with N = 400 units. Each population is noisy and has .

When present, the integration layer has N = 2000 units. The training targets are generated by obtaining representations from a K = 3-dimensional random RF code. When present, the hidden layers are two layers of 500 units, with no specific training target. The output population is N = 400 units, and the training target is given by samples from a random RF code for the two unique features in the input representation.