The encoding manifold reveals novel relationships between stimulus selectivity and neuronal classes

Encoding manifolds were calculated from the Allen Institute data using the responses to both drifting and static gratings. In brief, the encoding manifold arises from a machine learning algorithm that learns to place individual neurons in a relatively low-dimensional space according to their similarity in stimulus selectivity and response dynamics: neurons nearby on the manifold exhibit similar temporal responses to similar stimuli (Fig 1d) [1]. Responses to the natural scene stimuli were not included in the calculations of the encoding manifolds because the embedding algorithm requires stimuli for which neuronal responses do not depend on receptive field location (see Methods).

Consistent with prior results [1], the encoding manifold of VISp was relatively smooth and continuous, with neurons uniformly carpeting stimulus/response space (Fig 2a). Note that this topology is quite distinct from the manifold produced by this analysis when applied to retinal responses [1]. Representative peri-stimulus time histograms (PSTHs) for 5 temporal frequencies are shown for several neighborhoods of neurons; differences in selectivity for stimulus direction and in the temporal responses to drifting gratings across the manifold are hinted at by these PSTHs.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. The encoding manifold for VISp (V1) computed from responses to gratings.

a, A projection onto the first (dc0) and fourth (dc3) diffusion coordinates of the encoding manifold (see Methods) shows a smooth transition in ensemble PSTHs, from neurons sharply tuned for orientation (right side) to those broadly tuned (left side), indicating a continuous topology. The mean PSTHs for 10 different groups of neurons indicated on the manifold are arranged by temporal frequency from left to right then top to bottom at 1, 2, 4, 8, and 15 Hz. The PSTHs have been aligned by the preferred orientation to de-emphasize the specific orientation to which each neuron is tuned. b–d, Encoding manifold in a reproduced, with points colored according to orientation selectivity index (OSI) (b), firing rate (c), stimulus entropy (d), and preferred spatial frequency (e). Note the OSI values vary smoothly and continuously across the manifold, but are progressively less organized for stimulus entropy and preferred spatial frequency. f–g, Encoding manifold from a with points colored by the local density of broad-spiking neurons (f) or narrow-spiking neurons (g): note the distributions are complementary. h–i, Encoding manifolds reproduced showing the locations of broad-spiking and narrow-spiking neurons by cortical layers 2/3, 4, 5, and 6.

https://doi.org/10.1371/journal.pone.0356243.g002

To more clearly visualize how response properties are organized across the manifold, neurons (points) can be colored according to their preferred stimuli and/or response features. Clear gradients or geometries emerged along particular coordinates. For example, a gradient from left to right in the orientation selectivity index (OSI) of each cell was revealed by the manifold (Fig 2b), as well as one for firing rate increasing from lower right to upper left (Fig 2c). However, not all stimulus features were so well organized. For example, entropy, a measure of the range of stimuli for which a neuron gives robust responses, varied only slightly across the manifold (Fig 2d), while neurons were not obviously organized by their spatial frequency preference (Fig 2e; a point we will return to later). These observations underline an advantage of the encoding manifold: for a given firing rate, or orientation selectivity, there is a range of spatial frequencies and stimulus preferences. Without the global organization of the manifold, such co-variation would be difficult to discern.

Broadly, the organization of stimulus features across the manifold had a similar topology to that from our previous study that used a distinct stimulus ensemble [1,8]. This suggests that the VISp encoding manifold is not strongly dependent on the chosen stimulus set, provided that the set is sufficiently diverse. Of course, we cannot rule out that other topological features (i.e., distinct clusters) might emerge with yet-to-be tested stimuli. The continuous topology across the manifold is markedly different from that of the retina [1], in which morphologically and functionally distinct retinal ganglion cell types form distinct clusters on the manifold. This difference suggests that cell type diversity in visual cortex serves a distinct purpose than in the retina, as discussed previously [1].

The organization of features that were not used in the calculation of the manifold is more interesting. Neurons were classified on the basis of their electrical waveforms: broad-spiking or narrow-spiking (typically treated as putative excitatory and putative inhibitory neurons, respectively, although we note that some broad-spiking cortical neurons are known to be inhibitory and express somatostatin [9]). The broad-spiking neurons were concentrated on the right in this view of the manifold, with a secondary concentration salient at the lower left, and were present at a lower concentration throughout the middle of the manifold (Fig 2f). In contrast, the narrow-spiking neurons were concentrated in the upper left corner and were also present at a lower concentration throughout the middle of the manifold; they were less densely represented in the regions of the manifold in which the broad-spiking neurons were concentrated (Fig 2g). Thus, the encoding manifold recapitulates the observation that narrow-spiking neurons tend to be less selective for stimulus orientation and to fire at higher rates than broad-spiking neurons in mouse V1 [10,11]. Again, this confirms that the encoding manifold organizes neurons by their physiological properties.

The encoding manifold for V1 also organized neurons according to cortical layers even though layer information was not used to generate the manifold (Fig 2h, 2i). In layers 2/3, broad-spiking neurons were concentrated in the lower right, as were those in layer 4 though less tightly. In layer 6, broad-spiking neurons were much more tightly concentrated in the upper right. It is known that layer 6 broad-spiking neurons tend to have low firing rates [12], which is confirmed by the manifold (compare Fig 2h layer 6 to panel c). In layer 5, broad-spiking neurons exhibited two foci of density: one together with those of layer 6, but more tightly focused, and another in the salient concentration at the lower left. This bimodal distribution reveals that layer 5 contains (at least) two populations of broad-spiking neurons with quite different response properties [13] (see Discussion).

In comparison, the distributions of narrow-spiking neurons exhibited distinct patterns from the broad-spiking neurons across layers. In layers 5 and 6, narrow spiking neurons were most concentrated along the top of the manifold, whereas those in layers 2/3 and 4 were concentrated lower down except for a cluster at the very top left corner. While the layers differed from one another in the distributions of broad-spiking and narrow-spiking neurons, within each layer these two classes of neurons were largely complementary. Again, we emphasize that these features of the data were not utilized in generating the encoding manifold, and thus their recovery serves as validation that the manifold captures meaningful functional organization. At the same time, the finer within-layer structure demonstrates the ability of the manifold approach to reveal novel relationships between stimulus selectivity, response dynamics, and neuronal classes. As a control analysis, we tested whether the neurons’ receptive field position or size (see Methods) could trivially explain some of the organization found on the manifolds, but found that these properties were not well organized on any of the manifolds studied (S1 Fig).

Importantly, these results on broad-spiking and narrow-spiking neurons across layers recapitulate several observations from our previous use of encoding manifolds to understand mouse retina and VISp [1]. However, they add to that work by demonstrating that the approach is robust when the experiments use a different stimulus ensemble, different neuronal recording technologies, and are performed by different laboratories.

Encoding manifolds from higher cortical visual areas exhibit smooth organization similar to VISp

Do higher cortical visual areas in the mouse exhibit similar encoding manifolds with similar topology as VISp? Are these higher visual areas also continuous in their topology, and how are features such as spatial frequency, orientation, and firing rate organized across each of these neural populations? The Allen Institute data set allowed us to answer these questions for five higher visual areas: VISam, VISal, VISpm, VISlm, and VISrl.

We constructed the encoding manifolds for all five of the higher visual areas using stimuli and methods identical to those described for VISp. We found that the manifolds for all higher visual areas exhibited a smooth organization and shared many properties with VISp (Fig 3). When the orientation selectivity index (OSI) and firing rate were plotted as functions on the manifolds, they varied smoothly (Fig 3, left and center columns). High firing rates were organized with low OSI, and conversely. Finally, like VISp, spatial frequency preference did not appear well-organized for any of the higher visual areas (Fig 3 right column), although temporal frequencies were smoothly organized along one of the diffusion coordinates (S2 Fig). Note that the apparent shape of each manifold in the two-dimensional plots depends on the specific projection used for visualization. We chose coordinates to emphasize the smooth organization of neuronal properties over each manifold (see Methods).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Encoding manifolds for five higher visual areas (HVAs) computed from responses to gratings.

Rows indicate the HVAs and columns indicate different features: OSI and firing rate to drifting gratings (left and middle columns), and preferred spatial frequency (third column). Note the similarity of organization of these properties across areas and with VISp (Fig 2).

https://doi.org/10.1371/journal.pone.0356243.g003

We also analyzed how broad-spiking and narrow-spiking neurons were distributed across the manifolds for the higher visual areas. Similar to VISp, both classes of neurons exhibited layer-specific localization on their respective manifolds (see S4–S8 Figs), however the details of these arrangements varied somewhat across higher visual areas. One common trend was that broad-spiking neurons in layer 6 exhibited high OSIs across cortical areas. Another common trend was that broad-spiking and narrow-spiking neurons were located on complementary portions of the encoding manifold in each higher visual area and in each layer. A variable property was the bimodality of broad-spiking neurons in layer 5 observed in VISp: it varied across higher visual areas (S9 Fig).

The conclusion from this initial analysis of the encoding manifolds from the higher visual areas is that they exhibited many features in common with VISp in terms of manifold topology and geometry. Topologically, all the manifolds were continuous, distinct from the encoding manifold of retinal ganglion cells, which was more discrete (shown previously [1]). Their geometries were also similar to that of VISp, with OSI and firing rates being clearly organized across the manifolds, while preferred spatial frequencies were less clearly organized. We note these statements pertain to generating the manifolds from the responses to black-and-white gratings of various spatial and temporal frequencies; probing other stimulus features (e.g., color) could reveal greater differences across cortical visual areas (see Discussion).

Encoding manifolds organize natural scene responses relative to gratings responses

We next examined a novel feature of the Allen Institute dataset relative to previous applications of the encoding manifold: the inclusion of responses to natural scenes. This feature provided an opportunity to examine the extent to which neural responses to natural scenes exhibited any organization on these encoding manifolds, and whether this organization varied across cortical visual areas. While we did not utilize the responses to natural scenes in generating the encoding manifolds, we could display how neurons were organized on the grating-derived manifolds according to their responses to natural scenes [14]. Given the (perhaps) unbounded variability of natural scenes [15,16], it is difficult to know how to appropriately arrange the responses, so we began by simply plotting the mean firing rate to natural scenes across the manifold (Fig 4a). Surprisingly, the geometric organization of the natural scene responses, like that of the grating responses, varied rather smoothly across the manifold, approximately from right to left (Fig 4b). Comparing the kinetics of the natural scene to grating responses in individual neurons reveals similar dynamics (Fig 4c, 4d), but some neurons exhibited larger responses to natural scenes (Fig 4c), while others responded more strongly to gratings (Fig 4d).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. The encoding manifold inferred from gratings organizes the ratio of natural scene to static grating responses.

a–b, Neurons on the encoding manifold from Fig 2 colored by log mean firing rate to all natural scenes (a) and log mean firing rate to all static gratings (b). c–d, Firing rates (kernel-smoothed, see Methods) over at least 39 trials as a function of time for three representative neurons that respond better to static gratings than to natural scenes (c), or respond better to natural scenes than to static gratings (d). The labels shown within each plot correspond to the neuron identifiers used in panels e.1–e.10. e, Encoding manifold with points (neurons) colored by the ratio of their mean response to gratings versus natural scenes: note a gradient from top to bottom of neurons preferring gratings to those preferring natural scenes. e.1–e.10, PSTHs as in Fig 2 and bar plots of firing rates to gratings and natural scenes are shown for representative neurons whose positions are indicated on the manifold. Bar plots show the result is robust when using either ‘mean’ (using all stimulus variations) or ‘max’ (using the preferred stimulus variation) firing rates. Notice that the top-to-bottom organization of natural scene preference is approximately orthogonal to that of firing rate and orientation selectivity (cf. Fig 2).

https://doi.org/10.1371/journal.pone.0356243.g004

It should be noted that the static grating stimuli can be said to tile the (static) stimulus space for gratings, encompassing the relevant range of orientations and spatial frequencies so as to stimulate all grating-responsive neurons well, regardless of the location of their receptive fields on the stimulus screen. Therefore, the best response measured with these stimuli for each neuron is likely to be the near-optimal grating response. However, the natural scenes are a small selection from all possible natural scenes and, since they are flashed (not drifted), the responses to them presumably depend strongly on the location of each neuron’s receptive field in the visual field.

Nevertheless, the comparable co-variation in the geometries of organization of natural scenes to gratings in the manifolds (Fig 4a, 4b) inspired us to consider the ratio of each neuron’s responses to the two types of stimuli. This comparison produced a striking result (Fig 4e): the relative responses to natural scenes were organized along an axis orthogonal to the gradient of orientation selectivity on the manifold (Fig 2b). Specifically, the average response of each neuron to the 118 natural scenes (Fig 1c) was compared to its average response to all the static grating stimuli (Fig 1b). Note that the two types of stimuli were presented identically, in separate blocks that were interleaved between the repetitions of each block. Neurons that preferred natural scenes (green) were distributed along one side of the manifold while those that preferred static gratings were more prevalent on the other side (magenta). An alternative analysis compared the maximum response to any individual stimulus from among the sets of 120 static gratings and 118 natural scenes; this is potentially a noisier measure of selectivity than using the average responses to the entire stimulus set, but it produced the same result: when the manifold was colored by the ratio of each neuron’s response to the best natural scene versus its response to the best static grating, the picture was very similar (S11 Fig). This manifold’s surprising organization of natural scenes (relative to gratings) is not a trivial consequence of differences in overall firing rates: the distributions of average firing rates to static gratings and to natural scenes were similar (Fig 4a, 4b).

Despite the marked difference in the two stimulus sets, a relationship between grating preferences and natural scene preferences was revealed by the encoding manifold. We note that it has been notoriously difficult to find any consistent characterization of natural scene responses in visual cortex [3,17–20], especially when considering average responses [21]. Yet the encoding manifold has found an organization whereby across the entire range of firing rates there is a range of preferences for gratings versus natural scenes.

We next checked to see if the encoding manifolds for the higher visual areas exhibited a similar geometry, organizing the ratio of natural responses to grating responses. Remarkably, every higher cortical visual area exhibited a similar organization (Fig 5 and S10 Fig third column). In fact, a prediction from this observation is that if a manifold were generated by co-embedding all the neurons from all the visual areas, neurons from individual visual areas should be indistinguishably mixed (as opposed to clustered by visual area). We tested this prediction and, as expected, they were highly mixed, while main properties like OSI, FR, and ratios remaining largely well organized (see S12 Fig).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Encoding manifolds reveal preference for natural scenes vs. static gratings in VISpm (top) and VISal (bottom).

a, The encoding manifold for VISpm colored to indicate the ratio of responses to natural scenes versus gratings. Note the gradient varying from top left (preference for gratings, in purple) to bottom right (preference for natural scenes, in green). Details of responses of nine representative neurons (a.1–a.9) shown as in Fig 4e. b, The encoding manifold for VISal labeled by ratio of responses to natural scenes versus gratings. Note the gradient varying from left (preference for gratings) to right (preference for natural scenes). Details of responses of ten representative neurons (b.1–b.10) shown as in Fig 4e.

https://doi.org/10.1371/journal.pone.0356243.g005

The consistency of these results across visual cortical areas suggests something important about this organization. Furthermore, it had to be revealed by some property of the responses to only gratings because the natural scenes were not used in generating the encoding manifolds. We next sought to find this property of the grating responses that organized the preferences for natural scenes (relative to gratings).

Global relationship between spatial frequency tuning and natural scene preferences

To uncover the signifier in the grating responses that organized the natural scene preferences, a reasonable place to start is with the selectivity to spatial frequencies. We began with VISp: as noted above, preferred spatial frequency appeared mixed over the VISp manifold (Fig 2d). However, we examined the density of neural responses by frequency band (cf. S2 Fig) and a surprising organization was revealed upon closer inspection: there were complementary densities of neurons preferring either low or high spatial frequencies versus neurons preferring intermediate spatial frequencies (Fig 6a). Furthermore, the spatial distributions of these preferences over the manifold varied almost identically to the distributions of neurons that preferred natural scenes versus gratings (Fig 6b, 6c and S13 Fig). Thus, the population of natural-scene-preferring neurons was distributed on the manifold in a similar manner to neurons preferring either high or low spatial frequencies.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Comparing preference for spatial frequency and natural scenes versus gratings in VISp.

a, Encoding manifold for VISp colored by spatial frequency preference: neurons are colored according to their preferred spatial frequency for static grating stimuli, grouping 0.02 + 0.32 (extreme frequencies, in yellow) vs. 0.04 + 0.08 + 0.16 (intermediate frequencies, in blue). b, Spatial frequency preferences summarized by their local density over the manifold (see Methods); note that extreme frequencies (left) and intermediate frequencies (right) are largely complementary. c, The log ratios from Fig 4e, shown here for comparison with the distributions in b. Corresponding plots for higher visual areas are shown in S13 Fig. d, Distribution of number of units preferring a given spatial frequency (SF) in VISp. e, Comparison across neurons binned by their preferred spatial frequency (x-axis) of the mean ( s.e.m.) of the log ratios between mean firing rates to natural scenes vs. static gratings. Neurons that prefer low or high SFs tend to respond more strongly to natural scenes, while those that prefer intermediate SFs respond more strongly to gratings. f, Mean azimuthally averaged radial Fourier power spectra of the flashed gratings and natural scenes used in the experiments. Spectra were averaged across all stimulus instances within each stimulus class and are plotted as a function of spatial frequency on a logarithmic axis.

https://doi.org/10.1371/journal.pone.0356243.g006

To quantify this relationship between grating preferences and natural scene preferences, we first binned neurons in each visual area by their spatial frequency preferences (Fig 6d). Consistent with previous work, there was a preponderance of neurons that preferred lower spatial frequencies [22,23]. We then calculated the log ratio of the mean response across natural scenes to the mean response across static gratings for each neuron as a function of that neuron’s preferred spatial frequency (in response to drifting gratings); the result was a “U”-shaped curve, showing that neurons in the group preferring either low or high spatial frequencies tended to have the largest responses to natural scenes, while those preferring intermediate frequencies tended to respond more vigorously to gratings (Fig 6e). Importantly, the average radial spatial-frequency spectra of the grating and natural-scene stimulus ensembles were highly similar (Pearson correlation, r = 0.9037; cosine similarity, 0.9043; Fig 6f), indicating that the higher responses to natural scenes at the extreme SFs are not trivially explained by higher energy in those stimuli at 0.02 and 0.32 cpd. Rather, neurons in mouse V1 tuned to high or low spatial frequencies tend to respond better to natural scenes than gratings, while neurons tuned to intermediate spatial frequencies are more responsive to gratings. We return to this point and its potential significance in the Discussion.

Next, we determined to what extent this relationship observed in VISp between spatial frequency preferences and natural scene preferences held for the higher visual cortical areas. Indeed, each visual area exhibited a “U”-shaped relationship between spatial frequency preference and natural scene preference (Fig 7). Furthermore, as with VISp, this relationship was reflected in the encoding manifolds distributing neurons that preferred low+high spatial frequencies on one side, and neurons that preferred intermediate spatial frequencies on the other side of the manifold (S13 Fig). As a complementary visualization, S18 Fig shows the mean firing rates to natural scenes and static gratings separately as a function of preferred spatial frequency for each visual area.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Preference for spatial frequency and for natural scenes versus gratings across visual areas.

a, Distribution of number of units preferring a given spatial frequency (SF) in each visual area studied. b, Mean s.e.m. log ratio between mean firing rate to natural scenes (NS) vs. static gratings (SG) of the neuronal population in each area as a function of their preferred SF. Neurons that prefer low or high SFs tend to respond more strongly to natural scenes (positive ratios), while those that prefer intermediate SFs respond more strongly to gratings (negative ratios).

https://doi.org/10.1371/journal.pone.0356243.g007

Finally, to rule out the possibility that the observed relationship was merely a consequence of the gradient in firing rates to static gratings, we performed a permutation test. Shuffling natural-scene firing rates across neurons destroyed both the original manifold organization and the characteristic U-shaped relationship between preferred spatial frequency and the NS/SG ratio (S14 Fig). The observed U-statistic (see Methods) exceeded that expected under the permutation null distribution in five of the six visual areas studied, indicating that the observed U-shaped relationship is unlikely to arise from chance associations between preferred spatial frequency and natural-scene responses.

Natural scene preference differs by layer

We next checked for layer-specific preferences for natural scenes versus static gratings across visual cortical areas (Fig 8), because some of the laminar distributions observed showed clear overlap with grating- or natural-scene-preferring regions on the manifolds of several visual cortical areas (Fig 8a–8d). To quantify these observations, we generated histograms of the log ratio of mean firing rates to natural scenes versus gratings for neurons in each layer and visual area (layers 2/3 and 6 shown in Fig 8e). Histograms shifted toward positive values indicate a preference for natural scenes, while a shift toward negative values indicates a preference for static gratings. As with our previous comparisons between natural scene and static grating responses, the data were analyzed in two ways. First, the mean response across all displayed natural scenes was compared to the mean response across all static gratings (‘mean analysis’). Second, the mean response to the single natural scene that generated the largest response was compared to the mean response to the single static grating that generated the largest response (‘max analysis’). Statistical significance was assessed using one-sample t-tests against zero and corrected for multiple comparisons using the Benjamini–Hochberg false discovery rate (FDR) procedure across all layer, area, and analysis combinations. Effect sizes (Cohen’s d) are reported in S1 Table.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Preference for natural scenes versus gratings across cortical areas and layers.

a–d, Local density of broad-spiking neurons belonging to cortical layer 6 (left) and preference for gratings vs. natural scenes (right) for neurons in the VISpm, VISam, VISal, and VISlm manifolds. Note the overlap between the regions where layer 6 neurons concentrate and the regions with higher preference for gratings. e, Histograms of the distribution of log ratios computed from the maximum firing rates to natural scenes versus static gratings for all neurons in layer 2/3 (top row) and layer 6 (bottom row); each column shows a different visual cortical area; indicates the mean. Positive values indicate stronger responses to natural scenes, whereas negative values indicate stronger responses to gratings. Layer 2/3 distributions tend to be shifted toward natural-scene preference, whereas layer 6 distributions tend to be shifted toward grating preference, although substantial heterogeneity is present within each area and layer. f, Summary of one-sample t-test statistics comparing the log ratio between natural-scene (positive, green) and static-grating (negative, magenta) responses across cortical layers and visual areas. Values computed using either the mean (left column) or maximum (right column) firing rates are shown. Significance markers indicate Benjamini–Hochberg false discovery rate (FDR)-corrected p < 0.05 across all layers, areas, and analysis combinations. Corresponding effect sizes (Cohen’s d) are reported in S1 Table. Significant preferences for natural scenes are observed most consistently in superficial layers, whereas preferences for gratings in deeper layers are weaker and more variable across visual areas.

https://doi.org/10.1371/journal.pone.0356243.g008

The ‘mean analysis’ revealed relatively weak layer-dependent differences in the responses to natural scenes and static gratings (Fig 8f, left columns: magenta (resp., green) indicates grating (natural scene) preference). After correction for multiple comparisons, significant effects were largely restricted to VISp, which exhibited a preference for natural scenes in layers 2/3 and 4 and a preference for gratings in layer 6. By contrast, the ‘max analysis’ (Fig 8f, right columns) revealed a more consistent pattern across visual areas. Significant preference for natural scenes was observed in superficial layers (layers 2/3 and/or 4) in VISp, VISam, VISal, VISl, and VISrl. Preference for gratings in layer 6 remained significant in VISp, while VISam showed a significant grating preference in both layers 5 and 6. Although effect sizes in deeper layers were generally negative across visual areas, indicating a tendency toward stronger responses to gratings, these effects did not consistently reach significance after correction for multiple comparisons. Despite these population-level trends, there was substantial heterogeneity within each cortical layer and visual area, as illustrated by the broad distributions in Fig 8e).

The functional organization of laminar preferences differ across visual areas

Differences in grating selectivity between visual areas have been observed previously [22,23], and have been interpreted as a hint toward dorsal/ventral stream models [25–27]. The encoding manifold approach allowed us to refine the population analysis to individual layers, where further differences emerged, consistent with previous observations of laminar specialization in visual cortical responses [28]. For example, in VISp, the greatest concentrations of highly orientation-selective, broad-spiking neurons were found in layers 5 and 6 (Fig 2b, 2g). The same was true for narrow-spiking neurons in layers 4 and 2/3, even though they occupied regions practically disjoint from those of layers 5 and 6. In contrast, although in VISal the broad-spiking neurons of layer 2/3 were in the regions of the manifold with high orientation selectivity, those in layer 4 were not (S5c, S5f Fig). The same was true to a lesser extent in VISpm (S6c, S6f Fig). This is perhaps surprising considering that the dominant feedforward projection from superficial layers of VISp goes to layer 4 of higher cortical visual areas [3]; one would expect the recipient layer 4 populations not to lose selectivity.

The deeper layers are also interesting, since they involve both cortical and sub-cortical projections [12]. Projections to the dLGN may sharpen receptive fields in the mouse [29] and cat [30–32]), while the cortico-cortical projection could be involved in generating endstopping and other surround effects [33]. Layer 5 also projects to the pulvinar [34], a projection that could be relevant to coordinating natural scene responses [35,36].

Layers 5 and 6 showed different organizations for broad- and narrow-spiking neurons, depending on the area. In VISp (Fig 2g), layer 5 exhibited two distinct concentrations of broad-spiking neurons, one of which overlapped almost completely with layer 6 in the region of highest orientation selectivity. We do not know which if either of these broad-spiking clusters may contain somatostatin-expressing interneurons. Similarly, broad-spiking layer 5 neurons in the VISpm manifold were organized in two clusters, although the overlap of the cluster with the highly selective layer 6 neurons was less complete (S6 Fig). On the other hand, in VISal, layer 5 had practically no overlap with layer 6, even though the latter remained highly selective (S5 Fig). This difference between VISpm and VISal is striking, and consistent with previous reports of functional and anatomical differences between VISal- and VISpm-related visual pathways [37]. The distributions of laminar density for VISam, VISlm, and VISrl are shown respectively in S4 Fig, S7 Fig, and S8 Fig. Layer 5 in all three cases had a bimodal distribution on the respective manifolds, with one lobe remaining consistent with the density of broad-spiking neurons in layer 6. S9 Fig summarizes all of the above comparisons between layers 5 and 6.

In summary, building encoding manifolds from responses to diverse stimulus sets and then coloring the locations of broad- and narrow-spiking neurons from different layers can provide insights and generate novel hypotheses about the common themes and distinct specialization in the functional organization of visual cortical areas. Expanding these approaches to population recordings from primate visual cortical areas may also provide important insights to the conserved versus divergent features of visual processing across species.

A novel relationship between spatial frequency tuning and natural scene preferences

Natural scenes have been among the most difficult stimuli to analyze, and the encoding manifold revealed a unique invariant at the population level. Although the manifolds computed in this study were based solely on gratings, a consistent relationship emerged between the natural scene preferences of high- and low-spatial frequency preferring neurons compared to neurons preferring intermediate frequencies. Specifically, neurons tuned to low or high spatial frequencies tended to generate larger responses to natural scenes than gratings, while neurons tuned to intermediate spatial frequencies tended to generate larger responses to gratings (Fig 7b). This highlights the usefulness of the encoding manifold as an organizational and visualization tool: although in principle these relationships could have been found by directly plotting log ratios vs. preferred SF, inspection of how physiological properties and feature selectivity are distributed across the manifold provides a data-driven route to discovering such relationships and understanding how they are structured across cortical areas and layers (e.g., [38]). As we show in S15 Fig and S16 Fig, a naive analysis based on examining histograms based on standard properties like OSI and firing rate alone would not have revealed these relationships.

How might this curious relationship be explained (cf. [19,20])? One possibility is that it has to do with object content, or at least a proxy for it. Examining the statistics of image content is instructive (Fig 9a–9d and S17 Fig). The organization of neurons preferring low and high spatial frequencies relative to intermediate frequencies (Fig 9e, 9f) closely parallels the distribution of neurons preferring natural scenes versus gratings (Fig 9g), providing a visual interpretation of the U-shaped relationship shown in Fig 7b. Low frequencies signal “where” an object might be, while high frequencies signal “what” it might be made of. Such an identification of spatial frequency content with scene layout has long been proposed [39,40]. However, it is unlikely that simply extracting filtered features is the complete story (e.g., see [41]).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Natural scene statistics and neuronal preference on the encoding manifold.

a, An example natural scene image and its low-pass, intermediate (band-pass), and high-pass filtered versions, and b associated spectra. c–d, Two more natural scenes and their filtered versions. e, Encoding manifold for VISp colored by spatial frequency preferences of neurons (as Fig 6a): Neurons are colored according to their spatial frequency preference for static grating stimuli, grouping 0.02 + 0.32 (extreme frequencies, in yellow) vs. 0.04 + 0.08 + 0.16 (intermediate frequencies, in blue). f, Spatial frequency preferences summarized by the density over the data graph (see Methods) for extreme frequencies (left) and for intermediate frequencies (right). g, The log ratios from Fig 4e, shown here for comparison with the distributions in f. The different spatial frequency bands thus provide insight into the U-shaped curves from Fig 7b, suggesting a grouping of the low and high frequency content signals separated from the intermediate ones. For comparable plots in the higher visual areas, see S13 Fig.

https://doi.org/10.1371/journal.pone.0356243.g009

Classically, natural scene analysis has been thought to proceed by a coarse-to-fine visual processing [42]. Object perception, figure/ground discrimination and hunting behavior [43,44] are complex tasks, implicating computations across multiple visual areas [45], including feedback within and between them [46,47] and recurrence [48,49]. A recent study [50] found direct low- and high-frequency dynamics in natural scene viewing, extending classical results on dynamical frequency tuning [51–54]. Since the encoding manifold considers the entire time course of responses, it is able to capture these dynamics—a possible signature of coarse-to-fine processing [55,56]—in the manifold inference process. Perhaps while building biologically-motivated perceptual systems, following this study, one should consider coarse-and-fine processing.

Encoding manifolds

Encoding manifolds capture similarity in the responses over time of neurons exposed to a wide range of stimuli. They display neurons in relation to as many different stimulus features as one is able to illustrate. No other procedure of which we are aware lets one simultaneously visualize how the neurons that respond particularly well to one stimulus respond to many others. They disclose visually the relationships between, for example in the present data: firing rate, selectivity for stimulus orientation, spatial and temporal frequencies, cortical laminae, electrical waveform (related to excitatory versus inhibitory classes), overall selectivity (stimulus entropy), and preference for gratings versus natural scenes. This property should make encoding manifolds widely useful across biology when many relationships may be present but could go unnoticed.

Dataset

The “Neuropixels” single neuron recording dataset shared by the Allen Institute includes recordings from many cortical visual areas in alert wild type (C57BL6/J) mice [47]. Recording sites were localized to specific cortical areas by reference to intrinsic signal imaging maps and to cortical layer by transformation into the Allen Common Coordinate Framework. Details of all procedures are described at links from [7].

To summarize, visual stimuli were were displayed on an LCD monitor at a resolution of 1920 1200 pixels at 60 Hz refresh rate. Stimuli were presented monocularly, and the monitor spanned of visual space prior to stimulus warping. Each monitor was gamma corrected and had a mean luminance of 50 cd/m². To account for the close viewing angle of the mouse, a spherical warping was applied to all stimuli to ensure that the apparent size, speed, and spatial frequency were constant across the monitor as seen from the mouse’s perspective.

In our experiments, we analyzed responses to a subset of their “Brain Observatory 1.1” stimulus set, including 40 different drifting gratings, 120 stationary gratings, and 118 natural scenes [57]. We used data from all experiments that included these stimuli (32 mice in total). The drifting gratings were shown at a spatial frequency of 0.04 cycle/deg, 80% contrast, 8 directions of motion (0, 45, 90, 135, 180, 225, 270, and 315 degrees) and 5 temporal frequencies (1, 2, 4, 8, and 15 Hz), with 15 repeats per condition. The stationary grating set, also presented at 80% contrast, consisted of 120 gratings of 6 different orientations (0, 30, 60, 90, 120, and 150 degrees, clockwise from 0 = vertical), 5 spatial frequencies (0.02, 0.04, 0.08, 0.16, 0.32 cycle/degree), and 4 phases (0, 0.25, 0.5, 0.75), each presented for 0.25 seconds with no intervening gray period. The natural scene stimuli consisted of 118 natural images taken from the Berkeley Segmentation Dataset [58], the van Hateren Natural Image Dataset [59], and the McGill Calibrated Colour Image Database [60]. The images were presented in grayscale and were contrast normalized and resized to 1174 918 pixels. The images were presented in a random order for 0.25 seconds each, with no intervening gray period.

Data preprocessing

As in [1], responses of a neuron to each stimulus (PSTHs) were kernel-smoothed using the Improved Sheather-Jones algorithm [61], using a bandwidth of 25 ms (typical value obtained for all visual areas when using automatic bandwidth selection) for drifting gratings and 10 ms for static gratings and natural scenes (between 39 and 50 trials). The Python implementation [62] was used.

Responses to the drifting gratings were tested for statistical significance by performing a two-tailed Mann-Whitney test comparing the mean activity during the second half of the ISI immediately preceding the stimulus versus mean activity over any interval of the same length within the stimulus presentation. This strategy allowed transient responses to be able to reach the same level of significance as sustained responses. Responses were deemed significant if they had a p-value of at most 0.001. Only neurons that had a significant response to at least one variation of drifting grating were included in the data tensor (see below). The resulting number of neurons used were: n = 1261 for VISp, n = 910 for VISam, n = 961 for VISal, n = 525 for VISpm, and n = 810 for VISrl.

We were able to build meaningful encoding manifolds for gratings and flow stimuli in previous studies because these stimuli were globally homogeneous, so every neuron was simultaneously stimulated by similar visual features across time. This allows us to build meaningful tensors and generate a meaningful manifold embedding. Each natural scene, however, is really a different stimulus for different neurons depending on their receptive field positions; so aligning the same natural scene to all neurons in the tensor would not be feasible.

Data analysis

Tensor decomposition.

We organize the data as a 3-way tensor, neurons⁽¹⁾ stimuli⁽²⁾ response in time⁽³⁾. (For an introduction to tensors, see [65].) In standard notation [66]:

(1)

where ° denotes vector outer product and the factors are collected into the factor matrices X^(k), with individual factors as columns. A component is an associated set of factors, one from each tensor mode.

Factors are normalized by computing , where r indexes the component and k the factor mode. Collecting these scalars into a vector , we have:

(2)

Since neuronal firing rates are non-negative, we adopt a non-negative tensor factorization (NTF) algorithm [67] based on CP decomposition. This minimizes the squared reconstruction error [68]:

(3)

subject to the non-negativity constraint [68,69]. Non-negative tensor factorization is further advantageous in the sense that it imposes a part-like decomposition [68,69].

For implementation, we use a gradient-based direct optimization approach (OPT) [70] in the Tensor Toolbox [71], modified to allow permutation of the response factors. This allowed us to exploit data limitations, since two narrowly-tuned simple cells (for example) are similar except in their preferred orientations, effectively collapsing the orientation-preference coordinate on the manifolds.

Because drifting and static gratings have different trial lengths and a different number of stimulus variations, two tensors were constructed for each cortical area: one containing the drifting grating responses (in which the stimulus mode consisted of the 5 temporal frequencies) and another for the static grating responses (in which the stimulus mode consisted of the 5 spatial frequencies). The third (response) mode consisted of the vectorized peri-stimulus time histograms obtained by concatenating the responses to all directions (8, for drifting gratings) or orientations (6, for static gratings at the preferred spatial phase). Thus, the drifting-grating tensor had dimensions neurons temporal frequencies (directions time), while the static-grating tensor had dimensions neurons spatial frequencies (orientations time).

Importantly, orientation (for static gratings) and direction (for drifting gratings) were incorporated into the temporal mode rather than treated as independent stimulus modes. This design allows the permuted tensor decomposition to normalize away the particular preferred orientation or direction of individual neurons, so that neurons with similar tuning structure but different preferred orientations can be treated as functionally similar. By contrast, spatial and temporal frequencies were retained as explicit stimulus variables because their organization across the neuronal population was a primary object of study. Consequently, the manifold is designed to be insensitive to orientation/direction preference itself, while remaining sensitive to properties such as tuning sharpness, response dynamics, and frequency selectivity.

Finally, we also built a tensor combining all visual areas. An equal number of neurons (n = 500) were sampled uniformly at random from each area, for a total of n = 3000. All additional steps were kept the same as those for building the manifolds for individual areas. The results are shown in S12 Fig.

Neural encoding space.

The first mode of the tensor decomposition consists of the neural factors, which together form the neural matrix (factors as columns). Letting denote the diagonal matrix containing the component magnitudes , the reconstruction of the tensor can be written as

(4)

where the outer product between the stimulus and response factors of each component forms a vectorized stimulus-response basis. The matricized tensor with respect to the neural mode can therefore be expressed as

(5)

where each column of B is obtained by vectorizing the outer product of the corresponding stimulus and response factors. Thus, each neuron is represented as a linear combination of these stimulus-response vectors. We define the scaled neural matrix

(6)

whose rows provide a low-dimensional representation of the neurons in the neural encoding space.

The number of tensor components was selected using the normalized explained-variance criterion introduced in [1]. Briefly, candidate decompositions with different numbers of components were computed, and principal component analysis was applied to the corresponding neural matrices. The number of tensor components was chosen as the smallest value for which the normalized explained variance of the neural matrix reached a plateau, thereby avoiding underfactoring while also avoiding overfactoring, which tends to split meaningful components or model noise. As in [1], multiple random initializations were evaluated for each candidate number of components, and the solution with the lowest reconstruction error was retained.

For each visual area, separate neural matrices were obtained from the drifting-grating and static-grating tensor decompositions. These matrices were concatenated column-wise to form a single neural representation for each neuron. Principal component analysis (PCA) was then applied to the combined matrix. The minimum number of principal components required to explain at least 80% of the variance was retained for manifold inference.

Natural scene vs. static grating selectivity ratio.

Selectivity ratios for individual neurons were computed as the natural logarithm of the ratio between their firing rate to natural scenes and their firing rate to static gratings. Thus a selectivity ratio of 0 meant the neuron produced exactly the same magnitude of response to both stimulus classes; and a positive (respectively, negative) value indicated a stronger response to natural scenes (resp., static gratings). The magnitudes of responses to these two stimulus classes are readily comparable because both stimuli consist of static images presented during the same time interval of 250 ms.

Following the methodology from the Allen Brain Observatory SDK [75], mean firing rates (FR) for each stimulus class (namely, static gratings and natural scenes) were computed as the mean number of spikes per second of stimulus presentation across all trials of the same stimulus, averaged across all stimulus variations within the same class. For static gratings, ‘variations’ meant different orientation (6), spatial frequency (5), and phase (4); for natural scenes, ‘variations’ meant each of 118 scenes used. We also computed the maximum FR for a given stimulus class as the maximum averaged FR across all stimulus variations (in other words, the FR to the preferred stimulus variation). Thus, two ratios were computed: one using the ‘mean’ responses (“mean FR ratios”), and another using the ‘maximum’ responses (“max FR ratios”). As a complementary analysis, we also computed population mean FR to natural scenes and static gratings separately as a function of preferred spatial frequency (S18 Fig).

To assess whether the observed U-shaped relationship between preferred spatial frequency and the ratio of responses to natural scenes versus static gratings could arise by chance, we performed a permutation test. For each visual area, natural-scene firing rates were randomly shuffled across neurons while static-grating responses and preferred spatial-frequency labels were left unchanged. For each permutation, we recomputed the spatial-frequency-dependent ratio curve and quantified its U-shape using the statistic

(7)

where denotes the mean log(NS/SG) ratio of neurons preferring the indicated spatial frequency. Positive values indicate larger natural-scene preference among neurons preferring extreme spatial frequencies relative to intermediate spatial frequencies. A null distribution was obtained from 10,000 independent permutations, and permutation-test p-values were computed as the fraction of null U-statistics greater than or equal to the observed value.

For the layer-specific analyses shown in Fig 8, one-sample t-tests against zero were performed on the distributions of log response ratios. Because these analyses involved multiple simultaneous comparisons across visual areas, cortical layers, and ratio measures, p-values were corrected using the Benjamini–Hochberg false discovery rate procedure across all 48 tests. Effect sizes were quantified using Cohen’s d, computed as the mean log-ratio divided by its sample standard deviation (see S1 Table).

Additional metrics.

We used precomputed stimulus metrics available from the Allen Brain Observatory SDK [75] for obtaining firing rate, global orientation selectivity index (OSI), preferred spatial frequency (drifting gratings), and preferred phase (static gratings) for each unit. The Allen gOSI metric is defined as

(8)

where denotes the mean response to a drifting grating of orientation . Values range from 0 (non-selective) to 1 (maximally orientation-selective).

Receptive field position and area estimation followed the Allen SDK methodology of using response maps to localized Gabor stimuli and fitting those to a 2-D Gaussian distribution. Laminar position and mean electrical waveform were also obtained directly from the Allen SDK. Individual neurons were classified into broad spiking (putative excitatory) (VISp n = 944, VISpm n = 397, VISal n = 726, VISam n = 669, VISrl n = 634, VISlm n = 487) or narrow spiking (putative inhibitory) (VISp n = 306, VISpm n = 124, VISal n = 227, VISam n = 236, VISrl n = 171, VISlm n = 110) based on their extracellular spike waveform (see [10]). In addition, a stimulus entropy index (Fig 2c) was defined as , where H is the base-2 entropy of the vector containing the relative response magnitudes (divided by their sum) of a neuron to the 5 temporal frequencies used in the drifting gratings experiments. It therefore ranges between 0 (case in which the neuron responds to a single stimulus) and 5 (when it responds with uniform magnitude to all stimuli).