Intrinsic dimension estimation

Consider a set of P points embedded in an ambient space of dimension N. Assuming that the points are contained, exactly or approximately, within a manifold or hypersurface of intrinsic dimension D < N, one can try to estimate D. There exists a wide array of ID estimation methods (for a classic review, see [55]). To be suitable for characterizing neural activity manifolds, an estimator should ideally work well for non-linear, curved manifolds and a large range of dimensions, including high dimensions where the manifold is undersampled.

Linear ID estimation methods are based on the spectrum of the covariance matrix of the points. Let be the eigenvalues of the covariance matrix in decreasing order (). The corresponding eigenvectors are commonly termed principal components (PCs) [56]. The ID can be determined by assessing the ‘relevant number’ of PCs. The most common method consists in searching for a linear subspace accounting for a large fraction of the total variance , i.e., identifying the minimum K such that

(1)

(typical choices are ). Another method computes the participation ratio [57]

(2)

ID_PR corresponds to an ‘effective number’ of relevant dimensions, as typically with the entropy of the covariance matrix spectrum. Finally, parallel analysis (PA) [58] determines the number of significant PCs through a null model. Surrogate data are constructed by resampling the original data by shuffling each data coordinate independently. For each surrogate dataset, the eigenvalues of the covariance matrix are computed. By considering a large number of shuffling, one obtains a null distribution of eigenvalues. Let be the -th percentile of the null distribution (typically, ). One finally considers as significant all original eigenvalues exceeding this critical value,

(3)

Linear methods are appropriate only when the manifold is flat, i.e., it lies on a hyperplane. In the presence of curvature, they can largely overestimate the ID.

Among non-linear methods, the most commonly used are fractal [59,60] and nearest-neighbor methods [45–47]. Fractal methods are based on the fact that the number of points found within a spherical volume of radius r follows an ID-dependent scaling law. The most common example is the Correlation Dimension (CorrDim) method first proposed in the classic paper by Grassberger and Procaccia [59]. CorrDim focuses on the correlation integral (the density of points within a given cutoff distance r):

(4)

If the points are sampled from a distribution on a D-dimensional manifold, then for . Thus, one can extract the ID by measuring the slope of the linear fit of as a function of r in a log-log scale (). This method can work reasonably well for non-linear manifolds, but is fragile with respect to undersampling, as it requires a number of points scaling exponentially with the ID (“curse of dimensionality”), and thus underestimates high IDs. Nearest-neighbor methods are based on the fact that distances between nearest neighbors obey statistical relations that depend parametrically on the ID. The max-likelihood estimator [45] assumes that the data are distributed on a D-dimensional manifold and that the density is uniform in a neighborhood containing the first K neighbors of each point. Under this assumption, consider where r_i,k are the distances of the k-th neighbors of point x_i in the data. One can show that

(5)

from which one can estimate D with a maximum likelihood approach on the empirical . The two-nearest-neighbor (Two-NN) estimator [47] restricts the uniform density assumption to the neighborhood containing only the first 2 neighbors of each point. Under this assumption, one can show that

(6)

and one can infer D through a non-linear fit of the empirical distribution of . Also nearest neighbor methods incur the curse of dimensionality.

The FCI estimator

The full correlation integral (FCI) estimator proposed by Erba et al. [48] was shown to be remarkably robust to undersampling, being able to correctly estimate large IDs even in the extreme case P < D. The starting point of the method is provided by CorrDim. To address the curse of dimensionality, Erba et al. made the more restrictive assumption that points are sampled from a rotationally-invariant probability distribution on a D-dimensional hyperplane. In this case, after normalization of each point to unit norm, the data must lie on a D – 1-dimensional hypersphere (e.g., a circle for D = 2, a spherical surface for D = 3), and one can exactly calculate the average correlation integral as a function of r:

(7)

where ₂F₁ is the (2,1)-hypergeometric function and is the solid angle in dimension D. To stress that this analytic formula holds away from the small radius limit of CorrDim, the average correlation integral is named Full Correlation Integral (FCI). With this definition, the FCI, as a function of r, has a sigmoidal shape with a slope that depends on D. Performing a non-linear regression on the empirical density of neighbors (4) using the FCI Eq (7) results in an algorithm to determine the ID, shown in Fig 1. In detail, the steps are:

1. Center and normalize the data (Fig 1A,1B). Calculate the center of mass b of the empirical data as and subtract this quantity from each data point, obtaining a data set centered at the origin. Normalize each sample as .
2. Fit the FCI. Measure the empirical correlation integral of the dataset at different radii r (Fig 1C) and perform a non-linear regression of this empirical neighbors density using the average FCI in Eq (7) as the nonlinear model, and D as the free parameter (Fig 1D); increase the estimate by one, to reinstate the degree of freedom removed during the normalization step.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The Full Correlation Integral (FCI) method.

Panels A-D illustrate the basic steps of the FCI method for ID estimation. A The original D-dimensional dataset B Centering and normalization of the dataset: the data points are projected on an unitary sphere in D – 1 dimensions. C The empirical correlation integral of the dataset, , is measured as a function of the normalized radius r/r₀. This amounts to computing how many pairs of points are within a distance r. D ID estimation: is fitted with its theoretical model given by Eq (7), which depends parametrically on the ID. To obtain the true ID, the estimated ID is increased by one, compensating the removal of one degree of freedom during the normalization step.

https://doi.org/10.1371/journal.pcbi.1014162.g001

Strong deviations from the predicted functional form of the FCI can be detected by the goodness of fit (GoF), measured as the root-mean-squared deviation of the empirical FCI from its best fit using Eq (7) (in S2 Fig, we report examples of fits with different values of GoF).

By construction, the FCI method is exact for linearly embedded Euclidean spaces sampled with a rotationally-invariant probability distribution. Erba et al. showed that the method is remarkably robust with respect to violations of the rotational invariance assumption. In particular, they showed that FCI estimates work very well even in some cases of strongly non-isotropic distributions (e.g., points sampled from corners of D-dimensional hypercubes). In addition, estimates are very robust with respect to undersampling, as correct estimates of can be obtained from as little as points.

Recurrent neural networks performing cog-tasks

RNNs have become a customary tool of computational neuroscience research [61–64]. We trained RNNs to solve the ‘cog-task’ battery first introduced by Yang et al. [49]. For simplicity, we maintained the same RNN architecture and dynamics as originally proposed.

The network has N_rec = 256 recurrent units in the hidden layer coupled with N_in=65 input units and N_out=33 output units. The input layer is composed of one fixation unit and two rings of 32 units representing two stimuli. The output layer has one fixation unit and a ring of 32 units representing motor output.

In the rate model of network activity, each neuron is represented by a continuous variable representing the neuron’s membrane potential. The dynamics are described by the equation:

(8)

Here, represents the neuronal time constant, reflecting the slow synaptic dynamics driven by the NMDA receptors. The variable denotes input to the network, while is the bias or background input. The function captures the neuronal nonlinearity, and represents N_rec independent Gaussian white noise processes with zero mean and unit variance, and quantifies the noise intensity. A typical Softplus function is used, , which after re-parametrization is very similar to the neuronal nonlinearity (frequency-current curve) commonly used in neural circuit models. A set of output units z read out nonlinearly from the network,

(9)

where is the logistic function that limits output activities between 0 and 1. Wⁱⁿ, W^rec, W^out are the input, recurrent, and output connection matrices, respectively. After using the first-order Euler approximation with a time-discretization step , we have

(10)

Here, , and N(0,1) represent the standard normal distribution. We use a discretization step . We imposed no constraint on the sign or structure of the weight matrices Wⁱⁿ, W^rec, W^out. The network and training were implemented in TensorFlow [65]. The network receives three types of noisy input,

(11)

The input noise strength is set to . The fixation input u_fix is usually at the high value of 1 when the network is expected to maintain fixation, and changes to 0 when the network is supposed to react. The stimulus inputs u_mod1 and u_mod2 represent one-dimensional circular variables characterized by the angle around a circle. This mirrors what happens in typical experiments with primates, involving directional stimuli and responses (for instance, the stimulus is a cloud of dots moving in a given direction, and the animal is asked to move the arm in the same direction). There are 32 units in each of the corresponding stimulus rings, with their preferred directions evenly distributed from 0 to (). For unit i, which has a preferred direction , its activity in response to a stimulus at direction is given by

(12)

where represents the stimulus intensity. For multiple stimuli, input activities are summed.

The tasks used here are the same as those used by Yang et al. [49]. They selected 20 interrelated tasks, representing common tasks used in neurophysiological studies on nonhuman animals, and useful to understanding basic cognitive processes such as memory-guided response, simple perceptual decision making (DM), context-dependent DM, multisensory integration, parametric working memory, inhibitory control, delayed match-to-sample and delayed match-to-category tasks. The tasks can be divided into three families: the Go, Decision Making (DM), and Matching families.

Synthetic neural recordings

Altan et al. [31] created synthetic data to investigate the performance of ID estimators on neural manifolds. In particular, they constructed synthetic data of known dimensionality, mimicking the properties of multi-electrode neural activity recorded in the primary motor cortex of macaques. Data generation starts by extracting samples from an empirical distribution of firing rates obtained from multi-electrode array recordings of the macaque primary motor cortex [66] with a binning of 50 ms. Samples are extracted randomly across all recorded neurons and time bins, resulting in temporally uncorrelated samples. The d-dimensional data are then temporally smoothed (with a Gaussian kernel of s.d. 50 ms) and multiplied by a mixing matrix with entries randomly selected from a Gaussian distribution of zero mean and unit variance. This produces a dataset X composed of M samples of dimension N. This embedding step does not change the intrinsic dimension d. The value of d ranges from 3 to 40, while N = 96 is similar to the number of electrodes in multi-electrode measurements.

In order to produce a dataset with strongly non-linear properties, an exponential nonlinearity is computed on each of the N variables in X, yielding a data set

(13)

where determines the degree of nonlinearity. Also this non-linear embedding procedure preserves the ID. However, since the transformation is not an isometry (it does not preserve distances), it may significantly distort neighborhood relations (i.e., points that are near before the transformation may be mapped to points that are far apart), and for a finite sampling retrieving the correct ID may be significantly hard.

Responses to visual stimuli in mouse V1

We reanalyzed the data from [22], which were made available through Figshare [67]. For detailed methods we refer to the original study [22]. Here we provide a brief summary of the salient elements. Animals. While the dataset includes data from 8 mice, here we restricted attention to one animal for which responses to several stimulus sets were available. The mouse (‘M170714_MP032’) was bred to express tdTomato in inhibitory neurons, and GCaMP6s was expressed virally, indentifying excitatory neurons by lack of tdTomato expression. Imaging In each of 4 separate recordings, neural activity was recorded through a two-photon microscope using multi-plane acquisition with planes 30 to apart; 10 or 12 planes were acquired sequentially, scanning the entire stack repeatedly at 3 Hz or 2.5 Hz. The mouse was free to run on an air-floating ball; a field of view was selected such that neurons could be observed, with clear calcium transients and a retinotopic location. In total, (mean ± s.d.) neurons were obtained in the recordings. Stimuli. We analyzed four separate recordings. In two recordings, stimuli consisted of different normalized images from the ImageNet database [68], with 2,800 different images used in total (the researchers manually selected images that had a mix of low and high spatial frequencies and that did not consist of more than 50% uniform background. All images were uniformly contrast-normalized by subtracting the local mean brightness and dividing by the local mean contrast). The 2,800 natural image stimuli were displayed twice in a recording in two blocks of the same randomized order. All stimuli were presented for 0.5 s with a random interstimulus interval between 0.3 and 1.1 s consisting of a gray-screen. In one recording, a smaller set of 32 images from the same set was presented in a randomized order 114 times, with similar presentation parameters. Finally, in one recording drifting gratings of 32 directions, spaced evenly at 11.25°, were presented 117 times each, lasting 0.5 s each, and with a gray-screen inter-stimulus interval between 0.3 and 1.1 s. The spatial frequency of the stimuli was 0.05 cycles per degree and their temporal frequency was 2 Hz. Preprocessing. Calcium movie data were processed using the Suite2p toolbox to estimate spike rates of neurons. Underlying neural activity was estimated using nonnegative spike deconvolution. These deconvolved traces were normalized to the mean and SD of their activity during a 30-min period of gray-screen spontaneous activity. For further detail, please see the original study. Stimulus responses were computed from the first two frames acquired after stimulus onset for each plane. All analyses done in this paper were performed on the preprocessed data available on figshare [67].

ID and dimensionality reduction

Once an estimate of the ID is achieved, it can be coupled with a dimensionality reduction method to obtain a lower-dimensional representation of the data. For nonlinear manifolds, there are several well-established methods to do so. For instance, Locally Linear Embedding (LLE) [69] assumes the data lie on a smooth manifold of dimension D and that each point can be approximated by a linear combination of its nearest neighbors. For each data point x_i, LLE computes reconstruction weights w_ij by minimizing

These weights capture the local geometry of the data. LLE then finds a D-dimensional embedding y_i that preserves these relationships by minimizing

which leads to an eigenvalue problem. The resulting embedding preserves local neighborhood structure. Isomap [70] is another common nonlinear dimensionality reduction technique, similar to LLE in spirit. Isomap first uses a graph representation of the data to approximate geodesic distances on a manifold. It then applies classical multidimensional scaling on the geodesic distances to project the data in D dimensions. Recently, an extension of Isomap working for high-dimensional data sets has been proposed [71]. An alternative to classical methods is offered by deep autoencoders [72] - neural networks that learn compact, low-dimensional representations of data by training an encoder–decoder architecture to reconstruct the input. By stacking multiple nonlinear layers, they can capture complex, hierarchical structures that other methods cannot.

The local FCI pipeline

We start by presenting our pipeline for intrinsic dimension (ID) estimation. Our method builds on the Full Correlation Integral (FCI) estimator [48] (Methods). FCI assumes an isotropic point distribution on a hyperplane centered on the center-of-mass of the points. While FCI is exceptionally robust in high dimension compared to other methods, it can give suboptimal performance when its assumptions are violated, i.e., in presence of curvature or violations of the isotropy assumptions. Ref  [48] provided preliminary evidence that both limitations could be addressed by performing ID estimates locally on selected neighborhoods on the manifold. Indeed, i) local neighborhoods approximate the tangent plane to the manifold and are approximately flat and ii) the local distribution of points is typically approximately isotropic, unless a very strong density gradient is present. However, Ref  [48] did not turn this into a systematic procedure for local ID estimation. Here, we leverage such insight to propose a systematic and robust procedure for ID estimation.

The main limitation of FCI can be illustrated with a simple example. When points are sampled from a uniform distribution on a two-dimensional plane, FCI yields a very accurate estimate, D = 1.97 (Fig 2A). When points are uniformly sampled on a Swiss roll - a curved two-dimensional surface embedded in a three-dimensional space - FCI overestimates the dimension, yielding D = 2.73 (Fig 2E). Let us try to address this limitation by exploiting local ID estimates. For any given point x_i in the dataset, one can consider its first K neighbors,

where r_i,K is the distance of the K-th neighbor. Local neighborhoods of different size are shown for the plane in Fig 2B. By restricting the ID estimation to points within the neighborhood , one obtains a local ID estimate D_i,K. One can randomly subselect points and consider neighborhoods of increasing size K centered at each of these points, computing D_i,K for each point and neighborhood size. Estimates D_i,K as a function of the scale K can be collectively organized in a multiscale ID plot. This is shown for the 2-D plane in Fig 2C: estimates at all scales give values in the narrow range [1.7,2.2]. By collecting all local estimates for different K in a single local ID histogram, we obtain a sharp peak around the expected value, 2 (Fig 2D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The Local FCI method.

A A dataset with uniform distribution on a plane, ID = 2. FCI correctly identifies the ID of the dataset, yielding ID_FCI = 2.00. B Local ID estimates can be computed using FCI on neighborhoods of increasing size K. C In the multiscale ID plot, local ID estimates are shown as function of the neighborhood size K. Estimates are colored according to their reliability, as assessed by Goodness-of-Fit (GoF). D All reliable local ID estimates for different K are collected in the local ID histogram. The mode of local ID estimates (the peak of the histogram) yields a single overall ID estimate. We call this ID estimation method the local FCI estimation (lFCI) method. For the 2-D plane, we obtain ID_lFCI = 1.97 [1.89,2.05]. E A dataset with points uniformly distributed on a Swiss Roll. Even though the ID is 2, FCI gives ID_FCI = 2.73. The global FCI overestimates the ID due to the curvature of the manifold. F Neighborhoods of small size ( approximate well the local tangent plane to the manifold. The center-of-mass is located within the neighborhood (low ). G As the neighborhood size increases, neighborhoods depart from the local tangent plane, becoming curved and even possibly disconnected. As a result, the center-of-mass is located outside of the neighborhood (high ). H The local ID histogram for the Swiss Roll shows a prominent peak at 2. We obtain ID_lFCI = 2.04 [1.98, 2.11]. I Distribution of the ratio between distance of the center-of-mass and the average distance between points, , for the different neighborhoods sizes. The value of increases at the increasing of the neighborhood size and, around , most of the neighborhoods become curved with ( is the red line in the plot). In green the median value of . J Distribution of the GoF values for the different neighborhoods sizes. The red line is the threshold chosen as the smallest 99^th percentile of the GoF distributions for the different neighborhoods sizes. GoF_thr = 0.01. The neighborhoods with size are the less reliable. K In the multiscale ID plot for the Swiss Roll, we observe a sharp transition: for , local ID estimates are close to 2, while for larger K they approach 3. However, we can discriminate bona fide estimates ( and GoF < 0.01), shown in black, from unreliable ones ( or GoF > 0.01), shown in red. For , most estimates are unreliable.

https://doi.org/10.1371/journal.pcbi.1014162.g002

When replicating this procedure on the Swiss roll, we observe that estimates D_i,K remain around 2 for small neighborhood sizes (), while they progressively approach the value 3 for larger neighborhoods (Fig 2H). This is rather unsurprising: for sufficiently small K, the neighborhoods remain nearly flat, approximating the local tangent plane of the manifold, and local ID estimates correspond to the manifold’s proper ID; for larger K, the neighborhoods become curved and the estimates converge to the standard global FCI estimate. This suggests that a proper ID estimate could be obtained from the collection of local estimates, removing curved neighborhoods from the analysis.

Two metrics can track the departure of local neighborhoods from the tangent plane. The first is the position of the neighborhood’s center-of-mass, . For a flat neighborhood, the center-of-mass b_i,K lies among the points of the neighborhood: as a result, the center-of-mass’s distance from the closest point in the neighborhood is of the same order as the typical distance between the nearest neighbors within the neighborhood. For a curved neighborhood, instead, b_i,K is displaced from the neighborhood (for an extreme case, think of a spherical surface: its center-of-mass is the sphere’s center, located far off all surface points). This behavior is effectively captured by the metric

(14)

which is simply the distance between the center-of-mass and the closest neighborhood point, normalized by the average nearest-neighbor distance in b_i,K. For flat neighborhoods, we expect , while curved neighborhoods will give . The second metric is the FCI Goodness-of-Fit (GoF) (Methods), measured as the average discrepancy between the empirical FCI curve and its fit (i.e., larger values correspond to worse fit quality). Because the center-of-mass is located off the manifold, the points are not distributed isotropically around it, as assumed by the FCI model. Therefore, the fit quality is often poor.

The effectiveness of these metrics can be exemplified by considering two neighborhoods of different sizes on the Swiss Roll. In Fig 2F, we show a neighborhood with K = 500. The neighborhood approximates the tangent plane at x_i and it is nearly flat, yielding . Correspondingly, we obtain a good fit (GoF = 0.003) and an accurate ID estimate D_i,K = 2.04. In Fig 2G, we show a larger neighborhood with K = 4000. The neighborhood strongly deviates from the tangent plane at x_i, yielding . Correspondingly, we obtain a much worse fit (GoF = 0.022) and an excessive ID estimate D_i,K = 2.65. The just exemplified behavior is systematic, as shown in Fig 2J. For small neighborhoods () we invariably obtain . For larger ones (), progressively grows and the fit quality degrades.

Extensive tests on data sampled on flat manifolds showed that remains in the range [0,2] with near certainty (S1 Text and S8 Fig). Therefore, is a robust and conservative criterion for curvature detection, and we can consider local ID estimates as unreliable (due to curvature) if . The range of GoF values, instead, can widely vary within a dataset, even a flat one. Neighborhoods of specific sizes K can yield much worse GoF than others not only because they are more curved, as in the case of the Swiss Roll, but also because isotropy violations become more prominent (S1 Text and S9 Fig). When this wide variability occurs, in the sake of accuracy it is reasonable to restrict attention to the best local estimates. To this aim, we may identify a ‘reference’ GoF distribution by considering the size K at which the best (lowest) GoF is observed, and consider local estimates as unreliable (due to comparatively bad fit quality) if the corresponding GoF lies out of this distribution. In the multiscale ID plot for the Swiss Roll, (Fig 1K) we display reliable estimates in black and unreliable ones in red. Nearly all reliable ID estimates are around 2; conversely, excessive () estimates are nearly always spotted as unreliable. Congruently, while the local ID histogram is bimodal, with a peak at 2 (determined by small neighborhoods) and a peak at (determined by large neighborhoods), excluding unreliable estimates suppresses the peak at 3 (Fig 2H).

We thus propose the following local FCI (lFCI) pipeline:

1. compute local estimates D_i,K on randomly selected neighborhoods with different values of K.
2. identify the scale yielding estimates with the lowest GoF, and compute the 99% percentile of the GoF distribution for neighborhoods of size K.
3. Discard estimates corresponding to sizable curvature () or comparatively poor fit quality (GoF )
4. Collect all local ID estimates in a histogram and take its highest peak as the best estimate of the overall ID.
5. Specify a plausible range for ID by reporting the 10th and 90th percentile of the local ID estimates.

This pipeline is expected to work well for sufficiently well-sampled Riemannian manifolds, such that local neighborhoods will approximate an isotropic sampling on the manifold’s tangent plane.

The computational complexity of lFCI scales as (S1 Text). As N and D are fixed by the data set, the major free parameter entering the computational cost is the number of centers, M. For our results, we used M = 100 centers, which ensures an approximate covering of the whole dataset even when using small neighborhoods (). Running the code on a Desktop workstation with 8 Intel i7-8550U cores for a dataset of 10000 points took minutes.

ID of neural manifolds of recurrent neural networks performing the cog-task battery

We trained recurrent neural networks (RNNs) to perform the ‘cog-tasks’ battery [49] (Fig 3A). For each task, we trained M = 10 networks. After training, each RNN performed N = 200 trials of the task. For each network, we defined the neural manifold as the set of configurations (in the space of activities of the recurrent layer neurons) visited by the network across the N trials. We characterized each manifold with common ID estimators.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Intrinsic dimension of the neural manifold in the React-Go task.

A Schematic structure of the RNN used for the Cog-Task battery. The network has 65 input units: two sets of 32 channels each, , respectively encode two angular stimuli that jointly determine the network’s response; the fixation input u_fix instructs the network on when to respond. The recurrent layer has 256 units. The output layer has 33 units: 32 units represent an (angular) motor response, z, while the last unit represents a ‘fixation’ response z_fix. B Input, output and network activity for the RT-Go task. In this task, a single angular stimulus is presented, and the network should immediately give a response in the same direction. C Variance explained by successive principal components (PCs) of the network activity. D Projection of the network activity on the first three principal components. The color code stands for time-within-trial (time from trial onset). PC₁ is nearly aligned with time-within-trial. Starting from the origin (null activity), the trajectories reach a ‘ring’ of attractor points (yellow). E Projection of the network activity on the first three principal components. The color code stands for response angle. At any time-within-trial, responses lie on a ‘ring’ encoding the response angle. F Projection onto different principal components. The manifold is not entirely contained in the space spanned by the first 3 principal components, but ‘stretched’ along several orthogonal axes. G The multiscale ID plot shows ID estimates as a function of K, in black (reliable estimates) or red (unreliable ones). At high K > 500, all estimates become unreliable. H The local ID histogram shows a narrow peak around 2. Estimating the mode of the distribution yields ID_lFCI = 2.35 [2.09, 3.89]. A linear ID estimation method, parallel analysis (PA), finds ID_PA = 7. The classical geometric maximum-likelihood estimator (MLE) yields ID_MLE = 3.34, while Two-NN yields ID_TwoNN = 7.03, a severe overestimation due to its high sensitivity to local noise. I We used locally linear embedding (LLE) to find a global low-dimensional representation of the neural manifold for RT-Go. LLE flattens the manifold onto a circle spanned by a radial coordinate, representing time-within-trial, and an angular coordinate, representing response angle.

https://doi.org/10.1371/journal.pcbi.1014162.g003

Before analyzing how the manifold’s ID varies across tasks and trainings, we provide a detailed analysis of an exemplary case: the neural manifold of a RNN performing the React-Go task, where the network must respond with an output in the same direction of the input stimulus (Fig 3B). We first performed principal component analysis. The first principal component explains 64% of the variance, while four subsequent ones explain 5–10% (Fig 3C). As a result, parallel analysis finds ID_PA = 7. In the space defined by the first 3 principal components, the manifold appears to be topologically equivalent to a cone (Fig 3D and 3E). The cone axis is roughly aligned to the first principal component, while for fixed values of the first principal component, the points describe a (deformed) circle in the space spanned by the second and third principal component. In Fig 3D, points are colored according to time-within-trial (t = 0 start of trial; t = 152 end of trial); in Fig 3E, according to response angle. These representations allow a ready explanation for the manifold’s shape. Before the stimulus presentation, the network is silent (r = 0). This configuration corresponds to the vertex of the cone. As the stimulus is presented, the network reacts by reaching a set of attractor states that depend on the response angle and approximately describe a circle. These attractor points are not reached immediately, hence trajectories span a ‘conical’ surface. Importantly, the surface does not lie exactly in the linear space spanned by the first three principal components. In fact, the cone is ‘tilted’ in several dimensions. This is shown in Fig 3F, where we replace the third principal component with successive principal components. As a result, linear estimates of the ID end up to be considerably larger than 3.

In principle, the ‘conical’ surface could be described by an axial coordinate (encoding time-within-trial) and an angular one (encoding response angle). Thus, we would expect ID = 2. The best estimate by lFCI was ID_lFCI = 2.35 (C.I. [2.09,3.89]). A closer inspection through the multiscale ID plot (in 3G) shows that local ID estimates are low for sufficiently small neighborhoods (), while they become much higher for larger neighborhoods, eventually reaching values in the range [10,20]. Yet, estimates obtained for are discarded due to curvature and/or bad GoF. As a result, the local ID histogram has a clear peak around 2 (Fig 3H). The intrinsically 2-dimensional geometry of the manifold can be effectively captured using locally linear embedding (Methods), which yields an explicit 2-dimensional parametrization of the manifold. As shown in Fig 3I, the manifold can be ‘flattened’ on a two dimensional surface parametrized by a radial (time) and an angular (response angle) coordinate. However, the classical maximum likelihood ID estimator yields ID_MLE = 3.34 - considerably lower than parallel analysis, but definitely larger than 2. The Two-NN estimator gives ID_TwoNN = 7.03. As is well known, the maximum-likelihood estimator can be affected by the non-uniformity of the distribution, in this case probably causing an overestimate. The Two-NN estimate, in turn, is strongly biased by small-scale noise [47]. Only by reducing the density of the dataset through decimation (which reduces the effects of small-scale noise increasing the average distance between points), we obtained an estimate ID_TwoNN = 2.05 much more consistent with the lFCI one (S3 Text and S4 Fig).

In summary, even in this simple task, the neural manifold is highly curved and tilted in several dimensions. As a result, linear ID estimation methods considerably overestimate the dimension. On the contrary, lFCI yields an estimate that is congruent with the task structure and the possibility of parametrizing the manifold with two coordinates.

In Fig 4, we show ID estimates obtained when training 10 RNNs to solve each of the tasks. As for lFCI, we show the best estimate. For all tasks in the Go Family, we consistently obtain (Fig 4D). The classical maximum-likelihood estimator gave ID estimates between 3 and 4 (Fig 4B). Two-NN gave larger and quite variable ID estimates, (Fig 4C). When applying decimation, which removes small-scale noise, we obtained estimates consistent with those of lFCI, i.e., around 2.5 (S3 Text and S4D and S4F Fig). This result is consistent with the above analysis of the react-Go task, showing that the manifold is essentially spanned by a circular and a temporal variable. Linear ID estimates by parallel analysis are in general much larger (up to ID_PA = 10) and strongly depend on the task and training instance ( for Go and Anti; for Dly-Go and Dly-Anti; ID_PA = 7 for RT-Go and RT-Anti). Parallel analysis counts all directions on which the manifold has a non-vanishing projection, which broadly corresponds to the number of principal components explaining 90% of the variance (S4A Fig). Another linear ID estimation method, the participation ratio, effectively counts only the most relevant projections. The participation ratio yields indeed lower estimates than parallel analysis (in the range [1,5], but estimates again strongly depend on the task and training instance (S3 Text and S4B Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Intrinsic dimension of neural manifolds for the Cog-Task battery.

For each task in the Cog-Tasks battery, we trained 10 independent recurrent neural networks (RNNs) and computed their ID with different methods. Notably, we compare estimates obtained by the local FCI (lFCI) method with those obtained with a linear ID estimation method, parallel analysis (PA) and two commonly used geometric ID methods, the classical maximum likelihood estimator (MLE) and the two-nearest-neighbors (Two-NN) estimator. (A) ID as the number of significant PCs identified with the parallel analysis (PA). For most tasks, ID estimates are not consistent in different trainings. Most estimates give . (B) ID estimated by the classical maximum likelihood estimator (MLE). MLE gives very consistent estimates, with for most tasks. The only exception is given by the delayed-match-to-category (DMC) task, which, however, is very difficult to train, with most networks learning the task only approximately. (C) For tasks in the ‘Go-Anti’ family, Two-NN gives quite inconsistent estimates, with . For tasks in the ‘Decision Making (DM)’ family, estimates are consistent, with for the non-delayed and for the delayed tasks. For tasks in the ‘MS’ family, we obtain for D(N)MS, and for D(N)MC. (D) The local FCI (lFCI) method gives consistent estimates around 2.5 for all tasks in the ‘Go’ family, and estimates between 3 and 4 for tasks in the DM and MS family. (E) The ID identified by lFCI is always considerably lower than the ID estimate obtained by PA, except for the MS family. This is a reflection of the fact that neural manifolds for the Cog-Tasks battery are generally curved and extended across many linear dimensions, despite having an intrinsic 2- or 3-dimensional geometry. (F) The ID identified by FCI is always lower than the estimate obtained by Two-NN, which is severely affected by small-scale noise.

https://doi.org/10.1371/journal.pcbi.1014162.g004

For the React-Go and React-Anti tasks, the participation ratio can yield estimates close to 1, a direct consequence of the presence of a ‘dominant’ principal component explaining a large fraction of the total variance (the cone ‘axis’ representing time-within-trial).

For tasks in the DM family, most trainings gave for the non-delayed tasks (with a few outliers in the range [4,5]), and for the delayed tasks (with a few outliers around 2). We can trace back this increased dimension to a perturbation of the previously analyzed ‘conical’ manifold structure. Like in the Go family, neural trajectories must encode the response angle in the ‘Go’ phase to produce a correct response. However, in the delayed tasks (where the two stimuli are given in succession) the network must also transiently encode the first stimulus angle, which must be kept in memory until the second stimulus is presented and the correct response can be computed. Trajectories converging to the same attractor points thus form a smeared bundle. This bundle also includes ‘backward’ trajectories: as the stimuli are switched off in the ‘Go’ phase, trajectories do not remain in the vicinity of attractor points but slowly converge back to the origin before the end of the trial (for a detailed example and discussion, see S2 Text and S3 Fig). This ‘smearing’ of trajectories associated to the same attractor determines an additional ‘transversal’ degree of freedom, causing a rise in dimension. This phenomenon also occurs in the non-delayed tasks, where ‘smearing’ is only due to backwards trajectories not coinciding with the forward ones, causing only a minor dimension increase with respect to the Go tasks (). ID Estimates obtained with classical MLE are consistent and slightly above 2 for all DM tasks (4B), suggesting that MLE identifies only the two local dimensions with the strongest variability. ID Estimates obtained with Two-NN are consistently larger than those of lFCI ( for the non-delayed DM tasks, for the delayed ones; Fig 4C). Upon decimation, estimates are in good agreement with those of lFCI (S3 Text and S4D Fig). On the other hand, linear ID estimates are again quite inconsistent over tasks and trainings: parallel analysis generally yields much larger estimates (; S4A Fig), while the participation ratio yields estimates in a similar range (). The participation ratio estimates are not correlated with those of lFCI (S3 Text and S4E Fig), suggesting that the two methods capture different geometrical properties: ID_PR captures global variation along different orthogonal axes, and ID_lFCI captures local variation.

Finally, for tasks of the MS family, we consistently obtain for DMS and DNMS. This result is fully consistent with estimates given by maximum-likelihood estimator and Two-NN (after decimation). Parallel analysis gives inconsistent and high estimates, while the participation ratio gives estimates close to 1, due to a dominant PC. For the DMC and DNCM tasks, most trainings give (except one outlier around 3 for DMC, one outlier slightly below 2 and two outliers between 3 and 4 for DNCM). ID Estimates for DMC are thus more consistent than those of other estimators, while those for DNCM show some variability, which also emerges in maximum-likelihood and Two-NN estimates - perhaps reflecting a difficulty of properly training this task (the accuracy was compared to ).

ID of synthetic neural recordings

Altan et al. [31] introduced a benchmark data set of synthetic neural recordings. As detailed in Methods, the data were generated by considering firing rate data from d independent recording channels, with varying , embedding them in N = 96 dimensions with a linear transformation (linearly embedded data), and finally applying a non-linear transformation (nonlinearly embedded data).

The linearly embedded data present a challenge for classical geometric ID estimators, but not linear ones. While parallel analysis always finds , geometric estimators find for all , a reflection of their (known) difficulty in estimating high dimensionalities. Here, lFCI outperforms geometric methods, giving estimates broadly in line with those of parallel analysis: ID_lFCI =[3.00,5.63,8.87,16.74,29.41] for d = [3,6,10,20,40] (Fig 5B). In fact, this data does not pose significant challenges to lFCI, as reflected in the very well-behaved appearance of the multiscale ID plots and the local ID histograms for d = 6,20,40 (Fig 5D, 5F and 5H). Estimates are reliable (a natural consequence of the non-curved nature of the data) and consistent for all scales K, so that the local ID histogram is sharply peaked. The ID underestimation (which becomes evident for d = 20 and even more so for d = 40) is due to the presence of many dimensions with extremely small variance,  <10^-2 (Fig 5A). In this condition, it is essentially impossible to reliably discriminate these dimensions from small-scale noise. In fact, also parallel analysis tends to identify a number of dimensions lower than d, and ID_lFCI and ID_PA converge for d = 40. To validate this hypothesis, we variance-normalized the data before the embedding. For these equal-variance data, we obtained for all d (S6 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. ID estimation for the synthetic neural data with increasing dimension.

In Ref  [31], neural manifolds were constructed by randomly sampling firing rates from d independent channels in a multielectrode array. Data were then linearly embedded in 96 dimensions (linearly embedded data), and further subjected to a non-linear transformation (nonlinearly embedded data). A Variance explained by the principal components (PCs) of the linearly embedded data. B The ID was estimated by different methods, including parallel analysis (PA), Maximum Likelihood Estimator (MLE), Two-Nearest Neighbors (Two-NN), and the local FCI method; in red the lFCI estimates without our procedure to filter out the unreliable estimates, in green the lFCI results after the refinement. Dashed line indicates the identity. The linearly embedded data are stretched along d orthogonal axes, but the variance is very small () for several components. All ID estimation methods yield for . For , MLE and Two-NN show a non-monotonic behavior, with . PA and lFCI give monotonically increasing estimates for increasing d, with ; the presence of the unreliable results does not affect the final lFCI estimate. C The nonlinearly embedded data give rise to strongly non-smooth manifolds. All ID estimation methods yield for . For , MLE and Two-NN show a non-monotonic behavior, with . lFCI show a monotonically increasing trend, but the ID remains much lower than d (). Keeping the unreliable estimates lFCI gives for all d. D For the linearly embedded data at d = 6, local ID estimates are consistent at all K, with a slightly decreasing trend for increasing K, and we obtain lFCI = 5.51 [5.23, 5.71]. E For the nonlinearly embedded data at d = 6, local ID estimates are consistent at all K, and we obtain lFCI = 5.51 [4.98, 6.25]. F For the linearly embedded data at d = 20, most local ID estimates are consistently in the range [15,20] at all K, with a decreasing trend for increasing K. We obtain lFCI = 16.72 [16.45, 17.06], consistent with the number of PC components with variance >10^-2. G For the nonlinearly embedded data at d = 20, local ID estimates are very variable (in the range [5,20] for all K < 10⁴. Only for do estimates become more consistent with values around 20. H For the linearly embedded data at d = 40, most local ID estimates are consistently in the range [23,34] at all K, with a decreasing trend for increasing K. We obtain lFCI = 29.33 [28.77, 29.97], consistent with the number of PC components with variance >10^-2. I For the nonlinearly embedded data at d = 40, local ID estimates are very variable (in the range [5,40] for all K < 10⁴. Only for do estimates become more consistent with values around 40.

https://doi.org/10.1371/journal.pcbi.1014162.g005

The nonlinearly embedded data have some very peculiar characteristics, which we summarize here (only some were discussed in [31]): i) the data distribution along each dimension is nearly log-normal. ii) The upper tails of the distributions give rise to isolated low-density regions along orthogonal axes (S5A Fig). In these regions, the ‘tangent plane’ has not a dimension equal to the ID, and perhaps is not even well-defined. iii) the density varies dramatically across the dataset (to measure this quantitatively, we used the inverse of the distance of the 10-th neighbor as a proxy of the density). As a result, the manifold contains a highly dense ‘core’ and a large, rarefied ‘periphery’ (this is visualized in S5B and S5D Fig, where we projected the manifolds for d = 20,d = 40 in two dimensions using t-stochastic neighbor embedding [73], showing the local point density).

Due to these joint characteristics, ID estimation becomes a difficult endeavor. Due to i), the covariance structure of the data is strongly influenced by outliers: this makes linear ID estimation unreliable. In fact, parallel analysis first overestimates (for d < 20), then underestimates (for d = 40) the ID. Geometric estimators are bound to underestimate the ID (for d > 10) due to their intrinsic limitations but also the manifold geometry, as nearest-neighbor statistics in the peripheral regions strongly deviates from the predicted distribution. Indeed, the maximum-likelihood estimator and Two-NN give for . In this situation, lFCI can give accurate estimates up to d = 20, while it fails for d = 40, producing a sharp underestimation: ID_lFCI=[2.8,5.8,11.3,20.7,17.2] for d = [3,6,10,20,40]. The problem for d = 40 can be mitigated by relaxing a step in the pipeline, the removal of estimates corresponding to bad GoF, although this comes at the price of some overestimation: ID_lFCI=[2.5,5.8,11.2,22.5,48.7] for d = [3,6,10,20,40]. To better analyze the reasons for this (partial) success, in Fig 5E, 5G and 5I we show the multiscale ID plots and the local ID histograms for d = 6,20,40. For small d (d = 6), the behavior is similar to the one observed in the linear embedding case. Although local ID estimates are slightly more variable (in the range [5,7]), the local ID histogram is still sharply peaked For d = 20,40 local ID estimates appear to be highly variable for most neighborhood sizes K, except . This behavior depends on the dataset geometry: local ID estimates strongly correlate with the density (S5C and S5E Fig). Neighborhoods in the highly dense ‘core’ yield local ID estimates , whereas neighborhoods in the rarefied periphery cannot detect the full dimensionality, systematically yielding lower ID estimates. For large K (), all neighborhoods encompass a large fraction of the whole manifold. ID estimates become accordingly more consistent. However, estimates also become unreliable. This is not due to curvature (we always find in our data, reflecting the absence of curvature), but rather to a bad GoF. In fact, large neighborhood must include regions with variable density, leading to a violation of the isotropy assumption and hence a deviation of the FCI curve from its theoretical shape. The FCI fitting procedure, however, spots a reasonable if slightly overestimated dimension. For d = 20, the local ID histogram has a peak around 20 even if unreliable estimates are discarded, even though the peak is shifted to and becomes more pronounced if they are included (5G). For d = 40, the local ID histogram has a peak around 20 unless unreliable estimates are included (5G), which produces a peak around 45. In synthesis: when considering small neighborhoods, the presence of rarefied regions tends to produce ID underestimation; when using large neighborhoods, this problem is solved but density variations imply a bad GoF and some overestimation. Nevertheless, even for this very challenging manifold, lFCI can yield rich insight into the manifold’s structure and ID.

ID of high-dimensional visual responses

Having extensively benchmarked our method on artificial data in preceding sections, we now analyze real data from 2-photon imaging data of mice exposed to visual stimuli. Data were first discussed in the classical work by Stringer et al. [22,74] and made publicly available [67]. These data set includes responses of ~ 10,000 primary visual cortex neurons to several stimuli, including: 2,800 natural images, 32 natural images, 32 simple grating stimuli, and degraded natural images. Importantly, each visual stimulus was presented at least twice, allowing for a better discrimination of stimulus-related and stimulus-unrelated variability in neural recordings. Ref. [22] characterized the linear ID of neural responses. The ID found was always in line with the size of the stimulus set: for the small stimulus sets (32 images or gratings) it remained of the order of , while for the very high-dimensional stimulus (2800 images), it reached values of the order of . Importantly, Ref  [22] provided solid evidence that this large linear dimension truly represents stimulus-related variability. Leveraging the availability of (at least) two repetitions of the same stimulus, they computed the cross covariance of responses across repetitions, which is provably a good estimate of stimulus-related variance. They then characterized relevant (linear) dimensions as axes with high cross-covariance (a method called cross-validated PCA or cvPCA, S4 Text).

While Stringer et al.’s findings suggest that visual responses are not organized in a low-dimensional hidden manifold, this conclusion cannot be drawn without an analysis of the ID. In fact, a large linear dimension may emerge from a high-ID geometry, but also from a low-ID, highly nonlinear geometry. Note that the latter case is not merely a mathematical curiosity, as some common neural codes were shown to produce exactly this type of highly non-linear manifolds [42], and a recent study showed that a high linear dimension can emerge in recurrent neural networks with a low-dimensional latent dynamics [44]. To adjudicate between these two scenarios, we re-analyzed Stringer et al.’s data using lFCI alongside other ID estimators. We focused our analysis on data from a mouse for whom responses to multiple stimulus sets were recorded. To remove stimulus-unrelated variability, we adopted the same logic as Ref  [22], and we first projected the data onto a (linear) subspace containing 99% of stimulus-related variance, as estimated by cvPCA. While this step effectively removes stimulus-unrelated dimensions, it maintains the non-linear geometry of the stimulus-related responses - if any (projecting a nonlinear manifold onto a hyperplane yields a linear manifold only if all the nonlinearity is contained in the dimensions orthogonal to the hyperplane).

In the two experiments with 2,800 images, the linear dimension (as estimated via PCA with ) is very large, (Fig 6A). This is due to many of the linear dimensions having a very low variance: the effective linear dimension (as estimated by the participation ratio) is nearly one order of magnitude lower, around . We estimated the ID using Two-NN and lFCI. Two-NN yields inconsistent estimates in the two replications - respectively, and . While lFCI estimates display a relatively large interval for both repetitions (, ), the local ID histogram is peaked (Fig 6C) and the best estimates broadly in line with those of the participation ratio, . To properly interpret this discrepancy, one should take into account that Two-NN tends to systematically underestimate the ID when above , and radically so for (S1 Fig) Although it would be tempting to explain the difference between Two-NN estimates and linear dimension estimates in terms of curvature, this explanation is not supported by the data. In fact, lFCI failed to detect significant curvature in the data: the distribution of was entirely contained in the range [0,1] for all neighborhood sizes (the 95% percentile of the distribution was 0.65 in the first replication and 0.65 in the second). The comparatively low estimates given by Two-NN are then probably attributable to a method’s bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. ID of High-dimensional visual responses.

Results of the ID analysis on the mouse visual responses. The data have been processed with cvPCA before the dimensionality analysis. A ID results for the different experiments; B Distribution of the lFCI ID values for the different experiments; C ID values for the different neighborhood sizes for the experiment with 2800 images. The analysis for the curvature shows that the dataset is flat with for all the experiments.

https://doi.org/10.1371/journal.pcbi.1014162.g006

In the experiment with a reduced stimulus set of 32 natural images, both the linear dimension and the ID dropped dramatically with respect to what seen for the full stimulus set. PCA yielded a value , roughly equivalent to the maximum dimension (equal to the size of the stimulus set). Estimates by the other methods were broadly consistent: the participation ratio yields , Two-NN gives , lFCI . Despite the decrease in dimension, responses do not seem to be arranged in a truly low-dimensional manifold. In the experiment with gratings, results were qualitatively similar, except that we observed a further decrease of the effective linear dimension estimated by the participation ratio, which is . The linear dimension estimated with PCA is , close to the maximum possible value. Two-NN gave , .

In summary, our analysis suggests that visual responses to a high-dimensional set of natural images are organized within a high-dimensional, linear manifold with dimension . Even when considering a much smaller stimulus set, or radically simpler stimuli such as grating, there is no evidence of a truly low-dimensional organization. We remark that these conclusions remain valid if one does not preliminarily remove stimulus-unrelated variance (S4 Text and S7 Fig): the main differences are that (i) for the stimulus sets with 2,800 images, the picture is similar but stimulus-unrelated noise leads to even larger dimension estimates (ii) stimulus-unrelated variability outbalances stimulus-related one in the case of the stimulus sets with 32 stimuli, leading to much larger dimension estimates.

Methodology

Here, we introduced a novel ID estimation pipeline based on a local version of the full-correlation integral (Fig 1). Our pipeline, called local FCI or lFCI, computes the ID locally on neighborhoods of variable size, obtaining a robust ID estimate from the histogram of local estimates (Fig 2). Importantly, we prune the histogram from unreliable local estimates obtained on neighborhoods with significant curvature or density variations. As shown by the summary of results in Table 1, our pipeline is versatile enough to reliably cover a wide range of dimensionalities and non-linearities. In all cases analyzed, lFCI performed comparably or better than other methods. Importantly, no other method (among those included in our analysis) was able to produce consistently reliable results in a wide range of dimensionalities. Our pipeline encountered tangible difficulties only in the extreme case of a high-dimensional, highly skewed data distribution; even in this case, relaxing the pipeline yielded correct insight on the ID (Fig 5). Table 2 shows the lFCI results with the confidence interval. It is worth noticing that the True ID is included in the confidence interval. In the linear embedding case, the ID is underestimated due to the presence of some dimensions that have a very small variance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of the results.

https://doi.org/10.1371/journal.pcbi.1014162.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of the results for lFCI.

https://doi.org/10.1371/journal.pcbi.1014162.t002

In future developments, we might integrate this pipeline with suitable denoising procedures to further enhance its robustness. In Ref. [31], Altan et al. proposed the following denoising/estimation scheme: 1) use parallel analysis (PA) to obtain an upper bound on the ID, D_u; 2) denoise the data by first projecting (linearly, via PCA, or non-linearly, via an autoencoder) onto a D_u-dimensional space, and then projecting back into the original space; 3) if nonlinear denoising is more effective than linear denoising (indicating a nonlinear manifold) use a geometric ID estimator; otherwise, use a linear one. Our results show that FCI can yield a more robust upper bound than PA, also valid in a highly nonlinear case. We thus suggest the following modification of Altan et al.’s scheme: 1) use FCI to obtain an upper bound on the ID, D_u; 2) denoise the data 3) compute a handful of local ID estimates for small neighborhoods (e.g., M = 10 centers and ) and average the resulting estimates 4) if there is agreement with the global estimates (small relative error), stop; otherwise, perform to the full lFCI procedure to characterize the ID (note that step 3, which can be performed rapidly at a low computational cost, can spare the computational burden of the complete lFCI procedure when probably not needed).