Visual learning task and the ANN models

We studied visual learning in an orientation discrimination task in both convolutional ANN models (Fig 1A) and human observers (Fig 1B), where the models and observers were trained to classify the rotation direction of a Gabor stimulus relative to a reference orientation. During training, the reference orientation was 145°, and the stimuli could occur in two diagonally opposite positions on the screen, in the upper-left and bottom-right corners. To evaluate generalization in the human observers, we swapped the diagonal that stimuli were presented on to the upper-right and bottom-left corners of the screen and changed the reference orientation of the stimuli during the test to 55° (Fig 1C) [18]. However, testing on a different location in the visual field may not cause specificity in convolutional ANNs, as they may be equivariant to spatial location. To evaluate generalization in the models, we used a Gabor stimulus which maintained the training angle separation but with doubled spatial frequency. Separately, we also evaluated generalization for a different reference angle orientation to confirm the robustness of our results (S2 Fig). We did not use spatial frequency as a transfer condition for the human observers to avoid potential variability induced due to spatial frequency preferences across observers [19,20].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. An ANN model of visual learning.

(A) Schematic of the ANN model of the ventral visual pathway. The network was a convolutional ANN (based on AlexNet) pretrained on object categorization. (B) Schematic of the experimental task. Observers viewed a reference image, target image, and then provided a keypress response. Observers were given feedback during training. (C) Left: Train condition. Stimuli were presented in the upper-left or bottom-right corner, randomly chosen in each trial. The reference orientation was 145°. Right: Transfer condition. Stimuli were presented in the upper-right or bottom-left corner, with a 55° reference orientation. (D) Train accuracy throughout training for models trained on a single angle separation. (E) Transfer accuracy after training. This pattern of generalization (i.e., worse generalization for smaller angle separations) followed the pattern previously observed in humans. (F) Change in the L2-norm of skip connection weights (connecting convolutional layers to the output) throughout training on a 0.5° angle separation. (G) Change in the L2-norm of convolutional weights throughout training on a 0.5° angle separation. Note the difference in scale between (F) and (G). Error bars are not visible.

https://doi.org/10.1371/journal.pcbi.1014553.g001

We used a convolutional ANN model based on AlexNet [21] and pretrained on ImageNet [22] that has demonstrated strong alignment with both monkey [23] and human [24] visual systems and has been used to replicate several behavioural and neuronal attributes of visual learning [17]. Despite its relatively modest size and straightforward architecture, this model performs comparably to more sophisticated, larger architectures in terms of model-brain alignment [24]. Moreover, its relatively small size enables more thorough investigation of both its learning dynamics and post-training representations. We modified the model by incorporating readout weights, or “skip connections”, from every layer of the model to the output decision neuron (Fig 1A). These skip connections enable all hierarchical levels of the model to directly influence the final output, unlike pure feedforward networks where only the final processing stage directly contributes to the output [25]. This is consistent with neurophysiological evidence suggesting that visual areas across different processing stages can directly influence behavioural responses [26,27]. The modified model was fine-tuned on the orientation discrimination task using backpropagation.

First, our ANN model successfully reproduced the main behavioural characteristics of visual learning [3]. Accuracies increased more rapidly during training for tasks with larger angle separations (easier tasks) compared to those with smaller angle separations (harder tasks) (Fig 1D, S1A, S2A, S3A Figs). Moreover, our model replicated the generalization pattern previously observed in humans [3], namely, lower generalization to a transfer condition for smaller angle separations (Fig 1E, S2B, S3B Figs). These results are in general agreement with the previous findings of Wenliang and Seitz [17].

To identify the loci of plasticity underlying learning, we further examined changes in the model’s weights throughout the training process. Our analysis showed that the changes in the L2-norm of the weights in the skip connections (Fig 1F and S3C Fig) or in the gradients of the weight norms (S1D Fig) were approximately 200x larger than those in the between-layer convolutional weights (Fig 1G, S3D, S1E Figs). This observation suggests that learning predominantly occurred in the readout weights, consistent with previous neurophysiological studies indicating that VPL mostly alters the readout of sensory areas rather than their tuning properties [26,28,29]. This observation supports the hypothesis that direct skip connections to downstream association areas channel learning primarily into the readout weights from visual areas, rather than into the sensory representations of stimuli themselves [30]. To confirm this mechanism, we tested a variant of the model where all between-layer connection weights were frozen, allowing changes only in the readout connections. This restricted model exhibited identical behavioural characteristics and generalization patterns (S1B-S1C Fig). Therefore, our subsequent analyses of the model’s connection weights throughout this paper focus exclusively on the learned readout weights.

The effect of training curricula on ANN models and humans

We next investigated how different training curricula affect generalization in a task that typically exhibits poor transfer (i.e., orientation discrimination with small angle separation). We trained both our ANN models and human observers using three distinct approaches: 1) a baseline non-sequential paradigm focused only on the harder version of the task; 2) a shuffled curriculum with randomized difficulty levels; and 3) a sequential easy-to-hard curriculum (Fig 2A). In the sequential curriculum, the models or observers were trained first on an easier version of the task (5° angle separation) for the first half of training, and then a harder version (1° angle separation) for the second half of training. In the shuffled curriculum, we trained on randomly interleaved 5° and 1° angle separations for the entire training process. The baseline non-sequential condition involved training exclusively on the 1° angle separation for an equivalent duration.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. The effects of training curricula across models and observers.

(A) Visualization of curricula that were used to train models and human observers. Dark blue indicates training on 5° angle separations, light blue indicates training on 1° angle separations, and green indicates a transfer condition on 1° angle separations. In a non-sequential curriculum, models and observers are trained only on 1° angle separations; in the shuffled curriculum, they are trained on randomly interleaved 5° and 1° angle separations; and in the sequential curriculum, they are trained for the first half of training on 5° and for the second half of training on 1° angle separations. (B) Transfer accuracy for human observers across curricula. Observers trained on either the shuffled or sequential curricula significantly outperformed observers trained on the non-sequential curriculum, and there was no significant difference between the shuffled and sequential curriculum. (C) Transfer accuracy for models across curricula. Models trained on either the shuffled or sequential curricula significantly outperformed those trained on the non-sequential curriculum, but models trained on the sequential curriculum also significantly outperformed those trained on the shuffled curriculum.

https://doi.org/10.1371/journal.pcbi.1014553.g002

Our results showed that the shuffled and sequential curricula significantly improved generalization compared to the baseline non-sequential condition in both human observers (Fig 2B; shuffled: p = 0.00013; sequential: p = 0.00016) and the ANN model (Fig 2C, S2C, S3E Figs; shuffled: p<<0.0001; sequential: p<<0.0001). However, while the sequential curriculum significantly outperformed the shuffled curriculum in the model (p = 0.0008), interestingly, human observers showed comparable improvement with both curricula, with no significant performance difference between them (p = 0.84). Notably, the shuffled condition in human observers displayed much larger variance than the sequential condition, suggesting substantial individual variability. Further analysis of the learning dynamics provided insight into the discrepancy between the models and observers in the shuffled condition.

Implicit curricula in the shuffled condition predict generalization in ANN models and human observers

We noticed that the human observers in the shuffled condition had substantial variability in their performance relative to those in the sequential condition. We wanted to investigate the source of this variability and additionally attempt to determine the reason for the discrepancy between the performance on the shuffled condition between the human observers and the model.

We first found that observers trained on the shuffled condition appeared to follow an “implicit curriculum”: observers plateaued at approximately 90% accuracy on the easy condition about halfway through training but continued to learn from the hard condition until the end of training (Fig 3A). This suggests that when given randomly interleaved tasks, human observers develop their own implicit easy-to-hard curriculum, consistent with previous observations by Ahissar and Hochstein [3].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. An implicit curriculum arises in human observers and models through shuffled training.

(A) Accuracy throughout training for human observers on the shuffled condition shown separately for easy (5°) and hard (1°) samples. Accuracy on easy samples plateaued at around 90% starting approximately halfway through training, while accuracy on hard samples continued to improve. (B) Parameter gradients for models trained on the shuffled task condition in the first 100 steps of training. Parameter changes occurred predominantly in the easy condition (5°) early in training, then transitioned to greater changes in the hard condition (1°). (C) Transfer accuracy correlated with our curriculum metric across human observers – peak value from panel (D) (r = 0.45, p = 0.026). Our curriculum metric provided a proxy measure for the strength of the easy-to-hard implicit curriculum that the observer followed in the first n easy trials and the first n hard trials. (D) The correlation of transfer accuracy vs. curriculum metric, with varied n (number of trials used to calculate the curriculum metric). n was sampled logarithmically from the range [5, 100]. The correlation was robust for various values of n but decreased for very small and very large n. (E) Transfer accuracy of the models trained on a shuffled condition when different ratios of easy:hard samples were shuffled in for the first 25 steps of training (r = 0.91, p<<0.0001). Learning from a higher ratio of easy samples at the beginning of training correlated with a higher transfer accuracy. Shading on all plots represents 95% CI.

https://doi.org/10.1371/journal.pcbi.1014553.g003

Having observed implicit curricula in human observers, we asked whether similar implicit curricula also emerged in the ANN model during shuffled training. Unlike with humans, we can directly measure how much each task condition (hard vs. easy samples) changes the model’s parameters throughout training. We computed the gradients of the model’s training loss with respect to the model weights, which provided the direct learning signal. We used the mean of the absolute gradient values as a measure of the learning signal that drives the changes in a model’s performance. In the shuffled curriculum, parameter gradients peaked earlier for easy samples, and later for hard samples (Fig 3B and S4A Fig). We observed a similar pattern in the accuracy of each condition (S4B Fig). This revealed an implicit quasi-sequential curriculum in the model: the model learned from easy samples first and then shifted focus to harder samples as training progressed. This finding is consistent with previous work showing that ANNs implicitly learn tasks in a highly consistent order [31,32]. However, the precise dynamics of this implicit curriculum may differ among human observers and between humans and ANNs. While ANNs learn the task in a very consistent order with little variability among model instances, human observers show greater variability in their learning dynamics. We therefore investigated this variability and its relationship with learning generalization.

We defined a “curriculum metric” (see Eq. 1 in Methods) as the mean accuracy on the first n easy trials minus the mean accuracy on the first n hard trials, providing a proxy measure of each observer’s implicit curriculum. We found a positive correlation (r = 0.45, p = 0.026) between the transfer accuracy and our curriculum metric across observers (Fig 3C), indicating that the implicit curriculum is related to generalization: An observer who followed an easy-to-hard implicit curriculum was more likely to generalize better. Interestingly, we found an inverse-U relationship between the correlation value and the number of trials n used for calculating our curriculum metric (Fig 3D). For very small and very large numbers of trials, the measured implicit curriculum does not predict generalization. Performance in initial trials is affected by multiple factors that may not properly reflect observers’ implicit curriculum. Additionally, variability in performance in later training trials is also not predictive of learning generalization. To confirm that the relationship between the transfer accuracy and our curriculum metric is not merely a consequence of the random assignment of easy and hard trials early in training, we measured the correlation between the transfer accuracy and the ratio of easy to hard samples during the first n overall trials. We found no significant correlations (all p-values >0.05 for n sampled logarithmically from the range [5, 100]) indicating that the variability in performance is due to the observers own implicit curriculum, not the explicit random order of samples.

To more directly investigate whether the dynamics of the curriculum explain performance differences, we manipulated the ratio of sample difficulty at the beginning of model training to indirectly control the models’ implicit curriculum dynamics. Unlike humans, models learn tasks in a predictable and consistent order in the shuffled condition without much variation between model instances [31,32]. Therefore, we must manually introduce variability by explicitly manipulating the sample difficulty in order to force a human-like curriculum. Specifically, we repeated our shuffled model training as before, but instead of randomly selecting task difficulty conditions (i.e., angle separations) for each training step, we modified the ratio of easy:hard task difficulties for the first 25 steps of training. The number of steps was chosen because the models’ parameter gradients on the easy condition drop to approximately zero in 25 training steps (Fig 3B). This concept is similar to the idea of “pacing” in curriculum learning for ANNs, which controls the amount of training examples sampled from each difficulty as training progresses [33,34].

We found a very strong positive correlation (r = 0.91, p<<0.0001) between the ratio of easy:hard samples and the transfer accuracy (Fig 3E), replicating our findings in the human observers (Fig 3C). Additionally, models trained on ratios with a higher proportion of easy to hard samples performed better than a pure shuffled curriculum (1:1 ratio) (Fig 3E). This suggests that forcing a “more sequential” curriculum in the shuffled condition improves generalization, and the implicit curriculum that the model discovers through gradient descent-based learning (Fig 3B) is perhaps sub-optimal for the task.

Readout subspace dimensionality explains generalization variability across curricula

Most learning in our model occurred in the readout weights that connected different network layers to the output (Fig 1F-1G, S1D-S1E and S3C-S3D Figs). Throughout training, each neuron received readout weights that ultimately determined its contribution to the model’s output. The neurons with the largest contributions and their visual feature preferences form the representational space that determines the model’s behaviour. We hypothesized that generalization depends primarily on the dimensionality of this feature space – specifically, that learning curricula establishing low-dimensional readout subspaces (recruiting fewer distinct visual features) result in better generalization. We tested this hypothesis by analyzing the readout weights of our ANN model after training.

First, after applying dimensionality reduction (Principal Component Analysis) on the readout weight vectors, we observed that readout weights followed distinct optimization trajectories for each curriculum throughout learning (Fig 4A), indicating differences in the final readout weight patterns across the training paradigms. These readout weights determine neuronal contributions across different layers to the output (the readout subspace). Although all models, regardless of their training paradigm (non-sequential, sequential, or shuffled), were performing the same task of small-angle discrimination at the end of their training, they read out from distinct subspaces of the representational space to solve the task due to their different training histories. These distinct readout subspaces underlie the variability in generalization across learning curricula.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Readout subspace dimensionality determines generalization.

(A) Projection of model readout weights onto their first two principal components for all three curricula at the beginning (marked by a black star), middle, and end (marked by black circles) of training. Arrows show the direction of training progression in weight space; different curricula follow different trajectories. (B) Histogram of all non-zero readout weights at the end of training. The majority of weights were zero, with only a small portion of non-zero weights. Distributions for the three curricula visually appear to overlap almost completely, and standard error is not visible. (C) Accuracy drop per neuron lesioned, plotted against the number of neurons lesioned. The accuracy drop per neuron plateaued at about N = 150, meaning that the readout subspace was mostly affected by about 150 neurons. (D) Dimensionality of the 150-neuron readout subspace, measured by the effective rank, for non-sequential, sequential, and shuffled curricula (r = -0.69, p<<0.0001). Curricula leading to a low-dimensional readout subspace had a higher transfer accuracy. Results for other subspace sizes are shown in S5 Fig. (E) Dimensionality of the subspace created by sampling 150 random neurons from the readout. There is no apparent correlation in transfer accuracy and effective rank (r = 0.25, p = 0.058). (F) Negative correlation of the dimensionality of the readout subspace for shuffled curricula with different easy:hard sample ratios with transfer accuracy (r = -0.95, p<<0.0001). Models which were forced to learn from easy samples early in training had lower-dimensional readout subspaces and exhibited better generalization.

https://doi.org/10.1371/journal.pcbi.1014553.g004

To further characterize these readout subspaces, we examined the distribution patterns of the learned weights. Analysis of learned readout weight distributions revealed high skewness in all models, with only a small portion of non-zero weights (Fig 4B). This suggests that a small subset of neurons, with larger readout weights, mainly determined the readout subspace of each model. To identify these “influential” neurons, we lesioned the neurons with the largest absolute readout weights in increments by removing their weights to the readout and quantified the effect of this lesioning on transfer accuracy. We found that the contribution to transfer accuracy per neuron asymptotically reached zero after approximately 150 neurons were lesioned, indicating that ~150 influential neurons determined the model’s readout subspace (Fig 4C). However, our results are invariant to a range of subspace sizes (S5 Fig). We also repeated this analysis by choosing 150 neurons with the largest individual functional contribution to the transfer accuracy to confirm that our results are unaffected (S6 Fig).

Given these influential neurons, we could test our hypothesis: how many independent visual feature dimensions exist in the readout subspace of each model? In our non-stochastic model, this dimensionality reflects the number of neurons with dissimilar visual selectivity (different orientations, spatial frequencies, textures, etc.) that collectively form the model’s readout subspace. Larger numbers of neurons with diverse tuning properties contributing to the readout create higher-dimensional readout subspaces. According to our hypothesis, the observed generalization variability across training curricula stems from differences in readout subspace dimensionality.

To test this hypothesis, we estimated the dimensionality of readout subspaces across our three base curricula. From each model, we collected the activations of the 150 influential neurons with the largest absolute weight values in response to 100 natural images [22]. We estimated the dimensionality of each readout subspace using the effective rank [35,36]. We found that the effective ranks were negatively correlated with the transfer accuracies of the models – i.e., more generalizable readout subspaces also occupy a lower-dimensional representational space (r = -0.69, p<<0.0001) (Fig 4D, S2D, S3F, S6B Figs). When we repeated this analysis for 150 randomly chosen neurons, we did not see a correlation between transfer accuracy and effective rank between curricula (r = 0.25, p = 0.058) (Fig 4E). This confirms that the dimensionality of the readout subspace is what differs here, not the overall dimensionality of the readout.

We also measured the effective rank of models with different imposed easy:hard sample ratios for the first 25 steps of training, finding a strong negative correlation between effective rank and transfer accuracy (r = -0.95, p<<0.0001) (Fig 4F, S2E, S3G, S6C Figs). A higher easy:hard ratio also led to a lower effective rank, corroborating the relationship between curriculum and dimensionality. This reinforces our main theoretical claim: the curriculum that leads to the lowest readout subspace dimensionality (i.e., relies on a smaller number of independent feature dimensions) leads to the highest generalization. In scenarios with limited observations (as in most VPL paradigms), relying on fewer essential visual features is more robust in transfer conditions than relying on many features, some of which may not be useful under unseen conditions. Based on our results, easy-to-hard curricula appear able to discover this low-dimensional, essential feature space (Fig 4D, S2D, S3F, S6B Figs).

Lower-dimensional readout subspaces contain more orientation selective neurons

Motivated by our hypothesis that models trained on easy-to-hard curricula appear to be relying on fewer essential visual features and are therefore more robust to transfer conditions, we further investigated the properties of the readout subspaces formed when training on each curriculum. To determine whether the readout subspaces are composed of similar features, we investigated whether the specific neurons which composed the learned readout subspaces were the same across curricula. We compared the indices of the 150 neurons in the readout subspace across curricula using the Jaccard index, which measures the similarity of two sets. A Jaccard index of 1 means the sets are identical, and a Jaccard index of 0 means the sets have no elements in common. We found that models trained on the shuffled and sequential curricula have the most readout neurons in common, followed by the shuffled and non-sequential curricula, while the sequential and non-sequential curricula share the fewest readout neurons (Fig 5A). To confirm that within-model variability does not underlie this difference across curricula, we also measured the Jaccard index across curricula normalized by the average Jaccard index within a single curriculum and observed the same similarity pattern (Fig 5B). These results support our claim that the curricula read out from different subspaces at the end of training. However, this also indicates some presence of shared features in the learned solutions of the shuffled and sequential models, given their high similarity. We next investigated these shared and unique visual features through orientation tuning analysis.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Lower-dimensional readout subspaces contain highly orientation-selective neurons.

(A) Jaccard index for the set of indices of neurons in the readout subspace compared across curricula. Models trained on shuffled or sequential curricula have the most neurons in common. (B) Jaccard index for the set of indices of neurons in the readout subspace compared across curricula, normalized by the Jaccard index between models trained on the same curriculum. (C) Mean orientation selectivity of the 150 neurons in the readout subspace, measured by the orientation selectivity index, for sequential, shuffled, and non-sequential curricula. Curricula with a lower-dimensional readout subspace (Fig 4D) have a higher selectivity index. (D) Mean activations of each readout subspace neuron across stimulus orientations in the sequential curriculum. The mean across all neurons is shown as a thick line, individual neurons are shown as thin lines. (E) Same as (D) but in the shuffled curriculum. (F) Same as (D) but in the non-sequential curriculum. Both sequential and shuffled curricula have more consistent orientation selectivity across neurons than the non-sequential curriculum.

https://doi.org/10.1371/journal.pcbi.1014553.g005

To solve the orientation discrimination task, during training, the models must identify and rely on neurons that are maximally informative regarding stimulus orientation. These orientation-selective units detect the essential visual features necessary and sufficient for task performance. To investigate whether the various models differ in their reliance on these units, we quantified the orientation selectivity of the most influential neurons within each model. Orientation tuning analysis on the influential neurons in the readout subspace showed that there was a difference between the tuning properties of the neurons depending on the curriculum. We measured the difference in tuning properties using the orientation selectivity index (OSI), which measures the selectivity of each neuron for its preferred orientation compared to the selectivity for the orthogonal orientation [37]. More selective neurons have a higher OSI. The neurons in the readout subspace of the sequential model were more selective for stimulus orientation than the other two models, and the neurons in the readout subspace of the shuffled model were more selective than the non-sequential model (Fig 5C). When we visualized the tuning curve of each neuron, we found that qualitatively, the neurons in the sequential and shuffled readout subspaces have more consistent orientation selectivity than the non-sequential curriculum (Fig 5D-5F). This indicates that the sequential and shuffled curricula lead to low-dimensional readout subspaces which also rely mainly on stimulus orientation (i.e., more orientation selectivity), while the nonsequential curricula leads to a broader, high-dimensional set of readout neurons with less orientation selectivity. This supports our claim that lower-dimensional readout subspaces generalize better because they use fewer essential visual features and do not rely on non-essential visual features.

Early training steps set the dimensionality of the readout subspace

Given our main theoretical claim that generalization depends on the dimensionality of the readout subspace, we examined the formation of readout subspaces across models throughout training. Throughout training, we measured the Jaccard index for the set of the indices of the top 150 influential neurons, in comparison to the set of the indices of the most influential neurons from the previous timestep (Fig 6A). We found that the Jaccard index plateaued at 1 in the first 200 steps of training, meaning the readout subspace was mainly set in the first 200 steps of training. There was a small change in the readout subspace for the sequential curriculum at steps 500–600, which is expected when the task condition changes from easy to hard and the model adapts to the previously unseen samples. This indicates that the initial steps of training mainly set the most influential neurons that form the readout subspace. Because the dimensionality of the readout subspace is determined by the neurons included in the subspace, this means that the initial steps of training mainly set the dimensionality of the readout subspace.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Readout subspace dimensionality is set early in training.

(A) Jaccard index for the set of indices of the top 150 neurons, compared to the set of indices of the top 150 neurons from the previous step of training. The neurons involved in the readout subspace were set in the first 200 steps of training, and did not change much as training progressed. There was a small drop in the Jaccard index for the sequential curriculum at step 500, when the condition switched from easy to hard. (B) PCA of final readout subspace activations for a representative model. The activations in the readout subspaces form a continuum in PCA space from non-sequential → shuffled → sequential, with the shuffled and sequential curricula leading to very similar readout subspaces compared to the non-sequential curriculum. (C) Effective rank vs. task difficulty. Training on easy tasks led to a lower-dimensional readout subspace and greater generalization. (D) Comparison of a non-sequentially trained model constrained to use only the 150-neuron readout subspace transplanted from the forced sequential model. This allowed the model to learn and generalize on a hard task by searching within a pre-established low-dimensional subspace, improving generalization.

https://doi.org/10.1371/journal.pcbi.1014553.g006

We also examined the representational properties of the neurons influencing the readout subspace and compared them across our three base curricula (non-sequential, sequential, and shuffled). To this end, we measured the activations of the 150 influential neurons in each model in response to 100 natural images (see Methods for further details), and after dimensionality reduction using principal component analysis, we compared the response profiles of the three sets of influential neurons (Fig 6B). The representational profiles of the neurons for the three curricula appear to vary on a continuum. Interestingly, the non-sequential subspace had the least overlap with the other two readout subspaces. The shuffled and sequential subspaces, however, were much more similar to each other in terms of their representational profiles, indicating that, as expected, training curricula that involve both easy and hard examples lead to more similar readout subspaces.

What causes the differing dimensionalities of the readout subspaces across curricula? By training solely on easy or hard tasks, we found that the easy task (i.e., larger angle separation) led to a lower effective rank of readout subspace than the hard task (i.e., small angle separation) (Fig 6C). Because the readout subspace is mainly set during the initial steps of training, and the following phases of training appear to stay close to the formed initial readout subspace, it follows that beginning training with an easy task helps to set a lower-dimensional, more generalizable subspace that is then reused for more difficult tasks. This provides a mechanistic explanation for why certain observers in the shuffled condition outperformed others: the observers who performed better focused on easy tasks first (Fig 3A), establishing a low-dimensional subspace that could be reused for the rest of training. Intuitively, the low-dimensional readout subspace discovered during the easy training phase proves sufficiently effective for the subsequent hard task that the training process preserves rather than substantially alters this configuration. This leads us to the final part of our main theoretical claim: less generalizable tasks can “piggyback” [7] on the low-dimensional subspace established through a more generalizable training task, leading to better generalization.

We directly tested this claim by training a model non-sequentially on the hard task but enforcing a learned low-dimensional readout subspace on the model. This mimics the readout subspace set by an easy task at the beginning of training. We enforced the low-dimensional subspace by freezing the readout weights of all neurons except for 150 neurons, the locations of which we adopted directly from the low-dimensional readout subspace of a shuffled model trained with a 6:1 easy:hard sample ratio. We chose this ratio because it led to the highest transfer accuracy (Fig 3E). This allowed the “frozen” model to learn and generalize on a hard task by searching within a pre-established readout subspace. Indeed, we found that limiting the learning subspace of the model to a lower dimension significantly improved generalization on the hard task (Fig 6D; p<<0.0001). We also found that the frozen model performed significantly better than both the sequential (p = 0.025) and shuffled with 6:1 sample difficulty ratio (p = 0.0084) models (Fig 6D). The superior performance of the frozen model compared to the model from which the readout subspace was derived likely reflects an additional regularization effect created by constraining learning to fewer neurons.

Generalization in VPL

Generalization in perceptual learning has been investigated for decades [4]. As shown in previous works, many factors in tasks [38], paradigm design [8,39], and stimuli [6] influence generalization, but we yet lack a comprehensive theory of how all these factors together have an impact on learning generalization. Previous theories attributed learning generalization mainly to the receptive field properties of sensory neurons that were considered to contribute to the learned behaviour [30,40] or were re-tuned during learning [41]. Despite their success in explaining certain generalization variability, their explanatory power is mostly limited to the relationship between learning generalization and stimulus content. Other factors, such as learning sequence or curriculum, cannot be easily and directly explained by the existing theories. The problems of generalization and its underlying factors have also been a central research question in machine learning, and in particular, deep learning and ANNs [42]. Until recently [17], there has not been an explicit attempt to bridge between the two domains and connect the concepts of generalizations in machine learning and VPL, even though the observed phenomena are, at least from a behavioural standpoint, identical. Furthermore, potential empirical and theoretical contributions can easily be imagined in both directions. As a first step in this direction, as shown in this work and previous studies, we can start by investigating whether same patterns of generalization across training conditions and tasks emerge in ANNs and humans. For example, in the case of orientation discrimination and different angle separations, we observed that a similar pattern of generalization emerges in the two systems: an inverse relationship between learning generalization and angle separation is shared in the two systems [17]. We can then leverage the ANN models, and benefit from the accessibility to their whole inner machinery, to formulate a computational and mechanistic understanding of why different training paradigms, task designs, stimuli, or curricula lead to different generalization abilities. Theoretical studies of generalization in ANNs have shown fundamental advances in the last few years [42–45] that could offer new insights into perceptual learning generalization in humans. Equipped with new theoretical ideas, we can then develop new experiments that test neuronal and behavioural predictions following from this computational understanding.

Following this approach, we propose a theory of perceptual learning generalization: different training tasks, stimuli, and curricula lead to readout from different sensory populations with varying dimensionalities. As we demonstrated in our ANN model, this readout dimensionality strongly predicts learning generalization. Importantly, this theory has specific experimental predictions for how sensory representations in the brain are communicated to downstream sensorimotor or association areas. Specifically, the dimensionality of the communication subspace between sensory areas and downstream areas should depend on the training task, and we should be able to predict the learning generalization by measuring this dimensionality of the communication subspace. Although inter-area communication subspaces have been studied recently [46–48], the link between them, prior learning experience, and the characteristics of learned behaviour requires further investigation. The main challenge lies in investigating the communication subspace between sensory and sensorimotor areas that are causally involved in the learned task. Although experimentally challenging, such investigation is feasible [26,49]. This clear experimental prediction provides opportunities for further validation and refinement of our theory, which can bridge behaviour and its underlying neuronal mechanisms.

Learning curriculum

In deep learning literature, there are two views of the utility of learning curriculum. The first claims that easy-to-hard, or simple-to-complex, ordering of training data can help models learn faster and generalize better [33,50–53]. The other, more recent, view claims that in training regimes with large datasets and large models, the explicit ordering of the training samples does not play a significant role in generalization, and it does not benefit models to start with easier examples [54,55]. However, in training regimes with small datasets, sparse [56] or noisy [31,57] data, small models, or short training times [31], curriculum learning may still play a significant role. In particular, a curriculum can increase the efficiency of training, requiring smaller datasets and shorter training times [58–60]. Interestingly, as shown recently, learning curriculum may help models behave more similarly to biological organisms [61].

VPL is generally studied as a training regime with small training datasets and short training times, particularly when compared to the training regimes of large ANNs. If the same principle applies to human learning, perceptual learning should improve with curriculum learning. Our results indeed corroborate this expectation: a sequential learning curriculum improved learning speed and generalization. However, once human observers were offered an unstructured training curriculum with a shuffled order of task difficulty, they proved to be capable of following more complex learning strategies that were broadly as effective as an explicit sequential learning curriculum. The model, on the other hand, was unable to discover these more effective learning strategies from an unstructured training curriculum. As a future direction, it will be important to quantify curriculum efficacy as a function of training stimulus set size in humans. If the similarity with ANNs holds, we expect to see reduced curriculum effects as training stimulus sets increase in size.

In machine learning, previous work focusing on “active learning” has developed both heuristics and principled algorithms [62] for choosing an order for training examples to learn from. We find that our model benefited from an explicit sequential or quasi-sequential ordering, consistent with previous works that showed the benefits of an easy-to-hard curriculum for ANNs in limited training time and samples [31,56,58–60].

Inter-individual variability in learning

Inter-individual variability has not generally been emphasized in VPL, and common practice is to average over a large group of individual learning trajectories to discover shared learning strategies [63]. However, our results suggest that inter-individual differences may play a large role in visual learning. Our analysis of human observers suggests that there is substantial variability in the implicit curriculum learned from tasks with randomly interleaved difficulty levels. In this paper, we found that this variability led to different learning outcomes, and that the strength of the implicit curriculum predicted generalization (Fig 3C-3E). This interpretation could help to reconcile the differing conclusions of previous literature on the benefits of a sequential curriculum [6–9]. Averaging across participants using an interleaved curriculum removes the inter-individual variability, obscuring the conclusion that some participants may benefit from a shuffled curriculum while some benefit from an explicit sequential curriculum. Consistent with our results, recent work has shown that there are substantial inter-individual differences when learning visual tasks [63–65]. As shown in [65], mice displayed notable inter-individual differences in learning strategy and trajectory throughout training, and the strategy used later in training was predictable from the strategy used at the beginning of training.

It is important to note that, while we have studied inter-individual variability in the context of behavioural responses and generalization performance, there are likely multiple underlying factors at play. Previous work by Yang et. al focused on the baseline abilities of individuals to learn a perceptual task, which they hypothesized could underly individual differences in perceptual learning [63]. The authors were able to extract a subject-specific and task-invariant “perceptual learning ability” which measured an individual’s general baseline ability to learn the task, and which comprised a large part of the overall variance in learning [63]. This suggests that individual differences in learning strategy or ability may drive performance on perceptual learning tasks, consistent with our findings that the observers’ implicit curricula were a main driver of differences in generalization performance in the shuffled condition. In addition to varying behavioural responses and perceptual abilities, the computations performed by neurons in response to visual stimuli and during deliberation, which then lead to behavioural responses, also vary across individuals [66]. These varying neural computations could be responsible for the wide variability both in baseline perceptual learning ability [63] and in learning strategy [64,65,67] observed in our study and in previous work, as different neural computations lead to different behavioural responses and therefore may underly differing performance and generalization abilities.

Learning and reusing low-dimensional subspaces

In this paper, we hypothesized that learning from easy tasks early in training forms a low-dimensional subspace in which fewer independent visual features contribute to visual decisions. We showed that reuse of this low-dimensional subspace drives generalization to harder tasks learned later in training. This theory shares commonalities with previous work on visual learning in humans and ANNs [43,68,69], which converges on a similar theoretical framework from different perspectives.

Previous work in deep learning has provided a theoretical explanation for why different training tasks may lead to different generalization [43]. The spectral theory of learning generalization claims that the spectral properties of a representational space – namely, the distribution of eigenvalues and eigenvectors of neural responses to stimuli [68] – and how the training task (i.e., input-output mapping) aligns with the population’s eigenspectrum determines the resulting generalization. More specifically, tasks that align with the network’s top eigenvectors and that can be solved using these low-dimensional eigenvectors lead to better generalization in few-shot learning regimes. In our experiments, easier tasks (e.g., larger angle separation in orientation discrimination) may be better aligned with the top eigenvectors of the underlying population, hence are learned faster and generalize better. This is consistent with our observation showing that a larger angle separation led to a lower-dimensional readout (Fig 6C), representing the complementary perspective of the spectral theory. Prior experimental evidence in animal studies also supports the possibility of low-dimensional readouts under certain training conditions [49,70,71].

Beyond what was proposed by the spectral theory of learning generalization, we also showed that a hard training phase (low alignment with top eigenvectors, using the terms from [43]) that follows an easy training condition can leverage the same recruited low-dimensional subspace as the easy task and achieve better generalization compared to training only on the hard task. This is consistent with previous empirical observations in humans showing that a second training task can “piggyback” on the generalization outcome of a first training task [7].

Extending beyond visual learning, this theory is also consistent with work in dynamical systems which found that neuronal motifs (i.e., low-dimensional neural representations of task structure) are reconfigured and reused across similar tasks [72]. Previous work has shown that forcing a quasi-sequential implicit curriculum through gradual learning from a “teacher” model improves learning speed, generalization, and overall performance [73]. Learning from a teacher model in this way gradually scales up the complexity of the task and input representations, thereby enforcing a low-dimensional representation for the “student” model at the beginning of training that can be reused for more complex tasks [73]. This reinforces the benefits of our “forced” quasi-sequential curriculum in aiding the model to learn and reuse a low-dimensional readout subspace which it would not otherwise discover on its own.

Applications to real-world curricula

While the scope of this study is restricted to simple visual learning tasks, there are potential applications of our work in developing real-world curricula for perceptual generalization. For example, medical fields where diagnoses are made visually require a large amount of perceptual expertise and the ability to generalize to novel abnormal images across individuals [74,75]. For this reason, training the perceptual abilities of radiologists or other specialists who rely on their perceptual insights for diagnosis is extremely important. Although there is no generally accepted perceptual learning curriculum for these fields, it has been shown that initially training students on easier examples before progressing to complex examples can lead to significant improvements in recognition of abnormal images [76, 77].

Our findings can also extend beyond the purely perceptual realm. Studies on classroom curricula have found advantages in both easy-to-hard sequential training [78] as well as interleaving various aspects of learning [79–81]. In our results, we found no significant difference between interleaving, or shuffling, tasks vs. ordering them sequentially in an easy-to-hard manner. However, we also found that some observers can benefit from the shuffled curriculum, as it allows them to develop their own implicit sequential curriculum near the beginning of training that ultimately leads to better generalization than an explicit sequential curriculum. This suggests that in a classroom setting, interleaved curricula may be useful to allow students to develop their own individual implicit curricula. However, as shown in our results, not everyone will benefit as strongly from interleaved learning, and might be better suited to an explicit sequential curriculum. For example, an explicit sequential curriculum has been suggested for students with learning disabilities [78]. Therefore, the optimal curriculum for learning generalization may vary across individuals as well as across tasks.

Additionally, most education involves multiple sensory modalities and task types. Therefore, the optimal curriculum for learning classroom material may be much more complex than the optimal curriculum for visual learning generalization. Supporting this, studies have shown that more complex curricula [79,80,82] that may involve both sequential and interleaved components [83,84] are optimal for different tasks in both humans and ANNs. For example, recent work in ANNs has suggested that in both humans and ANNs, interleaving is advantageous during incremental learning and blocking is advantageous during rapid inference [84]. On the other hand, a recent study comparing transformers and ANNs on a rule-learning query response task found that humans benefit from observing many diverse examples at the beginning of training to promote in-context learning first, while ANNs do not benefit [85]. These studies point towards the need for a more complex curriculum as the complexity of the subject matter increases. Future work could build on our results and theory of generalization to develop and compare these optimal curricula across tasks and perceptual modalities, in addition to across individuals.

Limitations

One limitation of our study is the use of backpropagation as the learning algorithm. Backpropagation, which calculates loss gradients with respect to model parameters in the gradient descent algorithm used in virtually all machine learning models, including for modeling VPL [17], has known issues with biological plausibility [86,87]. Additionally, gradient descent learned through backpropagation violates Dale’s Law and produces biologically implausible weight distributions [88]. Therefore, the fact that our model did not closely follow the same implicit curriculum as human observers in the shuffled condition could be due or partially due to our use of backpropagation. To address this limitation, future work could reproduce our experiments using biologically plausible learning rules [89].

One other limitation of our analysis is our choice of curriculum metric. In this work, we used the difference between the mean accuracy on the first n easy trials and the mean accuracy on the first n hard trials as a proxy for the observers’ implicit curricula. While this metric isn’t a direct measure of curriculum, we chose it as a proxy in order to deal with the inherent noise present in the observer accuracy data. A rate of change metric would better approximate the actual implicit curriculum, as would a trial-by-trial metric instead of averaging over n trials, but due to the noisiness of the data these are technically infeasible. In future work, more advanced techniques for measuring the strength of the implicit curriculum could be effective.

A final limitation of our work is the pre-trained ANN architecture that we used in our modeling experiments. AlexNet [21] has often been used to model the ventral visual pathway in previous work on VPL [17,90,91] and has been shown to perform similarly to newer and more complex architectures in terms of model-brain alignment [24]. Further justification of our choice of model was presented in the results section. However, AlexNet and similar architectures remain rough approximations of visual system architecture and connectivity. For example, feedforward networks like AlexNet lack the lateral or feedback recurrent connections present in biological systems (91). Some have argued that models combining feedforward and recurrent connections are more plausible models of VPL than AlexNet or other strictly feedforward ANNs (91, 92). In an effort to address this limitation, we investigated several other model backbones (see S1 Appendix, S7 Fig). Although the correlation between readout subspace dimensionality and generalization holds for some models, it is not universal across all backbone architectures. This variability indicates that the relationship between readout dimensionality and generalization may interact with specific architectural properties. Further research is required to characterize how different architectural inductive biases influence the emergence of this dimensionality-generalization relationship.

Ethics statement

Observers gave written, informed consent before their participation, and the study was approved by the Ethics Committee of the Université de Montréal (CEREP, Project no. 2024–5806).

ANN model

Experimental design

Curricula

We trained both models and observers on three curricula: a baseline non-sequential curriculum, an easy-to-hard sequential curriculum, and a shuffled curriculum where easy and hard conditions were randomly interleaved. In the non-sequential curriculum, models and observers were trained on only the 1° condition throughout training. In the sequential curriculum, models and observers were trained on the 5° condition for the first half of training (500 trials for the models, 2 days or 640 trials for the observers) and then trained on the 1° condition for the second half of training for an equal number of trials. In the shuffled curriculum, models and observers were trained on randomly interleaved 5° and 1° trials. For the shuffled condition, the order of the 5° and 1° trials was randomized for each participant/model instance.

Analysis

Implicit curricula calculations.

We split the response time and accuracy for each observer by the difficulty of the trial (i.e., an angle separation of 5° or 1°). In Fig 3A, accuracy is averaged across each block of training (80 trials).

Our curriculum metric for each observer (Fig 3C and 3D) was calculated as follows:

(1)

where represents the mean accuracy on the first n easy trials and represents the mean accuracy on the first n hard trials. We chose to use this metric instead of, e.g., a rate of change metric, because calculating the difference in rate of change between consecutive trials considerably increased the amount of high-frequency noise in the data. Due to the high amount of noise between individual trials, we computed the metric over the first n easy/hard trials instead of computing the metric trial-by-trial.

To calculate the learning signal in the model, the mean was taken over all the absolute gradient values of the model’s training loss with respect to the model weights for the full model. The gradients were split by the difficulty of the trial. For Fig 3B, because only one difficulty condition occurs on each step of training, gradient magnitudes of the other condition were set to the previous value of that condition, or to zero if no previous value existed. For example, on a 1° trial, the gradient magnitude for plotting the 5° condition was set to the value calculated in the last 5° trial. For S4A Fig, we plotted both the active and non-active gradients at each timestep.

To approximate the range of implicit curricula found in human observers, we manipulated the ratio of sample difficulties while training the model on a shuffled curriculum (Fig 3E). We repeated the shuffled training as described previously, but instead of randomly selecting the task difficulty at each step with a 1:1 probability ratio of easy:hard samples, we modified this ratio to have a higher or lower probability of choosing each task difficulty. We used the modified ratio to choose samples for the first 25 steps of training and then returned to randomly sampling the task difficulties. We repeated the experiment with ratios ranging from 1:10–10:1 easy:hard samples.

Dimensionality analysis.

For Fig 4A, PCA was calculated on the full readout weight vectors taken at the beginning of training, halfway through training, and at the end of training. For Fig 6B, PCA was calculated on the final activations of the 150-neuron readout subspace for representative models from each curriculum. The weights or activations from each model’s readout subspace were concatenated before PCA and then coloured by the model they originated from after PCA. For both PCA figures, we used the StandardScaler from SciKit-Learn to standardize the data to zero mean and unit variance before computing the top two principal components.

For Fig 4C, neurons were lesioned incrementally 10 at a time by setting their readout weights to zero. This removes the connection from the neuron to the readout. Lesioning was based on the absolute readout weight value, i.e., first the largest 10 readout weights were lesioned, then the largest 20 readout weights, etc. Transfer accuracy was measured as usual after each set of neurons were lesioned. The transfer accuracy was divided by the number of neurons lesioned to calculate the accuracy drop per neuron. To determine the functional contribution of individual neurons, every weight in the readout was lesioned one at a time and the transfer accuracy was measured as usual. After determining the 150 neurons that contributed the most to the readout, S6A Fig was created as described for Fig 4C.

To calculate the dimensionality of the readout subspace, activations were first collected [36] from the readout layers of the models before finetuning while passing 100 natural images from a subset of the ImageNet dataset [22] through the model. We selected 1000 images from the validation set of ILSVRC 2012 [93] as our subset. We selected 150 neurons from the readout of each model instance with the highest absolute weight values (or highest functional contribution as described above) and selected the activations from the models based on the indices of these neurons. This yielded a 100x150 activation matrix. We then performed SVD on each activation matrix and calculated the effective rank according to Roy and Vetterli [35].

The Jaccard index shown in Fig 5A was calculated as follows for each pair of models, for each trial :

(2)

where A, B are models trained with different curricula, and is the set of indices of the 150 neurons in the readout subspace of model A at the end of training for trial .

In Fig 5B, the Jaccard index was normalized using the within-model variance:

(3)(4)

The Jaccard index shown in Fig 6A was calculated as follows for each model:

(5)

where S_t is the indices of the 150 neurons in the readout subspace at training step t.

Orientation tuning analysis.

The orientation selectivity index (OSI) shown in Fig 5C was calculated using the following formula for each neuron in the 150-neuron readout subspace:

(6)

To calculate , which is the firing rate, or activation, at the preferred orientation of each neuron, we first collected the activations from the readout layers of the models before finetuning while passing Gabor stimuli through the model. The Gabor stimuli had orientations of [-180, 180], and each was repeated 20 times. was then the mean maximum activation for that neuron (i.e., the maximum of the mean tuning curve). Then, is the mean activation of the neuron at the orthogonal orientation (preferred orientation + 90°). To plot the tuning curves shown in Fig 5D-5F, we plotted the mean activation from each neuron for all orientations, and the mean across all neurons for all orientations.