General discussion

Our study evaluated whether perceptual learning can lead to the acquisition of robustness to visual noise, both in human observers and DNNs. We obtained novel evidence that human object recognition can indeed become more robust to severe levels of noise, such that the learned representations are tuned to the trained category but also sufficiently flexible to generalize to entirely novel test images. These findings provide support for our hypothesis that the robustness of the human visual system is likely acquired via mechanisms of perceptual learning.

Unlike human observers, standard DNNs lacked robustness to these image corruptions, consistent with prior work from our lab [4] and others [2,26]. However, after training with noisy animate or inanimate objects, the DNN models showed moderate gains in robustness for the untrained category; moreover, they were able to match human-level robustness for noisy test objects from the trained category. Thus, standard DNNs showed both category-general improvement and an additional category-specific benefit of training with noisy object images.

Given that human observers must deal with many forms of real-world noise, as can arise from encounters with rain, snow, fog, or other noisy elements, it is perhaps not surprising that recognition performance was already quite robust before our participants underwent training in the lab. When the DNN models were provided with additional pre-training at moderate levels of noise, we found that subsequent training with noisy animate or inanimate objects resulted in category-specific improvements that closely matched the patterns of learning exhibited by human observers. These findings demonstrate that DNNs can be effectively used to model and to predict how humans will improve at a complex visual task when given opportunities for learning.

A layer-wise analysis of DNN responses provided further insights into the effects of training on information processing across the successive stages of these models. Whereas training with noisy animate or inanimate objects led to category-general improvements in the lower layers of VGG-19, category-specific benefits became more pronounced in the higher layers. Similar patterns of results were observed in networks pre-trained with moderate levels of noise, where category-specific training benefits were concentrated in the higher layers of the model. These findings were further supported by a series of layer-freezing experiments. When learning was restricted to the lowest layers of the DNN (i.e., layers 1–4), we mostly observed category-general improvements in noise robustness. By contrast, when learning was restricted to the higher layers (e.g., layers 13–19), category-general benefits disappeared and the DNN mostly showed category-specific improvements in SSNR threshold. These computational results strongly suggest that category-specific acquisition of noise robustness depends on learning that occurs at higher levels of visual representation.

Extrapolating from these modeling results to human observers, it can be inferred that our training protocol led to the refinement of higher-level category-specific representations in the human visual system. In comparison, low-level feature representations did not seem to improve from this training regime, otherwise human performance for the untrained category should also have improved to a considerable extent. Presumably, for our observers, these low-level representations were already well-tuned and quite robust to noise before training.

In this study, we compared perceptual training with animate versus inanimate categories of objects, motivated by the fact that these distinct superordinate categories [44] are known to engage well-differentiated representations in the human ventral visual system (e.g., [22,36–41]). While our results provide compelling positive evidence that training with noisy objects does indeed lead to category-specific learning in humans, it remains an open question as to whether such specificity would occur at finer scales among basic-level categories with greater shared similarity. For example, would training with one set of animal categories lead to successful generalization to other untrained animal categories, given that animals tend to share some overlap in their shape and textural properties? While the possibility of addressing this question in human participants extended well beyond the scope of our planned study, it was possible for us to explore this issue by training a new set of DNNs on a subset of animate or inanimate categories. Since we were interested in whether category specificity might be maintained at this finer level of granularity, we evaluated the performance of DNNs that initially received weak noise training with all 16 object categories, followed by strong noise training with only 4 of the basic-level categories drawn from either the animate or inanimate object sets. The results of this exploratory analysis revealed that training led to category-specific improvements at the basic level in 3 out of 4 cases, while we did also observe that DNNs trained with animate set 1 exhibited better generalization performance for animate set 2 (see S2 Fig). These findings suggest that perceptual training with noisy objects can lead to category-specific improvements at the basic level, although a certain degree of generalization can also occur at the broader superordinate level. Future studies with human observers could explore whether similar patterns of perceptual training emerge, although such investigations would likely require substantially larger sample sizes to detect potentially subtle generalization effects with adequate statistical power.

The learning benefits found in our study further demonstrate how human robustness to visual noise can be further improved by only a few hours of training in a laboratory setting, with 800 training trials in Experiment 1 or 400 trials in Experiment 2. Although Experiment 2 revealed these improvements to be category-specific, the acquired information was sufficiently flexible and robust to support successful generalization to novel real-world object images, thereby indicating a form of high-level visual learning. Likewise, it is notable that our DNN models showed improved robustness to noise after training with the same set of object images used for training in the human studies. Although it is commonly believed that DNN models require extremely large datasets to learn appropriate visual representations, we found that additional fine-tuning with only a few hundred noisy object images could lead to major improvements in DNN model robustness. If one assumes that DNNs can provide an informative measure of how much people might possibly learn in a given situation or from a limited set of training examples, then DNN models could prove useful for providing new insights into human perceptual learning.

Taken together, our human behavioral results, in conjunction with the DNN modeling results, provide strong support for our hypothesis that a limited number of encounters with highly challenging viewing conditions can provide the basis for the acquisition of visual robustness. Such conclusions also receive support from anecdotal reports by people who move to a much colder climate with snowy winters. While the first few occasions of driving through a snowstorm can be a terrifying white-knuckled experience, most individuals improve and become seasoned drivers after just one or two winters. Thus, even in adulthood, the human visual system appears capable of considerable plasticity, allowing for rapid and flexible learning, as was demonstrated by the participants in our study.

Our training paradigm, which involved presenting a novel object image on every trial, can be contrasted with traditional approaches to perceptual learning. These studies have typically focused on perceptual training with simple low-level visual features [34,45,46] or with a limited number of complex stimuli that are used for both training and test (e.g., [3]). In the absence of testing for generalization to novel stimulus conditions, it can be difficult to discern whether improvements in performance are due to the modification of low-level feature representations or higher-level object representations. Here, by evaluating human performance using heterogeneous and unpredictable object images, we could ensure that recognition performance had to rely on flexible, higher-level visual representations.

So far, only a few studies have used DNN models to characterize human perceptual learning, primarily focusing on tasks of orientation discrimination. These studies have found that DNNs are sensitive to the effects of training on high versus lower precision tasks, training with single or multiple spatial frequencies, and can help account for effects of generalization to new orientations, new spatial frequencies or other stimulus conditions [47–49]. Our study expands on this approach to characterize how humans and DNNs improve at complex tasks of object recognition that require dealing with ambiguity, noise, and stimulus uncertainty. While DNN models are only beginning to be introduced into studies of perceptual learning, we believe that this computational approach is highly promising. Indeed, convolutional neural network models are inspired by the feedforward architecture of the human visual system, and could help provide a computational framework for understanding the bases of many forms of visual learning.

While this study primarily focused on robustness to Gaussian noise, another important question concerns how robustness is acquired for multiple forms of image degradation. Some research has suggested that DNNs trained with one type of visual noise are unable to generalize to other types of noise (e.g., uniform vs. salt-and-pepper noise) [2], while other studies have reported partial generalization success [4]. Here, we found that DNNs trained with objects in Gaussian noise could generalize well to the impulse, shot and speckle noise conditions from ImageNet-C; however, such training only led to slight improvements in the ability to recognize blurred objects. These findings are consistent with the proposal that the generalization performance of DNN models may be linked to the spatial frequency content of the noise components used for training and test; if both have relatively greater power at high spatial frequencies, that could allow for a certain degree of generalization as those frequency components are down-weighted [50]. In other recent work, we have shown that DNNs trained with a mixture of clear and blurry images become more robust to both blur and noise, with significant improvements for 14 out of 19 ImageNet-C conditions [29]. Thus, when compared with the current findings we see evidence of an asymmetry: while blur training enhances robustness to Gaussian noise [29], Gaussian noise training does not confer the same level of robustness to blur (Fig 6). This discrepancy may highlight differences in the types of visual information that are learned from distinct image distortions. Further studies could reveal whether strategic exposure to a variety of visual distortions might enhance DNN robustness to a broader array of image challenges.

In the present study, we found that feedforward CNN models can capture important aspects of human perceptual learning when trained on a highly challenging task of recognizing objects in severe levels of visual noise. However, it is worth noting that standard deep neural networks typically lack top-down feedback and recurrent mechanisms—network properties that have been shown to markedly enhance visual recognition under complex, degraded visual conditions [51–54]. Moreover, perceptual learning has been found to modulate how top-down processing occurs in the visual system, thereby influencing how the brain integrates sensory input with prior knowledge [55–57]. Of considerable relevance, a recent study found that adding a top-down reconstructive attentional mechanism to DNN models led to improved recognition of MNIST digits in noise; moreover, this approach was successfully extended to classify objects in the presence of various forms of image degradation [58]. Incorporating such recurrent or top-down mechanisms into DNN architectures might not only improve their robustness, but further has the potential to provide insights into the mechanisms that underlie the robust nature of human vision.

In conclusion, the present study underscores the complex interplay between category-specific and category-general mechanisms in the perceptual learning of noise-robust visual representations. Through a systematic comparison of humans and DNN models, we provide key insights into the adaptive nature of visual learning in both systems. Our findings not only enhance current understanding of how the human visual system processes noisy or ambiguous visual inputs, they also provide guidance for how artificial vision systems can be made more robust and better aligned with human perceptual capabilities.

Ethics statement

This study was approved by the Institutional Review Board of Vanderbilt University (IRB #040945 “Studies of Human Visual Processing”). All participants provided written informed consent prior to participation and reported having normal or corrected-to-normal visual acuity. In Experiment 1, 20 participants were recruited, while Experiment 2 involved a separate group of 32 participants. Participants received either monetary compensation or course credit with supplementary monetary compensation for taking part.

Visual stimuli

Our training object images were obtained from the ImageNet validation set [35], and consisted of 50 images drawn from each of 16 categories. The stimulus categories included both animate objects (bear, bison, elephant, hamster, hare, lion, owl, tabby cat) and inanimate objects (airliner, couch, jeep, schooner, speedboat, sports car, table lamp, teapot). All images were converted to grayscale to eliminate color cues for recognition.

The level of noise in the images was manipulated by varying the signal-to-signal-plus-noise ratio (SSNR), defined as w in the equation , where S represents the source image, N denotes the noise image, and T is the target image with added noise  [59]. The SSNR ranges from 0 (noise only) to 1 (signal only). The pixel intensities for S and N were scaled between 0 and 1. For the noise N, random values were sampled from a Gaussian distribution centered at 0.5 with a standard deviation of 1/6, ensuring 99.73% of pixel intensities remained within the 0–1 range. Any pixels outside this range were clipped accordingly.

Experiment 1

In the first experiment, participants had to classify a novel training image embedded in noise on every trial, performing a total of 800 trials across 4 sessions, with each session lasting approximately one hour. On each trial, participants viewed an image where a target object gradually emerged from noise as the SSNR level increased from 0 to 1 in 0.025 increments every 400ms. Participants pressed a key to halt this animation sequence as soon as they were ready to make a sixteen-alternative forced-choice classification response. Additionally, participants used a virtual paintbrush controlled by a mouse pointer to annotate the portions of the noisy image that were most informative for their decision. To incentivize both fast and accurate responses, participants received a small monetary bonus for correct responses based on the speed of their response, and received feedback at the end of each trial. Prior to the first training session, participants completed 16 practice trials with a different set of novel images to familiarize themselves with the procedure.

Both before and after the training trials, we measured the threshold SSNR level at which participants could successfully classify noisy test objects that were briefly presented for 200ms. Specifically, we used an adaptive staircase procedure called QUEST [42] to estimate each observer’s recognition accuracy by SSNR curve, which was updated on a trial-by-trial basis to determine the SSNR level of the next test stimulus to present. Noise robustness was measured by the SSNR threshold level corresponding to a 57% performance criterion (inflection point) in a 48-trial run, with initial threshold and standard deviation estimates set at 0.5 and 0.4, respectively. For these pre-training and post-training test measures, we used 96 object images (6 per category) not included in the training sessions. Four independent raters, including one of the authors, determined the noise thresholds for all 96 images using our training protocol but without the annotation task. Based on these subjectively measured thresholds, two image sets (48 each) with closely matched difficulty levels were created, and these were assigned to serve as pre-training or post-training test stimuli, counterbalanced across participants. To minimize potential demand characteristics, participants were not informed that the entire study focused on testing for effects of perceptual learning.

Experiment 2

In Experiment 2, participants were randomly assigned to undergo training with noisy object images drawn from either the 8 animate categories or the 8 inanimate categories. Each of the two groups performed 400 training trials across 3 sessions (100 trials in the first session, 175 trials in the second session, and 125 trials in the final session). The noise sensitivity (or noise robustness) of both groups was assessed both before and after training using the adaptive staircase procedure, with separate SSNR threshold estimates obtained for animate objects and inanimate objects. Our experimental design sought to compare the effects of training with animate versus inanimate object images, which are highly visually distinct, to enhance the likelihood of detecting category-specific effects of training should they exist. That said, our study was not designed to test for possible effects of generalization within a trained superordinate category, as one could potentially test for generalization using visually similar categories (e.g., lion vs. puma) or visually dissimilar categories (e.g., lion vs. aardvark) and the potential problem space is vast.

To enhance the reliability of our adaptive staircase test, three raters evaluated the difficulty level of 768 individual images. These ratings were used to create 2 sets of animate object images of comparable difficulty, and 2 sets of inanimate objects with balanced difficulty (48 images per test set). The assignment of the image sets for pre- or post-training test was counterbalanced across participants. To ensure the consistency of training effects, most participants completed all three sessions within one week. Additionally, a criterion for inclusion was established: participants who did not achieve at least 90% accuracy in the first session were excluded from the remaining sessions.

Convolutional neural networks

We evaluated multiple trained versions of six CNN models: AlexNet [14], VGG16, VGG19 [15], GoogLeNet [16], ResNet50 [13], and Inception-v3 [17]. All modeling experiments were implemented using PyTorch. Each network was initialized with pre-trained weights from ImageNet, and noised-trained models were further trained on objects in visual noise using the same set of 800 training images that the human observers viewed in Experiment 1 or only 400 training images (i.e., animate or inanimate) for Experiment 2. During training, the input images were degraded with varying levels of Gaussian noise [4,59], with the specified signal-to-signal-plus-noise ratio randomly sampled from a uniform distribution from 0.2 to 1.0. In Experiment 2, a matching number of clear images from the untrained categories were concurrently used for training. To ensure that data augmentation did not introduce unintended interference with noise robustness, we applied only minimal augmentation techniques, specifically RandomResizedCrop and RandomHorizontalFlip, using their default parameters in PyTorch. (Methods such as color jitter were not used.) Images were resized to 224 × 224 pixels, converted to grayscale, and transformed to RGB by replicating the values across all 3 color channels to ensure compatibility with the pre-trained models. Each image was normalized using the mean (0.449) and standard deviation (0.226) of the ImageNet training set. The networks were optimized using a stochastic gradient descent algorithm with a fixed learning rate of 0.01 and a weight decay of 0.0001 for 100 epochs. Control CNNs were trained on noise-free training images using identical procedures.

Unlike human participants, CNNs could be repeatedly tested with the same object image across a full range of SSNR levels to assess threshold performance. Thus, noise thresholds before and after training were estimated from the complete SSNR-accuracy curve using the same test images as were used in the human experiments. Since category-specific noise training in Experiment 2 might lead to biased classification decisions in favor of the trained category for noisy stimuli, we restricted CNN responses by using the highest confidence response from the relevant trained or untrained categories to estimate SSNR thresholds.

Additionally, to investigate whether initial weak noise training on both animate and inanimate categories would elicit similar effects to those observed in humans (specifically, the absence of category-general effects), we provided modest noise pre-training to a subset of models using the 800 training images with SSNRs ranging from 0.6 to 1.0, which included both animate and inanimate categories. This was followed by training with SSNRs ranging from 0.2 to 1.0 for either animate or inanimate categories.

Layer-wise noise susceptibility analysis

For the layer-wise noise susceptibility analysis, we followed the methodology outlined by Jang et al. (2021) [4]. Specifically, we evaluated the consistency of activity patterns for objects with increasing noise levels by calculating the Pearson correlation coefficient between responses to noise-free images and the same images across varying SSNR levels. High robustness to noise was indicated by similar responses to both noisy and noise-free images. This analysis was conducted on all layers of VGG19, including 16 convolutional layers and 3 fully connected layers. The correlation-by-SSNR curve was rescaled to a range 0–1 to eliminate offsets at 0 SSNR. A logistic function was then fitted to the correlation-by-SSNR curve for each layer, and the SSNR level at which correlation strength reached 0.5 was identified as the SSNR threshold.

Diagnostic feature analysis in humans and neural networks

To compare diagnostic image regions utilized by CNNs with those obtained from human participants, we applied layer-wise relevance propagation [60] on VGG19 model responses to noisy object images. This technique identifies diagnostic features that contribute to network predictions by decomposing the network’s output into relevance scores, which are subsequently backpropagated to the input layer. To align with the resolution of human-drawn diagnostic regions, the relevance maps were smoothed using a Gaussian kernel with a standard deviation of 3 pixels. We assessed the similarity between the diagnostic regions of humans and CNNs by calculating Pearson correlation coefficients and overlap ratios. The specifics of parameter choices were based on Jang et al. (2021) [4]. Additionally, to evaluate human-to-human similarity in saliency maps, we randomly divided the data into two groups and computed the correlation and overlap ratios between the average maps from these groups. This sampling procedure was repeated 10 times, with the average values reported. The results of this additional analysis revealed that noise-trained VGG-19 performed better at predicting the diagnostic spatial regions reported by human observers, consistent with our prior work [4], and that spatial predictions were better for noisy objects from the trained category than the untrained category (see S3 Fig).