Fig 1.
Probing surround effects in CNNs.
Top left: A neuron was taken from either Alexnet or VGG16. Top middle: The optimal spatial frequency and grating orientation were found by grid search. Top right: Then the grating summation field (GSF) was read from the grating diameter tuning curve. Bottom left: We simulated a set of in-silico physiology experiments with the stimuli that were used in neurophysiology studies. Representative stimuli are shown. The responses of CNN neurons are compared with cortical neurons. Bottom right: We visualized surround effects in CNN neurons by a two-step optimization approach. First, the most facilitative center was optimized within the grating summation field. Then, the most suppressive and facilitative surround were optimized with the fixed most facilitative center.
Fig 2.
Grating orientation tuning of the CNN neurons.
A. Stimuli used in the experiments: rotating the center (left, black); rotating the surround (middle, purple); fixing the center at the optimal orientation and rotating the surround (right, cyan). B. Neurophysiology V1 data of the three types of orientation tuning curves (reproduced from [17]). The most suppressive surround orientation matches the optimal center orientation. The surround stimuli alone hardly elicit responses. 0° represents the optimal orientation (same for the following plots). C. Example orientation tuning curves of CNN neurons. D. Averaged orientation tuning curves in CNN layers. Shaded area indicates s.e.m.
Fig 3.
Visualizing the most suppressive and facilitative surround.
A. Center surround similarity can be quantified by the feature correlation, in which we took a feature map in another reference CNN, in this case, layer 1 in ResNet50, and computed the correlation between the center features and the average of the surround features. More specifically, four locations (top, bottom, left, right) in the reference feature map that were closest to the middle between the gsf and the theoretical receptive field were used to get surround feature vectors. Feature correlation was calculated as the correlation coefficient between the mean of the four surround feature vectors and the center feature vector. B. Feature correlations in two CNNs. The shaded area indicates standard deviation. Asterisks indicate p value smaller than 0.05 in paired t-test. Note that the feature correlation depends on the selection of reference CNN. Feature correlations calculated with other reference CNNs can be found in S6 Fig. The most facilitative center (left image with no frame), most suppressive surround (middle image with cyan frames), and most facilitative surround (right image with pink frames) are shown for each selected neuron. C. Example neurons in early layers that have recognizable features: color (left column) and frequency (middle and right column). The most suppressive surrounds appeared similar to the center, whereas the most facilitative surrounds appeared different from the center. D. Example neurons in late layers that have more complex patterns.
Fig 4.
Surround suppression tuning when the center is not at the optimal orientation.
A. Stimuli used in the experiments: the center was either fixed at the optimal orientation or rotated 15°, 30°, 45° off from the optimal orientation. The surround suppression tuning curve was acquired by changing the surround orientation. B. Neurophysiology V1 data of Surround suppression tuning curves, when the center was either optimal or rotated 45° away from the optimal (reproduced from [15]). Arrow indicates the center orientation. The most suppressive surround matched the center orientation. C. First row: example surround suppression tuning curves of CNN neurons. We chose these neurons to show the variety of behavior. Second row: Activation heat map of an extended experiment that used more surround and center orientation combinations. Low activation on the diagonal line indicates that the most suppressive surround orientations can follow the center. D. Averaged surround suppression tuning curves in CNN layers. Shaded area indicates s.e.m.
Fig 5.
The most suppressive surround can follow the center color change.
A. Averaged color correlation of the center the surround in VGG16 (left) and Alexnet (right). Higher values indicate higher color similarity between the center and surround. Four conditions are shown in the plot: the correlation between the optimal center and the most suppressive surround (solid blue); the optimal center and the most facilitative surround (solid red); the altered center and the most suppressive surround (dotted blue); the altered center and the most facilitative surround. The optimal center is defined as the most facilitative center. The altered center is the optimal center with three color channels permuted. The shaded area indicates the standard deviation. B. Two example neurons (VGG_L7_N7 and (VGG_L10_N55)) showing that the most suppressive surround can match the center color. For each neuron, the first row are the center stimuli; the second row are the center stimuli with the most suppressive surround; the third row are the center stimuli with the most facilitative surround. The first column is the optimal center; other columns are the optimal center with the three color channels permuted. The area of the red bars on the right of each image represents the normalized response (relative to the optimal center response).
Fig 6.
The most suppressive surround depends on the center pattern.
A. An example neuron in VGG16 layer 10. The first row is the center stimulus used to optimize the most suppressive surround; the second row is the most suppressive surround using the corresponding center in the first row. The first column is this neuron’s optimal center; the remaining columns are center patterns from the other neurons. B. Visualizations of most suppressive surround of 5 neurons with exchanged centers. The first column shows 5 optimal centers for the 5 selected neurons. Other columns are the most suppressive surround with different centers. The visualizations on the diagonal line used neurons’ own optimal center. The most suppressive surround strongly depends on the center pattern. Some most suppressive surrounds visually matched the center pattern.
Fig 7.
The most suppression by the homogeneous surround can be achieved by two layers with nonlinear activation, as shown in a conceptual model and toy model simulation.
A. A conceptual model. Top: The model has two input neurons with center-surround receptive fields, i.e. the optimal pattern in the center elicits a response of 1; the optimal pattern in the surround elicits a response of -0.8. We further assume that the optimal patterns for two neurons are orthogonal, i.e. the optimal pattern of one neuron does not elicit responses of the other neuron. The response of the output neuron, i.e. neuron 3, is the weighted sum of rectified responses of the two neurons. Neuron 1 has a higher weight than neuron 2. Bottom: input patterns that are created from combining optimal patterns of neuron 1 and 2. Activations of three neurons are computed as defined. The maximum surround suppression happens when the surround pattern matches the center. This is due to the nonlinear ReLU blocking the excessive suppression from the unmatched neuron. B. In a toy model simulation, the visualization results show similar behavior to the conceptual model. The two neurons are constructed by first taking two simple orientated filter maps, then convolving with a kernel with center weight 1 and surround weight -1. Equivalent filters for the two neurons are also shown as “deconvolved” input images. We simulated the visualization experiment in neuron 3. The optimal center for neuron 3 is horizontal and the most suppressive surround matches the optimal center. When the center is replaced by the neuron 2’s optimal pattern, the most suppressive surround looks vertical which is close to the optimal pattern of neuron 2, unlike the most suppressive pattern when the center is fixed to the optimal.
Fig 8.
Surround suppression of naturalistic textures and noise.
A. Naturalistic textures and spectrally matched noise used in the experiments. Naturalistic textures were synthesized using an algorithm described in Methods. B. Texture tuning of an example CNN neuron. The “optimal” textures for each CNN neuron was determined by the textures with the highest modulation index (see details in Methods). The “optimal” textures were then used to study the texture surround effects. C. Texture and noise diameter tuning curves for an example CNN neuron. D. Averaged naturalistic texture and spectrally matched noise diameter tuning curves in V2 neurophysiology data (Reproduced from [31]). Noise induces stronger surround suppression. E. Averaged diameter tuning curves in CNN layers. Noise appears to induce stronger surround suppression in most layers.
Fig 9.
Two mismatches between CNN neurons and cortical neurons.
A, C: Geometry effects of the surround suppression. A. Strength of the surround suppression depends on the location of the surround stimuli. Embed images show the stimuli used in this experiment. The center was fixed at the optimal orientation. The surround had two patches at different locations relative to the center stimuli. The surround was either at optimal orientation or orthogonal orientation. Surround patches that align with the center stimuli induce the strongest suppression when it is at optimal orientation and the strongest facilitation when it is at orthogonal orientation. The polar radius represents the normalized response where the gray circle represents 1 (reproduced from [15]). B. Plots of two example CNN neurons. C. Averaged plots of two CNN layers. P values were calculated from one-way repeated measure ANOVA. Though some neurons and layers showed significant modulation effects of surround location, the effect size and shape of the plots did not match the cortical neurons. D, E, F, G: Peak shift of the low contrast diameter tuning curve. D. We computed diameter tuning curves of each neuron with the normal contrast (high contrast, black line) and 17% of the normal contrast (low contrast, gray line). Dotted vertical lines indicate peaks of the diameter tuning curves. E. Two diameter tuning curves of an example V1 neuron (reproduced from [15]). The low contrast peak is shifted rightward. F. Three example CNN neurons with different directions of peak shift. G. Histogram of peak shift in three CNN layers. Peak shift is defined as low contrast peak diameter subtracting high contrast peak diameter. Positive values are more commonly found in cortical neurons. P values were calculated from paired t-test.
Table 1.
CNN architectures used in this study.
The input size of both networks is 224x224x3. Conv2D represents 2D convolutioanl layer. Three following numbers denotes the kernel size, stride size and channel numbers. BN represents batch normalization. MaxPooling represents 2D max pooling layer. The following numbers denotes the pool size and stride size. Dropout represent dropout layer. The following number denotes dropout rate.