Ethics statement

All animal procedures were approved by the Institutional Animal Care and Use Committee of the University of California, San Francisco (protocol numbers AN109411-03 and AN173817-01), and followed the guidelines of the National Institutes of Health.

Data collection

Electrophysiological signals were recorded from the auditory cortex of squirrel monkeys using a preparation described in detail elsewhere [13], which we summarize briefly here. Over the course of several months, recordings were made from penetrations into the core, belt, and parabelt areas of right auditory cortex of three animals (B, C, F). For monkey C, recordings were also made in left auditory cortex. We used a variety of probes with 1, 16, 32, 48, or 64 channels, selected for each recording session based on which probe was most suitable for accessing the target area on that day. As a result, probe layouts varied across sessions (see S2 Table for details). Our ideal approach was to advance the probe until the deepest channel no longer registered neural activity. While this generally allowed us to sample across all cortical layers at most recording sites, accessing cortical fields near the edges of the craniotomy often limited our ability to record from all channels on a given probe. Additionally, the electrode insertion angle was not always orthogonal to the auditory cortex, making it difficult to confidently assign specific channels to specific layers. Based on these constraints, we infer that the majority of the recorded neurons presented here are located within the supragranular and granular layers of the auditory cortex.

Auditory stimuli were played from a free-field speaker inside the soundproof chamber where recordings took place. During each recording session, two sets of stimuli were presented. The first set consisted of sentences from the TIMIT corpus, each approximately 1–3 seconds in length, which we have found to elicit strong responses from the auditory cortex of squirrel monkeys. A total of 499 unique sentences were presented: 489 were presented exactly once (in random order), while the remaining 10 were repeated 11 times each, for a total of 110 presentations (also in random order). The second set consisted of monkey vocalizations, each approximately 1 second in length, including grunts, screams, and coos. In this set, a total of 303 unique vocalizations were presented: 292 were presented once, and the remaining 11 were presented 15 times each, for a total of 165 presentations, also in random order.

Spikes were identified based on threshold crossings, defined as voltage deflections exceeding 3.5 standard deviations from the mean extracellular voltage on each probe. Unthresholded data were collected using Intan software and subsequently thresholded using custom MATLAB code. For the main analyses, spikes were counted in 50 ms bins; for Fig 4, spikes were binned at 20 ms. Because spike-sorting was not feasible, we work with multi-unit activity (MUA) throughout this study, and we refer to the source of spikes on a single channel as a “multi-unit.”

Trial-to-trial neural variability

Neurons do not respond identically to identical stimuli, whether because they are tuned instead or additionally to other, uncontrolled stimuli; as a consequence of feedback or attention; due to false alarms or misses in spike detection; or simply because of intrinsic noise in the spiking process.

To characterize this variability, we analyzed the response of each multi-unit to the repeated stimuli (see previous). In particular, since sample correlation coefficients converge on the true correlation as the number of samples (in our case, bins) increases, we first concatenated together responses from multiple trials. More precisely (see S1 Fig), for each of the M unique test stimuli, we randomly selected a single pair of responses, , with , for . We then concatenated together all M of the U_m into a single “long” sequence, U, and likewise for all M of the into a sequence V, in the same order. Thus for TIMIT, for which the repeat set was 10 unique sentences, each on average 1.6 seconds long, the two concatenated sequences U and V were each about 320 samples long (20 samples/second × 1.6 seconds/sentence × 10 sentences). For the monkey vocalizations, U and V had about 220 samples each (20 samples/second × 1.0 second/vocalization × 11 vocalizations). The sample correlation between U and V provides an estimate of this multi-unit’s reliability. To generate more such estimates, we repeated this entire procedure 100,000 times, yielding 100,000 correlation coefficients for each of the ∼1700 multi-units (recording channels).

To quantify this variability, we also constructed a null distribution of trial-to-trial correlations for each recording channel, analogous to the true distribution described above. The key difference is that, for the null distribution, we circularly shift the V responses before computing the correlations. This procedure preserves the marginal response statistics for each channel while disrupting any temporal alignment or correlation with U. We constructed a separate true and null distribution for each recording channel for each of the stimulus sets (speech and monkey vocalizations). These distributions were computed for 50 ms and 20 ms for, respectively, the main results and the analysis in Fig 4. We then put these distributions to three related but distinct uses.

Encoding models

Predicting spiking activity with ANNs.

The effective sampling rate in the ANN (imposed by the “strides” of the convolutions) declines across layers. Therefore, before fitting linear readouts maps, all layer activities were resampled to 20 Hz to match the 50-ms bins for the main results of this study (and for the analyses reported in Fig 4, to 50 Hz). We then fit a temporal receptive field (TRF) from each ANN layer to each well-tuned multi-unit (see Fig 1). In particular, the predicted neural response at one bin was allowed to depend linearly on a 250-ms window of ANN activity. TRFs were fit on the non-repeated stimuli (see Data collection above) by minimizing L₂-regularized squared error (∼20,000 samples and ∼6000 samples at 50-ms bins for speech and monkey vocalizations, respectively). The magnitude of the L₂ penalty was determined with three-fold cross validation over a range of regularization parameters (10⁻⁵ to 10¹⁵). All fits were made with the naplib python package [29], partly customized to run on GPUs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic of ANN-based encoding models.

Spiking activity (upper right) in response to stimuli (e.g., speech; waveform at left) is recorded from squirrel monkey auditory cortex (black dots on brain in upper left). The exact same stimuli are presented to the ANN (bottom). The spike sequences are then regressed separately (schematic linear plots, middle) onto each hidden layer’s responses to the same stimuli, yielding a temporal receptive field (TRF) for each layer of the ANN. Performance is evaluated on held-out stimulus-response pairs by computing the correlation between the TRF-based predictions and the spike-count sequences.

https://doi.org/10.1371/journal.pcbi.1013334.g001

We emphasize the rationale for using a (merely) linear readout: We want to be able to identify the layer of the network that best explains cortical neural activity. Allowing (arbitrary) nonlinear maps would break this correspondence, since the layers themselves are related by nonlinear maps. (We can say, colorfully, that if layer n makes the best predictions through a linear map, then layer 1 also makes the best predictions through the nonlinear map consisting of the neural network itself up to layer n, and then out through the linear map.) At the other extreme, we are uninterested in mapping single ANN units to single electrode channels because a layer is arbitrary up to a (full-rank) matrix multiplication (the subsequent weight matrix could absorb the inverse of this matrix).

Evaluating model predictions

Our main metric of model performance is the noise-corrected correlation coefficient (see above), which we computed for every combination of well-tuned units and encoding model. Note that an “encoding model” corresponds to a single layer from a single ANN, either trained or untrained, or to a STRF. Correlation coefficients were computed between the sequences of predicted and actual (biological) responses to the 110 and 165 held-out stimuli for speech and monkey vocalizations, respectively. Recall that these correspond, respectively, to 11 repeats of 10 unique sentences and 15 repeats of 11 unique vocalizations.

Since longer sequences yield better estimates of the correlation coefficient, we first assembled responses into “long sequences” (similar to the construction of the trial-to-trial correlations; see above). That is, for each multi-unit, we concatenated together the responses to the first instances of each of the 10 sentences; and likewise for the second instances; and so on up to the eleventh instance, yielding 11 long sequences of responses. For each model (ANN layer or STRF), we assembled a single long sequence of responses to these 10 unique sentences. We then computed the correlation between that model’s long sequence and each of a particular multi-unit’s 11 long sequences. For each multi-unit, we report the mean of these 11 correlation coefficients. (We used the same procedure for the responses to monkey vocalizations, mutatis mutandis.)

To compare models with each other, we compare the distributions of correlation coefficients across all well-tuned units. Since the distributions to be compared consisted of paired samples (each corresponding to the same multi-unit), but are not Gaussian, we tested for significant differences with the Wilcoxon signed-rank test. For each layer of an ANN, we make two comparisons: one with its untrained counterpart and another with the STRF, resulting in a total of comparisons for an L-layered model. To control the false discovery rate, we apply the Benjamini–Hochberg correction separately for each model.

Evaluating performance on the ecologically relevant pretext task

To relate the ANNs’ performances at predicting neural activity to their performances on the pretext task, we evaluated the word-error rates (WER) of all the six ANNs on the following datasets (see S6 Table):

• TED-LIUM 3 [30]: Test split of release 3 from the TorchAudio repository [31].
• Common Voice 5.1 [32]: Test split of Common Voice 5.1 for English language [33].
• VoxPopuli [34]: Test split of transcribed speech for English language from the Meta Research repository [35].

Performance of ANNs as models of auditory cortex

We expand our view to all six ANN architectures, all stimuli, and the entire set of tuned multi-units. We identified a multi-unit as tuned if its responses to multiple tokens of the same stimulus correlate with each other above chance (see Materials and methods). Note that this determination is sensitive to the width of the bin in which spikes are counted; unless stated otherwise, we used 50-ms bins. About 70% (1195/1718) of multi-units were tuned to speech under this criterion, and 72% tuned to monkey vocalizations (see S3 Table); this validates the use of speech audio as a stimulus. Nevertheless, many of these nominally tuned multi-units have trial-to-trial correlations quite close to chance (see S1 Fig), so we restrict our analyses to a subset of well-tuned multi-units (see Materials and methods), yielding 404 for speech and 489 for monkey vocalizations.

To make predictions, we fit a separate linear temporal receptive field (TRF) from each layer of each ANN to each multi-unit activity (across all electrode channels, all recording sites, and all three monkeys), i.e., to each sequence of spike counts in 50-ms bins. As a baseline, we also fit a spectrotemporal receptive-field (STRF) to each multi-unit activity. To evaluate these encoding models, we compute the correlation between sequences of spike counts and corresponding TRF predictions on a held-out set of stimuli. When reporting these correlations, we follow the standard practice of correcting for unexplainable trial-to-trial variability [36]. (An alternative [14] is to report the ratio of each ANN-neuron correlation to the corresponding STRF-neuron correlation. This has the advantage of canceling out the noisy normalizer in the correction, although it obscures the absolute correlations achieved. We re-present Fig 3 this way in S3 Fig.)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Model-neuron correlations.

All subpanels show correlations between model predictions and the multi-unit activity they are supposed to predict. A: Model-neuron correlations for speech (TIMIT) stimuli. Bar plot: median (across multi-units) correlation of the STRF (gray) and of the best layers of each of the neural networks, both trained (dark colors) and untrained (light colors). Line plots: the distributions of correlations as a function of ANN layer for each of six trained networks (colored) and their untrained counterparts (gray). The median (solid line) and interquartile range (shaded region) are shown. At starred layers, the trained ANN is significantly superior to its untrained counterpart (top row of stars) or a STRF (bottom row of stars; Wilcoxon signed-rank test with p < 0.01). To control the false discovery rate, we apply the Benjamini–Hochberg correction separately for each model. The layer type is indicated along the top of each plot: convolutional (brown), self-attention (blue), and recurrent (light blue). B: The same as A but using monkey vocalizations for stimuli.

https://doi.org/10.1371/journal.pcbi.1013334.g003

Fig 3A (colored shading and colored lines) shows the distributions of correlations across all multi-units and at each layer of each ANN, in response to English sentences. First, we note that, at their intermediate layers, all ANNs make superior predictions to the STRF (median correlation shown as gray bar in leftmost plot and as a dashed line in all other plots); i.e., the distributions of model-neuron correlations are significantly higher (gray stars; Wilcoxon signed-rank test, p < 0.01). Indeed, the most predictive ANN layers correlate with the median multi-unit at about 0.6 (noise-corrected). This is comparable to the best models of auditory neurons to date [21], which are complex neural networks fit directly to the neural data, whereas our models fit only the linear readout. Nevertheless, we caution that direct comparisons are difficult, since that study and ours differ in the animal model (ferret in theirs), number of data collected, neuron-inclusion criteria, and other details of the experiments.

Second, the distribution of correlations typically rises over the first few layers, peaks before the middle of each network, and then falls off. The rise is expected: For networks that take the raw waveform as input (wav2letter [15] and wav2vec2 [19]), it is not surprising that multiple layers of nonlinearities provide for better predictions, since the relationship between auditory-cortical responses and sound is known to be highly complex and nonlinear. But the same holds even for networks that take the spectrogram as input (speech2text [16], the Whisper models [18], and DeepSpeech2 [17]), which implies that even the STRF (which is linear in the spectrogram) is several nonlinearities away from the optimal response model. The post-peak fall off in correlations could be a result of ANN specialization for the speech-recognition task: the deepest layers predict phonemes or characters, a task that is arguably foreign to the squirrel monkey. Indeed, if we (pre-)train wav2vec2 on AudioSet [37], a large collection of sounds from internet videos, and then skip supervised fine-tuning altogether, the post peak fall-off mostly disappears (S4 Fig). We return to this theme in the Discussion.

The importance of task optimization

To verify that task optimization improves model predictions, we investigate the predictive performance of untrained networks. Because each layer of an artificial neural network contains nonlinearities, a linear readout from deeper layers can be more expressive, even when the network is untrained. To determine how much of the ANN’s predictive power is due to training on the task of automatic speech recognition (ASR), and how much merely to these stacked nonlinearities, we also attempt to predict cortical activity with untrained networks. In particular, we “reset” the weights to their initial values, i.e. before training began, and then fit a new set of linear maps (TRFs) from each layer to each cortical unit (see Materials and methods). The distributions of the resulting correlations are also shown in Fig 3A, in grayscale. Trained networks are superior to untrained networks almost everywhere (Wilcoxon signed-rank test; p < 0.01 indicated with top row of black stars), and the most predictive layers are all in trained networks.

In networks that take the spectrogram as input, the difference between trained and untrained networks is much smaller than in networks that take the waveform as input—presumably because in the latter, something like the spectrogram computation is learned by the trained network, but cannot be randomly assembled by the untrained network. This accords with the classical emphasis on STRFs and with the well understood transformations of the audio signal by the cochlea.

The ecological relevance of speech recognition

In order to avail ourselves of ANNs trained with supervision on large datasets, we have let the pretext task for our networks be speech recognition. This in turn has narrowed our focus (up to this point) to English-sentences stimuli, since it is unclear how such networks will respond to sounds not occurring in their training sets. Three findings support this choice: (1) Roughly as many multi-units—indeed, mostly the same multi-units—are well tuned to English and to monkey vocalizations; (2) our ANNs linearly predict the spiking responses to English sentences about as well as the best nonlinear models of auditory cortex [21]; and (3) the pretext training (on speech) improves model-neuron correlations beyond what can be achieved with random nonlinearities. Although English speech is not entirely irrelevant to these animals, these results together suggest that neurons in squirrel-monkey auditory cortex are tuned to audio features that are higher-order than spectrograms but generic enough to be useful (if not used) for speech recognition, as well as processing other sounds.

If this is the case, we also expect our ANNs to explain the responses of auditory neurons to monkey vocalizations. Fig 3B shows the results (in the same format as Fig 3A). Note that the linear readout was re-fit for these stimuli, but the ANNs are identical to the ones analyzed in Fig 3A: they have been pretrained only on speech tasks. We observe three important points:

• Like the responses to speech, responses to monkey vocalizations are best explained by the intermediate layers of the trained ANNs, which are likewise always better than their untrained counterparts. The best overall predictions are made by layer 2 of DeepSpeech2, just as for speech stimuli.
• The responses to monkey vocalizations are more easily predicted than the responses to speech, despite the ANNs being trained on a speech task. However, this also holds for STRFs, suggesting that monkey vocalizations are in some sense simpler stimuli than speech.
• Consequently, the weakest networks, wav2vec2 and speech2text, provide little to no improvement over STRFs in predicting responses to monkey vocalizations. But the stronger networks still do.

Still, there is a limit to the ecological relevance of speech to the squirrel monkey. The ANNs’ performance in speech recognition does not, for example, correlate with model-neuron correlations (S5 Fig), as has been observed in studies in humans [2,12,14]. And although in core we found about as many multi-units well tuned to speech as to monkey vocalizations, we found only about 60% as many in non-primary areas (see S3 Table) (see discussion).

The time scale of predictions

Up to this point, we have performed all analyses (Fig 2 and 3 as well as S5 Fig and S3 Table) with 50-ms bins, i.e. at 20 Hz. This represents a modeling decision: the various layers of the various ANNs operate at various sampling rates, which we have simply resampled to a single rate (20 Hz) in order to match a single bin size for counting spikes (50 ms), which is itself arbitrary. At the extreme, the biological and artificial activities could be summed over the entire stimulus-presentation period, and just one prediction made per stimulus. This is the approach that has (necessarily) been taken in the fMRI studies described previously [2,7–12]. However, the large values of correlations in Fig 3 and the examples in Fig 2A strongly suggest that the ANNs in this study can predict neural activity at much finer time scales than this. Here we ask more precisely what resolution yields the best correlations.

To answer this question, we low-pass filter the ANN activities with various cut-off frequencies before attempting to predict spikes binned at “high frequency” (50 Hz). If increasing the cut-off frequency—e.g., above 10 Hz—increases model-neuron correlations, then the model is capable of explaining aspects of the neural activity that are faster than 10 Hz. Fig 4 shows the result for a range of cut-off frequencies up to the Nyquist limit, for all six (trained) ANNs. More precisely, at each cut-off frequency, we plot the distribution of model-neuron correlations for speech stimuli from the layer that is most predictive with 50-ms bins. The pattern is the same for all networks: model-neuron correlations increase monotonically with cut-off frequency. This shows that the ANNs are predicting fine temporal structure in the spike-count sequences, all the way up to their Nyquist limits.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Distributions of model-neuron correlations as a function of maximum frequency of the model predictions.

Median (solid line) and interquartile range (shaded) are indicated. For each ANN, results are shown only for the layer that was most predictive at 50-ms bins. First, the layer’s responses were low-pass filtered at the frequency indicated on the horizontal axis. Then a linear readout (TRF) was fit to predict spiking activity binned at 20 ms. The vertical axis shows the resulting distribution of correlations on the test set. The red star indicates the cut-off frequency yielding the largest median (across multi-units) correlation; black dots indicate frequencies yielding correlation distributions indistinguishable from that at the red star (Wilcoxon signed-rank test with p < 0.01).

https://doi.org/10.1371/journal.pcbi.1013334.g004

Hierarchy

We now investigate the relationship between the hierarchies of auditory cortex and of the deep neural networks used to explain it. So far we have not distinguished between neurons recorded from core areas and those recorded in non-primary (belt, parabelt) regions of auditory cortex. Here we ask whether a multi-unit’s location in auditory cortex (and, by implication, the auditory hierarchy) is related to the depth of the ANN layer that best predicts its responses to speech and monkey vocalizations.

For each well-tuned multi-unit and each ANN, we find the depth of the most predictive layer, as a fraction of the total network depth. Fig 5 shows the distribution of these depths across all networks, broken out by primary (blue) and non-primary (orange) multi-units. (We break these out by network in S6 Fig.) Whether the stimuli are speech or monkey vocalizations, non-primary MUA is significantly more likely than primary MUA to be best predicted by deeper layers (Wilcoxon rank-sum test, p < 0.001). Nevertheless, the primary and non-primary distributions are broad compared to the differences between them. This suggests a “soft” anatomical hierarchy, with many neurons in belt and parabelt playing the role of lower-level neurons, and vice versa.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Distributions of most predictive layers (normalized) for primary (blue) and non-primary (orange) multi-unit activity.

Histograms and corresponding kernel density estimates are shown as a function of network depth (from shallowest to deepest), pooled across all neurons and all six ANNs. The distribution of preferred layers is significantly “deeper” for non-primary than primary neurons (Wilcoxon rank-sum test, p < 0.001).

https://doi.org/10.1371/journal.pcbi.1013334.g005

Untrained ANNs and STRFs as encoding models

Nevertheless, although worse than the best trained ANNs, STRFs and untrained ANNs provide perhaps surprisingly good encoding models. We believe this has sometimes been obscured in the literature, due to the different ways of constructing STRFs and “untrained” models. In short, the optimal STRF should predict neural responses better than untrained ANNs that take the waveform or a suboptimal spectrogram as input; but the same as or even worse than untrained ANNs that take the optimal spectrogram as input (recall Fig 3). The rationale is as follows.

In our study, untrained models were produced by (re-)initializing the network weights and biases according to their default initialization schemes. These typically set the weights to small random values (and biases to zero), which produces full-rank matrices with high probability. Furthermore, the activation functions in these ANNs are often linear (or even identity functions) near zero. Since the random weights are small and symmetrically distributed about zero, their product with inputs is also typically near zero, and therefore passes through the linear portion of the nonlinearity. So the “nonlinearities” in untrained networks are mostly information-preserving linear transformations.

Consequently, the first few layers of untrained networks that take the spectrogram as input are likely to linearly predict neural activity as well as the spectrogram itself, i.e. as well as a STRF. This is just what we find for the Whisper networks and DeepSpeech2 (Fig 3). It is not what we find for speech2text, whose early layers are much worse than the (optimal) STRF. But this is because speech2text takes a different spectrogram as input, one that is suboptimal for predicting neural responses to audio (S2 Fig, S5 Table). Similarly, untrained networks that take the waveform (rather than a spectrogram) as input will generally provide worse models of neural activity than the (optimal) STRF, because they are mostly linear in the waveform across layers. This can be seen for wav2letter and wav2vec2 in our study (Fig 3) and for wav2vec2 in (e.g.) the recent study using ANNs as encoding models for the electrocorticogram [14].

Results in the literature that seem to contradict this ordering of encoding models are sometimes misleading. For example, a recent fMRI study in humans listening to speech [9] found that untrained wav2vec2 (which takes only a waveform as input) outpredicts the mel spectrogram (figure S3, op. cit.). But that spectrogram does not logarithmically scale the amplitudes, which greatly reduces its predictive power (see again S2 Fig). More subtly, “untrained” models that are constructed by permuting the weights of trained models, rather than re-initializing them, can behave quite differently. Trained weights will typically be large, pushing outputs into saturating regions of some of the nonlinearities, losing information. This can degrade an ANN’s performance as an encoding model well below the level of the STRF, even though the ANN takes the spectrogram as input [12]. For example, in S7 Fig, we re-run our analyses on the cochresnet50 from Tuckute and colleagues, using both re-initialization and permutation to generate untrained networks. For monkey vocalizations, in particular, the permuted network produces much worse predictions than the re-initialized network.

The effect of the choice of ANN on encoding performance

Perhaps surprisingly, we did not observe systematic differences in model performance based on architectural choices. The best overall encoding model, for responses to both English speech and monkey vocalizations, was the first recurrent layer of DeepSpeech2. But the first self-attentional (transformer) layers of the Whisper networks provide only slightly worse predictive power, particularly for speech. And the purely convolutional wav2letter nearly matches DeepSpeech2’s performance on monkey vocalizations. This last fact likewise shows that the performance of trained networks, in contrast to untrained networks, is not determined by the choice of input (waveform vs. spectrogram). Nor does network size seem to matter much, since the two Whisper models are nearly identical as encoding models.

There is, however, some reason to believe that using more variable training data improves subsequent encoding performance. The two Whisper networks and DeepSpeech2 make consistently strong predictions across stimulus types, and these are the networks trained on large, heterogeneous data sets. Indirect evidence is also provided by the fact that pre-training wav2vec2 on AudioSet [37] (a diverse set of sounds from internet videos) rather than LibriSpeech, and skipping fine-tuning altogether, boosts the encoding performance of deeper layers (S4 Fig). On the other hand, this does not improve the peak encoding performance of wav2vec2. We consider this to be an important avenue for future research.

Speech as a stimulus and speech-trained ANNs as models

We chose speech-recognition for the pretext task because (1) we have found English sentences to elicit robust responses from neurons in squirrel-monkey core and belt; (2) monkey vocal calls contain somewhat similar spectrotemporal features; and (3) labeled training sets for English speech are several orders of magnitude larger than anything available for monkey calls or environmental sounds. Still, those more ecologically relevant sounds do lack the complex phonemic structure of English; and, conversely, contain sounds not found in English. From this perspective, it is not surprising that the deepest layers of the ANNs do not provide as good features for explaining the activity of neurons in the squirrel monkey. Furthermore, although we found nearly the same number of core multi-units tuned or well-tuned to speech as to monkey vocalizations, this ratio is much lower in non-primary areas (S3 Table). This is consistent with the observation that non-primary neurons are more specialized and sparsely firing. Similarly, the explanatory power of our ANNs is on average worse for neurons in non-primary rather than in primary areas. This suggests that we are still missing important features for explaining higher-order auditory cortex.

One way to address these issues would be to train ANNs unsupervised on monkey vocalizations or environmental sounds. However, comparing the results of such an experiment with the results of this study is not straightforward, because it is harder to evaluate what “good” performance is on the unsupervised task. For example, if such a network outperforms the networks evaluated in this study, is it the result of better task performance or of a better task? We have therefore not taken this approach in the present study, although we consider it the clear follow up to this investigation.

Correspondence between the hierarchies of ANNs and auditory cortex

We found a soft correspondence between (on the one hand) the hierarchy of the ANNs and (on the other) the distinction between primary and non-primary auditory cortex (Fig 5). A similar, although perhaps slightly less subtle distinction was found by Li and colleagues [14] in ECoG data—in this case between the homologous regions in humans, Heschl’s gyrus and superior temporal gyrus, respectively (figure 2a, op. cit.). In fMRI studies [9,12], the correspondence is more conspicuous still. Although all of these studies are in humans, rather than squirrel monkeys, the emergence of more robust hierarchical distinctions at coarser spatial and temporal granularities suggests that single neurons are simply more diverse in their functional role. Averaging over time and space may mask this diversity.

ANNs can predict the number of spikes in small bins

What the present study reveals, that cannot be observed from the electrocorticogram or fMRI, is that the representational correspondence with artificial neural networks emerges at the level of the single neuron, or at least the multi-unit (typically 1–5 neurons), not merely at the population level: The number of action potentials occurring in 20-ms intervals is well explained by features learned by the ANNs.

So ANNs can explain spiking activity in auditory cortex at time scales as fine as the networks’ Nyquist limits. But can they explain neural activity at time scales finer still? After all, neurons in the primate auditory cortex are known to encode information about stimuli at almost millisecond precision [38].

Here we are limited by our networks and, presumably, the pretext task. The Nyquist limits of the ANNs are set by architectural choices (e.g., stride lengths of convolutions), which could in theory be changed. However, these choices were made to achieve optimal performance on the pretext task; in particular, by the intermediate layers, sampling rates decline to about 50 Hz, which is on the order of the phoneme-production rate (10–20 Hz). Thus, complex features that require multiple layers of nonlinearities but are at finer time scales will not be predictable by such models, and the models probably cannot be changed to predict them without sacrificing performance on the pretext task.

This is another reason to explore non-speech ANNs. An interesting alternative, however, would be use sped-up speech as stimuli. This would retain the same enormous, labeled datasets for training; increase the optimal sampling rates for the ANNs; and (arguably) decrease only minimally the relevance of the stimulus to the monkey.

Biological implausibility of networks components

Transformer attention, bidirectional recurrence, and standard convolutions are all non-causal. In theory, all of these could be remedied (with causal masking, the use of unidirectional RNNs only, and causal convolutions, respectively), although these would have adverse affects on performance on the pretext task. We have made no attempt to address these here and consider this important future work. Similarly, none of the ANNs allow information to flow from deeper to earlier layers during processing, whereas primate cortex is known to have dense feedback connections from higher to lower areas of the processing hierarchy. A new class of ANNs known as “predictive coding networks” has attempted to capture this aspect of neural computation [39]; they would make potentially very interesting alternatives to standard ANNs as encoding models for the cortex.