Linear encoding models for fMRI

Encoding models for the visual cortex that attempt to predict the neural response to an image often rely on two-stage modeling [19]. First, a predetermined nonlinear transformation is used to extract relevant features from stimulus images. Second, for each voxel, a linear model is trained to predict the voxel response as a linear function of the image features. Typically, ordinary least squares or ridge regression [20] (-regularized mean-squared error minimization) are used as the feature covariance matrix inversion is shared across all voxels. The nonlinear feature mapping may vary based on what ROI the voxel is in, and can be informed by the hypothesized role of the ROI in question. The work by [6] used a receptive-field model based on Gabor wavelets to predict activity of voxels in early visual regions, namely V1, V2, and V3. Semantic labels that describe basic object and action categories for an image have been used as features for encoding ROIs higher in the visual processing stream [7]. Along similar lines, the work by [21] compared linear encoding models using Fourier power characteristics, subjective distance, and object categories as different possible feature representations and found that while the Fourier characteristics were more predictive of voxels in V1, the object categories were more predictive of voxels in ROIs higher in the visual processing stream like RSC, OPA, and PPA (an ROI associated with encoding information about the spatial content of scenes [22]). Multiple feature spaces can be used together through group ridge regression [19,23].

Neural network-based encoding models

More recently, representations from convolutional neural networks (CNNs) have been used as the basis for linear encoding models [24,25]. The work by [24] extracted the activations from one layer of a CNN, applied spatial subsampling for dimensionality reduction, then fit a ridge regression model using the features to predict the activations of each voxel. The work by [25] used a similar approach but without the intermediate dimensionality reduction. The works by [24] and [25] showed that the hierarchy of layers of CNNs trained on object recognition tasks had correspondences with and could be used to predict voxels throughout the human visual processing hierarchy. Specifically, they showed that the early convolutional layers of CNNs tended to be more predictive of voxels in early-visual regions, and the later layers in CNNs tended to be more predictive of ROIs higher in the visual processing stream.

Recently, more modern approaches that use various neural network architectures have been proposed for fMRI encoding in the visual cortex. These approaches allow for more flexibility in the learned feature representations of stimulus images. The work by [26] used the pretrained AlexNet as the feature extraction model for predicting voxel activations in early visual regions. The work by [27] trained a variational auto-encoder that can be used for both encoding brain responses and decoding viewed images, although they found lower encoding accuracies with this approach when compared to a CNN model. [28] used different deep neural networks with the same architecture trained for different vision tasks and found correspondence between the task and ROI, for example, edge detection for V1 and semantic tasks for later layers. At the same time as the writing of this paper, the work by [29] showed that neural networks trained from scratch to predict neural activity in V1-V4 using various types of hierarchies can outperform pretrained AlexNet. Our work differs in the choice of CL for fine-tuning, which has not been seen in any previous work on encoding models. However, for the purpose of fMRI decoding, a recent paper used CL to learn embeddings of fMRI responses in the image space of a pretrained CLIP model that could be used for decoding and reconstruction of the visual stimuli [1].

Contrastive learning

Contrastive learning (CL) [30] is a framework that has been adopted for self-supervised learning [31,32] and mutual information estimation [33,34]. The goal is that non-linear embeddings for one [4] or more [5] input spaces based on knowledge of positive pairings of input examples that share information and should be similar after embedding as contrasted to negative pairings of independently sampled examples. The positive and negative pairs may be constructed in various ways. In the SimCLR [4] formulation, CL is used for self-supervised learning on image datasets without labels. Positive pairs are created by associating two modified views of the same image, created by data augmentations like rotation, cropping, and color distortion. The augmentations are chosen such that they clearly encode the same information for downstream image classification tasks. The goal is to learn an embedding that ignores the variance due to augmentation and preserves the shared information between views. CL creates representations useful for downstream tasks [35], which has been theoretically justified in the recent work by [12]. In our case, the two views stem from the same image, but consist of the original stimulus images and the corresponding brain response. Therefore, a positive pair in our case is an image and its corresponding neural response (beta coefficients extracted from the fMRI BOLD response), and a negative pair is an image and a fMRI response for some other non-matching image. This is akin to pairing of an image with a caption, as in the CLIP model [5], which has been previously used for neural decoding of images from fMRI [36].

Several previous works have assessed the ability of neural networks which were trained using self-supervised methods such as CL to predict neural responses in the visual cortex. Prior work [37,38] found that linear encoding models that used image representations from ResNets [39] trained using CL achieved better encoding performance than encoding model that used features from models trained with supervised techniques, even across networks with similar architectures [40]. The key novelty of our approach is that we use CL to fine-tune a neural network using the neural responses themselves. Furthermore, we conduct analyses of how the CL fine-tuning affects the best intermediate feature layer for encoding, the image representations of the model, performance on downstream tasks, and the use of the model representations as prompts for a generative task.

Dataset

The dataset used for the modeling and majority of our analysis is the Natural Scenes Dataset [3], which consists of natural image stimuli and the average beta coefficients across multiple trials of a generalized linear model for the BOLD response captured by high-resolution 7T fMRI. Each of the eight subjects viewed approximately 9,000–10,000 distinct images across 3 trials (approximately 30,000 trials in total across multiple sessions). While beta coefficients were fit using voxel-specific hemodynamic response functions to each trial, our analysis uses the average across the 3 trials as provided by [3]. The images used as stimuli are from the Common Objects in Context (COCO) database [41]. Data is available for voxels in the visual cortex that were responsive to the visual stimuli, with approximately 20,000 total voxels per subject across the left and right hemispheres. A wide array of ROIs are available, which were manually segmented using functional properties. Most ROIs have both a left hemisphere (LH) and right hemisphere (RH) group of voxels, and we treat each hemispheric region as its own ROI. The data was organized as part of the Algonauts Challenge, an open competition where participants attempted to predict visual cortex responses to complex natural images [42]. We split the dataset (image and average response pairs) for each subject such that 85% of the data is used for training and validation of the neural network fine-tuning, and the remaining 15% is used for fitting and evaluating the encoding models, with the split consistent across all ROIs for a given subject (but not across subjects, since different subjects viewed different images during the trials). The size of the test sets for subjects 1–8 are 1477, 1477, 1363, 1317, 1477, 1363, 1477, and 1317 images, respectively.

Encoding model overview

In our proposed approach, for each subject and ROI we fine-tune the parameters of an instance of a pretrained AlexNet CNN to act as the feature extraction model. This results in a total of 280 feature extraction models, where each model is specific to a single subject and hemisphere-specific ROI.

For both our proposed approach and baselines we fit neural encoding models for each subject’s ROIs using three stages: first, feature extraction is performed by passing the stimulus image to the pretrained or fine-tuned CNN, the activations at a particular layer are treated as features; second, because early CNN layers have high-dimensional activations, the number of dimensions is reduced via principal component analysis (PCA); and third, a linear model is fit from these features to each voxel using ridge regression (-normalized linear regression).

To quantify the accuracy of the prediction, we evaluate the average correlation between the predicted responses and the true responses and average across all voxels in the ROI, as described in Accuracy Metric.

Baseline feature extraction models

As a baseline feature extraction model we consider the AlexNet CNN [9] pretrained on ImageNet [43]. All encoding models are specific to a particular subject- and hemisphere-specific ROI, but the base model (with or without fine-tuning) serves as a backbone for all image feature extraction models.

To match the expected input for AlexNet, the stimulus images are all resized to 224-by-224 pixels and their means and standard deviations are normalized so that the means across the 3 colors channels are 0.485, 0.456, 0.406, and the standard deviations across the color channels are 0.229, 0.224, and 0.225 (a standard preprocessing step for using an AlexNet model that was pretrained on ImageNet).

Pretrained AlexNet model.

We consider using the different layers of AlexNet (either after one of the five convolutional layers or after one of the 3 fully connected layers), the dimensions of which are listed in Table 1, for feature extraction.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Dimensions of the output at 8 different AlexNet layers, where the spatially organized convolutional layers (conv) have been vectorized. The final layers are fully connected (FC), with the final output being 1000 dimensions.

https://doi.org/10.1371/journal.pcbi.1014656.t001

The pretrained AlexNet model acts as the starting point for all functions performing image feature extraction. Tapping into the AlexNet model at layer with L = 8, we can select different amounts of processing of the original image for different ROIs.

Before the linear model is fit, for each layer , we apply PCA to reduce the dimensionality of the representation to be constant for all layers, taking the first d = 1000 principal components of the training images’ representations for a subject (as training images differ by subject).

For each ROI, we select the best representation layer for linear encoding from the pretrained AlexNet model by evaluating the cross-validated performance of an -regularized linear model fit for each voxel in the ROI using the dimension-reduced activations by evaluating the eight possible layers . After fine-tuning, we do the same search with the fine-tuned model to find the optimal layer for encoding with the tuned models.

Mathematical formulation of neural encoding framework

Let represent a single image (for AlexNet, ). For a given subject and ROI, let represent the corresponding v-dimensional cortical response in the ROI. The encoding model is a function that predicts the v-dimensional cortical response as a composition of two models: the ROI-specific feature extraction model , where d is the output dimension of the feature extraction model, and a linear model described by coefficients and biases . Specifically, let denote the activations at the output of f appended with a 1, then is prediction of the cortical response in the ROI to image x.

Let denote the family of feature extraction models. Ideally, the goal is to optimize the feature extraction model for the ROI that minimizes the expected sum of mean-squared error across voxels , giving an optimal regression for each voxel. This is stated as the following optimization problem:

(1)

In practice, limited training data is available to compute the expectation, and the capacity of , especially when it is a deep CNN, leads to overfitting.

To improve generalization, we use a three stage approach. First, we select a particular layer of a CNN after the CNN has been fine-tuned with a contrastive learning loss function to serve as the feature extraction model; second, we apply principal component analysis (PCA) to reduce the dimension of the features; and third, we apply -regularized linear regression to the reduced feature set to form the linear model. The PCA and regularization both reduce the effect of overfitting, but the PCA step is also essential to reduce the computational demands of forming and inverting a high-dimensional covariance matrix.

In the first stage of our proposed approach (and the regression-tuned baseline), we fine-tune an instance of the pretrained AlexNet model using contrastive learning for each ROI and individual. Next, given an ROI-tuned feature extraction model consisting of the fine-tuned AlexNet at the lth layer with output dimension , denoted , we apply PCA to reduce the dimension, yielding the dimension-reduced activation vector , where , is the matrix of the leading k eigenvectors defining the principal components of the activations in the training set . Given , an -regularized linear model is fit with voxel-specific penalty terms (selected by cross validation) as

(2)

The predicted neural response to image is then . The overview of our approach is depicted in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the proposed approach.

(Green arrows) The brain response to natural images (GLM beta coefficients fit to the fMRI) are organized into visual ROIs. (Blue arrows) Contrastive learning (CL) is used to fine-tune the AlexNet CNN f based on a loss that maximizes the cosine similarity sim of processed image-response pairs , contrasted to random pairs with (not depicted in the figure). is obtained by passing the image x through the AlexNet CNN f and then a linear projection and the shared non-linear projection head p. is obtained by passing the fMRI response through a linear projection and then the shared non-linear projection head p. f is updated by back-propagation using the gradient . (Purple arrows) An encoding model for the ROI uses the output from layer l of the fine-tuned AlexNet, applies PCA, and then fits a penalized linear model via ridge regression. Performance of the encoding predictions are assessed on held-out images via the correlation coefficient averaged across an ROI’s voxels.

https://doi.org/10.1371/journal.pcbi.1014656.g001

Accuracy metric

To assess the performance of a model, we compute the average correlation coefficient between the predicted response and the true response across all voxels in that ROI

(3)

where is the Pearson correlation coefficient between (the predicted responses for the jth voxel of the ROI) and the true responses . Accuracies over the entire test set are computed as the average over all test set images.

Neural network regression model.

Based on Eq 1, and to parallel the CL-approach, we create a fine-tuning baseline that tries to predict the voxel activity in a given subject and ROI from the final AlexNet activations of dimension d = 1000. The feature extraction model (and subsequent layers) is optimized using the mean squared error between the predicted and actual fMRI response as the loss function,

(4)

where the parameters of both the linear mapping at the output and the entire AlexNet are optimized during training, with the latter initialized as the pretrained AlexNet CNN. After fine-tuning, the activations at a specific layer is dimension-reduced via PCA and an encoding model is fit to predict voxel responses. While indirect, i.e., the encoding model built at the output of the fine-tuned CNN is not actually used, using regression to fine-tune the image feature extraction model specific to a subject and ROI has the potential to outperform the baseline pretrained model, but not overfit as much as a direct encoding model as in Eq 1. The hyper-parameters of the regression-tuned encoding model, layer and voxel-wise penalty are chosen using the same cross-validation approach as in Encoding Model Training, Hyper-Parameter Selection, and Evaluation.

Contrastive learning for ROI-specific feature extraction

The crucial technique in our proposed approach is the use of a contrastive loss function to fine-tune the ROI-specific feature extraction model. The goal is to learn representations of images that emphasize features that are relevant for a given ROI. Contrastive estimation uses machine learning to estimate log density functions [30] based on discriminating ‘positive’ true data from ‘negative’ data created from noise or a source of randomness. In the context of paired random variables, the noise is independent observations, and estimates of the log-density of the conditional distribution have been shown to lower-bound mutual information [33]. Maximizing this lower-bound provides an objective for optimizing the image feature extraction model.

Inspired by the SimCLR framework [4], which optimizes a feature extraction network by using differently augmented pairs of the same original image as positive pairs and augmented pairs of different images as negative pairs, we propose a multimodal CL approach akin to the CLIP model [5] that takes a given image paired with its corresponding fMRI response as a positive pair, and an image and a disparate fMRI response , , as a negative pair. The contrastive loss aligns , the response in the ROI, to the output of f. After fine-tuning, the feature extraction model f uses the features extracted at layer l of f.

Given a batch of K paired vectors in the same space , a contrastive loss inspired by SimCLR [4] is

(5)

where is the cosine similarity and is a temperature parameter. While slightly different due to the dropping of the positive pair in the denominator, this contrastive learning loss function can be related to a lower bound on mutual information [33], as discussed in S1 Appendix.

To map the ROI-specific image features and the ROI response to a common space where we can evaluate cosine similarity, we propose to use two linear mappings denoted by matrices and to map the fine-tuned activations and , respectively, to a shared vector space . As in the SimCLR architecture [4], this is followed by a shared non-linear function, specifically, a multi-layer ‘perceptron’ (MLP), referred to as a projection head (we note that CLIP [5] does not use a projection head). The vectors in the shared space are then and . The model architecture is summarized by Fig 1.

The optimization problem is

(6)

where the expectation is over mini-batches drawn from the training set defined by the mapping function created by sampling K integers from without replacement. Based on the bound on mutual information (see S1 Appendix), K should be as large as possible [4,33], but is limited in practice since the computation of has quadratic complexity with respect to the batch size.

Once the subject and ROI-specific model is fine-tuned, the same layer and penalty term cross-validation search as in Mathematical Formulation of Neural Encoding Framework is applied to build the final encoding model. Essentially the subsequent layers, which are also fine-tuned, are discarded for the purposes of encoding, as in the regression-tuned baseline. Dropping later fine-tuned layers has precedence in self-supervised learning [44]. As in the other baselines, the activations at this layer are projected onto the first k = 1000 principal components based on the training set and -regularized linear regression is used to fit the encoding model.

Implementation details

We set the dimensionality of the vector spaces based on v, the number of voxels in the ROI, with , , and the temperature as . In comparison, for SimCLR [4], , while for CLIP [5], is initialized to 0.07 and limited to 100 but updated in terms of the parameter . The projection head consists of three fully connected layers, with the first two layers each being followed by a batch normalization layer and then a ReLU activation function. The first two layers of the MLP projection head keep the dimension of the projection h the same, and the final layer reduces the dimension to .

We use Scikit-Learn to fit the regularized linear models [45] and PyTorch with CUDA to fine-tune the neural networks [46]. For all subjects and ROIs, the fine-tuning optimization uses batch sizes of K = 1024, 30 epochs for CL and 75 epochs for regression, and an Adam optimizer with a learning rate of for contrastive learning and 10^-4 for regression, betas of 0.9 and 0.999, an epsilon of 10^-8, and no weight decay. The number of training epochs for each method was selected by using a subset of 30 randomly selected ROIs from across the 8 subjects, splitting up the training data further into 85% training and 15% validation, and manually noting when the validation loss had converged for all the ROIs. Learning rates were selected based on the validation loss for a few ROIs, but the grid search was not extensive. We fine-tune the models using T4, V100, and P100 GPUs, on which the training takes roughly 2 minutes per epoch.

Pooled models

For each ROI, we also train a pooled model using the data from all subjects which have that ROI to fine-tune a single shared AlexNet backbone. The architecture is identical to the one described for subject-specific models in Contrastive Learning for ROI-specific Feature Extraction, but with a separate linear mapping for each ROI and subject s, which is required because of the varying voxel mappings and dimensionality for the same ROI across different subjects. To assess the impact of the dimensionality of the latent space , for each ROI we train one pooled model with equal to the average voxel dimension of all subjects with that ROI (rounded to an integer), and another pooled model with equal to the maximum voxel dimension over all ROIs in all subjects. The same training methodology and optimization hyper-parameters as in Implementation Details are used. Notably, one epoch of training involves a pass over the training images and corresponding fMRI responses for each subject, rather than just one subject, which increases the training set size. The performance of the pooled models gives an indication of how transferable the features learned from fine-tuning on one subject are to other subjects.

External dataset validation

A valid concern is whether the CL-tuned models are transferable beyond the Natural Scenes Dataset with its high-resolution fMRI and its ROI definitions. In order to investigate whether the CL-tuned models are extracting images features that are predictive of the fMRI response in corresponding ROIs, we validate them for brain encoding on the Natural Object Dataset (NOD) [11], which uses the Human Connectome Project Multi-Modal Parcellation (HCP-MMP) for ROI definitions [47]. Notably, NOD uses a 3T scanner and also uses stimulus images drawn from ImageNet for some subjects. For our validation, we use the responses to the ImageNet images for subjects 1–9, each of which have recorded responses to 4000 unique ImageNet images (whereas the remaining 21 subjects in NOD only have responses to 1000 images each). Because the fMRI responses are relatively noisy, we follow the encoding experiments in [11] and use only the 50 voxels in each ROI with the highest noise ceiling estimates, which were estimated in [11] as the 50 voxels which were best predicted by a population receptive field model fit to retinotopic mapping data.

We evaluate on the early visual ROIs V1-V4 from the ventral stream in both left and right hemisphere for each subject. For each ROI and subject, we use the CL-tuned pooled model for the corresponding ROI from NSD as the feature extraction model, and following Encoding Model Training, Hyper-Parameter Selection, and Evaluation, we evaluate the 5-fold cross-validated performance of an -regularized linear encoding model fit for each voxel in the ROI using the dimension-reduced activations from each of the 8 possible output layers from the CL-tuned AlexNet model, along with untuned AlexNet as a baseline. Note that this cross-validation technique is the same as the one used to evaluate the encoding models for NSD, but here we use the entire NOD dataset since none of the data was used for AlexNet fine-tuning.

Downstream classification tasks

To quantitatively analyze the CL-tuned feature extraction models, we build classifiers on top of the untuned and ROI fine-tuned models for standard image classification tasks. We use three different image classification datasets: ImageNet ILSVRC2012 [13], which is the dataset used in the AlexNet pre-training, Caltech256 [14], and Places365 [15]. ILSVRC2012 contains 1000 different classes, Caltech256 has 255 object classes (plus 1 catch-all “clutter” class), and Places365 has 365 classes. The Caltech256 and Places356 datasets provide external validation of the fine-tuned models, with the former containing classes focusing on specific objects or figures like bears, helicopters, and microwaves, while the latter’s categories are more general scenes like bedrooms, cafeterias, and staircases. We use the validation split of ILSVRC2012 containing 50,000 images, the entire Caltech256 dataset containing 30,607 images, and the validation portion of Places365, which has 36,500 images. For each, we randomly split the dataset into 85% for training and 15% for testing. We use the penultimate layer’s 4096-dimensional activations of the tuned or pretrained AlexNet CNN as feature representations of the images, fit a multinomial logistic regression model to predict the labels of the training data, and then evaluate the accuracy of the classifier in predicting the labels of the testing images. For ILSVRC2012, this allows us to probe how much fine-tuning (CL or regression) degrades the downstream performance as the model focuses on features relevant for predicting the activity in a subject’s ROI, possibly forgetting features useful for the classification task. For the Caltech256 and Places356 tasks, this accuracy indicates how well the features of the untuned and fine-tuned models can be used for datasets outside of the one that they were initially trained on. In summary, we train a classifier on top of the pretrained AlexNet CNN (replacing its own classification head), each subject-ROI-specific CL fined-tuned model, each pooled-subject, ROI-specific CL fine-tuned model (with both choices of embedding layer dimension), and each subject-ROI-specific regression-tuned model.

Model landscapes

Given a set of classification models, prior work [8] proposes an approach for visualizing the principal components of their pairwise dissimilarities based on the average similarity of their predictions across a test set. For M models, the pairwise dissimilarity matrix is computed as the average of the negative logarithm of the Bhattacharyya similarity B of the class prediction probabilities across a test set of size , , , where and represents the mth classification model’s probabilities estimates across the C classes for image . Multidimensional scaling (MDS) is applied to derive a 2D embedding for the mth model with coordinates , where is the mth entry of the eigenvector corresponding to the pth largest magnitude eigenvalue of the matrix , where is the centering matrix and is a matrix of all ones. Beyond visualization, the correspondence of the 2D embeddings to ROI groupings can be quantified by the well-known Silhouette index [48]. We estimate jackknife confidence intervals (95%) around the Silhouette index by calculating Silhouette indices after leaving out each subject in turn (we recompute MDS in this loop).

We also compute representational similarity analysis (RSA) to compare representations (e.g., model embeddings or the brain response in an ROI) [16]. In this case, where the Spearman rank correlation is computed between the entries of the representation similarity matrices using the representation , and symmetric dissimilarity measure . RSA is a general tool that can measure how similar the representation (brain or model) are in terms of the pairwise ordering of the choose 2 dissimilarities. Here, we adopt either the Euclidean distance or the ‘correlation distance’ dissimilarity , for two representations .

Generating stable diffusion images from CL-tuned models

As a simple example of the type of “in-silico” experiments that can be conducted with the brain-tuned models for different ROIs, we use the CL-tuned models to generate prompts for Stable Diffusion [17] to examine perceptual differences in the generated images from the models tuned on different ROIs. Stable diffusion models often use a CLIP text embedding as the prompt to handle image generation. Image-to-prompt [49] proposes a method to directly use an image embedding as the prompt. However, it requires that the image and text embeddings lie in the same learned latent space. Since the training of the stable diffusion model [17] is conditioned on CLIP ViT-L/14, it is necessary to align the image embedding spaces first. We formulate the space alignment problem as the well-known orthogonal Procrustes problem [50], which aims to find a matrix with orthonormal columns that minimizes the mean-squared error as a regression problem between the embeddings. It has a closed-form solution in terms of the singular value decomposition of the product of the two embedding matrices. For each CL-tuned model that is specific to a subject and ROI, we use the d = 1000-dimensional embeddings of the fine-tuned model f for the NSD training images and the corresponding embeddings produced by CLIP ViT-L/14. Note that CLIP embeddings of images capture semantics they may not capture information about spatial arrangement.

Pretrained AlexNet model

As described in Baseline Feature Extraction Models, we perform 5-fold cross validation to jointly select the AlexNet output layer and the penalty term for each voxel for the ridge regression for each ROI. We include the layers after each of the 5 convolutional layers and subsequent activation functions and the 3 fully connected layers for a search space of 8 possible output layers. Fig 2 shows the average test cross validation encoding accuracy for each ROI and AlexNet output layer, with the best output layer for each ROI highlighted.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Untuned AlexNet encoding scores per layer.

Matrix of encoding scores for each ROI and AlexNet layer, where the entries are the average test encoding accuracy over 5-fold cross validation when a particular AlexNet layer’s activations (after PCA for dimensionality reduction) are used to predict a particular ROI (using the best penalty term for each voxel and layer for ridge regression). Right hemisphere (rh) ROIs are omitted if they have the same best AlexNet layer as their corresponding left hemisphere (lh) ROI.

https://doi.org/10.1371/journal.pcbi.1014656.g002

We find that all of the early visual ROIs (V1-hV4) are best predicted by either the second or the third of the five convolutional layers, and that 24 of the 28 higher visual ROIs (counting left- and right- hemisphere splits as different ROIs) are best predicted by the output of one of the two fully-connected layers. Our findings for the most predictive AlexNet layer agree with previous works that suggest that early-visual regions are more similar to the middle convolutional layers (Conv-2 or Conv-3) in CNNs, and ROIs higher in the visual processing stream (PPA, ventral) have higher similarity with the later fully-connected layers [24,25,51].

Subject-specific models

We first analyze the performance of the subject-specific models described in Contrastive Learning for ROI-specific Feature Extraction on the test set. For each ROI, we evaluate the average Pearson correlation coefficient as described in Accuracy Metric for both baseline models and the CL model, and also judge the performance of the CL model versus the baseline models by computing the percentage of voxels in the ROI that had a higher average test correlation than the baseline model in question across the test images. Subject-specific results for a given subject for each of these groups are calculated as the mean over all ROIs in the group. Table 2 shows the results for the early and higher visual ROI groups.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Subject-specific encoding performance for early (E) and higher (H) visual ROIs using the untuned AlexNet (Ctrl.), see Baseline Feature Extraction Models; with CL-tuning (CL), see Contrastive Learning for ROI-specific Feature Extraction; and with regression-tuned (Reg.), see Neural Network Regression Model. (Avg. ) denotes the test set correlation , described in Accuracy Metric, averaged across ROIs. The last four columns show the percentage of voxels, averaged across ROIs in the group, for which encoding with the CL-tuned model was more predictive compared to either the untuned model or the regression-tuned model. Using a permutation test, lack of significance at a level of 0.01 is indicated by tilde above.

https://doi.org/10.1371/journal.pcbi.1014656.t002

Given the limited number of subjects, we only test for improvements of performance at the coarsest ROI grouping (early and higher). We provide other descriptive statistics at the ROI and voxel level, but do not test at these levels. We find that the CL-based fine-tuning produces significantly higher encoding correlation coefficients than the baseline pretrained AlexNet approach in the early visual ROIs (corrected p-value of 0.001 and effect size 0.465 (Cohen’s d) for a paired, one-tailed t-test with Bonferroni correction for 2 tests, sample size of 8 subjects, and significance threshold of ) as well as in the higher visual ROIs (corrected p-value of 0.009 and effect size of 0.17). While the regression-tuned models have a higher average encoding accuracy than the CL-tuned models in both early visual and higher visual ROIs, the difference is not significant in either the early visual ROIs (corrected p-value of 0.06 and effect size 0.147 (Cohen’s d) for a same test setup as for CL versus untuned), or the higher visual ROIs (corrected p-value of 0.05)).

As an alternative metric, we also examine the average percentage of improved voxels using CL versus the untuned baseline. For the early visual ROIs, the average percentage of improved voxels using CL versus the untuned AlexNet model across subjects is 92.9%, while for the higher visual ROIs, it is 77.6%. Using permutation test we compute the distribution of these statistics per subject for the early and higher ROI groups. As the number of voxels per ROI changes per subject, the threshold for significance at a level of 0.01 varies from 51.0–52.5%. Thus, all subjects improvements are significant accounting for Bonferroni correction for two tests (early and higher). As compared to the regression-tuned models, the CL-tuned models improve an average of 61.5% of voxels across subjects for the early visual ROIs and 57.7% of voxels across subjects for the higher visual ROIs. Using the significance thresholds, 5 of the 8 subjects were significantly better for the early visual and 7 of the 8 subjects for the higher visual ROIs. We attribute the relative strength of CL versus regression-tuned in this metric as compared to the mean encoding accuracy to the training objectives of the different approaches. While the regression-tuned models were fine-tuned by minimizing squared error, which heavily penalizes large errors, minimizing the contrastive loss used in CL may be done by predicting a large proportion of voxels well but having higher squared error on a few voxels.

We also visualize the difference in encoding accuracy for each voxel across the cortex of each subject in Fig 3, where we use the FreeSurfer template [52]. Fig 4 shows which layers of the CL-tuned AlexNet models are most predictive of the voxels in each ROI, while Fig 5 shows which layers of the regression-tuned AlexNet models are most predictive for each ROI. In general, the CL fine-tuning causes the most predictive layer to be 1 or 2 layers later, while the regression-tuning makes the last or second-to-last layer the most predictive in almost all ROIs.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Single-subject CL versus untuned cortical results.

Voxel-wise differences in encoding performance r visualized on FreeSurfer template for each subject. ‘Hot’ colors indicate voxel improvements with contrastive learning (CL) based fine-tuning versus the pretrained AlexNet.

https://doi.org/10.1371/journal.pcbi.1014656.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. CL-tuned AlexNet encoding scores per layer.

Matrix of encoding scores averaged across subjects for each ROI and CL-tuned AlexNet layer, where the entries are the average test encoding accuracy over 5-fold cross validation when a particular AlexNet layer’s activations (after PCA for dimensionality reduction) are used to predict a particular ROI (using the best penalty term for each voxel and layer for ridge regression). Right hemisphere (rh) ROIs are omitted if they have the same best AlexNet layer as their corresponding left hemisphere (lh) ROI.

https://doi.org/10.1371/journal.pcbi.1014656.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Regression-tuned AlexNet encoding scores per layer.

Matrix of encoding scores averaged across subjects for each ROI and regression-tuned AlexNet layer, where the entries are the average test encoding accuracy over 5-fold cross validation when a particular AlexNet layer’s activations (after PCA for dimensionality reduction) are used to predict a particular ROI (using the best penalty term for each voxel and layer for ridge regression). Right hemisphere (rh) ROIs are omitted if they have the same best AlexNet layer as their corresponding left hemisphere (lh) ROI.

https://doi.org/10.1371/journal.pcbi.1014656.g005

Pooled subject fine-tuning

Pooling subjects while fine-tuning the AlexNet for a specific ROI creates a shared representation space that benefits from more data, and has the potential to improve the subsequent encoding for ROIs. The encoding performance results with pooled models (using an embedding dimension equal to the average number of voxels in the ROI across all subjects) versus subject-specific models, summarized in Table 3, suggest that there is higher similarity in visual representations across subjects in the early visual areas than in the higher visual areas. As compared to the subject-specific models, encoding performance is significantly improved when pooling in the early ROIs, but not significantly improved when pooling in the later ROIs. For the early ROIs, the corrected p-value is 0.004 and the effect size is 0.39 (Cohen’s d) for a paired, two-tailed t-test with Bonferroni correction for 2 tests and significance threshold of ; and for higher ROIs, the corrected p-value is 0.36 with an effect size of 0.06. As an ablation to test the effect of the size of the latent dimension, in S1 Table, we show the pooled versus subject-specific results when using an embedding dimension equal to the maximum number of voxels across all ROIs and subjects, which has minimal impact on the pooled results.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Pooled versus subject-specific results for early (E) and higher (H) visual ROIs. For each subject, the average test set correlation r as described in Accuracy Metric across all early visual and higher visual ROIs is reported for CL-tuned AlexNet (Contrastive Learning for ROI-specific Feature Extraction) and pooled CL-tuned models (Pooled Models). The last two columns show the percentage of voxels, averaged across the early and higher visual ROIs, for which encoding with the pooled model was more predictive than the subject-specific model. Using a permutation test, lack of significance at a level of 0.01 is indicated by tilde above.

https://doi.org/10.1371/journal.pcbi.1014656.t003

External dataset validation

Table 4 contains the results of the external dataset validation experiment on the Natural Object Dataset [11], as described in External Dataset Validation. The ROI-specific feature extraction models fined tuned using CL on NSD’s Subject 1 improve the encoding prediction for voxels in the corresponding early ROIs in NOD. For all 9 subjects, more than half of the top 50 voxels per ROI with the highest signal to noise ratio in the early visual areas (V1, V2, V3, and V4) are better encoded (higher correlation) with the CL-tuned models. Overall, the average proportion of voxels improved is 70.6% and the average correlation increased from 0.142 to 0.153, an improvement of 8.1%. This supports the conclusion that the CL fine-tuning creates feature extraction models that process visual stimuli more similarly to the target ROI.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Encoding performance for early visual ROIs (V1-V4) for subjects 1–9 in the Natural Object Dataset using feature extraction models tuned on NSD as described in External Dataset Validation. For each subject, the average test set correlation r as described in Accuracy Metric across all early visual ROIs is reported for untuned AlexNet and the AlexNet models tuned using CL pooled across subjects on NSD. The percentage of voxels for which the CL-tuned model was more predictive on average across the test set compared to the untuned AlexNet is also shown. Using a permutation test, lack of significance at a level of 0.01 is indicated by tilde above (threshold is 58.5% for all subjects).

https://doi.org/10.1371/journal.pcbi.1014656.t004

The average encoding correlations for the NOD dataset are much lower than for the early visual ROIs in the NSD datasets (the best subject’s baseline is 0.222 in NOD versus 0.458 for the early visual ROIs in NSD). This is not unexpected due to three reasons: first, the scanner resolution for NOD is lower than NSD (3T instead of 7T); second, the responses to each image for NOD are only the result of a single trial rather than an average of 3 trials as in NSD; and third, NOD has fewer responses used for fitting the linear models than NSD (3200 instead of approximately 9000 per subject). However, the relative encoding performance for different methods still provides a meaningful metric for how well each feature extraction model is tuned for that ROI. Because the CL–tuned models were fine-tuned on a completely different dataset, the improved encoding accuracy for the brain-tuned models over the untuned models indicates that fine-tuning on NSD generalizes to other datasets.

Downstream classification tasks

Using the methodology from Downstream Classification Tasks we evaluate the test set classification accuracy for image classifiers built on top of the untuned, subject-specific CL-tuned and regression-tuned models for the ImageNet ILSVRC2012, Caltech256, and Places365 datasets. Fig 6 summarizes the performance of the subject specific tuned models versus the untrained baseline. See S5-S7 Figs for the classification accuracy per ROI for both the subject-specific and pooled models for each ROI.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Downstream classification task results.

Results for baseline and brain-tuned AlexNet on the validation portion of ImageNet ILSVRC2012 (the training portion was used for the pretraining the AlexNet baseline), Caltech256, and Places365. The bar height is the average accuracy across all ROIs tuned using that method for that subject. Error bars represent the full range of results across all ROIs (minimum to maximum).

https://doi.org/10.1371/journal.pcbi.1014656.g006

For both the ImageNet classification task and the non-training classification tasks (Caltech256 and Places365), we see a drop in classification accuracy when using the brain-tuned models, but the CL-tuned models suffer much less of a decrease than the regression-tuned models. For ImageNet, the baseline pretrained on ImageNet AlexNet achieves 45.8% accuracy. Across all possible subject-ROI pairs, the best CL model achieves a test accuracy of 43.6% (LH EBA in subject 1), see S5 Fig. This means there is only a 1.6% drop in performance after tuning to brain-activity (while starting from pretrained). For Places365, the most challenging task, the baseline pretrained model achieves 29.1% accuracy. Across all possible subject-ROI pairs, the best CL model achieves a test accuracy of 27.2% (LH OPA and LH PPA in subject 6), see S6 Fig. For Caltech256, the baseline pretrained model achieves 70.3% accuracy. Across all possible subject-ROI pairs, the best CL model achieves a test accuracy of 68.2% (LH EBA in subject 1), see S7 Fig. These results show that on average, the CL-tuning preserves most of the information content necessary for using the image features of the CNN for image classification tasks, especially in task-relevant ROIs. Quantitatively, the values are consistent across subjects.

Representational similarity analysis

While the Bhattacharya based dissimilarity metric allows for direct comparisons of the classification models, RSA allows direct comparisons of representations (brain or network based). At the individual subject level we use the held out test set (15% of the dataset with sample size ranging from 1317–1477) to compare each ROI’s CL-tuned model’s representation (both the full network f and the truncated , where l is the best layer selected for the encoding model) directly to the fMRI BOLD response (beta coefficients). We do the same for the untuned AlexNet representations and the regression-tuned representations (both full network and truncated). The results for the full networks are shown in Fig 7. (Additionally, we compare pretrained AlexNet to CLIP vision transformer, ViT-L-14) in S4 Fig, finding that there is generally lower RSA compared to untuned AlexNet when using correlation on the representation, but slightly higher RSA when using Euclidean dissimilarity.)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. RSA between fMRI to model.

Scatter plot comparing the representational similarity analysis (Spearman’s using either correlation distance (corr) or Euclidean distance (euc) as the dissimilarity measures) between each ROI’s fMRI response and the untuned AlexNet model’s representation, CL fine-tuned model, or regression-tuned (Reg) model. Error bars indicate standard errors across the subjects.

https://doi.org/10.1371/journal.pcbi.1014656.g007

The results show that the CL-tuned models have higher RSA with the fMRI responses than the untuned AlexNet model, especially as measured by the correlation distance. For early ROIs, the RSA are the same or slightly higher for the CL-tuned compared to regression-tuned using the correlation distance; however, using the Euclidean distance for dissimilarity yields higher RSA for the regression tuned. This is logical since the CL training ignores the norm of the embeddings, while for regression it is important. For the higher (later) ROIs, there are still gains for CL-tuned as compared to untuned, but regression-tuned representations have higher RSA for both correlation and Euclidean distance.

Fig 8 shows that the truncated representations, which use the best output layer for the encoding model, generally have lower RSA than the full model. This makes sense because there may be additional variation in the earlier layers that is then filtered out by the encoding weights. One exception is the truncated CL-tuned and regression-tuned models in higher (later) ROIs. In early ROIs, the RSA values with the fMRI are higher for the truncated CL-tuned compared to the truncated regression-tuned. This matches the encoding performance which showed CL outperformed regression in early ROIs.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. RSA between fMRI to truncated models.

Scatter plot comparing the representational similarity analysis (Spearman’s using either correlation distance (corr) or Euclidean distance (euc) as the dissimilarity measures) between each ROI’s fMRI response and the CL fine-tuned model, regression-tuned (Reg) model, or the corresponding network truncated at the layer used for the encoding model (Trunc-CL/T-CL and Trunc-Reg/T-Reg). Error bars indicate standard errors across the subjects.

https://doi.org/10.1371/journal.pcbi.1014656.g008

We also compare the representations among subjects on a subset of 1000 of the images from ImageNet that were used to test the downstream classifiers. We directly compare the Bhattacharyya-based dissimilarity to RSA of the CL-tuned network’s representation created by f in Fig 9. The results show dissimilarity based on the classification head trained for ImageNet classes correlates with RSA for the correlation distance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Bhattacharyya-based dissimilarity compared to RSA.

Scatter plots comparing the Bhattacharyya-based dissimilarity and RSA between all pairs of CL-tuned model across all subjects and ROIs.

https://doi.org/10.1371/journal.pcbi.1014656.g009

Table 5 details the statistics of the RSA values between ROI-specific CL-tuned (and regression-tuned) representations for different subjects. The models for early ROIs have lower intersubject RSA compared to models for higher (later) ROIs. While the regression-tuned representations appear more subject specific, the RSA values must be contextualized by the inter-ROI RSA values. To quantify this we calculate a test statistic (after first performing the Fisher transformation on the similarity values) that is the difference between average intersubject RSA within-ROI and the average inter-ROI RSA, using a permutation test of the ROI labels per subject to generate a surrogate distribution using 2000 permutations, and report the p-value in the table. The results for the two tuning methods match almost exactly, with the CL-tuning also indicating that V2d has more similarity between subjects than between ROIs. The ROIs with the lowest p-values are all higher (later) ROIs.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Intersubject RSA values. Statistics of intersubject RSA values among subjects for the CL-tuned models (top) and Reg.-tuned models (bottom) across ROIs using correlation distance on the log-softmax of 1000 ImageNet images. Mean, standard deviation, and min-max are across the N choose 2 unique pairs. The p-value is for the difference between average within-ROI, intersubject RSA and the average inter-ROI RSA, and is calculated by a permutation the ROI labels per subject across 2000 runs, with bold indicating significance at a level of 0.05.

https://doi.org/10.1371/journal.pcbi.1014656.t005

Model landscapes

To understand the similarity of the classification models trained for the ImageNet dataset on top of the feature extraction models we apply multidimensional scaling (MDS) of the Bhattacharyya based dissimilarity metric [8] between the prediction vectors from the downstream classifier for ImageNet for each model. When we apply this to the full set of models (untuned AlexNet, CL-tuned for each subject and ROI, CL-tuned pooled subject for each ROI, and regression tuned for each subject and ROI), we find that each group of models is distinct, as shown in S8 Fig. In comparison, models landscapes with different subsets of the models will use different components of the dissimilarity space as the axes of model embedding coordinates.

Fig 10 shows the landscape when taking only the untuned AlexNet and subject- and ROI-specific CL fine-tuned models. Models tuned with CL for different subjects, but the same ROI, appear in the same arrangement and are relatively similar based on the their embedding coordinates. The landscape is arranged such that the first axis separates early versus late ROIs, but the second axis separates the left and right hemispheres in the early ROIs (V1–4). This indicates that predictions for left and right ROI models are systematically different across the images—due to the lateralized presence of objects related to the classification task and the retinotopic tuning in V1–4. Higher level (late) ROIs have more similar predictions to untuned AlexNet predictions, and are more accurate.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Overall ROI model landscapes.

Model landscapes for subject-specific ROI-specific CL fine-tuned models and untuned AlexNet using Bhattacharyya dissimilarity based on classification head (Left) or RSA with correlation distance (Right). The early visual ROIs are separated from from higher ROIs along the first dimension (MDS 1) and early ROIs separate into left and right along the second dimension (MDS 2).

https://doi.org/10.1371/journal.pcbi.1014656.g010

Fig 11 shows the landscape when taking only the untuned AlexNet and subject- and ROI-specific CL fine-tuned models for higher level ROIs. Here we see the two axes separate the models by ROI group related to representation of body, face, place, and words. The ROIs associated to face and place are generally worse for classification, but clearly distinct in this embedding. Again, models tuned for different subjects but the same ROI appear in the same arrangement and are relatively similar based on the their embedding coordinates. Notably the untuned AlexNet is not most similar to the highest performing fine-tuned model: subject 1’s EBA (left hemisphere), as detailed in S5 Fig. This reveals that fine-tuned models may have deviations in the predictions from the untuned AlexNet while still retaining features useful for downstream classification tasks.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Higher visual ROI model landscapes.

Model landscape for higher level (late) subject-specific ROI-specific fine-tuned models.

https://doi.org/10.1371/journal.pcbi.1014656.g011

Fig 12 shows the landscape when only embedding the untuned AlexNet and pooled-subject CL-tuned models matches the landscape for the subject-specific CL fine-tuning.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. Model landscapes with pooled-subject CL-tuning.

Model landscapes for untuned AlexNet (star marker) and pooled-subject ROI-specific ROIs. Left plot is all ROIs and right plot is higher level (late).

https://doi.org/10.1371/journal.pcbi.1014656.g012

Finally, we note that the landscape for the regression-tuned, subject-specific and ROI-specific models differs from the CL-tuned model landscapes, as shown in Fig 13. Notably, while the ROIs of different subjects are similar and the primary axis of dissimilarity is associated with early versus later ROIs, the second axis separates place and face ROIs in the later areas. This indicates that the regression tuning yields a different landscape. Looking at only the later ROIs, there is less clustering than in the CL tuning. To quantify this, we compute the average silhouette index for the distances in the 2D MDS landscapes for different ROI groupings and a jackknife confidence interval for each. The results in Table 6 show that the regression-tuned Silhouette score is lower but still inside the 95% CI of the CL-tuned for higher-only. However, using fine-grained ROI labels and grouping higher ROIS the CL-tuning is better and the regression-tuned is outside the CI. Using a coarse grouping of early layers, regression-tuned has higher score and CL is outside the CI. Finally, there is not much difference for early versus late (which has the highest Silhouette scores for both).

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Silhouette scores for ROI groups in landscapes. Four different ROI groupings are explored. Jackknife computed by leaving out one subject.

https://doi.org/10.1371/journal.pcbi.1014656.t006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 13. Regression-tuned model landscapes.

Model landscape for subject-specific and ROI-specific regression fine-tuned models and AlexNet (star marker). Left plot is all ROIs and right plot is higher level (late).

https://doi.org/10.1371/journal.pcbi.1014656.g013

Generating stable diffusion images from CL-tuned models

To illustrate how the CL-tuned models can reveal visual processing of different visual ROIs, we visualize the Stable Diffusion output for the CL-tuned models for left and right hemisphere V1v and EBA for all 8 subjects for a single prompt image in Fig 14. We find that there are distinct differences in the output for V1v and EBA which are consistent across subjects. Whereas the images generated from the EBA-tuned models feature close-ups of people or groups of people (where their faces are visible), the images generated from the V1v-tuned models mostly contain people riding motorcycles, bicycles, or doing other sports, without clearly visible faces. The similarity across subjects and distinctiveness between the outputs from V1v and EBA indicate the the CL fine-tuning induces changes in processing that can be used for “in-silico” experiments with the ROI-tuned models.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 14. Image-to-Prompt Generative Examples.

The shared prompt image (a center cropped image from John Thomas, The National Library of Wales, Creative Archive Licence) are shown with 6 generations of Stable Diffusion for untuned AlexNet, CLIP, and 32 subject-ROI specific CL-tuned models (8 subjects and left/right hemisphere V1v and EBA).

https://doi.org/10.1371/journal.pcbi.1014656.g014

To validate the cyclic consistency of the in-silico framework, for each of Subject 1’s CL-tuned ROI models (and the untuned AlexNet) we generate 1000 images using the CLIP-aligned embeddings of the 1000 images from ImageNet as prompts. We then embed them using the corresponding model. Ideally, for each prompt image embedding, the corresponding generated image embedding would be the closest. Whereas a random permutation have the generated image as the 500th nearest image. We use rank of the distance to the nearest neighbor (using correlation distance) to quantify cyclic consistency for each ROI. We find that the average ranks (denoted ) are generally low (31.5–123) across the ROIs, compared to 30.6 for AlexNet and 11.7 for CLIP (ViT-L-14). CLIP provides a performance bound We also apply RSA to compare how similar the dissimilarity is between the original images and the generated images with values ranging from 0.44–0.74 compared to 0.76 for AlexNet and 0.45 for CLIP. Both quantitative measures are in Table 7. The lower RSA for CLIP is unexpected, as generally the two measures agree across subject 1’s 33 ROIs (Spearman rho of -0.841). To understand the topology of the embeddings of the original versus generated images, we also estimate the intrinsic dimensionality (ID) using the TwoNN [18] method implemented in scikit-dimension [53] using the original images and the generated ones. The values, shown in Table 7, indicate that the generated image representations have a lower ID estimate, indicating that using the embeddings for conditioning and the generating process produces images that could be described as lying on a lower dimensional manifold. Nonetheless, across the ROIs ID estimates of the original and generated latent representations correlate (Spearman rho of 0.718). AlexNet’s ID estimates are 17.8 for the original and 11.3 for the generated. CLIP’s ID estimates are 18.3 and 16.4, respectively. This is the highest ID for the generated images and the lowest gap of 1.9, confirming that CLIP preserves the most information, through this generative process but the generated images exist on a lower-dimensional manifold than the original images. The higher ID for CLIP could explain why its RSA is lower, since there is more variety in the generated images, but Stable Diffusion was originally trained with text embedding conditioning. Thus, while the relative topological arrangement is preserved for CLIP (small ) there is a distributional shift after generation, which the CLIP embedding is sensitive to. In contrast the ROI-tuned embeddings are tuned more “narrowly”, filtering out variation created by Stable Diffusion. Furthermore, it is notable that the ID for many ROI tuned models is higher than CLIP or AlexNet, indicating that they are preserving more information about the original image, which is subsequently lost by the Stable Diffusion generation. In Fig 15, we also show the model landscape derived from the RSA representations of the generated images recapitulates aspects of the subject-specific landscape. We note that for this subject, the second dimension of landscape captures dorsal-ventral differences and the third dimension corresponds to left-right lateralization.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Cyclic Consistency of Image-to-Prompt Generative Images. The average rank () of corresponding generated image (lower is better: 1 is ideal and 500 is chance), RSA values (higher is better: 1 is ideal) using original versus generated images, and the intrinsic dimensionality (ID) estimates using TwoNN [18] of original and generated representations across ROIs.

https://doi.org/10.1371/journal.pcbi.1014656.t007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 15. Subject 1 ROI model landscapes with Generative AI.

Model landscapes for the Subject 1’s ROI-specific CL fine-tuned models using RSA with correlation distance using the original embeddings (Left) compared to embeddings of images generated with StableDiffusion (Right). The early visual ROIs are separated from from higher ROIs along the first dimension (MDS 1) and early ROIs separate into ventral and dorsal along the second dimension (MDS 2), and left and right along the third dimension (MDS 3).

https://doi.org/10.1371/journal.pcbi.1014656.g015

Limitations

We note that a number of benchmarks were set for encoding on the Natural Scenes Dataset (NSD) [3] based on the Algonauts Project 2023 competition [42]. We do not claim to achieve state of the art, but rather we use NSD as a uniquely large dataset to understand how contrastive learning can yield ROI and/or subject-specific tuning of feature extraction models. Our contribution is a method to compare ROI-specific fine-tuned models by examining their transferability and their similarity and performance on different downstream tasks. Likewise, we have used the Natural Object Dataset [11], which uses a different parcellation scheme [47] than the manual ROI definitions in NSD, to confirm that the transferability of the feature extraction models tuned to early visual ROIs which outperform the baseline, but do not directly compare to the previous encoding experiments [11] that using population receptive field models for retinotopic mapping and regularized linear regression.

The design choices we have used may not be optimal. In particular, using ridge regression is not ideal as it does not account for the correlation of neighboring voxels. Partial least squares or rank constraints across the weight matrices may further improve the encoding [57,58].

The choice of a nonlinear projection head, while used in SimCLR [4], is not used by CLIP [5]. Prior work, [4] and [44], showed that more information may be preserved if a non-linear projection head is used, but then dropped. However, for our purposes the choice of a dimension reducing linear projection and/or non-linear projection head on the voxel response is questionable since all of the information in the response is relevant. It may be the case that with the linear projection (of reduced dimension) and non-linear projection head the contrastive learning optimizes the model for a subset of the voxels in an ROI that best aligns with the feature extraction. This could explain why the encoding results are not improved across all voxels. For linear encoding, using the neural response and network activations directly (or through a variance preserving projection) may be more optimal. Nonetheless, it is necessary to have at least one linear map so that the fMRI response and AlexNet embedding are of the same dimension. Future work should examine the benefit of dimensionality reduction and/or the non-linear projection head.

Additionally, our choice of scaling the embedding dimension in the contrastive learning loss to be proportional to the voxels in an ROI for the subject-specific ROI-specific models may not be optimal. With the pooled tuning, we tested whether using a constant embedding dimension in the CL loss performs better than an embedding dimension proportional to the number of voxels. As shown in S1 Table, while not significant in terms of encoding performance, the constant embedding dimension had marginally better performance in terms of both average test set correlation and percentage of voxels improved. This means that the particular choice of proportional scaling may not be an optimal hyper-parameter, but is not degrading performance and does not affect any of the conclusions.

Similarly, we have used a fixed temperature for the loss function. Although the temperature is in the range of prior work [4] it may not be optimal and may limit the lower bound estimate of mutual information. In contrast, CLIP [5] treats the temperature as a learnable parameter.

While we tuned a separate model for each ROI and selected a single layer per ROI, it could be argued that one could use a whole brain approach that tunes a single feature extraction model useful for predicting all ROIs. However, it is not clear how the subsequent analysis of this feature encoding would be approached. Instead, we explicitly disentangle the representation by creating a differently fine-tuned model for each ROIs. The benefit of our approach is that we can analyze the ROI-specific models. However, a limitation of our approach is the computation required—which has limited our hyper-parameter searches. By selecting a single layer from the network for the feature extraction of the encoding model, we essentially assume that each ROI is homogeneous in the complexity of the processing. Given the limited granularity of AlexNet layers and relatively fine-grained ROIs, this assumption seems reasonable.

Future work could, firstly, fine-tune the image feature extraction using information pooled across the fMRI responses across different regions of the visual processing stream (i.e., a whole-brain contrastive learning), and secondly, incorporate cross-attention mechanisms to use different ROIs to tune different parts of the CNN, similar to the approach of [29] used to build models predictive of V1-V4, but extended to more ROIs.

More generally, the contrastive learning approach could be used with neural data from other experimental modalities to tune other deep neural network architectures, such as fMRI responses to spoken or read text being used to modify language models. We note that a limitation of our work is that we used a low-capacity network compared to modern architectures. This was necessary since the computational budget of training all subject and ROI specific models would be higher with networks such as ResNet [39] or vision transformers as in CLIP [5]. However, the fact that CL-tuning still selects earlier layers after fine-tuning indicates that the capacity of the network is not limiting for early vision ROIs. Nonetheless, a higher capacity network may be necessary for higher ROIs or for regression tuning. Although we compared results to CLIP, we did not compare to modern pretrained feature extraction networks such as DINOv2 [59].