1.1. Related work

[3] Basic neuroscience has demonstrated an extensive overlap in the representation of seen and mental images [4–8]. Nonetheless, differences between vision and mental imagery make cross-decoding from vision to imagery challenging [9,10]. Compared to vision, brain activity during mental imagery has a lower signal-to-noise ratio (SNR) [11], varies along fewer signal dimensions [12], and encodes imagined stimuli with less spatial resolution [13,14].

Previous studies have reported successful n-way classification of mental images [9,15,16], retrieval of mental images of natural scenes using a visual encoding model [7,17], and reconstruction of simple blobs, letters, or singular natural objects [6,18–22]. With the open releases of CLIP [23], Stable Diffusion [24], and large-scale fMRI datasets like NSD [25], newer vision decoding models now yield highly accurate reconstructions of natural scenes [26–36]. These methods map fMRI brain activity patterns to embeddings of pre-trained deep learning models that are used to drive a diffusion model [2,37–39] to generate image reconstructions of the content present in visual cortex.

Tests of open-source vision decoding models [2,21,26–29] on mental imagery activity in the NSD-Imagery dataset [1] showed that improved performance on vision decoding does not necessarily translate to mental imagery decoding. For example, MindEye2 [2] was the SOTA model on the test set of NSD, but not on the imagery trials of NSD-Imagery (Fig 1). Furthermore, in that study it was observed that methods with linear backbones, compact representations, and multimodal guidance yielded the best performance. We interpreted these observations as follows: First, [40] showed that linear mappings from CLIP to brain activity explains much of variance in activity in visual cortex. Thus, we should expect linear methods to work quite well when decoding CLIP. Second, as models increase in complexity, the risk of overfitting to noise in the input increases; this danger is especially acute in our case, given that imagery has very low SNR. Thus, representations with fewer dimensions are to be favored, all else being equal. Third, imagery and visual activity are most aligned in brain areas with language-like representations ([3,9,13]). Thus, it would make sense to include high-quality language-like representations in the decoding pipeline for mental imagery. These ideas informed the design of MIRAGE. Our results, including detailed ablation analyses, support our reasoning, and explain why MIRAGE is SOTA for mental image reconstruction.

2.1. Feature metric evaluations

We provide performance benchmarks against existing methods evaluated on NSD-Imagery [1] using the metrics in Table 1. For all methods, we output 10 reconstructions per test sample from each method and report averaged metrics across them. Metrics can fluctuate due to the stochastic nature of fMRI-to-image models, and this averaging step increases the reliability of results. Statistical significance measures can be found in Table A in S1 Text. Across the majority of metrics, MIRAGE shows SOTA performance on mental image reconstructions. We note that although these metrics are often used as a proxy for human judgment, many research efforts have established that these metrics do not closely approximate or align with human assessments of content [49] or quality [50]. We observe that they are particularly volatile with a dataset as small as NSD-Imagery. For this reason, we provide extensive behavioral evaluations of our results by human raters in Section 2.2. Benchmarks separated by stimulus type are provided in Tables C and D in S1 Text. Results of our method on the NSD shared1000 test set are also provided in Table B in S1 Text.

2.2. Human ratings of reconstruction quality

To ensure downstream applicability, reconstruction methods must produce outputs that are meaningful to human observers. Although we report standard image feature metrics, prior work indicates that these automated scores often dissociate from human perceptual judgments of quality and semantic content [2,49,50]. These factors, in addition to the high volatility of automated metrics on small datasets such as NSD-Imagery, lead us to treat human evaluation as the definitive standard for assessing model performance. We thus conducted several large-scale online behavioral experiments in which human raters (n = 500) assessed the quality of the reconstructions (See Appendix A.14 in S1 Text).

Experiment 1 To perform a systematic controlled experiment analyzing our results, we had human raters perform a 2-alternative forced choice (2AFC) judgment about whether a reconstruction was more similar to the ground truth image than a randomly selected “distractor” reconstruction of a brain activity pattern originating from a different stimulus that was sampled from the same stimulus type, method, and NSD subject. Results (Table 2) confirm MIRAGE as SOTA for every stimulus type (p < 0.001)‌‌ (Fig 3).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3.

Human similarity scores for simple and complex stimuli: X-axis = vision, Y-axis = imagery; each point is the mean over 12 samples (larger bold points are the overall means), colored/shaped by method.PCA-fit slopes closer to unity indicate tighter imagery–vision correspondence; dashed unity line shown.

https://doi.org/10.1371/journal.pcbi.1014263.g003

Experiment 2 Human raters viewed a ground truth image, a reconstruction of that stimulus from a vision trial, and a reconstruction of the same stimulus from an imagery trial. Raters provided continuous measures of similarity between each reconstruction and the ground truth image. The rating provided a direct comparison of the similarity between vision and imagery reconstructions. MIRAGE shows SOTA generalization from vison to imagery [40].

Experiment 3 Mental image reconstructions of the conceptual stimuli are particularly difficult to evaluate, as they do not have associated ground truth images or a meaningful match to vision trials. To compare the performance of our method on these stimuli relative to other models trained on NSD, we conducted a third behavioral experiment that pitted reconstruction methods head-to-head by presenting human raters with a ground truth conceptual stimulus and two reconstructions of that stimulus sampled at random from the collection of methods we evaluate against in this work. Every trial in this experiment is a head-to-head comparison between two random reconstruction methods, and so over the course of the whole experiment, we gather lots of trials between all combinations of two methods. The “similarity score advantage” is the average difference between one method and another across the trials where both methods were presented. Fig 4A plots the average “similarity score advantage” for all combinations of methods. Reconstructions from our method are the most strongly preferred in head-to-head comparisons (p < 0.001) and have the largest advantage in all comparison cases evaluated (Table 2).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. (A) Head-to-head human similarity score results for the conceptual stimuli.The Y-axis represents the similarity score advantage (difference between target method’s score and the alternative, on the radial X-axis); a larger colored polygon area indicates a stronger advantage, and the dashed circle at unity denotes equal performance. MIRAGE outperforms all other methods (p < 0.001). (B) Ablation analyses: model variants (numbered circles) under each ablation type (color) are assessed via the normalized average of all feature metrics (Table 1), with vision on the x-axis and imagery on the y-axis. The standard error of each point along each axis is visualized with a shaded ellipse of the same color. For details on how we compute the normalized average metric scores, see Appendix A.6.1 in S1 Text.

https://doi.org/10.1371/journal.pcbi.1014263.g004

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Human identification accuracy scores for the vision and mental imagery trials of the NSD-Imagery benchmark.

Scores are provided for each method and stimulus type; best values are bolded and second best underlined (chance = 50%, all p < 0.001).

https://doi.org/10.1371/journal.pcbi.1014263.t002

2.3. Ablation study

We systematically ablated model components to identify which were most important for mental imagery reconstruction. Colored numeric identifiers refer to Fig 4B. MIRAGE is identified as (1). We also provide a hyperparameter search over the ridge parameter in Appendix A.1 in S1 Text.

Low-Level Module We found that despite the relatively low spatial resolution of mental imagery, the low-level decoding module provides much of the needed structure to accurately reconstruct the target stimulus (3). We also observe that our image filtering technique induces a small performance increase for mental imagery reconstructions (2), suggesting that this technique partially mitigates the loss of structural detail in mental vs. seen image reconstructions.

Retrieval Module We see a notable boost from our retrieval procedure (4), demonstrating that high-dimensional image embeddings can be decoded and used to guide a retrieval module to improve performance, even if those same embeddings are not suitable to drive an image generator.

Caption Types We examine the results of using COCO captions (5), as well as a set of synthetic “short” captions (6) that aim to replicate the length of the COCO captions (the median word count is 8 vs 10 for COCO), and evaluate the quality of captions provided by LLaVA v1.5-13B. These short synthetic captions did not provide a boost over COCO, but the longer synthetic captions (average word count = 34) utilized in our final method (1) provided an improvement.

Diffusion Features We replaced the 1 × 768 image embeddings and 77 × 1280 text embeddings used to drive Stable Cascade with the features used to drive Versatile Diffusion in MindEye1 and Brain Diffuser (7) [39] (257 × 768 image embedding, 77 × 768 text embedding), and SDXL unCLIP [2], the embedding used to drive MindEye2 (8,9) (257 × 1664 image embedding). Our smaller image embedding and larger text embedding provided the most robust guidance for reconstructing mental images (1).

Guidance Modality For both vision and mental imagery, image-only guidance (10) to Stable Cascade afforded better performance than text-only guidance (11). In both cases, but especially for mental imagery, the combination of text and image guidance (1) provided a large performance boost, suggesting that multimodal guidance plays an important role for mental imagery reconstruction.

Backbone Architecture The MLP backbone and diffusion prior architectures of MindEye1 (14,15) and MindEye2 (12,13) perform worse than our ridge regression backbone (1) for both vision and mental imagery, suggesting that these architectures tend to overfit to the training data in the core NSD experiment. Evidently, the potential for increased expressivity afforded by an MLP is a disadvantage when attempting to generalize vision reconstruction performance to new sessions and stimulus types, or to mental imagery.

Scaling Behavior We assess the scaling behavior of MIRAGE and other decoding pipelines with respect to the number of trial repetitions folded into the brain activity averages (Appendix A.10 in S1 Text), and the number of training data hours (Appendix A.11 in S1 Text). In both cases, the scaling behavior of MIRAGE outperforms all other decoding pipelines.

Contribution of the Natural Image Prior To assess the relative contributions of the natural image prior on reconstruction performance, we implemented two systematic controls. First, in Experiment 1, distractor images were generated by decoding randomly selected brain activity patterns. Thus, both target and distractor outputs were equally shaped by the diffusion prior. The high identification accuracy observed (p < 0.001; see Table 2) therefore reflects genuine signal decoding rather than the prior’s semantic bias. Second, we assessed reconstruction performance across varying strengths of diffusion guidance. We found that reconstructions with minimal guidance (with nearly pure VDVAE decoding) remained highly identifiable, while increasing guidance strength primarily enhanced image quality rather than semantic content (Fig P in S1 Text), mitigating recent concerns about the authenticity of fMRI-to-image reconstructions [51,52].

3.1. Neuroscientific interpretations

Our ablation studies (Section 2.3) identify two critical factors for generalization: reduced feature dimensionality and the inclusion of text-based guidance. These empirical results align with known properties of the visual cortex. First, the success of lower-dimensional image embeddings likely reflects the reduced signal-to-noise ratio (SNR) and coarser spatial resolution of mental imagery compared to vision [12,13]. High-dimensional embeddings, while expressive for high-SNR vision trials, are empirically prone to overfitting and harming generalization to the noisier imagery signal.

Second, the efficacy of text guidance supports our hypothesis that mental imagery relies heavily on semantic representations. Previous work has demonstrated that natural language supervision improves encoding models of higher-level visual cortex [3], suggesting that these regions encode visual information in a format that is semantically aligned with language. Higher-level visual cortex is also the area of the brain that most heavily overlaps with activations created by mental imagery [13], and so by incorporating text embeddings, we speculate that MIRAGE taps into this semantic overlap, allowing the model to stabilize reconstructions even when fine-grained visual details in the brain signal are degraded.

3.2. Societal Impact

Our work is a necessary step towards applications of mental image reconstruction, including diagnostic instruments for psychiatric conditions [53] and disorders of consciousness [54–56], as well as expressive alternative communication methods for patients with traumatic brain injuries [57], amyotrophic lateral sclerosis (ALS) [58], and locked-in syndrome [59]. While a significant fraction of this communicative value could be provided by a “concept decoder” whose output is linguistic or otherwise compressed, however, visual representations of internal states could complement linguistic representations, for example by revealing how a specific concept or text prompt is visually interpreted. As the saying goes, ’a picture is worth a thousand words’. While our work makes no claims about being able to decode what is unique and idiosyncratic about a specific individual’s mental imagery, it does represent a first step towards customized bespoke representations of internal states, and is a very literal attempt at this goal motivated by the overlap between representations of vision and mental imagery, and measured against a known ground truth visual stimulus image.

Of course, the development of this technology raises concerns about the potential for misuse [60]. We propose that when deployed in a clinical setting, brain decoding should be defined as a medical procedure that yields private health information and should therefore be subject to all relevant laws pertaining to patient consent, risk / benefit assessment, and the protection of privacy. In all other settings, it seems obvious that laws governing brain decoding should require informed consent and, where necessary, parental guidance.

3.3. Current limitations

The MIRAGE method does exhibit some notable weaknesses and biases when applied to NSD-Imagery. The median and worst case reconstructions from MIRAGE and other methods can be seen in Appendix A.4 in S1 Text, and in particular we note how the method is heavily influenced by the prior of the NSD training dataset. Objects that are more well represented in NSD, such as surfers, tend to be reconstructed much more successfully than objects more sparsely observed, such as birds. This is seen even more clearly on the simple stimuli, which are not represented at all in NSD (which contains only natural scenes), and so MIRAGE and other methods tend to produce reconstructions with consistent low-level structural features, but seemingly random semantic content reconstructed in the details of the image.

The NSD-Imagery dataset utilized as a validation set for this work also presents a limitation, as it contains only 18 stimuli, and thus does not allow for large-scale model training or fine-tuning of existing vision decoding models, necessitating a cross-decoding approach. Future models for downstream applications would surely be improved by training or fine-tuning on mental imagery datasets. Such datasets must be a priority for research in this space.

In addition to dataset availability, practical clinical implementation faces the challenge of data acquisition time per subject. Collecting the volume of fMRI data typically required to train robust decoding models is often infeasible for patient populations, however, recent advancements in the “novel subject” problem, such as MindEye2 [2], have demonstrated the potential to fine-tune models on as little as one hour of data by leveraging pre-training on large public datasets. While current data-efficient methods struggle to generalize to mental imagery, we anticipate that future research will bridge this gap, and our results suggest a linear backbone with multimodal guidance to be a promising direction for future subject-adaptive mental imagery decoding.

Currently, the computational requirements to run these models are also an obstacle to realizing medical applications. We trained our ridge regression modules on computing hardware with 512GB of RAM and performed inference for our models using an NVIDIA A100 with 40GB of VRAM. The requirement of such hardware limits the use of our method to researchers with considerable compute resources.

All of these limitations we believe to be rich ground for future work, and we look forward to follow-up research addressing them in more detail.

4.1. NSD-imagery dataset

To evaluate model performance on internally generated visual representations, we utilize the NSD-Imagery dataset, an extension of the Natural Scenes Dataset (NSD) [25] where the same eight subjects completed a session of additional mental imagery tasks. For a full description of the dataset, please see [1]. This dataset consists of high-resolution 7T fMRI responses collected using identical acquisition protocols to NSD.

4.1.1. Stimuli.

Prior to scanning, participants in the NSD-Imagery experiment memorized a unique single-letter cue associated with each of the 18 stimuli (Fig 5), which span three distinct categories to assess reconstruction across varying levels of visual and semantic complexity:

1. (A) Simple Stimuli: Six geometric shapes constructed from black bars on a gray background, including four oriented bars (0°, 45°, 90°, 135°) and two crosses (“+” and “ד).
2. (B) Complex Stimuli: Five natural scenes selected from the NSD shared1000 set and one artwork (“The Two Sisters” by Kehinde Wiley), chosen based on recognizability scores from the original NSD sessions.
3. (C) Conceptual Stimuli: Six abstract single-word concepts (e.g., “stripes”, “mammal”, “banana”) rather than specific fixed images. As the vision trials for these concepts involved viewing multiple variant images, we exclude the vision trials for this condition and evaluate only on the imagery trials.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Overview of the tasks utilized for the NSD-Imagery benchmark.

https://doi.org/10.1371/journal.pcbi.1014263.g005

4.2. MIRAGE

4.2.2. Methodology.

We propose a model, (MIRAGE), that respects two important desiderata for decoders that generalize well from vision to mental imagery:

Reduced model complexity to accommodate the relatively low SNR of mental imagery activity. Complex, expressive models trained in a high SNR regime may exhibit unacceptably high levels of error variance when tested in a lower SNR setting [62]. Brain activity is known to have much lower SNR during mental imagery than it does during vision, and recent work [1] has found that relatively complex SOTA vision decoding pipelines generalize poorly when applied to mental images. Our decoding pipeline therefore prioritizes robustness over expressivity. In particular, we implemented a linear ridge regression backbone (Fig 6)—in contrast to the non-linear MLP backbone use more commonly in vision decoding. Ridge regression is effective in high-dimensional, low-SNR ‌‌settings [66], and is singularly effective for aligning noisy brain activity patterns across individual brains [33].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. MIRAGE training pipeline.

(1) Brain activity (7T fMRI) acquired as NSD subjects view > 10K stimuli. (2) Stimuli are passed to VDVAE encoder [37] yielding (1 × 91168) latents (3) LLaVA v1.5-13B [63,64] generates synthetic captions. (4) Captions are encoded into CLIP ViT-bigG/14 text embeddings (77 × 1280) [65]. (5) Stimuli are also passed through the CLIP ViT-L/14 image encoder [23] to generate both CLS token (1 × 768) and hidden layer (257 × 1024) image embeddings. (6) Parallel ridge regression modules are trained from the measured fMRI brain activity to the various feature spaces.

https://doi.org/10.1371/journal.pcbi.1014263.g006

Representational alignment to mental images. Brain activity patterns that represent seen and mental images, respectively, overlap in early visual cortex [7] where structural details of stimuli are encoded, but are most closely aligned in higher-level visual brain areas [13] that are known to represent semantic and/or linguistic aspects of stimuli [67]. Accordingly, our method drives an image generator with the CLS token of a CLIP ViT-L/14 image embedding, and incorporates multi-modal guidance from decoded CLIP ViT-bigG/14 text features (Fig 7).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. MIRAGE inference pipeline.

(1) The NSD subjects imagine stimuli from letter cues under 7T fMRI. (2) A set of feature embeddings is predicted by passing the measured fMRI brain activity through our frozen ridge regression models. (3) The VDVAE [37] latents are reconstructed into a low-level image. (4) The image is filtered to boost its structure. (5) The filtered low-level image, decoded image embedding, and decoded text embedding are used as input to Stable Cascade [68] to generate a retrieval pool of 16 candidate reconstructions. (6) Each of the candidate reconstructions is encoded into a CLIP ViT-L/14 hidden layer image embedding [23]. (7) The final reconstruction is automatically selected as the image with the highest cosine similarity to the decoded CLIP ViT-L/14 hidden layer image embedding. (8) The same hidden layer image embedding is passed through the GiT [69] captioning model. (9) A decoded caption is generated along with our final reconstruction.

https://doi.org/10.1371/journal.pcbi.1014263.g007

4.2.3. Ridge regression backbone.

To map preprocessed fMRI data to feature representations in our decoding pipeline, we employ parallel L₂ regularized ridge regression models trained on each individual feature set. In keeping with our first constraint, ridge regression is chosen because it is known to be effective in low signal-to-noise regimes, in contrast to MLPs that offer advantages for capturing nonlinear relationships, but can also be fragile in brain decoding contexts when the input data are of low SNR. For each set of features, we train a parallel ridge regression model to predict the feature value from our fMRI responses, minimizing the loss function in:

(1)

where is the fMRI data matrix of n fMRI trials by v voxels, is the matrix of target features d for each trial n, is the weight vector mapping fMRI voxels to the feature dimension d, is the bias vector added to each trial n, and is the ridge parameter controlling the strength of the L₂ penalty. To select an optimal value for , we treat the four NSD subjects who are not selected for reconstruction as a hyperparameter tuning set, and perform a grid search over possible values of on this set to pick the optimal value for generalizing to fMRI responses of mental images. Using this process, we select for all modules of the regression backbone. Details on our hyperparameter search are in Appendix A.1 in S1 Text.