1.1 Contributions

The primary contributions of this research are:

• Developed a comprehensive ML framework comprising multiple 2D mammographic views, available patient-level clinical information, and retrieval-augmented generation (RAG) for capturing clinical reasoning patterns, providing interpretable evidence, and improving diagnostic accuracy.
• Constructed a specialized Vision Transformer (ViT) encoder to capture both global and fine-grained features from individual mammographic images using attention-based representation learning.
• Developed a fold-wise retrieval memory-bank module for storing training-derived mammographic embeddings and retrieving diagnostically similar historical cases during inference.
• Implemented a retrieval-augmented generation (RAG) module to facilitate retrieval of prototypes that are diagnostically similar from a historical case repository via multi-headed attention and used to augment features associated with current images to provide stronger predictive decisions.
• Performed a comparative assessment of the public mammograms which provided consistent improvements of the proposed retrieval-augmented 2D mammography model relative to CNN-based and transformer-based baseline approaches, particularly in difficult-to-diagnose cases.
• Aligned the proposed retrieval-augmented transformer architecture with real world mammography processes thereby advancing automated classification of breast cancer, segmentation of lesions and evidence-based diagnostic interpretation.

The novel aspects of ViT-MultiRAGNet are,

• Retrieval-Augmented 2D Mammography Framework with RAG: A deep learning architecture that combines multi-view 2D mammographic images, available patient-level clinical information, and historical case retrieval into a single end-to-end trainable model.
• Vision Transformer-Based Global Feature Extraction: A specialized Vision Transformer encoder is implemented to capture global and fine-grained mammographic features using self-attention-based representation learning.
• Retrieval-Augmented Generation Mechanism: A memory-guided RAG module is developed to retrieve diagnostically similar training cases and enrich the current mammographic representation during inference.
• Fold-Wise Memory Bank Construction: A fold-specific memory bank is constructed using only training-derived mammographic embeddings, ensuring that validation and test samples are excluded from retrieval and preventing data leakage.
• Multi-Head Cross-Attention Fusion: A multi-head cross-attention fusion module dynamically integrates current mammographic features with retrieved contextual embeddings from the memory bank.
• Evidence-Based Interpretable Diagnosis: The RAG module improves diagnostic transparency by identifying reference cases that influence predictive decisions and by supporting evidence-based interpretation.

2.1 Summary table: Field status overview

The past decade has seen marked improvement within the field of medical imaging Artificial Intelligence (AI) related to accuracy, generalizability, and architectural sophistication. The results of this progress can be shown through representative studies done using mammograms where representative studies done using mammograms are summarized in Table 2 by the type of model used, performance metrics, and an overview of key strengths and limitations for each approach.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Technological progression in AI-based mammography: Timeline and performance evolution.

https://doi.org/10.1371/journal.pone.0349865.t002

The advancement of AI-driven mammography has occurred at a steady pace over time. The first documentation demonstrating that AI-driven mammography was feasible in the application of the transformer architecture was recorded during the period of 2020–2021. Using multi-view and transfer learning methodologies, mammography performance from 2022–2023 improved; however, resolution issues and excessive overfitting still existed. At present, as of 2024–2025, hybrid transformer/retrieval-supported (ViT-MultiRAGNet) architectures are achieving improved performance for mammograms. ViT-MultiRAGNet progresses this advancement by adding memory guidance through the utilization of retrieval-augmented memory, thereby enhancing both interpretability, contextual comparison to results, and facilitating evidence-based reasoning for analysis of 2D mammograms.

2.2 Research gap

Despite much research into transformer models and their applications in mammography from 2020–2025, there remain many significant gaps. The first of which is the vast majority of existing research focuses on limited architectural examples (ViT, Swin Transformer) without systematically investigating the use of retrieval-augmented paradigms. Another gap is the majority of the research has focused on improving dataset-specific accuracy, limiting the ability for many transformer models to generalize/investigate a broader range of application and real-life clinical settings, and have not provided any insights into the robustness of transformer models for clinical situations outside of traditional datasets (e.g., rare lesions, higher density tissue types, institutional differences). Many of the current transformer models are built as closed architecture which means they do not have access to external knowledge or reference historical cases during the inference phase, so they cannot use the types of comparative reasoning used by a trained clinician when making a diagnosis. Additionally, there is a large gap in having any systematic validation of retrieval-augmented transformers applied to real-life clinical situations. Addressing this issue is an important step toward developing a reliable transformer-based system that meets essential criteria of transparent, interpretable, actionable, evidence-based reasoning, and clinician trust for everyday mammographic use.

2.3 Motivation

Detecting breast cancer is an ongoing problem that requires a lot of experience and skill. There are many difficulties in diagnosing breast cancer due to different image resolutions, the large variety of tissue types seen, and the subtle change in size or shape of the lesion being looked at. Although AI developments have been made when it comes to algorithms, most still tend to be built for only one type of data type per algorithm, don’t provide an easily usable way for interpreting the results or comparing the results with past cases (thereby providing less reliability regarding a diagnosis and providing lower confidence levels for a radiologist making a diagnosis). This study focuses on developing an AI-assisted mammography framework that provides richer diagnostic information and helps radiologists understand how a prediction is supported by similar historical cases. By combining multi-view 2D mammographic feature extraction with retrieval-augmented memory guidance, the proposed framework aims to improve diagnostic reliability, interpretability, and confidence in automated breast cancer assessment.

2.4 Proposed work

ViT-MultiRAGNet is a clinically relevant retrieval-augmented framework for breast cancer classification and lesion segmentation using 2D mammography. Unlike conventional models that process mammograms using only fixed learned parameters, ViT-MultiRAGNet incorporates a memory-guided retrieval mechanism that retrieves historically similar training cases during inference. The framework processes 2D mammogram images using Vision Transformer based encoders create a fold wise memory bank using the embeddings generated by the training step, determine the top-k similar cases using cosine similarity, and then use multi-head cross-attention to merge the retrieved/contextual representation with the current image features (image). ViT-MultiRAGNet was evaluated for its diagnostic ability, interpretability and operational feasibility in performing evidence-based analysis of mammography (CBIS-DDSM) and RTM datasets.

The research objectives include:

• Integrated Diagnostic Framework Development: To develop a unified 2D mammography-based diagnostic framework that combines ViT-based image representation, fold-wise memory-bank retrieval, and retrieval-guided feature fusion for breast lesion classification and segmentation.
• Context-Aware AI Implementation via Retrieval: To implement ViT-MultiRAGNet for visual pattern recognition and retrieval of historically similar mammographic cases using Vision Transformer and RAG mechanisms.
• Diagnostic Performance Enhancement: To improve classification and segmentation performance for breast lesions in challenging cases through ViT-based global feature extraction and retrieval-augmented contextual fusion.
• Decision Transparency and Clinical Utility: To support transparent and interpretable diagnostic decision-making by using retrieved reference cases as evidence for model predictions.
• Clinician-Centered AI Development: To develop an evidence-supported AI framework that assists radiologists by improving diagnostic confidence, interpretability, and consistency in breast cancer assessment.

Overall, these objectives address the limitations of existing mammography AI systems by developing a framework that is accurate, transparent, context-aware, evidence-supported, and suitable for further clinical validation.

3.1 Data collection

The experiments used two datasets for mammography images collected at 2D. There are two mammography datasets included in the study for this research project; they are the Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM) dataset from the United States, which contains 2-dimensional human mammograms, and the RTM mammography datasets from Mexico, which contain 2-dimensional human mammograms. Both datasets are made of human-approved 2-D mammograms, which were de-identified prior to their public release or use in this study [19]. Each of the datasets provides a some-data, clinical, and imaging metadata associated with each image, such as unique identifiers of the patients in the collection (i.e., Patient Identifier, Diagnostic Label,etc.), laterality (i.e., left breast, right breast), View Type (i.e., 4-view, true-view, etc.), and lesion annotation (if available). Patient Identifier was used for patient-level partitioning during cross-validation (i.e., images from the same patient were not stored in the same training/validation/test sets). For each image in each dataset, the Diagnostic Labels were standardized, the Common Variables were mapped to a standard schema, and preprocessing techniques such as one-hundred percent resolution normalization and one-hundred percent intensity normalization were performed to minimize variability between datasets. The example of the CBIS-DDSM data set consists of high quality digitised and annotated 2D screening mammograms that have both types of pathology (i.e., CAD/CAM) confirmed for cases involving masses or microcalcifications. Experts have also provided annotating of lesions in the form of both classification and segmentation labelling (where possible). Standard screening formats of mammograms were used, e.g., craniocaudal (CC) and mediolateral oblique (MLO), and all images collected from screening were used for pre-processing the data, training the models, and building memory banks on a fold-wise basis at both an image and patient level.

RTM (Real Time Mammogram) Dataset:

• The RTM dataset consists of 2D mammographic images collected from multiple clinical sources.
• The mammograms are categorized into diagnostic groups such as normal, benign, and malignant according to available clinical annotations.
• Available metadata and radiological notes were used only for dataset organization, label verification, and contextual interpretation.
• For model evaluation, the dataset was split using patient-level five-fold cross-validation. Within each fold, only training samples were used to construct the RAG memory bank, while validation and test samples were excluded from retrieval storage.

3.2 Data preprocessing

Mammographic images are challenging to process because of background artifacts, low contrast, scanner-dependent intensity variation, and differences in breast-tissue density. Therefore, the following preprocessing steps were applied before model training and evaluation:

1. Standardization: All images are resized to be 512 x 512 pixels with aspect ratio preserved through appropriate padding techniques.
2. Removal of Noise: Background noise is removed using Wiener filtering with a 3 x 3 window size while still preserving features of edges.
3. Contrast Enhancement: Contrast-Limited Adaptive Histogram Equalization (CLAHE) was applied using a clip limit of 2.0 and an 8 × 8 grid size to improve local contrast visibility.
4. Breast Region Extraction: Otsu’s thresholding was used to separate the breast region from the background, followed by morphological filtering to remove non-tissue artifacts.
5. Pixel Intensity Standardization: Pixel intensity values were normalized to the range [0, 1] to reduce scanner-dependent intensity variation.

All mammogram images were first normalized using min–max scaling to map pixel intensities to the range [0,1]. Subsequently, z-score normalization was applied during model input preparation to standardize pixel distributions across the dataset. These preprocessing steps help reduce inter-image intensity variations and improve training stability.

3.3 Data normalization

Histogram normalization was applied to reduce intensity variation across images acquired from different sources. A reference template was derived from representative high-quality mammograms, and the intensity distribution of each image was adjusted toward this reference. This step helped reduce inter-dataset variability and improved the stability of feature extraction. Fig 2 summarizes the normalization and preprocessing pipeline.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Data normalization and preprocessing pipeline.

https://doi.org/10.1371/journal.pone.0349865.g002

3.4 Data augmentation

Data augmentation was applied to improve model generalization and reduce overfitting under limited medical imaging data. Table 3 summarizes the augmentation strategies and their parameter settings (Fig 3).

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Data augmentation strategies and parameterization.

https://doi.org/10.1371/journal.pone.0349865.t003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Data augmentation process flow, demonstrating geometric, intensity, and lesion-aware transformations applied during model training.

https://doi.org/10.1371/journal.pone.0349865.g003

Data augmentation was applied only to the training subset within each fold of the cross-validation procedure. Augmentation operations included horizontal flipping, rotation, scaling, and intensity perturbation. These transformations were implemented using on-the-fly augmentation during training, meaning that augmented images were generated dynamically at runtime rather than stored as additional samples. Consequently, the number of original images in the dataset remained unchanged, while the model observed multiple augmented variations of each training image during different training iterations. Validation and test samples were not augmented to ensure fair performance evaluation.

3.5 Feature extraction

Feature extraction was performed using a Vision Transformer backbone. The proposed model uses the ViT-B/16 architecture to encode 2D mammographic images into global feature representations. Learnable positional embeddings were added to patch embeddings to preserve spatial relationships between image regions before transformer encoding. To capture features at different semantic levels, representations from the 4th, 8th, and 12th transformer encoder layers were selected and aggregated using channel-wise concatenation followed by linear projection. The resulting unified feature representation was used for memory-bank retrieval, cross-attention fusion, classification, and segmentation.

1. Patch Embedding: Each input image is separated into non-overlapping patches of (16 x 16). An image of (512 x 512) therefore separates into (32 x 32 = 1024) different patches.
2. Global Feature Learning: The ViT backbone uses pure self-attention and does not have a hierarchical windowing mechanism used for feature learning; this provides the attention model with true global context to enable global context processing.
3. Multi-Scale Extraction: Intermediate transformer features from selected encoder layers were used to capture low-level, mid-level, and high-level semantic information.
4. Position Encoding: Learnable positional embeddings were added to patch embeddings to preserve spatial information across mammographic image regions.
5. Classification and Segmentation Feature Extraction: The encoded feature representation was used for both classification and segmentation, with the global representation supporting class prediction and spatial feature representations supporting lesion-mask generation.

Pretraining weights were obtained from publicly available ViT checkpoints provided by the original Vision Transformer implementation. Table 4 and Fig 4 provide the feature extraction process in detail. Table 5 provides the training configuration of the proposed model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Feature extraction strategy and transformer mechanisms.

https://doi.org/10.1371/journal.pone.0349865.t004

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Training configuration of the proposed ViT-MultiRAGNet framework.

https://doi.org/10.1371/journal.pone.0349865.t005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Feature extraction process via Vision Transformer, showing patch embedding, self-attention mechanisms, and multi-scale hierarchical feature learning.

https://doi.org/10.1371/journal.pone.0349865.g004

The feature extraction stage uses a ViT-B/16 backbone initialized with ImageNet-21k pretrained weights. The model was fine-tuned end-to-end on the mammography datasets within each fold of the five-fold patient-level cross-validation setting. Training was performed using the AdamW optimizer with an initial learning rate of , a weight decay of 0.01, and cosine annealing learning-rate scheduling. The classification branch was trained with a batch size of 16 for 100 epochs, while the segmentation branch was trained with a batch size of 8 for 200 epochs using a hybrid Dice and binary cross-entropy loss.

3.6 Segmentation

The segmentation framework uses the ViT encoder and retrieval-augmented fused representation to delineate lesion regions in 2D mammograms. After the encoder extracts the global feature set using self-attention, the decoder converts the features extracted by the encoder into segmentation maps / lesion segmentations while maintaining the important spatial characteristics of those features. This design enables the decoder to progressively integrate global contextual information from the transformer encoder with local spatial details required for accurate lesion boundary reconstruction. The decoder module reconstructs the spatial resolution of the feature representations obtained from the transformer encoder. The decoder consists of a sequence of hierarchical upsampling stages, where each stage performs bilinear upsampling followed by convolutional refinement to recover spatial details. Skip connections are introduced between intermediate encoder features and the corresponding decoder layers to preserve fine-grained spatial information. Each decoder block includes a convolution layer followed by batch normalization and ReLU activation to refine the upsampled feature maps. Finally, a 1 × 1 convolution layer with sigmoid activation generates the final segmentation probability map.

3.7 Retrieval-Augmented Generation (RAG) module

The RAG module is the central retrieval component of the proposed ViT-MultiRAGNet framework. For each cross-validation fold, a memory bank is constructed using only the encoded feature representations of training samples. Each memory entry contains a key embedding, a value embedding, and the corresponding diagnostic label. During validation or testing, the encoded representation of the current 2D mammogram is used as a query to retrieve the top- most similar training cases from the fold-specific memory bank. The retrieved contextual representation is then fused with the current image representation using multi-head cross-attention.

Let denote the input 2D mammogram and denote the ViT encoder. The query embedding is obtained as

For each fold, the memory bank is represented as , where , , and denote the key embedding, value embedding, and diagnostic label of the -th training sample. The cosine similarity between the query and each memory key is computed as

The top- most similar entries are selected, and softmax normalization is applied only over these selected scores:

The retrieved context is then calculated as

This retrieved context is fused with the current mammographic feature representation using multi-head cross-attention before classification and segmentation.

Here, denotes the indices of the top- retrieved memory entries. By applying softmax only within this selected set, the model assigns higher weights to the most relevant historical cases. The resulting retrieved context provides evidence-guided support for the final classification and segmentation predictions. Given a query embedding , cosine similarity is computed between the query and all key embeddings in the fold-specific memory bank. Only the top- most similar training cases are selected for retrieval. Softmax normalization is applied only within this selected top- set, ensuring that the retrieved context is computed from the most relevant historical cases. The resulting contextual representation is fused with the current mammographic feature representation using multi-head cross-attention before final classification and segmentation. Table 6 provides the components of RAG module.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. RAG module components and mechanisms.

https://doi.org/10.1371/journal.pone.0349865.t006

Retrieved context subsequently undergoes fusion with current query features via multi-head cross-attention, producing enriched representations for final classification and segmentation.

3.8 Data leakage

The RAG memory bank is constructed exclusively from embeddings of the training dataset. Validation and test samples are strictly excluded from the knowledge base, ensuring that retrieval during inference only accesses training-derived representations and prevents any form of data leakage.

To prevent retrieval bias, the memory bank is constructed exclusively from training samples within each cross-validation fold. Since dataset splitting is performed at the patient level, no images from the same patient appear in both the training and evaluation subsets. Therefore, the Top-K retrieval mechanism cannot recover samples belonging to the same patient as the query image during validation or testing. This design ensures that retrieval provides contextual information from independent training cases while avoiding patient-level information leakage.

3.9 Data splitting and cross-validation

All experiments were performed using 5-fold cross-validation with patient-level splitting. All images belonging to the same patient were assigned to a single fold only, ensuring that no patient contributed images to more than one of the training, validation, or test subsets within a fold. This prevented subject-level leakage and ensured unbiased evaluation on unseen patients.

Since mammography datasets contain multiple views per patient (e.g., CC and MLO views of both breasts), dataset splitting was performed at the patient level rather than at the image level. All images belonging to the same patient were grouped using the patient identifier and assigned to the same fold during cross-validation. This ensured that no views from a patient appeared simultaneously in training and evaluation subsets. Stratification was performed based on patient-level diagnostic labels to maintain class balance across folds.

3.10 Ethics statement

All mammographic data used in this study were anonymized before analysis. The publicly available CBIS-DDSM dataset was used according to its research-use guidelines. The RTM dataset was handled in accordance with institutional ethical and privacy requirements, and no personally identifiable patient information was used during model training, validation, or testing. Patient-level partitioning was applied during cross-validation to prevent subject-level data leakage.

4.1 ViT-MultiRAGNet architecture

The proposed ViT-MultiRAGNet architecture consists of five main components: 2D mammogram preprocessing, ViT-based feature extraction, fold-wise memory-bank retrieval, cross-attention-based feature fusion, and task-specific prediction heads. First, each mammogram is resized, normalized, and divided into non-overlapping patches. These patches are encoded using the ViT-B/16 backbone to obtain mammographic feature representations. During training, embeddings from the training subset of each fold are stored in a memory bank. During validation and testing, the encoded query image retrieves the top- most similar training cases from this memory bank using cosine similarity. The retrieved context is fused with the current query feature through multi-head cross-attention and is then used for benign/malignant classification and lesion segmentation.

Fig 5 illustrates the proposed ViT-MultiRAGNet architecture for 2D mammogram classification and lesion segmentation. The input 2D mammogram is first preprocessed and divided into non-overlapping image patches, which are encoded using the ViT-B/16 backbone. The encoded query representation is compared with the fold-wise training memory bank to retrieve the top- most similar training cases using cosine similarity. The retrieved contextual embeddings are fused with the current mammographic feature representation through multi-head cross-attention. The fused representation is then passed to the classification head for benign/malignant prediction and to the segmentation decoder for lesion-mask generation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Proposed ViT-MultiRAGNet architecture for 2D mammogram classification and segmentation.

https://doi.org/10.1371/journal.pone.0349865.g005

4.2 Architecture components

This fusion mechanism permits selective attention to most salient retrieved cases and clinical context while processing current visual information.

1. 1. Vision Transformer Encoder: The ViT encoder processes each 2D mammographic image through patch partitioning, linear projection, positional embedding, and stacked transformer encoder blocks. Let denote the initial patch-token sequence after linear projection and positional embedding. For the -th transformer layer, the encoded representation is computed as

where denotes multi-head self-attention, denotes layer normalization, and denotes the feed-forward multilayer perceptron block. The final ViT representation is used as the query embedding for memory-bank retrieval.

1. 2. Memory-Bank Retrieval Module: The encoded mammographic representation is used as the query vector . Cosine similarity is computed between and all key embeddings stored in the fold-specific memory bank. The top- most similar training cases are retrieved and used to construct the contextual representation .
2. 3. Optional Clinical Feature Embedding: When patient-level clinical metadata are available, they are encoded using a lightweight fully connected embedding layer and may be fused with the mammographic representation. If such metadata are unavailable for a dataset, the model operates using only 2D mammographic image features and retrieved memory-bank context.
3. 4. RAG Module: For a query mammogram , the ViT encoder generates the query embedding . The RAG module compares with the memory-bank keys , selects the top- most similar entries, and computes the retrieved contextual representation using softmax-normalized similarity weights over the selected entries only.
4. 5. Multi-Head Cross-Attention Fusion: The retrieved contextual representation is fused with the current mammographic feature representation using multi-head cross-attention:

where represents the current image query feature, represents the retrieved contextual representation, and denotes multi-head attention. The fused feature is used for final classification and segmentation.

1. 6. Classification and Segmentation Heads: The fused representation is passed to a classification head for benign/malignant prediction and to a segmentation decoder for lesion-mask generation. The classification head uses fully connected layers, while the segmentation decoder uses upsampling and convolutional refinement blocks followed by a convolution with sigmoid activation.

4.3 Hyperparameter tuning

The optimal network configuration was determined through systematic hyperparameter search and validation. Hyperparameter tuning was conducted independently for the CBIS-DDSM and RTM datasets within the 5-fold cross-validation framework. For each fold, the training data were further split into training and validation subsets. The validation subset was used to evaluate candidate hyperparameter configurations, while the test subset remained unseen during model selection. The explored ranges and selected optimal hyperparameters for the segmentation and classification branches are summarized in Tables 7 and 8. Model selection was primarily based on Dice score for segmentation and accuracy/AUC-ROC for classification. When multiple configurations produced similar primary metric values, secondary metrics such as HD95 or AUC were used to resolve the trade-off and select the final model configuration.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Hyperparameter optimization results for the segmentation network using validation-based selection within cross-validation.

https://doi.org/10.1371/journal.pone.0349865.t007

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Hyperparameter optimization results for the classification network using validation-based model selection.

https://doi.org/10.1371/journal.pone.0349865.t008

The Vision Transformer encoder used in the proposed architecture is initialized with ImageNet pre-trained weights. During training, the backbone is fine-tuned jointly with the retrieval-augmented module using the CBIS-DDSM and RTM datasets. This transfer-learning strategy helps stabilize training and improves convergence when learning from relatively limited medical imaging data.

4.4 Workflow diagram

The workflow of the proposed ViT-MultiRAGNet model is shown in Fig 6. The process begins with 2D mammogram preprocessing, followed by patch embedding and ViT-based feature extraction. During training, encoded features from the training subset are stored in a fold-wise memory bank. During validation and testing, the query embedding of the current 2D mammogram is compared with memory-bank keys, and the top- most similar training cases are retrieved. The retrieved context is fused with the current image feature using multi-head cross-attention. The fused representation is then used for benign/malignant classification and lesion-mask segmentation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Workflow of the proposed ViT-MultiRAGNet model showing 2D mammogram preprocessing, ViT feature extraction, fold-wise memory-bank construction, top- retrieval, cross-attention fusion, classification, and segmentation.

https://doi.org/10.1371/journal.pone.0349865.g006

4.5 Algorithm: VIT-MultiRAGNet Framework

Input: 2D mammogram image , training set , retrieval size

Output: Classification label , segmentation mask

1. Preprocess each 2D mammogram using resizing, normalization, noise reduction, and contrast enhancement.

2. Split the dataset using patient-level five-fold cross-validation.

3. For each fold, construct training, validation, and test subsets without patient overlap.

4.  Encode each training mammogram using the ViT encoder .

5.  Store the resulting training embeddings in the fold-specific memory bank .

6.  For a query mammogram , compute the query embedding .

7.  Compute cosine similarity between and all memory-bank key embeddings.

8.  Select the top- most similar training embeddings.

9.  Apply SoftMax normalization only over the selected top- similarity scores.

10. Compute the retrieved contextual representation as the weighted sum of the selected value embeddings.

11. Fuse the current query feature and retrieved context using multi-head cross-attention.

12. Pass the fused representation to the classification head to predict the benign/malignant label .

13. Pass the fused representation to the segmentation decoder to generate the lesion mask .

14. Evaluate classification and segmentation performance on validation/test samples only.

To ensure reproducibility, fixed random seeds were used for dataset partitioning, model initialization, and training. The memory bank was constructed only from the training subset of each fold. Validation and test samples were excluded from the memory bank to prevent retrieval-based data leakage.

4.6 Training and evaluation

The proposed model training implements two-stage sequential approach:

Segmentation Training Protocol:

• Loss Function: Combined Dice loss and Binary Cross-Entropy (BCE) with boundary-aware weighting
• Optimizer: AdamW with weight decay coefficient 0.01
• Learning Rate: Initial 1e-4 with cosine annealing schedule
• Batch Size: 8 samples per gradient update
• Epochs: 200 with early stopping using a patience of 20 consecutive non-improving epochs
• Regularization: Dropout , weight decay, and gradient clipping with maximum norm of 1.0

Classification Training Protocol:

• Loss Function: Weighted Cross-Entropy with consistency regularization
• Learning Rate: Initial 1e-4 with linear warm-up and cosine decay schedule
• Batch Size: 16 samples per update
• Epochs: 100 with early stopping (patience = 15)
• Data Sampling: Stratified balanced sampling addressing class imbalance

Both training stages employ 5-fold cross-validation, ensuring robust evaluation and reduced metric variance.

4.7 Performance metrics

Comprehensive evaluation encompasses:

Segmentation Evaluation Metrics:

• Dice Similarity Coefficient (DSC): Spatial overlap between predicted and ground truth segmentations
• Intersection over Union (IoU): Jaccard index quantifying regional overlap
• Hausdorff Distance (HD95): 95th percentile boundary accuracy assessment
• Sensitivity and Specificity: Pixel-level true positive and true negative rates

Classification Evaluation Metrics:

• Area Under ROC Curve (AUC): Primary classification performance indicator
• Accuracy: Overall classification correctness rate
• Sensitivity/Recall: True positive rate for malignant lesions
• Specificity: True negative rate for benign lesions
• F1-Score: Harmonic precision-recall mean
• Precision: Positive predictive value

Computational Efficiency Metrics:

• Inference Latency: Mean processing time per image
• Model Size: Parameter count and memory footprint
• FLOPs: Floating-point operations per inference

4.8 Processor/Hardware configurations

The experiments were conducted using a high-performance workstation equipped with an NVIDIA A100 40 GB VRAM GPU, an AMD Ryzen Threadripper PRO 32-core CPU, and 128 GB RAM. The use of PyTorch 2.0 allowed all development, training, and inference activities to be performed using flexible CPU/GPU resource management. Given the high-performance attributes of this hardware configuration, the infrastructure was designed to support the processing of large amounts of mammography data efficiently, constructing a large memory bank efficiently, and executing efficient computational retrieval tasks during inference.

5.1 Computational performance of the proposed model

ViT-MultiRAGNet was evaluated for computational feasibility under the proposed 2D mammography classification and segmentation setting. On an NVIDIA A100 40 GB GPU, the average training time was 11.4 minutes per epoch for the segmentation branch and 7.2 minutes per epoch for the classification branch. The total training duration was approximately 14.6 hours across the reported experimental setting. Peak memory consumption was 35.2 GB during model training and memory-bank construction, while the average training throughput was 23.1 samples per second. Table 9 summarizes the computational resource utilization and performance metrics.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Computational resource utilization and performance metrics.

https://doi.org/10.1371/journal.pone.0349865.t009

An epoch is one complete cycle through all the training data in each cross-validation fold. The training data were used to perform experiments using 5-experiment folded cross-validation. Each reported training time is an average time required to train the model in a single fold and includes times for the forward and backward training processes, but not times associated with preprocessing, such as splitting the dataset and constructing the memory bank. A “sample” in this context represents one 2D mammogram image that is input into the neural network; therefore, it does not represent an instance of a patient. All timing measurements were conducted using the same hardware configuration to provide for common performance comparisons.

ViT-MultiRAGNet generated predictions within 0.28 seconds per image while requiring less than 7 GB of inference memory. The model contained 89.1 million parameters with a disk footprint of 342 MB, and the RAG memory bank required approximately 2.3 GB of additional storage. These results indicate that the proposed framework is computationally feasible for further validation in clinical decision-support workflows.

5.2 Quantitative evaluation of segmentation on both datasets

The segmentation performance of the proposed model was evaluated on both datasets, and the results are presented in Tables 10 and 11.

Download:

• PNG
  larger image
• TIFF
  original image

Table 10. Segmentation performance: CBIS-DDSM dataset (Mean ± SD over 5-fold CV).

https://doi.org/10.1371/journal.pone.0349865.t010

Download:

• PNG
  larger image
• TIFF
  original image

Table 11. Segmentation performance: RTM dataset (Mean ± SD over 5-fold CV).

https://doi.org/10.1371/journal.pone.0349865.t011

Lower Dice scores on CBIS-DDSM than RTM is likely since digitised film mammograms have heterogeneous imaging properties, there is boundary uncertainty in the annotations made by radiologists and there is a domain shift in the appearance of lesions. Because the Dice score is highly sensitive to contour alignment, even minor boundary misalignments cause measurable reductions in overlap metrics. The segmentation performance of the CBIS-DDSM dataset is worse than that of the RTM dataset, which is supported by the Dice scores shown in Tables 10 and 11. For example, the proposed model has a Dice score of 0.795 ± 0.009 on CBIS-DDSM and a Dice score of 0.882 ± 0.008 on RTM, i.e., a difference of approximately 9.9%. As will be demonstrated, several dataset-specific features can explain the differences in performance between the two datasets. First, the CBIS-DDSM dataset is made up of digitised film mammograms with lower contrast and much more noise, thus making it very difficult to delineate the boundaries of lesions. In comparison, the RTM dataset has much higher quality digital mammographic images, where the structures of lesions are usually visible and well-defined. Second, there is a higher inter-annotator variability in CBIS-DDSM than in RTM, which increases the difficulty of segmenting lesions because the CBIS-DDSM lesion boundaries are often irregularly shaped and have greater amounts of inter-observer variability; therefore, increasing difficulty in developing true ground-truth masks for segmenting lesions within the dataset. Despite these challenges, the proposed ViT-MultiRAGNet consistently demonstrates strong segmentation performance across both datasets, outperforming the baseline architecture indicating that the retrieval-augmented transformer framework is robust to variations across different datasets.

Figs 7 and 8 present bar-chart comparisons of segmentation performance across the evaluated models on the CBIS-DDSM and RTM datasets, respectively. Each bar represents the mean value of a specific metric, allowing direct visual comparison between baseline methods and the proposed ViT-MultiRAGNet framework. The bar-chart representation provides a clear overview of performance differences across Dice, IoU, HD95, sensitivity, precision, and AUC.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Bar chart comparison of segmentation performance metrics across different models on the CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0349865.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Bar chart comparison of segmentation performance metrics across different models on the RTM dataset.

https://doi.org/10.1371/journal.pone.0349865.g008

The bar charts in Figs 7 and 8 provide direct visual comparisons of segmentation metrics across the evaluated models. ViT-MultiRAGNet shows consistently higher values for Dice, IoU, sensitivity, precision, and AUC, while also achieving lower HD95 values. These results indicate improved lesion-overlap accuracy and boundary localization compared with the baseline models.

5.3 Quantitative evaluation of classification on both datasets

To prevent data leakage, all dataset partitions were generated at the patient level. Images belonging to the same patient were grouped using the patient identifier and assigned to the same fold during cross-validation. The retrieval memory bank was constructed independently for each fold using only the training samples of that fold. Validation and test samples were excluded from the memory bank to ensure that retrieval operations during inference accessed only training-derived embeddings. The proposed model was evaluated on both datasets, and the classification results are presented in Tables 12 and 13,

Download:

• PNG
  larger image
• TIFF
  original image

Table 12. Classification performance: CBIS-DDSM dataset (Mean ± SD over 5-fold CV).

https://doi.org/10.1371/journal.pone.0349865.t012

Download:

• PNG
  larger image
• TIFF
  original image

Table 13. Classification performance (RTM dataset) values are Mean ± SD over 5-fold CV.

https://doi.org/10.1371/journal.pone.0349865.t013

ViT-MultiRAGNet achieved the highest classification performance on the RTM dataset, with an accuracy of 0.978 ± 0.009 and an AUC of 0.998 ± 0.003. This improvement over the baseline models indicates the effectiveness of retrieval-augmented contextual reasoning. The graphical representation was presented the same at Fig 9 as follows.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Classification performance comparison on the CBIS-DDSM and RTM datasets.

https://doi.org/10.1371/journal.pone.0349865.g009

The graphical comparison of classification accuracy and AUC is presented in Fig 9. Bars represent the mean values across the five folds of cross-validation. The metrics shown include classification accuracy and AUC for the evaluated models. Table 14 provides classification performance including specific details. Table 15 provides the ablation study on retrieval-augmented module details.

Download:

• PNG
  larger image
• TIFF
  original image

Table 14. Classification performance including specificity on the CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0349865.t014

Download:

• PNG
  larger image
• TIFF
  original image

Table 15. Ablation study on retrieval-augmented module.

https://doi.org/10.1371/journal.pone.0349865.t015

To further evaluate the reliability of the predicted probabilities, we performed calibration analysis using Expected Calibration Error (ECE). The proposed ViT-MultiRAGNet demonstrated low calibration error, indicating that the predicted probabilities are well aligned with the observed outcome frequencies.

The ablation results demonstrate that incorporating the retrieval-augmented module improves segmentation accuracy and boundary localization. The RAG-enhanced model achieves higher Dice and IoU scores while reducing HD95, indicating more accurate lesion boundary reconstruction.

These results confirm that the retrieval component contributes directly to improved overlap accuracy and boundary localization.

5.4 F1-score comparative analysis

The F1-score comparison and cross-validation performance of the proposed model are presented in Tables 16–18 and Figs 10–13.

Download:

• PNG
  larger image
• TIFF
  original image

Table 16. F1-score performance comparison across models and datasets.

https://doi.org/10.1371/journal.pone.0349865.t016

Download:

• PNG
  larger image
• TIFF
  original image

Table 17. Cross-validation results: CBIS-DDSM dataset with 95% bootstrap confidence intervals.

https://doi.org/10.1371/journal.pone.0349865.t017

Download:

• PNG
  larger image
• TIFF
  original image

Table 18. Cross-validation results: RTM dataset with 95% bootstrap confidence intervals.

https://doi.org/10.1371/journal.pone.0349865.t018

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. F1-score comparison across evaluated 2D mammography models on the CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0349865.g010

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. F1-score comparison across evaluated 2D mammography models on the RTM dataset.

https://doi.org/10.1371/journal.pone.0349865.g011

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. Cross-validation metrics distribution and proportional representation for CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0349865.g012

Download:

• PNG
  larger image
• TIFF
  original image

Fig 13. Cross-validation metrics distribution and proportional representation for RTM dataset.

https://doi.org/10.1371/journal.pone.0349865.g013

Tables 17 and 18 report the cross-validation performance of the proposed ViT-MultiRAGNet model on the CBIS-DDSM and RTM datasets, respectively. A consistent reporting format of mean ± standard deviation with 95% bootstrap confidence intervals is used.

Receiver operating characteristic and precision-recall curves.

The ROC curves in Fig 14 show strong discrimination performance for ViT-MultiRAGNet on both datasets. The Precision–Recall curves in Fig 15 demonstrate consistently high precision across recall ranges, with AUC-PR values of 0.975 for CBIS-DDSM and 0.994 for RTM (Figs 16 and 17).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 14. ROC curves illustrating ViT-MultiRAGNet superior discrimination capacity across CBIS-DDSM and RTM datasets.

https://doi.org/10.1371/journal.pone.0349865.g014

Download:

• PNG
  larger image
• TIFF
  original image

Fig 15. Precision-Recall curves demonstrating ViT-MultiRAGNet precision retention across recall spectrum.

https://doi.org/10.1371/journal.pone.0349865.g015

Download:

• PNG
  larger image
• TIFF
  original image

Fig 16. Confusion matrix of the proposed ViT-MultiRAGNet model on the CBIS-DDSM dataset showing class-wise recall values of 98.1% for benign and 90.7% for malignant cases.

https://doi.org/10.1371/journal.pone.0349865.g016

Download:

• PNG
  larger image
• TIFF
  original image

Fig 17. Confusion matrix of the proposed ViT-MultiRAGNet model on the RTM dataset showing class-wise recall values of 98.1% for benign and 97.6% for malignant cases.

https://doi.org/10.1371/journal.pone.0349865.g017

5.5 Ablation studies and component contribution analysis

The segmentation task is formulated as a binary segmentation problem, where the model distinguishes lesion regions from background tissue. In this setting, the Dice coefficient and F1-score are mathematically equivalent. Therefore, Dice coefficient is reported as the primary overlap metric in the revised manuscript to avoid redundancy. The segmentation task is evaluated using Accuracy, Dice coefficient, and Area Under the ROC Curve (AUC). Since segmentation produces pixel-wise predictions, the AUC is computed at the pixel level by comparing the predicted probability map with the ground-truth binary segmentation mask. Specifically, each pixel is treated as an independent prediction, and the ROC curve is generated by varying the decision threshold applied to the predicted probability values. The AUC value is then calculated as the area under this ROC curve.

For model evaluation, k-fold cross-validation is employed. Metrics are computed separately for each fold using the test subset of that fold. The final reported results correspond to the mean and standard deviation across all folds, ensuring that the performance estimates reflect variability due to data partitioning. In addition to Dice coefficient, we also report Intersection over Union (IoU) and 95th percentile Hausdorff Distance (HD95) to provide a more comprehensive evaluation of segmentation performance. IoU measures the overlap between predicted and ground-truth segmentation masks, while HD95 quantifies the boundary distance between the predicted and reference contours, making it less sensitive to outliers than the maximum Hausdorff distance.

The ablation study was performed on both datasets, and the results are presented in Tables 19 and 20.

Download:

• PNG
  larger image
• TIFF
  original image

Table 19. Ablation study: CBIS-DDSM dataset component impact on segmentation.

https://doi.org/10.1371/journal.pone.0349865.t019

Download:

• PNG
  larger image
• TIFF
  original image

Table 20. Ablation study: RTM dataset component impact on segmentation.

https://doi.org/10.1371/journal.pone.0349865.t020

Ablation analysis indicates that removal of the RAG module results in substantial performance degradation, validating the importance of retrieval-augmented contextual reasoning. Removal of the ViT encoder causes severe deterioration, confirming the importance of global attention-based mammographic feature extraction. The reduced performance after removing the cross-attention fusion module further demonstrates that effective integration of retrieved context with current image features is essential for improved classification and segmentation. Overall, the systematic component analysis validates the architectural design choices of the proposed 2D mammography-based ViT-MultiRAGNet framework are graphically represented in the Figs 18–21 respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 18. Ablation study results visualizing component contribution to model performance on CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0349865.g018

Download:

• PNG
  larger image
• TIFF
  original image

Fig 19. Boundary error comparison (HD95) for CBIS-DDSM.

https://doi.org/10.1371/journal.pone.0349865.g019

Download:

• PNG
  larger image
• TIFF
  original image

Fig 20. Ablation study results visualizing component contribution to model performance on RTM dataset.

https://doi.org/10.1371/journal.pone.0349865.g020

Download:

• PNG
  larger image
• TIFF
  original image

Fig 21. Boundary error comparison (HD95) for RTM.

https://doi.org/10.1371/journal.pone.0349865.g021

Segmentation performance

The segmentation performance of the proposed model is summarized in Tables 10 and 11. As shown in Table 10, the proposed ViT-MultiRAGNet achieves the best performance on the CBIS-DDSM dataset, with a Dice score of 0.795 ± 0.009 and IoU of 0.660 ± 0.011, outperforming U-Net, TransUNet, and Swin-UNet. In addition to improved overlap accuracy, the proposed model achieves the lowest boundary error with an HD95 of 3.13 ± 0.22 mm, compared to 4.89 ± 0.35 mm for Swin-UNet and 6.57 ± 0.48 mm for U-Net. These results indicate that the proposed architecture produces more accurate lesion boundary delineation.

Similarly, the segmentation results on the RTM dataset, presented in Table 11, demonstrate the superior performance of the proposed method. The ViT-MultiRAGNet achieves a Dice score of 0.882 ± 0.008 and IoU of 0.789 ± 0.007, which are higher than the corresponding values for U-Net, TransUNet, and Swin-UNet. Furthermore, the proposed model achieves the lowest boundary distance with an HD95 of 2.81 ± 0.19 mm, indicating more precise lesion boundary localization. These results confirm that the integration of transformer-based feature extraction with retrieval-augmented contextual learning improves segmentation accuracy and boundary reconstruction across datasets.

Classification performance

The classification results further validate the effectiveness of the proposed architecture. As presented in Table 12, the proposed ViT-MultiRAGNet achieves an AUC of 0.987 ± 0.010 and an accuracy of 0.961 ± 0.011 on the CBIS-DDSM dataset, outperforming both ResNet-50 and Swin-T. The improved F1-score of 0.933 ± 0.006 indicates a better balance between sensitivity and precision, which is essential for medical screening systems where both false positives and false negatives must be minimized.

On the RTM dataset, the performance improvement becomes even more pronounced. As shown in Table 13, the proposed model achieves an AUC of 0.998 ± 0.008 and an accuracy of 0.978 ± 0.009, substantially surpassing the baseline models. The high sensitivity (0.976 ± 0.010) demonstrates the model’s strong ability to correctly detect malignant cases, while the high precision (0.964 ± 0.011) indicates reliable classification decisions. The additional comparison in Table 14 confirms that the proposed model also achieves the highest specificity (0.948), indicating improved discrimination between benign and malignant samples.

Contribution of the retrieval-augmented module

The effectiveness of the retrieval-augmented mechanism is highlighted by the ablation study results. As shown in Table 15, incorporating the RAG module increases the Dice score from 0.864 to 0.882 while simultaneously reducing the boundary error from 3.45 mm to 2.81 mm. This improvement demonstrates that retrieval-augmented contextual information can significantly enhance segmentation performance by providing additional reference information during feature representation.

The detailed ablation analysis in Tables 17 and 18 further confirms the importance of each architectural component. Removing the RAG module results in a noticeable decrease in Dice and IoU scores and an increase in HD95, indicating reduced segmentation accuracy and boundary quality. The most significant performance drop occurs when the ViT encoder is removed, where Dice decreases to 0.712 ± 0.015 on CBIS-DDSM (Table 17) and 0.692 ± 0.014 on RTM (Table 18). This observation highlights the importance of transformer-based feature extraction in capturing global contextual information.

Model efficiency and practical implications

Despite incorporating multiple components, the proposed architecture remains computationally efficient. As shown in Tables 10–13, the proposed model achieves the best performance while maintaining a relatively low inference time (0.31 s), which is significantly faster than transformer-based baselines such as Swin-UNet and TransUNet. This efficiency makes the proposed model suitable for practical deployment in clinical decision support systems where both accuracy and computational speed are important.

Stability of the model

The cross-validation results reported in Tables 15 and 16 further demonstrate the stability and robustness of the proposed approach. The relatively small standard deviations across folds indicate consistent performance across different data partitions. For example, on the RTM dataset (Table 16), the model achieves an accuracy of 0.978 ± 0.007 with a 95% confidence interval of (0.966, 0.991), confirming that the performance gains are not due to overfitting but are reproducible across different validation splits.

Overall, the results demonstrate that the proposed ViT-MultiRAGNet architecture effectively combines transformer-based feature extraction with retrieval-augmented contextual learning. This integration leads to improved segmentation accuracy, better boundary localization, and superior classification performance across multiple datasets while maintaining computational efficiency. These findings suggest that retrieval-augmented transformer architectures can provide a promising direction for improving medical image analysis systems.

6.1 State-of-the-art comparative benchmark

ViT-MultiRAGNet has the highest joint segmentation/classification accuracy when compared to current state-of-the-art models. Some current models excel at localization or diagnosis only, while very few do both simultaneously. The balanced performance of the proposed model shows that there is an architectural synergy among global attention and retrieval reasoning. Even though there is a moderate increase in computational complexity, performance gains in Dice, AUC, and boundary accuracy due to this trade-off outweigh the costs. Table 21 provides comprehensive comparison with contemporary state-of-the-art models.

Download:

• PNG
  larger image
• TIFF
  original image

Table 21. Comprehensive comparison with contemporary state-of-the-art models.

https://doi.org/10.1371/journal.pone.0349865.t021

ViT-MultiRAGNet achieves optimal classification AUC (0.998 RTM) among all evaluated methods while maintaining competitive segmentation performance (Dice 0.882 RTM) and reasonable computational resource requirements. Table 22 provides the comparison of the proposed model with other existing models.

Download:

• PNG
  larger image
• TIFF
  original image

Table 22. Computational complexity comparison of CNN-, transformer-, and hybrid-based models. GPU memory usage, parameter count, FLOPs, and inference time are reported for a single input image. Training time (s) denotes the average training time per batch during model training under the same hardware configuration.

https://doi.org/10.1371/journal.pone.0349865.t022

All computational measurements reported in Table 20 were obtained using an NVIDIA RTX 3090 GPU with 24 GB memory, an Intel Xeon CPU, and 32 GB RAM. Training time represents the average time per batch, while inference time denotes the average processing time per image during testing. The ViT-MultiRAGNet is a new hybrid Segmentation Model Figure that has achieved the best combination of model performance and computational efficiency. Pure CNN-based architectures (usually good at being efficient models) have many limitations when it comes to capturing long-range dependencies and therefore cannot be used to produce accurate segmentation maps for medical images where there’s often much more than just local detail present. Similarly, transformers as dominant architectures perform well in capturing global context. However, such architectures are typically too computationally intensive to be utilized in real-time (or near real-time) clinical practice. The integration of both types of models via hybrid architecture offers the best of both worlds.

ViT-MultiRAGNet incorporates the advantages of using a convolutional backbone and combining the convolutional model with attention-based modules. This enables the proposed model to capture both fine-grain local details as well as global context, which is essential for accurate fractional segmentation to occur in medical images. The hybrid design of the proposed model has allowed the model to have a high level of accuracy and robust feature representation while also avoiding the high memory usage and long inference times that are commonly seen with the heavier transformer architecture. In comparison to other state-of-the-art models, the proposed model has a reduced number of training parameters and FLOPs, which leads to reduced training/inference times without compromising accuracy of the resulting segmentation maps. The combined efficiency of the proposed model will allow it to scale to higher resolution medical images (typically, when viewed from 300 pixels up) making the model appropriate for use in real-time or near real-time clinical settings. Overall, the proposed model is an outstandingly constructed state-of-the-art architectural framework for efficient, hybrid segmentation of medical images, thus establishing a new performance standard for efficient medical image segmentation.

6.2 Model interpretability and visualization of performance

Fig 22 illustrates comprehensive model output visualization, including segmentation masks, retrieved historical case contexts, and visualizations of the attention mechanism.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 22. Comparative visualization of proposed ViT-MultiRAGNet performance versus established models across segmentation and classification metrics.

https://doi.org/10.1371/journal.pone.0349865.g022

Fig 23 illustrates comprehensive model output visualization, including segmentation masks, retrieved historical case contexts, and visualizations of the attention mechanism. The model accurately attends clinically significant features while identifying similar historical cases supporting diagnostic reasoning, thereby furnishing clinicians with explicit interpretable evidence supporting automated decisions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 23. Representative segmentation results produced by the proposed ViT-MultiRAGNet model on the CBIS-DDSM and RTM datasets.

The images illustrate example predictions generated during the testing phase and correspond to the same experimental evaluation used to compute the quantitative performance metrics reported in Tables 10 and 11.

https://doi.org/10.1371/journal.pone.0349865.g023

6.3 Clinical application

The proposed ViT-MultiRAGNet framework is designed not only to achieve high predictive performance but also to support clinical interpretability. The attention maps generated by the transformer encoder highlight image regions that contribute most significantly to the model’s prediction, enabling clinicians to visually verify whether the detected regions correspond to suspicious lesions. In addition, the retrieval-augmented module provides similar previously observed cases from the memory bank, allowing radiologists to compare the current mammographic findings with clinically validated examples. This combination of visual attention explanation and case-based retrieval provides a transparent decision-support mechanism that can assist clinicians in evaluating ambiguous lesions and improving diagnostic confidence.

Future research directions

Future investigations will systematically advance framework applicability and clinical deployment:

• Multi-Center Knowledge Base Development: Expanding memory repositories to encompass cases from diverse institutions and populations, improving generalizability and reducing institutional bias.
• Federated Retrieval Learning Implementation: Deploying privacy-preserving federated approaches enabling knowledge sharing while safeguarding patient privacy across health systems.
• Prospective Clinical Validation: Conducting radiologist-in-the-loop prospective studies assessing real-world clinical utility, workflow integration, and user acceptance.
• Multimodal Extension: Incorporating supplementary imaging modalities (ultrasound, MRI, PET) into unified analysis framework.
• Advanced Interpretability Tools: Developing sophisticated visualization methodologies for articulating retrieval mechanism decisions and attention patterns to clinicians.