Context-enhanced hard negative contrastive loss

To address the limitations of current contrastive learning methods in medical image segmentation—particularly their inefficiency in data utilization and inability to distinguish subtle differences—we introduce the CEHNC loss. This newly designed loss function incorporates additional components aimed at improving the model’s discriminative capability and training efficiency.

Unlike traditional contrastive losses such as InfoNCE [29], which typically treat all negative samples equally or use a fixed weighting scheme, CEHNC dynamically adjusts the contribution of each negative sample based on its context and difficulty. Traditional contrastive losses may fail to effectively utilize hard negatives—samples that are visually similar but belong to different categories—which are crucial for enhancing model generalization and learning efficiency.

Adaptive hard negative selection

In contrastive learning, conventional approaches often rely on random or fixed strategies to select negative samples—those semantically different from the anchor (i.e., positive) sample. However, such methods may fail to identify hard negatives, which are visually similar yet belong to different categories and are essential for improving model generalization and learning efficiency. To address this limitation, the CEHNC loss introduces an adaptive mechanism to select hard negatives.

AS show in Fig 2, the core idea of this mechanism is to dynamically adjust the weight assigned to each candidate negative sample within a training batch. Specifically, at the start of each batch, the similarity between the anchor and all other samples is computed—typically using metrics such as cosine similarity. Based on these similarity scores, the method identifies the most challenging negative samples for the current anchor. Then, considering the overall distribution of the batch, the algorithm adaptively updates two parameters, and , which control the difficulty curve of the selected negatives. These parameters ensure that sufficient attention is given to hard cases while still allowing for the inclusion of easier samples. As a result, the CEHNC loss enables the model to better focus on distinguishing fine-grained differences, ultimately leading to improved segmentation performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of the proposed CEHNC loss pipeline.

This figure presents the CEHNC loss mechanism, which dynamically selects hard-negative samples and updates difficulty parameters and to enhance fine-grained feature discrimination during training.

https://doi.org/10.1371/journal.pone.0344978.g002

Integration of contextual information

Beyond conventional image-text pairs, the CEHNC loss further incorporates domain-specific contextual information from medical data. This is achieved by integrating relevant clinical metadata, such as diagnostic reports or patient history, into the feature learning process. These additional contextual features are encoded using a dedicated context encoder and subsequently fused with both image and text embeddings before being fed into the contrastive loss function.

To implement this, all auxiliary data sources are first processed through a specialized context encoder, which transforms them into high-dimensional feature representations. These encoded features are then combined with the original visual and textual embeddings to construct a more comprehensive and semantically rich feature space. This integration not only enhances the model’s understanding within each individual modality but also facilitates cross-modal consistency and complementarity.

For example, when a patient’s prior medical records and personal health context are incorporated, the model can better interpret ambiguous findings in a single X-ray image, leading to more accurate identification of pathological conditions. By embedding such rich contextual knowledge, the CEHNC loss significantly improves the system’s ability to recognize complex disease patterns. Moreover, it enhances the robustness and interpretability of the model, enabling predictions that are not only more accurate but also more transparent and clinically meaningful.

Loss function formulation

The CEHNC loss is designed as a dual-direction contrastive objective that jointly optimizes image-to-text and text-to-image alignment. It is defined as:

(1)

where each term represents a directional component responsible for pulling positive pairs closer while pushing apart negative ones based on their embedding distances. The revised formulation now includes the positive pair term in the denominator (highlighted in the equations below), following standard contrastive learning conventions. Specifically:

(2)

(3)

with z_n and s_n denoting the n-th normalized embeddings from the vision and language encoders respectively, M the batch size, and a learnable scaling factor analogous to temperature. To dynamically modulate the influence of negative samples, we introduce two adaptive weighting functions:

(4)

(5)

where , are hyperparameters that control the sensitivity to hard negatives. These weights suppress the contribution of distant negatives exponentially with respect to their Euclidean distance in the joint embedding space. Unlike temperature scaling (), which uniformly scales all similarity scores, the weighting functions and apply sample-specific, distance-dependent weights that selectively emphasize hard negatives (samples closer in embedding space). For normalized embeddings, Euclidean distance and cosine similarity are monotonically related ( for unit vectors), so using Euclidean distance provides a direct hardness signal while maintaining consistency with the cosine similarity used in the contrastive objective. The learnable scaling factor is initialized to , following the default temperature setting in the original CLIP model, and is optimized jointly with other model parameters during training. In summary, CEHNC loss leverages dynamic weighting and context-aware hard negative mining to enhance the model’s ability to handle subtle differences in medical images, offering significant improvements over traditional contrastive losses.

To fine-tune the BiomedCLIP model [22] using the CEHNC loss function, we utilized the public MedPix dataset containing multiple radiological modalities. In this process, we chose the base version of Vision Transformer as the image encoder, while PubMedBERT [34] was used as the text encoder. To ensure the quality and consistency of the data, we performed a detailed preprocessing work on the MedPix dataset, which included removing all special characters, trimming the front and back margins, and excluding data samples with caption lengths of less than 20 characters. These steps help reduce noise and improve the efficiency of model training.

Next, all images are uniformly resized to a resolution of pixels and normalized using the mean and standard deviation values of the RGB channels as defined in the original CLIP model. This normalization strategy ensures better alignment and comparability of visual features across different modalities. Following preprocessing, the dataset is split into training and validation subsets using an 85%–15% ratio, yielding 20,292 images for training and 3,515 for validation. This partitioning strategy guarantees a sufficiently large training set while maintaining a representative validation set for reliable performance evaluation.

During the fine-tuning stage, a relatively low initial learning rate of is employed, combined with a stepwise learning rate decay of 50%, allowing the model to converge more steadily. Additionally, a batch size of 64 is used throughout training, which provides a balanced trade-off between computational efficiency and accurate gradient estimation. Through these carefully selected hyperparameters and training configurations, we aim to enhance the BiomedCLIP model’s effectiveness in medical image analysis tasks [35].

Zero-shot segmentation guided by DPFEM-enhanced textual cues

Using the fine-tuned BiomedCLIP model, we propose a zero-shot generalized medical image segmentation strategy. The strategy integrates the XAI technique, gScoreCAM [8], with BiomedCLIP to generate text-related visual saliency maps guided by anatomical or pathological cues. While gScoreCAM has shown superior accuracy and specificity over Grad-CAM [36] for natural images, we extend its use to radiology tasks and further enhance the generated saliency maps through the DPFEM (Fig 3) before post-processing them with Conditional Random Field (CRF) filters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Architecture of the proposed DPFEM.

This figure presents the DPFEM architecture, which captures both local details and global contextual information through parallel perception branches and dynamic gated feature fusion to enhance medical image segmentation.

https://doi.org/10.1371/journal.pone.0344978.g003

The gScoreCAM heatmaps are converted to SAM bounding box prompts through the following steps: (1) the heatmap is normalized to [0,1] using min-max normalization; (2) a threshold of 0.5 is applied to obtain a binary mask; (3) morphological closing with a 3×3 kernel is applied to fill small holes and smooth boundaries; (4) the tightest axis-aligned bounding box enclosing all non-zero pixels is computed; (5) this bounding box is provided as the prompt input to SAM for mask generation.

Fig 3 shows an overview of the proposed DPFEM. Our model leverages DPFEM to effectively capture both local details and global contextual information. The operation flow of the framework can be divided into several key steps: first, in the input image preprocessing stage, the original image is processed by an initial convolutional layer, a process that not only initially extracts spatial features, but also appropriately reduces the data dimensionality and lays the foundation for subsequent deeper feature learning. The next step is the local perception branch, where Depth-wise Separable Convolution (DSC) is used, combined with activation functions such as SiLU, which enables the model to retain as much spatial structure information as possible while maintaining high computational efficiency, which is crucial for understanding the subtle but important local features in the image. characteristics of an image. Meanwhile, in the global perception branch running in parallel, the Single-head Self-Attention (SHSA) mechanism is introduced, which allows the model to explore the correlations within the whole image region to better grasp the semantic content of the image as a whole and the interactions between its parts. The last step is dynamic gated attention fusion, where we innovatively combine feature maps from the local perception and global perception branches through a dynamic gating mechanism. This approach dynamically weighs the importance of local and global features based on their relevance and confidence scores, significantly enhancing the model’s ability to handle complex medical images. Unlike traditional fusion methods that simply concatenate or sum features, our dynamic gate adjusts weights in real-time, providing a more nuanced and context-aware feature representation. This dynamic gating mechanism not only improves segmentation accuracy but also boosts the model’s generalization capabilities across diverse imaging modalities.

The comprehensive representation rich in both fine-grained information and macroscopic vision is thus formed, which greatly enhances the model’s comprehension of the complex medical image as well as the expressiveness of the segmentation task.

Input image preprocessing.

The input image I first undergoes preprocessing through an initial convolutional layer [26]. This step extracts preliminary spatial features and reduces the data dimension for subsequent processing. The output of this step can be represented as:

(6)

where denotes the convolution operation.

Local perception branch.

This branch utilizes Depth-wise Separable Convolution (DSC) along with an activation function such as SiLU (Sigmoid Linear Unit), to preserve spatial structure information. DSC decomposes a standard convolution into two stages: a depth-wise convolution followed by a point-wise convolution. The process can be formulated as:

(7)

where and represent the depth-wise and point-wise convolution operations, respectively.

Global perception branch.

In parallel with the local perception branch, the global perception branch incorporates a Single-head Self-Attention (SHSA) mechanism, which focuses on learning global dependencies. The SHSA mechanism can be expressed as:

(8)

where represents the single-head self-attention operation. We adopt SHSA rather than multi-head attention for computational efficiency, as our dual-path architecture already provides feature diversity through parallel local and global perception branches. Empirically, SHSA achieves comparable performance to multi-head attention at lower computational cost for our zero-shot segmentation task, while still effectively capturing long-range dependencies between different regions of the image.

Feature fusion.

The features from both the local and global perception branches are then fused together. A common approach to fusion is to concatenate the feature maps and pass them through a fully connected layer or another convolutional layer. The fusion process can be described as:

(9)

where represents the fully connected or convolutional operation, respectively.

We apply a Conditional Random Field (CRF) filter to the fused feature map F^fused for post-processing, to obtain the initial coarse segmentation result. The CRF filter can smooth the segmentation boundaries and remove noise, thereby improving the segmentation quality. By integrating these two complementary paths, the network is not only able to precisely capture local details but also efficiently learn the global structural information within the images. This overcomes the limitations inherent in traditional CNNs and Vision Transformers (ViTs). Ultimately, the fused features result in a more comprehensive and richer representation, enhancing the model’s performance in tasks such as medical image segmentation. The dual-path design ensures computational efficiency while enhancing the model’s ability to parse complex medical images.

Weakly-supervised segmentation for potential labeling improvement

On the basis of zero-shot segmentation, we further introduce a weakly supervised learning strategy to improve the accuracy and robustness of the segmentation results. The specific implementation steps are as follows: first, the initial rough segmentation results are obtained by using the fine-tuned BiomedCLIP model and the saliency map generated by gScoreCAM, combined with the DPFEM and the Conditional Random Field (CRF) filter for post-processing. These rough segmentation results are used as pseudo-labels. More accurate pseudo-masks are generated by passing these pseudo-labels to the SAM. These pseudo-masks are used as the results of zero-shot segmentation to provide the basis for subsequent weakly-supervised segmentation training with potential labeling improvements.

In the weakly supervised segmentation training, we use the pseudo-masks generated above as training data. These pseudo-masks are not fully accurate labeling, but they can provide enough information to guide the learning of the model. Meanwhile, we choose residual UNet [23] as the model for weakly supervised training. Residual UNet is an efficient convolutional neural network architecture that can provide high-quality segmentation results while maintaining computational efficiency. During the training process, we use the Cross-Entropy Loss function (CELF) to measure the difference between the model predictions and the pseudo-labeling. Parameter updates are performed using the Adam optimizer with a learning rate of 1E-4 and appropriate decay strategies are set. To increase the generalization ability of the model, we applied various data enhancement techniques such as random rotation, flipping, and scaling during the training process. After weakly supervised training, the model is able to generate finer and more accurate segmentation results. These improved segmentation results not only increase the boundary accuracy, but also reduce noise and mis-segmentation.

Experimental setup, datasets, and validation metrics

We evaluate MedZeroSeg on three public medical imaging datasets: ACDC for cardiac MRI, Synapse for multi-organ CT, and COVID-QU-Ex for chest X-ray (pulmonary segmentation). All models are implemented in PyTorch and trained on NVIDIA A100 GPUs with the same hyperparameter settings for fair comparison. All quantitative results are reported as mean ± standard deviation over three independent runs with different random seeds to account for training variability.

Notably, MedZeroSeg performs 2D slice-wise inference for inherently 3D datasets (ACDC and Synapse), due to the 2D-only input requirement of the pre-trained CLIP and SAM models. Consequently, 3D/2.5D inference or volumetric post-processing is not applied. This design choice preserves the strict zero-shot setting without introducing task-specific heuristics.

To evaluate the quality of BiomedCLIP fine-tuning, we performed tests on the ROCO dataset [37], which contains about 7,042 multimodal medical images covering a wide range of clinical cases. We validated the top-1 and top-2 matching retrieval accuracy in both image-to-text and text-to-image directions. For the experiments, we performed five runs, each using a batch size of 50, and ensured diverse combinations of text and images within each batch by randomly shuffling the batch (generating a total of 70,420 rearranged examples). During the fine-tuning process, we compared several different loss functions, including the InfoNCE loss [29], DCL [38], HN-NCE [39], and our proposed CEHNC loss. To ensure fairness, all strategies are trained with the same hyperparameters (temperature , learning rate ). For HN-NCE and CEHNC, we set a uniform difficulty coefficient . In addition, we also consider the pre-trained BiomedCLIP [22], PMC-CLIP [16], and CLIP [10] as the baseline models.

To evaluate the segmentation performance under zero-shot and weakly supervised conditions, as well as the effectiveness of different design components of MedZeroSeg, we selected three public datasets (covering three different imaging modalities) that provide ground truth segmentation labels (cardiac structures, abdominal organs, and lung fields) and are divided into training, validation, and test sets. The details are as follows:

• Synapse [32]: Synapse is a public multi-organ segmentation dataset. There are 30 contrast-enhanced abdominal clinical CT cases in this dataset. Following the settings in [40], 18 cases are used for training and 12 for testing. The annotation of each image includes 8 abdominal organs.
• ACDC [31]: ACDC is a public cardiac MRI dataset consisting of 100 exams. For each exam, there are two different modalities, and the corresponding label includes left ventricle, right ventricle, and myocardium. The dataset is split into 80 for training and 20 for testing.
• Chest X-ray (COVID-QU-Ex) [33]: The COVID-QU-Ex database includes 16,280, 1,372, and 957 chest radiographs (including normal, lung opacities, viral pneumonia, and COVID-19 cases) for training, validation, and testing, respectively.

Based on the above datasets, we conducted a detailed comparative analysis to evaluate the segmentation quality of the initial labels, zero-shot pseudo-masks, and weakly supervised methods generated from the gScoreCAM results after conditional random field processing guided by the DPFEM on each test set. For the ablation study of the zero-shot segmentation task, we paid particular attention to two aspects: first, the effect of BiomedCLIP model fine-tuning on the final segmentation results; and second, comparing the performance differences between two different classes of activation map (CAM) techniques, gScoreCAM and Grad-CAM, in generating segmentation masks. These ablation experiments were all performed on a test subset of each of the three previously mentioned independent datasets.

To ensure the consistency of the experimental conditions, we uniformly used the same SAM architecture, a fixed selection of target layers, a consistent text cueing strategy, and considered only the top 60 most salient channels from each input image for the CAM analysis, regardless of the settings. In addition, several widely recognized quality evaluation metrics were used to measure the performance of the segmentation algorithms throughout the study, including Intersection over Union (IoU),Dice Similarity Coefficient (DSC), and Area Under the Curve of the Receiver Operating Characteristic (AUC). Their formula is as follows.

(10)

(11)

(12)

By implementing a paired-samples t-test, we were able to scientifically verify whether the observed phenomena and trends were statistically significant. Specifically, when the p-value is less than 0.05, it can be considered that there is a significant difference between the two sets of data, thus supporting our experimental conclusions. Metrics such as HD95 and ASSD were not reported, as they are highly sensitive to small boundary discrepancies and annotation noise, which are pronounced in medical images and exacerbated in zero-shot settings. Therefore, Dice, IoU, and AUC were selected as more reliable and clinically meaningful evaluation metrics.

Data preprocessing.

Prior to model training and evaluation, comprehensive data preprocessing was applied to ensure data quality and consistency across all datasets. For the ROCO dataset, images were resized to pixels and normalized using ImageNet statistics. Text captions were lowercased, tokenized using the BiomedBERT tokenizer, and truncated or padded to a maximum sequence length of 77. Duplicate or ambiguous image-text pairs were removed through automated filtering and manual inspection. For the segmentation datasets, we performed modality-specific preprocessing: (1) In Synapse and ACDC, CT and MRI volumes were resampled to isotropic spacing (1 mm³ voxels), intensity-normalized using z-score normalization per volume, and sliced into axial 2D slices for training; (2) For Lung X-ray, chest radiographs were resized to , and pixel intensities were scaled to [0,1]. To reduce bias, all datasets were checked for patient overlap between train/validation/test splits, and data augmentation (random horizontal flipping, rotation , and brightness jittering) was applied only on the training sets during fine-tuning.

Quantitative and qualitative results

Table 1 demonstrates the performance of the BiomedCLIP model on the ROCO dataset in cross-modal retrieval tasks including text-to-image and image-to-text after fine-tuning the model using a number of different loss functions. To provide a benchmark comparison, we also list three pre-trained CLIP models as references, namely, the standard CLIP [10], the medical domain-specific PMC-CLIP [16], and the original version of BiomedCLIP [22]. Through a paired McNemar statistical test, we found that the BiomedCLIP model fine-tuned using our CEHNC loss function significantly outperforms the other available loss functions and all pre-trained baseline models. This result demonstrates the effectiveness of the CEHNC loss function in improving cross-modal retrieval accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Top-K cross-modal retrieval accuracy for CLIP models on the ROCO dataset.

https://doi.org/10.1371/journal.pone.0344978.t001

Table 2 shows in detail the accuracy assessment of MedZeroSeg under different settings in the zero-shot segmentation task. Specifically, we compare the performance of the model in two cases: first, using pre-trained BiomedCLIP focuses on context-aware hard negative mining; second, the effectiveness of gScoreCAM versus the Grad-CAM method in generating SAM bounding box cues. The experimental results show that the bounding box cues generated using gScoreCAM are significantly better than those of the Grad-CAM method, suggesting that gScoreCAM is able to improve segmentation accuracy more effectively. In addition, by fine-tuning BiomedCLIP using our proposed CEHNC loss function, not only its superior performance across multiple task types and different image modalities was further verified, but also an overall improvement in segmentation quality was observed. This confirms the effectiveness of the CEHNC loss function for enhancing the performance of BiomedCLIP in medical image processing tasks. In Table 2, we also include the MIS-Net method [41], a SAM-based weakly-supervised baseline, for comparison, against which our improvements are clearly demonstrated.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of different models and CAM techniques with standard deviation and training time.

https://doi.org/10.1371/journal.pone.0344978.t002

In Table 3, we show in detail the segmentation accuracy of the proposed methods under zero-shot and weakly-supervised settings, with fully-supervised segmentation as a reference benchmark. This comparison helps to comprehensively evaluate the performance of different methods in real applications.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance of various methods compared to fully supervised baseline.

https://doi.org/10.1371/journal.pone.0344978.t003

For the zero-shot segmentation task, we compare two different ways of generating initial labels: one is based on the initial labels generated by the gScoreCAM saliency map (“saliency map”), and the other is a pseudo-mask generated via SAM (“saliency map + SAM”). The experimental results show that the method combining BiomedCLIP and SAM exhibits significant superiority in all evaluation metrics, significantly improving segmentation quality (p < 0.05). This suggests that by integrating these two techniques, critical regions in the image can be captured more efficiently, thus improving the accuracy and reliability of segmentation.

Under the weakly supervised setting, the performance comparison between zero-shot and weakly supervised methods shows dataset- and metric-dependent trends. For the Chest X-ray dataset, the weakly supervised ResUNet substantially outperforms the zero-shot approach across all metrics (IoU: 75.95 vs. 48.85, DSC: 85.92 vs. 64.24). In contrast, for ACDC and Synapse, the zero-shot method (Saliency Maps + DPFEM + CRF) achieves competitive or superior performance compared to the weakly supervised ResUNet in terms of IoU and DSC. Specifically, on ACDC, the zero-shot method achieves IoU of 57.52 vs. 40.99 for weakly supervised, and on Synapse, IoU of 49.91 vs. 41.55. It should be noted that the statistical significance markers in Table 3 indicate comparisons against the fully supervised ResUNet baseline, not against the weakly supervised method.

Although current fully-supervised deep learning models [23] provide state-of-the-art accuracy in the field of medical image segmentation, our MedZeroSeg zero-shot segmentation method still demonstrates competitiveness on some specific tasks. Specifically, MedZeroSeg outperforms the ResUNet-based fully supervised method in the ACDC and Synapse segmentation tasks. This finding suggests that our method can provide high-quality segmentation results even in the absence of large amounts of labeled data.

However, the fully-supervised method still performs well in the lung radiograph segmentation task, outperforming the zero-shot method both in terms of evaluation metrics such as IoU, DSC, and AUC. This suggests that sufficient labeled data is still a key factor in improving segmentation accuracy in certain application scenarios.

Fig 4 presents representative qualitative results for the zero-shot segmentation setting on the ACDC cardiac MRI dataset. Each example includes the original image, the corresponding ground-truth label, and the predictions produced by BiomedCLIP, the SAM-based weakly supervised baseline MIS-Net, and the proposed MedZeroSeg framework. Under the same zero-shot segmentation configuration, MedZeroSeg yields cardiac masks that more faithfully follow the contours of the left ventricle, right ventricle, and myocardium, with more complete coverage of thin myocardial walls and clearer separation between adjacent structures. In contrast, BiomedCLIP often under-segments the ventricular cavities or fails to capture subtle myocardial boundaries, while MIS-Net tends to generate fragmented or overly smooth contours that deviate from the ground truth. Across diverse cardiac views and varying contrast levels, MedZeroSeg produces fewer spurious predictions in the background and better preserves small anatomical details. These qualitative observations are consistent with the quantitative improvements in IoU, DSC, and AUC reported in Table 2, and jointly demonstrate the superiority of MedZeroSeg over BiomedCLIP and MIS-Net for zero-shot cardiac segmentation on ACDC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The cardiac MRI segmentation results on the ACDC dataset.

Representative qualitative comparison among BiomedCLIP, the SAM-based weakly supervised baseline MIS-Net, and the proposed MedZeroSeg in the zero-shot segmentation setting. For each cardiac MRI slice, the original image, ground-truth label, and predicted masks from the three methods are shown. MedZeroSeg yields more accurate and complete delineations of the left ventricle, right ventricle, and myocardium, with clearer boundaries and fewer spurious predictions than BiomedCLIP and MIS-Net.

https://doi.org/10.1371/journal.pone.0344978.g004

Fig 5 shows qualitative zero-shot segmentation results on the Synapse multi-organ abdominal CT dataset, again comparing MedZeroSeg with BiomedCLIP and MIS-Net. The examples cover several representative abdominal organs, including both large structures and smaller, shape-irregular targets. Under identical zero-shot segmentation conditions, MedZeroSeg produces organ masks that more closely align with the ground-truth annotations, particularly around complex boundaries and regions with low contrast or ambiguous intensity transitions. BiomedCLIP frequently under-segments target organs or confuses neighboring tissues, while MIS-Net tends to suffer from boundary leakage and missing small structures, leading to incomplete or noisy segmentations. By contrast, MedZeroSeg maintains coherent organ shapes and reduces both false positives and false negatives across different slices and patients. Together with the performance gains summarized in Table 2, the qualitative results in Fig 5 further highlight that MedZeroSeg generalizes more effectively than BiomedCLIP and MIS-Net to challenging multi-organ CT segmentation tasks in the zero-shot setting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The multi-organ CT segmentation results on the Synapse dataset.

Qualitative zero-shot segmentation results comparing BiomedCLIP, MIS-Net, and the proposed MedZeroSeg on representative abdominal CT slices. Each example includes the original image, ground truth annotations, and the corresponding predictions from the three methods. Across multiple abdominal organs, MedZeroSeg produces masks that better align with the ground truth, particularly around challenging boundaries and small structures, while BiomedCLIP and MIS-Net tend to under-segment targets or leak into adjacent tissues.

https://doi.org/10.1371/journal.pone.0344978.g005

Beyond cardiac MRI and abdominal CT, we further evaluate MedZeroSeg in the zero-shot segmentation setting on the COVID-QU-Ex chest X-ray dataset, which contains chest radiographs from normal, lung opacity, viral pneumonia, and COVID-19 cases. As illustrated in Fig 6, each example shows the original radiograph, the ground-truth lung or lesion mask, and the corresponding predictions from BiomedCLIP, the SAM-based weakly supervised baseline MIS-Net, and the proposed MedZeroSeg. Under identical zero-shot segmentation conditions, BiomedCLIP tends to under-estimate the spatial extent of diffuse pulmonary opacities or miss faint peripheral lesions, while MIS-Net often produces fragmented or overly smooth masks with noticeable leakage into ribs, heart shadows, or mediastinal regions. In contrast, MedZeroSeg generates more contiguous and anatomically plausible segmentations, better delineating the bilateral lung fields and pathological opacities while suppressing background structures. These qualitative findings are consistent with the quantitative results in Table 2, where MedZeroSeg achieves the highest IoU, DSC, and AUC among the three methods on the chest X-ray dataset, further demonstrating its advantage in zero-shot chest X-ray segmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The chest X-ray segmentation results on the COVID-QU-Ex dataset.

Representative qualitative comparison of lung or lesion segmentation in the zero-shot segmentation setting using BiomedCLIP, the SAM-based weakly supervised baseline MIS-Net, and the proposed MedZeroSeg. For each chest radiograph, the original image, ground-truth mask, and predicted masks from the three methods are presented. Compared with BiomedCLIP and MIS-Net, MedZeroSeg more accurately captures the extent of pulmonary opacities and preserves anatomically plausible lung boundaries, while reducing false positives in ribs, cardiac regions, and other background structures.

https://doi.org/10.1371/journal.pone.0344978.g006