2.1 Medical image segmentation

Medical image segmentation has long been recognized as a critical foundation for computer-aided diagnosis and treatment planning, particularly in thoracic oncology where accurate delineation of lung nodules and lesions directly impacts diagnostic reliability and therapeutic outcomes. Traditional deep learning-based segmentation frameworks such as UNet and V-Net have achieved considerable success, yet their performance is often limited by boundary ambiguity, inter-patient variability, and complex morphological characteristics of pulmonary structures. Recent research has therefore focused on improving feature representation and spatial precision through network architectural innovations. For instance, Lu et al. [9] developed a deep learning-based lung nodule detection and recognition model that effectively combines multi-level contextual learning, while Wu et al. [10] proposed IAD-VNet to enhance feature aggregation and suppress false positives in pulmonary nodule segmentation. Similarly, Jiang et al. [11] introduced a boundary-aware dual-branch neural network that emphasizes contour-sensitive feature extraction to refine segmentation edges, and Lian et al. [12] integrated PET/CT data within a CFC-MSPCNN framework to exploit multimodal information for more accurate tumor localization.

In addition to architecture-level innovations, recent works have increasingly emphasized clinical relevance, interpretability, and robustness of segmentation models across varying imaging modalities and disease contexts. Xie et al. [13] demonstrated the clinical value of multiscale CT radiomics for differential diagnosis between benign and malignant lung nodules, highlighting the potential synergy between segmentation and radiomics analysis. Riahi et al. [14] further reviewed how machine learning and artificial intelligence are revolutionizing lung cancer management by enabling more personalized and precise treatment strategies. Beyond lung cancer, similar deep segmentation techniques have been extended to related thoracic diseases such as COVID-19, where Arora et al. [15] proposed a multi-label deep network capable of segmenting and detecting pulmonary abnormalities from chest radiographs. Collectively, these studies underscore a growing trend toward hybrid, context-aware, and clinically interpretable segmentation methods that bridge the gap between algorithmic accuracy and practical medical application.

2.2 Uncertainty modeling in medical image segmentation

Uncertainty modeling in medical image segmentation has gradually become a research hotspot, aiming to characterize the sources of uncertainty in model predictions under conditions such as boundary ambiguity, annotation variability, and limited training data. Kendall et al. [16] proposed to distinguish and model aleatoric and epistemic uncertainties within Bayesian deep learning, providing a foundational framework for tasks such as semantic segmentation. Subsequently, Kohl et al. [17] introduced the Probabilistic U-Net, which employs a conditional variational autoencoder to generate diverse segmentation outputs, thereby capturing images with multiple plausible solutions. Wang et al. [18] further combined test-time augmentation strategies to estimate uncertainty induced by observational noise in convolutional neural network-based segmentation. Mehrtash et al. [19] approached the problem from the perspective of model calibration, proposing confidence calibration methods to enhance the consistency between predicted probabilities and ground truth distributions, thus improving the reliability of uncertainty estimation. These methods collectively advanced uncertainty modeling from a theoretical concept to practical applications in medical image segmentation.

In recent years, extensive studies have focused on expanding the capacity of uncertainty modeling across diverse application scenarios. Zou et al. [20] provided a systematic review of uncertainty estimation methods in medical image segmentation, covering Bayesian networks, ensemble learning, probabilistic models, and test-time augmentation. Buddenkotte et al. [21] proposed a scalable ensemble-based framework for uncertainty quantification, which effectively improved reliability in large-scale segmentation tasks through model calibration. Roshanzamir et al. [22] linked inter-rater variability with both aleatoric and epistemic uncertainties, revealing how subjective annotation differences influence the trustworthiness of segmentation results. Huang et al. [23] integrated Bayesian U-Net with active learning strategies to guide sample selection in multi-label dental CT segmentation tasks based on uncertainty. Meanwhile, Li et al. [24] introduced the QUBIQ challenge, which provides benchmark datasets for uncertainty quantification, and Chlap et al. [25] conducted empirical studies using a 3D probabilistic U-Net on small-scale radiotherapy trial datasets. Collectively, these works demonstrate that uncertainty modeling not only enhances segmentation performance and generalization but also provides critical insights into the trustworthiness and interpretability of model predictions in clinical scenarios.

2.3 Morphology-aware regularization and structural priors

In medical image segmentation tasks, incorporating morphological priors and structural constraints has been regarded as an effective approach to enhance the plausibility of predictions and maintain structural consistency. Zheng et al. [26] modeled conditional random fields as recurrent neural networks and proposed CRF-RNN, which can be trained end-to-end to refine segmentation boundaries and ensure global consistency. Krähenbühl et al. [27] developed an efficient inference method for fully connected CRFs, which preserves fine boundary details through high-dimensional filtering. Oktay et al. [28] proposed Anatomically Constrained Neural Networks (ACNNs), embedding prior anatomical shapes into segmentation models to guarantee anatomical consistency. Kervadec et al. [29] designed a boundary-based loss function to alleviate contour deficiencies in highly imbalanced segmentation tasks. Karimi et al. [30] optimized the Hausdorff distance, explicitly reducing the maximum deviation between predictions and ground truth boundaries, thereby improving the geometric stability of segmentation results.

Further studies have introduced topological constraints and geometric regularization into deep learning frameworks to strengthen structural consistency. Hu et al. [31] proposed a topology-preserving segmentation method, where topological constraints were integrated into the training process to prevent connectivity violations. Clough et al. [32] introduced a topological loss function based on persistent homology to ensure that segmentation predictions align with global topological priors. Weigert et al. [33] proposed a star-convex polyhedra-based shape modeling method for microscopy image segmentation, enforcing geometric constraints to guarantee morphological plausibility. Chen et al. [34] integrated Euler’s elastica model with deep networks, achieving learnable morphological regularization based on curvature and length. Kang et al. [35] provided a comprehensive survey on fast algorithms for elastica-based models, offering systematic insights into elastica priors for image segmentation. These studies collectively demonstrate that morphological priors and structural constraints not only improve the robustness and interpretability of segmentation results but also provide crucial solutions to address complex boundaries and morphologically implausible predictions.

2.4 Summary and motivation

In summary, while existing research has made substantial progress in generic medical image segmentation, studies specifically targeting lung nodule and lesion segmentation in CT images still face persistent challenges. Conventional CNN-based methods often struggle with blurred and irregular tumor boundaries caused by low contrast and imaging noise, leading to over-smoothed or discontinuous contours that compromise diagnostic accuracy. Transformer-based approaches, though capable of capturing long-range dependencies, tend to neglect local structural priors and thus suffer from morphological distortion and fragmented predictions in complex thoracic regions. Moreover, most uncertainty-aware or morphology-regularized methods are designed for general anatomical structures rather than the highly heterogeneous and irregular patterns of lung tumors, resulting in limited generalization to real-world clinical cases. To bridge this gap, the proposed OncoSeg2D framework integrates uncertainty-aware boundary modeling and morphology-preserving regularization within a SegFormer backbone, effectively coupling probabilistic boundary refinement with global shape constraint. This design enables the model to achieve precise boundary delineation and structural coherence simultaneously, thereby enhancing the reliability, interpretability, and clinical applicability of 2D CT lung cancer segmentation.

3.1 Overall model architecture

The overall structure of the OncoSeg2D framework proposed in this paper is shown in the Fig 1. Its core consists of three components: a feature extraction backbone network (Backbone), uncertainty-aware boundary modeling, and morphology-preserving regularization. The Backbone uses SegFormer’s MiT-B2 as the backbone network to extract semantic and boundary features from lung cancer CT slices at multiple scales, while the key novelty of OncoSeg2D lies beyond the backbone architecture. Specifically, SegFormer-style Transformer models primarily improve segmentation by designing efficient multi-scale feature encoding and lightweight decoding, but they typically treat boundaries as deterministic targets and do not explicitly quantify boundary ambiguity in medical lesions. In contrast, OncoSeg2D introduces an uncertainty-aware boundary modeling module that represents boundary predictions as a probabilistic distribution and explicitly captures boundary ambiguity, enabling more reliable boundary learning under weak contrast and irregular lesion contours. Moreover, although edge-aware Transformer variants such as EDAFormer enhance boundary perception through explicit edge/attention designs, they still focus on strengthening edge responses in a deterministic manner and lack an explicit mechanism to model boundary uncertainty and to constrain the predicted masks to satisfy clinically plausible morphology. OncoSeg2D further incorporates morphology-preserving regularization to enforce shape-consistent predictions, which complements uncertainty-aware boundary modeling by reducing unrealistic fragmentations and distortions, thereby forming a boundary-uncertainty–morphology synergy that is not present in SegFormer or EDAFormer.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The overall framework of OncoSeg2D employs SegFormer MiT-B2 as the backbone to extract multi-scale features, incorporates UBM for uncertainty-aware boundary modeling, and integrates MPR for morphology-preserving regularization.

This design enhances both boundary refinement and global structural consistency in lung cancer CT segmentation.

https://doi.org/10.1371/journal.pone.0348719.g001

Given an input 2D CT image , it first passes through a linear projection layer (Patch Embedding) to divide the image into fixed-size patches and project them into a low-dimensional feature space:

(1)

where P is the patch size and C is the embedding dimension.

Next, MiT-B2 progressively extracts multi-scale features through four stages (Stages 1–4) of Transformer encoders, each consisting of an attention layer and a feed-forward network (FFN). In the lth layer, the input feature is updated via the multi-head attention (MHA) mechanism and residual connections to:

(2)

Then it is further transformed by the feedforward network:

(3)

After four stages, a set of feature maps at different scales is obtained:

(4)

Where These correspond to low-level, fine-grained boundary features and high-level semantic features, respectively, providing a representational foundation for subsequent uncertainty-aware boundary modeling and morphology-preserving regularization.

In this architecture, the Backbone module is responsible for capturing the global context and local boundary patterns of CT slices. These multi-scale features not only enhance sensitivity to fuzzy tumor boundaries but also provide prior support for morphological constraints. Subsequently, the uncertainty-aware boundary modeling performs probabilistic boundary modeling in fuzzy regions, while morphology-preserving regularization constrains the morphological plausibility of the segmentation results through regularization, thereby achieving refined and structurally consistent lung cancer segmentation.

3.2 Uncertainty-aware boundary modeling

In lung cancer CT segmentation, tumor boundaries often exhibit inherent ambiguity, making it difficult even for experienced physicians to accurately delineate pixel-level contours. This ambiguity arises not only from imaging device noise and contrast deficiencies, but also from the morphological uncertainty of the lesions themselves. If traditional one-hot labeling is used for supervision, the model will forcibly learn hard labels at the boundaries, resulting in jittery predictions. Therefore, in OncoSeg2D, we propose Uncertainty-Aware Boundary Modeling to explicitly model this ambiguity in feature space, making the model’s representation of boundary transitions more flexible and robust. Its module architecture is shown in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Schematic structure of the UBM module.

Multi-scale features from the backbone are fused through a pyramid structure to generate the initial boundary probability representation, which is then combined with uncertainty parameter branches and gradient field information to produce feature representations that incorporate both boundary sensitivity and uncertainty modeling.

https://doi.org/10.1371/journal.pone.0348719.g002

The idea behind uncertainty-aware boundary modeling is to leverage the multi-scale features output by the backbone to generate a probability distribution for the boundaries rather than a single binary prediction. First, we align and fuse the feature maps at a unified scale to obtain the fused features:

(5)

Where represents the multi-scale feature fusion operation, which ensures that the model can simultaneously utilize low-level, fine-grained boundary information and high-level semantic context information.

After obtaining the fused features, we want to map them to a boundary response space. To this end, we introduce the boundary saliency function:

(6)

Where is the convolution kernel, * represents the convolution operation, and is the sigmoid function, which constrains the output to the range [0,1] and represents the boundary probability.

However, boundary probabilities alone are insufficient to describe uncertainty. Therefore, we further introduce mean and variance parameterized features through variational methods:

(7)

(8)

Where and represent the mean and variance of the boundary region, respectively, reflecting the degree of uncertainty in the boundary location.

Based on the above parameterization, we can sample the potential boundary distribution in the feature space:

(9)

Where is Gaussian noise, and ⊙ represents element-wise multiplication. This sampling mechanism enables the model to generate different possible contours at the boundary, thereby simulating the differences in manual annotations.

Furthermore, to further refine boundary modeling, we combine the original boundary probability with the sampled latent representation:

(10)

Where is the balancing coefficient, and serves as the normalization function. This equation indicates that the final boundary estimate is a weighted combination of the deterministic boundary probability and the uncertain sampling results.

At the spatial level, boundary uncertainty can be expressed using a gradient field. We define the gradient magnitude of the fused feature as:

(11)

Where (x y) is the spatial coordinate. This gradient field characterizes the region with the most dramatic boundary changes, which often corresponds to the transition zone at the edge of the tumor.

Combining the uncertainty parameter with the gradient field, we can further construct a boundary-sensitive weight:

(12)

This weight indicates that the model is more confident about the boundary location in regions with low uncertainty and large gradients, while remaining flexible in regions with high uncertainty.

To explicitly separate boundary and non-boundary regions in the feature space, we introduce a boundary mask:

(13)

Where is the threshold and is the indicator function. This mask divides the feature map into potential boundary and non-boundary regions.

Furthermore, we weight the fused features F under the mask constraint:

(14)

Where represents boundary features, and represents non-boundary features. By explicitly separating them, the model can learn stronger discriminative capabilities for boundary regions during training.

Finally, to maintain boundary consistency across scales, we align the boundary representations of features at each scale:

(15)

Where represents the multi-scale fusion function, and represents the uncertainty boundary estimate obtained at the ith scale. This operation ensures consistent boundary prediction at different resolutions.

In summary, the uncertainty-aware boundary modeling transforms boundary ambiguity in lung cancer CT images into explicit uncertainty modeling through probabilistic boundary modeling, mean-variance parameterization, random sampling, and gradient guidance, thereby improving the model’s robustness and generalization under complex boundary conditions. Tightly integrated with the multi-scale features extracted by Backbone, the uncertainty-aware boundary modeling not only avoids overfitting to hard labels but also produces smoother and more clinically meaningful segmentation results during inference.

3.3 Morphology-preserving regularization

In 2D CT slices, the visual morphology of lung cancer tumors often exhibits a coexistence of an overall nearly elliptical shape and localized spurs, with breaks and gaps often present at their boundaries. Relying solely on pixel-by-pixel classification often results in illogical fragmentation and morphological distortion. Therefore, in OncoSeg2D, we introduce a morphology-preserving regularization module without changing the basic decoder architecture. This module uses geometric and morphological operators to enforce consistent constraints on multi-scale semantic features from Backbone and boundary cues from the uncertainty-aware boundary modeling. Using soft segmentation probabilities as a carrier and representations such as level sets and structure tensors, it performs projective updates in the feature and mask spaces. This approach, without explicitly introducing supervision, establishes soft constraints on connectivity, density, convexity, and spur preservation. Its module architecture is shown in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Schematic diagram of the MPR module architecture.

It first utilizes uncertainty-aware boundary modeling output and multi-scale features to generate geometric representations such as level set functions, curvature, and area. Subsequently, through geometric morphological operations and regularization units, structural priors are embedded in the segmentation process to maintain morphological plausibility.

https://doi.org/10.1371/journal.pone.0348719.g003

First, the fused features F and the uncertainty boundary estimate from the uncertainty-aware boundary modeling are combined to form the initial region probability P. This step serves only as the input representation for subsequent geometric operations:

(16)

Where ⊕ represents feature-level concatenation, * represents convolution, and is the compression function. The resulting P encodes both global semantics and boundary confidence.

To perform geometric morphological computation, we use the signed distance transform to map P to the level set function , where positive values represent the inside of the target and negative values represent the outside:

(17)

provides a continuously differentiable isosurface representation, facilitating a unified measure of curvature, connectivity, and contour length.

The curvature of can be used to characterize the curvature of local boundaries, thereby selectively smoothing noise-induced aliasing:

(18)

is a curvature scalar field whose magnitude indicates the strength of local convexity and concavity.

Combined with the boundary uncertainty weight provided by the uncertainty-aware boundary modeling, we use a first-order explicit update to perform a boundary-sensitive regularized flow on to suppress unreasonable small fluctuations and stop at the clear boundary:

(19)

Where is the step size, comes from the uncertainty-aware boundary modeling gradient uncertainty map, and reduces the flow rate in high uncertainty regions to avoid oversmoothing truly fuzzy boundaries.

At the global scale, we define compactness metrics using area and contour length. The higher the compactness, the closer the contour is to a reasonably dense form:

(20)

A is the target area calculated by the Heaviside function H.

(21)

L is the contour length calculated by the Dirac function .

(22)

is a classic compactness measure, used to indicate whether the target consists of long, fragmented edges.

To describe the gap-filling tendency, we introduce a convexity measure based on the convex hull. First, we calculate the convex hull area of the probability map:

(23)

is the minimum convex hull cover area.

(24)

represents the degree of consistency between the area cover and the convex hull, and can be used to guide the closing of small gaps at credible boundary locations.

At the local structural level, we measure directional anisotropy using the structure tensor to characterize radial morphologies such as burrs and thin protrusions:

(25)

Where is a Gaussian kernel of scale , and the eigendecomposition of J gives the principal directional energy.

(26)

r is the anisotropy ratio, and are the eigenvalues of J. When r is large, it indicates the presence of thin radial structures, and the smoothing strength needs to be reduced to preserve the semantics of the burrs.

By integrating multiple morphological variables such as curvature, compactness, convexity, and anisotropy, we project the initial probability P onto a feasible set that satisfies the morphological prior using a projection scheme, resulting in a morphologically consistent soft mask :

(27)

Where represents the morphological projection operator, which internally performs curvature-guided smoothing on the isosurface in a small number of iterations, performs a weak closure along the convex hull’s trusted directions, and adaptively adjusts the balance between smoothing and detail preservation based on the spatial distribution of r.

At the implementation level, the above process follows a unidirectional information flow from semantics to geometry and back to probability. First, the multi-scale features generated by Backbone and the boundary confidences of the uncertainty-aware boundary modeling form a probabilistic representation of the region. This is then transformed into a continuous geometric domain via a distance transform. Within this domain, local smoothness is described using curvature, global density using area, length, and compactness, gap-filling propensity using convexity, and radial and burr directionality using a structure tensor. Finally, projection is performed back to the probabilistic domain to obtain . This design allows semantics and morphology to interact on a unified intermediate representation, where the uncertainty-aware boundary modeling weight field controls the regularization flow, maintaining tolerance near true blur boundaries while suppressing noise artifacts.

From another perspective, morphology-preserving regularization and uncertainty-aware boundary modeling complement each other: the uncertainty-aware boundary modeling provides probabilistic uncertainty modeling of boundaries at the pixel-to-probabilistic level, while the morphology-preserving regularization performs global shape uniformity and restoration at the geometric-to-regional level. By collaboratively utilizing , we introduce structural properties that balance connectivity, density, convexity, and directionality to the segmentation results without changing the backbone architecture. The resulting is both semantically informed and consistent with morphological priors, making it easy to perform subsequent thresholding or integrate with other modules.

3.4 Algorithm novelty and differences from other Transformer architecture segmentation algorithms

The proposed OncoSeg2D framework introduces a structurally novel design that diverges from existing Transformer-based segmentation models in three key aspects. First, instead of solely relying on self-attention for spatial feature refinement as in classical architectures such as TransUNet or Swin-Unet, OncoSeg2D explicitly integrates an uncertainty-aware boundary modeling mechanism that probabilistically characterizes edge ambiguity through learnable mean–variance estimation, thereby improving delineation of irregular tumor margins. Second, it incorporates a morphology-preserving regularization module that imposes geometric constraints derived from curvature, compactness, and convexity priors, ensuring global shape consistency that standard Transformer decoders fail to guarantee. Finally, the framework achieves a synergistic balance between pixel-level accuracy and topology-level robustness via joint optimization of uncertainty-aware boundary modeling and morphology-preserving regularization branches, enabling stable training and enhanced generalization across heterogeneous CT datasets. Collectively, these innovations differentiate OncoSeg2D from prior Transformer-based segmentation networks by embedding explicit structural reasoning and shape-aware constraints into the Transformer learning paradigm.

3.5 Dataset

The experimental datasets were collected from two publicly accessible sources. The first dataset was obtained from the Medical Segmentation Decathlon (MSD) lung task, which provides paired 3D CT volumes and expert annotations for lung lesions. We included patient cases that (i) contained complete CT volumes with corresponding voxel-wise lesion masks, and (ii) were successfully parsed without missing slices or corrupted metadata; cases with incomplete annotations or severe artifacts were excluded. The MSD lung task contains 96 cases in total, including 64 training cases and 32 official test cases; however, the 32 test cases are not publicly released with annotations, and thus we used only the 64 publicly available training cases in our experiments. Among these 64 training cases, 1 case (1.56%) was excluded due to a parsing failure, resulting in 63 eligible patient cases. All 63 eligible cases were used to avoid selection bias.

Each 3D CT volume was resampled to an isotropic spacing of 1.0 mm (linear interpolation for images and nearest-neighbor interpolation for masks), followed by HU clipping to [−1000, 400] to suppress air and high-density structures. The volumes were then sliced along the axial plane into 2D images. To ensure that 2D training focuses on clinically relevant regions while preserving local anatomical continuity, we retained slices containing visible lesions and additionally included a symmetric context window of K adjacent slices above and below each lesion slice. This strategy reduced the dominance of non-lesion slices and mitigated extreme class imbalance. In total, 8,640 2D slices were generated. Each slice was normalized before being fed into the 2D network. Slice-wise normalization was adopted to reduce inter-scan intensity shifts caused by heterogeneous acquisition protocols, which is common in CT-based 2D segmentation. To prevent data leakage, the dataset was split strictly at the patient level into training/validation/testing sets with a ratio of 7:1:2, ensuring that slices from the same patient appeared in only one split. Random flipping, rotation, and contrast perturbation were applied for augmentation.

The second dataset was collected from a Kaggle lung cancer segmentation dataset, which originally contained 2,535 2D CT images with corresponding expert-annotated lesion masks. Unlike MSD, this dataset is released directly as 2D samples rather than full 3D volumes; therefore, the total number of slices is lower even though the dataset aggregates images from multiple public sources. To reduce potential overlap between the two datasets and avoid evaluation bias, we further excluded those 2D slices that were visually identified as originating from MSD. This filtering process was performed manually, by inspecting and removing the relevant slices one by one. After removing these MSD-related samples, 1,933 2D CT images were retained in the final second dataset. Following the same approximate 7:1:2 split ratio, 1,353 images were used for training, 193 for validation, and 387 for testing. All images underwent the same intensity normalization and geometric/photometric augmentations as those used for MSD to maintain a consistent preprocessing pipeline.

Furthermore, we randomly selected representative examples from both datasets for visualization in Fig 4. This figure is provided solely to illustrate the overall data appearance and typical imaging characteristics of the datasets used in this study, so that readers—especially those unfamiliar with these datasets—can quickly understand the visual style and acquisition variability and more easily assess whether these two datasets are suitable for their own subsequent research or application scenarios.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Medical Segmentation Decathlon Challenge & lung cancer segmentation dataset examples.

https://doi.org/10.1371/journal.pone.0348719.g004

3.6 Experimental setup

In this study, we evaluated the OncoSeg2D framework using a publicly available lung cancer CT segmentation dataset. All experiments were conducted on the same hardware environment and maintained a consistent training configuration to ensure fairness. Training and inference were implemented in PyTorch, using the SegFormer MiT-B2 backbone network [8]. To ensure the reliability of the reported results, we adopted a five-fold cross-validation protocol, and the final performance was reported as the mean and standard deviation across the five folds. All hyperparameter/model selection was performed within the training and validation partitions of each fold only, and the corresponding held-out fold was used solely for final evaluation. We did not conduct an automated hyperparameter search; instead, we followed a controlled manual tuning protocol over a small set of candidate configurations. Due to the computational training cost, we did not perform large-scale or wide-range hyperparameter tuning, and focused on a limited but representative set of settings. All tuning runs followed the same five-fold cross-validation setting, and we trained models on the training partition while selecting configurations based only on validation performance within each fold. During tuning, the model architecture, preprocessing, and augmentation were kept unchanged, and we varied one factor at a time while keeping all other settings fixed. The final training configuration was chosen according to the best validation segmentation performance (primary criterion: IoU). Although OncoSeg2D exhibits slightly higher architectural complexity compared to certain lightweight frameworks such as SegMan, its integration of uncertainty-aware boundary modeling and morphology-preserving regularization provides distinct advantages in clinical scenarios that demand high boundary fidelity and structural interpretability. Specifically, the model demonstrates superior robustness when dealing with irregular, low-contrast lesions and offers more reliable shape-consistent predictions, which are essential for quantitative assessment and treatment planning. The specific model hyperparameters are shown in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Experimental model hyperparameter settings.

https://doi.org/10.1371/journal.pone.0348719.t002

3.7 Evaluation metric

In medical image segmentation, common evaluation metrics include Intersection over Union (IoU), mean IoU (mIoU), Hausdorff Distance (HD), Dice Similarity Coefficient (DSC), and mean Accuracy (mAcc). These metrics jointly assess the overlap, boundary accuracy, and classification consistency between predicted and ground-truth regions.

IoU quantifies the overlap between prediction P and ground truth G:

(28)

mIoU represents the mean of foreground and background IoUs:

(29)

HD measures the maximum boundary deviation:

(30)

In practice, we report the Hausdorff distance computed using the 95^th percentile of all point-to-boundary distances to reduce sensitivity to outliers.

DSC, equivalent to the F1-score, evaluates region overlap:

(31)

mAcc denotes average pixel-level accuracy:

(32)

In summary, IoU and DSC focus on overlap quality, HD on boundary precision, and mIoU and mAcc on overall classification balance, providing a comprehensive assessment of segmentation performance.

4.1 Experimental results of evaluation indicators changing with epoch

First, this paper presents the evaluation metrics and loss changes across epochs during training on the two datasets, as shown in Fig 5. The training process was conducted for the full scheduled number of epochs without employing an early stopping strategy. The final model was selected by choosing the epoch that achieved the highest validation IoU after convergence.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Evaluation indicators and loss curves on two datasets.

(a) Evaluation indicators and loss curves on the Medical Segmentation Decathlon dataset. In this study, the Hausdorff Distance was computed on the validation set using the 95th percentile Hausdorff Distance metric. (b) Evaluation indicators and loss curves on the lung cancer segmentation dataset. In this study, the Hausdorff Distance was computed on the validation set using the 95th percentile Hausdorff Distance metric.

https://doi.org/10.1371/journal.pone.0348719.g005

Experimental results show that both training and validation losses for both datasets show a rapid decline with increasing iterations, ultimately reaching a plateau, demonstrating good model convergence during training. Corresponding evaluation metrics (such as IoU, DSC, and mAcc) continue to rise and remain at high levels, demonstrating the effective improvement in the model’s segmentation performance. Furthermore, the Hausdorff distance gradually decreases, further verifying the model’s refined improvements and stability in boundary localization.

4.2 Comparison of experimental results with other models

To comprehensively evaluate the effectiveness of our proposed method, we compared OncoSeg2D with several representative segmentation models using the same dataset and training configuration. These included a classic convolutional neural network architecture (UNet++ [36]), a multi-scale feature aggregation model (DeepLabv3+ [37], nnU-net [38]), Transformer-based methods (Mask2Former [39], SegFormer [8], SegNeXt [40], VWFormer [41], and EDAFormer [42]), and a recently proposed hybrid architecture (SegMan [43]). These comparisons covered a diverse range of designs, from traditional CNNs to novel Transformers and hybrid architectures. This helped validate the competitiveness and applicability of our method across various mainstream frameworks.

We first present experimental results on the Medical Segmentation Decathlon Challenge dataset, as shown in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Experimental results on the Medical Segmentation Decathlon Challenge dataset (mean ± standard deviation across five-fold cross-validation).

https://doi.org/10.1371/journal.pone.0348719.t003

As shown in Table 3, the proposed OncoSeg2D outperforms existing mainstream models across all evaluation metrics. Compared to classic CNN architectures (such as UNet++ and DeepLabv3+), this method significantly improves IoU, mIoU, and DSC, while significantly reducing HD. This demonstrates that OncoSeg2D is able to better model boundary locations while maintaining overall segmentation accuracy. This suggests that relying solely on convolutional architectures fails to fully capture the complex boundary characteristics of lung lesions. The proposed uncertainty-aware boundary modeling plays a key role in addressing boundary ambiguity and annotation uncertainty.

Further comparison with Transformer-based methods (Mask2Former, SegFormer, SegNeXt, VWFormer, EDAFormer, and SegMan) reveals that, in addition to its global modeling capabilities, OncoSeg2D significantly improves the structural consistency and morphological plausibility of its segmentation results by introducing morphology-preserving regularization. MPR’s contributions to geometric constraints and topology preservation enable the model to achieve superior numerical performance while also effectively avoiding undesirable morphologies such as fragmentation or gaps in the generated results. Consequently, this method demonstrates enhanced robustness and clinical application potential in addressing the two core challenges of blurred boundaries and complex morphology in lung cancer CT images.

The experimental results of lung cancer segmentation Segmentation dataset are further given, and the experimental results are shown in Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Experimental results on the lung cancer segmentation dataset (mean ± standard deviation across five-fold cross-validation).

https://doi.org/10.1371/journal.pone.0348719.t004

As shown in Table 4, the proposed OncoSeg2D also achieved leading performance in various indicators on the Kaggle lung cancer segmentation dataset. In particular, it outperformed mainstream CNN and Transformer architectures in IoU, mIoU, and DSC, while maintaining the lowest value for the HD metric. This shows that this method not only has advantages in overall segmentation accuracy, but also exhibits stronger robustness and generalization capabilities in boundary modeling and morphology preservation, effectively addressing the complex changes of lung nodules and lesions in different clinical scenarios.

4.3 Ablation experiment results

This paper also presents ablation results, which an ablation analysis is performed on the baseline used in this paper. First, the module ablation results are presented, as shown in Table 5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Ablation study results on the Medical Segmentation Decathlon Challenge dataset and the lung cancer segmentation dataset (mean ± standard deviation across five-fold cross-validation).

https://doi.org/10.1371/journal.pone.0348719.t005

Ablation experiments show that introducing both the uncertainty-aware boundary modeling and morphology-preserving regularization modules significantly improves segmentation performance on two different datasets. For example, compared to the baseline model SegFormer, introducing either the uncertainty-aware boundary modeling or morphology-preserving regularization alone improves metrics such as IoU, mIoU, and DSC, while reducing HD, demonstrating the positive effects of both modules on feature extraction and boundary modeling. The performance gain of uncertainty-aware boundary modeling is more pronounced on boundary-sensitive metrics, reflecting its capability to refine ambiguous edge regions, whereas morphology-preserving regularization mainly enhances the global shape regularity and segmentation consistency.

Further examining the results for Setting 3, we find that this combination achieves optimal performance across all metrics: IoU, mIoU, and DSC reach the highest values, while HD is minimized, demonstrating strong complementarity between the two modules. This synergy indicates that uncertainty-aware boundary modeling focuses on local uncertainty refinement while morphology-preserving regularization constrains global morphology, resulting in a balanced improvement across precision and stability. Experiments on the lung cancer segmentation Dataset also confirm this conclusion. While the overall performance is lower than on the previous dataset, the combined uncertainty-aware boundary modeling and morphology-preserving regularization still significantly outperform both the individual modules and the baseline model. This fully demonstrates the robustness and effectiveness of the proposed method in different datasets and task scenarios.

4.4 Qualitative analysis of experimental results

In this section, we present the results of the segmentation experiment and the Grad-Cam experiment, also conducted on two datasets.

4.4.1 Qualitative experimental results on the Medical Segmentation Decathlon Challenge dataset.

First, we give the experimental results of the Medical Segmentation Decathlon Challenge dataset, as shown in Fig 6.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Medical Segmentation Decathlon Challenge dataset segmentation results.

https://doi.org/10.1371/journal.pone.0348719.g006

Experimental visualization results show that our proposed method demonstrates superior region coverage and boundary consistency across multiple case segmentation tasks. Compared to Mask2Former, Unet++, and Deeplabv3 + , our model achieves more complete coverage of lesion regions, avoiding the under-segmentation and over-segmentation observed in other methods. Furthermore, our method achieves closer alignment with the ground truth for boundary details, reducing jagged or irregular prediction errors, demonstrating its superior morphology preservation and edge refinement capabilities.

Further comparison of magnified regions reveals that while other models tend to miss or inaccurately segment small lesions or regions with blurred boundaries, our method more consistently captures this fine-grained information, ensuring smoother and more consistent segmentation results. This improvement not only enhances visual quality but also provides more reliable structural information for subsequent clinical analysis, demonstrating the practical value of our proposed model in real-world applications. Furthermore, this paper gives the Grad-Cam experimental results.

Fig 7 show that the model achieves relatively accurate segmentation predictions on the Medical Segmentation Decathlon Challenge dataset, effectively covering lesion areas. Grad-Cam visualizations show distinct hotspots of interest, which are highly consistent with the distribution of the predicted mask. This demonstrates that the model not only generates reliable segmentation results at the output layer but also effectively focuses on lesion-related areas at the feature level, demonstrating strong interpretability and robustness, thereby enhancing its credibility for practical clinical applications.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Medical Segmentation Decathlon Challenge dataset Grad-Cam experimental results.

https://doi.org/10.1371/journal.pone.0348719.g007

4.4.2 Qualitative experimental results on the lung cancer segmentation dataset.

Similarly, this paper also gives the segmentation results of the lung cancer segmentation dataset and the Grad-Cam experimental results. First, the segmentation results are given, as shown in Fig 8.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Segmentation results of the lung cancer segmentation dataset.

https://doi.org/10.1371/journal.pone.0348719.g008

The results in Fig 8 demonstrate that our method demonstrates strong robustness and accuracy in the task of lung cancer segmentation. A comparison with Ground-Truth shows that the model effectively captures the boundaries and morphology of lesion regions, maintaining high consistency, especially in the presence of blurred boundaries and low contrast. This demonstrates the model’s strong discriminative ability in lesion region identification and localization, mitigating overexpansion and omission issues.

Compared to other comparison methods, Mask2Former, Unet++, and Deeplabv3 + , our method demonstrates clearer lesion outlines and closer regional alignment. While Mask2Former and Unet++ perform reasonably well in gross segmentation, they are prone to incomplete edges and overprediction, while Deeplabv3 + ’s results tend to be more fuzzy. Overall, the proposed method strikes a balance between regional integrity and boundary accuracy, making it suitable for clinically assisted segmentation of lung CT nodules/tumors.

Finally, Grad-Cam is also used for analysis, and the experimental results are shown in Fig 9.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Lung cancer segmentation dataset Grad-Cam experimental results.

https://doi.org/10.1371/journal.pone.0348719.g009

From the experimental results, it can be observed that Grad-Cam effectively highlights the critical regions of interest in lung CT images, particularly forming distinct high-intensity heatmap distributions around nodules or lesions, which are largely consistent with both the predicted results and the actual lesion areas. This indicates that the model demonstrates good interpretability in identifying lesion locations, with its decision-making focused on clinically relevant anatomical structures rather than irrelevant regions. At the same time, although certain samples still exhibit some blurring or incomplete coverage, the overall consistency between the Grad-Cam visualizations and the predictions provides strong support for the reliability of the model and its interpretability in clinical auxiliary diagnosis.

4.4.3 Learning rate hyperparameter sensitivity experiment.

This paper also gives the experimental results of the learning rate hyperparameter sensitivity on two datasets, and the experimental results are shown in Table 6.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Effect of learning rate on model performance across two datasets.

https://doi.org/10.1371/journal.pone.0348719.t006

The results presented in Table 6 demonstrate that the proposed OncoSeg2D framework maintains stable and consistent performance across different learning rates on both datasets. As the learning rate decreases from 1e-3 to 1e-4, the model shows a steady improvement in IoU, mIoU, and DSC, while the Hausdorff Distance continuously declines, indicating enhanced boundary precision and convergence stability. The optimal performance at 1e-4 learning rate on both the Medical Segmentation Decathlon and lung cancer segmentation datasets confirms the robustness of the proposed optimization strategy, suggesting that the model achieves a balanced trade-off between convergence speed and generalization ability under varying training conditions.

4.5 Restriction analysis

This paper also provides verification of the model’s generalization performance. Specifically, for two datasets, training is performed on one dataset and testing is performed on the other dataset. The experimental results are shown in Table 7.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Cross-dataset transfer evaluation: lung cancer segmentation → Medical Segmentation Decathlon (top) and Medical Segmentation Decathlon → lung cancer segmentation (bottom).

https://doi.org/10.1371/journal.pone.0348719.t007

From the cross-domain transfer results in Table 7, it is apparent that all models suffer a marked performance drop under out-of-domain evaluation, indicating that cross-dataset generalization in this task remains highly challenging. For example, in the Lung CT → MSD direction, SegFormer achieves an IoU of only 0.201, SegMan 0.247, and EDAFormer 0.228, while our model—although still performing the best comparatively—reaches only 0.276. Meanwhile, the HD values increase substantially, for example, SegFormer reaches an HD of 32.8, indicating a clear increase in boundary deviation and hard-to-segment outlier regions. The reverse transfer direction (MSD → Lung CT) exhibits a similar overall degradation, although the absolute metric values differ slightly. Notably, in both transfer directions, OncoSeg2D consistently maintains comparatively superior performance with lower HD values, suggesting that the proposed morphology-preserving regularization and uncertainty-aware modeling do contribute to robustness under cross-domain scenarios. Nevertheless, the absolute performance decline remains obvious, which also highlights an important limitation of the current framework for real-world deployment.

This limitation is not solely attributable to the model itself, but is mainly associated with the substantial domain gap between the two public datasets. Potential sources of this discrepancy include differences in imaging acquisition protocols, such as slice thickness, reconstruction kernels, and sampling intervals; variations in image intensity characteristics and preprocessing settings; differences in noise patterns caused by scanners, hospitals, and acquisition equipment; inconsistencies in annotation protocols and labeling criteria across datasets; and heterogeneity in lesion type, morphology, and size distribution. These factors jointly make it difficult for representations learned from the source domain to transfer effectively to the target domain. Therefore, the observed performance drop should be understood as a consequence of cross-dataset distribution mismatch, rather than as evidence of failure in the core model design.