2.1 Semantic segmentation

Semantic segmentation aims to provide a comprehensive understanding of visual scenes and precise object localization. Early approaches, such as Fully Convolutional Networks (FCNs) [12], pioneered the development of end-to-end networks tailored specifically for semantic segmentation tasks. These networks utilized dense forward computation and backpropagation to generate outputs that match the input image size, effectively allowing for pixel-wise classification. By integrating semantic information from various convolutional layers, FCNs improved the accuracy of semantic segmentation networks significantly.

As the field evolved, more sophisticated architectures emerged. Encoder-decoder frameworks and dilated convolution techniques, exemplified by models like U-Net [13], SegNet [14], and the DeepLab series [15], made substantial strides in enhancing segmentation precision. These models not only excelled in detail restoration but also improved contextual understanding, which is crucial for accurately identifying and delineating objects within complex scenes. To address the inherent challenges posed by multi-scale targets—where objects may appear at various sizes—Pyramid Pooling Modules (PPM) [16] and Atrous Spatial Pyramid Pooling (ASPP) [17] were introduced. These techniques significantly bolster the extraction of global contextual features, enabling models to recognize and segment objects more effectively, regardless of their scale.

Simultaneously, attention mechanisms gained traction, including spatial attention and both position and channel attention modules [18–20]. These mechanisms are instrumental in enhancing the aggregation of long-range contextual information, thereby enriching the model’s ability to represent semantic information more robustly.

Despite these advancements, traditional semantic segmentation methods encounter significant challenges, particularly in scenarios where annotated data is scarce or when there is a need to adapt to new categories that were not included in the training dataset. This dependency on large-scale annotated datasets constrains the scalability and generalization capabilities of these models, prompting researchers to explore alternative strategies such as Few-Shot Segmentation (FSS). FSS techniques aim to bridge this gap by enabling effective segmentation with minimal labeled examples, thus broadening the applicability of semantic segmentation in real-world scenarios.

2.2 Few-shot segmentation

Few-shot segmentation overcomes the limitations of traditional methods by enabling segmentation of objects using a minimal number of annotated examples for each class. The primary goal is to generalise from a small support set (labelled examples) to accurately classify objects within a query image.

Recent years have seen substantial advancements in prototype-based semantic segmentation for few-shot images [21,22].The SG-One network [23] optimizes feature maps and segmentation loss by utilizing two branches of shared convolutional layers. It employs Masked Average Pooling to extract prototype features from support images, measuring the prototype distance between support and query images using cosine similarity. The PANet [24] framework leverages prototype-based metric learning and incorporates prototype alignment regularization to maximize the information derived from support images. ASGNet [25] introduces a superpixel-guided clustering approach, which extracts multiple prototypes from support images and reconstructs their feature maps through a novel assignment strategy, thereby enhancing semantic extraction and representation. PFENet [26] aggregates multi-scale information from input samples to capture global feature information, significantly improving the model’s predictive capability. To address information loss issues, HSNet [27] focuses on feature interrelationships, converting dense feature-related tensors into segmentation results through high-dimensional convolutions. Additionally, advancements in attention mechanisms have further improved few-shot segmentation models. DENet [28] introduces a novel attention module that enhances algorithm generalization by adjusting the weights of the metric classifier. MCE [39] enhances few-shot segmentation by capturing shared visual properties and learning inter-image dependencies, improving pixel-wise labeling of unseen classes with limited annotated images. BAM [29] optimizes prediction results through a fully supervised semantic segmentation model, discarding erroneous predictions and proposing a new model based on base classes. The methods mentioned above primarily rely on prototype-based approaches, where features are extracted and prototypes are generated using Masked Average Pooling (MAP). However, the averaging operation in MAP fails to preserve the details and diversity information in the support set images, leading to the loss of fine spatial and structural information. As a result, the generated prototypes lack sufficient discriminative power. These methods often lead to a reduction in segmentation accuracy when dealing with complex and highly variable objects, particularly in cases where there are significant changes in object scale and appearance, causing the segmentation results to lack fine-grained details.

To address the limitations of current Few-Shot Segmentation (FSS) methods, we propose the Multi-Scale Prototype Convolutional Network (MPCN). The MPCN enhances feature interaction between the support set and query images through the PMG module, utilizing dynamic kernels at multiple scales to capture fine details and large-scale object features, thereby improving segmentation accuracy and robustness. Subsequently, the MPE module is employed to strengthen the support set features, apply downsampling to generate multi-scale features, and refine prototypes to achieve better query segmentation, thus overcoming the limitations of Masked Average Pooling (MAP).

3.1 Few-shot segmentation

The task of few-shot segmentation aims to address image segmentation challenges under conditions of limited annotated data. In a typical few-shot semantic segmentation dataset , the data is divided into training samples and test samples , with the classes in and being mutually exclusive, i.e., . During the training phase, the few-shot semantic segmentation model is trained on a large set of annotated training samples , under the guidance of the ground truth labels for query images. The goal is to construct a segmentation model f ( S , Q )  that can utilize K annotated images and their corresponding support set masks to perform semantic segmentation on query images , producing predictions that approximate the ground truth (where S and Q represent the support and query sets, respectively). Consequently, during the inference phase, the model is able to rapidly generalize to new class data using the limited annotated data, thereby alleviating the dependency on extensive labeled datasets.

3.2 Overview

As illustrated in Fig 1, our Multi-Scale Prototype Convolutional Network comprises two key modules: the Prior Mask Generation (PMG) module and the Multi-Scale Prototype Extraction (MPE) module. Specifically, given the support and query images, and , we employ a shared-weight backbone to extract mid-level and high-level features. The PMG module subsequently generates an initial mask for the target object in the query image and applies foreground filtering to the query set features. Subsequently, the MPE module extracts support prototypes and generates similarity map by matching these prototypes with the filtered query features. This process produces category-aware prototypes with contextual dependencies and rich semantics. During the decoder stage, we fuse the category-aware features, query activation maps, and intermediate query features to generate a refined query pseudo-mask. The technical specifics of each module will be addressed in the subsequent sections.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The overall architecture of our proposed Multi-Scale Prototype Convolutional Network (MPCN) for few-shot semantic segmentation is illustrated.

Initially, high-level support and query features are fed into the Prior Mask Generation (PMG) module to generate the initial mask for the query image , which is then used to perform foreground filtering on intermediate query set features. Subsequently, the Multi-Scale Prototype Extraction (MPE) module extracts support prototypes and matches these prototypes with the filtered query features to generate a similarity map and category-aware prototypes . Finally, category-aware features, query activation maps, and intermediate query features are fused to produce a refined query pseudo-mask.

https://doi.org/10.1371/journal.pone.0319905.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The specific structure diagram of Prior Mask Generation module.

https://doi.org/10.1371/journal.pone.0319905.g002

3.3 Prior mask generation module

To align the features of the support set and query set, and to guide the segmentation model to focus on the most relevant regions in the query image, we introduce the PMG module. Specifically, we first use the high-level features from both the support set and the query set to generate a prior mask [8]. This prior mask, typically a matching between feature maps, represents an approximate location of the target object without considering the overall contextual relationships. To address this, we employ three sliding windows to achieve global and regional matching, as shown in Fig 2. Specifically, we utilize the high-level features from the support set and query set, denoted as and , alongside binary masks as inputs. Here, C represents the channel dimension, while and denote the height and width of the support and query features, respectively. We then extract the regional features and using the designed sliding windows:

(1)

(2)

where  ⊗  denotes the Hadamard product, W represents the sliding window operation, and U , V are the height and width of the windows. We use three multi-scale windows, specifically (1,1), (3,3), and (5,5), to capture features corresponding to small, medium, and large objects. We then compute the cross-correlation features based on similarity of and :

(3)

where Ein denotes the Einstein summation convention [9], icj represents the index of , and ick represents the index of ; the final cross-correlation feature has the index ijk, which indicates summation over index c. Finally, we average and normalize the cross-correlation features to obtain the activation maps for the target object masks. Given that we use three different windows to obtain , , and , we average these to produce , which represents the approximate location of the target object in the query image.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The specific structure diagram of Multi-Scale Prototype Extraction module.

https://doi.org/10.1371/journal.pone.0319905.g003

3.4 Multi-scale prototype extraction module

The initial mask from the previous section only provides an approximate location of the target object. To capture more details, we construct a Multi-Scale Prototype Extraction (MPE) module to generate sufficiently informative and scale-sensitive support prototypes, and use a similarity spectrum module to filter out background from the query features, retaining the foreground to produce the final query class semantic feature . Specifically, to mitigate interference from irrelevant classes, we perform a pixel-wise multiplication between the initial mask and the query image features to obtain . Similarly, we multiply the support mask with the support image features to obtain . The MS-CPE module then extracts rich semantic information from , as shown in Fig 3.

First, we apply average pooling to , followed by convolutional layers and activation functions to obtain the semantic-enhanced features :

(4)

where denotes the support features after average pooling, represents the convolutional layers and a rule layer, and σ is the Sigmoid function. To address the challenge of scale diversity within class samples, we obtain multi-scale feature sets by applying different down-sampling factors r ∈ { 1 , 2 , 4 } . Next, using , we retain the foreground features in and flatten them into a set of feature vectors:

(5)

where  ( x , y )  denotes spatial positions, I is the indicator function that outputs 1 if the input condition is met and 0 otherwise, and represents the set of foreground feature vectors derived from the support image. Unlike methods that use direct mask average pooling, we use average pooling to extract general features, maximum pooling to retain significant features, and average the two pooled vectors to obtain the initial support prototype :

(6)

where Avg and Max denote average pooling and maximum pooling, respectively. effectively summarizes the class semantic information. To achieve pixel-level predictions, we use a prototype matching model to construct a non-parametric similarity measure across spatial locations.

First, we calculate the cosine similarity between the feature vector at each spatial location and the support prototype . The cosine similarity is used as the exponent in an exponential function to obtain the similarity spectrum S:

(7)

where exp ⁡   and cos ⁡  -simi denote the exponential function and cosine similarity, respectively;  ( x , y )  represents the spatial location; denotes the feature vector of at the location  ( x , y ) ; and S ( x , y )  represents the similarity score between and . We use the similarity spectrum S to reweight the query image features , enhancing the foreground regions while suppressing the background, to obtain the category-aware feature :

(8)

Finally, the category-aware feature , the query activation maps , and the intermediate query features are reshaped to the same spatial dimensions and fused together to produce the output feature :

(9)

where denotes concatenation along the channel dimension. The final output is then input to the decoder to generate the segmentation mask for the query image :

(10)

where , , and are the consecutive modules constituting the decoder.

3.5 Training loss

Inspired by previous work [8,30,31], we employ the Binary Cross-Entropy (BCE) loss between the predicted mask of the query image and the ground truth mask as the primary loss function for our model:

(11)

Additionally, to ensure that the multi-scale contextual prototype extraction module generates more accurate support prototypes, we introduce an auxiliary loss . This auxiliary loss is computed by predicting the support mask using the corresponding query prediction mask , with the generation of being similar to the method described in Equation (10):

(12)

Here, represents the BCE loss between and . Thus, the final loss function is:

(13)

where λ is a balancing factor between the segmentation loss and the auxiliary loss . In our experiments, λ is set to 1.0.

4.1 Dataset

PASCAL- [32] is a few-shot segmentation variant derived from the PASCAL VOC dataset [33], designed to evaluate the performance of few-shot learning methods. It divides the 20 object categories covered by the PASCAL dataset into 4 subsets, with each subset containing 5 classes for testing and the remaining 15 classes for training.

COCO- [34] is a variant of the COCO dataset [35] specifically designed for few-shot segmentation tasks. It partitions the 80 object categories in the COCO dataset into 4 subsets, with each subset containing 20 classes for testing and the remaining 60 classes for training.

4.2 Experimental settings

Metrics and Evaluation. Consistent with previous work, we utilize mean intersection over union (mIoU) and foreground-background IoU (FB-IoU) as our evaluation metrics. mIoU is a widely accepted standard for assessing segmentation performance, as it computes the ratio of the intersection to the union of predicted and ground truth regions for each category, averaging the results across all categories to gauge overall performance.

(14)

where N is the number of categories, is the ground truth for category i, and is the prediction for category i. Moreover, FB-IoU offers a comprehensive assessment of the model’s segmentation capabilities by calculating the IoU between the predicted foreground and background regions and their corresponding ground truth segments. This metric provides a more precise evaluation of the model’s performance in managing complex scenes.

(15)

where and represent the foreground and background in the ground truth, respectively, and and are the corresponding predictions for the foreground and background.

Implementation Details. In our few-shot segmentation experiments, we selected VGG-16 [36] and ResNet-50 [37] as backbone networks, both pretrained on the ImageNet classification task, with their weights fixed during training. The implementation was carried out using PyTorch 1.9.0, and experiments were executed on an NVIDIA RTX 3090 GPU. All images were cropped to a size of 473 × 473 pixels for training. We utilized stochastic gradient descent (SGD) for optimization, setting the initial learning rate to 0.005, a batch size of 8, weight decay to 0.0001, and momentum to 0.9. Within the PMG module, sliding window sizes were configured to (1,1), (3,3), and (5,5). For the MPE module, downsampling factors r ∈ { 1 , 2 , 4 }  were employed. Training was conducted for 200 epochs on the PASCAL-5i dataset and for 50 epochs on the COCO-20i dataset.

4.3 Comparison with state-of-the-art methods

PASCAL-: Table 1 provides a comparison of our model, MPCN, with various state-of-the-art methods on the PASCAL- dataset. Using the VGG16 backbone, MPCN surpasses most of the evaluated methods, attaining 63.2% mIoU and 75.6% FB-IoU in the 1-shot setting, and 68.6% mIoU and 79.1% FB-IoU in the 5-shot setting. Likewise, with the ResNet50 backbone, MPCN exhibits notable improvements. When compared to the most recent methods, SiGCN and MCE, MPCN achieves the highest mIoU and FB-IoU scores with the ResNet50 backbone. In the 1-shot setting, MPCN enhances mIoU by 2.3% compared to SiGCN and by 1.7% relative to MCE. In the 5-shot setting, MPCN increases mIoU by 2.6% compared to SiGCN and by 1.1% compared to MCE. Notably, MPCN also showed the highest foreground-to-background intersection ratio (FB-IoU) value, further emphasizing its effectiveness in distinguishing between foreground and background.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The performance of the metrics on PASCAL- was evaluated using mean Intersection over Union (mIoU) (%) and Foreground-Background Intersection over Union (FB-IoU) (%). Results are presented with bold text indicating the best performance. The baseline results represent performance achieved without incorporating any additional modules (e.g., PMG and MPE).

https://doi.org/10.1371/journal.pone.0319905.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The performance of the metrics on COCO- was evaluated using mean Intersection over Union (mIoU) (%) and Foreground-Background Intersection over Union (FB-IoU) (%). Results are presented with bold text indicating the best performance. The baseline results represent performance achieved without incorporating any additional modules (e.g., PMG and MPE).

https://doi.org/10.1371/journal.pone.0319905.t002

COCO-: COCO- presents a more challenging environment, featuring a greater number of categories and more complex scenes. Table 2 illustrates the performance of MPCN in comparison to other methods on the COCO- dataset. Notably, MPCN excels in both the 1-shot and 5-shot settings. Using the VGG16 backbone, MPCN achieves an average Intersection over Union (mIoU) of 42.4% in the 1-shot setting and 47.8% in the 5-shot setting, significantly outperforming other methods such as DPCN and BAM. When using the ResNet50 backbone, MPCN’s performance is even more pronounced. In the 1-shot setting, MPCN reaches an average mIoU of 45.1%, outperforming SiGCN and MCE by 3.7% and 0.9%, respectively. In the 5-shot setting, MPCN achieves an average mIoU of 52.3%, exceeding SiGCN and MCE by 4.3% and 1.3%, respectively. Additionally, MPCN excels in FB-IoU, with scores of 69.4% and 71.2% in the corresponding settings.

The Multi-Scale Prototype Convolutional Network (MPCN) addresses intra-class variability and prototype representation issues through the Prior Mask Generation (PMG) and Multi-Scale Prototype Extraction (MPE) modules, achieving significant improvements in segmentation performance on the PASCAL- and COCO- datasets. However, the use of multiple dynamic kernels with varying sliding window sizes and multi-scale feature extraction may lead to a substantial increase in computational cost and memory usage.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Ablation studies of main model components.

https://doi.org/10.1371/journal.pone.0319905.t003

4.4 Ablations

To assess the impact of various modules on the model’s performance, we conducted an ablation study utilizing the ResNet-50 backbone in the 1-shot setting on the PASCAL- dataset. As presented in Table 3, the baseline model, which excludes the two modules (configuration (a)), achieves an mIoU of 63.4% and an FB-IoU of 74.6%. Introducing only the PMG module (configuration (b)) increases the average mIoU to 65.1% and the average FB-IoU to 76.3%. Adding only the MPE module (configuration (c)) further improves the average mIoU to 66.0% and the FB-IoU to 77.4%. The most notable performance enhancement occurs when both the PMG and MPE modules are utilized together (configuration (d)), resulting in a 4.2% increase in mIoU and a 4.1% rise in FB-IoU compared to the baseline model. The ablation study results indicate that both the PMG and MPE modules significantly enhance the performance of the MPCN model. While each module individually contributes to improving the mIoU and FB-IoU metrics, their combined application yields the highest performance.

4.5 Impact of hyperparameters and comparison with other refinementmethods

In this subsection, we provide a detailed explanation of the selection of key hyperparameters, as well as a comparison with other refinement methods to highlight the advantages of the MPE module.

Fig 4a investigates the impact of different sliding window sizes on model performance. A1 represents the use of a sliding window of size (3,3), A2 represents a sliding window of size (5,5), A3 represents a sliding window of size (7,7), and A4 corresponds to the original configuration with sliding windows of sizes (1,1), (3,3), and (5,5). As shown in Fig 4a, the A4 configuration achieves the optimal result, demonstrating that the multi-scale sliding window setting significantly enhances the model’s performance.

Fig 4b explores the impact of different downsampling factor combinations on model performance. In Fig 4b, B1 represents the use of downsampling factors [1,2,4], B2 uses [1,4,4], B3 uses [1,4,8], and B4 employs [1,8,8]. As observed from Fig 4b, the B1 configuration yields the best performance, indicating that this downsampling factor setting allows the model to effectively capture the key features of the data, thus improving performance.

Additionally, Fig 4c shows a comparison between MPE and other refinement methods. Here, DCM [8] represents dynamic convolution techniques, SGC [46] refers to superpixel-guided clustering, and PAM [47] corresponds to prototype activation methods. As seen in Fig 4c, our method achieves the best mIoU results. Compared to dynamic and weighted adaptive clustering techniques such as DCM, SGC, and PAM, the MPE module significantly improves segmentation accuracy. By leveraging dynamic kernels for multi-scale feature extraction, our approach not only enhances feature alignment but also adapts more effectively to variations in object size and structure.

4.6 Qualitative results

Fig 5 illustrates the segmentation results of MPCN in comparison to the baseline model on the PASCAL- and COCO- datasets. As depicted, MPCN demonstrates greater accuracy in segmenting target objects and captures finer details more effectively than the baseline model. Specifically, in columns 1, 3, 4, and 6 of Fig 5, our proposed MPCN accurately segments the target objects, whereas the baseline model incorrectly includes irrelevant background elements in the segmentation.

As shown in Fig 6 we Visualization of different ablative results on the PASCAL- and COCO- datasets and compare the comparative results of the other methods. It can be seen that by adding PMG and MPG modules to the baseline model, the network can filter out irrelevant background regions and discover more target parts. For example, adding the fifth line of MPG removes the redundancy of the segmentation of birds and cars compared to the base prediction. In addition, adding the MPG module results in a more refined segmentation. For example, compared to the base prediction, the sixth row can filter out a large number of irrelevant background regions and localize the target object accurately. Finally, we compare our method with MCE, and as shown in Fig 6, our approach accurately segments the bird and the car. In contrast, MCE erroneously includes the nearby motorcycle in its vehicle segmentation and fails to fully capture the car as a whole.