3.2.1 Path-searching method.

In SAM 2, the mask with the highest predicted IoU is selected. While this greedy strategy may increase the risk of errors when processing challenging frames to cause error accumulation.

To mitigate this problem, SAM2Long [3] enhances SAM 2 by integrating multiple memory pathways, as illustrated in Fig 2(A). This design enables the storage of diverse masks as memory at each time step, thereby improving the accuracy of predictions for subsequent frames. Within each pathway, an object-aware memory bank is constructed, this bank selectively incorporates the frames with confidently detected objects and high-quality segmentation masks. The specific mechanism of this process is formally described in Eq (1).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Existing methods based on SAM2 for video object segmentation.

(A) Path-searching method. In which the multiple memory pathways are searched to include the confidently detected objects. (B) Motion-estimation method. Where the objects motion is modeled to enhance the score system. (C) Memory-adaptive method. In which the enhanced memory mechanisms are designed for object information consistency.

https://doi.org/10.1371/journal.pone.0353156.g002

(1)

where ε is a small constant to prevent the logarithm of zero, denotes the features of t-th frame, is the mask index from SAM2 output three masks, represents the memory features encoded to the memory bank, and N = 7, and is the indicator function, when the condition is met, it equals to 1, otherwise 0. In each memory pathway, an object-aware memory selection mechanism is designed to retrieve frames containing discriminative objects as described in the second formula in Eq (1). Concurrently, the calculation of memory attention is modulated to further enhance the model's focus on target objects, which is depicted in the third formula in Eq (1).

3.2.2 Motion-estimation method.

The fixed-window memory mechanism in the SAM 2 model fails to account for the quality of memories. Specifically, objects with low confidence may be incorrectly encoded by the model, which may lead to error propagation in long video sequence. To tackle this problem, the SAMURAI [4] introduces two key enhancements, as illustrated in Fig 2(B). First, a Kalman Filter (KF)-based motion modeling module is applied to estimate the object motion, and second, an improved memory selection strategy is designed, which fuses both affinity and motion scores. These modifications are explicitly devised to boost the model’s capability to track objects accurately in complex video scenarios. The mathematical formulation of this process is provided in Eq (2).

(2)

In which, z denotes the state of Kalman filter, which is consist of x, y, the top-left coordinate of the bounding box, a denotes aspect ratio, height h, and their corresponding velocity , A is the linear state transition matrix, K represents the Kalman gain, and H is the observation matrix. Furthermore, to ensure the robustness of the motion modeling after the targeted object reappears or the poor mask qualities for a certain period of time, a stable motion state is maintained with confident threshold, as describe in the last formula in Eq (2).

3.2.3 Memory-adaptive method.

When the tracking object is surrounded by similar objects in crowd scene, current VOS methods still face significant challenges. To address this issue, DAM4SAM [5] proposes a novel distractor-aware memory (DAM) module for the SAM2 model, which composes the recent appearance memory (RAM) and distractor resolving memory (DRM) , as described as Eq (3).

(3)

As formulated in Eq (3), the memory is updated every ∆ = 5 frames and always incorporates the most recent frame containing object . For DRM, a bounding box is fitted to al the masks, and if the ratio between the area of the two bounding boxes drops below 0.7, the current frame is considered as a potential candidate for updating DRM. In DAM4SAM, the DRM is updated during periods with sufficiently stable tracking, as depicted in the last formula in Eq (3). The criteria consist of the predicted mask with a threshold and the mask area with . And the denotes the median mask area in last frames.

3.3.1 Appearance uncertainty prediction.

In SAM 2, the memory values associated with the query frame are computed based on the similarity between the query and memory keys, with the underlying assumption that all query tokens carry equal importance. Usually, compared with frame t-1, only a small subset of tokens in current frame t exhibit significant changes. To address challenges posed by appearance variations, query tokens are weighted by the appearance uncertainty to enhance the model’s stability.

Specifically, the appearance uncertainty prediction module is introduced to estimate the uncertainty for each query token, enabling the adaptive memory and image feature fusion. The prediction module is designed to be lightweight, comprising merely three convolution layers. We formulate the prediction task as a segmentation problem, optimized using a segmentation loss . The supervision label is derived from the mask differences between frame t − 1 and frame t, and the sigmoid function is applied to transform the prediction as a probability. The final features are obtained via a soft merge of memory readout and last features, formally defined in Eq (4):

(4)

Where represents the predicted appearance uncertainty, is the current features queried from memory bank via memory attention, and denote the features from the last iteration. And the is calculated at each pixel as a function of the current frame image feature and retrieved feature using Eq (5):

(5)

In which, denotes the Sigmoid activation function to constrain the uncertainty between 0 and 1, and is a convolutional layer that learns to identify regions where features are inconsistent. Furthermore, the uncertainty is learned with loss function formulated as Eq (6):

(6)

Where the is the ground truth mask label for pixel , is predicted mask from model, acts as a learnable weight, if the model prediction is high uncertainty, the penalty decreases for a wrong prediction, and the is the regularization term.

Specifically, to derive the current features fused from memory bank, the module leverages a scaled L2 similarity metric to retrieve similar features by pixel-wise attention across the keys from memory bank, as formulated in Eq (7).

(7)

In which, denotes the shrinkage term, which scales the similarity values, represents the selection term, which regulates the relative importance to ensure that more attention is allocated to the more discriminative channels. This enables the algorithm to focus on target-specific region while retaining information of surrounding regions, as elaborated in [20]. Specifically, the function is applied to the shrinkage factor to constrain its range to [1, ∞), and the sigmoid function is used to limit the selection factor to be in [0, 1].

3.3.2 Motion uncertainty estimation.

Following [4], the Kalman Filter-based motion modeling is employed to resolve association ambiguities. The process can be defined as Eq (8):

(8)

Where z is the state vector, x, y represents the top-left coordinate of the bounding box, a denotes aspect ratio, height h, and their respective velocities. Specifically, A is the state transition matrix to predict the next state based on the current state, and the state is updated via Kalman gain , which depends on the motion uncertainty , is the observation matrix to map the state vector the measurement space, here extracting box position from the full state , and the denotes the predicted mask score. In this module, assuming the object motion adheres to a linear constant velocity model, and further, and the motion uncertainty is represented by the updated covariance matrix with Eq (9):

(9)

In which, represents the posterior uncertainty from the previous frame, and Q is the process noise, is defined as a diagonal matrix that represents the deviation from the linear const model, and denotes the measurement noise, representing the covariance of the SAM2 detections. is estimated the Kalman Gain, and uncertainty is updated based on . Under the linear model assumption, the position is set to 0.05, and the velocity is 0.00625 in Q, and ratio and its velocity are set to 0.01 and 1e-5, respectively. The measurement noise from SAM2 model is also diagonal configured to 0.05 to prevent overfitting to a certain mask. Because the initial position, the first prompt, is given, the initial position of the uncertainty matrix is set to 0.1 and the velocity is 0.0625. If the SAM2 is accurate, the Kalman Gain increases, the module will pull the state closer to the new observation. While if the motion prediction is very certain, which means is close to zero, decrease, the module will ignore the noisy jump of the detections and stick to the predicted trajectory.

Furthermore, the KF-IoU score is computed by calculating the Intersection over Union (IoU) between the predicted masks from SAM2 and the bounding box derived from the predicted states, as described in the third formula of Eq (8). The predicted mask is selected according the hybrid score of mask score and KF-IoU score , and the updated features can be obtained with Eq (10):

(10)

Where denotes the weight for fusing the enhanced features from appearance and motion estimation modules. Similar to [4], to ensure the robustness of the motion modeling, the motion module is activated if and only if the tracked object is being successfully update in the past frames.

3.3.3 Memory management.

To effectively maintaining both fine-grained local temporal dynamics and global semantic consistency, improving the robustness and generalization ability of the model, the long-term memory updating and short-term memory selection mechanism is utilized. Further, in the long-term memory design, the amount of old memory to be forgotten is dynamically determined by a linear attention gating mechanism that relies entirely on input data to automatically compute gating weights, achieving a balance between performance preservation and computational cost reduction.

Short-term memory selection. To perform the consistent feature memory propagation of the target, the short-term memory selection mechanism similar to [5] is applied. Specifically, we select memories from the feature candidates based on the set of scoring functions, which composes of , as defined in Eq (11):

(11)

where is the maximum number of historical frames to be traced back. Subsequently, the short-term memory bank is fed into the memory attention layer for feature retrieval. Specifically, following [5], we set the for selected units, others for long-term memory, and the short-term memory is dynamically updating and the same strategy of only being updated when target is present is utilized with . In this manner, the most relevant memory units are retained, which can resist the distractions from crowd background or visual similar objects to avoid introducing invalid or noisy memory information.

Long-term memory updating. In SAM 2, the original features are stored in the memory bank, in long videos, the memory storage will grow rapidly and consume more computation. And for dynamic object, the longer features play less importance on current represent. Therefore, the long-term memory updating mechanism is designed to balance the computation and performance, as illustrated in Fig 4. Specifically, the linear attention is applied to compress the features of the historical representation. The standard attention-based memory mechanism accepts an input sequence and outputs the sequence by Eq (12):

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. An illustration of the propped memory management module.

It splits the memory bank of SAM2 into long-term memory and short-term memory, where the long-term memory is updated with linear attention, and the short-term memory is updated using the proposed mechanism.

https://doi.org/10.1371/journal.pone.0353156.g004

(12)

In which, L is input length and d is the model hidden dimension, , and represent the query, key and value matrix respectively. Based on the query , key, and value token, the attention is performed which grows with the input length and has quadratic computing complex. To reduce the computing complexity and memory consuming, the Linear attention mechanisms [27] is utilized, which expand the vector hidden state with matrix representation which is calculated with Eq (13):

(13)

Where denotes the kernel operation, for linear attention, the linear kernel, setting to be the identity, without a normalizer is employed in practice [28]. And further the linear attention layer can be described as Eq (14):

(14)

In which, the query and key vectors are normalized before the attention calculation as shown in Eq (14). It can be observed that the linear attention layer is essentially a recurrent layer with matrix-valued hidden states , that is updated via the outer-product . Moreover, to dynamically updated the states automatically, the data-dependent and learnable 2D forget gate is further applied, as shown in Eq (15):

(15)

where denotes the Hadamard product. Follow [28], to make the learnable more stable, the parameter-efficient parameterization is applied base on the outer-product-based low-rank approximation: , which has parameters versus parameters of original matrix form. And the and are data-dependent to balance the short-term and long-term memory, which are calculated via a linear projection of the input with Eq (16):

(16)

In which, and , and is the sigmoid function. This gating updating mechanism determines how much the new information from the short-term memory is written into the memory bank of the SAM2 model, and how much the old state of the long-term memory is preserved.

3.3.4 Loss function.

For frame t, let be the predicted segmentation mask, and be the corresponding ground-truth. To optimize the proposed model, we adopt the common loss functions widely used in video object segmentation (VOS) tasks [8,12,46], as formulated in Eq (17):

(17)

And the overall loss function is summarized as Eq (18):

(18)

where and are the coefficients for the cross-entropy and dice losses, respectively.

This whole procedure is illustrated in Fig 3. Starting from the initial prompt, UAMP propagates the consistent features across the entire video with the appearance and motion uncertainty estimation. The proposed pseudo-label enhancement algorithm is summarized in Algorithm 1.

Algorithm 1. Proposed UAMP Method.

Input: video frames , initial prompt

Output: masks

1: Initialize memory bank and long-term memory

2: Initialize motion state from , covariance and the object token

3: for t = 1 to N do

4:  get image feature, prompt embedding, mask score, object score and masks.

5: if t > 1 then

6:  Query current features from memory bank based on the input image feature: using Eq (7)

7:  # The appearance uncertainty prediction

8:  Predict the appearance uncertainty using Eq (5)

9:  Update feature using Eq (4) appearance-uncertainty based updating

10:  # The motion uncertainty estimation

11:  calculate the state based on , and the posterior uncertainty based on using Eq (8)

12:  Estimate the motion uncertainty using Eq (9) and get the motion score

13:  Get feature using Eq (10) motion-uncertainty based updating

14:  # memory management

15:  Update short-term memory using Eq (11) short-term memory selectin based on the and

16:  Calculate the gating using Eq (16)

17:  Update long-term memory using Eq (15) long-term memory updating

18:  # parameter optimizations

19:  Calculate the appearance uncertainty loss using Eq (6) and segmentation loss using Eq (18)

20:  Optimize the model parameters using AdamW optimizer

21: else

22:  put to memory bank and

23: end if

24: end for

4.1.1 Dataset.

To comprehensively evaluate the effectiveness of our proposed method, we conduct extensive experiments on five VOS benchmarks. The specific evaluation setup is to systematically increase the difficulty of VOS tasks by targeting core challenging scenarios in video sequences, including occlusions, long videos, extreme transformations, object diversity, and scene diversity. For quantitative assessment, the following three metrics are employed: J for region similarity to measure the accuracy of region-level segmentation, F for contour accuracy to capture the fine-grained target boundaries, and the combined J&F for harmonic mean of J and F [1]. All evaluations are conducted in a semi-supervised setting, and the detailed information about datasets is presented as follows:

DAVIS 2017 [2] is a multi-object benchmark extended on DAVIS 2016, where the test split contains 30 more challenging videos with 89 objects. YouTube-VOS [29] is a large-scale diverse VOS benchmark, containing 3471 videos in the training split with 65 categories, additional 26 unseen categories are given.

MOSE [19] is a challenging VOS dataset focused on complex, real-world scenarios, and 2,149 video clips with 431,725 segmentation masks are provided. LVOS v1 [30] is collected as the long-term video object segmentation in realistic scenarios. It contains 720 video clips with 296,401 frames and 407,945 annotations, and the average video duration is over 60 seconds. LVOS v2 [18] expends LVOS v1 and convers 44 categories. Specifically, 12 of these categories are designated as unseen classes to evaluate the generalization capabilities of VOS models. SA-V [1] comprises 50.9K video clips, 642.6K masklets with 35.5M annotated masks, which is a large-scale video segmentation dataset for VOS. The key challenge lies in the presence of small, occluded, and reappearing objects in the video sequences.

4.1.2 Model details.

For image encoder, the input image is first scaled to the resolution of 1024, and the image embeddings are obtained by fusing the stride 16 and 32 features from Stages 3 and 4 of the Hiera [26] image encoder with the dimension and respectively. These multi-scale features are then fused via a pyramid network following [1], and model dimension of Hiera is 256. The positional embeddings are calculated with the windowed absolute positional embeddings, and the positional information spanning across windows is obtained by interpolating the positional embeddings. For memory attention, the input feature map is the embeddings generated by the memory encoder, the 2d spatial Rotary Positional Embedding (RoPE) [31] with the dimension of 256 is utilized in both self-attention and cross-attention layers, and this module composes 4 layers. For mask decoder, the stride 4 and 8 features from Stages 1 and 2 are added to the up-sampling layers to get high-resolution features. Following [7], the appearance uncertainty prediction module comprises one 1 × 1 convolution and two 3 × 3 convolutions. It takes the input of a concatenation of current frame feature, last frame feature, and last mask prediction, and outputs a one-channel uncertainty mask of the query token. For memory bank, the memory features are projected to the dimension of 2048 for 8 memory banks via a linear layer, and the short-term memory selection is designed follows DAM [5] as discussed in Sect 3.3. The long-term memory updating composes 2 layers of the transformer-like structure, in which the self-attention layer is replaced with the linear-attention layer. And the matrix state is projected to the same dimension as short-term memory with a linear projection layer.

4.1.2 Train details.

During training, the backbone of the SAM 2 model is frozen and one Nvidia A100 40G GPU is used. To boost the segmentation quality on long videos, the proposed method is fine-tuned where the 16-frame sequences are sampled on videos from LVOS, MOSE and YouTube-VOS training datasets, which is different from the 8-frame sequences of the SAM 2 training strategy. Specifically, The SA-V dataset is composed of the middle 20 videos from sav_000491 to sav_000510, the first 300 videos are extracted from the MOSE dataset according to the meta_train.json file, and the last 50 videos are chosen from the LVOS v1 dataset and last 150 videos from the LVOS v2 dataset according to the meta.json. Following [1], only the mask with the lowest segmentation loss is taken as a supervision, none of the mask outputs is supervised if there are no masks in current frame. For optimization, the AdamW method is employed. The learning rate is set to 1e-5 with a weight decay of 0.05, and the momentum is set to 0.9 and 0.998. the main training process runs for 60K iterations with batch size 16, and we drop the learning rate by a factor of 2 after first 20K iteration. And Table 1 further details the hyperparameter settings for training.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The training hyperparameter settings of the proposed methods.

https://doi.org/10.1371/journal.pone.0353156.t001

For inference, due to the recurrent refinement characteristic of the memory-based paradigm, the first-frame warm-up strategy is adopted. Specifically, the first frame with prompts is repeated 3 times, with each repetition treated as the initial frame to initialize the memory bank, to enhance robustness against the given prompts.

4.2.1 Quantity results.

Semi-supervised video object segmentation. We evaluate the proposed UAMP with the semi-supervised video object segmentation (VOS) setting. In which, we compare our proposed method, UAMP, with various state-of-the-art VOS methods on both the SA-V [1] and MOSE [19] datasets, as shown in Table 2. Specifically, our method achieves a J &F score of 81.6 on the SA-V validation dataset, a 1.9-point improvement over SAM2Long, 81.7 on SA-V testing dataset, and 77.1 on MOSE validation dataset.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance comparison with the state-of-the-arts methods on SA-V and MOSE datasets.

https://doi.org/10.1371/journal.pone.0353156.t002

To evaluate the performance of proposed UAMP in unseen categories, we also compare our method with various state-of-the-art VOS methods on both LVOS 1 [37] and LVOS 2 [18] datasets, which challenges models with difficult visual cues, as shown in Table 3. It can be seen that, UAMP attains J &F scores of 83.7 and 86.2 on the v1 and v2 subsets, respectively, outperforming SAM2Long by 2.4 and 2.0 points. Notably, UAMP particularly excels in unseen categories, achieving J and F scores of 83.2 and 87.5. The significant improvements of 4.1 and 3.3 points over SAM2Long highlight its robust generalization capabilities. These results showcase the effectiveness of the proposed consistent memory propagation in long-time video segmentation scenarios.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Performance comparison with the state-of-the-arts methods on LVOS datasets. Subscript s and u denote scores in seen and unseen categories.

https://doi.org/10.1371/journal.pone.0353156.t003

To further evaluate the effectiveness of UAMP, we also apply it to three widely used object tracking benchmarks, namely LaSOT [38], LaSOText [39], and GOT-10k [40]. We first generate an initial mask from the bounding box provided for the first frame using the SAM2 Mask decoder. Subsequently, all subsequent masks are transformed into bounding box format by identifying the minimum and maximum coordinates of their non-zero indices. For comparison, we include recent state-of-the-art methods specifically designed for VOT, including SAMURAI [4] and DAM2SAM [5].

Based on the strong foundation of SAM2, we achieve competitive performance with state-of-the-art VOT methods, as shown in Table 4. Specifically, our method, UAMP, delivers results that are comparable to the performance of SAMURAI and DAM2SAM, two recent methods specifically optimized for VOT. Where, UAMP’s uncertainty-aware feature fusion and linear attention design achieves a more balanced adaptation to VOT’s core demands—robustness to drift, precision in localization, and efficiency in long-sequence processing, validating the effectiveness and rationality of our proposed UAMP framework.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. State-of-the-art comparison on three standard bounding box benchmarks. The results are reported in terms of AUC (Area Under the success rate Curve) for LaSoT and LaSoText, and AO (Average Overlap) for GOT-10k.

https://doi.org/10.1371/journal.pone.0353156.t004

4.2.2 Ablation study.

Effect of the individual modules. We demonstrate the effect of the with or without appearance uncertainty prediction, motion uncertainty estimation and long-term memory updating on different settings in Table 5. Both of the proposed modules have a positive impact on the SAM 2 model, while combining both can achieve the best J&F on the SA-V, MOSE, LVOSv1 and LVOSv2 datasets with the score of 81.6%, 77.1%, 83.7% and 86.2%, respectively. For these datasets with its inclusion of small, occluded, and reappearing objects, the proposed modules showed the robust performance. Specifically, for the long sequences, such as LVOS datasets, the long-term memory updating design played an important role, which is more validated in subsequent qualitative results.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Ablations on individual design. The performance is evaluated on SA-V, MOSE, LVOSv1 and LVOSv2 datasets with J&F metric.

https://doi.org/10.1371/journal.pone.0353156.t005

Effect of the uncertainty weight. We showcase the effect of the motion uncertainty estimation, where the weight is utilized to fuse the features from memory and appearance uncertainty enhanced features in Table 6. It can be seen that the trade-off between motion and appearance demonstrates a significant impact on tracking and segmentation performance. Our experiments reveal that setting the weight to 0.24 yields the best J&F score on the MOSE, LVOSv1, LVOSv2 dataset, where motion information is needed in the occlusion and reappearing scenes. For further analysis,we give the attention map visualization, predicted object score and motion score in Fig 5. It shows that the estimated motion score has larger variance than visual object score, which also needs the guidance of visual features.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Effect of the uncertainty weight. The larger is, the more motion information is contained.

https://doi.org/10.1371/journal.pone.0353156.t006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. An illustration of the visual object score and estimated motion score from a sample of the MOSE dataset.

The first row is the visual attention and the second row is the predicted object score and motion score, where the x axis is the frame index, y axis is the score value.

https://doi.org/10.1371/journal.pone.0353156.g005

Effect of the long-term memory updating. We investigate the effectiveness of feeding the memory features to a linear attention before adding them to the memory bank. In this ablation, we evaluate on LVOSv1 and LVOSv2 as the benchmark for long-term object segmentation. While prior works have commonly employed GRU states as a means of incorporating memory into the tracking process, our findings in Table 7 suggest that the GRU does not provide a small improvement. Instead, we find the proposed module achieves a comparable performance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Effect of the long-term memory updating. Two different updating mechanism is compared with GRU and linear attention.

https://doi.org/10.1371/journal.pone.0353156.t007

Effect of IoU threshold for short-term memory selection. The choice of the IoU threshold is crucial for selecting frames with reliable object cues. As shown in Table 8, setting yields the highest J&F, indicating an effective trade-off between filtering out poor-quality frames and retaining valuable segmentation information.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Effect of IoU threshold for short-term memory selection. The higher the threshold is, the more reliability the object has.

https://doi.org/10.1371/journal.pone.0353156.t008

Runtime analysis. The incorporation of the uncertainty modeling and an enhanced memory management into the SAM2 segmentation framework introduces minimal computational overhead. Specifically, the appearance uncertainty is composed of merely three convolution layers as shown in Eq (5) and Eq (6), and the motion uncertainty is estimated with 7-dimensional matrix using Kalman filter shown in Eq (8) and Eq (9). Further, the short-term selection maintains a queue with , and is updated by specific strategy shown in Eq (3). Therefore, the memory and computation of these operations is almost negligible. The main complexity comes from the long-term memory updating module, which composes a 2-laryer transformer block with linear attention, and the training parameters are about 3.51M, 0.93G FLOPs, and with 7.37MB memory consumption. Compared to the SAM2 model with 80.8M parameters, 96.1GFLOPs computation, the memory footprint and inference latency of the proposed method remain almost consistent, as shown in Table 9.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Runtime analysis on A100. w/indicates with the module, the uncertainty module consists of the appearance and motion uncertainty, and the memory management includes the long-term memory updating and short-term memory selection.

https://doi.org/10.1371/journal.pone.0353156.t009

4.2.3 Qualitative result.

We present the qualitative results on DAVIS 2017, MOSE, YouTube-VOS, LVOS1, and LVOS2 datasets, as shown in Figs 6, 7, 8, 9, and 10. UAMP demonstrates a significant consistency in long-video object segmentation, maintaining more accurate and consistent tracking of objects across various frames. And it achieves an adaptive balance between short-term and long-term memory for long videos and crowded scenes via structurally distinct yet mutually complementary designs.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Qualitative results on DAVIS 2017 validation set.

Where the tracked objects are colored with green.

https://doi.org/10.1371/journal.pone.0353156.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Qualitative results on MOSE validation set.

Where multiple similar objects are in the vicinity of the tracked target.

https://doi.org/10.1371/journal.pone.0353156.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Qualitative results on YouTube-VOS validation set.

In which the tracked targets have large shape transformations.

https://doi.org/10.1371/journal.pone.0353156.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Qualitative results on LVOS v1 validation set.

(A) The sample video sequence contains 2681 frames, and the tracked target is far away and completed occluded in the process. (B) A video sequence contains 5526 frames, and the tracked target is distracted by similar objects and completed occluded in the process.

https://doi.org/10.1371/journal.pone.0353156.g009

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Qualitative results on LVOS v2 validation set.

(A) The tracked target has large shape transformation and heavy occlusion. (B) The tracked target is surrounded and occluded by the objects with the similar appearance in the video sequence.

https://doi.org/10.1371/journal.pone.0353156.g010

Fig 6 presents the segmentation results on DAVIS 2017 validation video sequences, demonstrating its enhanced ability to handle subtle and ambiguous segmentation tasks. In the first two rows, a person is tracked and segmented with another person around. When the person is partially occluded, by taking advantages of the motion uncertainty estimation and short-term memory, the proposed UAMP still could track part of the target during occlusion and stays on until the end of the sequence. In which, the persons are moving in different direction. The same as the third example, where the monkey far away can be consistently segmented.

More results on MOSE dataset and YouTube-VOS dataset are performed, as shown in Fig 7 and Fig 8 respectively, which involves complex real-world scenes. However, YouTube-VOS, with an average duration of only 4.5 seconds, offers limited opportunities to showcase long-term tracking capabilities. But these results illustrate that we have preserved the fundamental segmentation capabilities of SAM 2, demonstrating the robustness and versatility of UAMP across a range of VOS benchmarks. Particularly, the tracked targets are surrounded by similar objects and have shape transformations, which need the appearance information, motion clues and memory information to be tracked.

To demonstrate the long-term abilities of the proposed method, some results on LVOS 1 dataset are given in Fig 9. The UAMP enhances the SAM2 to handle subtle and ambiguous segmentation tasks. Specifically, In Fig 9(A) the video sequence has 2681 frames, the person is completed occluded and disappeared, and in the 576-th frame the person is occluded again. And further in Fig 9(B) the video length is 5526, and in 1926-th frame, the target is disappeared for about 105 frames, and the proposed method with consistent memory propagation enables it to successfully re-track the correct target, which further validates the long-term memory module.

Furthermore, more segmentation results are demonstrated in Fig 10, where the objects with identical or same appearance surrounds the tracked targets. Specifically, in these situations, the large ambiguity is introduced, challenging the tracking and segmenting. For example, in the second sequence of Fig 10(A), the tracked fish has large shape transformation and occluded by the other fishes with the similar shape. The same as the first sequence of Fig 10(B), the bottle is moved in arbitrary direction, where the consistent memory propagation is needed to handle the appearance changes, movement and dependence on history features.