Adaptive token division-based transformer for visual object tracking

Dan Tian; Dongxin Liu; Xiao Wang

doi:10.1371/journal.pone.0351112

Abstract

In visual object tracking, one-stream trackers typically use all search tokens to interact with templates across all encoder layers. However, the search area usually contains a lot of interference information, such as distractors with similar appearance to the tracking object, which will cause the distractors in the search area to be misjudged as interactive objects, establish wrong cross-relation modeling, and reduce the accuracy of tracking. To alleviate this issue, this paper proposes a transformer-based visual object tracking framework with adaptive token division. First, our tracking framework is a simple encoder-decoder structure without any post-processing. Second, we propose an adaptive token division module, which enables search tokens and template tokens to perform the most appropriate cross-relationship modeling, and improves the model ‘s ability to distinguish between object and background. At the same time, we introduce an attention masking strategy and Gumbel-Softmax technique. The strategy enables efficient and parallel attention calculation between different categories of tokens, and the technique facilitates the end-to-end optimization of the division module. Finally, we conduct tests on six tracking benchmarks, and the experimental results prove the effectiveness of our method.

Citation: Tian D, Liu D, Wang X (2026) Adaptive token division-based transformer for visual object tracking. PLoS One 21(8): e0351112. https://doi.org/10.1371/journal.pone.0351112

Editor: Shrisha Rao, International Institute of Information Technology Bangalore, INDIA

Received: November 10, 2025; Accepted: May 24, 2026; Published: August 7, 2026

Copyright: © 2026 Tian et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are available from public repositories. The COCO, LaSOT, GOT-10k, TrackingNet, UAV123, NFS, and TNL2Kdatasets can be accessed at https://cocodataset.org/#download, http://vision.cs.stonybrook.edu/~lasot/download.html, http://got-10k.aitestunion.com, https://github.com/SilvioGiancola/TrackingNet-devkit, https://ivul.kaust.edu.sa/benchmark-and-simulator-uav-tracking-dataset, http://ci2cv.net/nfs/index.html, and https://github.com/wangxiao5791509/TNL2K_evaluation_toolkit.

Funding: This work was supported by Liaoning Science Foundation of China under Grant 2023-MS-322.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

Visual object tracking [1] is an important research content in the field of computer vision, which aims to locate a specific object and track its motion trajectory in real time within consecutive video sequences. With the rapid development of artificial intelligence technology, visual object tracking has shown a wide range of application prospects in many fields such as autonomous driving, intelligent monitoring, and human-computer interaction [2].

At present, the mainstream visual object tracking methods are mainly divided into two-stream [3–5] and one-stream [6–8]. These methods typically treat the tracking problem as a per-frame template matching task, generally divided into three stages: (i) feature extraction from both search and template images; (ii) feature matching and fusion using convolutional or attention mechanisms; (iii) use customized head networks to perform operations such as center point localization and scale estimation. Compared with the two-stream methods, the one-stream methods have a simpler network structure and higher computational efficiency. The one-stream methods leverage attention mechanisms to jointly extract visual features and model relationships between template regions and search regions. However, the use of a customized head network increases the complexity of the network framework and may require additional training. SeqTrack [9] and ARTrack [10] modeled tracking as a sequence generation problem. They use the encoder-decoder transformer structure, which does not need to design complex head networks and simplifies the network structure.

References [7,8,11] indicate that early interaction between the template and the search region facilitates the generation of features with greater object discrimination. However, not all parts within the search region are suitable for interaction with the template. In fact, the search region usually contains a lot of interference information, such as interferences similar to the appearance of the tracking object, which will lead to the interference in the search area being misjudged as interactive objects, establish wrong cross-relationship modeling, and reduce the accuracy of tracking.

Although attention mechanisms can suppress inappropriate feature interactions to some extent, the application of global interaction modeling to all encoder layers inevitably introduces some interference. Specifically, this interference is mainly reflected in two aspects: First, when the distractor in the search region interacts incorrectly with the template, the distractor may aggregate the object features, which will make it difficult to locate the actual object in the search region accurately. Second, in the process of feature updating, improper interaction between the template tokens and distractors in the search region will reduce the quality of the template feature. The above situation will weaken the ability of the one-stream tracking methods to distinguish between the object and the background, which reduces the accuracy of the algorithm.

To address the above issues, we propose a transformer-based visual tracking framework with adaptive token division, which only uses the encoder-decoder transformer architecture. By introducing the adaptive token division module into the transformer encoder, the search tokens and the template tokens can be used to model the most suitable cross-relationship, suppress the influence of background information or similar interferences, and improve the model ‘s ability to distinguish between object and background. Specifically, we divide all tokens into three categories: one is template tokens, and the other two are different types of search tokens. Among them, only the search tokens judged as the tracking object can interact with the template tokens, and the search tokens judged as the background cannot interact with the template tokens.

The categorization of search tokens relies on the adaptive token division module, but two problems need to be solved in the actual design: one is how to realize the interaction between different token categories efficiently and parallelly, and the other is how to ensure the end-to-end optimization of discrete token classification. For the first issue, we use the attention masking mechanism to integrate the original multiple attention operations into a unified attention operation. For the second issue, we introduce the Gumbel-softmax technique [12] to make the discrete classification process differentiable. In addition, compared with [9] and [10], our model adopts a similar transformer structure, only adds an adaptive token division module to the encoder. This paper makes three main contributions:

(1) We design an adaptive token division module and introduce it into the transformer encoder, so that the search tokens and the template tokens can be used for the most appropriate cross-relationship modeling. This can improve the model ‘s ability to distinguish between object and background. Meanwhile, we use a simple encoder-decoder structure without designing complex head networks for post-processing.
(2) We introduce the attention masking mechanism and Gumbel-softmax technique to enable efficient and parallel interaction between different categories of tokens, and facilitate end-to-end optimization of the adaptive token division module.
(3) Compared to other state-of-the-art trackers, our method achieves more advanced performance on datasets such as Lasot, GOT-10k, and TrackingNet.

2. Related works

2.1. Visual tracking framework

In the past, object tracking methods based on siamese network architecture [13–16] have made some progress in the field of visual tracking. Such methods usually use a two-branch structure to extract features from the template and the search region respectively, and then perform feature fusion through specific modules. With the rapid development of transformer in the field of computer vision, it has shown excellent performance in object tracking tasks. Consequently, numerous state-of-the-art trackers [2–6,17,18] now adopt transformers as their core architecture.

In recent years, one-stream tracking frameworks [6–8] have used self-attention mechanisms for joint feature extraction and relationship modeling, which has effectively enhanced their feature interaction capability. Although these one-stream methods are different from the previous two-stream methods in structure, they all treat the search area as a whole, that is, the template features interact with the entire search area. However, not all parts of the search area are suitable for interacting with the template. When the feature expression ability is insufficient, it is easy to cause confusion between the object and the background information. To solve this problem, we propose an adaptive token division module. This model enables search tokens and template tokens to establish the most appropriate cross-relationship modeling in each encoder layer, which helps suppress background interference.

In addition, these methods usually need to design a specific head network for post-processing operations such as object location and scale estimation, which increases the complexity of the model. The design of the head network also needs to introduce multiple loss functions, and more hyperparameters need to be adjusted, which increases the difficulty and uncertainty of model tuning. To this end, SeqTrack [9] and ARTrack [10] model tracking as a sequence generation task. They adopt the encoder-decoder transformer structure, and do not need to design a specific head network and multiple loss functions. Our model uses a similar transformer structure, only adding an adaptive token division module to the encoder.

2.2. Tracking based on dynamic design

Some tracking algorithms achieve diversity objectives by introducing dynamic design at different stages of the model. LightTrack [19] and AutoMatch [20] leverage neural architecture search techniques to dynamically optimize network structures during training. LightTrack focuses on searching lightweight backbone network and prediction head architecture. This goal is to achieve a balance between computational efficiency and tracking accuracy. AutoMatch focuses on automatically designing diverse matching operators to enhance the robustness of related modules. However, the network structures of these methods are fixed once training is completed. Consequently, they lack the ability to dynamically adjust based on input information during inference. In contrast, our method supports dynamic adjustments during inference. Through the adaptive token division module, it dynamically selects the most suitable search tokens for interaction with the template in each encoder layer. As a result, our approach offers significantly greater flexibility.

To optimize the performance of visual transformers, researchers [21–23] have proposed various token organization strategies. These approaches fully leverage the high flexibility of attention mechanisms. SparseTT [24] employs a sparse attention mechanism. Through this method, the model can selectively focus on the most similar token features, which enhances foreground-background discrimination. OSTrack [8] designs an early candidate elimination module. With this module, the model can remove irrelevant tokens in the search region. This significantly improves computational efficiency. GRM [25] uses a lightweight prediction module to classify input tokens, which enables flexible relationship modeling. Although these methods have achieved excellent performance, their structures are complex. They need to design specialized head networks for subsequent object localization.

To address the above problems, this paper proposes a transformer-based visual tracking framework with adaptive token division, which only uses encoder and decoder without any post-processing. We integrate an adaptive token division module into the encoder. This allows flexible interaction between search and template tokens, suppresses interference from irrelevant information in the search region, and achieves more accurate tracking.

3. Method

In this section, we present a detailed description of the proposed method. First, we briefly outline the algorithmic framework. Then, we describe the model architecture in detail. Finally, we introduce the training and inference strategies.

3.1. Overview

Our tracker framework, as shown in Fig 1, adopts an encoder-decoder transformer architecture. First, both the search image and template images are divided into image patches. These patches are then transformed into visual embeddings via linear projection and fed into the encoder. The input image is divided into three categories by the adaptive token division module, where G_A represents the object tokens in the search region, G_S represents the remaining tokens in the search region, and G_T is all template tokens. The visual features are output through three parallel attention calculations. During training, the decoder takes [start, x_min, y_min, x_max, y_max] as the input sequence and [x_min, y_min, x_max, y_max, end] as the output sequence. Here, [x_min, y_min, x_max, y_max] is converted from the object bounding box, and the start and end tokens control the start and end of the sequence generation task, respectively. During inference, the input sequence begins with [start], and newly generated bounding box tokens are appended sequentially at each step. The inference task concludes once all four bounding box tokens are generated. To ensure that each token can only focus on the previous information during its generation, a causal attention mask is applied to the self-attention module of the decoder.

Download:

Fig 1. Overall architecture of our method.

The encoder is responsible for extracting visual features, while the decoder uses the extracted visual features to generate the final prediction sequence. Through the proposed attention masking strategy, three parallel attention calculations are performed. For simplicity, we only mark the queries and keys in parallel attention computing. The values are identical to the keys and are omitted here.

https://doi.org/10.1371/journal.pone.0351112.g001

3.2. Image and sequence representation

The input of the encoder includes a search image and a template image . Before inputting the image into the encoder, the search image and the template image are first divided into image patches and , where and represents the number of image patches, and is the size of the image patch. Then, the image patch is converted into visual embeddings and by linear projection, where is the embedding dimension. Finally, the visual embeddings are input to the encoder for subsequent processing.

The input sequence of the decoder is [start, x_min, y_min, x_max, y_max], which is converted from the object bounding box. We discretize all continuous coordinate values and map them into a unified vocabulary V. Each coordinate value is converted into integers between [1, n], which are considered as words in vocabulary V. Specifically, each value in the input sequence and the object sequence (i.e., the coordinate value of the bounding box) corresponds to a word in the vocabulary. During training, the model learns how to generate the object sequence from the input sequence. During inference, the model maps the generated integers to coordinate values from its vocabulary, and finally outputs the prediction of the bounding box.

3.3. Model architecture

3.3.1. Encoder.

The encoder’s structure is the same as the vision transformer [26], except for one modification: an adaptive token division module is added to encoder layers. First, both the search and template images are divided into image patches. These image patches are converted into visual embeddings through linear projection, and then fed into the encoder. Second, the input tokens are divided into three categories: G_T, G_A and G_S by using the adaptive token division module. G_T can aggregate the tokens information in G_T and G_A, G_S can aggregate the tokens information in G_S and G_A, and G_A can aggregate all the tokens information. Finally, the visual features are output through three parallel attention calculations.

3.3.2. Decoder.

Each decoder layer consists of masked multi-head self-attention, multi-head cross-attention, and a feedforward network. The word-to-embedding layer converts the input sequence into word embeddings, adds positional information, and feeds it into the decoder’s masked multi-head self-attention layer. Due to the masking effect, the generation of each sequence element depends solely on its preceding elements. Subsequently, multi-head cross-attention fuses the visual features extracted by the encoder with the word embeddings. Finally, the feedforward network layer feeds the output of the multi-head cross-attention into the next decoder block.

3.3.3. Adaptive token division module.

In order to adaptively divide the search tokens into G_S and G_A, we propose an adaptive token division module and add it to encoder layers. Specifically, to provide object-related context information to search tokens, we first aggregate all template tokens through a global maximum pooling. This operation will generate an object-aware global representation. Subsequently, the object-aware representation is concatenated with the features of each search token. Finally, the concatenated features are processed by a lightweight multi-layer perceptron (MLP). This MLP predicts classification probabilities for each search token. The outputs represent the likelihood of belonging to G_S and G_A:

(1)

Where and denote the template tokens embedding and the i-th search token embedding respectively, and denote the probability that the i-th search token belongs to G_S and G_A, respectively. Based on the predicted values and , the model selects the larger probability to determine the search token’s category. Although it seems easy to apply this module for search token division, two obstacles need to be solved. That is how to achieve efficient parallel attention computing, and how to achieve end-to-end optimization learning of this module.

The first issue stems from how to model the cross-relations of the three token categories. The number of queries and keys in these three categories is different, which makes it difficult to perform three attention calculations in parallel. This will reduce the tracking speed. To address this, we design an attention masking strategy. Specifically, we first convert the probability into one-hot encoding . Then we define two one-hot tensors: denotes the category assignment of template tokens, and denotes the category assignment of search tokens. In , the first column is all set to 1, and the other two columns are set to 0, indicating that all template tokens are divided into G_T. is obtained by adding a column of zeros before , which ensures that the search tokens are not categorized into G_T. Finally, the merged is used to construct the attention mask , where the value of each element is calculated as follows:

(2)

The value of determines whether token can aggregate information from token . By element-wise multiplying this attention mask matrix with the original complete attention weight matrix, the attention weight of the interactive position prohibited will be set to zero. We can effectively merge the original three independent attention calculations into a single calculation, and then achieve parallel attention calculation.

The second issue is that the classification of search tokens is essentially a discrete decision-making process. This discrete operation is non-differentiable, which prevents gradient calculation during backpropagation. As a result, the parameters of the division module cannot be directly optimized through gradient descent. To address this, we introduce the Gumbel-Softmax technique, which adds Gumbel distribution noise to the classification probability , so that the sampling process becomes differentiable:

(3)

Where . The is used for the classification of search tokens in forward propagation. In the back propagation, the softmax operation with temperature parameters is used to approximate the argmax operation:

(4)

where represents the temperature parameter and represents the gradient. The gradient is calculated by the continuous output of softmax, so that the adaptive division module can achieve end-to-end optimization. For a complete derivation of this technique, please refer to reference [12].

3.4. Training and inference

3.4.1. Training.

During training, the adaptive token division module calculates the category probability for each search token, then uses Formula (3)to generate one-hot encoding , so as to construct the attention mask matrix and realize parallel attention calculation. In the back propagation, the model uses the softmax approximation in Formula (4)to calculate the gradient, and combines the loss function to realize the end-to-end optimization of the division module.

We use the cross-entropy loss function for training, which aims to maximize the log-likelihood between the generated object sequence and the actual object sequence. The loss function for the training process is as follows:

(5)

Where represents the probability of softmax, and represent the search image and the template image respectively, represents the currently predicted object sequence, represents the position of the currently generated token in the sequence, represents the object sequence before , and represents the length of the object sequence.

3.4.2. Inference.

During inference, the adaptive token division module performs a binary classification of each search token, and directly takes argmax to generate one-hot encoding without Gumbel noise. The input sequence of the decoder is [start], which marks the beginning of the task. Then, the model generates the object bounding box sequence [x_min, y_min, x_max, y_max, end] in an autoregressive manner. In each generation step, the model is sampled from the vocabulary V based on the maximum likelihood, that is, , where is the word in the vocabulary V.

4. Experiments

4.1. Implementation details

The encoder of our model adopts ViT-B [26] architecture, which is pre-trained using the MAE [27] model. The resolution of the search image and the template image is set to 256 × 256 and 128 × 128, respectively, and the size of the image patch is 16 × 16. The decoder layer is set to 2, the attention head is set to 8, the dropout rate is 0.1, and the FFN hidden size is 1024. The vocabulary size is set to 4000, and both the word embedding dimension and the decoder’s hidden dimension are set to 256. All adaptive token division modules adopt the same structure, that is, a lightweight MLP, which consists of two hidden layers that use the GELU activation function [28] and a final output layer. Their channel sizes are 384, 192, and 2, respectively.

The training datasets include COCO [29], LaSOT [30], GOT-10k [31], and TrackingNet [32]. According to the official requirements, we remove the 1k video sequences prohibited in the GOT-10k dataset during training. We use brightness jitter and horizontal flip to achieve data enhancement. The entire model is optimized for 300 epochs using the AdamW optimizer [33]. The learning rate for the backbone network is set to 4 × 10⁻⁵, while for other parameters it is set to 4 × 10⁻⁴. After 240 epochs, the learning rate is reduced by a factor of 10. For the GOT-10k dataset, we train for 150 epochs, with the learning rate decreased by a factor of 10 after 120 epochs. Our experiments are performed on Intel Core i9-13900KF CPU @ 3.00GHz, 32GB RAM and a single GeForce RTX4090 GPU. The running speed of our tracker is about 65 fps, which is an average value obtained by directly performing continuous inference on a 1000-frame long video using a single GeForce RTX4090 GPU.

4.2. Main results and comparisons

We compare our method with some state-of-the-art trackers on six tracking benchmarks: LaSOT [30], GOT-10k [31], TrackingNet [32], UAV123 [34], NFS [35], and TNL2K [36].

LaSOT is a large-scale benchmark dataset designed for single object tracking tasks. It contains a total of 1,400 video sequences. The average length of each sequence is about 2,500 frames, and the total number of frames exceeds 1.5 million. The dataset covers a variety of object categories and complex scenes, which involve various challenging factors such as illumination changes, object occlusion, and rapid movement. It is widely used to evaluate the robustness and accuracy of tracking algorithms. We evaluate our method on the LaSOT test set and compare it with some mainstream trackers. As shown in Table 1, our method achieves a success rate of 72.6%, which is 1% and 1.2% higher than ARTrackV2 [37] and AQATrack [38], respectively.

Download:

Table 1. Comparison with other methods on LaSOT.

https://doi.org/10.1371/journal.pone.0351112.t001

GOT-10k is a large-scale benchmark dataset for general object tracking. It contains more than 10,000 video frame sequences, and the total number of frames is more than 1.5 million. The dataset covers a variety of object types and scenes, and contains a variety of complex challenges such as motion states and occlusions. Therefore, it is of great significance to test the generalization performance of tracking algorithms. According to the official requirements, we only use the training set of the dataset for model training, and evaluate our method through the official evaluation server. As shown in Table 2, our method achieves an AO score of 76.0%, which is 1.3% higher than that of SeqTrack [9].

Download:

Table 2. Comparison with other methods on GOT-10k.

https://doi.org/10.1371/journal.pone.0351112.t002

TrackingNet is a large-scale short-term object tracking benchmark. The dataset is divided into training and testing subsets, which contain more than 30,000 video sequences. We evaluate our method on its test set and use an online evaluation server to obtain performance metrics. As shown in Table 3, our method achieves a success rate of 85.0%, which is 1.2% and 0.8% higher than AQATrack [38] and ARTrack [10], respectively.

Download:

Table 3. Comparison with other methods on TrackingNet.

https://doi.org/10.1371/journal.pone.0351112.t003

We also evaluate our method on three other tracking benchmarks, including UAV123, NFS, and TNL2K. UAV123 is a long-term tracking benchmark and comprises 123 video sequences captured from a drone perspective, which include diverse complex environments and challenges such as fast motion, viewpoint changes, and occlusions. NFS is a tracking benchmark consisting of 100 fast-motion video sequences. Similar to previous trackers, we use the 30fps version of NFS to evaluate our method. TNL2K is a large-scale dataset with natural language description, which contains 700 test sequences. Each sequence not only has visual data, but also has a natural language description. It is a challenging tracking benchmark. We compare our method with ARTrackV2 [37], SeqTrack [9], OSTrack [8], TransT [4], ARTrack [10], DiMP [40], ATOM [13], and Ocean [41]. As shown in Fig 2, our method achieves AUC scores of 70.7%, 68.2% and 59.3% on three datasets, respectively, which demonstrates excellent performance.

Download:

Fig 2. Comparison of AUC scores with other trackers on UAV123, NFS, and TNL2K datasets.

https://doi.org/10.1371/journal.pone.0351112.g002

4.3. Ablation studies

In order to prove the effectiveness of the adaptive token division module, we use the GOT-10k dataset for comparative experiments. Both methods employ identical pre-training strategies, input resolutions, training data compositions, and evaluation protocols. As shown in Table 4, after introducing this module into the encoder, the model achieves an AO score of 76.0% with a slight loss in inference speed, which is 3.3% higher than that of the model without it.

Download:

Table 4. Comparison experiment of adaptive token division module.

https://doi.org/10.1371/journal.pone.0351112.t004

Subsequently, to determine the optimal encoder layer for adaptive token division, we conduct three variant experiments: incorporating the module into all encoder layers (variant 2), into the last eight encoder layers (variant 3), and into all encoder layers except the first one (variant 4). The results of variant 2 in Table 5 show that it is not optimal to apply adaptive token division in the first encoder layer, because the initial token embeddings lack sufficient feature refinement. Specifically, the input tokens in the first encoder layer have only undergone linear projection. They have not yet been refined through attention-based feature aggregation or non-linear transformations. Their representations remain at a local, low-semantic patch level, which lacks global context and awareness of target structure. Making token division decisions based on such raw features is prone to interference, which often comes from local noise or background regions. As a result, it leads to unstable segmentation outcomes. This instability not only degrades the quality of cross-modal interaction, but also harms the target feature extraction in subsequent layers. Variant 4 has the best tracking performance. Therefore, in our method, we add the adaptive token division module to all encoder layers except the first layer.

Download:

Table 5. The ablation experiment on which layers of the encoder apply the adaptive token division module.

https://doi.org/10.1371/journal.pone.0351112.t005

Additionally, the separate processing of attention for different categories slows down the model. To address this issue, we introduce an attention masking strategy that unifies the computation. In our comparative experiments, the model with separate computation of attention runs at 50 fps, while our method with the masking strategy increases the speed to 65 fps.

5. Conclusion

In this paper, we propose a transformer-based visual tracking framework with adaptive token division. Our framework only uses encoders and decoders without any post-processing operations. In addition, this paper introduces an adaptive token division module in the encoder to achieve the better interaction between search tokens and template tokens. We employ an attention masking strategy to achieve parallel attention calculation between different token categories, and introduce Gumbel-softmax technique to promote end-to-end optimization of adaptive token division module. Experiments show that our method achieves excellent performance on the mainstream tracking benchmarks, which proves the effectiveness of our method in dealing with complex scenarios such as similar object interference.

References

1. Javed S, Danelljan M, Khan FS, Khan MH, Felsberg M, Matas J. Visual Object Tracking With Discriminative Filters and Siamese Networks: A Survey and Outlook. IEEE Trans Pattern Anal Mach Intell. 2023;45(5):6552–74. pmid:36215368
- View Article
- PubMed/NCBI
- Google Scholar
2. Mohamed I, Elhenawy I, Sallam AW, Gatt A, Salah A. A practical evaluation of correlation filter-based object trackers with new features. PLoS One. 2022;17(8):e0273022. pmid:36006906
- View Article
- PubMed/NCBI
- Google Scholar
3. Wang N, Zhou W, Wang J, Li H. Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1571–80. https://doi.org/10.1109/cvpr46437.2021.00162
4. Chen X, Yan B, Zhu J, Wang D, Yang X, Lu H. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. https://doi.org/10.1109/CVPR46437.2021.00803
5. Gao S, Zhou C, Ma C, Wang X, Yuan J. Aiatrack: Attention in attention for transformer visual tracking. In: Proceedings of the European Conference on Computer Vision, 2022. https://doi.org/10.1007/978-3-031-20047-2_9
6. Yan B, Peng H, Fu J, Wang D, Lu H. Learning Spatio-Temporal Transformer for Visual Tracking. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 10428–37. https://doi.org/10.1109/iccv48922.2021.01028
7. Cui Y, Jiang C, Wang L, Wu G. MixFormer: End-to-End Tracking with Iterative Mixed Attention. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 13598–608. https://doi.org/10.1109/cvpr52688.2022.01324
8. Ye B, Chang H, Ma B, Shan S, Chen X. Joint feature learning and relation m-odeling for tracking: A one-stream framework. In: Proceedings of the Europ-ean conference on computer vision. 2022.
- View Article
- Google Scholar
9. Chen X, Peng H, Wang D, Lu H, Hu H. SeqTrack: Sequence to Sequence Learning for Visual Object Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 14572–81. https://doi.org/10.1109/cvpr52729.2023.01400
10. Wei X, Bai Y, Zheng Y, Shi D, Gong Y. Autoregressive Visual Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 9697–706. https://doi.org/10.1109/cvpr52729.2023.00935
11. Xie F, Wang C, Wang G, Cao Y, Yang W, Zeng W. Correlation-aware deep tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. https://doi.org/10.1109/CVPR52688.2022.00855
12. Jang E, Gu S, Poole B. Categorical reparameterization with gumbel-softmax. 2016. https://doi.org/10.48550/arXiv.1611.01144
13. Danelljan M, Bhat G, Khan FS, Felsberg M. ATOM: Accurate Tracking by Overlap Maximization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4655–64. https://doi.org/10.1109/cvpr.2019.00479
14. Li B, Yan J, Wu W, Zhu Z, Hu X. High Performance Visual Tracking with Siamese Region Proposal Network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 8971–80. https://doi.org/10.1109/cvpr.2018.00935
15. Guo D, Wang J, Cui Y, Wang Z, Chen S. SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 6268–76. https://doi.org/10.1109/cvpr42600.2020.00630
16. Voigtlaender P, Luiten J, Torr PH, Leibe B. Siam r-cnn: Visual tracking by r-edetection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. https://doi.org/10.1109/CVPR42600.2020.00661
17. Lin L, Fan H, Zhang Z, Xu Y, Ling H. SwinTrack: A Simple and Strong Baseline for Transformer Tracking. In: Advances in Neural Information Processing Systems 35, 2022. 16743–54. https://doi.org/10.52202/068431-1218
18. Mayer C, Danelljan M, Bhat G, Paul M, Paudel DP, Yu F. Transforming model prediction for tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. https://doi.org/10.1109/CVPR52688.2022.00853
19. Yan B, Peng H, Wu K, Wang D, Fu J, Lu H. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. https://doi.org/10.1109/CVPR46437.2021.01493
20. Zhang Z, Liu Y, Wang X, Li B, Hu W. Learn to Match: Automatic Matching Network Design for Visual Tracking. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 13319–28. https://doi.org/10.1109/iccv48922.2021.01309
21. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 9992–10002. https://doi.org/10.1109/iccv48922.2021.00986
22. Rao Y, Zhao W, Liu B, Lu J, Zhou J, Hsieh CJ. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems. 2021;34:13937–49.
- View Article
- Google Scholar
23. Yu T, Zhao G, Li P, Yu Y. BOAT: Bilateral local attention vision transformer. arXiv preprint arXiv. 2022.
- View Article
- Google Scholar
24. Fu Z, Fu Z, Liu Q, Cai W, Wang Y. SparseTT: Visual tracking with sparse transformers. arXiv preprint. 2022.
- View Article
- Google Scholar
25. Gao S, Zhou C, Zhang J. Generalized Relation Modeling for Transformer Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 18686–95. https://doi.org/10.1109/cvpr52729.2023.01792
26. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
- View Article
- Google Scholar
27. He K, Chen X, Xie S, Li Y, Dollar P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 15979–88. https://doi.org/10.1109/cvpr52688.2022.01553
28. Hendrycks D, Gimpel K. Gaussian error linear units (gelus). arXiv preprint. 2016.
- View Article
- Google Scholar
29. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common Objects in Context. Lecture Notes in Computer Science. Springer International Publishing. 2014. p. 740–55.
- View Article
- Google Scholar
30. Fan H, Lin L, Yang F, Chu P, Deng G, Yu S, et al. LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5369–78. https://doi.org/10.1109/cvpr.2019.00552
31. Huang L, Zhao X, Huang K. GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild. IEEE Trans Pattern Anal Mach Intell. 2021;43(5):1562–77. pmid:31804928
- View Article
- PubMed/NCBI
- Google Scholar
32. Müller M, Bibi A, Giancola S, Alsubaihi S, Ghanem B. TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild. Lecture Notes in Computer Science. Springer International Publishing. 2018. p. 310–27.
- View Article
- Google Scholar
33. Loshchilov I, Hutter F. Decoupled weight decay regularization. 2017.
- View Article
- Google Scholar
34. Mueller M, Smith N, Ghanem B. A Benchmark and Simulator for UAV Tracking. Lecture Notes in Computer Science. Springer International Publishing. 2016. p. 445–61.
- View Article
- Google Scholar
35. Galoogahi HK, Fagg A, Huang C, Ramanan D, Lucey S. Need for Speed: A Benchmark for Higher Frame Rate Object Tracking. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017. 1134–43.https://doi.org/10.1109/iccv.2017.128
36. Wang X, Shu X, Zhang Z, Jiang B, Wang Y, Tian Y, et al. Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 13758–68. https://doi.org/10.1109/cvpr46437.2021.01355
37. Bai Y, Zhao Z, Gong Y, Wei X. Artrackv2: Prompting autoregressive tracker where to look and how to describe. In: Proceedings of the IEEE/CVF confer-ence on computer vision and pattern recognition. 2024.
- View Article
- Google Scholar
38. Xie J, Zhong B, Mo Z, Zhang S, Shi L, Song S, et al. Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 19300–9. https://doi.org/10.1109/cvpr52733.2024.01826
39. Cai W, Liu Q, Wang Y. HIPTrack: Visual Tracking with Historical Prompts. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 19258–67. https://doi.org/10.1109/cvpr52733.2024.01822
40. Bhat G, Danelljan M, Van Gool L, Timofte R. Learning Discriminative Model Prediction for Tracking. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 6181–90. https://doi.org/10.1109/iccv.2019.00628
41. Zhang Z, Peng H, Fu J, Li B, Hu W. Ocean: Object-aware anchor-free tracking. In: Proceedings of the European Conference on Computer Vision, 2020. https://doi.org/10.1007/978-3-030-58589-1_46

[ref1] 1. Javed S, Danelljan M, Khan FS, Khan MH, Felsberg M, Matas J. Visual Object Tracking With Discriminative Filters and Siamese Networks: A Survey and Outlook. IEEE Trans Pattern Anal Mach Intell. 2023;45(5):6552–74. pmid:36215368
View Article
PubMed/NCBI
Google Scholar

[2] View Article

[3] PubMed/NCBI

[4] Google Scholar

[ref2] 2. Mohamed I, Elhenawy I, Sallam AW, Gatt A, Salah A. A practical evaluation of correlation filter-based object trackers with new features. PLoS One. 2022;17(8):e0273022. pmid:36006906
View Article
PubMed/NCBI
Google Scholar

[6] View Article

[7] PubMed/NCBI

[8] Google Scholar

[ref3] 3. Wang N, Zhou W, Wang J, Li H. Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1571–80. https://doi.org/10.1109/cvpr46437.2021.00162

[ref4] 4. Chen X, Yan B, Zhu J, Wang D, Yang X, Lu H. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. https://doi.org/10.1109/CVPR46437.2021.00803

[ref5] 5. Gao S, Zhou C, Ma C, Wang X, Yuan J. Aiatrack: Attention in attention for transformer visual tracking. In: Proceedings of the European Conference on Computer Vision, 2022. https://doi.org/10.1007/978-3-031-20047-2_9

[ref6] 6. Yan B, Peng H, Fu J, Wang D, Lu H. Learning Spatio-Temporal Transformer for Visual Tracking. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 10428–37. https://doi.org/10.1109/iccv48922.2021.01028

[ref7] 7. Cui Y, Jiang C, Wang L, Wu G. MixFormer: End-to-End Tracking with Iterative Mixed Attention. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 13598–608. https://doi.org/10.1109/cvpr52688.2022.01324

[ref8] 8. Ye B, Chang H, Ma B, Shan S, Chen X. Joint feature learning and relation m-odeling for tracking: A one-stream framework. In: Proceedings of the Europ-ean conference on computer vision. 2022.
View Article
Google Scholar

[15] View Article

[16] Google Scholar

[ref9] 9. Chen X, Peng H, Wang D, Lu H, Hu H. SeqTrack: Sequence to Sequence Learning for Visual Object Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 14572–81. https://doi.org/10.1109/cvpr52729.2023.01400

[ref10] 10. Wei X, Bai Y, Zheng Y, Shi D, Gong Y. Autoregressive Visual Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 9697–706. https://doi.org/10.1109/cvpr52729.2023.00935

[ref11] 11. Xie F, Wang C, Wang G, Cao Y, Yang W, Zeng W. Correlation-aware deep tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. https://doi.org/10.1109/CVPR52688.2022.00855

[ref12] 12. Jang E, Gu S, Poole B. Categorical reparameterization with gumbel-softmax. 2016. https://doi.org/10.48550/arXiv.1611.01144

[ref13] 13. Danelljan M, Bhat G, Khan FS, Felsberg M. ATOM: Accurate Tracking by Overlap Maximization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4655–64. https://doi.org/10.1109/cvpr.2019.00479

[ref14] 14. Li B, Yan J, Wu W, Zhu Z, Hu X. High Performance Visual Tracking with Siamese Region Proposal Network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. 8971–80. https://doi.org/10.1109/cvpr.2018.00935

[ref15] 15. Guo D, Wang J, Cui Y, Wang Z, Chen S. SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 6268–76. https://doi.org/10.1109/cvpr42600.2020.00630

[ref16] 16. Voigtlaender P, Luiten J, Torr PH, Leibe B. Siam r-cnn: Visual tracking by r-edetection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. https://doi.org/10.1109/CVPR42600.2020.00661

[ref17] 17. Lin L, Fan H, Zhang Z, Xu Y, Ling H. SwinTrack: A Simple and Strong Baseline for Transformer Tracking. In: Advances in Neural Information Processing Systems 35, 2022. 16743–54. https://doi.org/10.52202/068431-1218

[ref18] 18. Mayer C, Danelljan M, Bhat G, Paul M, Paudel DP, Yu F. Transforming model prediction for tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. https://doi.org/10.1109/CVPR52688.2022.00853

[ref19] 19. Yan B, Peng H, Wu K, Wang D, Fu J, Lu H. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. https://doi.org/10.1109/CVPR46437.2021.01493

[ref20] 20. Zhang Z, Liu Y, Wang X, Li B, Hu W. Learn to Match: Automatic Matching Network Design for Visual Tracking. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 13319–28. https://doi.org/10.1109/iccv48922.2021.01309

[ref21] 21. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 9992–10002. https://doi.org/10.1109/iccv48922.2021.00986

[ref22] 22. Rao Y, Zhao W, Liu B, Lu J, Zhou J, Hsieh CJ. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems. 2021;34:13937–49.
View Article
Google Scholar

[31] View Article

[32] Google Scholar

[ref23] 23. Yu T, Zhao G, Li P, Yu Y. BOAT: Bilateral local attention vision transformer. arXiv preprint arXiv. 2022.
View Article
Google Scholar

[34] View Article

[35] Google Scholar

[ref24] 24. Fu Z, Fu Z, Liu Q, Cai W, Wang Y. SparseTT: Visual tracking with sparse transformers. arXiv preprint. 2022.
View Article
Google Scholar

[37] View Article

[38] Google Scholar

[ref25] 25. Gao S, Zhou C, Zhang J. Generalized Relation Modeling for Transformer Tracking. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 18686–95. https://doi.org/10.1109/cvpr52729.2023.01792

[ref26] 26. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
View Article
Google Scholar

[41] View Article

[42] Google Scholar

[ref27] 27. He K, Chen X, Xie S, Li Y, Dollar P, Girshick R. Masked Autoencoders Are Scalable Vision Learners. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 15979–88. https://doi.org/10.1109/cvpr52688.2022.01553

[ref28] 28. Hendrycks D, Gimpel K. Gaussian error linear units (gelus). arXiv preprint. 2016.
View Article
Google Scholar

[45] View Article

[46] Google Scholar

[ref29] 29. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common Objects in Context. Lecture Notes in Computer Science. Springer International Publishing. 2014. p. 740–55.
View Article
Google Scholar

[48] View Article

[49] Google Scholar

[ref30] 30. Fan H, Lin L, Yang F, Chu P, Deng G, Yu S, et al. LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5369–78. https://doi.org/10.1109/cvpr.2019.00552

[ref31] 31. Huang L, Zhao X, Huang K. GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild. IEEE Trans Pattern Anal Mach Intell. 2021;43(5):1562–77. pmid:31804928
View Article
PubMed/NCBI
Google Scholar

[52] View Article

[53] PubMed/NCBI

[54] Google Scholar

[ref32] 32. Müller M, Bibi A, Giancola S, Alsubaihi S, Ghanem B. TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild. Lecture Notes in Computer Science. Springer International Publishing. 2018. p. 310–27.
View Article
Google Scholar

[56] View Article

[57] Google Scholar

[ref33] 33. Loshchilov I, Hutter F. Decoupled weight decay regularization. 2017.
View Article
Google Scholar

[59] View Article

[60] Google Scholar

[ref34] 34. Mueller M, Smith N, Ghanem B. A Benchmark and Simulator for UAV Tracking. Lecture Notes in Computer Science. Springer International Publishing. 2016. p. 445–61.
View Article
Google Scholar

[62] View Article

[63] Google Scholar

[ref35] 35. Galoogahi HK, Fagg A, Huang C, Ramanan D, Lucey S. Need for Speed: A Benchmark for Higher Frame Rate Object Tracking. In: 2017 IEEE International Conference on Computer Vision (ICCV), 2017. 1134–43.https://doi.org/10.1109/iccv.2017.128

[ref36] 36. Wang X, Shu X, Zhang Z, Jiang B, Wang Y, Tian Y, et al. Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 13758–68. https://doi.org/10.1109/cvpr46437.2021.01355

[ref37] 37. Bai Y, Zhao Z, Gong Y, Wei X. Artrackv2: Prompting autoregressive tracker where to look and how to describe. In: Proceedings of the IEEE/CVF confer-ence on computer vision and pattern recognition. 2024.
View Article
Google Scholar

[67] View Article

[68] Google Scholar

[ref38] 38. Xie J, Zhong B, Mo Z, Zhang S, Shi L, Song S, et al. Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 19300–9. https://doi.org/10.1109/cvpr52733.2024.01826

[ref39] 39. Cai W, Liu Q, Wang Y. HIPTrack: Visual Tracking with Historical Prompts. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 19258–67. https://doi.org/10.1109/cvpr52733.2024.01822

[ref40] 40. Bhat G, Danelljan M, Van Gool L, Timofte R. Learning Discriminative Model Prediction for Tracking. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 6181–90. https://doi.org/10.1109/iccv.2019.00628

[ref41] 41. Zhang Z, Peng H, Fu J, Li B, Hu W. Ocean: Object-aware anchor-free tracking. In: Proceedings of the European Conference on Computer Vision, 2020. https://doi.org/10.1007/978-3-030-58589-1_46

Figures

Abstract

1. Introduction

2. Related works

2.1. Visual tracking framework

2.2. Tracking based on dynamic design

3. Method

3.1. Overview

3.2. Image and sequence representation

3.3. Model architecture

3.3.1. Encoder.

3.3.2. Decoder.

3.3.3. Adaptive token division module.

3.4. Training and inference

3.4.1. Training.

3.4.2. Inference.

4. Experiments

4.1. Implementation details

4.2. Main results and comparisons

4.3. Ablation studies

5. Conclusion

References