3.1.1 Research area and subject selection.

The research was conducted at Youzhi Ranch, located in the Luquan District of Shijiazhuang City, Hebei Province, China. This site was chosen as the primary location for data collection and flight verification. Video recordings captured Holstein cows that exhibited well-developed hindquarters, wedge-shaped body profiles, distinct black-and-white markings, and characteristic white spots on the lower legs and tail. The designated sampling zone was outlined at an interval of 10 meters within aerial photographs (Fig 1a). The primary aim of the study was to observe the natural behaviors of Holstein cows without causing disturbances or making environmental changes. Following multiple sampling events, subjects were chosen based on criteria such as large body size, symmetrical shape, thin skin, slender skeletal structure, and minimal subcutaneous fat (Fig 1b).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the study area and selected study subjects.

(a) Aerial image of the study area captured by UAV. (b) Selected study subjects.

https://doi.org/10.1371/journal.pone.0331850.g001

3.1.2 Acquisition platform.

The experimental framework utilized the DJI Mavic 3T thermal imaging drone, a robust industrial UAV characterized by its compact design. The drone’s specifications include a bare weight of 920 grams and fuselage dimensions of 347.5 × 283 × 107.7 mm (length × width × height). It provides a maximum flight duration of 45 minutes, an operational range of 32 kilometers, a payload capacity of 1050 grams, and a service ceiling of 6000 meters, making it well-suited for a wide range of UAV applications. The platform includes three integrated camera systems: wide-angle, telephoto, and thermal imaging. The thermal camera offers a temperature detection range from −20°C to 150°C, with a measurement accuracy of ±2°C at the higher end of this range. Data processing was performed using the NVIDIA Orin NX computing platform, the specifications of which are detailed in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Nvidia Orin NX parameters table*.

https://doi.org/10.1371/journal.pone.0331850.t001

3.1.3 Data preparation.

To enhance the practical utility of the dataset, data collection was conducted from April to May 2024 across several Holstein cow farms in the Luquan District, Hebei Province. The UAV was equipped with an array of cameras including wide-angle, telephoto, and downward-facing thermal imaging capabilities, enabling the synchronized capture of RGB and thermal videos without necessitating post-processing. Recordings were executed during three distinct daily periods—morning, midday, and evening—to capture a diverse range of lighting and temperature conditions. A variety of environmental settings were purposefully selected, encompassing clear daylight, moderate fog (Fig 2a), low-light nighttime conditions (<5 lux) (Fig 2c), and scenarios featuring natural occlusions such as fences or overlapping cows (Fig 2b). The drone maintained altitudes between 10–15 meters and a maximum speed of less than 1 m/s to ensure the stability of the imagery. After excluding footage that was distorted or lacked the target, a collection of 75 high-quality MP4 videos (1920 × 1080 @ 25 fps), each spanning 25–35 minutes, was compiled.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Representative environmental conditions during data collection.

(a) Moderate fog, (b) Natural occlusions, (c) Low-light nighttime condition.

https://doi.org/10.1371/journal.pone.0331850.g002

To establish robust ground truth, keyframes were manually extracted and annotated to depict typical behaviors such as feeding, walking, resting, and excretion, as well as challenging conditions like occlusion and deformation. Annotations were performed independently by two trained experts using the LabelImg and VIA tools, with discrepancies being resolved through a review conducted by a third annotator. The final labels were stored in both YOLO and Pascal VOC formats, with each object instance tagged with one or more of seven difficulty attributes as per the Generic Thermal Object Tracking (GTOT) standard: occlusion (OCC), large scale variation (LSV), fast motion (FM), low illumination (LI), thermal crossover (TC), small object (SO), and deformation (DEF). The RGB–thermal paired image dataset was subsequently divided into training (6,300), testing (1,800), and validation (900) subsets, adhering to a 7:2:1 ratio to ensure balanced evaluation and effective model generalization.

3.3.1 Dual-stream siamese network for unimodal feature extraction.

The tracking framework incorporates a dual SiamCMA architecture to independently process RGB and thermal inputs. Although both networks maintain structural similarity, they utilize distinct parameters tailored to enhance feature extraction for each respective modality. Each Siamese neural subnetwork comprises two branches: the template branch and the detection branch. These branches are structurally identical and share parameters. The primary function of the template branch is to extract features from the template patch.

In the development of the feature extraction module for unimodal analysis, the architectural framework of the RGB branch of the SiamCMA was employed as a reference. This decision was prompted by the consistent architecture observed across all four branches of the dual-stream SiamCMA network. As depicted in Fig 4, the RGB SiamCMA template branch integrates a ResNet50 backbone with a Feature Pyramid Network (FPN).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Schematic depiction of the template branch of the RGB SiamCMA, consisting of a backbone network merged with a FPN.

https://doi.org/10.1371/journal.pone.0331850.g004

To address the performance constraints inherent in Siamese trackers, our investigation employs a methodology advocated by Li et al. [65], which incorporates the advanced deep learning architecture ResNet50 [66] as the primary backbone network. This choice is based on the premise that deeper networks can significantly enhance tracking performance. As depicted in Fig 4, the architecture we propose features several critical modifications to the standard ResNet50 design. A notable innovation in our approach involves eliminating the downsampling operations in the final two convolutional blocks (conv4 and conv5) by setting the stride to 1. This adjustment improves the resolution of the feature maps, thereby facilitating a more precise capture of fine-grained details within the template patch. Additionally, we incorporate dilated convolutions in these blocks with dilation rates of 2 and 4 for conv4 and conv5, respectively. This strategy increases the receptive field while maintaining feature map resolution. To further enhance efficiency, a 1 × 1 convolutional layer is added after each of the last three blocks (conv3, conv4, and conv5), reducing the channel count to 256. For computational optimization, feature extraction is confined to the central 7 × 7 region of the template, as empirical evidence suggests that this area harbors critical target features while minimizing computational demands. We intentionally omit features from the first two blocks (conv1 and conv2) due to their tendency to introduce noise, which impairs tracking performance.

The template (or detection) patch processes information through a three-tiered feature extraction mechanism within the backbone network. At the primary level, the extracted features concentrate on visual components such as edges, corners, colors, and shapes, which are crucial for precise object localization. Conversely, features at a higher level capture more semantic details essential for object identification. Integrating features from both levels markedly improves the PR of object tracking. To augment the feature extraction capabilities of the backbone network, an FPN was integrated into its last three blocks. This modification enables the integration of cross-level features derived from the hierarchical layers labeled }. In our network configuration, the } features, which maintain consistent spatial resolutions and channel counts, were amalgamated using a top-down approach to generate the FPN output for each corresponding layer, represented as }. The resulting FPN output was then designated as the output features of the template branch in the RGB SiamCMA, identified as = {}. Similarly, the output features of the detection branch in the RGB SiamCMA, denoted as = {}, were derived using a similar summation strategy. Upon securing these features from the respective branches, the CCFF module was devised to effectively fuse the features extracted from the dual-mode SiamCMA.

(1)

(2)

(3)

3.3.2 Complementary coupling feature fusion module (CCFF).

Building on the approach proposed in the study by [40], which introduced coupled convolutional filters to enhance modality interaction in RGB-T tracking, our work expands this concept through the design of a CCFF module. This module not only facilitates enhanced integration of cross-modal features but also incorporates adaptive weighting to augment tracking robustness under challenging conditions. Our CCFF module employs coupled filters within the convolutional layers at a coupling ratio of 0.5. This configuration enables the filters to concurrently update the weights of both visible and thermal infrared features, thus optimizing cross-modal feature integration. As depicted in Fig 5, the CCFF module integrates coupled convolutional filters to concurrently enhance RGB and TIR features. Each convolutional layer is equipped with both coupled and uncoupled filters, each having a kernel size of 3 × 3. The coupled filters undergo updates twice per iteration, generating weight maps that regulate the extent of information exchange between modalities. These weights are normalized to a range of [0,1] using a sigmoid function, which facilitates the enhancement across modalities. Subsequently, the enhanced features are concatenated and processed through a 1 × 1 convolutional layer to standardize the channel dimensions, thereby creating a robust fused representation. For instance, within the template branch, the weight maps were derived as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. CCFF module, adaptively integrating RGB and thermal infrared features via coupled and uncoupled convolutional filters.

https://doi.org/10.1371/journal.pone.0331850.g005

(4)

(5)

where conv(*,) signifies a convolutional layer characterized by parameter θ, encompassing both uncoupled and coupled filters with identical parameters. Here, (·) denotes the sigmoid function. Following the computation of the weight map, both visible light and thermal infrared features are enhanced through cross-modal interactions involving and .

(6)

(7)

Subsequently, the enhanced features and were obtained, and their fusion was accomplished by concatenation. A 1 × 1 convolutional layer was then utilized to integrate the channel information, resulting in a combined feature.

(8)

3.3.3 Multi-modal weight penalty module (MWPM).

The MWPM algorithm assesses the characteristics of both modalities in a unified manner and assigns weights to the deep features based on this comprehensive evaluation. To compute the attention weights, MWPM utilizes scaling factors obtained from BN [28], which function to suppress less significant features by incorporating regularization terms. These scaling factors, within the channel attention submodule, quantify the degree of variation in each channel, thus indicating the channels’ relative significance.

(9)

Here, denotes the normalized eigenvalue, while and serve as adjustable reconstruction parameters. These parameters facilitate the network’s ability to adaptively restore the feature distribution akin to that of the original network. The application of the scaling factor, derived from the variance observed in BN, follows a specific rationale: a higher variance indicates a more substantial change in the channel, suggesting that the information within that channel is of greater importance, whereas channels with less variance carry less critical information.

(10)

(11)

In Equation (10), the input feature, denoted , undergoes processing to produce the output feature, labeled , with acting as the scaling factor for each respective channel. By employing a consistent normalization strategy across all pixels within the spatial domain, we can establish the spatial attention mechanism as delineated in Equation (11).

As depicted in Fig 6, the RGB and TIR features are combined along the channel dimension to create a cohesive representation of both the template and search features within the channel attention module.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The MWPM connects the deep features of the RGB mode and the thermal infrared mode, and then assigns weights to all channels.

https://doi.org/10.1371/journal.pone.0331850.g006

(12)

Subsequently, is penalized with characteristic weights, and the resulting output is expressed as follows:

(13)

The function δ(·) denotes the sigmoid activation, while “Split” refers to the division of features according to channel size. In the integration of the two submodules, channel attention is initially applied, followed by spatial attention. The channel attention mechanism reduces the prominence of less pertinent feature channels, while the spatial attention mechanism aims to attenuate background noise.

Following the processing by the attention module, the enhanced feature images of the template and search images are further processed by MWPM. Initially, the algorithm correlates the two branches of visible light to produce the response map R2. A parallel correlation is conducted on the two branches of thermal infrared, resulting in another response map, denoted R3. Through a tri-layer channel compression process, certain features are extracted, followed by the fusion of R1*, R2*, and R3*. Additional channel compression subsequently leads to the final feature map, represented as R-final (25 × 25 × 256).

(14)

3.3.4 Reward-driven learning mechanism.

In this section, we discuss the application of reinforcement learning for optimal feature selection. By leveraging the feature map generated by the score branch of a Siamese-based tracking model, the complexity of the input image was effectively reduced. This reduction was achieved through the maximization of input simplification directed towards the reinforcement learning model. Subsequently, the features extracted from the score map were utilized as input variables for the reinforcement learning model.

1. (1). Action

The state was defined by utilizing both the score map and the movement direction of the object. The score map was modified by integrating the object’s movement direction as a weighting factor. Upon receiving the target image, denoted as z, and the search image, referred to as x, as inputs, the score branch identified the position with the highest score. Action A is delineated as a 9-dimensional vector, comprising the central position as defined by Equation 15 and the eight adjacent positions within the same channel as delineated by Equation 16. Here, a₀ = (m − 1, n − 1), a₁ = (m − 1, n),..., a₄ = (m, n),..., a₇ = (m + 1, n), a₈ = (m + 1, n + 1).

(15)

(16)

The output feature map produced by the score branch was processed using the softmax function to derive score probabilities, which were then employed to select an action for training. The chosen index was subsequently inputted into either a bounding box or a mask branch, where a bounding box was predicted to delineate the object’s location corresponding to that index.

1. (2). Status

As defined by Equation 17, the term “State S” encapsulates a combination of a score map and a vector indicating the object’s movement direction.

(17)

The score map serves to condense the information present in the original image. The orientation of the object was inferred by estimating the bounding box based on the previously selected action. The motion direction, represented by a unit vector, was derived from the position of the bounding box. The trajectory of movement over the previous 10 frames up to the current frame was formulated as a vector. In Equation 17, denotes the similarity score map, and refers to the vector encapsulating the movement direction of the bounding box. Consequently, the state included the location details of the area that exhibited the highest similarity to the target.

1. (3). Reward

In numerous tracking algorithms that incorporate offline learning, a tracking error at the onset of a sequence may precipitate cumulative errors, ultimately causing the tracker to erroneously follow an incorrect object or background element. The reward function within our model was established based on the Intersection over Union (IoU) between the actual bounding box () and the predicted bounding box () observed in the final frame of the sequence. This design aims to foster effective learning even when the object is partially occluded (PO). Consequently, the reward was structured as described in Equation (18), paralleling the ADNet approach, where an IoU of 0.7 or greater is achieved at the conclusion of the sequence.

(18)

1. (4). State Transition

The process of selecting an action within a particular state initiates a transition to the next state. This state transition is dictated by a function that is contingent on the action selected, as demonstrated in Equation (19).

(19)

1. (5). Realization

Within the architectural framework of our reinforcement learning model, feature extraction is executed through two layers of 3x3 convolution applied to the score map. This is followed by a dense layer that produces outputs corresponding to nine predefined actions. Over a period of 10 frames, the average movement direction, denoted as , is calculated as the mean vector of directional movements. The decision-making mechanism involves a point-wise multiplication of the weight vectors (k ∈ [0,8]) with the average direction across the score map to select an action, as illustrated in Equation (20) of Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Implementation of the reinforcement learning-based dynamic tracking decision mechanism integrating score maps and motion direction.

https://doi.org/10.1371/journal.pone.0331850.g007

(20)

In Fig 7, when the prevailing movement direction is to the right, the algorithm assigns an enhanced weighting to this rightward direction (). This decision leverages an average of directional data over the preceding 10 frames to compute actions on the score map. For assessing tracking failures, the method of dynamic template switching is employed, which considers the tracking output that achieved the highest score in the previous frame and uses it as the template for the current frame. This template is then correlated with the search image in the correlation layer of the target image. The tracking result that receives the highest score during this process is designated as the definitive output. This dynamic switching method is designed to activate after several frames, thereby accommodating the initially low probability of target loss and minimal target variation.

1. (6). Train

In instances of tracking failure, the dynamic template switching strategy involves selecting the highest scoring tracking result from the preceding frame as the new reference template. This template is then correlated with the search image in the correlation layer to pinpoint the target. The highest scoring result is subsequently adopted as the definitive output. Given that the initial stages of tracking typically present minimal risk of target loss and that significant changes to the target are unlikely, this method is implemented at specific intervals following the commencement of tracking. The experimental conditions were established using a randomly selected sequence, and the learning parameters were adjusted based on the rewards obtained from the final frame of that sequence.

In the final frame of the sequence associated with the environment, the reward was secured during the training phase. Consequently, as illustrated in Equation (21), the implementation of Stochastic Gradient Ascent (SGA) within ADNet facilitated the necessary parameter updates for training the reinforcement learning model.

(21)