Background

Simultaneous Localization and Mapping (SLAM) represents the forefront of spatial localization technology and has exhibited remarkable progress over the last three decades. SLAM involves the process in which a robot or vehicle equipped with a specific sensor constructs a model of its surrounding environment while in motion, without prior knowledge of the environment. It simultaneously estimates its own movement, enabling autonomous positioning and navigation. Visual SLAM (VSLAM) has emerged as a prominent area of research due to its cost-effectiveness, ability to capture abundant information, and ease of installation, attracting substantial attention in the research community [1]. Finding representative image features from images is a key research problem for visual SLAM [2]. The extraction and matching of image feature points play a crucial role in various tasks, including image retrieval, object tracking, 3D reconstruction, image registration, pose estimation, map creation, and loop detection. The significance of feature point detection-based image matching methods has been increasingly emphasized with the advancement of visual SLAM technology [3, 4].

Current challenges or limits of prior SOTAs

Historically, researchers have developed many prominent feature extractors and descriptors, each of which exhibits different performance in terms of rotation and scale invariance as well as speed [5]. David Lowe introduced the Scale Invariant Feature Transform (SIFT [6] method in 1999, and its subsequent refinement in 2004 has established it as a highly renowned and influential feature extraction algorithm. SIFT features are local image descriptors that maintain robustness against a wide range of transformations, including rotation, scale changes, viewpoint changes, lighting variations, affine transformations, and noise. However, SIFT features obtain the local extreme values of the space through a pixel-by-pixel traversal in DOG (Gaussian First-order Difference) space. Even though SIFT features perform well in most cases of transformation, its large computational cost leads to low extraction efficiency, which seriously limits its application in real-time systems. Meanwhile, the effectiveness of SIFT may also be affected in images with strong noise or complex background. Herbert Bay proposed the Speeded-Up Robust Features (SURF) [7] algorithm in 2006 by referring to the SIFT algorithm. Compared with SIFT, the algorithm uses Hessian matrix and dimensionality reduction feature descriptors, which improves the execution efficiency to some extent. SURF overcomes the limitations of SIFT in working in high-dimensional environments and is also robust to lighting changes. However, it is important to note that this advantage comes with a trade-off, as it has lower performance in terms of resource consumption. Additionally, while SURF has improved efficiency, it is still limited for real-time applications due to its computational complexity. This method was applied to the first monocular visual SLAM (Mono SLAM) [8] system proposed by Davison et al. in 2007. In 2006, Edward Rosten first proposed the FAST (the Features from Accelerated Segment Test) algorithm [9], which uses the gray of each pixel in a particular region and the gray of the center to determine whether the central point is a corner, greatly improving speed. Due to the fact that the algorithm only detects corner points, it does not have scale invariant and rotation invariant. Compared with SIFT and SURF algorithms, this algorithm is susceptible to noise and has lower accuracy.

Binary Robust Independent Elementary Features (BRIEF) [10, 11] is a description of the detected feature points, which is a binary encoded descriptor, and these comparisons produce a binary string that can be matched very quickly using the exclusive OR (XOR) operation. BRIEF is constructed using comparisons of the intensities of random pixel pairings within a patch centered on the feature point and discards the traditional method of describing feature points using a regional gray histogram, so it is much faster than SIFT-like descriptor extraction [12]. In the field of computer vision, the research community has introduced numerous binary descriptor algorithms in recent years, aiming to enhance the performance of image matching algorithms. Binary descriptors leverage efficient bit operations to expedite the process of feature point matching. Compared with non-binary descriptors, binary descriptors offer significant improvements in computational efficiency and storage space. The Oriented FAST and Rotated BRIEF (ORB) matching algorithm, initially proposed by Rublee et al. [13], is a fusion of the FAST feature detection algorithm and the enhanced BRIEF feature description algorithm. The ORB algorithm generates binary descriptors by employing machine learning-based selection of point pairs, while incorporating rotational invariance by calculating the corner points’ orientation using the gray centroid. It is about 100 times faster than SIFT algorithm and 10 times faster than SURF algorithm [13]. Traditional ORB algorithms, however, do not provide good matching quality and anti-interference capabilities.

The advantage of the ORB algorithm is the fast computation speed, however, the stability of the ORB algorithm is weaker than SIFT descriptor and SURF descriptor, but considering that the visual SLAM algorithm needs high real-time performance, ORB feature points are more suitable for feature point SLAM algorithm. Since the ORB algorithm pursues the speed advantage too much, the ORB algorithm does not work well on scale invariance, illumination invariance, and rotational invariance, which will lead to a large number of mismatches. Numerous enhanced image-matching algorithms have been proposed based on the ORB algorithm. For instance, BRISK (Binary Robust Invariant Scalable Keypoints) [14] is a feature detection and description algorithm introduced by Leutenegger et al. at the 2011 ICCV, which is known for its robustness to changes in scale, rotation, and viewpoint, making it suitable for various image matching and recognition tasks. The FREAK algorithm [15] was presented at the 2012 International Conference on Computer Vision and Pattern Recognition, which is based on the BRISK algorithm with the introduction of feature descriptors in binary. The FREAK algorithm not only has the characteristics of high matching rate and high operation rate than the traditional detection algorithm, but also has the scale transformation invariant property of stronger stability and stronger robustness. The feature descriptors of the FREAK algorithm are made up of the central feature point and the sampled feature points combined. In 2015, according to Yang and Cheng [16], the Local Difference Binary (LDB) algorithm is proposed as an ultrafast and distinctive feature description method that enhances traditional binary descriptors by incorporating gradient information in both horizontal and vertical directions. LDB partitions feature point regions into different-sized sub-blocks to capture local information, enabling smaller subblocks to gather localized information for improved descriptor discrimination. Conversely, larger subblocks effectively suppress noise but may result in reduced distinctiveness due to decreased sensitivity to subtle changes. In 2017, Levi et al. proposed Learned Arrangements of Three Patch Codes (LATCH) [17]. LATCH improves upon the original local binary feature description by utilizing multi-image block F norm comparison instead of pixel-level gray level comparisons employed by ORB. This modification enhances the algorithm’s robustness against noise, contributing to improved anti-noise performance.

Traditional sparse feature extraction methods typically adopt a “detect-then-describe” strategy, where keypoint detectors [6, 7, 18–20] are used to extract keypoints, and feature descriptors [6, 7, 10, 13, 14] are subsequently extracted from these keypoints. However, in recent years, deep learning-based methods have gained attention, and some of these methods integrate the learning of descriptors and detectors. LIFT (Learned Invariant Feature Transform) [21] is the first fully learning-based architecture that reconstructs the key steps of the SIFT algorithm using neural networks to achieve stable feature extraction. SuperPoint [22], inspired by LIFT, treats keypoint detection as a supervised task. In the initial stage, it trains the model using labeled synthetic data to perform keypoint detection before description. Later, SuperPoint further extends to an unsupervised version [23]. DELF (Deep Local Features) [24] and LF-Net (Local Features Network) [25] employ distinct methods for unsupervised learning, with DELF utilizing attention mechanisms, while LF-Net adopts an asymmetric gradient backpropagation scheme. D2-Net [26], unlike previous approaches that separately learn descriptors and detectors, devises a joint optimization framework based on non-maximum suppression, allowing simultaneous optimization of descriptor and detector performance. To enhance the reliability and repeatability of keypoints, R2D2 [27] introduces a list-wise sorting loss function based on differentiable mean average precision to improve keypoint ranking. Similarly, aiming for the same objective, ASLFeat [28] introduces deformable convolutions to enhance the reliability and repeatability of keypoints. F. Luo et al. [29] propose a multiscale diff-changed feature fusion network (MSDFFN) for change detection (CD) in hyperspectral images (HSI). This network enhances the feature representation by learning fine-grained change components between dual-temporal HSI at different scales. In 2013, Wang et al. [30] introduced FeatureBooster, a lightweight network designed to enhance keypoint descriptors within the same image. Utilizing both MLP-based (Multi-layer perceptron) and Transformer-based stages, the network boosts the performance of both hand-crafted and learning-based descriptors. Wang and Liu [31] developed SSG-Net, an innovative approach for registering multimodal images. This technique incorporates an adaptive training framework and initially employs both the SuperPoint and SuperGlue networks for coarse alignment. Subsequently, the DEGENSAC algorithm is applied to identify the best registration points, using a judiciously chosen threshold.

In 2017, the quadtree approach was employed by ORB SLAM 2 for the homogenization of ORB feature extraction [32, 33]. While this strategy undeniably enhances the uniformity of feature point distribution, there is still room for improvement in terms of efficiency. Building upon this approach, YU Xinyi et al. [34] proposed the Qtree-ORB algorithm in 2018, which effectively eliminates redundant feature points. However, this method still relies on the traditional quadtree structure, necessitating further enhancements to improve computational efficiency. In order to extract additional feature points, Xin-Nan et al. [35] adopted adaptive thresholding in 2019 as opposed to fixed thresholding in the original ORB method. Nonetheless, the aforementioned approach requires the computation of a threshold value on a per-pixel basis, leading to increased computation time. Ma et al. [36] presented a feature description technique aimed at enhancing the accuracy of the ORB algorithm in 2020. Their method incorporates local grayscale difference information alongside the original grayscale intensity information. In 2013, Zhu et al. [37] developed an improved ORB-RANSAC algorithm tailored for Unmanned Aerial Vehicle (UAV) remote sensing image registration. The method employs a simplified image pyramid for scale invariance and an adaptive FAST corner detection algorithm. It also utilizes bidirectional matching and an enhanced RANSAC for fine feature point matching.

While traditional algorithms like SIFT and SURF have been foundational in the field, they often suffer from computational inefficiency and lack of robustness in varying conditions. Deep learning-based methods like LIFT and SuperPoint have shown promise in terms of performance but require large amounts of labeled data and computational resources. The integration of these methods into existing SLAM systems remains an open challenge.

Research motivation

The motivation behind this research is centered around addressing certain limitations associated with the ORB (Oriented FAST and Rotated BRIEF) algorithm’s feature point detection and description techniques. Through a comprehensive analysis of the ORB algorithm, several noteworthy shortcomings have been identified:

1. The ORB algorithm incorporates the FAST method for feature point detection, which examines the gray value in the vicinity of each candidate feature point. By analyzing the pixel values within a circular region centered on the candidate point, the algorithm determines if there are sufficient pixel points exhibiting significant deviations from the grayscale intensity of the candidate point. The FAST method, on the other hand, employs a randomly chosen globally fixed threshold, which results in the identified feature points being too concentrated or even overlapped. In regions where the picture texture is not visible, this may result in the extraction of only a limited number of feature points or none at all. Such dense distributions of feature points can significantly impact system accuracy in various scenarios, including image navigation, video tracking, and SLAM systems [38].
2. The ORB algorithm employs the enhanced BRIEF algorithm to compute feature point descriptors. The BRIEF algorithm generates binary descriptors by comparing the intensities of two points within a window centered on the feature point. However, this process solely relies on the size relationship between gray values, thereby neglecting the utilization of valuable image information contained in the pixel intensity differences. This limitation leads to a potential loss of image information.

Addressing these limitations is of paramount significance in bolstering the precision and performance of the ORB algorithm within image matching. This enhancement in capabilities renders the algorithm more adept at tasks like image navigation, video tracking, and SLAM systems. The core objective of this research is to architect strategic measures that effectively mitigate these challenges, thereby contributing substantially to the advancement of cutting-edge techniques in feature point detection and description.

Main contributions

This paper aims to address the aforementioned limitations of the ORB algorithm. The main contributions of this study can be summarized as follows:

1. In the ORB SLAM system proposed by Mur-Artal [32, 33], a fixed two-threshold approach was used to detect FAST feature points. A higher threshold of 20 was first applied for feature point detection. If no points were detected with this threshold, a lower threshold of 7 was then used. However, this dual-thresholding method relied on preset fixed threshold values without considering variations in local pixel intensities. As a result, it may lack robustness under changing lighting conditions compared to an adaptive thresholding approach. To address this limitation, we propose an adaptive thresholding method based on the analysis of image grayscale contrast. This enhancement improves the feature point extraction capability of the original ORB algorithm, especially in regions with weak textures.
2. In the work by Mur-Arta [32, 33], a quadtree approach was employed to address the issues of over concentration and overlapping of feature points. However, the quadtree splitting process is based on the total number of feature points extracted from each pyramid image layer. This approach may lead to a decrease in computational efficiency, as it requires further sorting and splitting of nodes when the total number of feature points can still be satisfied. To improve the efficiency of feature extraction, our proposed method reduces the estimation of redundant feature points while enhancing the overall feature extraction efficiency.
3. During the feature point description stage, we propose a novel approach to form a binary descriptor. Building upon the descriptor of ORB SLAM 2, we introduce a Gaussian blurring step followed by a comparison of the average gray size of pixel blocks in BRIEF pattern point pairs to generate the binary descriptor. The pixel block difference between feature point pixel blocks and BRIEF pattern points is calculated to form a dynamic threshold, and then the gray difference between the pattern pixel block pairs is compared with the dynamic threshold to form a new binary code string. The two feature descriptors mentioned above are fused into a 64-bit binary descriptor, which makes the feature descriptors less sensitive to noise.

Our work addresses the existing gaps by introducing an adaptive thresholding method for feature point detection and a novel binary descriptor formation technique. Unlike traditional methods, our approach considers local pixel intensity variations, making it more robust under varying lighting conditions. Additionally, our feature descriptor is less sensitive to noise, addressing a common limitation in existing binary descriptors. This work represents a significant step towards achieving real-time, accurate, and robust feature point detection and description, particularly in the context of SLAM systems.

oFAST algorithm

FAST is an efficient feature point (corner point) detection algorithm that does not involve feature point description. Taking the candidate feature point p as the central reference of the circle, the 16 pixels of the circle formed by the radius of 3 pixels around it are calculated and compared with the point p. When there are enough pixel points with grayscale values that differ from the center point p, the image point p may be a corner point. The algorithm consists of the following steps:

1. Choose any point p in the image and assume the intensity is I_p. A circle is drawn with the center of point p and a radius of 3. In Fig 1, the circle consists of 16 pixels. A candidate corner point p is identified if there are N consecutive pixels (usually 12 pixels) on the circle template that are either brighter than a threshold value plus t or less bright than I_p − t. To optimize efficiency in practical applications, only the pixel differences between the center p and the four pixels at positions 1, 5, 9, and 13 (representing the four cardinal directions) are examined for corner detection. If at least three of their absolute values exceed the threshold value, they are further compared with other pixels; otherwise, they are directly discarded. All pixel points in the image are traversed using the above method to obtain the candidate feature points for initial screening
2. Non-maximum suppression of the image: To identify if there are multiple feature points in a neighborhood (e.g., 3x3 or 5x5) centered on feature point p, a comparison is made. In this comparison, the FAST score value (referred to as the s-value) is calculated separately for each feature point. The s-value is obtained by summing the absolute differences between the 16 points and the center. Subsequently, the feature point with the higher response value is preserved while suppressing the others. The score calculation formula is as follows (V is used in the formula to denote the score and t to denote the threshold value). (1) where: S(p) is the cumulative accumulation of pixel intensities that are greater than or equal to p, and S(v) represents the accumulation of pixel values that are less than p.
3. To achieve scale invariance, an image pyramid is constructed. Feature points are extracted at each level of the pyramid. These feature points from different levels are then mapped back to the original image, resulting in a collection of oFast feature points for the entire image. This approach ensures that the selected feature points maintain scale invariance.
4. The intensity centroid method is used to determine the feature point orientation and ensure the image rotational invariance. The specific calculation method is as follows: To ensure image rotational invariance and determine the orientation of feature points, the intensity centroid method [39] is employed. The calculation method for this purpose is as follows: (2) The centroid of an image block can be determined using moments. (3) The direction vector , which is obtained by connecting the geometric center O of the image block with the centroid C, represents the orientation of the feature point. (4) To ensure rotational invariance, a common practice is to use an image block size of 31 31 pixels.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. FAST corner extraction diagram.

https://doi.org/10.1371/journal.pone.0293111.g001

rBRIEF feature descriptor

After acquiring feature points with orientation, BRIEF is employed to calculate the binary descriptors for these points. Within an image block, a localized self-centered patch serves as the space for x and y point pairs. A binary test τ is defined using the following equation. (5) where p(x) represents the intensity of p at a point x.

By comparing the intensities of the two points, the features are encoded as n-bit binary vectors according to the binary descriptor: (6)

The ORB algorithm improved the BRIEF descriptor. First, using the integral image in ORB, random point pairs are selected in a 31×31 pixels patch, and the selected random points are used as the center to calculate the gray mean value in a 5×5 sub-window, and the neighboring gray mean values of the random point pairs are compared for binary encoding, instead of deciding the encoding result only by the grays of two random point pairs, which can effectively solve the noise-sensitivity problem. Meanwhile, the BRIEF algorithm does not have rotational invariance, thus ORB adds a rotation factor called rBRIEF based on the original.

For an n-bit test binary feature description, define a 2 × n matrix: (7)

The corresponding rotation matrix R_θ of angle θ is obtained by Eq (8). Subsequently, the matrix S is rotated by R_θ. (8)

In this case, a binary test descriptor with direction is obtained, which is defined as follows. (9)

In the above algorithm, the intensity of 256 pairs of pixel points is generally compared to generate a 32-byte feature descriptor.

Adaptive threshold determination

The traditional FAST algorithm employs a fixed threshold for feature point extraction, with more feature points extracted as the threshold decreases and vice versa. However, in the ORB algorithm, a predefined global threshold is artificially set for FAST, resulting in the extraction of numerous redundant feature points in regions with clear image changes. While a fixed threshold simplifies calculations and reduces processing time, it falls short in meeting the requirements of feature point extraction in various scenes. In scenarios where the image quality is poor or significant changes occur in the external environment, the number of extracted feature points can sharply decrease. This reduction can lead to map distortion or image stitching failure. To address this issue, Mur-Artal [32, 33] introduced a fixed dual-threshold method for feature extraction. When the high threshold fails to extract feature points, the low threshold is used for feature extraction. This dual-threshold mechanism effectively solves the problem of insufficient feature point extraction under challenging conditions, ensuring a more robust and reliable feature detection process. However, the fixed artificially set threshold value still has some limitations and cannot be well adapted to different scenes.

The algorithm for the proposed adaptive threshold method is as follows. Similar to the grid-based approach used in the ORB SLAM2 system, we divide the image into grids and calculate the threshold for each grid using the following method. (10) where and are the i-th largest gray value and the i-th smallest gray value within each grid of the image, respectively; n is the first n largest (smallest) gray values in the image, and n is taken as 10.α is the scale factor, generally taken as 0.15 to 0.30, which represents 15% to 30% of the absolute contrast of the image.

Feature point homogenization strategy

The original ORB algorithm tends to focus on regions with rich texture, resulting in feature point aggregation and redundancy in areas with less or no texture. To address this issue and achieve a more uniform distribution of feature points throughout the image, this article employs the following homogenization methods:

• Step 1. In order to obtain multi-resolution scale information, we convert the input RGB image to grayscale, build the image pyramid, and use the Adaptive Threshold Method proposed earlier in this article to extract feature points on each layer. The number of feature points for each layer is: (11) In the equation, N, n, N_α, and s represent the expected total feature points, the pyramid layers, the feature points to extract, and the inverse scaling factor of each pyramid image, respectively. We use empirical values of 8 and 1.2 for n and s.
• Step 2. Use the whole image as the initial extraction node for quadtree segmentation. Generally, for a 640*480 pixel image, the node count is initially 1.
• Step 3. Identifying all nodes in the image. If a node has no feature points, it is deleted. If it has only one feature point, the node is considered as a terminal node and is not further divided. However, if a node has more than one feature point, it continues to split into four children nodes.
• Step 4. Continue executing Step 3 iteratively until the number of nodes in the image reaches the specified number of feature points N_α, and terminate the splitting process at that point.
• Step 5. Select the most salient feature point in each node using the Harris response value. If a node has more than one feature point, pick the one with the highest value as the node’s representative point.

However, the above steps do not limit the quadtree split depth, which increases the algorithm’s computation time. To reduce redundant feature computation, we set different depths based on the total feature points extracted from each pyramid layer. Algorithm 1 shows the detailed process. In Algorithm 1, I_ter is the current quadtree’s iteration number or split depth; D_max is the maximum depth for the current pyramid layer; N_kp is the node feature point count; bFinish is the flag to stop splitting; and N_store is the node storage count. The quadtree’s maximum depth should follow this equation: (12) I_ini indicates the number of initial extraction nodes. Depending on the aspect ratio of the input image, I_ini is usually 1 or 2. For example, the image in the KITTI dataset is 1241*376 pixels, so the initial number of nodes is 2.

Algorithm 1: Improved quadtree method algorithm

Data: Input: Image

Result: Output: ORB Keypoints Extracted

1 Construct the image pyramid;

2 for each pyramid layer i do

3  Adaptive threshold-based ORB feature point extraction;

4  Split quadtree nodes on level i pyramid;

5  for each subnode j do

6   while !bFinish do

7    if N_kp(j) > 1 and I_ter < D_max then

8     Divide child quadtree nodes;

9    end

10    if N_kp = 1 then

11     Store child node;

12    end

13   end

14   if N_store > N_α then

15    bFinish = true;

16   end

17  end

18  Retain features with large Harris response value;

19 end

Feature description method combined with gray difference information

Our work references patch-based approaches in recognition. The texture segmentation system in [40] demonstrated the ability to detect local texture features by analyzing the cross-correlation of a central patch with nearby patches. In [41], a descriptor was formed by comparing a central patch with surrounding patches, thereby extending the shape-context [42] descriptor for intensity images. This descriptor has shown strong invariance to image style and local appearance.

In [43], two related descriptor families were proposed, namely the Three-Patch LBP descriptor and the Four-Patch LBP descriptor. LBP, initially introduced as a global representation for the entire image by [43–45], has proven to be a successful method for describing texture features. One specific application is the generation of TPLBP (Three-Patch Local Binary Pattern) codes, where the values of three patches are compared to generate a single bit value in the assigned code for each pixel. Similarly, FPLBP (Four-Patch Local Binary Pattern) codes are generated by comparing the values of four patches to produce a single-bit value in the assigned code for each pixel. While the algorithm proposed in [43] is primarily designed for face recognition and may not be directly applicable to image matching, its core concept involves generating feature descriptions by comparing pixel patches. Furthermore, in [36], the ORB-TPLGD algorithm utilizing the Three-Patch Method and local gray difference was introduced. Chaoqun Ma et al. [46] proposed a Quadtree ORB (QTORB) algorithm to enhance the feature point extraction ability in uniform regions. This paper refers to the feature description method that combines gray difference features from [46], and redesigns the algorithm by incorporating the gray information of the feature points themselves. Hence, in this paper, we used a feature descriptor creation method that utilizes both grayscale intensity values and pixel patch difference information.

In the feature description phase, the ORB algorithm’s descriptors primarily focus on the size relationships of random pixel dot pairs within the local area of the feature point. This means that the distribution of size relationships among the selected pixel dot pairs is crucial. However, the algorithm overlooks the actual gray values of the pixels themselves and only takes into account the size relationships of the random dot pairs.

In this paper, based on the improved BRIEF in ORB, we propose to use the size and the difference of gray values for feature point description. We introduce the intensity difference information to construct the descriptors and focus on the gray information around the feature points. The specific schematic diagram is shown in Fig 2. Assuming there is a candidate feature point P on the smoothed image, a specific pattern is used to select n pairs of pixel blocks for grayscale comparison, resulting in the generation of binary codes, denoted as f_n(p).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Binary descriptor of the combination of gray size and gray value difference information.

https://doi.org/10.1371/journal.pone.0293111.g002

Additionally, to generate another binary code, the average difference between each pixel block and the feature point pixel block (Fig 2) is calculated. The gray difference H_i can be defined as: (13) where g denotes the grayscale sum of the pixel block and denotes the average gray level of the pixel block. Each feature point has n corresponding gray differences. Binary code conversion requires non-floating point data. Therefore, to generate a binary code, it is necessary to determine a threshold value and compare the gray difference value with this threshold. By calculating the gray differences and evaluating them against the threshold, the binary code for each feature point can be derived. The threshold value T_average can be defined as: (14)

Compare n gray difference and threshold T_average: (15)

According to Eq (15), f_n(p)^∧ is generated: (16)

In the procedure depicted in Fig 2, the two binary descriptors, f_n(p) and f_n(p)^∧, are merged together to form a novel binary descriptor. (17)

The overall workflow of our algorithm

In this paper, the ALGD-ORB algorithm enhances three key stages of ORB: feature extraction, feature point allocation, and feature point description. The complete procedure of the algorithm is outlined in Fig 3, while Algorithm 2 provides a breakdown of the specific steps involved in ALGD-ORB.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Illustrates the comprehensive workflow of the ALGD-ORB algorithm.

https://doi.org/10.1371/journal.pone.0293111.g003

Algorithm 2: Detailed explanation of ALGD-ORB operation

Data: Input: Image

Result: Output1: Keypoints, Output2: Keypoints description

1 Pre-compute the scale pyramid based on the input gray image. The total number of pyramid layers and the scaling factor are set to empirical values of 8 and 1.2

2 Gridding each layer of the pyramid image. Set the size of the image grid to, calculate the adaptive threshold Th according to Eq 10, traverse and calculate the feature points in all grids of all pyramid images, and note that the extracted N feature points are p = {p₁, p₂,…, p_i, …, p_N};

3 Use the improved quadtree method to average and distribute feature points in each image pyramid layer;

4 The number of all feature points after homogenization by the improved quadtree algorithm is denoted as K. Then P = {P1, P2,…,PK};

5 for i = 1, 2, …, K do

6  Define a 31 × 31 pixel image block M centered on P_i;

7  Select random point pairs in, and the coordinates of each random point relative to the feature point are R = {R_t}_t=1…n = {[(x_t1, y_t1), (x_t2, y_t2)]}_t=1…n;

8  Define 5 × 5 pixel patches A_t1, B_t1, P_i centered on point (x_t1, y_t1), (x_t2, y_t2) and p_i;

9  for t = 1, 2, …, n do

10   if g(A_t1) < g(B_t2) then

11    Assign 1 to corresponding bit;

12   else

13    Assign 1;

14   end

15  end

16  Calculate the gray difference H_i and threshold T_average;

17  for t = 1, 2, …, n do

18   if H_i > T_average then

19    Assign 1 to corresponding bit;

20   else

21    Assign 1;

22   end

23  end

24  The final Description(p) = f_n (p) + f_n (p)^∧ for each feature point is obtained by joining f_n (p) and f_n (p)^∧;

25 end

Evaluation metrics

In the experiments, four evaluation criteria were utilized to assess the algorithm’s performance. The first criterion is Evenness [47], which involves dividing the image into 10 regions: top, bottom, left, right, top left, bottom right, top right, bottom left, center, and periphery. These regions are derived from four directions (vertical, horizontal, 45°, and 135°) as well as the center and periphery (as depicted in Fig 4). The number of feature points within each region is counted, and the variance (V) of the group is calculated based on these data. Subsequently, the uniformity is determined using a specific formula. (18)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Schematic diagram of five-directional segmentation.

https://doi.org/10.1371/journal.pone.0293111.g004

A smaller value indicates a more balanced distribution of feature points across different regions, indicating better uniformity in their distribution.

The second evaluation metric is precision [48], which calculates the ratio of the number of correct matches Num_{correctmatches} to the total number of matched points Num_allmatches. The equation for precision is as follows: (19)

The root mean square error (RMSE) is a third evaluation metric that measures the overall positional accuracy between detected and true matching points in image matching. By calculating the differences between matched point pairs, RMSE quantifies the error between the matching results and the ground truth. A lower RMSE value indicates a higher accuracy and stability of the image matching algorithm. Consider a pair of matching points A_a(x_a, y_a) and B_b(x_b, y_b) obtained by an image matching algorithm. Here, A_a(x_a, y_a) represents a feature point in image A, and B_b(x_b, y_b) represents a feature point in image B. The true position of feature point A_a(x_a, y_a) in image B is denoted as . The RMSE calculation formula for this pair of points is as follows: (20)

The fourth evaluation metric focuses on the computational efficiency of the image matching algorithm. It includes various processing times, such as feature point extraction, descriptor generation, and matching. Notably, tasks like image loading and the calculation of other algorithm performance metrics are excluded from this evaluation.

Oxford optical image dataset

To validate the proposed algorithm’s effectiveness, we utilized images from a publicly available database established by K. Mikolajczyk and C. Schmid [48]. This comprehensive database comprises eight distinct image categories, encompassing variations such as blur (Trees and Bikes), lighting (Leuven), JPG compression (Ubc), scale and rotation (Bark and Boat), and visual enhancements (Wall and Graf). Refer to Fig 5 for a visual representation of these categories. Within each category, the first image can be paired with any of the other five images to form input image pairs. The dataset provides the homography matrix (referred to as H1to2p) between the two images in each pair, enabling the calculation of ground truth information for comparison purposes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Optical images from Oxford dataset.

(a) Bikes (Blur) (b) Trees (Blur) (c) Graf (Viewpoint) (d) Wall (Viewpoint) (e) Bark (Zoom & Rotation) (f) Boat (Zoom & Rotation) (g) Leuven (Light) (h) UBC (JPEG Compression).

https://doi.org/10.1371/journal.pone.0293111.g005

Experiment result based on Oxford dataset

The experiment was conducted on eight image categories from the dataset, including Bikes (blur), Trees (blur), Graf (viewpoint), Wall (viewpoint), Bark (zoom and rotation), Boat (zoom and rotation), Leuven (light), and Ubc (JPEG compression). For each image pair, 30 experiments were conducted using the feature extraction algorithms proposed in Table 1. To ensure generality, the average value of the 30 experiments was used as the experimental result.

Impact of feature points distribution on matching performance: A study on the Oxford dataset.

In this study, feature point detection and matching algorithms including SIFT, SURF, BRISK, FREAK, LATCH, LDB, ORB, Mur-ORB, LF-Net, D2-Net, R2D2 and ALGD-ORB were selected to conduct experiments on feature point extraction and image matching using images from the Oxford dataset. In addition to evaluating the performance differences of various feature point detection and matching algorithms on the Oxford dataset, we also analyzed the distribution of feature points using each algorithm to intuitively observe the performance differences in feature point uniformity distribution.

Fig 6 provides a visual representation of the experimental results obtained by testing nine different algorithms on the Leuven image set from the Oxford dataset. It offers a scientific and experimental perspective on the results. The experimental results indicate that the performance of LATCH, LDB, ORB LF-Net and D2-Net algorithms differ from other algorithms in terms of feature point extraction. They tend to extract image regions with more prominent edges, while not extracting many feature points in homogenous regions. Although SIFT, SURF, BRISK, FREAK and R2D2 algorithms have the ability to extract feature points from homogenous regions, the density of feature points in such areas tends to be lower. While most feature points still tend to be concentrated in edge areas, resulting in significant overlap, the distribution of feature points in the Mur-ORB, ALGD-ORB, and R2D2 algorithms appears to be more uniform compared to other tested algorithms. Visual observation alone does not provide a clear distinction regarding which algorithm exhibits the most uniform distribution. However, according to Table 2, the ALGD-ORB algorithm demonstrates a comparatively more uniform distribution of feature points when compared to the Mur-ORB and R2D2 algorithms. ALGD-ORB shows the ability to extract feature points from specific uniform regions where other algorithms may struggle.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Illustrates the distribution of feature points extracted by different algorithms.

(a) SIFT, (b) SURF, (c) BRISK, (d) FREAK, (e) LATCH, (f) LDB, (g) ORB, (h) Mur-ORB, (i) LF-Net, (j) D2-Net, (k) R2D2, and (l) ALGD-ORB, as demonstrated on the Leuven group image from the Oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Average feature point distribution uniformity across image groups in the Oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.t002

Fig 7 presents the image matching results obtained using the nine tested algorithms on the Leuven image set from the Oxford dataset. The matching pairs are depicted with red lines for incorrect matches and green lines for correct matches after applying RANSAC. Similar to the feature point distribution observed in Fig 6, the feature point matches obtained from SIFT, SURF, BRISK, FREAK, LATCH, LDB, and ORB algorithms in Fig 7 exhibit a noticeable concentration. Conversely, the Mur-ORB, ALGD-ORB, LF-Net, D2-Net, and R2D2 algorithms produce more uniformly distributed matching results, accurately capturing matches in both uniform and edge regions. Notably, the ALGD-ORB algorithm demonstrates a higher degree of uniformity in the matching results compared to the Mur-ORB and R2D2 algorithms. For a more accurate analysis of algorithm performance, statistical analysis is conducted on the feature point distribution uniformity across all images in the Oxford dataset, as presented in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Illustrates the distribution of feature points extracted by different algorithms.

(a) SIFT, (b) SURF, (c) BRISK, (d) FREAK, (e) LATCH, (f) LDB, (g) ORB, (h) Mur-ORB, (i) LF-Net, (j) D2-Net, (k) R2D2, and (l) ALGD-ORB, as demonstrated on the Leuven group image from the Oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.g007

Table 2 demonstrates the performance of different algorithms across the eight scene transformations available in the Oxford dataset. Both Mur-ORB and ALGD-ORB exhibit superior uniformity in feature point extraction compared to the other algorithms tested. While ALGD-ORB may show slightly lower uniformity than Mur-ORB in specific scenarios such as the Bark and Leuven group images, it consistently achieves more uniform feature point distributions across the remaining six image groups. Notably, ALGD-ORB demonstrates the ability to extract feature points in certain uniform regions where Mur-ORB falls short. Compared to Mur-ORB, ALGD-ORB demonstrates better adaptability under multiple scenes.

Matching precision analysis of the Oxford dataset.

This study conducted image matching tests on all images within the Oxford dataset. Each image group consisted of five pairs of images that required matching (ranging from 1-2 to 1-6, with increasing variations between the pairs). Fig 8 presents the matching accuracy for each image pair, while the symbol “Mean” represents the average accuracy of image matching across the eight image groups. Further details regarding the average accuracy can be found in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Matching precision results on the Oxford dataset.

(a) Bikes (blur). (b) Trees (blur). (c) Graf (viewpoint). (d) Wall (viewpoint). (e) Bark (zoom and rotation). (f) Boat (zoom and rotation). (g) Leuven (light). (h) Ubc (JPEG compression).

https://doi.org/10.1371/journal.pone.0293111.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Mean precision for image groups based on the Oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.t003

Based on Fig 8 and Table 3, we can see that for the Wall image set, SIFT achieves the highest precision. For the Bark image set, BRISK achieves the highest precision. For the Bikes, Graf, and Leuven image sets, the R2D2 algorithm has the highest precision. For the Boat image set, ORB achieves the highest precision. For the Trees and Ubc image sets, Mur-ORB has the highest precision. Mur-ORB also achieves the highest average precision.

The ALGD-ORB algorithm is an improved and optimized version based on the ORB algorithm. However, it does not demonstrate exceptional performance in certain image sets. The main reason for this is that, in order to improve computational efficiency, the feature detector focuses only on small image regions and specifically on low-level structures such as corner points. Subsequently, the descriptors capture higher-level information in the larger surrounding regions of the keypoints. In the presence of strong appearance variations, the detection of keypoints becomes unstable due to the susceptibility of the low-level information used by the detector to variations in low-level image statistics, such as pixel intensity. Nevertheless, despite the lack of significant advantages in certain image sets, the ALGD-ORB algorithm still exhibits acceptable performance in terms of accuracy.

RMSE analysis of image matching on the Oxford dataset.

According to Fig 9, the RMSE (Root Mean Square Error) metrics of SIFT, SURF, BRISK, FREAK, LATCH, LDB, ORB, Mur-ORB, LF-Net, D2-Net, R2D2, and ALGD-ORB algorithms on the Oxford dataset under different image variation scenarios are presented. In Table 4, LF-Net algorithm has the smallest average RMSE value in the Bikes image set. R2D2 algorithm achieves the smallest average RMSE value in the Boat image set. Similarly, in the Leuven and Ubc image sets, R2D2 algorithm exhibits the smallest average RMSE value. ALGD-ORB algorithm demonstrates the smallest average RMSE values in the Bark, Graf, Trees, and Wall image sets. However, the RMSE value for the Bark image set is missing for the R2D2 algorithm, indicating that it failed to match correct features in this particular image set, resulting in a considerably large RMSE.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Mean RMSE analysis on the Oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Feature points are extracted from images with various scenes and lighting conditions.

The mean RMSE of each group is shown in units of pixels.

https://doi.org/10.1371/journal.pone.0293111.t004

Overall, the ALGD-ORB algorithm exhibits the smallest average RMSE value of 1.0285 across all image sets. This suggests that the ALGD-ORB algorithm performs well in terms of feature point extraction in various image variation scenarios within the Oxford dataset.

Matching operation time analysis on the Oxford dataset.

To demonstrate the efficiency of the ALGD-ORB algorithm, we compared the average runtime of various algorithms, including SIFT, SURF, BRISK, FREAK, LATCH, LDB, ORB, Mur-ORB, LF-Net, D2-Net, R2D2, and ALGD-ORB, on the Oxford dataset. As shown in Fig 10 and Table 5, the ALGD-ORB algorithm has a remarkable runtime efficiency. during runtime. Among the tested algorithms, the R2D2 algorithm, based on deep learning, has the longest runtime, while the ORB algorithm has the shortest. Deep learning-based feature extraction algorithms generally require more time compared to traditional feature detection algorithms. Despite the improvements made to the FREAK, LATCH, LDB, and Mur-ORB algorithms derived from the ORB algorithm, they still consume more time. In comparison to the Mur-ORB algorithm, the ALGD-ORB algorithm experiences a slight increase in runtime due to the inclusion of additional processes, such as adaptive threshold selection and 64-bit descriptor generation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Statistical analysis of mean operation time on the oxford dataset.

https://doi.org/10.1371/journal.pone.0293111.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Shows the mean operation time of each algorithm on Oxford dataset in units of seconds.

https://doi.org/10.1371/journal.pone.0293111.t005

Overall, the ALGD-ORB algorithm demonstrates excellent efficiency on the Oxford dataset. Despite the slight increase in runtime compared to the Mur-ORB algorithm, the ALGD-ORB algorithm maintains competitive performance while incorporating additional processes for adaptive threshold selection and 64-bit descriptor generation. The advantages of the ALGD-ORB algorithm in terms of runtime efficiency are clearly depicted in Fig 10 and summarized in Table 5. In contrast to LF-Net, D2-Net, and R2D2, which are deep learning-based algorithms, the ALGD-ORB algorithm does not rely on GPU acceleration yet still provides efficient matching operations. The ALGD-ORB algorithm provides a balanced performance in terms of runtime and feature extraction capabilities, making it a suitable choice for a variety of applications, especially in re-source-constrained scenarios where GPU acceleration is not available.