Image Retrieval(IR) and feature extraction.

Recent deep-learning-based models have become state-of-the-art in connection with various tasks in computer vision and have great leverage in extracting significant visual representations from images in the image retrieval tasks. Specifically, convolutional neural network (CNN)-based methodologies are widely studied because traditional bag-of-word techniques have demonstrated improvements by utilizing feature vectors from the activation of well-known pre-trained CNNs as image descriptors.

R-MAC [17] proposed a method for extracting the representation of images by combining max-pooling at the back of the CNN architecture, which does not include a fully connected layer. This makes it possible to simultaneously obtain both global and local representations. However, it has the limitation of including information of unnecessary regions, such as the background, distant objects, and other trivial aspects when exploring local features within the image. To address this problem, Gordo et al. [18] presented a method for obtaining local features of meaningful object localization using region proposal networks (RPNs). Furthermore, they described a Siamese network that computes similarities between images while sharing the weights of convolutional layers by feeding the similar positive, non-similar negative, and query images, which was also constructed in the end-to-end deep learning architecture DIR [19]. Ahmad Faiyaz [20] introduced a deep image retrieval method that combines neural network interpolation and similarity-based indexing to enhance image retrieval. By integrating deep learning techniques, the proposed approach improves accuracy and effectiveness efficiently in retrieving images.

Approaches with transformer models combined with a retrieval module have embraced image retrieval in computation elapsed time, extracting localized (detail) information. Nath et al. [21] utilized a pre-trained big transfer (BiT) model with triplet loss. The BiT model was trained using a large, supervised dataset (tf_flower) to extract features and later fine-tuned for the target task. The extracted features were later used in searching for a similar target using a trained K-nearest neighbor (KNN) network for image retrieval. FIRe [22] proposed a local feature integration transformer (LIT), which has a transformer-like architecture that extracts super-features that are local features along with global features from image data. The ASMK model [7] was used to support the extraction in local-features, which is used from the beginning of the training phase instead of only at the retrieval phase. SSL-ViT-16 [23] showed that it is possible to extract meaning from unlabeled data by applying the feature embedding method with a self-supervised vision transformer in the zero-shot image retrieval task. Revaud et al. [24] proposed a new method to optimize the global mean average precision (mAP) for image retrieval by leveraging recent advances in listwise loss formulations. They proposed a new method and benchmark by simultaneously considering multiple images at each iteration and eliminating the need for ad-hoc tricks.

As the components that make up the query required for image retrieval are becoming more diverse, many recent works have studied how to combine multi-modal queries. Gordo et el. and PCME [11, 25] used combination of visual and textual embedding based on similarities. ARTEMIS [13] proposed to use two attention mechanisms, one executing a target image and reference image with text as input, and the other computing a score for how well the target image matches the text. CAISE [14] introduced a conversational agent that can not only edit the visual properties of images but also retrieve images from user requests. It can embed images and utterances via faster RCNN / positional encoding and the LSTM architecture and then generates an executable command for editing or retrieving images via attention mechanisms that can compute the similarity of each of the two embedding vectors. Considering the improvement by adopting additional data, multi-modal is surely a considerable domain.

Feature extraction, an important underlying technology in the field of image retrieval, is utilized in various fields such as object recognition, detection, and medical analysis. Damaneh et al. [26] presented a method for recognizing static hand gestures in sign language. It utilizes a CNN along with feature extraction techniques using the ORB descriptor and Gabor filter for accurate hand-gesture recognition. Yang et al. [27] examined integrating deep learning techniques into face perception technology to improve accuracy in diverse lighting conditions. By leveraging advanced algorithms, the proposed approach enhances face recognition in varying illumination intensities. Our strategy closely aligns with the approach of leveraging advanced deep learning algorithms. Qureshi et al. [28] focused on a method called radiogenomic classification, which uses a combination of different types of data (multi-omics fused feature space) to determine the MGMT promoter methylation status. Their goal was to develop a non-invasive diagnostic approach using mpMRI scans that requires minimal invasiveness.

Image Quality Assessment (IQA).

The quality evaluation of the image could be classified as; full-reference(FR) and no-reference based on whether the reference image(clean, pristine image) is required. PSNR and SSIM [29] approaches are two popular traditional full-reference image quality assessment(FR-IQA). PSNR is a pixel-based metric that measures the difference between two images in pixel-level values. SSIM measures the structural similarity considering the luminance, contrast, and structural information of the images. DeepSim, LPIPS, and IQT apply deep learning as an approach [30–32]. Each combined VGGNet, transformer, and attention to improve the accuracy.

No-reference image quality assessment(NF-IQA) methods evaluate the image without having access to a pristine image. Natural image quality evaluator (NIQE), BRISQUE [33, 34] are traditional no-reference approaches. NIQE is a statistical approach that captures and learns features of pristine images, such as texture and contrast, and predicts the image quality. BRISQUE uses a support vector regression (SVR) model to predict the image quality. Deep learning-based approaches have also been proposed for NF-IQA [35–42]. Multi-scale architecture [38, 39], transformer model [38, 40, 41], and contrastive learning [42, 43] have recently been introduced.

Adversarial attacks.

Studies have been conducted on the security of the ranking system. In particular, in the case of images, various definitions of distortion exist. The general approach for an adversarial attack is distortion by adding noise to the image in the query stage [44–47].

ZQBA [44] has beenintroduced a zero-query attack method for attacking content-based image retrieval(CBIR) systems in a black-box setting where no knowledge about the system is available. The method is based on an ensemble of models that perform optimization using some surrogate feature-extraction models and complement the optimized results. QAIR [45] performs a query-based attack against the image retrieval system under black-box settings. It has a relevance-based objective function for quantifying the attack effects and a recursive model stealing method to improve the query-attack efficiency. The system is capable of fooling commercial image retrieval systems such as Bing Visual Search with a few queries. NAG [46] is a method to achieve the most challenging black-box attacks in deep hash-based image retrieval. The relations between adversarial subspace and black-box transferability have been explored by using random noise as a substitute. The proposed model is an algorithm to estimate the adversarial region by introducing random noise, which is used to assess the capacity of different attacks. In PIRE [47], data perturbation is defined as a change that could not be easily caught by the human eyes but altered to disrupt the content-based retrieval system. It analyzes of adversarial queries in unsupervised methods, focusing on neural, local, and global features.

Additional approaches include attacking the ranking system to return the unexpected results as a prediction [48, 49]. UAP [48] includes a set of universal attack methods against image retrieval systems to cause the system to return unrelated images as top results in the ranking list. The authors generated universal adversarial perturbations (UAPs) that can be applied to all query images using gradient-based optimization algorithms. This attack can significantly reduce the accuracy of the system by disrupting the neighborhood relationships between features in the images used for retrieval. DAIR [49] is an efficient query-based attack method for image search systems, using projected natural evolution strategies (PNES) to generate adversarial perturbations that flip the top-K search results. PNES is an optimization algorithm that uses natural evolution strategies to search for the optimal perturbation vector, incorporating new projection operators, utility functions, and optimization techniques to ensure that the perturbed images remain visually similar to the original images.

Waterloo exploration database.

Waterloo Exploration Database contains 4744 original and 94880 distorted images. It is proposed to compensate for the limited content variations of previous benchmarks. The original image is distorted into four distortion types and five distortion levels.

PieAPP.

PieAPP benchmark is utilized for training image-error prediction algorithms by providing reference images and their distorted versions labeled with probability of preference. It uses a subset of 200 reference images from the Waterloo dataset, and 40 human subjects were queried to ensure reliable probability labels. Total 19,680 distorted images were generated, and we captured distortion in aspects of common image artifacts (e.g., additive Gaussian and speckle noise), HVS (e.g., non-eccentricity, contrast and sensitivity); and complex algorithms (e.g., deblurring, denoising, super-resolution, and compression).

Feature extraction.

In this stage, two features should be collected by the module: representational and quality. in the aspect of the representational feature, it contains information such as shape and color. this information has already been utilized in pre-existing image retrieval systems; therefore, it will also be utilized in this framework. The quality information is obtained by the quality assessment models. To ensure that the original image is ranked at the top and the distorted image is ranked lower than the original, two features are normalized and balanced.

Representation feature.

Feature extraction methods vary depending on the purpose of the task, such as classification or detection. For simple CBIR system, we chose the self-supervised learning models for the feature extraction. However, the extraction module of GuaRD is highly compatible with other methods and could be replaced by other models as required.

The BYOL model is learned by leveraging the idea of “positive pairs” consisting of an original image and its augmented version. We expected a model that could distinguish it from other data while focusing on the features of each data itself, even if similar data existed. Utilizing self-supervised methodologies, several models were evaluated on a library framework [52]. Table 1 presents the prediction accuracy of each model measured by mean precision, as indicated in the form of ‘mp@k’. The values were edited to compare performance against BYOL, set to 1, and correspondingly compared with the others. It is a preliminary predictor that accurately shows results. Regarding the top-1 prediction, the SupCon model was twice as accurate as the other models. In the top-3 and top-5 predictions, when compared with other models, the accuracy level was inferior. BYOL model, by contrast, showed stable predictive performance. Hence, the final feature extraction model was designed based on the BYOL model.

Because the original purpose of the system was to focus on simple similarity comparison rather than classification, the used data also do not have separate label values. As an adequate methodology for the feature extraction module, BYOL, one of the self-supervised learning models, was utilized.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Mean precision at rank K (mP@K) with pre-trained models.

https://doi.org/10.1371/journal.pone.0288432.t001

Fig 4 and the following equation are based on the schematics of the existing study on BYOL [53]. It consists of two networks (online and target) and passes the same image through different data augmentation steps. The network learns the essence feature by predicting the projection result of the target network with q_θ and (z_θ). Each (θ, γ) describes trained weights and the exponential moving average. To prevent model collapse, the target network updates the values of the online network using the exponential moving average method [53]. Eq 1 obtains the loss value and normalizes the batch. L_θ,γ is a loss function optimized by minimizing the difference between the projection of γ and prediction result of q_θ(z_θ). Because the network is symmetrical, the loss function of the other half network can be expressed as .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Proposed feature extraction network based on the BYOL structure.

The backbone of the feature extraction network is based on the BYOL network. Through the network, a given image is converted into an embedded vector, which is a vector learned by the projector of the model.

https://doi.org/10.1371/journal.pone.0288432.g004

The loss function of BYOL presented in Eq 2 is optimized for the online network. η denotes the learning rate. In training, the model is intended to learn through the stochastic optimization step with the total set of loss functions. The parameter values of the target network are updated in Eq 3. The weighted target value is aggregated with the updated value θ from the online network. (1) (2) (3)

Quality feature and integration.

To obtain the quality information of the image, CONTRIQUE, which is no-reference image quality assessment model is adapted. The feature extracted from CONTRIQUE contains information on the scale and types of distortion which are applied in the model. The distances calculated from two aspects (representation and quality) are balanced and merged to execute the similarity of the images. In other words, the quality of the image is used as a regularization term. Eq 4 is the equation that calculates the integrated distance. Determining the appropriate α value depends on the specific characteristics and configuration of a given dataset. For this experiment, we used an the value of 0.9. (4)