With great potential for assisting radiological image interpretation and decision making, content-based image retrieval in the medical domain has become a hot topic in recent years. Many methods to enhance the performance of content-based medical image retrieval have been proposed, among which the relevance feedback (RF) scheme is one of the most promising. Given user feedback information, RF algorithms interactively learn a user’s preferences to bridge the “semantic gap” between low-level computerized visual features and high-level human semantic perception and thus improve retrieval performance. However, most existing RF algorithms perform in the original high-dimensional feature space and ignore the manifold structure of the low-level visual features of images. In this paper, we propose a new method, termed dual-force ISOMAP (DFISOMAP), for content-based medical image retrieval. Under the assumption that medical images lie on a low-dimensional manifold embedded in a high-dimensional ambient space, DFISOMAP operates in the following three stages. First, the geometric structure of positive examples in the learned low-dimensional embedding is preserved according to the isometric feature mapping (ISOMAP) criterion. To precisely model the geometric structure, a reconstruction error constraint is also added. Second, the average distance between positive and negative examples is maximized to separate them; this margin maximization acts as a force that pushes negative examples far away from positive examples. Finally, the similarity propagation technique is utilized to provide negative examples with another force that will pull them back into the negative sample set. We evaluate the proposed method on a subset of the IRMA medical image dataset with a RF-based medical image retrieval framework. Experimental results show that DFISOMAP outperforms popular approaches for content-based medical image retrieval in terms of accuracy and stability.
Citation: Shen H, Tao D, Ma D (2013) Dual-Force ISOMAP: A New Relevance Feedback Method for Medical Image Retrieval. PLoS ONE 8(12): e84096. https://doi.org/10.1371/journal.pone.0084096
Editor: Xi-Nian Zuo, Institute of Psychology, Chinese Academy of Sciences, China
Received: July 19, 2013; Accepted: November 12, 2013; Published: December 31, 2013
Copyright: © 2013 Shen et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported in part by the following projects: 1. National Natural Science Foundation of China under grant number 61003017, 2. Project of State Key Laboratory of Software Development Environment under grant number SKLSDE-2013ZX-30, 3. ARC Discovery Project under grant number DP140102164. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Medical image interpretation is a process which incorporates subjective perception and objective reasoning. Typically, radiologists obtain superficial visual features from medical images and render diagnostic conclusions based on personal knowledge and experience. Due to differences of perception, training and fatigue, different conclusions about the same medical image will be drawn by different professionals or by the same professional under different circumstances , . The goal of content-based medical image retrieval (CBMIR) is to enable radiologists to make better diagnosis about a given case by retrieving similar cases from a variety of semantically annotated medical image archives.
It is well-known that “semantic gap” is one of the issues faced by content-based image retrieval (CBIR). The fact that medical images contain varied, rich and subtle visual features  is an additional challenge to the use of CBIR in radiology. Unlike from regular image understanding, medical image diagnosis is dependent on case-specific interpretation. It is common for visually similar medical images to convey different semantic meanings, while semantically-alike images have different visual features. Let us take medical images obtained from IRMA medical image dataset  as an example. The IRMA medical image dataset is a widely used test bed for performance evaluation of CBMIR –. The new version of IRMA dataset  contains 12,677 fully annotated gray value radiographs in a training set. These images are categorized into 193 classes according to a mono-hierarchical multi-axial classification standard called the IRMA coding system . The system classifies a medical image from four orthogonal axes: imaging modality, body orientation, body region examined and biological system examined. Figure 1 and Figure 2 illustrate the scenario of semantic gap. As shown in Figure 1, two chest radiographs have a similar visual appearance, but their semantic meanings are different. The IRMA code  of the left image is “1123-127-500-000”, while the IRMA code of the right image is “1123-110-500-003”. By contrast, though their visual appearance is different, the two mammograms shown in Figure 2 have the same IRMA code “1124-310-610-625”.
The chest radiographs shown in this figure have a similar visual appearance but belong to different semantic categories. Images are taken from IRMA medical image dataset.
The mammograms shown in this figure belong to the same semantic category, though they have a different visual appearance. Images are taken from IRMA medical image dataset.
Relevance feedback (RF) is a promising solution to fill the semantic gap in CBIR . Under the assumption that every user’s need is different and time varying , , RF provides a user-in-the-loop mechanism to allow a user to interact with the retrieval system to refine the retrieval results. The basic process of RF in CBIR is as follows: 1) the retrieval system returns the initial retrieval results to the user; 2) the user labels query-relevant images and query-irrelevant images as positive feedback and negative feedback, respectively; 3) based on the labeled feedback, the retrieval system learns to improve the retrieval performance and returns new results; 4) if the user is satisfied with the new results, the RF process ends; otherwise, go to 2).
Over the past decades, many representative RF approaches have been proposed in the context of CBIR –. A comprehensive survey of these methods can be found in , . In , RF methods are categorized into four groups: subspace selection-based schemes, support vector machine (SVM)-based schemes, random sampling-based schemes and feature reweighting-based schemes. Performance evaluations of several RF approaches are reported in , .
Many RF methods have also appeared in CBMIR in recent years. Rahman et al.  utilized positive feedback to update the optimal query point for medical image retrieval. They proposed a RF-based dynamic similarity fusion approach for biomedical image retrieval  in which RF information is utilized to reweight features at each iteration. Xu et al. ,  utilized RF to update feature weights for X-ray image retrieval. To solve the small sample size problem, Hoi et al.  proposed a method called semi-supervised SVM batch mode active learning for both medical and regular image retrieval. In addition, Ko et al.  integrated the RF scheme into CBMIR to boost retrieval performance. Though the approaches mentioned above achieve promising results, there is room for performance enhancement because most of these methods do not consider the manifold structure of low-level image features.
In this paper, we formulate a new RF method termed dual-force ISOMAP (DFISOMAP) for CBMIR. DFISOMAP is proposed in the context of precisely exploring the manifold structure of low-level image visual features –. DFISOMAP operates in the following three stages: 1) the local geometry preservation stage, 2) the margin maximization stage, and 3) the similarity propagation stage. First, the local geometry of the positive examples in the high-dimensional feature space is preserved according to the isometric feature mapping (ISOMAP) criterion . To precisely model the geometric structure of positive examples in the low-dimensional embedding, a reconstruction error constraint according to locally linear embedding (LLE)  is also added. Second, negative examples are pushed away from positive examples by a force driven by the maximization of average pairwise distances between the positive and negative examples. Finally, negative examples are pulled into the negative sample set by another force generated by similarity propagation. We conduct experiments to demonstrate the effectiveness of DFISOMAP. Compared to conventional RF methods, e.g., linear discriminant analysis (LDA) , locality preserving projections (LPP) , biased discriminant analysis (BDA) , constrained similarity measure using support vector machine (CSVM) , ISOMAP and exponential locality preserving projections (ELPP) , DFISOMAP differ in the following ways: 1) DFISOMAP precisely preserves the geometric structure of positive feedback examples, and 2) DFISOMAP does not suffer from the undersampling problem.
In this section, we detail the proposed DFISOMAP. To better present the method, Table 1 lists important notations used in this paper.
Consider a set of medical images in low-level feature space, and a query image Following the query-by-example paradigm of the CBIR system, there are top returned images for each query, from which we obtain images which are from the same semantic class as We term them positive examples: Putting these examples together, we get a positive feedback set Meanwhile, we obtain images, which are from different semantic classes with respect to We term them negative examples: Putting these examples together, we get a negative feedback set The relevance feedback set is constructed by putting and together as where the first are positive examples and the remaining are negative examples, For convenience, we use to represent all examples, and denote and
DFISOMAP assumes that medical images lie on a low-dimensional manifold and are artificially embedded in a high-dimensional ambient space, i.e., the low-level feature space The objective of DFISOMAP is to learn a mapping function from to based on the relevance feedback set The learned mapping should effectively separate positive examples from negative examples. For simplicity, we assume that is linear. The problem of DFISOMAP is then converted to find a projection matrix that maps to i.e., where Here, each column of
DFISOMAP operates in three stages which containing two forces to separate negative examples from positive examples. In the first stage, the local geometric structure of the positive examples is preserved according to the ISOMAP criterion . To make the local geometry preservation more precise, an error reconstruction constraint is added. This stage is termed “local geometry preservation”. In the second stage, a margin maximization function is defined to maximize the gap between positive examples and negative examples. The margin maximization function acts as a force to push negative examples away from positive examples, and this stage is termed “margin maximization”. In the final stage, termed “similarity propagation”, the similarity propagation technique  is employed to build a similarity matrix which quantifies similarities between the intraclass examples contained in the relevance feedback set. Based on the similarity matrix, the distance between the intraclass examples is minimized to shrink the distance between image pairs from the same semantic class. The procedure acts as another force to pull negative examples away from positive examples.
2.1. Local Geometry Preservation
ISOMAP preserves the local geometry of positive examples by the following objective function 
where is the geodesic distance between image in high-dimensional space And is the corresponding Euclidean distance between image and in low-dimensional embedding
Let us denote Where and are matrices. According to , and can be converted to inner product matrix and respectively. Operator is defined as
where is an identity matrix, Thus, equation (1) can be transformed to
where stands for the trace operator, Assuming that is a constant matrix, equation (5) can be converted to
To minimize reconstruction error of the local geometry preservation presented above, we further assume each can be reconstructed by its neighbors. Thus, we have
where is an identity matrix. is obtained via locally linear embedding (LLE) :
Putting equation (6) and (7) together, we obtain the objective function for local geometry preservation
where is the trade-off parameter.
2.2. Margin Maximization
In the low-dimensional embedding, we expect that the average pairwise distances between negative and positive feedback examples will be as large as possible, and the average pairwise distances among positive feedback examples will be as small as possible, i.e.,
Where is the gap factor. Considering equation (10) reduces to:
2.3. Similarity Propagation
Equation (11) only takes into account the distances between the positive examples and coarsely treats negative examples. To remedy this, we need the average pairwise distance among the intraclass examples to be rendered as small as possible.
The straightforward way to shrink the pairwise distance between interclass examples is to minimize the average weighted square distance between all sample pairs in the low-dimensional embedding:
where is termed similarity matrix.
In this paper, we defineas
quantifies the similarity relationship among positive and negative examples, respectively. In our implementation, we setas 0.5.
Putting equation (11) and (12) together, we have
where is the trade-off parameter. Let us denote
Then equation (14) can be rewritten as
where is a diagonal matrix,
2.4. Objective Function
Combining equation (9) with (16), we obtain the objective function of DFISOMAP
where is the margin factor,
Because the real matrix is symmetric (the proof is given in Appendix S1), can be solved by standard eigenvalue decomposition on By imposing on (17), is formed by the eigenvectors associated with the first largest eigenvalues.
We utilize the framework depicted in Figure 3 for CBMIR. Any RF feedback algorithm can be integrated into this framework. As shown in this figure, when a query image is provided, its low-level visual features are extracted. All images contained in the medical image database are then sorted in ascending order according to their distance from the query image measured by Euclidean metric. If the user is not satisfied with the result, s/he labels some semantically relevant images as positive feedback examples and some semantically irrelevant images as negative feedback examples. Based on these feedback examples, a RF model can be trained. All images, including the positive feedback, the negative feedback and the remaining images contained in the medical image database, are re-sorted based on the updated similarity metric and the top-ranked images are returned. If the user is not satisfied with the result, the RF process is repeated.
For DFISOMAP, we learn a projection matrix according to equation(17). Then we use to project all the images to the low-dimensional embedding. In the projected embedding, each image is re-sorted in ascending order with respect to its Euclidean distance from the query image and the top-ranked images are returned to the user. The RF procedure stops when the user is satisfied with the results.
We use LBP , SIFT  and pixel intensity descriptors respectively to extract features from the medical image. For the LBP descriptor, we divide each medical image into 3×3 equal regions. On each region, a 59-bin LBP histogram is built. Then we concatenate these 59-bin LBP histograms into a 531-D vector. For the SIFT and intensity descriptors, we follow bag of features  scheme to represent the image. In detail, we first densely sample each image with SIFT and the intensity descriptor, respectively. We set the sampling space as 8, and the patch size as 16×16. Then we use K-means clustering to learn two 500-word dictionaries, i.e., SIFT and intensity visual word dictionary. Finally, for each image, we obtain a 500-bin SIFT and intensity histogram, respectively.
We represent each image by concatenating the 531-bin LBP histogram, 500-bin SIFT histogram and 500-bin pixel intensity histogram into a 1531-D long vector. To get rid of redundant information contained in the concatenated vector and reduce the computational complexity in the next section, we normalize the concatenated 1531-D vector into a normal distribution with zero mean and one standard deviation. Then we use principal component analysis (PCA) to reduce the normalized vector to a 500-D feature vector.
In this section, we report performance of the proposed DFISOMAP for CBMIR comparing with that of other methods, i.e., LDA, LPP, BDA, CSVM, ISOMAP, LLE and ELPP.
This section is organized as follows. In section 4.1, we introduce the dataset used for evaluation. Section 4.2 presents experimental setup. In section 4.3, we compare DFISOMAP with other RF approaches using mean average precision (MAP) and standard deviation (SD). Section 4.4 reports performance evaluation results of RF methods in terms of precision and recall. Finally, we explore effects of parameters on the performance of DFISOMAP in section 4.5.
4.1. IRMA Medical Image Dataset
The IRMA medical image dataset is widely used for CBMIR evaluation. In our experiment, we select the first 57 categories from the new version of IRMA dataset as test bed. The selected images contain a total of 10,902 images. Figure 4 shows example images from the dataset. Figure 5 illustrates three query images.
4.2. Experimental Setting
We conduct 338 independent experiments to evaluate performance of DFISOMAP and other RF methods. In detail, we randomly select 338 images from the IRMA data set as query examples. These images belong to different IRMA categories. In general, five or six images are selected from each IRMA category. In initial retrieval, for each query sample, there are five to eight relevant images in top30 ranked results. For each selected image, a “leave one out” query is conducted: Rest images contained in the data set are ranked according to their Euclidean distance to the query sample.
Different RF algorithms are embedded into the framework depicted in Figure 3. The RF process is automatically performed by the computer. A computer-simulated query for each query image is performed on all the other 10,901 images contained in the dataset. The computer marks all query relevant images as positive feedback in the top 30 images and the rest as negative feedback. In general, we have between two and eight images as positive feedback. The procedure is close to a real-world application scenario, because typically the user does not want to label many feedback examples in the iteration process. We set the number of RF iterations as 10. For the first iteration, the returned images are ranked according to their Euclidean distance from the query image. Starting from the second iteration, different RF algorithms learn different projection matrices based on positive and negative feedback, respectively. In the projected low-dimensional embedding, other images in the dataset are re-ranked according to their Euclidean distance from the query image.
We parameterize the settings of all baseline methods according to the descriptions in corresponding papers. In the experiments, the parameters of different methods are tuned to obtain the best results. For CSVM, we choose the Gaussian kernel with LibSVM  is utilized to achieve an optimal hyperplane to separate negative and positive examples. For ELPP, we set parameters as what is described in .
4.3. Performance Evaluation Using MAP and SD
In this section, we use MAP and SD to measure the performance of DFISOMAP and other RF algorithms. MAP is the mean of average precision values of the 338 independent queries. MAP value measures the retrieval precision of RF algorithms. SD value is computed from AP values of the 338 independent queries. SD value assesses the stability of RF algorithms.
Figure 6 and Figure 7 illustrate performance of the proposed DFISOMAP compared to LDA, LPP, BDA, CSVM, ISOMAP, LLE and ELPP-based RF algorithms. In Figure 6, subfigures (A), (B), (C), (D) and (E) show MAP values for the top 10, 20, 30, 40 and 50 results, respectively. The eight curves in each of these subfigures illustrate performance of the RF algorithms. The x-coordinate represents number of iterations, which varies from 0 to 9. Iteration 0 represents the initial retrieval measured by Euclidean distance in the high-dimensional feature space without RF, while iteration 1 refers to the first round RF based on feedback examples labeled in the 0th iteration, and similarly other iterations (from iteration 2 to 9). The y-coordinate indicates MAP values of different RF algorithms after each iteration. In Figure 7, subfigures (A), (B), (C), (D) and (E) detail SD values in the top 10, 20, 30, 40 and 50 results, respectively. SD indicates stability of the RF algorithm: the smaller the SD value, the more stable the algorithm.
Subfigures (A), (B), (C), (D) and (E) detail MAP values in the top 10, top 20, top 30, top 40 and top 50 results, respectively.
Subfigures (A), (B), (C), (D) and (E) detail SD values in the top 10, top 20, top 30, top 40 and top 50 results, respectively.
From the figure we can see that, in all experiments, and after any number of iterations, the proposed DFISOMAP consistently outperforms other conventional RF algorithms in terms of MAP. The DFISOMAP also shows good stability, as demonstrated by the SD value and tendency of the SD curve. At each level (top 10 to 50), it can be seen that SD values of DFISOMAP for further iterations decrease after one iteration, and are much lower than those of other RF algorithms.
4.4. Performance Evaluation Using Precision and Recall
In this section, we utilize average precision (AP) and average recall (AR) to evaluate performance of DFISOMAP and other methods. In the context of CBMIR, precision refers to percentage of relevant medical images in top retrieved results. AP is calculated as the averaged precision values obtained via all queries. And recall refers to percentage of relevant medical images in all relevant examples contained in the test bed. AR is averaged recall values of all queries.
Figure 8, Table 2 and Table 3 show AP of different methods. In detail, Figure 8 (A), (B), (C), (D) and (E) present AP of different methods in the top 10, 20, 30, 40, and 50 results, respectively. As we can see from the figure, it is evident that DFISOMAP subsequently outperforms other algorithms. Details of the AP values of top ranked results for different approaches after the fifth and ninth feedback are presented in Table 2 and Table 3, respectively. From these two tables, we can draw the conclusion that DFISOMAP achieves more promising results compared with other methods.
Subfigures (A), (B), (C), (D) and (E) detail AP in the top 10, 20, 30, 40, and 50 results, respectively.
Figure 9, Table 4 and Table 5 present AR of different algorithms. Specifically, Figure 9 (A), (B), (C), (D) and (E) demonstrate AR of different approaches obtained in the top 10, 20, 30, 40, and 50 results, respectively. We can conclude from the figure that DFISOMAP is more effective than the other compared methods. Moreover, AR values of top ranked results for different methods after the fifth and ninth feedback are given in Table 4 and Table 5, respectively. According to these two tables, we can see that DFISOMAP is more effective than other approaches.
Subfigures (A), (B), (C), (D) and (E) detail AR in the top 10, 20, 30, 40, and 50 results, respectively.
4.5. Effects of Parameters
(1) Effects of α. As shown in equation (17), parametercontrols the contribution of to Wherestands for utilizing LLE to preserve local geometry of positive feedback examples.
With the same experimental setup detailed above, we conduct experiments to evaluate effects of In our experiments, we increasefrom 0 to 100 with step 10, and setas 1400. Table 6 and Table 7 show AP and AR of DFISOMAP in top50 results, respectively. From which we can draw the following conclusions. 1) DFISOMAP achieves best performance whenis set as 10. 2) With the increasing of performance of DFISOMAP degrades. 3) Whenis set as 0, i.e.,has no contribution to performance of DFISOMAP is worst. The conclusion verifies the effectiveness of applying LLE to minimize reconstruction error within positive feedback examples.
(2) Effects of γ. Equation (17) demonstrates thatcontrols the contribution of toWherestands for similarity propagation in positive and negative examples.
With the same experimental setup mentioned above, we conduct experiments to explore effects of In our experiments, we increasefrom 0 to 2000 with step 200, and set as 10. Table 8 and Table 9 detail AP and AR of DFISOMAP in top50 results, respectively. From the table we can draw the following conclusions. 1) DFISOMAP achieves best performance whenis set as 1400. 2) Whenis set as 0, i.e., there is no similarity propagation, performance of DFISOMAP is worst. The conclusion confirms effectiveness of similarity propagation.
Starting from the assumption that medical images are artificially embedded in a high-dimensional visual feature space, we propose the dual-force ISOMAP (DFISOMAP) to map medical images from high-dimensional feature space to low-dimensional embedding. In the framework of CBMIR, DFISOMAP precisely preserves the geometric structure of positive feedback examples according to the ISOMAP criterion, and effectively separates negative examples from positive examples by utilizing two forces. The evaluation results on a subset of the IRMA medical image dataset show that DFISOMAP outperforms popular dimensionality reduction-based RF algorithms, e.g., LDA, BDA, LPP, ISOMAP, LLE, ELPP and support vector machine-based RF algorithms, e.g., CSVM.
Proof of is symmetric.
We thank Prof. Dr. T. M. Deserno of the Dept. of Medical Informatics, RWTH Aachen, Germany, for providing us with the IRMA medical image dataset. We also thank Mr. Adams Wei Yu at Carnegie Mellon University for his helpful discussions.
Conceived and designed the experiments: HS DT DM. Performed the experiments: HS. Analyzed the data: HS DT DM. Contributed reagents/materials/analysis tools: HS DT DM. Wrote the paper: HS DT DM.
- 1. Siegle RL, Baram EM, Reuter SR, Clarke EA, Lancaster JL, et al. (1998) Rates of disagreement in imaging interpretation in a group of community hospitals. Academic Radiology 5: 148–154.
- 2. Barlow WE, Chi C, Carney PA, Taplin SH, D'Orsi C, et al. (2004) Accuracy of screening mammography interpretation by characteristics of radiologists. Journal of the National Cancer Institute 96: 1840–1850.
- 3. Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, et al. (2011) Content-based image retrieval in radiology: current status and future directions. Journal of Digital Imaging 24: 208–222.
- 4. Deserno TM, Ott B (2009) 15,363 IRMA images of 193 categories for ImageCLEFmed 2009. V1.0 ed. http://www.irma-project.org/datasets_en.php?SELECTED=00009#00009.dataset.
- 5. Wang JY, Li YP, Zhang Y, Wang C, Xie HL, et al. (2011) Bag-of-features based medical image retrieval via multiple assignment and visual words weighting. IEEE Transactions on Medical Imaging 30: 1996–2011.
- 6. Dimitrovski I, Kocev D, Loskovska S, Dzeroski S (2011) Hierarchical annotation of medical images. Pattern Recognition 44: 2436–2449.
- 7. Yang L, Jin R, Mummert L, Sukthankar R, Goode A, et al. (2010) A boosting framework for visuality-preserving distance metric learning and its application to medical image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 32: 30–44.
- 8. Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Information Retrieval 11: 77–107.
- 9. Lehmann TM, Schubert H, Keysers D, Kohnen M, Wein BB (2003) The IRMA code for unique classification of medical images. Medical Imaging 2003: PACS and Integrated Medical Information Systems: Design and Evaluation 5033: 440–451.
- 10. Tao DC, Tang X, Li XL, Wu XD (2006) Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 28: 1088–1099.
- 11. Zhou XS, Huang TS (2003) Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems 8: 536–544.
- 12. Kurita T, Kato T (1993) Learning of personal visual impression for image database systems. Second International Conference on Document Analysis and Recognition: IEEE. pp. 547–552.
- 13. Fu Y, Huang TS (2008) Image classification using correlation tensor analysis. IEEE Transactions on Image Processing 17: 226–234.
- 14. Tao DC, Li XL, Wu XD, Maybank SJ (2009) Geometric mean for subspace selection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31: 260–274.
- 15. Sugiyama M (2007) Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. Journal of Machine Learning Research 8: 1027–1061.
- 16. Xu D, Yan SC, Tao DC, Lin S, Zhang HJ (2007) Marginal Fisher analysis and its variants for human gait recognition and content-based image retrieval. IEEE Transactions on Image Processing 16: 2811–2821.
- 17. Tao DC, Tang XO, Li XL, Rui Y (2006) Direct kernel biased discriminant analysis: A new content-based image retrieval relevance feedback algorithm. IEEE Transactions on Multimedia 8: 716–727.
- 18. Guo GD, Jain AK, Ma WY, Zhang HJ (2002) Learning similarity measure for natural image retrieval with relevance feedback. IEEE Transactions on Neural Networks 13: 811–820.
- 19. Yong R, Huang T (2000) Optimizing learning in image retrieval. IEEE Conference on Computer Vision and Pattern Recognition: IEEE. pp. 236–243.
- 20. Kherfi ML, Ziou D (2006) Relevance feedback for CBIR: A new approach based on probabilistic feature weighting with positive and negative examples. IEEE Transactions on Image Processing 15: 1017–1030.
- 21. Zhou XS, Huang TS (2001) Small sample learning during multimedia retrieval using BiasMap. IEEE Conference on Computer Vision and Pattern Recognition: IEEE. pp. 11–17.
- 22. Tong S, Chang E (2001) Support vector machine active learning for image retrieval. Ninth ACM International Conference on Multimedia. Ottawa, Canada: ACM. pp. 107–118.
- 23. Chu-Hong H, Chi-Hang C, Kaizhu H, Lyu MR, King I (2004) Biased support vector machine for relevance feedback in image retrieval. IEEE International Joint Conference on Neural Networks: IEEE. pp. 3189–3194.
- 24. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys 40: 5–60.
- 25. Bian W, Tao DC (2010) Biased discriminant Euclidean embedding for content-based image retrieval. IEEE Transactions on Image Processing 19: 545–554.
- 26. Huiskes MJ, Lew MS (2008) Performance evaluation of relevance feedback methods. International Conference on Content-based Image and Video Retrieval: ACM. pp. 239–248.
- 27. Doulamis N, Doulamis A (2006) Evaluation of relevance feedback schemes in content-based in retrieval systems. Signal Processing: Image Communication 21: 334–357.
- 28. Rahman MM, Bhattacharya P, Desai BC (2007) A framework for medical image retrieval using machine learning and statistical similarity matching techniques with relevance feedback. IEEE Transactions on Information Technology in Biomedicine 11: 58–69.
- 29. Rahman MM, Antani SK, Thoma GR (2011) A learning-based similarity fusion and filtering approach for biomedical image retrieval using SVM classification and relevance feedback. IEEE Transactions on Information Technology in Biomedicine 15: 640–646.
- 30. Xu X, Lee D-J, Antani SK, Long LR, Archibald JK (2009) Using relevance feedback with short-term memory for content-based spine X-ray image retrieval. Neurocomputing 72: 2259–2269.
- 31. Xu X, Antani S, Lee DJ, Long LR, Thoma GR (2006) Relevance feedback for shape-based pathology in spine X-ray image retrieval. Medical Imaging 2006: PACS and Imaging Informatics: SPIE. pp. 61450K–61450K.
- 32. Hoi SCH, Jin R, Zhu J, Lyu MR (2009) Semisupervised SVM batch mode active learning with applications to image retrieval. ACM Transactions on Information Systems 27: 16 11–16: 29.
- 33. Ko BC, Lee J, Nam J-Y (2012) Automatic medical image annotation and keyword-based image retrieval using relevance feedback. Journal of Digital Imaging 25: 454–465.
- 34. Zhou TY, Tao DC (2013) Double shrinking sparse dimension reduction. IEEE Transactions on Image Processing 22: 244–257.
- 35. Liu WF, Tao DC (2013) Multiview hessian regularization for image annotation. IEEE Transactions on Image Processing 22: 2676–2687.
- 36. Hong ZB, Mei X, Tao DC (2012) Dual-force metric learning for robust distracter-resistant tracker. ECCV 2012: Springer Berlin Heidelberg. pp. 513–527.
- 37. Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290: 2319–2323.
- 38. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290: 2323–2326.
- 39. Duda RO, Hart PE, Stork DG (2001) Pattern classification: Wiley-Interscience.
- 40. He XF, Niyogi P (2004) Locality preserving projections. Advances in Neural Information Processing Systems: MIT Press. pp. 153–160.
- 41. Wang SJ, Chen HL, Peng XJ, Zhou CG (2011) Exponential locality preserving projections for small sample size problem. Neurocomputing 74: 3654–3662.
- 42. Liu W, Tian XM, Tao DC, Liu JZ (2010) Constrained metric learning via distance gap maximization. Twenty-Fourth AAAI Conference on Artificial Intelligence: Association for the Advancement of Artificial Intelligence. pp. 518–524.
- 43. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24: 971–987.
- 44. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60: 91–110.
- 45. Li FF, Perona P (2005) A Bayesian hierarchical model for learning natural scene categories. IEEE Conference on Computer Vision and Pattern Recognition: IEEE. pp. 524–531.