https://orcid.org/0000-0002-2351-2557
Figures
Abstract
Graph-based Laplacian regularization techniques have been extensively applied in unsupervised feature selection due to their capability in capturing the inherent structure of data. However, Laplacian regularization frequently results in a solution that is biased towards a constant geodesic function, which leads to inadequate extrapolation capabilities and an inability to robustly maintain the data’s topological structure. Aiming to tackle the drawback, we propose a new framework named Hessian-based Unsupervised Feature Selection using Low-Rank Approximation (HLR-UFS). First, our method introduces Hessian regularization to address the inability of graph-based Laplacian regularization to effectively preserve complex topological structures and nonlinear geometric information due to null space constraints. Second, to explicitly eliminate feature redundancy, we use low-rank approximation techniques to identify latent correlations among features. Furthermore, we employ -norm regularization to suppress noise and outliers. Finally, we design an efficient algorithm and rigorously validate its convergence through theoretical analysis. Comprehensive assessments on nine standard datasets show the validity and superior performance of HLR-UFS.
Citation: Zhang J, Lin W, Liu Y, Tu S, Peng M (2026) HLR-UFS: Hessian-based Unsupervised Feature Selection using Low-Rank Approximation. PLoS One 21(9): e0356837. https://doi.org/10.1371/journal.pone.0356837
Editor: Khan Bahadar Khan, Islamia University of Bahawalpur: The Islamia University of Bahawalpur Pakistan, PAKISTAN
Received: July 24, 2025; Accepted: August 9, 2026; Published: September 3, 2026
Copyright: © 2026 Zhang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The source code and benchmark datasets used in this paper are publicly available at Zenodo: https://doi.org/10.5281/zenodo.19722505.
Funding: This work was supported in part by the Fujian Natural Science Foundation under Grants 2023J01979 and 2024J01855, in part by the Open Fund Project of Fujian Provincial University Key Laboratory of Big Data Mining and Applications (Longyan University) under Grant DSJZDSYS202605, and in part by the Doctoral Research Project of Longyan University under Grant LB2023008. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Rapid development in the field of computer technology has led to the emergence of high-dimensional data in many practical application areas, including image processing, video surveillance systems, speech data, and biological information processing [1–3]. This has made dimensionality reduction an increasingly important yet challenging problem in machine learning, computer vision, and pattern recognition applications [4,5]. In practice, dimensionality reduction techniques can be broadly categorized into two main paradigms: feature selection, which identifies optimal feature subsets, and feature extraction, which constructs new feature representations. Feature extraction techniques, including manifold learning [6] and principal component analysis (PCA) [7], transform data from a high-dimensional to a low-dimensional space, which may reduce the interpretability of the original features. In contrast, feature selection specifically identifies and retains key subsets from the original feature set, safeguarding the vital information inherent in the full-dimensional space [8–10]. Consequently, feature selection has become a crucial technique in high-dimensional data analysis, offering a balance between dimensionality reduction and the preservation of semantic meaning.
Feature selection techniques are typically categorized into two types, contingent upon the availability of class labels: supervised and unsupervised. The supervised approaches leverage explicit label annotations to identify maximally discriminative feature subsets, benefiting from well-defined guidance that enables efficient optimization for specific learning objectives [11–13]. Nevertheless, supervised methods face fundamental implementation challenges, as large-scale data annotation typically incurs substantial time and resource costs, especially in domains requiring specialized expertise. The development of methods for unsupervised feature selection is propelled by this inherent constraint. Unsupervised feature selection techniques eliminate reliance on labeled data by leveraging intrinsic data properties, such as manifold structures or cluster relationships, to construct pseudo-supervisory signals for feature selection. Although unsupervised feature selection holds significant theoretical promise, its practical effectiveness critically depends on the quality of pseudo-supervision derived from learned structural patterns. Therefore, accurately capturing data structures remains an important and ongoing research frontier in machine learning.
Intrinsic data structures are often used as pseudo-supervisory signals in unsupervised feature selection. Thus, structure learning is frequently employed in unsupervised feature selection, and a particularly efficient method is graph-based regularization. It relies on the standard assumption that data points exhibiting similarity in the original data space should still demonstrate similarity in the transformed space [14,15]. From this principle arises the widely recognized graph-based Laplacian regularization term, critical for depicting the data’s intrinsic structure [16]. Despite its widespread use in feature selection, the graph-based Laplacian regularization faces certain limitations when it comes to geodesic and null space considerations. Furthermore, graph-based Laplacian regularization primarily captures pairwise linear relationships between samples and fails to model high-order geometric properties such as manifold curvature or topological connectivity [17].
In addition, the problem of feature selection can be conceptualized as seeking out the subset of features that optimally mirrors the initial feature space in a lower-dimensional subspace, which is fundamentally consistent with the low-rank approximation approach [18,19]. Low-rank approximation constitutes an effective structural prior that promotes compact latent dependency patterns while mitigating feature redundancy. Existing low-rank-based unsupervised feature selection methods usually exploit the intrinsic low-dimensional structure of data through the low-rank constraint [20], Schatten-p norm regularization [21], or nuclear-norm regularization [22]. In this work, low-rank approximation is employed not as a strict assumption on the raw input data matrix, but as a regularization principle to induce a compact and informative feature-selection structure.
Recent research in unsupervised feature selection has also extended toward deep representation learning, where autoencoder-based architectures jointly incorporate manifold preservation and feature selection within a unified training framework [23]. These methods are effective in modeling nonlinear latent structures, but they typically optimize representation learning and feature selection simultaneously. In this work, we instead focus on an interpretable, optimization-based formulation.
Against this backdrop, we aim to address the limitations of existing graph-based regularization while integrating the advantages of low-rank approximation. To this end, we propose a novel method termed Hessian-based Unsupervised Feature Selection using Low-Rank Approximation (HLR-UFS). First, we harness Hessian regularization to capture the local geometric structure of complex nonlinearities, consequently effectively preserving the data’s topological structure. Second, building upon self-representation principles, we impose a low-rank prior on the coefficient matrix to capture a compact global dependency structure among features, while -norm regularization is employed to improve robustness to noise and outliers. Finally, we compare our novel algorithm with both the baseline and several leading-edge feature selection methods using nine datasets that are openly available from a machine learning database. Broad-ranging experiments confirm that our method achieves superior clustering performance relative to other methods across the tested datasets. In summary, the primary contributions of this paper are highlighted as follows:
- We replace traditional Laplacian regularization with Hessian regularization to capture high-order geometric structures, overcoming the linearity limitation of traditional graph-based methods.
- We impose low-rank approximation on the coefficient matrix to capture shared latent structure among features and promote a compact feature-selection structure, while
-norm regularization enhances robustness to noise and outliers.
- We validate HLR-UFS extensively on nine datasets spanning images, biology, and mass spectrometry. Extensive experiments convincingly demonstrate the superiority of HLR-UFS.
This paper is structured as follows: Sect 2 provides a comprehensive review of existing methods. Sect 3 introduces the proposed HLR-UFS framework. Sect 4 presents the IRLS-based optimization algorithm, algorithmic procedure, and computational complexity analysis of HLR-UFS. Sect 5 analyzes the algorithm’s optimization convergence. Sect 6 displays and analyzes the results of experiments. Finally, Sect 7 concludes with an overview of the paper and future research directions.
2. Related work
Over the past few decades, a variety of unsupervised feature selection techniques have been proposed to address high-dimensional data when label information is unavailable or limited. Zhu et al. employ the self-representation property and -norm regularization to effectively identify representative features [24]. Tang et al. integrate latent representation learning within the feature selection process to leverage the interrelations among data and conduct feature selection in the latent space, thereby enhancing noise resilience [25]. Ma et al. incorporate a latent relationship penalty term into unsupervised feature selection to account for sample uniqueness, using an attribute score strategy to mitigate noise interference and preserve data structure [26]. Liu et al. employ
-norm feature reconstruction with an adaptive distance metric, graph learning, and inner product regularization to enhance robustness, preserve data structure, and ensure feature sparsity [27].
A primary concern of unsupervised feature selection is maintaining the inherent local geometric relationships within the dataset [28]. Therefore, the graph-based Laplacian regularization term is frequently adopted in embedded unsupervised feature selection methods. He et al. propose the Laplacian score, which evaluates features based on their ability to preserve local structures [29]. Hu et al. leverage a Laplacian-based graph regularizer within a feature-level self-representation framework to capture local manifold structure and conduct unsupervised feature selection via -norm sparsity [30]. Cai et al. leverage the Laplacian matrix to capture the multi-cluster structure of data and effectively select features preserving the data’s intrinsic manifold structure through sparse regression [31]. However, graph-based Laplacian regularization is frequently employed to preserve local geometric structures by enforcing smoothness over similarity graphs, which leads to the fact that Laplacian regularization mainly captures the pairwise linear relationships between samples, and cannot model high-order geometric attributes such as manifold curvature or topological connectivity [32,33].
Hessian regularization offers superior extrapolation capabilities compared to Laplacian regularization because of its broader null space. This regularization prefers functions that undergo linear transformations based on geodesic distance, thus effectively retaining the data’s topological structure [34]. The Hessian locally linear embedding technique employs second-order derivative constraints to maintain manifold curvature and effectively captures the intrinsic curvature as well as the topological connectivity of data manifolds [35]. Sheikhpour et al. employ a generalized uncorrelated constraint with mixed -norm regularization, leveraging Hessian regularization to preserve the data’s topological structure [36]. Although Hessian-based graph learning is often regarded as particularly suitable for semi-supervised settings, where limited label information can guide structure preservation, its ability to capture high-order local geometry also makes it useful for unsupervised representation learning. Moslemi et al. effectively preserve the geometrical structure of data and select the most discriminative features by utilizing Hessian regularized latent representation learning with an
-norm constraint [17].
In addition, low-rank approximation seeks to capture the global correlation structure among features, making it an important method in feature selection. Wang et al. tackle the feature selection problem by employing matrix factorization coupled with low-rank constraints, thereby capitalizing on the benefits of both the low-rank characteristics and the structure learning aspects [20]. Zheng et al. learn a data dictionary with low-rank constraints and perform feature reconstruction to effectively reduce redundant features and noise while preserving the sample distribution in multiple subspaces [37]. Chen et al. minimize the weighted distance between simplified matrix representations to address issues in existing methods that neglect the local structure of sample data [38].
Besides classical model-driven unsupervised feature selection, recent studies have explored deep unsupervised feature selection by coupling feature selection with latent representation learning. Han et al. propose an autoencoder-inspired unsupervised feature selection method that combines autoencoder regression with group lasso regularization, enabling the model to identify informative features by exploiting nonlinear relationships among features [23]. Moslemi and Jamshidi develop an autoencoder-based unsupervised feature selection framework with sparse manifold learning, in which both the encoding and decoding processes are regularized and the geometric structure of the input data is preserved through Laplacian graph constraints [39]. Song et al. propose a dual-graph autoencoder for unsupervised feature selection, which incorporates graph regularization in both the data space and feature space while imposing sparsity-promoting norm constraints on the encoder and decoder to improve robustness and structural preservation [40]. These methods generally integrate autoencoder-based reconstruction, manifold preservation, and graph regularization into neural architectures, thereby enhancing nonlinear representation learning and improving feature discrimination.
3. The proposed method
3.1 Notations
To start, we review some essential symbols that will be consistently applied in this paper. For a column vector , its Euclidean norm is defined as
. For an arbitrary matrix
, the
-norm is given by
. The rank operator
applied to
returns its matrix rank. When
represents a square matrix in
, its trace is computed as
.
3.2 Problem formulation
We denote the data matrix as where
is a feature vector and
is a sample. Employing the regularized self-representation of the data matrix [24], the unsupervised feature selection problem can be mathematically expressed as
where ,
is a d-dimensional column vector, and
is a parameter that balances the accuracy of the model fit with the regularization strength. Here,
denotes the coefficient matrix used to identify the selected features, the term
acts as a regularization factor aimed at promoting sparsity and addressing outliers, while
represents the significance of the i-th feature. A higher value of
indicates a higher importance of the corresponding feature.
3.3 Low-rank approximation
To explicitly reduce feature redundancy and avoid degenerate self-representation solutions, we impose a low-rank constraint on the coefficient matrix . It can not only identify latent correlations among features but also help avoid degenerate or trivial solutions in the self-representation model. The fundamental objective of unsupervised feature selection lies in identifying a compact subset of features that optimally preserves the essential characteristics of the initial high-dimensional space. The low-rank approximation is imposed on the coefficient matrix
to promote a compact and informative low-dimensional structure for feature selection [20]. Then we can obtain the following model:
where l denotes the number of selected features, and the condition imposes a low-rank constraint ensuring that
serves as a pseudo-indicator matrix for feature selection. By explicitly incorporating the rank constraint into the optimization objective, the learned coefficient matrix is encouraged to exhibit a compact structure for feature selection.
3.4 Hessian regularization
Structure learning plays a pivotal role in addressing unsupervised learning challenges. Nonetheless, the aforementioned optimization problem overlooks the inherent structural characteristics of the input data. Motivated by the principles of manifold learning, we integrate a manifold regularization term to preserve local similarity within the data [17,36]. It solves the problem of Laplacian regularization being unable to effectively preserve complex topological structures and nonlinear geometric information of data due to null-space constraints.
Following standard Hessian regularization, we introduce a local geometric smoothness assumption in constructing the regularizer. This assumption is introduced as a local approximation for neighborhood structure learning, rather than a global smooth-manifold requirement on the data.
Let denote a manifold. The Hessian energy for the mapping
is defined as:
where represents the local cotangent space at point
, f is the mapping function, and
denotes the second covariant derivative of f. The term
signifies the natural volume element on the manifold M. Since the Hessian energy depends on the manifold geometry, it can be locally evaluated using normal coordinates on
. The normal coordinates for a designated point
are defined as the coordinates on M such that the manifold is locally approximated as Euclidean around x. Consequently, normal coordinates
and
centered at
are derived through the following methodology:
Eq. (4) elucidates that at the point , the norm of the second covariant derivative corresponds to the Frobenius norm of the Hessian matrix of f within the normal coordinates.
We utilize the k nearest neighbors to establish the local normal coordinate system. Then, the local tangent space
is approximated using the PCA technique, where the top m eigenvectors serve as the estimated orthogonal basis for
. Following the classical Hessian regularization construction, this step provides a local linear approximation of the underlying manifold. Therefore, its quality depends on the neighborhood quality: when the neighborhood samples lie close to a low-dimensional manifold, the approximation is reliable; however, heavy noise or severe outlier contamination may deteriorate the estimated tangent space and consequently affect the resulting Hessian matrix. Consequently, the normal coordinates
and
of a neighboring point
, for
, can be approximated through the local tangent space
. The Hessian of f at
is approximated as described below:
where represents the local Hessian operator for the sample
in the normal coordinates
and
of a point
. The operator
can be determined by fitting a second-order polynomial
in these normal coordinates to the values
for
. Moreover, following the Hessian-energy estimator in [41], we fix the zeroth-order term at
in the local second-order polynomial fitting, which improves the stability of the subsequent Hessian estimation and helps avoid degenerate polynomial fits, as detailed below:
where denotes the zeroth-order term,
denotes the first-order coefficient, and
denotes the second-order coefficient. Eq. (6) can be interpreted as the second-order Taylor expansion of f centered at
, as the neighborhood size diminishes towards zero. Subsequently,
and
are:
with . In order to estimate the value of the function at point
and its adjacent points by using the basis function in the local coordinate system, and then calculate the local Hessian matrix, we utilize the method of ordinary least squares to fit the polynomial as depicted in Eq. (8).
where represents the design matrix with
. The corresponding basis functions
consist of the monomials of the normal coordinates (centered at
) of
up to the second order. The solution to Eq. (8) is given by
, where
,
, and † denotes the Moore-Penrose inverse.
To avoid confusion, we assume and
. The Frobenius norm of the Hessian of f at
is approximated by the following expression:
where . Ultimately, the Hessian energy function is approximated by the following formula:
where and
represents the data in the local tangent space.
Then, can be expressed through
and
, that is,
. Here,
is the coefficient matrix that maps the original feature space
to the learned representation space. Substituting
into the Hessian energy function, we obtain
. It is worth noting that the effectiveness of this regularizer depends on the quality of local neighborhood approximation. In cases where the local geometry is highly irregular or non-smooth, the Hessian term may offer weaker structural characterization, although the overall framework remains applicable.
Incorporating the Hessian regularization into our framework Eq. (2), the final objective function is presented as follows:
where is a coefficient that balances local geometric-structure preservation and reconstruction error.
4. Optimization and algorithm
4.1 Optimization
Utilizing the property that , and given that
is positive semi-definite, the magnitude of
corresponds to the number of non-negative eigenvalues of
. Therefore, based on
, it follows that
has
eigenvalues equal to zero. Let
denote the i-th singular value of matrix
. We subsequently use the expression
to approximate the rank of
. By penalizing the sum of the
smallest eigenvalues of
, the low-rank constraint in Eq. (11) is relaxed into a penalized approximation. In accordance with Ky Fan’s Theorem [42], we introduce a variable
to approximate the sum of the
smallest singular values, thus ultimately leading to the following relationship:
Employing the method outlined above to convert the low-rank constraint into an objective function for optimization, Eq. (11) can be reformulated as follows:
4.1.1 Update 
with fixed 
.
The optimization problem mentioned above, concerning , is equivalently reformulated in the subsequent expression:
To address this, we apply the Lagrange multiplier method to this constrained optimization problem:
where is the Lagrange multiplier matrix. Next, we compute the derivatives with respect to
and
:
Setting and
yields the optimal solution
and the orthogonality of
.
4.1.2 Update 
with fixed 
.
Owing to the non-smooth nature of the -norm, the iterative reweighted least-squares (IRLS) algorithm was introduced to tackle this issue [43,44]. We define the diagonal weighting matrices
and
, where
and
.
Furthermore, according to the operational properties of matrix trace , we have
Subsequently, is updated by addressing the following weighted least squares issue:
The closed-form solution for is given by
where denotes the identity matrix. A sufficiently small value
is incorporated by defining
and
to avoid division by zero.
4.2 Algorithm and computational complexity analysis
The preceding discussion has addressed each variable under optimization individually. Building on these analyses, the procedure for our method is encapsulated in Algorithm 1.
Algorithm 1. Hessian-based Unsupervised Feature Selection using Low-Rank Approximation (HLR-UFS)
Input: Data matrix , target feature number l, parameters
.
Output: Selected feature subset I.
1 Initialize ;
2 Construct Hessian regularization matrix via Eq. (10).
repeat
3 Update as eigenvectors of
corresponding to the smallest
eigenvalues.
4 Compute diagonal matrices (
) and
(
).
5 Update .
until Convergence
6 Sort (
) in a descending sequence and select top-l indices as I.
The proposed HLR-UFS algorithm involves two key subproblems: updating the coefficient matrix and the low-rank projection matrix
. The time complexity of each iteration is dominated by matrix operations.
- Updating
: To solve the orthogonal constraint problem via eigenvalue decomposition of
, the time complexity is O(d3) per iteration.
- Updating
: The predominant computational cost rests in calculating the inverse of a
matrix in Eq. (19). Using the Woodbury identity, the complexity is reduced from O(d3) to
. Overall, the complexity for updating
is
.
Combining these steps, each iteration of the HLR-UFS algorithm has the time complexity of . The overall complexity depends on the number of iterations T needed for convergence, thereby yielding a total time complexity of
.
5. Convergence analysis
In this section, we theoretically analyze the convergence of the proposed HLR-UFS algorithm. Let denote the original objective function:
To handle the non-smoothness of L, we construct the following surrogate function J at the t-th iteration:
where and
are diagonal matrices with their diagonal entries defined as:
Theorem 1. is a surrogate function for
. More specifically, for any
, we have
, and the equality holds if and only if
.
Proof of Theorem 1. For any vector and any nonzero vector
, the following inequality holds:
where equality holds when .
We apply the inequality in Eq. (22) to each row of the residual matrix . Let
and
denote the i-th row of
and
, respectively. Then for each row
, we have:
Similarly, applying the inequality to each row of the coefficient matrix gives:
Let . In the following, we will prove that for any
, there is
. First,
can be evaluated as:
Then, can be rewritten as:
Because and
(in practice, an infinitesimal constant is added to avoid division by zero), and the numerators are perfect squares, we have
. Consequently, J is a valid surrogate function satisfying
.
Theorem 2. The proposed HLR-UFS algorithm yields a monotonically non-increasing original objective function in each iteration, i.e.,:
Proof of Theorem 2. The alternating minimization process can be decomposed into two steps. We will prove that neither updating nor updating
will increase the objective function.
- Step 1: Updating
. Given
, the t-th iteration aims to solve
. Because
is the minimizer of J, we have
. Combining this with Theorem 1, we obtain:
(28)
Thus, updating does not increase the objective function.
This is a standard eigenvalue problem, and its closed-form solution is given by the eigenvectors corresponding to the smallest eigenvalues of
. Due to the optimality of
, we have:
Since the other terms in L are independent of , it yields:
where the first inequality stems from the fact that is a surrogate function, and the second inequality results from the optimality of
.□
6. Experiments
Since HLR-UFS is an optimization-based unsupervised feature selection method, the main experimental comparisons are conducted against representative non-deep baselines under a unified evaluation framework in which clustering performance is assessed on the selected features.
6.1 Datasets
Experiments have been carried out across nine distinct benchmark datasets: the COIL20 object image dataset, the Arcene mass spectrometry dataset, the USPS digit image dataset, and four facial image datasets (warpAR10P, JAFFE, orlraws10P, and Yale_32x32), as well as two biomedical datasets (TOX-171 and Prostate-GE). The characteristics of these datasets are summarized in Table 1. Given their diverse data sources, these datasets provide an effective platform for thoroughly evaluating the performance of various methods.
6.2 Comparison algorithms
We compare the proposed method with several state-of-the-art unsupervised feature selection methods to demonstrate its effectiveness and efficiency.
- Baseline: All original features are used without feature selection.
- LS [29]: It relies on Laplacian eigenmaps and the projection that maintains localities, and it assesses the features by their ability to preserve locality.
- RUFS [45]: It integrates robust non-negative matrix factorization with local learning regularization and employs joint
-norm minimization to enhance feature selection performance.
- LRSL [20]: It formulates feature selection as a matrix decomposition problem, where low-rank constraints and graph-based Laplacian regularization are jointly optimized.
- MRSR [46]: It preserves local data structures via
-norm reconstruction and sparsity constraints, optimized by iterative reweighted least squares.
- SLNMF [47]: It uses convex nonnegative matrix factorization to reduce dimensionality and enhance sparsity, while soft-label learning guides the feature selection process by capturing the ambiguous associations between data points and the centroids of clusters.
- RAFG [48]: An unsupervised feature selection method using adaptive
-norm graph learning. It employs
-norm regression and
-norm sparsity for noise-robust feature ranking and structure preservation.
- LRPFS [26]: It introduces a new latent relationship penalty term to preserve the data structure and uniqueness of samples, and employs an
-norm sparsity constraint to enhance computational efficiency.
6.3 Parameter setting
In order to evaluate the efficacy of different feature selection methods, it is necessary to set certain parameters beforehand. First, for our proposed HLR-UFS algorithm, we set the structure learning parameter to . The low-rank parameter
is selected from
. Moreover, to control the sparsity of feature selection, we tune
over
. Since the local Hessian estimate relies on local normal coordinates constructed from k-nearest neighborhoods and PCA-based tangent-space approximation, we tune the k-nearest-neighbor parameter within the range of
for each dataset when constructing the Hessian matrix, and select the PCA subspace dimensionality accordingly. Second, for all comparison algorithms, the parameters of the different feature selection algorithms are set according to the default or recommended settings reported in their original papers. For LS, k = 5 is set to specify the number of neighbors. For LRSL, the structure learning parameter and the penalty weight parameter are fixed
and
. For MRSR and RUFS, the regularized coefficient is tuned to 10. For SLNMF, the regression coefficient and the regularized term coefficient are set as
and
. For LRPFS, the penalty weight parameter and the sparse term parameter are set as 1. For RAFG, the sparse regularization parameter and the sparse level parameter are fixed as
,
, p = 0.7, respectively. Finally, we configure the number of selected features for the feature selector to range from 20 to 100 in increments of 10. Additionally, the maximum number of iterations is capped at 20 for all tested datasets. We evaluate the clustering performance using the k-means clustering algorithm with a specified number of features. Notably, the number of clusters is set to match the number of true classes for each dataset. Considering that distinct initializations can lead to diverse clustering outcomes, each experiment is conducted 30 times with random initial values. We present the mean value and standard deviation for each comparative algorithm.
6.4 Evaluation metrics
We evaluate the efficacy of unsupervised feature selection algorithms through the application of two widely recognized evaluation metrics: clustering accuracy (ACC) and normalized mutual information (NMI).
Here, represents the ground-truth labels provided by the dataset, while
denotes the clustering labels obtained from the unsupervised feature selection algorithms. The ACC is defined as follows:
Here, the delta function is defined such that
if a = b, and
otherwise. Additionally,
denotes the mapping function that aligns the obtained clustering label
with the corresponding label in the dataset.
Consider two random variables P and Q, where P represents the true labels and Q denotes the clustering results. The normalized mutual information (NMI) between P and Q is defined as follows:
Here, I(P; Q) denotes the mutual information between P and Q, while H(P) and H(Q) represent the entropies of P and Q, respectively. It is important to note that higher values of ACC and NMI indicate better clustering performance.
6.5 Result and analysis
In this section, we evaluate the performance of different methods on nine datasets, with a focus on ACC and NMI as the principal metrics for assessment. The ACC results are presented in Table 2, and the NMI results are detailed in Table 3. Moreover, the t-SNE visualizations and running time comparison are shown in Figs 1 and 2, respectively. These tables and figures offer a comprehensive comparison of the performance metrics and computational efficiency across various methods and datasets. From the experimental results depicted above, the following observations are made:
- 1) It can be seen that our method demonstrates superior performance compared to the state-of-the-art methods on nearly all tested datasets and delivers better outcomes than the Baseline, suggesting that our method can effectively exclude irrelevant or redundant features.
- 2) Our HLR-UFS method has better performance than the LRSL and MRSR algorithms that employ the Laplacian matrix to preserve geometrical information. For example, our method outperforms the LRSL algorithm by 6.10% in ACC on the warpAR10P dataset. This may be because compared to these two technologies, Hessian regularization possesses a richer null space than Laplacian regularization, which results in superior preservation of topological information. To further provide a direct comparison, we visualize the t-SNE embeddings of the feature subsets selected by LRSL, MRSR, and HLR-UFS on the orlraws10P dataset. As shown in Fig 1, HLR-UFS produces more compact and better-separated clusters, with higher silhouette coefficients (Sil) than the Laplacian-based methods. These results provide additional evidence that the proposed Hessian-based regularization better preserves the intrinsic higher-order geometric structure of the data.
- 3) HLR-UFS is slightly inferior to RAFG, the top-performing method on the arcene dataset. One possible reason is that RAFG employs a more flexible
-norm sparsity regularization, which may better adapt to the sparsity structure of the data and more effectively remove redundant features.
- 4) From Fig 2, HLR-UFS exhibits substantially shorter running time than RAFG on both TOX-171 and arcene. This is likely because RAFG combines adaptive
-norm graph learning with
-norm regularization and requires alternating iterative updates. HLR-UFS is also slightly faster than LRSL, possibly because LRSL optimizes a low-rank approximation model with a fixed structure-learning regularizer through alternating updates of multiple variables, which introduces additional iterative overhead.
Additionally, to demonstrate the impact of feature selection on clustering performance, Figs 3 and 4 provide detailed performance comparisons of all algorithms across different numbers of selected features for various datasets. On one hand, from Fig 3, the ACC results of HLR-UFS consistently outperform those of the compared algorithms across all feature numbers on the COIL20, JAFFE and Yale_32x32 datasets. On the orlraws10P and Prostate-GE datasets, when selecting a larger quantity of features, our method shows superior performance to other algorithms. On the other hand, as depicted in Fig 4, HLR-UFS also obtains the best NMI results for most feature numbers on the COIL20, JAFFE, USPS and arcene datasets. Moreover, HLR-UFS remains competitive with other methods on TOX-171 and Prostate-GE. The experimental findings consistently confirm the efficacy and advantages of the proposed HLR-UFS algorithm.
6.6 Parameter sensitivity
To assess the impact of the regularization parameter and the k-nearest-neighbor parameter on the clustering performance of HLR-UFS, we conduct a sensitivity analysis for each parameter individually. Experiments are conducted by fixing
and varying k from {2,3,...,10} or fixing k = 5 and varying
from
, respectively.
From Fig 5, we observe that the ACC values of HLR-UFS method decrease slightly when is larger on the COIL20 and orlraws10P datasets and the ACC values fluctuate slightly when
is smaller on the warpAR10P and arcene datasets. From Fig 6, the ACC values fluctuate slightly within a certain range on the warpAR10P and arcene datasets while remaining more stable on the other datasets. Notably, Fig 7 shows significant fluctuations in NMI with changes in
on datasets like Prostate-GE, TOX-171 and arcene, indicating that the corresponding regularization terms play a vital role in the proposed model. Fig 8 indicates that the NMI exhibits pronounced fluctuations with varying k values on Prostate-GE, warpAR10P and arcene datasets, while it demonstrates robust performance on other datasets. Overall, these results indicate that the proposed method is generally robust to variations in both
and k, as the ACC and NMI values remain relatively stable on most datasets. Meanwhile, the observed fluctuations on several datasets suggest that the parameter selection may still influence the balance of regularization terms and the reliability of Hessian-based local structure estimation.
6.7 Ablation study
The objective function of HLR-UFS contains three core components: Hessian regularization (controlled by ), low-rank approximation (controlled by
), and the
-norm sparsity term (controlled by
). To quantify the contribution of each component, we conduct an ablation study by systematically removing one or more components and examining the resulting change in clustering performance. Specifically, we evaluate the full model and seven ablated settings on the Prostate-GE and JAFFE datasets:
- Full Method: the complete HLR-UFS model with all three components enabled (
).
: the Hessian regularization is removed, while the low-rank approximation and sparsity term are retained.
: the low-rank approximation is removed, while the Hessian regularization and sparsity term are retained.
: the sparsity term is removed, while the Hessian regularization and low-rank approximation are retained.
: both the Hessian regularization and low-rank approximation are removed.
: both the Hessian regularization and sparsity term are removed.
: both the low-rank approximation and sparsity term are removed.
: all three components are removed.
Additionally, the number of neighbors k is fixed at 5 in all settings. Table 4 reports ACC and NMI results.
According to Table 4, removing any single component leads to clear performance degradation on both datasets. For example, when , the ACC on Prostate-GE decreases from 69.51% to 60.78%, and the ACC on JAFFE decreases from 76.74% to 74.19%, indicating that the low-rank approximation contributes positively to the overall performance. Similar degradation can also be observed when the Hessian regularization or the sparsity term is removed.
When multiple components are removed, the performance generally deteriorates further. For instance, when , the ACC on JAFFE drops to 72.52%, which is lower than that of the corresponding single-component removals. Removing all three components further reduces the ACC/NMI to 57.97/2.99 on Prostate-GE and 72.82/77.37 on JAFFE. Overall, these results suggest that Hessian regularization, low-rank approximation, and
-norm sparsity are complementary within the proposed framework, and their joint use yields the best performance among the compared settings.
6.8 Fixed Hessian regularization vs dynamic graph learning
Graph-based regularization techniques traditionally rely on a precomputed affinity matrix that remains fixed throughout optimization. Although effective for preserving local geometry, such a static structure may be sensitive to noise and inaccurate neighborhood estimation, since errors in the initial graph may be propagated during learning. Recent studies have therefore explored dynamic graph learning mechanisms that iteratively adapt the affinity matrix [49]. To further examine whether explicit graph updating improves robustness, we compare our fixed-Hessian framework with a representative dynamic Laplacian-based structure learning method. In the dynamic baseline, the affinity matrix is updated in each iteration via:
In Eq. (31), denotes an all-ones vector,
is the i-th row of the affinity matrix,
denotes the connectivity weight between samples i and j, and
is introduced to avoid trivial solutions. The resulting adaptive affinity matrix is denoted by
and is iteratively refined during training.
We compare this dynamic Laplacian-based structure learning (DSL) baseline with our fixed structure learning (FSL) framework, with empirical results summarized in Tables 5 and 6. For a cleaner comparison, the sparse regularization term is removed from our formulation, yielding the FSL objective:
Results in Tables 5 and 6 show that the dynamic baseline does not exhibit a consistent advantage over FSL. In fact, FSL achieves better performance on three of the five datasets, indicating that the proposed fixed-Hessian framework remains competitive even without iterative graph refinement.
6.9 Qualitative analysis
To further investigate the robustness of HLR-UFS under structured non-Gaussian perturbations, we provide a qualitative case study on the warpAR10P dataset. We consider sunglasses as a representative form of structured perturbation and visualize the learned feature weights together with the original and reconstructed faces.
As shown in Fig 9(a), the learned feature weights still emphasize several salient facial regions, especially around the eyes, nose, and mouth. Correspondingly, the reconstructed clean face in Fig 9(c-1) remains visually close to the original face in Fig 9(b-1), suggesting that HLR-UFS can preserve the main facial structure in the clean case.
(a) learned feature weights; (b-1) clean face; (b-2) face with sunglasses occlusion; (c-1) reconstruction of the clean face; (c-2) reconstruction of the occluded face.
We then examine the case of sunglasses occlusion. By comparing Fig 9(b-2) and Fig 9(c-2), it can be observed that the occluded region remains largely visible in the reconstructed face. This suggests that improving robustness to structured perturbations may require more adaptive and robust Hessian-based structure learning mechanisms in future work.
6.10 Convergence study
In this subsection, we carry out an empirical analysis to examine the convergence characteristics across all the datasets that have been tested. For this purpose, we set the parameter to 1 and k to 5. Fig 10 visually depicts the convergence behavior of the HLR-UFS objective function across multiple datasets. The objective function value is observed to decrease monotonically and subsequently stabilizes following a specific number of iterations, which substantiates the accuracy and efficacy of the convergence analysis detailed in Section 5. Moreover, it is worth noting that HLR-UFS rapidly reduces the objective value and converges within five iterations on all datasets. This indicates that the optimization algorithm used for training our model demonstrates strong convergence properties.
7. Conclusions and future work
In this paper, we introduce an innovative framework for unsupervised feature selection, Hessian-based Unsupervised Feature Selection using Low-Rank Approximation (HLR-UFS). The HLR-UFS method addresses critical limitations in existing approaches by unifying three synergistic components: Hessian regularization to model intrinsic manifold curvature and topological connectivity, low-rank constraints to eliminate feature redundancy, and -norm regularization to suppress noise and outliers. An efficient optimization algorithm based on iterative reweighted least squares is developed to solve the non-convex problem, together with convergence and complexity analysis. Finally, through comprehensive experiments conducted on nine datasets, the proposed HLR-UFS algorithm demonstrates superior performance and enhanced robustness.
Although HLR-UFS achieves competitive performance, future work will mainly focus on two aspects. First, we will investigate more adaptive and robust Hessian-based structure learning, such as dynamic neighborhood refinement and adaptive weighting, to better handle unreliable local neighborhoods, noise, and non-uniform sampling effects. Second, we will explore more scalable implementations for ultra-high-dimensional data, particularly approximate Hessian construction and distributed optimization.
References
- 1.
Khadse S, Gourshettiwar P, Pawar A. A review on meta-learning: how artificial intelligence and machine learning can learn to adapt quickly. 2025 International Conference on Electronics and Renewable Systems (ICEARS). IEEE; 2025. p. 2038–43.
- 2. Safarpour MH. SimGroup-FS: multi-way redundancy modeling for interpretable feature selection in supervised learning. IEEE Access. 2026;14:30327–38.
- 3.
Chen C, Zhang X, Niu Y, Li B, Zhou L. SE-RSRNet: rain streak removal by feature selection and feature extraction. Appl Soft Comput. 2026:114702.
- 4. Seo M, Oh S. CBFS: high performance feature selection algorithm based on feature clearness. PLoS One. 2012;7(7):e40419. pmid:22792310
- 5.
Huang Y, Yan J, Lu X, Zhao L. Explainable reinforcement learning with adaptive feature selection for stock trading. Appl Soft Comput. 2026:114543.
- 6. Zhou J, Zhang Q, Zeng S, Zhang B, Fang L. Latent linear discriminant analysis for feature extraction via isometric structural learning. Pattern Recognit. 2024;149:110218.
- 7. Lu X, Long J, Wen J, Fei L, Zhang B, Xu Y. Locality preserving projection with symmetric graph embedding for unsupervised dimensionality reduction. Pattern Recognit. 2022;131:108844.
- 8. Moslemi A, Ahmadian A. Dual regularized subspace learning using adaptive graph learning and rank constraint: unsupervised feature selection on gene expression microarray datasets. Comput Biol Med. 2023;167:107659. pmid:37950946
- 9. Wang X, Wu P, Xu Q, Zeng Z, Xie Y. Joint image clustering and feature selection with auto-adjoined learning for high-dimensional data. Knowl-Based Syst. 2021;232:107443.
- 10. Fu Y, Wu Q, Liu K, Gao H. Feature selection methods for extreme learning machines. Axioms. 2022;11(9):444.
- 11. Odhiambo Omuya E, Onyango Okeyo G, Waema Kimwele M. Feature selection for classification using principal component analysis and information gain. Expert Syst Appl. 2021;174:114765.
- 12. Solorio-Fernández S, Carrasco-Ochoa JA, Martínez-Trinidad JF. A review of unsupervised feature selection methods. Artif Intell Rev. 2019;53(2):907–48.
- 13. Fira M, Goras L, Costin H-N. Evaluating sparse feature selection methods: a theoretical and empirical perspective. Appl Sci. 2025;15(7):3752.
- 14. Ali M, Ali SI, Kim D, Hur T, Bang J, Lee S, et al. uEFS: An efficient and comprehensive ensemble-based feature selection methodology to select informative features. PLoS One. 2018;13(8):e0202705. pmid:30153294
- 15. Zhu X, Hu R, Lei C, Thung KH, Zheng W, Wang C. Low-rank hypergraph feature selection for multi-output regression. World Wide Web. 2017;22(2):517–31.
- 16. Hu R, Cheng D, He W, Wen G, Zhu Y, Zhang J, et al. Low-rank feature selection for multi-view regression. Multimed Tools Appl. 2016;76(16):17479–95.
- 17. Moslemi A, Shaygani A. Subspace learning via Hessian regularized latent representation learning with l2,0-norm constraint: unsupervised feature selection. Int J Mach Learn Cyber. 2024;15(11):5361–80.
- 18. Zhou Q, Wang Q, Gao Q, Yang M, Gao X. Unsupervised discriminative feature selection via contrastive graph learning. IEEE Trans Image Process. 2024;33:972–86. pmid:38241117
- 19.
Chenakkod S, Dereziński M, Dong X, Rudelson M. Optimal embedding dimension for sparse subspace embeddings. Proceedings of the 56th Annual ACM Symposium on Theory of Computing; 2024. p. 1106–17.
- 20. Wang S, Wang H. Unsupervised feature selection via low-rank approximation and structure learning. Knowl-Based Syst. 2017;124:70–9.
- 21. Peng M, Zhang H. Unsupervised feature selection via Schatten-p norm and feature self-representation. Comput Eng Appl. 2020;56(23):45–52.
- 22. Yang X, Che H, Leung M-F. Tensor-based unsupervised feature selection for error-robust handling of unbalanced incomplete multi-view data. Inf Fusion. 2025;114:102693.
- 23.
Han K, Wang Y, Zhang C, Li C, Xu C. Autoencoder inspired unsupervised feature selection. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2018. p. 2941–5.
- 24. Zhu P, Zuo W, Zhang L, Hu Q, Shiu SCK. Unsupervised feature selection by regularized self-representation. Pattern Recognit. 2015;48(2):438–46.
- 25. Tang C, Bian M, Liu X, Li M, Zhou H, Wang P, et al. Unsupervised feature selection via latent representation learning and manifold regularization. Neural Netw. 2019;117:163–78.
- 26. Ma Z, Huang Y, Li H, Wang J. Unsupervised feature selection with latent relationship penalty term. Axioms. 2023;13(1):6.
- 27. Liu W, Ning Q, Liu G, Wang H, Zhu Y, Zhong M. Unsupervised feature selection algorithm based on L2,p-norms feature reconstruction. PLOS ONE. 2025;20(3):e0318431.
- 28. Huang Y, Shen Z, Cai F, Li T, Lv F. Adaptive graph-based generalized regression model for unsupervised feature selection. Knowl-Based Syst. 2021;227:107156.
- 29. He X, Cai D, Niyogi P. Laplacian score for feature selection. Adv Neural Inf Process Syst. 2005;18:507–14.
- 30. Hu R, Zhu X, Cheng D, He W, Yan Y, Song J, et al. Graph self-representation method for unsupervised feature selection. Neurocomputing. 2017;220:130–7.
- 31.
Cai D, Zhang C, He X. Unsupervised feature selection for multi-cluster data. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2010. p. 333–42.
- 32. Belkin M, Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering. Adv Neural Inf Process Syst. 2001;14:585–91.
- 33. Shi C, Ruan Q, An G, Zhao R. Hessian semi-supervised sparse feature selection. IEEE Trans Multimedia. 2014;17(1):16–28.
- 34. Laurian-Ioan P, Ali A, Cătălin B, Alkhaldi AH. The χ-Hessian quotient for riemannian metrics. Axioms. 2021;10(2):69.
- 35. Moslemi A. A tutorial-based survey on feature selection: recent advancements on feature selection. Eng Appl Artif Intell. 2023;126:107136.
- 36. Sheikhpour R, Berahmand K, Forouzandeh S. Hessian-based semi-supervised feature selection using generalized uncorrelated constraint. Knowl-Based Syst. 2023;269:110521.
- 37. Zheng W, Xu C, Yang J, Gao J, Zhu F. Low-rank structure preserving for unsupervised feature selection. Neurocomputing. 2018;314:360–70.
- 38. Chen T, Chen X. Sparse low-rank approximation of matrix and local preservation for unsupervised image feature selection. Appl Intell. 2023;53(21):25715–30.
- 39. Moslemi A, Jamshidi M. Unsupervised feature selection using sparse manifold learning: Auto-encoder approach. Inf Process Manag. 2025;62(1):103923.
- 40. Song Z, Chen M, Xie L, Fang X. Unsupervised feature selection via a dual-graph autoencoder with l2,1/2-norm for [68Ga]Ga-Pentixafor PET imaging of glioma. Appl Sci. 2025;15(11):6177.
- 41.
Kim K, Steinke F, Hein M. Semi-supervised regression using Hessian energy with an application to semi-supervised dimensionality reduction. Adv Neural Inf Process Syst. 2009:979–87.
- 42. Fan K. On a theorem of Weyl concerning eigenvalues of linear transformations I. Proc Natl Acad Sci U S A. 1949;35(11):652–5. pmid:16578320
- 43. Kümmerle C, Maly J. Recovering simultaneously structured data via non-convex iteratively reweighted least squares. Adv Neural Inf Process Syst. 2023;36:71799–833.
- 44. Wright SJ. Coordinate descent algorithms. Math Program. 2015;151(1):3–34.
- 45.
Qian M, Zhai C. Robust unsupervised feature selection. Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. Citeseer; 2013. p. 1621–7.
- 46.
Liang S, Xu Q, Zhu P, Hu Q, Zhang C. Unsupervised feature selection by manifold regularized self-representation. 2017 IEEE International Conference on Image Processing (ICIP). IEEE; 2017. p. 2398–402.
- 47. Zhou S, Song P, Song Z, Ji L. Soft-label guided non-negative matrix factorization for unsupervised feature selection. Expert Syst Appl. 2023;216:119468.
- 48. Jiang K, Cao T, Zhu L, Sun Q. Adaptive and flexible l1-norms graph embedding for unsupervised feature selection. Appl Intell. 2024;54(22):11732–51.
- 49. Nie F, Zhu W, Li X. Unsupervised feature selection with structured graph optimization. AAAI. 2016;30(1):1302–8.