Random LASSO and its limitations

Random LASSO introduced bootstrapping for robust analysis with high-dimensional data. Random LASSO consists of two procedures of bootstrapping [6]. The first procedure computes importance scores of predictors while approximating weights of variables by drawing multiple bootstrap samples. The second procedure estimates coefficients of the linear model on bootstrapping samples with weights of the importance scores, where predictors having higher importance scores have higher chances to be selected than lower ones. Then, the final estimations of the coefficients are computed by taking the average of the multiple estimates from the bootstrapping. Random LASSO deals with multicollinearity of different signs and identifies non-zero variables more than the sample size.

However, there are still several issues to improve. First, Random LASSO sets coefficients of predictors to zeros, even though the predictors are never selected in the bootstrapping. The unselected predictors may be possibly estimated as non-zero coefficients if being selected in the bootstrapping. Therefore, it introduces a systematic bias regardless its importance of the predictors. Moreover, the lower number of bootstrapping in Random LASSO generates the more systematic bias, where there may be more variables never selected. Note that the bootstrapping number directly affect computational costs in Random LASSO. Second, Random LASSO does not take advantage of global oracle property. Although Random LASSO uses bootstrapping with weights being proportional to importance scores of predictors in the second procedure, the final coefficients are estimated without the weights. Random LASSO may adopt adaptive LASSO to take the oracle property. However, Adaptive LASSO takes local weights of each bootstrapping in Random LASSO, where the local oracle property may vary depending on which predictors are involved in the bootstrapping. Finally, Random LASSO does not provide a statistical test to identify a set of features from multiple bootstrapping results. Random LASSO considers a heuristic threshold, for example, the reciprocal of the sample size, without statistical test, although the results of the feature selection substantially depend on the threshold.

Hi-LASSO improves Random LASSO

Hi-LASSO tackles the aforementioned limitations of Random LASSO. The contributions of Hi-LASSO are follows. (1) Hi-LASSO rectifies the systematic bias that Random LASSO introduces, by refining the process to compute importance scores. To prevent the systematic bias, Hi-LASSO considers the coefficient estimation of the unselected predictors as missing values on the bootstrapping in the procedures. (2) Hi-LASSO computes importance scores of variables by averaging absolute coefficients. The coefficient of a predictor may be assigned a different value or opposite sign with its estimate in different linear models with other predictors, specifically multicollinearity with different signs may often cause coefficient estimates of different signs over bootstrapping. Therefore, taking the absolute value of the sum of the coefficient estimates of bootstrapping in Random LASSO may reduce the importance score. (3) Hi-LASSO provides a statistical strategy to determine the number of bootstrapping. The determination of the number of bootstrap samples is crucial to ensure the performance for high-dimensional data. Some predictors may never be considered due to the nature of random sampling no matter how important they are in the model. Thus, Hi-LASSO considered a sufficient number of bootstrap for all predictors to be taken into account. (4) Hi-LASSO takes advantage of global oracle property by adopting Adaptive LASSO [3] in the second procedure. (5) Hi-LASSO uses parametric statistical tests for feature selection in bootstrap regression modeling (PSTFSboot). PSTFSboot allows Hi-LASSO to robustly perform feature selection from multiple bootstrapping results, as a filter feature selection, while most LASSO models are wrapper-based feature selection [9].

Hi-LASSO packages

We provide efficient solutions for performing Hi-LASSO with high-dimensional data, as Python and Apache Spark packages. We reduce the high-cost computations from a number of independent bootstrapping by using parallel processing. We also improve the scalability of Hi-LASSO by implementing the algorithm based on the Apache Spark engine. The Python package for Hi-LASSO (https://pypi.org/project/hi-lasso) and its Apache Spark version (https://pypi.org/project/hi-lasso-spark) are available through PyPI and can be easily installed using Python PIP:

pip install hi-lasso // installation in Python

pip install hi-lasso-spark // installation in Apache Spark (Spark 3.0.0+)

Sample codes and troubleshooting guide are provided in the web pages (https://hi-lasso.readthedocs.io/en/latest/). Hi-LASSO includes the following hyper-parameters: ‘q₁’, ‘q₂’, ‘L’, ‘alpha’, ‘logistic’, ‘random_state’, and ‘n_jobs’. In Hi-LASSO, each procedure repeats a random selection of q_i predictors B times, where the subscript i denotes the first or second procedure, i ∈ {1, 2}. Smaller q_i causes more bootstrapping, which requires more computation cost. On the other hand, large q_i may make coefficient estimation less accurate, because the random selection of a large number of predictors is likely to include more multicollinearity. B is determined by L, which is a desired average number of times that a predictor is selected during the bootstrapping. At the end of the second procedure, the parametric statistical test for feature selection is performed with a threshold alpha (e.g., 0.01 or 0.05). logistic indicates if the model is based on a logistic regression model for binary classification or a linear regression model for regression problems. n_jobs sets the number of cores for parallel processing.

Simulation study

We assessed the feature selection performance of Hi-LASSO with PSTFSboot by comparing with current state-of-the-art LASSO methods, including LASSO, ElasticNet, Adaptive, Relaxed, Random, Recursive, and Precision. We generated synthetic data with six different scenarios: Dataset I—Dataset VI, where numbers of predictors and samples are varied and the ground truth of relevant features are known (S1 File). We measured F1-scores, where relevant variables (|β|>0) were considered as positive, and irrelevant variables (|β| = 0) as negative in a confusion matrix. F1-scores show how accurately a model identifies the set of relevant features as feature selection [11]. Note that the original paper of Hi-LASSO assessed the feature selection performance using F1-scores by a threshold that maximizes the Root Mean Square Error (RMSE) of the validation data without a parametric statistical test, whereas this study conducted the experiments with further feature selection process that statistically combines bootstrapping results (i.e., using PSTFSboot), which does not require validation data.

We tuned the hyper-parameters of the benchmark models. We optimized the hyper-parameters of Precision and Relaxed LASSO as their original papers proposed. For Hi-LASSO, we set L as 30 and q₁ and q₂ as the sample size. We set the hyper-parameters of q and B in Random LASSO and Recursive LASSO as Hi-LASSO did for the fair comparison. For the other benchmark models, the optimal hyper-parameters of L1 or L2-norm regularization (λ) were obtained to minimize the prediction error with inner 5-fold cross validation in the training.

We repeated the experiments ten times by randomly generating simulation data for reproducibility. The experimental results are shown in Table 1 and Fig 1A. Overall, Hi-LASSO outperformed the benchmark models on most of the datasets. Hi-LASSO produced the highest F1-scores in all experiments except Dataset III. However, when sample sizes are increased, Hi-LASSO constantly showed superior performance, at least 10–20% higher F1-scores, for feature selection with high-dimensional data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Experimental results.

(A) Comparison of F1-scores with the simulation data, (B) improvement of efficiency on the Hi-LASSO Python package and the Spark version, (C) comparison of F1-scores with the semi-real simulation data, and (D) robustness of feature selection.

https://doi.org/10.1371/journal.pone.0278570.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Experimental results for feature selection performance (F1-scores) with the simulation data.

https://doi.org/10.1371/journal.pone.0278570.t001

We also assessed the efficiency of the Hi-LASSO packages. Fig 1B shows the improved speedup of Hi-LASSO on a parallel processing in Python and Spark, comparing to Hi-LASSO’s implementation in the original paper, using a large-scale simulated data (Dataset V) on a machine with Intel Xeon Gold (24 cores × 2). The experimental results show that the packages in Python and Apache Spark enhanced the speedup to 3.75 and 4.83 times faster, respectively. The Spark version was approximately two times faster than the Python version. The details of the execution time are in S2 File.

Semi-real simulation study based on TCGA cancer data

We further conducted a semi-real simulation study based on cancer data, including Glioblastoma Multiforme (GBM), Low Grade Gliomas (LGG), breast cancer (BRCA), and ovarian cancer (OV) in The Cancer Genome Atlas Program (TCGA) repository (https://www.cancer.gov/tcga). We downloaded the cancer genomic data from https://www.cbioportal.org. For the semi-real simulation study, we used gene expression (e.g., microarray or RNA-seq) as independent variables and generated a response variable as follows: (1) We performed correlation analysis between survival months and gene expression; (2) We selected 20 genes with the highest correlation with survival months; (3) The regression coefficients () of the 20 genes were randomly generated from the normal distribution, (μ=4, σ²=1), preserving the sign of the correlation coefficient corresponding to the regression coefficient; (4) The coefficients of the other genes were set to 0; and (5) The synthetic survival months (the response variable) were generated from the linear combination of the gene expression (X) and the ground truth’s coefficients () and errors (ϵ) from the normal distribution with a mean of zero and the standard deviation of the logarithmic survival months, i.e., .

We then calculated F1-scores to evaluate the performance of feature selection with the semi-real data (Table 2 and Fig 1C). The experimental results showed that Random, Recursive, and Precision produced F1-scores close to zeros, whereas LASSO, ElasticNet, and Adaptive showed F1-scores between 0.0 and 0.4. Remarkably, Hi-LASSO presented F1-scores between 0.5 and 0.72. The numbers of non-zero variables identified by the benchmark methods are shown in the parentheses in Table 2. Hi-LASSO constantly identified 10∼17 relevant variables, which is closed to the 20 non-zero variables in the ground truth. Whereas, Random and Precision tended to select large numbers of variables (>1,000 non-zeros), which caused recall relatively higher but precision closed to zero. It may be because Random LASSO uses a reciprocal of sample size as a threshold for feature selection, and Precision LASSO is optimized to regression problems rather than feature selection. Recursive LASSO tends to introduce extreme bias in the first bootstrapping for feature selection, which makes it often fail to identify relevant features. LASSO, ElasticNet, Adaptive, and Relaxed showed unstable performance with large variance on the numbers of non-zero variables in the semi-real data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experimental results for feature selection performance (F1-scores and numbers of non-zero variables in parentheses) with the semi-simulated data based on four TCGA cancer datasets.

The average of the experiments are shown, where bold-face indicates the best performance.

https://doi.org/10.1371/journal.pone.0278570.t002

Tuning the hyper-parameters

The optimization of the hyper-parameters, q₁, q₂, and L, is often critical to the performance of feature selection in Hi-LASSO. We investigated how the hyper-parameters affect the performance of Hi-LASSO using the two simulation data (S3 File). We compared F1-scores by varying values of the hyper-parameters, where we set the identical values for q₁ and q₂ (i.e., q = q₁ = q₂) for the sake of simplicity. We empirically found that the optimal values of q were around of the sample size. Generally, the larger L improved the performance in the experiments. However, L > 50 does not improve the performance significantly in the experiments. Empirically, the optimal value of L was 30, which approximates normal distribution by the central limit theorem, when the distribution is unknown.

Robustness for feature selection

Finally, we evaluated the Hi-LASSO’s robustness on the feature selection by calculating pairwise Kuncheva Index (KI) on the semi-real simulation datasets [12]. Specifically, Hi-LASSO, Random, and Recursive LASSO are based on bootstrapping, which may produce different results on every execution. KI calculates how much two sets are overlapped, where one indicates two sets are identical, and zero shows no overlap between the two sets. Hi-LASSO showed KI of 0.751 on average on the four cancer semi-real data, whereas Random and Recursive showed 0.650 and 0.733, respectively (Fig 1D). Note that Hi-LASSO produced the highest F1-scores on the semi-real simulation data. The best F1-scores and robustness with the highest KIs demonstrated reliable feature selection performance of Hi-LASSO.