LASSO.

The least absolute shrinkage and selection operator (LASSO) [8] is a well-known regularization method for estimating parameters in a high dimensional regression model. The method imposes a L1 penalty on the parameter, resulting in the “shrinking” of small parameter estimates to zero, and some shrinkage to non-zero estimates. When applying this L1 regularization to the partial likelihood, the regression parameter estimate is

where denotes the set of observations at risk of failure or experiencing failure at time y (the risk set), and denotes the L1 (LASSO) penalization term. In the R package glmnet [9,16] that we use to implement the method, the optimal is estimated by 10-fold cross-validation to maximize the concordance of observations. When performing the LASSO, it is advised that all covariates are approximately on the same scale with similar means and variances.

ALASSO.

The adaptive LASSO (ALASSO) [10] is more general than the LASSO penalization method, where regularization is allowed to vary over the features. This adaptive regularization helps in selecting important features in the presence of correlation. The ALASSO log-hazard ratio vector estimate is

where is a vector of nonnegative p-dimensional known weights.

The goal of the inclusion of these weights is to increase the shrinkage of the parameter estimates corresponding to noninformative features and minimize the shrinkage of parameter estimates for significant features. Among several choices we set the weight , where is the j-th estimated log-hazards ratio when the Cox proportional hazards model is fit to the data including all features with the ridge penalty (L2). A smaller value of leads to higher penalty (or more shrinkage) for the j-th regression parameter. Once again, an optimal value for is obtained via minimizing a ten-fold cross-validation. We use the R package Coxnet [17] to obtain the ALASSO estimator.

Elastic net (ENET).

ENET [11] is a generalization of the LASSO and Ridge regularized methods, where a mixture of L1 and L2 penalties are applied to the regression parameters. This encourages both complete shrinkage of noninformative features (L1 penalization) and more stable incomplete shrinkage of correlated features (L2 penalization). The ENET estimate is

where

The parameter determines the mixture of L1 and L2 penalties, and may be either estimated by cross-validation or fixed a priori. We fix to 0.5. However, we estimate through ten-fold cross-validation. The R package glmnet [9,16] is used to perform the ENET.

CoxBoost (CB).

CB [12] is a method of estimating the regression parameters of the proportional hazards model (1) using a regularization and gradient boosting method. Although a machine learning technique is used in the estimation, CB is not a model-free approach. Suppose that and are the parameter vector and its r^th component at the t^th iteration. Define , where

represent the gradient and double derivatives of the partial log-likelihood function with respect to the rth component of -vector with

and is the penalty term to avoid overfitting. Define . Determine

This index r^* maximizes the squared scaled gradient, which is somewhat equivalent to maximizing the log-partial likelihood [18]. Then set , for , where M denotes the number of boosting steps. In the accompanying R package CoxBoost [19], the default choice of is nine times the number of failures in the data, and the default for M is 100. We use these default hyperparameter values for all studies.

Random survival forest (RSF).

The random forest, proposed by Breiman [20], seeks to capture the strengths of the nonparametric decision tree while reducing the overfitting that commonly occurs when fitting a single tree model. To accomplish this, a technique known as bootstrap aggregation (“bagging”) is employed, where many bootstrapped samples of the original data are used to fit weak learner decision trees which are then aggregated (normally through averaging) to form a single strong learner. Ishwaran et al. [13] extended this idea to censored survival data and provided a corresponding R package [21]. The main steps to fit a RSF are as follows.

1. Draw B (default 500) bootstrap samples from the original data, each excluding approximately 37% of the data, called “out-of-bag" (OOB) data.
2. Grow a survival tree for each bootstrap sample. For each node, randomly select 403 candidate variables. The node is split using the candidate variable and location that maximizes mean survival difference between child nodes.
3. Grow the tree to full size under the constraint that every terminal node should have at least eight failures.
4. Calculate the survival function for each terminal node (which is briefly described below).

At every terminal node h of all trees b in the forest, a Kaplan-Meier survival function is approximated using all (even OOB) observations that lie within h. Let denote the Kaplan-Meier estimator at node h of tree b. To approximate the survival function for a new observation with feature vector X₀, it is passed down all B trees until a terminal node h(X₀) is reached. The arithmetic mean is the aggregated survival function estimate for this observation. The median of this estimate () is used when predicting the survival time of this observation.

For a feature, minimum depth is the distance (or the number of steps) from the root node when the feature first appears in any given tree. Note that an important feature generally branches out the data closer to the root node than at further distanced nodes, so that important features are expected to be split upon early in a majority of trees in the forest. Once the RSF is fit, each feature’s average minimum depth is calculated over all trees. Features whose average depth is less than the mean of all average depths are selected as important features [22,23].

Tuning parameters for the number of candidate variables at each node split and the minimum node size were chosen by a short simulation study, where optimal hyperparameters were estimated by minimizing out-of-sample error on 200 synthetic datasets. The ceiling of mean optimal hyperparameters (403 candidate variables, minimum node size of 8) are used in all studies. R code for this procedure is in TuneRSF.R of our repository. This a priori hyperparameter tuning was done to preserve computational resources in our large-scale simulation studies where RSF is already intensive, and may have resulted in some performance loss. We have performed an additional study that quantifies this loss in S1 Appendix, titled “Impact of RSF tuning parameters on performance.”

We chose to evaluate out-of-sample predictive metrics using observations that were unseen by the RSF during the training process (held-out data). Alternatively, out-of-sample predictive ability can be quantified by using OOB data, which acts as a proxy for out-of-sample data. For each observation, only the trees that did not include it during training (its OOB trees) are used to generate predictions. We have performed a simulation study comparing these procedures in S1 Appendix, titled “Comparing held-out and out-of-bag predictive metrics in RSF.”

Screened random survival forest (sRSF).

We first ran univariate Cox regression of the response on each individual covariate. Any covariate with a p-value greater than 0.1 was excluded. The remaining covariates were then used as inputs to the RSF model, which was fitted as described previously. For evaluating feature selection capability of the method, features were selected using the minimum depth criteria after fitting the RSF.

Benjamini-Hochberg procedure (BH).

The Benjamini-Hochberg (BH) [14] and q-value (QV) [24] procedures are two variable screening methods that have been proposed to control the rate of false discovery during repeated hypothesis testing on high-dimensional data. Both of these procedures are applied on p p-values arising from p hypothesis tests, one for each predictor through univariate Cox regression, and the methods return FDR-adjusted measures, which are used for feature selection. Without these adjustments, using standard p-values and independent -level significance tests results in approximately type-1 errors (false discoveries), where s denotes the number of true significant features. The details for these methods are outlined below.

Suppose that p null hypotheses correspond to p-values such that . Let

for a given desired FDR r. BH advises rejecting . In our context, , where is the log-hazards parameter corresponding to the predictor with associated p-value P_i for the Cox proportional hazards model fit to the i-th predictor. Rejecting H_i is analogous to considering the i-th predictor to be significant. This procedure ensures that the expected FDR is smaller or equal to r when the p-values are independent [14] or positively dependent [25]. For arbitrary dependence (e.g., features might be correlated in an arbitrary fashion), a more conservative threshold has been proposed [25], which is not considered here.

We then used the selected features in a multivariate Cox regression model and calculated predictive metrics based on the fitted model. The R package mutoss [26] was used to perform BH.

Q-value (QV).

In this procedure, for each test a q-value is calculated, which is the minimum FDR at which that test would be called significant [24]. For that calculation, the proportion of true null hypotheses is estimated from the data by the “smoother” method provided in the WGCNA [27] package implementation of QV. Assume p hypotheses are being tested. Corresponding to p-value P_i, the q-value is

where is an indicator function taking on one or zero if the statement • holds or not. If we decide to reject H_i, the expected FDR among all features with q-values less than or equal to q_i is . The q-value method assigns an FDR estimate to each individual test (analogous to a p-value), allowing identification of significant features across a range of FDR thresholds. In contrast, BH requires repeating the significance decision process for each chosen FDR cutoff. QV is generally more powerful than BH.

For prediction, the selected features are then used to fit in a multivariate Cox regression model to the time-to-event data.

Correlation-adjusted regression survival (CARS) scores.

The CARS filter is an extension of the correlation-adjusted regression scores filter from Zuber et al. [28] adapted to right-censored data. Welchowski et al. [6] defined the vector of CARS scores as

where is a shrinkage estimator of the correlation matrix of X, and estimates the correlations of the columns of X with with inverse-probability-of-censoring weighting to counteract biasing by censored observations. The general inverse-probability-of-censoring weight of the i-th observation is defined as

where approximates the censoring process, considering C as a random variable independent of T_i and X_i, i.e., the censoring process is noninformative. The matrix R_shrink is used in place of the correlation matrix between the columns of X to simplify computation of the inverse square root for large p. is found using singular value decomposition of R_shrink. For further details, see Welchowski et al. [6]. Predictors with CARS scores further away from zero are considered more strongly associated with time-to-event than those with the scores closer to zero, and their signage determines the direction of the association. The corresponding R package [29] advises selecting features with the top 5% of as important predictors. Since the 5% could be very small or quite large, depending on the number of features, we employed two techniques on to identify important features.

First, order the absolute value of , and call the ordered values . Let L be the fitted straight line through the points and , and define D(i, L) as the shortest (perpendicular) distance between the line and the point . The elbow point is

Features having are considered to be important features. We refer to this process of elbow point estimation as maximal Euclidean distance (MED). The MED procedure tends to select more features. Therefore, we also propose an ad-hoc approach to determine this elbow point. For all k in , define the sets and . Fit two least-square linear regressions, one to points in P_k,1 and one to P_k,2, and collect the residuals. Let e_{1:k, i}, be the residuals from the first regression and e_{(k+1):p, i}, from the second regression. We define the elbow point as

Using lower powers than 6 gives a larger estimate of E, which we opt against. We refer to this procedure for estimating the elbow as the minimal sextic residuals (MSR).

Fig 1 shows an example of the elbow point estimates from these two methods on absolute CARS scores calculated on a real bladder cancer dataset.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Elbow point estimation using the MED and MSR methods on the plot of decreasing absolute CARS scores for 3,000 features from the bladder cancer cohort (TCGA-BLCA).

https://doi.org/10.1371/journal.pone.0351429.g001

We considered both methods for elbow point estimation in selecting predictors after computing CARS scores. The selected features are used to fit a multivariate proportional hazards model.

A brief summary table of the examined methods’ characteristics is provided in Table 1. Recall that embedded methods combine the steps of feature selection and model-building while filter methods perform these steps sequentially, and that parametric methods assume that the underlying distribution of survival times follows (1), so that the regression parameter vector is estimated by the method.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Methods at a glance.

https://doi.org/10.1371/journal.pone.0351429.t001

Synthetic data structure (setting-I).

The p-dimensional feature vector X was simulated from , where , where I_p and 1_p respectively denote the identity matrix of order p and the p-vector with all entries being one. We considered two different values of , 0 (the independent features case) and 0.5 (the correlated features case). For every X, we simulated the time-to-event T from the exponential distribution with scale . Then censoring time C was simulated from , where Q(r) denotes r^th sample quantile of the T vector. Then observed time was set to and censoring indicator . The procedure resulted in about 15% right-censored observations. The sample size for every simulated dataset was n = 300 and p was 1000.

We considered three different levels of sparsity (the percentage of nonzero components of the regression vector ), s = 2%, 5%, and 10% and set , , , for . We took three different values for , 0.5 for weaker signals, 1, and 2 for moderate to strong signals. The simulation design was somewhat similar to that of Zhu et al. [30].

For every unique combination of , s, , we generated 200 datasets through the process outlined above. In every dataset, 200 of the 300 total observations were analyzed by each of the methods described earlier, and the remaining 100 were used to evaluate the out-of-sample predictive ability of each resulting model. A data-flow diagram for this study is presented in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Data-flow diagram of simulation setting-I.

Light green cells are numeric scalars or vectors, orange cells represent tabular data, white cells are processes/algorithms, and purple cells are statistical or machine learning models. 18 unique cases of data characteristics are considered, each using (sparsity), (predictor correlation), and (strength of true signals). Methods that do not require hyperparameter tuning do not undergo the cross-validation step.

https://doi.org/10.1371/journal.pone.0351429.g002

Synthetic data simulation imitating the bladder cancer data (setting-II).

In this setting, we simulated data mimicking a real bladder cancer cohort from The Cancer Genome Atlas (TCGA-BLCA). The sample size was set to n = 423 (the real data sample size) and the number of features was 3,000, which was a pre-screened subset of a larger number of mRNAs standardized to mean 0 variance 1. We set the true regression parameter to 10 times the log-hazard estimates of the real data using the CoxBoost method, which had 53 nonzero coefficients out of the 3,000 coefficients. All nonzero coefficients were between 0.15 and 0.9 in absolute value. The time-to-event was simulated from the Weibull distribution with shape 2 and scale , i.e., , so that the median survival time was approximately 250. Censoring time C was randomly simulated from , where Q(r) is the r-th sample quantile of T. Each method was applied for analyzing the data , , where and . We generated 200 datasets with the above process for this setting of simulations. In each dataset, 43 (approx. 10%) observations were chosen at random to be for evaluating out-of-sample predictive performance. The remaining observations were used for training of all methods. A data-flow diagram for this setting of simulations is shown in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Data-flow diagram of simulation setting-II.

Light green cells are numeric scalars or vectors, orange cells represent tabular data, white cells are processes/algorithms, and purple cells are statistical or machine learning models. The TCGA-BLCA cohort cleaning and the preliminary feature selection step are discussed in the real data analysis background. Methods that do not require hyperparameter tuning do not undergo the cross-validation step.

https://doi.org/10.1371/journal.pone.0351429.g003

Performance evaluation metrics.

Feature selection metrics

We employed FDR = FP/(TP + FP) and to evaluate feature selection performance, where TP, FP, and FN denote the number of true positives, false positives and false negatives, respectively. Both FDR and F1-score lie between zero and one. FDR is an increasingly important metric to consider in the field of bioinformatics [4], as falsely classifying irrelevant variables as significant can result in a major waste of resources in pursuit of false leads in biological research. We also seek to succinctly evaluate the precision = TP/(TP + FP) and recall = TP/(TP + FN) of the methods, so we employ F1-score, which is their harmonic mean. Optimal methods will have a low FDR and a high F1-score.

Predictive capability metrics.

Concordance index (CI)

We evaluated the predictive ability of the methods with out-of-sample CI [31]. Two observations are considered comparable if one has experienced an event before the other either experiences an event or is censored. A comparable pair of observations is concordant if the observation that experiences an event first is predicted to have a shorter survival time (or equivalently be at greater risk), and otherwise the pair is discordant. CI is defined as the proportion of concordant pairs among the comparable pairs.

Here is the set of comparable observation pairs (i,j) such that Y_i < Y_j and , and is the i-th observation’s (with covariate X_i) inverse risk estimate where is the method’s log-hazard ratio vector estimate when performed on the training data. Here denotes set cardinality. For the nonparametric RSF and sRSF, is the predicted survival time from the ensemble of decision trees. CI can take on values between 0 and 1 inclusive, where higher values show better predictive ability, and a score of 0.5 shows no improvement over randomly guessing each observation’s risk. We calculate CI using only out-of-sample observations.

Brier score

The Brier score [32] is analogous to the mean squared error between survival past a time point t₀, and the predicted probability of survival past t₀, . To account for event censoring, time-specific inverse-probability-of-censoring weighting of observations is used. The Brier score at time t₀ is

where denotes the inverse-probability-of-censoring weight of the i-th observation at time t₀, and n_o denotes the size of the out-of-sample data (test data). Again, approximates the censoring process. In the first setting of simulations, we report the Brier score at t₀ = 0.5, and t₀ = 250 in the second setting, both of which are near the population medians of their respective studies. This metric is calculated on out-of-sample observations in all studies.

RMSE

We define the root mean squared error (RMSE) for the logarithm of the time-to-event as

For parametric methods, is the estimated median survival time under the Cox proportional hazards model fitted using only features with nonzero estimated regression coefficients. For RSF and sRSF, denotes the predicted survival time using the ensemble of trees. Here denotes the size of the out-of-sample (testing) data. A lower value of RMSE indicates lesser difference between predicted and true event times, i.e., better predictive performance. This metric is calculated on out-of-sample observations. Notice that the true event times T_i are needed to compute the metric, making it exclusive to simulation studies.

In brief, CI quantifies how well observation risks are ranked, Brier score measures the gap between observed survival beyond a time point and the model’s predicted survival probability, and RMSE is the deviation between true event times and predicted event times.

Background.

The Cancer Genome Atlas (TCGA) is a publicly available cancer genetics program containing genomic, epigenomic, transcriptomic, and proteomic data from cancer patients. The genomic data in particular have been used in benchmark studies (e.g., Bommert et al. [5]) and are generally accepted for demonstrating method efficacy on right-censored data. We used patients’ mRNA transcript abundance data from the bladder cancer (BLCA) cohort to illustrate how the examined methods can be performed on real data. We obtained the normalized (debiasing due to differences in sequencing depth or gene length) and log2 transformed (reducing skewness) mRNA data (HiSeqV2) prepared by the RSEM method [33] from the Xena Browser [34].

We obtained BLCA patients’ mRNA and time-to-event (days from diagnosis to patient failure) with censoring indicator and sex. We removed observations that experienced an event on the same day as diagnosis or contained missing values. Further, we removed all constant features. After these exclusions, we had 423 observations (patients), 308 of which were male and the remaining 115 female, and 20,240 mRNA features. Estimated Kaplan-Meier survival functions for male and female BLCA patients are shown in Fig 4, with corresponding log-rank test p-value for testing for differences between survival distributions. We also provide a histogram of the million absolute pairwise correlations (APC) of the genes (Fig 5). To reduce to the high dimensionality and multicolinearity of the features, we define a preliminary feature selection (PFS) step.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Estimated Kaplan-Meier estimator of the survival functions for BLCA cohort females and males.

Shaded regions show 95% confidence intervals and vertical dashed lines are the estimated median survival times. The log-rank test’s p-value is inside the figure, which indicates no significant difference in survival distributions between the groups.

https://doi.org/10.1371/journal.pone.0351429.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Absolute pairwise correlations (APC) before and after our preliminary feature selection (PFS).

(Top) Histogram of APC between 20,240 mRNA features prior to PFS. (Bottom) Histogram of APC between the 3,000 features chosen by PFS. The number of observations within each bin are shown in the counts above each bar.

https://doi.org/10.1371/journal.pone.0351429.g005

Preliminary feature selection (PFS).

The 20,240 mRNA features were standardized to have mean 0 variance 1 among all observations, and then CARS scores were calculated. The mRNAs corresponding to the 3,000 greatest values of were kept and used by all methods in the real data analysis and setting-II of our simulations. CARS was chosen for this process as it enjoys good computation scaling with respect to p and selects features that have strong marginal correlation to the outcome and lesser correlation with other features [6]. We performed PFS once on all observations instead of per-fold so that methods select features each fold from a consistent feature set. Of the features kept by PFS, we examined existing literature to identify a feature subset whose elements likely contribute to differences in cancer progression and patient survival times. We found 30 features in our search, which are presented in Table 2 with corresponding literature citations.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. List of 30 mRNA features likely associated with differences in cancer patient survival time found within existing literature.

https://doi.org/10.1371/journal.pone.0351429.t002

Additional studies concerning PFS.

Three additional studies were performed that examine the quality of PFS and its downstream effects in the real data analysis, all described in S1 Appendix.

To examine the downstream effects of PFS, we evaluated the embedded methods’ differences in predictive accuracy and runtime in the real data analysis when performed using either the size 20,240 feature set (no PFS) or the reduced set of 3,000 features (with PFS). This study is titled “Effect of PFS on embedded methods’ performances in the real data analysis.”

We also examined the overall quality of the PFS step through two studies. Firstly, we observed the relationship between the signal-to-noise ratios of truly significant features (for which is a proxy) and their retention rates when PFS is performed through a simulation study titled “Retention of true signals by PFS under varying signal-to-noise ratios.” Secondly, we performed a separate study that examines PFS with the real survival times to determine how sensitive the procedure is and if/how the downstream modeling may have been affected by its application on the full data instead of per-fold, titled “Sensitivity analysis of PFS.”

Analyses techniques.

Let denote the set of 3,000 mRNA features kept by PFS, and be the subset of 30 true signals identified within existing literature. We randomly partitioned the 423 observations into 10 approximately equally sized and mutually exclusive folds. Every method was performed 10 times in this analysis, each time using the observations from 9 of the 10 folds for training. All features were shifted and scaled to have mean 0 variance 1 among the observations in the training folds. This process of repeatedly performing a method on a subset of all observations and evaluating predictive performance on the remaining observations (the outer loop) and performing hyperparameter cross-validation within the training set (the inner loop) during each iteration of the outer loop is known as nested cross-validation. We use this process to achieve replications in lieu of the ability to generate new datasets. The sets of selected features from the i-th model (denoted for ) were recorded, and out-of-sample CI and Brier score at 365 and 1,000 days post-diagnosis were calculated using the fold of observations left out of the training. For quantifying feature selection capability, we recorded and (i.e., the total number of selected features and the number of selected features among the features identified within existing literature respectively) for . These values are presented as “Selected mRNAs” and “True positives.” Since the uncensored event times of all observations are not known, we cannot compute RMSE. Further, we cannot compute FDR nor F1-score since knowledge of the exhaustive list of true signals is not present in this analysis of real data. We instead report feature selection stability with the Dice coefficient which is described later. A data-flow diagram for this individual study is shown in Fig 6.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Data-flow diagram of the real data analysis.

Light green cells are numeric scalars or vectors, orange cells represent tabular data, white cells are processes/algorithms, and purple cells are statistical or machine learning models. Nested 10-fold cross-validation of the full cohort was done so that methods were ran multiple times on the data and predictive performance was evaluated with out-of-sample observations. Methods that do not require hyperparameter tuning do not undergo the inner cross-validation step.

https://doi.org/10.1371/journal.pone.0351429.g006

Dice coefficient.

The Dice coefficient [65] measures how “similar” two sets are by comparing the sizes of the individual sets with the size of their intersection. We calculate the Dice coefficient between all pairs of S_i and S_j where for each method as

which varies between 0 when no features are in common between the sets and 1 when sets are identical. If one of the compared sets is the list of true signals then the resulting Dice coefficient is equivalent to F1-score. We report the median and inter-quartile range of the Dice coefficients for each method in the real data analysis results section.

Feature selection performances.

Boxplots of the F1-score and FDR for setting-I are shown in Fig 7. For each method, the boxplot displays the distribution of the metric across simulation replications, with the black horizontal bar indicating the median value. For example, when , s = 2%, and , the median FDR for LASSO is about 75%.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Boxplots of feature selection metrics and computation time in setting-I of the simulation studies.

(Top) Boxplots of FDR. (Middle) Boxplots of F1-score. (Bottom) Boxplots of computation time, measured in seconds. Plots are faceted by data characteristics: horizontally by correlation between predictors and sparsity of informative features (), and vertically by signal strength ().

https://doi.org/10.1371/journal.pone.0351429.g007

Under most cases, the BH and QV procedures exhibit the lowest median FDR, while RSF shows the highest. For any given sparsity level, the FDR for ALASSO decreases notably as the sparsity s increases, especially when the signal strength is greatest. In contrast, increasing the correlation from 0 to 0.5 does not result in a noticeable change in FDR values. The FDR values from the sRSF procedure are substantially lower than those from RSF, indicating that the screening step helps reduce false discoveries. Unsurprisingly, MSR maintains lower FDR than MED when performing CARS due to its more conservative threshold for feature selection. Of the examined embedded methods, CB consistently has the lowest FDR when s = 2%, and is about tied with sRSF when s = 10%.

Note that a lower FDR indicates a method is more effective at avoiding the selection of unimportant features.

For sparsity 2% and 5%, ALASSO and CB show the highest F1-scores, with LASSO also performing competitively in the case. Overall, F1-scores increase with signal strength, while they decrease as the correlation among features increases. At the 10% sparsity level, LASSO shows the highest F1-scores, followed by ALASSO and CB. As expected, the BH and QV methods show the lowest F1-scores due to selecting very few features. Importantly, the F1-scores for sRSF are noticeably higher than those of RSF in most cases, indicating that the inclusion of the additional screening step improves the FDR and F1-score of the RSF method.

Note that a higher F1-score indicates a better method for detecting all relevant features while avoiding unimportant ones.

Predictive performances.

Boxplots of CI, Brier score, and RMSE for setting-I are presented in Fig 8. A higher value of CI and lower values of Brier score and RMSE indicate better predictive capability.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Boxplots of predictive metrics in setting-I of the simulation studies.

(Left) Boxplots of CI. (Middle) Boxplots of Brier score. (Right) Boxplots of RMSE. Plots are faceted by data characteristics: horizontally by correlation between predictors and sparsity of informative features (), and vertically by signal strength ().

https://doi.org/10.1371/journal.pone.0351429.g008

In terms of CI, LASSO and ALASSO and ENET seem to perform the best across all scenarios. CB was generally around 4th in this metric. BH and QV maintain median CI of 0.5, suggesting these methods have no noticeable improvement over randomly guessing observation risk scores. Additionally, the MSR threshold outperforms MED when performing CARS. As expected, CI generally increases with and decreases with s and .

Recall that methods with higher CI have superior predictions of relative risk between observations (i.e., observations with greater predicted risks generally experienced events prior to ones with lesser predicted risks).

When evaluating Brier score, the embedded methods generally outperformed the filter methods under the cases with low s. However, both classes of methods performed similarly when s = 10%. The ALASSO and CB were consistently the best performers among the embedded methods. CARS with the MSR threshold was the best of the filter methods for the case of low s, and BH and QV performed better when s was large.

Recall that Brier score quantifies the difference between predicted probability of an event occurring after a given time and whether an event was observed or not.

For smaller values of s, ALASSO is generally the best in terms of RMSE. In terms of CI and RMSE, sRSF performed better than RSF. In most cases, the MSR threshold produces lower RMSE than MED when performing CARS.

Note that methods with lesser RMSE more accurately estimate observation event times.

Computational time.

Boxplots of methods’ computational time on our synthetic data (setting-I) are shown in Fig 7. Little variability in computation time was present as data characteristics changed, and relative rankings remained almost constant. RSF without screening was consistently the most computationally taxing, followed by the LASSO and sRSF. Interestingly, ALASSO is among the top performers. CARS also performed consistently well when using the MED threshold, but less so when using MSR.

Cumulative method rankings.

To summarize each method’s overall relative performance in our first setting of simulations, we ranked the median metric values of all methods within all combinations of data characteristics, totaled the ranks over the combinations of characteristics, and finally ranked the totals. The resulting aggregate method rankings for each metric are presented in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Aggregate relative method rankings in simulation settings I and II.

https://doi.org/10.1371/journal.pone.0351429.t003

BH and QV consistently lead for FDR control, while ALASSO and LASSO achieved the highest F1-scores. For CI, LASSO and ENET generally outperformed, though BH had the best overall Brier score. LASSO and ALASSO also yielded the lowest RMSE. Regarding runtime, CARS with the MED threshold proved fastest.

A table containing the medians and inter-quartile ranges of metric values across each method’s 200 replications under each combination of data characteristics () is provided in S2 Table. Each column of this table corresponds to a unique combination of data characteristics, and every row is an individual method and metric. The best median metric values for each case of data characteristics are boldfaced.

Feature selection performances.

A table of summary statistics (median and inter-quartile range) of all metrics for setting-II simulation studies are shown in Table 4. Well-performing methods have lower FDR, Brier score, RMSE, and computation time, and higher F1-score and CI.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Summary statistics of setting-II of the simulation studies.

https://doi.org/10.1371/journal.pone.0351429.t004

CoxBoost performed the best in controlling FDR, followed by ALASSO and LASSO. In contrast, BH and QV surprisingly had the worst performance under this metric. All embedded methods with the exception of RSF and sRSF had similarly good F1-scores, while the filter methods generally underperformed. The regularized methods LASSO, ALASSO, and ENET had very good CI and Brier score. CB also performed well in Brier score. ALASSO performed best in RMSE by a decent margin, and CARS with MED thresholding had the shortest computation times.

For quick reference, the bottom panel of Table 3 shows relative method rankings in this second setting of simulations.

BLCA cohort analysis.

The Kaplan-Meier estimators for BLCA males and females in Fig 4 show no significant differences in the survival functions between the two sexes (p = 0.53) at the 5% level of significance. Survival probability dropped rapidly during the first 500 days following initial diagnosis, and tapered off after about 3,000 days, which suggests that many diagnoses may have occurred late in disease progression. Of 423 patients, 2, 133, 146, and 141 were at stages I through IV respectively (one patient’s stage was not recorded) at the time of initial surgical pathology or biopsy, and 208 of those procedures were within a year of the initial diagnoses. The median survival time was approximately 1,000 days for both genders (i.e., about half of the cohort experienced failure prior to this point). The presence of more males than females within the cohort leads to the 95% confidence intervals for the survival function being tighter for the males than for the females.

mRNA feature pairwise correlations.

The histograms of APC between the 20,240 mRNA features prior to PFS and the 3,000 features following PFS are shown in Fig 5. Although a majority of feature pairs are not highly correlated, there still exist over 1.2 million feature pairs with APC exceeding 0.5 prior to PFS, as shown in the top panel. The distribution is somewhat more left-leaning following PFS, i.e., some feature pairs that had high APC were removed, as well as features with little to no association with the outcome.

mRNA feature selection.

Summary statistics (median and inter-quartile range) of all feature selection metrics are provided in Table 5. Methods with higher ratios of true positives to selected mRNAs, higher Dice coefficient, and lesser computation times are better performing.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Feature selection and computation time metrics for the real data (TCGA-BLCA) analysis.

https://doi.org/10.1371/journal.pone.0351429.t005

Both RSF and sRSF selected the largest number of features, and RSF had the largest variation in this quantity over the ten outer folds. CB and CARS with MSR thresholding had consistently fewer selected features, and CB generally had at least one true positive. CARS with the MSR threshold also had the most consistent selected features (high Dice coefficient), followed by BH and QV. The LASSO had the worst Dice coefficient. CARS with the MED threshold had the least computation time.

The selection frequencies of the 30 mRNA features listed in Table 2 over the ten outer folds of the real data analysis are provided in S3 Table. The features DLEU1 and TOP3B were the most commonly selected by a majority of the methods.

Prognostic modeling.

Predictive performances based on the 10 folds of analysis on the real data are shown in Table 6. Methods with greater CI and lesser Brier score at both time points are better performing.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Predictive metrics for the real data (TCGA-BLCA) analysis.

https://doi.org/10.1371/journal.pone.0351429.t006

Although the LASSO and ALASSO were the best performing in CI, sRSF was competitive in performance. CARS with MSR thresholding was the best in Brier score at both time points, and tied with both RSF methods at the 365 day horizon and slightly outperformed them at 1,000 days. Interestingly, although the parametric embedded methods (LASSO, ALASSO, ENET, CB) performed competitively in Brier score at 365 days, they greatly underperformed at 1,000 days.

Model calibration checks.

We performed an additional heuristic check of the prediction performance by each method in the ten outer folds of the real data analysis. For each method and fold, observations were partitioned into quintiles based on predicted risk. We assessed model calibration by plotting predicted versus observed survival probabilities at the 1, 3, and 5-year clinical horizons. These plots are provided in S4 Fig, faceted by method and clinical horizon. The curves from the ten folds are colored and overlayed in each individual subplot. Well-calibrated models predict each group’s survival probability to be near their respective observed survival probabilities, shown visually by having survival curves near the solid black y = x line. Least squares regression lines are shown per method and horizon as dashed black lines, with their intercepts and slopes provided in each subplot.

Generally, the observed survival probabilities at the 1 year clinical horizon exceed 0.5, but they are more uniformly distributed at the 3 and 5-year horizons. Further examining the 1 year plots (the first and fourth columns), some methods have curves that are nearly vertical or even slightly downwards-sloping, suggesting that survival predictions may be volatile at this horizon, as there is greater deviation in predicted survival probabilities than in the observed ones. In the 3 and 5-year plots, many of the curves are biased above the y = x line, suggesting that the predicted survival probabilities are optimistic when the observed survival probability is about 0.5. This is largely the case with the parametric methods. The RSF and sRSF methods avoid this optimism, and have very polarized observed survival probabilities, indicative that the nonparametric forests have highly segmented the high and low risk groups. CARS with the MSR threshold has downward-sloping curves, which suggests that informative features may be missing from the final models. Overall, with few exceptions, most of the methods’ predictions are stable between the ten folds, shown by the colored curves being near each other in each subplot.

Note that the calibration we observe is largely dependent upon the underlying data characteristics. Methods’ performances change depending upon the data sparsity, strength of signals, predictor correlations, and censoring, many of which are unknown in the real data case. Further, it is important to examine calibration at multiple horizons, otherwise observed survival probabilities may be only high (as is the case in our 1 year plots) or only low.

Variance filter.

The variance filter, which excelled in the benchmark study from Bommert et al. [5], is not applicable to our studies. In short, the variance filter gives each feature a score equal to its sample variance and selects the features with scores exceeding a specified threshold, which trivially makes it ineffective in our studies where features are standardized to uniform variance.

RSF with variable importance (VIMP).

A third variation of RSF from Ishwaran et al. [13] utilizes a different variable selection criterion called variable importance (VIMP). VIMP is the increase in prediction error if a particular variable had not been present within the data. Trivially, predictors with high VIMP are considered important. We examined the variable selection capability of RSF by choosing predictors with VIMP exceeding a threshold, but it performed worse than RSF by minimum depth with respect to both examined variable selection metrics.

Stepwise variable selection.

A family of wrapper methods was also examined that utilized a screening procedure followed by stepwise variable selection. First, data dimensionality was reduced by selecting variables with the lowest p-values from univariate Cox regression, or the variables with the lowest mean minimum depths after fitting a RSF, where , i.e., the number of uncensored observations within the data. Then a proportional hazards model was fit using iterative forward stepwise variable selection with either AIC or BIC as a model selection criterion. These methods were extremely computationally intensive and performed poorly in all metrics.

Bayesian additive regression trees (BART).

Much like the RSF, BART is a nonparametric tree ensemble method for estimating relationships between feature vectors and their associated outcomes. The model was introduced by Chipman et al. [66] in 2010 and adapted to right-censored survival data by Sparapani et al. [67] in 2016 with the corresponding R package survbart.

The general paradigm has inspiration from boosting methods where several weak learners (decision trees in this case) are fit sequentially, each successive one fitting to the residuals of the former so that the sum of these trees better captures the variations in the data. To prevent overfitting, regularized prior distributions of the model parameters (tree complexities and terminal node value estimates) are assumed, i.e., less complex decision trees are preferred. The corresponding posterior distributions are estimated using a Markov chain Monte Carlo (MCMC) algorithm, which repeatedly samples from the posterior [66].

We briefly examined the performance of BART in our predictive metrics (CI and RMSE) using the survbart package implementation. In the RMSE calculation, the estimated survival time was the median of the estimated survival function. In the first setting of our simulation study, BART was applied on 18 total synthetic datasets, one for each possible unique combination of our examined data characteristics.

The predictive performance of BART was promising and regularly outperformed the RSF/sRSF methods, but was extremely computationally taxing, and was therefore removed from the remainder of the study.

Park et al. [68] benchmarked the RSF and BART methods and some of their variations in terms of their predictive performances on synthetic linear regression data featuring missing values, high-dimensionality, and high predictor correlations. In their studies, they found that the DART, a variant of BART that utilizes Dirichlet prior distributions which encourage sparse models, outperformed the RSF in most of their predictive metrics, but suffered extreme computational costs.

The BART paradigm appears promising once improvements to the efficiency of the MCMC algorithm are made.