Time-to-event analysis with right-censored data

Consider a test sample of n_test independent and identically distributed individuals, and let T_i and C_i denote the true event and censoring times, respectively, of individual i ∈ {1, …, n_test}. Here we consider right-censored time-to-event data, meaning that an individual is considered to be censored with respect to the event of interest if their censoring time C_i is less than their true survival time T_i. Consequently, the observed event time is defined by . In this paper, we only consider the case of a single target event without competing events. Let δ_i denote the status indicator variable, whereby δ_i = 1 if the individual is observed to experience the event, otherwise δ_i = 0. We also define a p-dimensional vector of time-independent predictors Z_i = (Z_i1, …, Z_ip)^⊤, which are assumed to influence the event-free survival time of individual i. The marginal probability of event-free survival is defined as S(t) = P(T_i > t), whereas the individual-specific probability of event-free survival is defined conditionally on the predictors, i.e., S_i(t) = P(T_i > t|Z_i = z_i), where t corresponds to some point in time. Here we assume that the event and censoring times are conditionally independent, i.e., that the process responsible for generating the target event and that responsible for the occurrence of right censoring are independent of each other given the predictors [20]. Both processes may therefore be modeled as separate probability distributions, where we additionally assume that the two models do not share any common parameters. In this framework, the censoring survival function quantifies the time-dependent probability of “surviving” a censoring event: analogously to the definition of S(t), the marginal probability of censoring-free survival is defined as G(t) = P(C_i > t), whereas the individual-specific censoring survival function is defined as G_i(t) = P(C_i > t|Z_i = z_i). In the following, the index i may be omitted in some instances for the sake of simplicity. While estimating the survival function S(t) is naturally of primary concern, especially in predictive modeling, estimation of the censoring distribution is typically of little practical interest. However, as mentioned above, certain methods such as IPC weighting require the estimation of the censoring survival function G(t).

Inverse-probability-of-censoring weighting

To address the issue of right censoring, we may assign weights to individuals depending on their respective probabilities of “surviving” the censoring event. This weighting method is known as IPC weighting [18, 19]. The intuition behind IPCW is that censored individuals cannot directly contribute to the analysis since the time of the event is unobservable for these individuals. However, it would be unwise to completely ignore censored individuals, since the probability of losing individuals to follow-up increases over time. Subjects with large event times T_i are therefore more likely to be censored than those with smaller event times. Hence, simply excluding censored individuals would lead to a biased sample in which individuals with smaller event times are over-represented, and would thus lead to an underestimation of the survival probabilities. IPCW prevents individuals who are censored at or prior to a particular time point t from contributing to the analysis directly by assigning them a weight of zero, however, the remaining subjects are re-weighted accordingly to account for the excluded censored individuals. Concretely, IPCW requires the estimation of G(t) (or, if a set of predictors is considered, G_i(t)), as the weights assigned to each individual depend on the censoring process. More specifically, for each t the IPC weights ω_i, i = 1, …, n_test, assigned to each individual are defined as (1) where is the indicator function, and where refers to a time point that is infinitesimally smaller than . By definition, the IPC weights do not have fixed values but rather vary over time and are thus defined relative to a particular time point t. Since the value of G(t) decreases with increasing t, individuals with larger observed event times are assigned larger weights. This type of weighting scheme has been adapted for a wide range of methods, including ML techniques [35] as well as the estimation of performance metrics like the Brier score and the concordance statistic for time-to-event data ([36], see also the Discussion for a brief overview).

The Brier score

The Brier score is a scoring metric frequently used for evaluating the accuracy of predictions in classification, time-to-event analysis and other settings. In the case of survival data with a single target event, the Brier score is equivalent to the mean squared error (MSE), which is defined by the mean squared difference of the true and the estimated survival probabilities, S_i(t) and , evaluated at time t: (2)

While the estimated survival probabilities may be derived from the data, the true survival probabilities S_i(t) are generally unknown and must therefore be approximated through the use of step functions which are equal to one prior to the event times and zero afterwards. The empirical Brier score for uncensored data, which we will refer to as the uncensored Brier score in this paper, is therefore defined as (3)

Replacing the true survival function with an approximation inevitably introduces some amount of bias. However, it can be shown that this bias depends on S_i(t) but not [21, 37]. More specifically, the expectation of the uncensored Brier score is equal to (4) where the second term in (4) is the irreducible loss. The proof of (4), as well as a decomposition of the uncensored Brier score into calibration and refinement (i.e., discrimination) terms, is given in S1 File.

Obviously, the computation of the uncensored Brier score in (3) is only feasible in datasets where all of the event times are observed, i.e., where ∀i = 1, …, n_test. When dealing with right-censored data, we may use inverse-probability-of-censoring weighting to approximate the Brier score for uncensored data. The IPCW Brier score is defined as (5) where corresponds to the sum of the IPC weights in (1). Note that when estimating the IPC weights on the test set with a simple method such as the Kaplan-Meier estimator, is equal to n [21]. It can be shown that the expectation of the IPCW Brier score is equal to the expectation of the Brier score for uncensored data if the functions G_i(⋅), i = 1, …, n_test, are known. When the functions G_i(t) are unknown, which is most often the case, we may replace G_i(t) in Eqs (1) and (5) above with an estimator . If the censoring model is correctly specified, i.e., if the estimator is weakly consistent for G_i(t), the expectation of (5) will converge to (4) for n_test → ∞ [20]. Note that, unlike in other settings with a binary outcome where the Brier score corresponds to a single value, the Brier score in time-to-event analysis consists of a series of scores, as it depends on the time t. These scores may be plotted over time to illustrate the change of prediction error with increasing t (“prediction error curve”, [22]).

Training and test data

As stated in the introduction, one of the persisting questions regarding the application of the IPCW Brier score concerns the estimation of the censoring distribution. Here, we aim to determine whether the training set, the test set or the combined dataset is best suited for this purpose. We generally assume that each pair of training and test sets is the result of randomly subsampling from a single “parent” dataset. Additionally, we assume that fitted models of the survival function are obtained on the training sample of size n_training. Denoting the estimated survival function by (or alternatively by , depending on the context), predictions are then computed on an independent test sample of size n_test. Models of the censoring survival function are obtained on either the training, test or combined sample, depending on the considered scenario, and predictions for the estimated censoring survival probabilities (or , depending on the context) are computed on the test set. Further details regarding the model fitting and estimation process are given in Section “Simulation study” and Section “Notations and Definitions”. In some instances, we also employ bootstrapped datasets of the training and test data, see also Section “Data preparation and modeling” of the SEER data example. In cases where prior tuning of hyperparameters is necessary, e.g., for the random forest and extreme gradient boosting models, we perform cross-validation and Bayesian optimization on the dataset used for model fitting.

Generating right-censored time-to-event data

We used the Weibull and the exponential distribution to generate our right-censored datasets. Let Y_i equal the natural logarithm of Weibull-distributed time, i.e., Y_i = log(T_i), and let and γ^⊤ = (γ₁, …, γ_p) represent the matrix of predictors and the vector of corresponding regression coefficients. For the purpose of our study, we represent the link between log time and the predictors by a linear relationship of the form (6) where and σ > 0 correspond to the intercept and scale parameter of the model, respectively. As shown by Klein & Moeschberger [38], W_i, which may be viewed as a residual term, follows a standard extreme value distribution with density f_W(w) = exp(w − exp(w)). In particular, it may be shown that when the natural logarithm of the event time log(T_i) follows an extreme value distribution, then T_i follows a Weibull distribution with shape parameter α = 1/σ and scale parameter λ = exp(μ + γ^⊤Z_i). In the special case of the exponential distribution, σ is set to 1. By scaling the negative regression coefficients −γ, we obtain a proportional hazards model with regression coefficients β = −γ/σ and a Weibull hazard function of the form (7) where αλt^α−1 corresponds to the baseline hazard function h₀(t). We use this kind of model to generate both the event time and the censoring distributions. In the case of the censoring distribution, the random variable T_i is simply replaced by C_i and the values of the intercept and scale parameters μ and σ are readjusted to control the rate of censoring and the coefficient of determination R², which we defined as var(β^⊤Z)/var(log(T)) (omitting the index i for the sake of simplicity).

Simulation parameters and settings

Our datasets included a set of p = 10 informative predictors. The values of the Weibull regression coefficients were fixed at γ = (−0.5, −0.4, −0.3, −0.2, −0.1, 0.1, 0.2, 0.3, 0.4, 0.5)^⊤. These were identical for the event and censoring distributions. The p predictors in each dataset were drawn from a multivariate normal distribution with zero mean and a compound symmetry covariance structure (var(Z_j) = 1, 1 ≤ j ≤ 10, cov(Z_j, Z_k) = 0.5, j ≠ k, 1 ≤ j, k ≤ 10). In some of our scenarios we added a set of q = 50 non-informative predictors which neither contributed to the generation of the event times nor of the censoring times (for an overview of the simulation scenarios see Section “Model fitting and prediction” of the simulation study). These were designed to emulate “noisy” data. For the distribution of the event times, we set the value of the scale parameter σ_T to approximately 0.58, which resulted in a value for R² of approximately 0.5 across all of our experiments, while the value of the intercept parameter μ_T was set to 0. We used two different censoring models, either a marginal model or a “full” Weibull model. In the marginal model, the censoring times C_i were independent of the predictors. In this case, we set the value of the scale parameter σ to 1, resulting in an exponential distribution. In the “full” model on the other hand, the values of the predictors influenced the censoring times C_i, leading to hazard functions that varied from individual to individual. For these models, we set the value of σ_C to approximately 0.58, resulting in R² ≈ 0.5 for the censoring distribution. We set the level of censoring to approximately either 20%, 50% or 80%, depending on the scenario. This was done by offsetting the value of the location parameter μ_C for the full censoring models and by adjusting the rate parameter λ_C for the marginal (predictor-free) censoring models. The location parameter was set to either 0.800, −0.803, or −0.002 and the rate parameter to either 0.219, 0.864, or 3.013. Since the level of censoring is also marginally affected by the scale parameter σ, the values of these three parameters were jointly determined in advance using large external datasets of size 10⁶.

Model fitting and prediction

For each scenario we generated m = 1000 time-to-event datasets with n observations each. Each dataset of size n was randomly split into a training and a test dataset. In order to determine whether the relative size of the training set (n_training) and of the test set (n_test) had any effect, we varied the size of one of these datasets, with values chosen among {25, 50, 100, 250, 500}, while keeping the size of the other fixed at 500. Thus, the total size of each combined dataset (training and test) varied between n = 525 and n = 1000. Subsequently, a time-to-event model was fit to the training data. We used several popular and well-established methods, including regression-based models such as the Cox proportional hazards model [23] and regularized Cox regression (here, the Lasso, [24–26]), as well as ML methods such as random survival forests [27] and extreme gradient boosting (XGBoost, [28]). With the exception of the Cox proportional hazards model, which does not require prior tuning of hyperparameters, all models were tuned using cross-validation and/or Bayesian optimization, see [39, 40]. Further details regarding the tuning algorithm, priors and the chosen hyperparameters are provided in S2 File. The resulting model was then used to predict the survival function on the test set. The censoring model was fit to either the training set, the test set, or the combined dataset. This was done for both the marginal model, using a Kaplan-Meier estimator, as well as for the full model using either a Cox proportional hazards model, Lasso, random forest or XGBoost. Each of the three resulting models, obtained on either the training, test or combined dataset, was then used to provide a prediction of on the test set. As a result, we obtained an estimate of the survival function , as well as three distinct estimates of the censoring survival function (, and ), corresponding to the censoring survival function fitted to the training, test or to the combined dataset, respectively. In total, we considered five different scenarios in our simulations:

1. (i). a baseline scenario with an exponentially distributed and predictor-free censoring process, where the event time model was fitted using a Cox regression model and the censoring model using a Kaplan-Meier estimator,
2. (ii). a Weibull-Cox scenario with Weibull-distributed and predictor-dependent censoring processes, where both the event time and the censoring model were fitted using Cox regression models,
3. (iii). a misspecification scenario with Weibull-distributed and predictor-dependent censoring processes, where the event time model was fitted using a Cox regression model but the censoring model was “erroneously” fitted using the (marginal) Kaplan-Meier estimator,
4. (iv). a “low-noise” scenario with Weibull-distributed and predictor-dependent censoring processes, where both the event time and the censoring model were fitted using statistical modeling and ML methods (Cox regression, Lasso, random forest and XGBoost), and
5. (v). a “high-noise” scenario, which extended the low-noise scenario by 50 additional non-informative predictors.

Performance metrics

We assessed the effect of the different estimates of G_i(t) on prediction error by comparing the IPCW Brier score (Eq 5) obtained for each estimate of G_i(t) to the expectation of the uncensored Brier score (Eq 4). This was possible due to the fact that in simulation settings the true event times T_i, i = 1, …, n, and the true survival functions S_i(t) are known for all individuals. In order to standardize the comparison of differently parameterized distributions and to allow for an unbiased comparison of Brier score curves obtained on different datasets, we introduce a weighted and standardized cumulative measure of prediction error aimed at providing a single score for performance comparison. This measure represents the weighted squared difference of the IPCW Brier score obtained on right-censored data and the expectation of the uncensored Brier score obtained on a comparable set of uncensored data. In the following, we will refer to this measure as the weighted squared error (WSE), which is formally defined as (8)

In practice, each instance of the squared error (E(BS) − BS_IPCW)² is weighted by an estimate of the underlying marginal probability density of the event times. In this particular case, the term “instance” refers to the Brier score or squared error evaluated at any time t_j, j = 1, …, k. We chose a grid of 10⁴ equidistant time points for the calculation of the Brier scores and the WSE, therefore k = 10⁴ in all of our simulations. In order to ensure a fair comparison, we used a separate external dataset of size 10⁶, obtained using the same data-generating process as that used for generating the original dataset, in order to (numerically) determine the uncensored Brier score. By summing over all k instances and by taking the square root, we obtain a time-independent score defined by (9) which was used as the main performance measure in our simulations. In order to prevent large individual IPC weights from skewing the results, we capped the value of the weights ω_i(t) at 5 (see [21, 41]).

Results

The main results for our baseline scenario (i) are presented in Fig 1. As with the following scenarios, we chose to visualize the Monte-Carlo error using violin plots. It can be seen that the RWSE does not systematically differ depending on the dataset used for fitting the censoring distribution. This seems to be the case regardless of the sample size of each dataset (n), the relative size of the training and test sets (n_training and n_test, respectively), as well as the censoring rate. Unsurprisingly, the estimation accuracy of the IPCW Brier Score improves with an increase in the total sample size n and a lower censoring rate. The reduction of the RWSE is more pronounced following an increase in the size of the training set n_training with a constant n_test (Fig 1b), compared to an increase in the size of the test set n_test with a constant n_training (Fig 1c). This may be due to the fact that a larger training set leads to a better estimate of the survival function S(t), which is likely more important for improving the Brier score estimate than an improved estimate of the censoring survival function G(t). Further numerical results for this baseline scenario may be found in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Baseline scenario with an exponentially distributed and predictor-free censoring process.

The plots present values of the log-transformed RWSE score obtained for 1000 simulation runs as a function of the dataset used for fitting the censoring survival function (test, training or the combined dataset), for different values of n, which corresponds to the size of the combined dataset (with n_training = n_test, panel a), for different values of n_training and a constant value for n_test = 500 (panel b), for different values of n_test and a constant value for n_training = 500 (panel c), and for different values of the censoring rate (20%, 50% and 80%, where n_training = n_test = 500, panel d). The survival function was fitted using a Cox proportional hazards model, whereas the censoring survival function was fitted using a marginal Kaplan-Meier estimator.

https://doi.org/10.1371/journal.pone.0318349.g001

Fig 2 presents the results for the Weibull-Cox scenario (ii) with 10 informative predictors. Similarly to the results obtained for the baseline scenario, the estimation accuracy of the IPCW Brier score is again strongly dependent on the overall size of the dataset, with smaller sample sizes leading to a particularly large deviation from the expected prediction error, regardless of the dataset used for fitting the censoring distribution, see Fig 2. Interestingly, unlike in the baseline scenario, the data used to fit the censoring distribution does seem to affect overall estimation accuracy. Concretely, the estimators where the censoring survival function was fit to the test set slightly outperform those fit to the combined dataset, which themselves outperform those fit to the training set. Additionally, increased rates of censoring once again lead to an increase in estimation accuracy. At low to medium rates of censoring (≤50%), the estimators where the censoring distribution was fit to the test set slightly outperform those where the censoring distribution was fit only to the training set. Further numerical results for the Weibull-Cox scenario are presented in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Weibull-Cox scenario with a Weibull distributed and predictor-dependent censoring process.

The plots present the values of the log-transformed RWSE score obtained for 1000 simulation runs as a function of the dataset used for fitting the censoring survival function (test, training or the combined dataset), for different values of n, which corresponds to the size of the combined dataset (with n_training = n_test, panel a), for different values of n_training and a constant value for n_test = 500 (panel b), for different values of n_test and a constant value for n_training = 500 (panel c), and for different values of the censoring rate (20%, 50% and 80%, where n_training = n_test = 500, panel d). Both the survival and the censoring survival function were fitted using a Cox proportional hazards model.

https://doi.org/10.1371/journal.pone.0318349.g002

Another important aspect that merits attention is the effect of model misspecification on prediction error. We consider a model to be misspecified if it is structurally incapable of modeling some or most of the complexity contained in the true data generating process. We examined this by comparing the results of the misspecification scenario (iii) with those of the Weibull-Cox scenario (ii). As expected, the correctly specified Cox model in the Weibull-Cox scenario outperforms the Kaplan-Meier estimator in the misspecification scenario, especially for larger datasets (n ≥100), see Fig 3, stressing the importance of incorporating all of the available covariate information when modeling the censoring process. Importantly, as in the case of a correctly specified model (scenario (ii)), the estimators fit to the test data also slightly outperform those fit to the training set in the misspecification scenario. Further numerical results for the misspecification scenario are presented in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Weibull-Cox and misspecification scenarios with a Weibull distributed and predictor-dependent censoring process.

The plot presents the values of the log-transformed RWSE score obtained for 1000 simulation runs as a function of the dataset used for fitting the censoring distribution (test, training or the combined dataset), and as a function of the type of model used to fit the censoring function (Cox regression [solid lines, correctly specified] or Kaplan-Meier estimator [dashed lines, misspecified]). The total size of the dataset is given by n = n_training + n_test, where n_training = n_test.

https://doi.org/10.1371/journal.pone.0318349.g003

In order to assess the effects of estimates of the censoring distribution in the context of ML and noisy data, we selected four popular methods widely used in the field, namely Cox regression, regularized Cox regression (Lasso), random forest, and extreme gradient boosting (XGBoost). Details on the tuning of the hyperparameters of the Lasso, random forest and XGBoost models are presented in S2 File. Each type of model was applied separately to fit both the survival and the censoring function, on either the training, test or the combined dataset, whereby n = 1000, n_training = 500, n_test = 500, and the censoring rate ≈50% for all models. In our low-noise scenario (iv), the underlying survival and censoring functions were both Weibull distributed and included a set of 10 informative predictors each. The results are presented in Fig 4. Similarly to the Weibull-Cox scenario (Fig 2), there is an advantage in using the test set in order to fit the censoring survival function, as opposed to using the training set or the combined dataset when dealing with data with a low signal-to-noise ratio. However, we only found this effect for the Cox proportional hazards and random forest models. Further numerical results for the low-noise scenario are presented in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Low-noise scenario.

The plots present the values of the RWSE score obtained for 1000 simulation runs as a function of the dataset used for fitting the censoring distribution (test, training or the combined dataset), and as a function of the type of model used to fit the survival and the censoring survival functions (Cox proportional hazards, panel a; Lasso, panel b; random forest, panel c; XGBoost, panel d). The total size of the dataset is given by n = n_training + n_test = 1000, where n_training = n_test = 500. The censoring rate was set to approximately 50%.

https://doi.org/10.1371/journal.pone.0318349.g004

We subsequently repeated the above analysis for the high-noise scenario (v). Again, as seen in Fig 5, there is an advantage in using the test set in order to fit the censoring survival function, as opposed to only using the training set when dealing with data with a low signal-to-noise ratio. This is particularly evident for Cox proporional hazards (panel a), random forest (panel c) and XGBoost models (panel d), but not for the Lasso (panel b). Further numerical results for the high-noise scenario are presented in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. High-noise scenario.

The plots present the values of the RWSE score obtained for 1000 simulation runs as a function of the dataset used for fitting the censoring distribution (test, training or the combined dataset), and as a function of the type of model used to fit the survival and the censoring survival function (Cox proportional hazards, panel a; Lasso, panel b; random forest, panel c; XGBoost, panel d) on a dataset with a signal-to-noise ratio among the predictors of 1:5 (10 informative and 50 non-informative predictors). The total size of the dataset is given by n = n_training + n_test = 1000, where n_training = n_test = 500. The censoring rate was set to approximately 50%.

https://doi.org/10.1371/journal.pone.0318349.g005

Data preparation and modeling

For our analysis, we selected female breast cancer patients aged between 18 and 75 years at the time of diagnosis who first entered the database between 1998 and 2010. We excluded patients with distant metastases and those without a clear histological confirmation of the diagnosis. Of the 143 variables included in the original dataset, we selected 20 to use in our analyses [42]. The continuous and ordinal predictor variables included in our dataset consisted of the age of the patient at diagnosis (years), the size of the tumor (mm), the number of positive and examined lymph nodes, and the number of primaries. Categorical features included cancer stage according to the TNM classification system (12 T-stage and 4 N-stage categories), tumor grade (I—IV), estrogen and progesterone receptor status (positive/negative), primary tumor site (9 categories), surgery of primary site (yes/no), type of radiation therapy and sequence of radiation and/or surgery (7 and 6 categories, respectively), laterality (left/right), ethnicity (white, black, American Indian/Alaska Native, Asian or Pacific Islander, unknown), Spanish origin (nine categories), and marital status at diagnosis (single, married, separated, divorced, widowed). For the TNM T-stage, Hispanic origin and radiation/surgery sequence variables, similar categories with relatively few observations were collapsed into larger classes (see [42]). Individuals with missing observations for either the predictor or the outcome variables were excluded from the analysis. The final analysis dataset included 121, 798 observations, with a median survival time of 63 months (min: 1, max: 156) and a censoring rate of approximately 93.1%.

We randomly split the dataset into a training and a test set of equal size (n_training = n_test = 60, 899, to rule out possible effects linked to different sample sizes) and generated 10 bootstrap datasets each. Using either Cox proportional hazards regression, random forests, or extreme gradient boosting, we fitted a survival model to the bootstrapped training data and a censoring model to either the bootstrapped training, test or combined dataset, analogously to the steps described previously for the simulated survival data. Note that in contrast to the simulation study, we did not consider a penalized Cox model, i.e., Lasso, because of the lack of a natural definition of an untuned model with “default” hyperparameters, so that a comparison of a tuned and untuned model would not have been possible in a natural way. For the tuned models, prior to fitting the models, the main hyperparameters of the random forest and XGBoost algorithms were tuned using a Bayesian optimization algorithm [39, 40] on each dataset. For the untuned models we used the default settings provided in the respective R packages. Further details regarding the tuning algorithm, priors and the chosen hyperparameters are provided in S2 File. Predictions for , , and were obtained on the test data. The IPCW Brier score was then computed for each of the censoring models and averaged over the 10 bootstrapped datasets. Further information on the SEER program and the available data can be found on the SEER website (https://seer.cancer.gov). The R scripts with the pre-processing steps and the computation of the IPCW Brier score are available on Github.

Results

The following results provide an analysis of the predictive performance of three estimation methods—Cox proportional hazards, random forest and XGBoost—as a function of time and of the estimated censoring survival function using the IPCW Brier score. In contrast to the simulation setting, where the true underlying survival probabilities S_i(t) are known and the uncensored Brier score may thus serve as a point of reference (ground truth), here, the true survival probabilities remain unknown. Calculating the WSE is therefore not possible. The assessment of possible differences in performance as a result of fitting the censoring function to either the training set, test set or both thus remains purely qualitative. However, it is still possible to make some interesting observations. Firstly, the predictive performance does not seem to be severely impacted by the choice of the data used to fit the censoring function. Using the training or the test set, or the combined dataset comprising both, leads to very similar results, see Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Analysis of the SEER breast cancer data using a Cox proportional hazards model.

The plot shows the mean (bold center line) and standard deviation (shaded area) of the IPCW Brier score obtained on 10 bootstrap test samples, with IPC weights estimated from either the training, test, or the combined dataset. A Cox proportional hazards model was used for estimating the survival and the censoring survival functions.

https://doi.org/10.1371/journal.pone.0318349.g006

Secondly, it is important to stress the importance of prior tuning of the ML models. In order to illustrate this, Figs 7 and 8 show the prediction error curves for both the untuned models (with the hyperparameters set to the default values pre-specified in the respective R packages, see S2 File), as well as for the models where the most important hyperparameters were tuned. As seen in Figs 7 and 8, we can observe differences in performance with respect to the data used for fitting the censoring survival function when the models are not properly tuned. This changes with prior tuning of the models, whereby we observe a convergence of the Brier score error curves. However, as we do not know the “ground truth”, we cannot definitively say which dataset is best suited for model fitting. Interestingly, the differences between the predictive performance of the tuned and the untuned models are less pronounced for the XGBoost models than for the random forest models. This may be because the default settings of the XGBoost algorithm could have been closer to the “optimal” values than those of the random forest for this particular dataset. Additional figures showing only the mean values of the IPCW Brier Score for better legibility are available in S1 Fig.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Analysis of the SEER breast cancer data using random forest.

The plots show the mean (bold center line) and standard deviation (shaded area) of the IPCW Brier score obtained on 10 bootstrap test samples, with IPC weights estimated from either the training, test, or the combined dataset. A random forest was used for estimating the survival and censoring survival functions. The left panel shows the results of an untuned model, with the number of trees set to 500 and all other hyperparameters set to their default values, whereas the model in the right panel was tuned using Bayesian optimization.

https://doi.org/10.1371/journal.pone.0318349.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Analysis of the SEER breast cancer data using XGBoost.

The plots show the mean (bold center line) and standard deviation (shaded area) of the IPCW Brier score obtained on 10 bootstrap test samples, with IPC weights estimated from either the training, test, or the combined dataset. XGBoost was used for estimating the survival and censoring survival functions. The left panel shows the results of an untuned model, with the number of boosting rounds set to 500 and all other hyperparameters set to their default values, whereas the model on the right was tuned using cross-validation and Bayesian optimization.

https://doi.org/10.1371/journal.pone.0318349.g008

In addition to analyzing the preprocessed SEER data described above, we repeated our study using a modified semi-synthetic version of the SEER data, following the methodology of Qi et al. [43]. Further details regarding the methodology and the results of the semi-synthetic analyses can be found in S3 File.