2.1 Setup

Let be a multistate survival process with states. We assume a multistate model has already been developed and we want to assess the calibration of the predicted transition probabilities, , in a cohort of interest. The transition probabilities are the probability of being in state at time , if in state at time , where . To assess the calibration of the multistate model, we must estimate observed event probabilities:

In a well calibrated model, the transition probabilities will be equal to the observed event probabilities.

In the absence of censoring, can be estimated using cross sectional calibration techniques in a landmark [25,26] cohort of individuals who are in state at time (i.e., methods to assess the calibration of models predicting binary or multinomial outcomes). In the presence of censoring, calibration must be assessed in this landmark cohort of individuals either using cross sectional techniques in combination with inverse probability of censoring weights, or through pseudo-values. These approaches were previously proposed and evaluated in a simulation study [24], but were restricted to assessing calibration out of the starting state at time . The theory is summarised and revised to allow assessment of calibration out of any state at any time in sections 2.2–2.6.

2.2 Binary logistic regression with inverse probability of censoring weights (BLR-IPCW) calibration curves

The first approach produces calibration curves using a framework for binary logistic regression models in conjunction with inverse probability of censoring weights to account for informative censoring (BLR-IPCW). Let be an indicator for whether an individual is in state at time . is then modeled using a flexible approach with as the sole predictor. This model is fit in the landmark cohort (in state at time ) of individuals who are also still uncensored at time . This cohort is weighted using inverse probability of censoring weights (see section 2.4). We suggest using a loess smoother [29]:

(1)

or a logistic regression model with restricted cubic splines [30]:

(2)

Any flexible model for binary outcomes could be used, but these are the most common and are implemented in this package. Observed event probabilities are then estimated as fitted values from these models. These are commonly referred to as ‘predicted-observed risk’ values. The calibration curve is plotted using the set of points . To obtain unbiased calibration curves, the assumption that each outcome is independent from the censoring mechanism in the reweighted population must hold.

2.3 Multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW) calibration scatter plots

The second approach produces calibration scatter plots using a framework for multinomial logistic regression models with inverse probability of censoring weights (MLR-IPCW). Let be a multinomial indicator variable taking values such that if an individual is in state at time . The nominal recalibration framework of Van Hoorde et al. [31,32], is then applied in the landmark cohort of individuals uncensored at time , weighted using inverse probability of censoring weights (section 2.4). First calculate the log-ratios of the predicted transition probabilities:

Then fit the following multinomial logistic regression model:

where is an arbitrary reference category which can be reached from state , takes values in the set of states that can be reached from state , and where is a vector spline smoother [33]. Observed event probabilities are then estimated as fitted values from this model. This results in a calibration scatter plot rather than a curve due to all states being modeled simultaneously, as opposed to BLR-IPCW, which is a “one vs all” approach. The scatter occurs because the observed event probabilities for state vary depending on the predicted transition probabilities of the other states. This is a stronger [9] form of calibration than that evaluated by BLR-IPCW, and will also result in observed event probabilities which sum to 1. In future iterations of calibmsm functionality will be added to produce smoothed curves estimated from these scatter plots. To obtain unbiased calibration curves, the assumption that the outcome is independent from the censoring mechanism in the reweighted population must hold.

2.4 Estimation of the inverse probability of censoring weights

The estimand for the weights is , the inverse of the probability of being uncensored at time if in state at time :

(3)

where denotes the history of the multistate survival process up to time , including the transition times, and is a set of baseline predictor variables believed to be predictive of the censoring mechanism. Note that may be the same as, but is not restricted to, the variables used for prediction when developing the multistate model. First the estimator is calculated by developing an appropriate survival model. The outcome in this model is the time until censoring occurs. Moving into an absorbing state prevents censoring from happening and is treated as a censoring mechanism in this model (i.e., a competing risks approach is not taken when fitting this model). is explicitly conditioned on when defining because the weights must reflect that censoring can no longer be observed for an individual if they enter an absorbing state at some time . Therefore, in calibmsm, the weights are estimated as:

Note that if the censoring mechanism is not conditionally independent from the outcome process given , i.e., the rate of censoring changes depending on outcome state occupancy, then this approach will be invalid. Instead, the outcome history up until time must be conditioned on when estimating the weights, as specified in equation (3). By default, is estimated in calibmsm using a cox proportional hazards model where all predictors are assumed to have a linear effect on the log-hazard. These conditions are highly restrictive. Users can therefore also input their own vector of weights which is strongly recommended. calibmsm has been developed to provide a framework for assessing calibration, and the packages’ primary purpose is not estimation of inverse probability of censoring weights, however, future versions may allow for more flexible models to estimate the weights. Given the BLR-IPCW and MLR-IPCW approaches are both reliant on correct estimation of the weights, we encourage users to take the time to carefully estimate these using a well specified model. The limitations of using the calibmsm internal functions for estimating the weights in this clinical example (section 4) are discussed in more detail later, and explored in vignette Sensitivity-analysis-for-IPCWs [27].

Stabilised weights can be estimated by multiplying by the weights by the mean probability of being uncensored:

The numerator can be estimated using an intercept only model, and note there is no dependence on .

Another option is to estimate , which is the inverse of the probability of being uncensored at time if uncensored at time :

This can be estimated using the same approach as for , except there is no requirement to be in state when landmarking at time . If the censoring mechanism is conditionally independent from after conditioning on , then , and any consistent estimator for will be a consistent estimator of . The advantage is that is calculated by developing a model in the cohort of individuals uncensored at time , which is a larger cohort than those uncensored and in state at time . Therefore will be a more precise estimator than . On the contrary, if the rate of differs depending on , there is a risk of bias in estimation of the weights. We therefore recommend using the estimator unless sample size (number of individuals in state at time ) is low, which may be assessed using sample size formula for prediction models with time-to-event outcomes [34]. If the sample size is deemed insufficient, one may consider using , but the risk of bias associated with this estimator must be carefully considered.

We note the importance of using inverse probability of censoring weights, even if the censoring mechanism is believed to be completely at random. When a multistate model has an absorbing state (which is the case for most), entry into this state will prevent censoring from happening. This induces a dependence between the outcome and the censoring mechanism which must be adjusted for using inverse probability of censoring weights. This issue was highlighted in the supplementary material of previous work [24].

2.5 Pseudo-value calibration plots

The third approach produces calibration curves using pseudo-values [35,36]. Pseudo-values can be used in place of the outcome of interest in a regression model if some outcomes are not observed due to right censoring. This is the case in models (1) and (2). For certain estimators (where estimates the expectation of the outcome it is replacing), the pseudo-value for individual is defined as:

,

where is equal to estimated in a cohort without individual . One such estimator for the outcomes in models (1) and (2) given the underlying multistate survival process, is the Landmark Aalen-Johansen estimator [37], which estimates the expectation of in the landmark cohort of individuals in which calibration is being assessed. The resulting pseudo-values are a vector with elements, one for each possible transition, for every individual . These pseudo-values can replace the outcome in equations (1) and (2) in order to estimate .

Pseudo-values are based on the same assumptions as the underlying estimator . The Landmark Aalen-Johansen estimator is valid for both Markov and non-Markov multistate models. However, it does make the assumption of random censoring. According to Kleinbaum and Klein [38], this means that “subjects who are censored at time should be representative of all the study subjects who remained at risk at time with respect to their survival experience”. The approach to alleviate this is to estimate the pseudo-values within sub-groups of individuals, now making the assumption of random censoring within the specified subgroups. This can be done by calculating the pseudo-values within subgroups defined by baseline predictors, or subgroups defined by the predicted transition probabilities . Both options are implemented in this package. When pseudo-values are calculated within subgroups, they are still used as the outcome in models (1) and (2) in the same way. Note that the pseudo-values are continuous, as opposed to binary , but the link function in model (1) remains the same to ensure are between zero and one. Here, are commonly referred to as ‘pseudo-observed risk’ values.

2.6 Estimation of confidence intervals

Confidence intervals for both BLR-IPCW and pseudo-value calibration curves can be estimated using bootstrapping. While theoretically feasible, it is currently unclear how to present confidence intervals for each data point in the calibration scatter plots produced by MLR-IPCW, and therefore these are omitted. A process for estimating the confidence intervals around the BLR-IPCW calibration curves is as follows:

1. Resample validation dataset with replacement
2. Landmark the dataset for assessment of calibration
3. Calculate inverse probability of censoring weights
4. Fit the preferred calibration model in the landmarked dataset (restricted cubic splines or loess smoother)
5. Generate observed event probabilities for a fixed vector of predicted transition probabilities (specifically the predicted transition probabilities from the non-bootstrapped landmark validation dataset)

This will produce a number of bootstrapped calibration curves, all plotted over the same vectors of predicted transition probabilities. Taking the and percentiles of the observed event probabilities for each predicted transition probability gives the required confidence interval around the estimated calibration curve. To estimate confidence intervals for the pseudo-value calibration curves using bootstrapping, the same procedure is applied except the third step is replaced with ‘calculate the pseudo-values within the landmarked bootstrapped dataset’. This will be highly computationally demanding as the pseudo-values must be estimated in every bootstrap dataset. This process is integrated into calibmsm internally, however code for how to implement these steps manually is also provided in the vignette BLR-IPCW-manual-boostrap [27].

If using the pseudo-value method, confidence intervals can however be calculated using closed form estimates of the standard error when making predictions of the observed event probabilities (i.e., when obtaining fitted values from models (1) and (2)). We recommended this due to the computational burden of bootstrapping the confidence intervals around the pseudo-value calibration curves. There are a number of issues with estimating parametric confidence intervals for the BLR-IPCW calibration curves. Firstly, a robust sandwich-type estimator should be used to estimate the standard error [39], which are known to result in conservative confidence intervals. On the contrary, the size of the confidence interval will be underestimated as uncertainty in estimation of the weights is not considered. Due to the impact of these two factors, we recommend using bootstrapping to estimate the confidence intervals for BLR-IPCW calibration curves.

4.1 Clinical setting and data preperation

We utilise data from the European Society for Blood and Marrow Transplantation [28], containing multistate survival data after a transplant for patients with blood cancer. The start of follow up is the day of the transplant and the initial state is alive and in remission. There are three intermediate events (: recovery, : adverse event, or : recovery + adverse event), and two absorbing states (: relapse and : death). This data is available from the mstate package. We assume the user of calibmsm has experience with handling the type of data used to develop a multistate model as outlined by De Wreede et al [15].

Four datasets are provided to enable assessment of a multistate model fitted to these data. The code for deriving all these datasets is provided in the source code for calibmsm [27]. The first is ebmtcal, which is the input for argument data_raw. This data is the same as the ebmt dataset provided in mstate, with two extra variables derived: time until censoring (dtcens) and an indicator for whether censoring was observed (dtcens_s = 1) or an absorbing state was entered (dtcens_s = 0). This dataset contains baseline information on year of transplant (year), age at transplant (age), prophylaxis given (proph), and whether the donor was gender matched (match), and information on time until entry into the different outcome states. Such a dataset would need to be constructed by the user after prospective data collection or manipulation of observational data.

The second dataset provided is msebmtcal, which is the input for argument data_ms. This is a dataset of class msdata, and has been derived from the ebmt dataset by applying the processes and functions from the package mstate [15]. It contains all transition times, an event indicator for each transition, as well as a trans attribute containing the transition matrix. Note that while the data on transition times is not required to be present in ebmtcal in order to run calib_msm, it is required in order to derive the dataset msebmtcal from ebmtcal.

In the work of De Wreede et al. [15], the focus is on predicting transition probabilities made at times and days, across a range of follow up times , and comparing prognosis for patients in different states . In this study we also focus on assessing the calibration of the transition probabilities made at these times. We assess calibration of the transition probabilities at years, a common follow up time for cancer prognosis, but calibration of the model may vary for other values of . We estimate transition probabilities for each individual by developing a model as demonstrated in de Wreede et al. [15], following the theory of Putter et al [2].

The predicted transitions probabilities from each state at times and are contained in stacked datasets tps0 and tps100 respectively. A leave-one-out approach was used when estimating these transition probabilities. This means each individual was removed from the development dataset when fitting the multistate model to estimate their transition probabilities. This approach allows validation to be assessed in the same dataset that the model was developed with minimal levels of in-sample optimism. Note that for tps100 the predicted probabilities for some states are equal to . This is because no individuals in state at time transition into states or . This may be due to the definition of an adverse event having to occur within a certain number of days post transplant.

4.2 Calibration plots for the transition probabilities out of state at time

We start by producing calibration curves for the predicted transition probabilities out of state at time . Given all individuals start in state , there is no need to consider the transition probabilities out of states at . Calibration is assessed at follow up time ( days). We start by extracting the predicted transition probabilities from state at time from the object tps0. These are the transition probabilities we aim to assess the calibration of.

We first evaluate calibration using the BLR-IPCW approach by specifying calib_type = “blr”. We choose to estimate the calibration curves using restricted cubic splines, although the use of loess smoothers would be equally valid. When using restricted cubic splines, the number of knots must always be specified by the user, and 3 knots are chosen here given the reasonably small size of the dataset. Weights are estimated using the internal estimation procedure with baseline predictor variables year, agecl, proph and match. The w_max_follow=t_eval argument censors individuals at t_eval before fitting the model used to estimate the weights, i.e., a “stopped cox” approach [42]. This decision was made to help meet the proportional hazards assumption as there is differential follow up for individuals in different year groups (see vignette Sensitivity-analysis-for-IPCWs [27] for more details). The w_landmark_type argument assigns whether weights are estimated using all individuals uncensored at time , or only those uncensored and in state at time , as discussed in section 2.4 The maximum weight (w_max = 10) and stabilisation of weights (w_stabilised = TRUE) are left as default. Weights can also be manually specified using the weights argument. We request 95% confidence intervals for the calibration curves calculated through bootstrapping with 200 bootstrap replicates.

The first element of dat_calib_blr (named plotdata) contains 6 data frames. One for the calibration curves of the transition probabilities into each of the six states, Each data frame contains five columns, id: the identifier of each individual; pred: the predicted transition probabilities; obs: the observed event probabilities; obs_lower and obs_upper: the confidence interval for the observed event probabilities. The second element (named metadata) is a metadata argument containing information about the data and chosen calibration analysis. The plot data and metadata can be viewed using the print and metadata commands respectively. However, it is recommended to get acquainted with the underlying object structure, as accessing the plot data will be useful if wanting to customise plots or apply bootstrapping manually.

The BLR-IPCW calibration curves can then be generated by applying the plot function to the output from calib_msm. Note, when marginal density curves are requested (default), grid::grid_draw is the preferred approach to display this plot in the viewer. The calibration curves (Fig 3) indicate the level of calibration is different for the transition probabilities into each of the different states. The calibration into states and 6 looks the best. State has good calibration over the majority of the predicted risks but over predicts for individuals with the highest predicted risks. Transition probabilities into states and are over and under predicted respectively over most of the range of predicted risks. Importantly the calibration of the transition probabilities into state (Relapse), a key clinical outcome in this clinical setting, is extremely poor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. BLR-IPCW calibration curves out of state j= 1 at time s = 0.

https://doi.org/10.1371/journal.pone.0320504.g003

Next we use the pseudo-value approach to assess calibration by specifying calib_type = “pv”. Instead of specifying how the weights are estimated, we now specify variables to define groups within which pseudo-values will be calculated (see section 2.5). The goal is to induce uninformative censoring within the chosen subgroups. We chose to calculate pseudo-values in individuals with the same year of transplant (pv_group_vars = c(“year”)), and then split individuals into a further three groups defined by their predicted risk (pv_n_pctls = 3). The number of percentiles should be increased in bigger validation datasets, although guidance on specific numbers is currently lacking. Year of transplant was identified as a subgrouping variable because a later transplant resulted in a shorter possible follow up, an earlier administrative censoring time, and it was therefore highly predictive of being censored. Your data should be explored to identify appropriate variables for subgrouping (see vignette Evaluation-of-estimation-of-IPCWs). A parametric confidence interval is estimated as recommended in section 2.6.

The pseudo-value calibration curves (Fig 4) are largely similar to the BLR-IPCW calibration curves (Fig 3). The agreement in the calibration curves from two completely distinct methods provides some reassurance on the validity of the assessment of calibration. This is with the exception of state , where the pseudo-value calibration plot indicates the transition probabilities are well calibrated, but the BLR-IPCW calibration plot indicates the transition probabilities under predict. In a situation like this, we recommend testing the assumptions made by each of the methods to try and diagnose which are most likely to hold, and what may be driving the difference. In this particular example, we hypothesised that the cox-proportional hazards model for estimating the inverse probability of censoring weights may be misspecified, in particular the proportional hazards assumption, due to the strong effect of year of transplant on the censoring mechanism. We explored this theory in more detail (see vignette Sensitivity-analysis-for-IPCWs), but found little change when estimating the weights using a flexible parametric survival model. Instead, we identified that this may be caused by a difference in the censoring mechanism for individuals in the adverse event state, as it appeared these individuals were less likely to be censored. This will bias the results from the BLR-IPCW and MLR-IPCW methods unless the weights are conditional on the amount of time spent in each outcome state, something which calibmsm is not currently set up to do. Although it’s not possible to be certain that the individuals in the adverse event state were less likely to be censored purely from looking at the data, we concluded it was a strong possibility, and that the BLR-IPCW calibration curves may be biased in this particular clinical example.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Pseudo-value calibration curves out of state j= 1 at time s = 0.

https://doi.org/10.1371/journal.pone.0320504.g004

Next we use the MLR-IPCW to evaluate calibration which produces a calibration scatter plot by specifying calib_type = “mlr”. The inputs for calculating the weights are the same as for the BLR-IPCW approach, but a confidence interval is no longer requested which is not possible for the MLR-IPCW approach.

The MLR-IPCW calibration scatter plots, produced using plot are contained in Fig 5. Within each plot for state , there is a large amount of variation in calibration of the transition probabilities depending on the predicted transition probabilities into states . One valuable insight from these plots is that the variance in the calibration of the transition probabilities into state , is considerably smaller than that of state , despite these two states both having good calibration according to the BLR-IPCW plots (arguably state looked better calibrated). This means the calibration of the transition probabilities into state remains reasonably consistent, irrespective of the risks of the other states. On the contrary, the calibration of the predicted transition probabilities into state is more dependent on the predicted transition probabilities of the other states. These plots can also be used to identify groups of individuals where calibration is poor. For example, there is a cluster of individuals with a predicted-observed risk > 17.5% for state 3, where the model is under-predicting. These are all individuals aged 20–40, year group 1995–1998, no gender mismatch and no prophylaxis.

In practice, we recommend researchers produce calibration curves using all three methods. The MLR-IPCW approach is a stronger [9] form of calibration assessment than the BLR-IPCW and pseudo-value approaches, but all three provide important contextual information. Note that we hypothesized that the inverse probability of censoring weights may not be accurate in this example due to a censoring mechanism which changes depending on outcome state occupancy. Using the current approach for estimating the weights, this will result in biased calibration plots for the BLR-IPCW and MLR-IPCW methods, having a particular effect on the plots for state 3 (adverse event). For the following section, we therefore proceed using only the pseudo-value method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. MLR-IPCW calibration scatter plots out of state j= 1 at time s = 0.

https://doi.org/10.1371/journal.pone.0320504.g005

4.3 Calibration plots for the transition probabilities out of states and at time

In the work of De Wreede et al. [15], focus then shifts to comparing transition probabilities when depending on whether an individual has had an adverse event (state $3$) or remains in state (post transplant). Our focus therefore now shifts to assessing the calibration of these transition probabilities. This is done through landmarking as described in section 2. We start by extracting the predicted transition probabilities from state and at time from the object tps100. These are the transition probabilities we aim to assess the calibration of.

The process for estimating the calibration curves remains the same, changing the inputted values j and s, and specifying the appropriate predicted transition probabilities to the argument tp_pred. We start by producing the calibration plots for and using pseudo-value (Fig 6) method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Pseudo-value calibration curves out of state j= 1 at time s = 100.

https://doi.org/10.1371/journal.pone.0320504.g006

There are only four calibration plots because no individuals in state at time are in states (adverse event) or (recovery + adverse event) after days. We believe this is due to the definition of an adverse event occuring within 100 days, but as secondary users of the data, cannot be sure about this. The calibration of the predicted transition probabilities according is very poor. Only for state is the observed risk a monotonically increasing function of the predicted transition probabilities. The confidence intervals are very large. For states and , we cannot rule out that the poor calibration is a result of sampling variation as opposed to a poorly performing prediction model. A larger validation dataset would be required to get to the bottom of this. There is a major issue with the calibration of the transition probabilities of staying in state , as the predicted risk is inversely proportional to the observed event rate.

Next we produce calibration plots for and (Fig 7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Pseudo-value calibration curves out of state j= 3 at time s = 100.

https://doi.org/10.1371/journal.pone.0320504.g007

Again there are only four possible states that an individual may transition into, although this includes states (adverse event) and (recovery + adverse event), instead of (post transplant) and (recovery). This is because once an individual has entered state , they cannot move backwards into states or . The calibration plots are better than for . For transitions into states and , the calibration curves are monotonically increasing and comparatively close to the line of perfect calibration, although the confidence intervals are still quite large. Again the calibration of state is very poor. This makes it difficult to base any clinical decisions on the predicted transition probabilities for relapse out of states or at time , whereas making clinical decisions based on the risk of death () after survival for days is more viable, as this was well calibrated for both and .

We provide an overview of interpretating the calibration curves for this clinical example in Box 1.