Background

Treatment decisions for people with relapsing-remitting multiple sclerosis (PwRRMS) are based on clinical but also paraclinical disease activity [1–5]. MRI measures are often used to predict the disease course of RRMS and to design individual treatment strategies. MRI measures also indicate treatment failure during the future disease course [5]. Clinical decision-making can be based on individual-level surrogacy (ILS) concepts between a prognostic surrogate endpoint (SEP) and a clinical endpoint (CEP). Precise individual predictions of the future disease course (CEP) are essential. The paper explains that evidence on high-quality surrogacy cannot be reproduced within a set of randomized clinical trials.

Our paper discusses the likelihood reduction factor (LRF) as a measure to quantify ILS quality. LRF is related to mutual information (MI) [6] and is to our knowledge rarely applied in MS literature, but often discussed in the methodological literature on surrogacy. We explain why LRF is the method of choice and apply it to data from ten phase II/III clinical trials in PwRRMS.

If the SEP deteriorates in a PwRRMS, the treating physician may conclude that the future CEP is also deteriorating. Subsequently, he/she may suggests more effective treatment as a preventive measure. However, there is rarely reliable information on which physicians and PwRRMS can weigh up the pros and cons of the decision.

A less successful effort is underway to develop individual prediction models that are supposed to provide this information. Reviews in this field question the methodological state-of-the-art of models developed for PwRRMS [7,8]. Before outlining the rationale for choosing LRF as our preferred measure, we briefly review how ILS quality is addressed in the RRMS literature. One central weakness is the use of measures that capture associative effects instead of those that reflect individual predictive accuracy (see also section 4 in S1 Text).

Some papers [9–16] quantify ILS quality by demonstrating a significant difference in SEP between treatment response and non-response groups (as measured by the SEP outcome; see S1 Table for more details). Here, a significant p-value is considered proof of ILS. Unfortunately, it is not possible to convert the p-value into a quantitative risk statement. Significance provides qualitative evidence of a potential ILS effect and depends on the size of the population studied. A large population may even render a clinically irrelevant weak ILS effect significant. Additionally, authors model the SEP-CEP relationship using logistic regression or Cox proportional hazards models. Typically, they do not report their models except for the odds ratio (OR) or hazard ratio (HR) attributed to the SEP [9–16]. Again, significant ORs or HRs are considered proof of ILS, even though the significance may depend solely on the size of the study sample. Clearly, large OR or HR values influence the predictive value of the SEP. The positive predictive value (PPV) for an individual is higher with larger OR values of the predictor. However, a large OR does not help to quantify individual risk precisely, as it reflects a relative ratio which transforms a baseline risk. Different individual baseline risks give different individually transformed risks even for the same OR. If we have a risk of 10% or 30% for a severe disease course in the group of a favorable SEP, the respective risks are 18% and 46% in the group of unfavorable SEP given an OR of 2. This missing link between OR or HR and risk statements is rarely reflected in MS literature.

One study provides correlation values between SEP and CEP [17]. However, a significant correlation is not sufficient to establish clinically relevant ILS quality. As we know from linear regression, the coefficient of determination (R²) measures the percentage of variability explained. R² values close to 1 are essential for accurately predicting the dependent variable given the independent variable(s). The formula for R² is independent of the sample size used in the linear regression analysis. An informative report on the quality of ILS in linear regression settings should also include confidence intervals for R² measures.

Accordingly, we require a type of R² measure for more general settings between SEP and CEP. LRF is such a candidate and represents a generalization of R². However, the important point here is not whether the LRF is different from 0, but how high the LRF is. In order to make precise predictions, a threshold value must be determined to establish the quality and confidence intervals for the corresponding estimators. The threshold defines a minimal quality measure above which the metric is considered clinically useful. For example, an SEP with a corresponding LRF value above a pre-defined threshold may be indicative for treatment failure. Although correlation coefficients (r) and R² values are reported, the concept of applying pre-defined thresholds for clinical utility is not used in MS literature. To our knowledge, no studies to validate threshold values for r or R² has been conducted for PwRRMS. To define such a threshold for MS is out of the scope of this article.

When considering dichotomous SEP and CEP values (favorable/unfavorable), many papers report differences in SEP values between PwRRMS with favorable and unfavorable CEP responses. These differences are often expressed in terms of sensitivity (SE: percentage of people with an unfavorable SEP within people with an unfavorable CEP) and specificity (SP: percentage of people with a favorable SEP within people with a favorable CEP). However, sensitivity and specificity are not predictive concepts. Predictive statements are made using predictive values. The positive predictive value (PPV: percentage of people with an unfavorable CEP within people with an unfavorable SEP) and the negative predictive value (NPV: percentage of people with a favorable CEP within people with a favorable CEP) can be derived from sensitivity (SE) and specificity (SP), together with the prevalence of unfavorable CEPs (see section 5 in S1 Text). For example, the title of Table 1 in MAGNIMS (2015) may be misinterpreted (“MRI criteria for predicting treatment response”) as the data shown does not allow PPVs or NPVs to be calculated. In contrast, Table 1 in Rio et al. [15] enables such a calculation. Nevertheless, it is still not standard practice in ILS RRMS literature to report predictive values, although some papers report not only predictive values but also accuracy values.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Trial information.

https://doi.org/10.1371/journal.pone.0337893.t001

Predictive values are conditional measures on sub-populations with favorable or unfavorable SEP outcomes. An ILS quality measure should assess ILS for the entire population, like the R² measure does. The accuracy percentage is a global measure for the entire population which is , where P_CEP is the prevalence of unfavorable CEP outcome. It is equal to , where is the prevalence of unfavorable SEP outcome. However, accuracy cannot easily be generalised to settings involving non-binary endpoints and is rarely reported.

The literature also contains information on the quality of prediction models by reporting its area under the receiver operating curve (AUC under the ROC) [18]. The AUC is a discrimination measure. This information is not helpful when discussing precise individual risks during shared decision making. The quality of the prediction is also determined by calibration. Calibration of risk models assesses whether accurately predicted probabilities of risk events align with observed outcomes, ensuring the model’s outputs are reliable for decision-making. Good calibration is essential to shared decision making. In the MS literature, information on model calibration is generally lacking. External validation is the state-of-the-art, but rarely provided for prediction models in the RRMS literature [7,8]. This means that the generalisability of the evaluated prediction models to new persons remains unproven, and their clinical applicability cannot be assumed. We conclude that the availability of high-quality quantitative individual risk estimates is sparse.

A prominent assessment strategy for establishing ILS for PwRRMS uses the Prentice criteria [12] and the proportion of treatment effect explained (PTE), introduced by Freedman [19]: . Here, is the regression coefficient when CEP is regressed on the treatment variable, and is the regression coefficient for treatment when CEP is regressed on treatment and SEP. If SEP contains all (or none) of the information on treatment, then = 0 and PTE = 1 ( = 1 and PTE = 0). PTE is a group-based measure derived from group-specific regression coefficients. However, it ignores the variability around the regression lines, which actually determines the quality of the CEP prediction given the SEP. Despite its frequent use [20,21], high PTE values do not guarantee accurate predictions at the individual level (see sections 1 and 4 in S1 Text).

Another qualitative approach to assessing the relevance of the SEP in a CEP prediction model is the likelihood ratio test (based on −2LLR, where LLR is the log-likelihood ratio) comparing models with and without the SEP. A significant test result is often considered evidence of ILS. However, even if the SEP has a small influence on CEP prediction, a large sample size may lead to significant test results. Again, a significant likelihood ratio test indicates potential ILS, but does not provide information on ILS quality.

Mutual information (MI) is a general concept that quantifies the amount of information shared between two random variables [6]. It is a powerful tool for detecting and measuring statistical dependence in fields such as machine learning, information theory, and data analysis. In likelihood models, MI is the mean log-likelihood difference between the model with and without the SEP: . In the ILS context, MI can be interpreted as the average change in individual likelihood of the model including the SEP compared to the model without the SEP. MI represents a form of likelihood alteration that is independent of sample size, and it quantifies the shared information between the SEP and the CEP at an individual level within the given population. In general, MI quantifies the increase in log-likelihood difference when independence between two outcomes is assumed.

Alonso and colleagues [6] introduced the LRF as a function of MI, as well as an instrument to understand the relationship between the SEP and the CEP.

(1)

The mutual information (MI) between an SEP and a CEP is . Here, is the entropy of the CEP and is the entropy of the CEP given the SEP. If the SEP predicts the CEP well, is small compared to and makes large, resulting in an LRF value closer to 1. If SEP and CEP are independent (SEP carries no information on CEP), the LRF equals zero since . Alonso et al. demonstrated that the LRF has properties of a R² measure and equals R² in case of a linear model [6].

Section 3 of the S1 Text provides examples of how to calculate the MI or LRF for 2-by-2-tables (3a) or for a logistic regression model (3b). While the PPV only reflects those with an unfavorable SEP, the LRF applies to the entire population, regardless of the value of the SEP. An LRF close to 1 suggests that the examined prognostic factor(s) add relevant information to a predictive model. A high MI suggests that predictive models may exist, but whether such a model with high performance can be found depends on how the information is structured and exploited by the modeling approach. Exploring the LRF should be part of any predictive model development.

From a clinical perspective, the LRF expresses how much percentage of the uncertainty in an individual prediction is explained by considering the SEP. A higher reduction means that the prediction becomes more precise, for example when the expected range of a deterioration in EDSS values is narrower instead of wide. As mentioned above, a low variability in individual predictions is essential for informed clinical decision making.

We demonstrated the need for predictive models for PwRRMS. Many researchers have taken up this challenge, claiming to possess such models [7,22]. Often, however, the published models yield low to moderate performance metrics. One reason could be that the models cannot be any better due to the limited methodology used.

The paper is organised as follows: 1) We introduce the methodological concepts and present a simulation study to assess the LRF as a quality measure for ILS; 2) We analyse and interpret ILS in the data of ten MS trials; 3) We evaluate ILS as discussed in the literature; 4) We perform PTE analyses with our data and compare them to the results in the literature; 5) As a practical application we provide an assessment of the evidence of T2L as an individual-level SEP and treatment failure indicator based on literature cited in two MS guideline articles.

Materials and methods

Surrogate validation methods

Alonso and colleagues [6] proposed to estimate mutual information (MI) by the mean contribution of the log-likelihood ratio between two models. The first model includes the longitudinal SEP as a covariate, and the second model does not (Equation (2)).

(2)

where:

• = Clinical endpoint vector
• = Surrogate endpoint vector
• = number of subjects
• , , and are the associated design matrices and trial treatment effects
• are the trial specific effects of the SEP
• and link functions relate the linear predictors of the models to the mean value of the outcome.
• and are functions to describe the time trajectories

Following Equation (1), = .

Kent et al. and Alonso et al. [36,37] describe asymptotic confidence intervals for LRF. See section 2 in S1 Text for more details. The LRF was estimated for each pair of control and active study arm combinations (considering multi arm trials) as given in column Analysis Groups in Table 1.

Results

Simulation studies

Fig 1 displays the results of the simulation study (rows: two or four time points; columns: analysis strategies). Columns one to six show LRFs for transformed count data, the last three columns display LRFs for original count data. For each setting, the red horizontal line indicates the true LRF values. The top of each panel shows the number of numerically unstable estimations out of 1000 simulation iterations. The results demonstrate that the LRF estimation performs reliably, with only two instances of numerical instability observed (column six, rows one and five). The estimates of the LRFs slightly underestimated the true LRF, particularly for a small number of simulated individuals ( = 100 or 300) and two time points. In scenarios with low correlations, the LRF can become negative indicating that the reduced model without the SEP has a higher log-likelihood than the model including it, suggesting the SEP lacks detectable predictive ability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Simulation (1000 iterations): LRF derived from models of different distributional families.

https://doi.org/10.1371/journal.pone.0337893.g001

Results from the simulation study, conducted with n = 1000 iterations. Red horizontal lines in the graphs indicate the true LRF values. The numbers displayed at the top of each panel correspond to the instances of convergence issues observed during the simulation study. To derive the LRF, Gaussian, Poisson, or ordinal models were employed. It is important to note that the results of the Poisson family were analyzed in their untransformed form. The study considered four combinations of SEP and CEP: 1) Gaussian – Gaussian, 2) Gaussian – (transformed) Poisson, 3) transformed Poisson – Gaussian, and 4) transformed Poisson – (transformed) Poisson. The datasets included 100, 300, and 600 subjects (labeled at columns title), contributing to a comprehensive analysis of model performance. Abbreviations: LRF: Likelihood Reduction Factor; tr.: transformed; Pois: Poisson; Tp: time point.

S5 Fig presents additional results from the simulation scenario (SEP and CEP simulated as count data and analyzed using negative binomial and zero-inflated Poisson regression models). Instances of numerical instability were observed. S6 and S7 Figs display the absolute differences between the true LRF values and those estimated from the simulated data. These differences range from −0.203 to 0.023 for original (S6 Fig) and from −0.104 to 0.023 for transformed count data (S7 Fig), indicating a small to moderate bias. When both SEP and CEP are normally distributed, the LRF shows minimal bias, though smaller sample sizes or transformed endpoints can underestimate its true value (S7 Fig).

The second simulation study uses the time-lagged model (S8 Fig). Results indicate that a higher number of measurement time points lead to higher (= LRF) values. is not adjusted to the number of longitudinal measurements.

Surrogacy in the trial data

Seven of the ten trials were multi-arm trials, and 19 pairs of treatment/control arms were available for analysis (see Table 1). S5 Table provides median and interquartile ranges (IQR) aggregated over trial and time. During the trial, median changes in T2V ranged from −0.031 to −0.024, while EDSS showed a median change of 0. The mean relapse rate per time point ranged from 0.12 to 0.21, and the mean rate of T2L ranged from 1.45 to 1.46.

Fig 2 presents the LRFs derived from 19 pairings (Table 1 and S2 Table). The figure focuses on the four combinations of the SEPs and CEPs of interest, considering both transformed count data (columns one and two) and non-transformed count data (column three). For the T2V - EDSS outcome combination, LRFs for WA1_1 and WA2_1 exhibit values above 0.2, but all upper bounds of the LRF 95% CIs remain below 0.5, indicating a lack of strong evidence for ILS. Across all 68 scenarios, we observed two weak clinically non-relevant signals for ILS. The instabilities observed in Fig 2 (blue asterisks) indicate that the distribution of the number of relapses given T2V (P [Number relapses | T2L]) does not always perfectly follow a Poisson distribution. Since ordinal models are generally more prone to numerical instability, our results suggest that models with Gaussian-Gaussian endpoint combinations appear more reliable. Nevertheless, the observed numerical instabilities had minimal impact, as LRF values were highly consistent across model families with and without instability, showing comparable results between Gaussian, Poisson, and ordinal models (compare columns 1 to 3 in Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. LRF derived from different distributional families (transformed count data).

https://doi.org/10.1371/journal.pone.0337893.g002

LRFs were computed for 19 trial arm combinations (intervention vs. control). Gaussian, Poisson, or ordinal models were employed to derive the LRFs. Results from the Poisson model family were analyzed from non-transformed data. Four combinations of SEP and CEP (rows) were considered: 1) T2 Volume – EDSS, 2) transformed T2 lesion count – transformed relapses, 3) T2 Volume – transformed relapses, and 4) transformed T2 lesion count – EDSS. The calculated LRFs are represented by a point with 95% confidence interval bars. Blue asterisks indicate convergence issues encountered during the derivation of the LRFs from the corresponding models. Notably, the content in column 3 of Fig 2 and column 4 of S9 Fig, both utilizing the Poisson distribution family, is identical. Furthermore, the combination of T2V and EDSS outcomes remain consistent between Fig 2 and S9 Fig. Abbreviations: LRF: Likelihood Reduction Factor; tr: transformed; EDSS: expanded disease status scale.

The results for original count data (S9 Fig) align closely with those obtained for transformed count data. Convergence issues arose, particularly for the combination of T2V and relapse count, when using models with negative binomial distribution family.

Discussion

We focus on the likelihood reduction factor (LRF) as a quality measure for individual-level surrogacy (ILS). Currently, surrogacy between MRI-derived disease activity markers (SEPs) and patient-relevant clinical disease outcomes (CEPs) is one of the central paradigms in the therapeutic decision-making process for people with remitting-relapsing multiple sclerosis (PwRRMS). With this work, we wanted to reproduce evidence for this established and recommended practice (see S1 Table).

We explored ILS in longitudinal independent individual-level data from 4773 to 5673 PwRRMS in three phase II and seven phase III trials. We could not find generalisable, robust scenarios with noteworthy ILS across trials (see Fig 2, S9, and S10 Figs). LRFs were mostly below 0.2 (Fig 2) indicating that in most examined scenarios less than 20% of the CEP variability is explained by the SEP. This means that the SEP provides relatively little information for predicting the CEP on the individual level, and such a level of explained variability may be insufficient to support informed treatment decisions. To provide an intuitive understanding of the LRF, Figs A and B in S1 Text may serve as illustrative examples. While data variability is low in Fig A and high in Fig B, this difference is reflected in the R² and LRF. Overall, the scenario shown in Fig A provides a more suitable basis for individual predictions than that in Fig B.

As shown in S7 Fig, this limited predictive power should also be interpreted in light of potential bias. While scenarios with transformed or non-normal endpoints tend to underestimate the true LRF (up to −0.104 for results corresponding to Fig 1), the bias is minimal when both SEP and CEP are approximately normal, and sample sizes are moderate to large (N ≥ 300). Accordingly, the LRF values for T2V and EDSS (Fig 2, row 3) are not biased, while values in other scenarios may be underestimated. Even so, LRFs remain low. Overall, the results suggest that Gaussian-Gaussian endpoint combinations provide unbiased and less numerically unstable estimates compared to Poisson or ordinal models.

Another question is, if there is an appropriate cut-off point to differentiate between strong and weak ILS? In oncology, IQWIG (the German Institute for Quality and Efficiency in Health Care) has defined LRF values below 0.50 as being clinically irrelevant [43]. However, future research has to establish a clinically meaningful LRF threshold to differentiate between clinically relevant and irrelevant ILS for PwRRMS.

First, imagine a simple 2 × 2 table where 20% of the people have an unfavorable SEP. In this example, the PPV is 63% and the NPV is 90%, similar to cases reported by Rio et al. [15]. In this situation, the LRF is around 0.30, indicating a moderate reduction in uncertainty when predicting outcomes based on SEP. Second, consider how SEP is related to the CEP. Suppose CEP at time t depends on α times the SEP plus some random variability: representing a regression model, where α is the effect of SEP on CEP and σ is the variability of . If the variability is small compared to , the SEP is informative, and the prediction is more precise. If the variability is large, SEP explains little about the clinical endpoint and prediction quality is low (see sections 1b and 4 in S1 Text). For example, with four time points, the LRF can be calculated as , where is the ratio of the SEP effect to its variability. An LRF of 0.2 corresponds to a high variability , while LRF = 0.5 corresponds to a smaller variability . Third, consider a logistic regression model predicting the probability of an unfavorable clinical endpoint based on SEP and additional patient information X. Examples in sections 3a and 3b in S1 Text show that ORs of predictors between 2 and 5 result in an LRF values of about 0.25, illustrating that even moderately strong predictors on the population-level only partially reduce uncertainty in individual predictions.

Results based on the whole population or subgroups are very different and must be considered differently. Our ILS assessment of ten independent clinical trials gave low LRF values. We showed that in the MS literature on individual prediction, mainly population-wide association measures were used. In most cases significant PTEs, ORs, and HRs (S1 Table) were reported instead of metrics suited for assessing the quality of predictions at the individual level, such as PPVs, NPVs, calibration, correlation, or the LRF. The unsuccessful search for population-wide prediction models may be explained by our weak ILS signals of the predictors used to forecast specific RRMS clinical outcomes. To avoid being misguided in their search for prediction models by too optimistic and inappropriate ILS statements, we recommend that predictive research in the field of MS focus on state-of-the-art ILS quality measures to assess settings on which future individual prediction models may be built. The use of PTE or the Prentice Criteria is misleading and can distort ILS assessments. See sections 1 and 4 in S1 Text or elsewhere [6,38].

The Prentice criteria and the PTE became more prominent. Two studies [20,21] assessed PTE in PwRRMS treated with interferon-beta-1 for the accumulated T2L values as SEP and the cumulative relapses as CEP (aggregated associative): PTE = 0.53 (0.28–1.01). Investigating first-year change in T2L as SEP and the cumulative relapses within the second year of treatment (aggregated predictive setting) resulted in PTE of 0.8 (0.34–1.86). Our data (data set W2_1, see Table 1) gave a PTE of 0.58 for the aggregated associative setting and PTEs below 0.8 for the aggregated predictive analysis (S10 Fig). By using comparable designs within the available trial data, we were unable to replicate these findings which have been seminal for ILS understanding in RRMS.

The analyses of prediction research that exist to date obviously influence guidelines and their application in everyday clinical practice. For example, the MAGNIMS guidelines recommend specific high-quality MRI T2L scans to monitor MS disease activity since T2L should have a high prognostic power for disease activity [1,2]. The CMSWG also recommends T2L as an indicator of treatment non-response, informing treatment decisions [1–4]. In a survey with US American neurologists, 97% (monitoring of RRMS within 12 months) and 67% (treatment switch to intravenous treatment, when ≥2 T2L occurred) of the neurologists follow these recommendations [44].

MAGNIMS and CMSWG recommendations claim a high predictive ability of T2L on relevant CEPs, based on various methodologies (see background). However, MAGNIMS and CMSWG also report a few predictive values which can be considered as ILS information. S1 Table provides a summary of our assessment.

From a clinical perspective, the utility of the LRF quantifying individual-level SEP quality can be illustrated in the following example: Consider a screening test with low sensitivity but high specificity. In this case, positive results are generally reliable (high specificity - > low false positives), but many true cases remain undetected, meaning the test alone is insufficient to guide decisions for all individuals. In a screening context, false positives can be clarified through subsequent testing before any intervention. In contrast, when a therapeutic decision is made directly based on this initial test, there is no opportunity to correct for misclassifications, which underscores the need for robust and reliable individual-level SEPs to guide treatment decisions. However, current evidence on T2L as a SEP detecting for treatment failure is frequently assessed using metrics that are not appropriate for prediction quality. Additionally, if a treated PwRRMS shows an unfavorable SEP and a treatment switch is recommended, it remains unclear whether the new therapy will be beneficial. There is uncertainty whether an unfavorable CEP will result from the unfavorable SEP and if the person responds to the new treatment. We found no guidelines discussing a benefit/risk trade-off for this decision-making process where information about the PPV of the SEP has to be combined with models that predict the best therapy to be switched to [45].

Conclusions

We critically discussed methodologies commonly used in prediction research in the field of MS to claim ILS quality and explored ILS between MRI-derived markers and clinical outcomes within RRMS cohorts using independent individual-level data from ten MS trials. We examined ILS for T2L measurements and found weak signals in some trials, but no stable, generalisable ILS patterns between T2L as SEP for disease activity and disability progression in RRMS. To quantify ILS, we introduced the LRF as a reliable and generalized ILS quality measure for normally distributed longitudinal data. In some scenarios where endpoints followed a Poisson distribution, were ordinal, or had been transformed, the LRFs were underestimated by up to 0.104. However, this did not affect the overall interpretation of the LRF values derived from the ten clinical trials.

Supporting information

Acknowledgments

This publication is based on research using data from “Sanofi”, “Roche” and “Novartis” that has been made available through Clinical Study Data Request. We thank anonymous reviewers of the trials’ sponsors for their helpful and constructive comments.

Consent for publication

Before the publication, the sponsors of the trials reviewed the manuscript and consented with its publication.