Introduction

Benchmarking an observational analysis against a randomized trial increases confidence in the use of observational data to complement inferences made in trials [1]. If one can first replicate the results of a randomized trial using observational data, clinicians and other decision makers may be more confident in the use of these data to address questions not answered by the trials, e.g., the effect of treatment in populations excluded or underrepresented in the trials [2–4]. And, when benchmarking fails and observational data cannot reliably estimate similar results to a trial, we enhance the understanding of the limitations of observational data to answer causal questions.

Until now, few examples of benchmarking have been within oncology [5,6]. However, benchmarking an observational analysis against a trial of a cancer treatment poses a unique set of challenges. First, composite outcomes such as disease-free survival are commonly used, which can be hard to define in observational data. Second, correct alignment of time zero in sequences of treatment (e.g., surgery and first-line treatment) is crucial, as treatment delays may lead to a worse prognosis [7]. Third, intended treatment strategies are not always clearly distinguishable at baseline in observational data, thus the risk of selection bias due to immortal time is high without appropriate study design considerations (i.e., cloning) [7].

Here, we design a target trial [2,4,8,9] with a protocol as similar as possible to the B-31 and N9831 trials [10–12], which estimated the effect of adjuvant trastuzumab plus chemotherapy compared with chemotherapy alone on disease-free and overall survival in individuals with early human epidermal growth factor receptor 2 (HER2)-positive breast cancer. We then emulate the target trial to understand if we can undertake an observational analysis that successfully replicates estimates of the effect of trastuzumab on both overall and disease-free survival using the Swedish National Quality Register for Breast Cancer and the routinely collected data from the Swedish population-based registers.

Methods

Index randomized trials: B-31 and N9831

The protocol used for benchmarking is outlined in Table 1. It essentially merges the B‑31 and N9831 trials (S1 Table), as they were similar and analyzed in a joint analysis approved by the Food and Drug Administration and the National Cancer Institute before any data cut-off [10–14]. Both trials were open-label, multicenter, randomized, controlled clinical trials in individuals with early HER2-positive breast cancer who underwent surgery for complete resection of the primary tumor (Table 1). Participants in both studies were randomly assigned to receive either an initial anthracycline-based combination chemotherapy for 12 weeks followed by a taxane-based chemotherapy plus trastuzumab, or the same chemotherapy regimen without trastuzumab. The primary endpoint was the composite of disease-free survival consisting of the earliest of (1) local or regional recurrences, (2) distant recurrences, (3) contralateral breast cancer, (4) other second primary cancer, or (5) death from any cause. Secondary endpoints were the individual components of the composite outcome.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of protocol used for benchmarking, the target trial and its emulation to compare trastuzumab plus chemotherapy with chemotherapy, NKBC and seven further Swedish registers, 2008–2015.

https://doi.org/10.1371/journal.pmed.1004661.t001

After approximately five years in each study (B-31: February 2000 to February 2005; N9831: May 2000 to March 2005), an intention-to-treat analysis was performed [12]. A total of 3,676 individuals were randomized, of whom 3,351 had at least one follow-up examination. 1,672 individuals were assigned to the trastuzumab plus chemotherapy arm (B-31: 864; N9831: 808) and 1,679 to the chemotherapy only arm (B-31: 872; N9831: 807). The 5-year disease-free survival proportion was 92.0% (1539/1672) in the trastuzumab group and 84.5% (1418/1679) in the no trastuzumab group (hazard ratio [HR]: 0.48, 95% confidence interval [CI] [0.39, 0.59]). Overall survival proportion at 5 years were 96.3% (1610/1672) and 94.5% (1587/1679), respectively (HR: 0.67, 95% CI [0.48, 0.93]).

The observational analysis

Results

Fig 1 depicts a flowchart of patient selection for the target trial emulation. The baseline characteristics of all 1,578 eligible individuals are shown in Table 2. All but 308 individuals were assigned to both strategies at baseline (i.e., they were only given chemotherapy, and it was not clear if they would go on to be given trastuzumab). Compared with those in the B-31 and N9831 trials, individuals in the target trial emulation were older, but slightly fewer had a tumor larger than 5 cm in diameter. For the overall survival analysis, there were 159,972 and 61,733 person-weeks of follow-up in the trastuzumab plus chemotherapy and chemotherapy arm, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Baseline characteristics of individuals included in the emulation of a target trial comparing trastuzumab plus chemotherapy with chemotherapy, NKBC, and seven further Swedish registers, 2008–2015.

https://doi.org/10.1371/journal.pmed.1004661.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of patient selection into the emulation of a target trial comparing trastuzumab plus chemotherapy with chemotherapy, NKBC, and seven further Swedish registers, 2008-2015.

https://doi.org/10.1371/journal.pmed.1004661.g001

Table 3 shows the estimated survival, RDs, and RRs adjusted for baseline and time-varying variables, and Figs 2, S1, and S2 show the inverse probability weighted survival curves for all outcomes. For the primary analysis 74.7% (95% CI [70.7%, 78.5%]) were disease-free and alive at year 5 in the trastuzumab plus chemotherapy group and 53.5% (95% CI [46.6%, 60.3%]) in the chemotherapy group, resulting in an RD of −21.2% (95% CI [−29.2%, −13.8%]) and an RR of 0.54 (95% CI [0.44, 0.67]). S5 Table shows the distribution of final inverse probability weights.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Survival, risk differences, and risk ratios at 5 years from baseline estimated in the observational emulation of a target trial comparing trastuzumab plus chemotherapy with chemotherapy, NKBC, and seven further Swedish registers, 2008–2015 (main analysis based on 1578 individuals).

https://doi.org/10.1371/journal.pmed.1004661.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Inverse-probability weighted survival curve of disease-free survival to compare adjuvant chemotherapy + trastuzumab with chemotherapy from an observational emulation of a target trial, NKBC, and seven further Swedish registers, 2008–2015 (main analysis based on 1,578 individuals).

https://doi.org/10.1371/journal.pmed.1004661.g002

Results for the individual components of disease-free survival were as follows. Overall survival at 5 years was 90.7% (95% CI [88.1%, 93.2%) in the trastuzumab plus chemotherapy group and 67.5% (95% CI [60.1%, 74.0%]) in the chemotherapy group, resulting in an RD of −23.2% (95% CI [−30.8%, −16.0%]) and an RR of 0.29 (95% CI [0.20, 0.40]). The 5-year distant recurrence free survival was 80.7% (95% CI [76.9%, 84.1%]) in the trastuzumab plus chemotherapy group and 73.8% (95% CI [68.2%, 79.0%]) in the chemotherapy group, resulting in an RD of −6.9% (95% CI [−13.6%, −0.9%]) and an RR of 0.74 (95% CI [0.56, 0.96]). There were few events for the remaining components of disease-free survival (local recurrence, contralateral breast cancer, other second primary cancer). All results from sensitivity analyses were similar to the main analysis (see S6, S7, and S8 Tables, and S3, S4, S5, S6, S7, and S8 Figs).

Discussion

We used routinely collected data from Swedish population-based registers to emulate a target trial that asked a similar clinical question to the B-31 and N9831 randomized trials. Both the trials and observational analysis estimated similar effects of trastuzumab on disease-free survival and overall survival in women with HER2-positive breast cancer, i.e., a clear increase in survival. Data from both the trials and observational analyses, therefore, support the recommendation of trastuzumab.

Symmetry of conclusions between our observational analysis and the index trials, i.e., successful benchmarking, increases confidence in our observational analysis. However, this does not mean there are no differences between the studies. In fact, several differences affecting estimates in different directions may average out creating a false impression of successful benchmarking. Therefore, we will consider important differences and their potential impact.

A central challenge was to define the individual cancer-specific components of the disease-free survival outcome in the available routinely collected data using definitions similar to the trials. Consider the following two examples: (1) we defined distant recurrence using International Classification of Diseases, Tenth Revision, Tenth Revision (ICD-10) codes that capture secondary cancers, which, in fact, could be a local, regional or distant recurrence; and (2) we defined local recurrences using surgical procedure codes, meaning a local recurrence that was not removed is missed. As there are several unavoidable misclassification problems with these definitions of local and distant recurrences, the survival estimates for trastuzumab of these specific outcomes are difficult to interpret. However, for the composite outcome of disease-free survival, the effect of this misclassification is less problematic. This is because misclassification occurs between the individual components (i.e., capturing local recurrence instead of distant recurrence) or a missing diagnosis for one component could be captured in another component (i.e., missing local recurrences can be captured within the distant recurrence definition). However, the absolute risks for disease-free survival may be lower if any diagnoses of the cancer-specific components are not recorded in the registers.

A differential distribution of treatment effect modifiers between the trial and observational analysis could also be a reason for different estimated effects. In our observational analysis, individuals were, on average, older than those in the trials, and the proportion of individuals with lymph node involvement was not comparable between studies. However, it is unlikely that this contributes to a meaningful difference between effect estimates for disease-free survival as it has been suggested that the effect of trastuzumab on recurrence risk does not differ by age or nodal status [33]. Notwithstanding, an older age is often accompanied by further comorbidities, which we might not have captured in our analyses, and could lead to a poorer response to treatment and/or more treatment-unrelated deaths than in the randomized trials, explaining the more pronounced absolute risk estimates for overall survival.

As with all observational analyses, there is a risk of residual and unmeasured confounding. Our analysis required accounting for both baseline reasons individuals would or would not receive trastuzumab, and post-baseline factors associated with adhering to the assigned treatment strategy (i.e., whether or not an individual initiated trastuzumab over follow-up). We relied on diagnoses of clinical events and drug use to account for this potential confounding. Ideally, we would also use information on biomarkers (e.g., laboratory or radiology parameters), as changes in these recordings could be a reason someone would not receive trastuzumab, but such test results are only routinely captured in the register data for selected conditions. Also, we would need information on treatment discontinuation and dose modifications, which is not captured in the registers. However, if we consider the null result for other second primary cancers as a negative control to assess the impact of potential unmeasured factors (i.e., confounding), there is enhanced certainty that cancer-specific outcomes are not affected.

One notable result from our observational analysis is the large estimated effect of trastuzumab on overall survival, which is considerably larger than the effect estimated in the trials. This difference could be a consequence of unmeasured confounding in the observational analysis, i.e., there were factors associated with overall survival (but not particularly associated with disease-free survival) not captured in the data, such as frailty, that were also reasons someone was withheld trastuzumab. However, there were very few non-breast cancer-related deaths, and the estimated effect on breast cancer-specific survival is similar to the effect on overall survival. So, it is likely that the effect estimates for overall survival reflect a difference in deaths due to the breast cancer.

Another factor contributing to the dissimilarity in the effect estimate for overall survival may be the estimation of the analog of the per-protocol effect in the observational analysis and the estimation of the intention-to-treat effect in the trials. If individuals assigned to the chemotherapy arm in the trials switched to additionally receive trastuzumab, this could lead to an attenuation of the effect estimate on overall survival in an intention-to-treat analysis. It is then plausible that the effect estimate derived from a per-protocol analysis would be larger than that obtained from an intention-to-treat analysis. As the trials did not conduct a per-protocol analysis, the comparison of effect estimates from the observational analysis and the trials should focus more on the direction and the definitiveness of the superiority of trastuzumab plus chemotherapy irrespective of the data source.

A limitation of the observational analyses was that there was 20% missing data for the start date of chemotherapy. To overcome this, we imputed the start date for those with missing data. Although such imputation relies on the untestable missing at random assumption, we also carried out a sensitivity analysis in which we excluded all individuals with missing data, which returned similar results. The consistency between these two approaches to deal with missing data enhances the certainty in our results.

Of note, the person-weeks of follow-up was higher in the trastuzumab plus chemotherapy arm compared with the chemotherapy arm. This is a consequence of more individuals receiving trastuzumab plus chemotherapy than chemotherapy alone (i.e., at week 15, there were 1,050 individuals in the trastuzumab plus chemotherapy arm and 518 in the chemotherapy only arm) and the survival being considerably higher in the trastuzumab group. But this is not expected to bias effect estimates, particularly because we adjust for time-varying factors related to censoring.

If investigators using routinely collected data can successfully benchmark analyses that estimate the effect of cancer treatments on both overall and disease-free survival against a randomized trial, it would increase confidence in the ability of these data to ask questions that could not be answered in the initial trial. For example, routinely collected data can provide evidence more relevant to everyday clinical practice situations. Individuals who are regularly underrepresented or excluded from randomized trials often represent a majority of affected individuals [34,35]; routinely collected data, therefore, provide a unique opportunity to explore heterogeneity in treatment effectiveness across subgroups.

We successfully benchmarked an observational analysis of trastuzumab on disease-free and overall survival against the B-31 and N9831 trials. By aligning protocols and using appropriate methodological approaches, we show that observational data can be used to estimate similar results as randomized trials of cancer treatments like trastuzumab. This opens the door to using observational data to complement results from randomized trials of cancer treatments which can provide quick, cheap, and robust evidence to support decision-making where trials leave evidence gaps.

Supporting information

Acknowledgments

We thank Mats Talbäck for his aid in data management.

The information reported in the manuscript was previously presented at the 2024 Conference of the Society for Epidemiologic Research.