The HISTORIC trial design

HiSTORIC was a stepped-wedge cluster randomized controlled trial enrolling consecutive patients with suspected acute coronary syndromes across 7 acute secondary or tertiary care hospitals in Scotland [3]. Cluster randomization of hospital sites was necessary to avoid the risk of clinical error attributable to simultaneous use of different diagnostic pathways. The trial was registered (www.clinicaltrials.gov; Unique identifier: NCT03005158), approved by the Scotland A Research Ethics Committee (REC number 12-SS-0115) and the conduct of the trial was periodically reviewed by an independent trial steering committee. We received approval to use routinely collected National Health Service (NHS) data for research without the need for individual patient consent. This approval was granted by the NHS National Services Scotland Privacy Advisory Committee (PAC), now called the NHS Scotland Public Benefit and Privacy Panel for Health and Social Care (NHSS HSC-PBPP). All data were collected from the patient record and national registries, de-identified, and linked in a data repository within secure National Health Service safe havens.

The trial design is illustrated in Fig 1. The trial consisted of three phases over a 24-month period—labelled “validation”, “randomisation” and “implementation”. Patients presenting during the validation phase were managed according to the established assessment protocol (standard care). This was followed by a 6-month randomisation phase where participating centres were randomised to cross over to use the early rule out pathway (intervention), with stratification by hospital size. However, the sites did not (or were unable to) adhere to the pre-planned randomised crossover time points (see comparison between “Planned” and “Actual” in Fig 1). As is common in pragmatic trials evaluating changes in care pathways, the actual date of implementation of the intervention differed from the planned date for clinical reasons that were out with the control of the investigators. These included a directive from one Health Board to implement earlier than planned to address capacity issues in the Emergency Department, and delays at other sites due to a change in the provision of ambulatory care services and a request for an additional governance review prior to implementation. A final “implementation phase” followed in which all sites used the early rule out pathway. Recruitment was continuous with patients experiencing either the standard care pathway (control) or early rule out pathway (intervention), but not both. Within the typology schema of Copas et al. [5], this trial has a “continuous recruitment with short exposure” design. Patients were included in the study on their first attendance with those who reattended during the study period not enrolled twice.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram showing planned and actual implementation of the intervention (the HighSTEACS pathway).

https://doi.org/10.1371/journal.pone.0271027.g001

As a stepped wedge trial, the study design allowed both (i) horizontal before-and-after comparisons of outcomes (i.e. comparisons within rows/sites in Fig 1), and (ii) vertical cross-sectional comparisons based on contemporaneous controls (i.e. comparisons across sites for patients presenting at the same or similar times). Hereafter for simplicity, we refer to these as “horizontal” and “vertical” comparisons, respectively.

In this study, the clusters were seven individual sites, which were randomized to begin the intervention at three distinct time points. In the actual study design (Fig 1), although all sites implemented the early rule-out pathway during the randomisation phase, four of the sites crossed over in the first cross over point, one in the second, and two in the final cross over point.

We have previously described this trial as a “hybrid before-and-after and stepped wedge design” [6].

The HISTORIC trial used routinely collected data to identify, characterise and follow up all patients presenting at the Emergency Department and record study outcomes. Patients did not individually consent to participate in the HISTORIC trial. The primary and secondary safety outcomes were adjudicated as previously described [3]. Source data sets were linked together within a secure Safe Haven environment (the Public Health Scotland National Safe Haven [7]), and release of outputs was tightly controlled to prevent release of sensitive or confidential patient-level information. Exporting raw data or details about specific events was prohibited under patient confidentiality and data protection regulations in case this identified individual patients. Raw datasets were only accessible by approved individuals who had undertaken the necessary governance training.

Pre-specified statistical analysis methods

A linear mixed-effects regression model was fitted to the efficacy co-primary outcome of length of hospital stay, adjusting for hospital site, season, time of patient presentation since start of study (in days), and an indicator variable for whether the new intervention pathway was implemented or not. The reason for adjusting for season in the model is that there is known to be an increased risk of myocardial infarction (and death from myocardial infarction) in the winter months [8]. Season featured in the model as a four category factor variable, with three indicator dummy variables for Spring (March April May), Summer (June July August) and Autumn (Sept Oct Nov); relative to the reference category Winter (Dec Jan Feb). Time of patient presentation since start of study appeared as a linear term in the model, centred on the mid-point of the study enrolment period. Hospital site was included as a random effect. A compound symmetry variance-covariance structure was assumed.

A logistic mixed-effects regression model was fitted to the safety co-primary outcome, using an adaptive Gauss-Hermite quadrature method to approximate the log-likelihood [9]. As for the efficacy outcome, this was adjusted for hospital site (as a random effect), season, time of patient presentation since start of study (in days), and an indicator variable for the intervention.

In the pre-specified primary analysis, data were analysed according to the actual timing of the introduction of the new pathway rather than the randomised time the new pathway was to be introduced. Pre-specified sensitivity analyses involved various modifications of the primary analysis model and analysis population, details of which are presented in Table 1. Additional post-hoc sensitivity analyses were used to assist interpretation of the findings from the pre-specified analyses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of all pre-specified primary and secondary analysis models.

https://doi.org/10.1371/journal.pone.0271027.t001

Statistical analysis methods

Table 1 describes the different pre-specified models fitted, as per the original statistical analysis plan. A key pre-specified secondary analysis (analysis B in Table 1) involved comparing the validation and implementation phases only of the trial in a before-and-after study context, where the validation and implementation phases were calendar matched to avoid any bias due to a seasonality effect. The rationale for this before-and-after analysis was that it enabled us to compare the intervention once it had been fully embedded into the Emergency Department with standard care. It was anticipated that during the randomisation phase of the trial the effect of the intervention may have been diluted since clinicians might still have been learning to adopt the new procedure. However, this analysis assumes that there were no natural changes in the safety outcome over the evaluation period, which is a reasonable assumption given the study population primarily consists of patients who have myocardial infarction ruled out and receive few if any specific treatments.

For post-hoc analyses, our model choices were informed by the stepped wedge literature [10–15] and with consideration given to our chosen study design (see Table 2). These included using different ways of modelling the time trend (e.g. as cubic spline or categorical variable) and adjusting for the length of time that the site had been using the intervention pathway (exposure time). Note that for all post-hoc models of the binary safety endpoint we used the Laplace approximation to increase the speed of convergence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of all post-hoc advanced primary and secondary analysis models.

https://doi.org/10.1371/journal.pone.0271027.t002

In addition to the above models, there was a need to control for any natural changes in key demographic variables across the duration of the study. This was at least partly motivated by a peer reviewer of the primary trial report who suggested that adjusting for potential confounders through using “propensity scores either through matching or weighting would strengthen the study substantially.”

Therefore, in all pre-specified and post-hoc models, we performed covariate adjustment to attempt to control for any shift in patient characteristics over time, and we did this using four alternative methods of adjustment listed in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of adjustment methods with justification.

https://doi.org/10.1371/journal.pone.0271027.t003

Data augmentation

The event rate for the binary safety outcome was only 0.36% overall, and so there was a concern that sparse data bias may have contributed to the divergent results. We therefore considered methods to address this problem. One of the methods considered was a data augmentation approach as suggested by Greenland et al. [21]. We adapted this method for the context of stepped wedge trials by first generating synthetic data to encode our prior information, and then combining this synthetic data with the real HISTORIC data. For generating the prior data, we fixed the number of events before and after the cross over point in each site (and across all sites) to be equal, and the proportion of events was simulated based on a Bernoulli distribution with pre-specified probability of 0.4%, which was the event rate assumed in the original sample size calculations. Within each site, the dates of presentation (days since start of study) were fixed to be one day before and after the crossover point. The rationale for choosing these dates of presentation is that prior research suggests that cluster-periods closest to the treatment switch date are most informative [22, 23] and therefore we conjecture that the effect of sparse data would be more acute at the crossover points. We also found it easier to generate the necessary priors using these dates of presentation. The amount of prior information (i.e. number of simulated patients) was calibrated to produce an absolute risk difference of approximately 0%, and 95% CI within the bounds of -2% to 2%. This means that we are hypothetically assuming a priori that it is very unlikely that the absolute risk difference would exceed 2% in either direction. This prior information was then combined with the real data to observe the effect on the intervention effect estimates. We then fitted the primary analysis model (a generalised linear mixed effect model utilizing an adaptive Gauss-Hermite approximation to the log-likelihood) without intercept term, adjusting for the type of data (i.e. we specified a binary factor variable called “synthetic” taking the value 1 for synthetic data and 0 otherwise). A dramatic change in the results compared to the original analysis would have indicated the presence of sparse data bias.

Examination of the raw data

As a post-hoc analysis, the raw safety endpoint data was examined within the secure Safe Haven analysis platform. This is consistent with the recommendation of Greenland et al. [21] to perform “tabular examination” of basic data to identify sparse data problems. Safety outcome data was categorised into monthly periods per site, called “cluster periods”, and the data shown in a table. We also tested the sensitivity of the results to small changes in the numbers of events per cluster-period.

As indicated in Table 1, Nickless et al. [13] recommend adjusting for exposure-time in stepped wedge analyses. This was a particular concern in our trial because the long before and after periods made it difficult to precisely estimate the intervention effect independently of any natural overall time trends or local exposure time effects using the standard model. We therefore also visually examined the raw data for any obvious exposure-time effects.

Software

R software version 3.6.1 was used to conduct all analyses [24]. Statistical models were fitted using the lme4 package [9] and forest plots constructed using the metafor package [25].

Data augmentation

After performing data augmentation, the odds ratios for the primary analysis model reduced slightly from 1.97 (95% CI 0.95 to 4.08) down to 1.76 (95% CI 0.96 to 3.25), based on a synthetic prior data with a sample size of N = 1274 imaginary patients and risk difference (RD) of 0% (95% CI -0.71% to 1.99%). For a highly specific prior of N = 3220 (RD 1, 95% CI -0.45% to 0.94%), the odds ratio reduced to 1.56 (95% CI 0.92 to 2.64). Note that the number of sites and length of the randomisation phase remained unchanged in this analysis.

Examination of the raw data

Fig 4, which was presented in the supplementary material of Anand et al. [3] and reproduced here, shows how the binary safety outcome changed over the duration of the trial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Event rate for the primary safety outcome of myocardial infarction or cardiac death at 30 days by month of presentation from the start of the trial.

https://doi.org/10.1371/journal.pone.0271027.g004

The safety outcome event rates were 0.5% (49/10724), 0.4% (37/9336), and 0.2% (27/11432) for the first, second and third phases of the trial respectively, corresponding to the validation, randomisation, and implementation phases of the trial, respectively.

It was informative to further examine the raw safety data after splitting the number of safety events (type 1 or type 4b myocardial infarction or cardiac death) by cluster-period. Here we let each cluster-period be defined as each site and month combination, so this means we split the number of safety events by site and by month.

After examining the raw safety endpoint data, we discovered that results were strongly driven by the safety outcome values of one or two sites under the intervention condition in the randomisation phase. Although it was not possible to export a full table of raw safety endpoint data to show in this article in case this identified individual patients, we are able to give some limited details below.

The median number of events per month at each site (cluster-period) in the randomisation phase was 1. However, at one site in one month during the randomisation phase, the number of events was notably higher (7 events) than in all other cluster periods, but still within the realms of random chance. It was instructive to assess the sensitivity of the results to the number of events in this cluster period. After reducing the number of events from 7 down to 1 (the median), and repeating the primary analysis, the odds ratio was reduced from 1.97 (95% CI 0.95 to 4.08) to 1.20 (95% CI 0.56 to 2.57). This is a substantial change after modifying the events from a single site during a single month of the trial.

We then investigated: “How many events do we need to subtract from those occurring under the intervention condition in the randomisation phase in order to reverse the direction of the odds ratio?” In the HiSTORIC trial, there was a 0.3% event rate under the intervention condition compared to a 0.4% event rate under the standard care condition. In contrast, when considering the randomisation phase only, we observed a 0.6% event rate (29 events out of 5248 presentations) under the intervention condition; compared to a 0.2% event rate (8 out of 4,088 presentations) under the control condition. If hypothetically the event rate under the intervention condition were 21 out of 5248, the results would have been: OR 1.00 (95% CI 0.46 to 2.16). Therefore, eight fewer events would have been needed to switch the direction of the odds ratio from 1.97 to just below 1.

Regarding possible exposure time effects, there was no clear evidence of changes in the event rate over time after implementation of the intervention within each site when examining the raw data.

Results interpretation

Post-hoc results from extensive exploratory analyses of the HiSTORIC trial data are presented in this article. These results have confirmed the findings presented in the main trial paper [3]. In particular, we have confirmed the divergence between the calendar-matched and other analyses, even after utilising three distinct covariate adjustment methods.

Aside from the calendar-matched and “as-randomised” analyses, the length of stay outcome results were very similar regardless of the model specification or method of covariate adjustment. Although the calendar-matched analysis gave different point estimates and confidence intervals, our overall conclusion about the effectiveness of the intervention in reducing length of stay was unchanged. There was strong evidence of a reduction in length of stay based on the new pathway.

The binary safety endpoint results exhibited a similar pattern, except that results were less robust to the model specification and varied slightly more across the different methods of covariate adjustment. In general, the IPTW method provided more precise results (narrower confidence intervals) compared to the other methods, and regression adjustment for propensity score was not always equivalent to standard covariate adjustment.

There was a clear difference observed between the “as-randomised” and primary analysis results for both the efficacy and safety outcomes. Whilst all sites implemented during the 6-month randomisation phase, some implemented one or two months earlier and others later than planned due to local site pressures. Therefore the “as randomised” sensitivity analysis for the efficacy outcome measure is less helpful because the duration of stay will remain unchanged if the pathway has not actually been implemented, thereby diluting the intervention effect. Nevertheless, the magnitude of the difference observed between the primary analysis and the “as randomised” analysis for the safety outcome is potentially informative and suggested that the primary statistical model was overly influenced by data near the cross-over points or by the pattern of the cross-over points.

The raw safety outcome data showed no clear evidence of changes in the event rate over time after implementation of the intervention within each site. This appears to contradict the fact that after adjusting for exposure time in the regression models, the odds ratio point estimate (and 95% CI) reduced dramatically to be closer to 1. However, these results may be due to the cluster-period event count of 7 leading to a raised event rate in the randomisation phase and subsequent lower monthly event rates rather than any real exposure time effect. The event rate was only 0.2% in the third (implementation) phase, which was lower than the 0.6% event rate observed in the randomisation phase for patients recruited under the intervention condition. The exposure time model may therefore be overfitting the data.

Sparse data can in general lead to issues with model fitting and parameter estimates which are biased away from the null [21]. We explored whether a modified data augmentation method could be used to address this possible bias. However, the data augmentation method suggested that the effect of sparse data alone on the primary analysis model was insufficient on its own to explain the divergent results. The odds ratio point estimate and confidence intervals remained dissimilar to the calendar matched analysis results even after data augmentation.

Evaluation of the study design

The HiSTORIC design was unusual in that it included a small number of sites and short duration of the randomisation phase. With a small number of sites, we are vulnerable to any confounding bias occurring within a single site, and this may have affected the results. Furthermore, the long before and after periods in the HiSTORIC trial meant that there was limited opportunity for contemporaneous comparisons as less than a quarter of the total trial duration covered periods in which sites were in both the control and intervention conditions. However, as the patient population is stable and no other changes in clinical practice for the assessment or treatment of patients without myocardial infarction were introduced during the study period it is unlikely that comparisons before and after the intervention was implemented are confounded by time varying trends.

Evaluation of the statistical modelling approach

We performed a simulation analysis of the binary safety outcome (see S1 File) which showed no evidence of systematic bias in the intervention effect and no evidence of an inflated Type 1 error rate due to the very low number of clusters. This finding differs from Taljaard et al. [26], possibly because the between-cluster variability in our trial was effectively zero and the number of patients per site (cluster size) was so large. Furthermore, our choice of primary analysis model is supported by the literature; generalized linear mixed model (GLMM) analysis is the recommended approach for dealing with binary outcomes with a small number of clusters [27]. The use of Generalised Estimating Equations (GEE) with small sample correction has also been recommended in the literature [28, 29]. However, the models were either extremely slow to converge or failed to converge completely in R software, even after fitting a very simple GEE model with exchangeable correlation structure (and no small sample correction). Our problems with fitting the GEE models were presumably due to the very large cluster sizes.

Horizontal versus vertical bias

The avoidance of “horizontal bias” in stepped wedge trials in the form of adjusting for secular changes over time is strongly emphasised in the literature. In particular, under-specification of the secular trend will lead to bias in the intervention effect estimates [12]. For stepped wedge trials with high numbers of sites (i.e. over 50), bias across sites within cluster periods (i.e. “vertical bias”) is less of a concern because any such bias is likely to have lower influence on the intervention effect estimate. However, we would suggest that “vertical bias” is likely to be of a much greater concern in stepped wedge trials of binary outcomes with a low number of clusters, particularly if the outcome event rate is low.

In the HISTORIC trial, we believe that the vertical comparisons in the randomisation phase and binary event rates near the cross-over points were highly influential for the primary analysis method. Our examination of the raw data supports this to some extent. Only eight fewer events were needed to switch the direction of the odds ratio from 2 to below 1 in the primary analysis method. It is possible that the reasons why the sites did not conform to the “as randomised” cross-over points affected the binary event rates near the cross-over points and introduced confounding bias. Indeed, this is perhaps likely as some sites implemented the early rule-out pathway early to address over-crowding in the Emergency Department at the start of the winter period.

Research by Kasza, Forbes and Taljaard [22, 23] suggests that the information content is not equal across cluster periods and that cluster-periods closest to the treatment switch date are most informative and contribute the most to the statistical analysis. Our concern was that the primary analysis model, although valid in the general sense, was too sensitive to any bias or random chance differences occurring near the crossover points. Specifically, our primary analysis model may have over-adjusted for unreal time effects in the short period where both the standard care and intervention conditions were running.

An advantage of the calendar matched before and after analysis is that it does not use vertical comparisons (which may be affected by confounding bias), and avoids the problem of attempting to model both the secular time trend and intervention effect based on a low event rate. We accept however the limitations of the calendar-matched analysis such as the potential for confounding by natural secular changes over time and reduced sample size. Nonetheless, the calendar-matched analysis was particularly suitable as a sensitivity analysis in the HiSTORIC trial due to the long before and after periods and small number of clusters in the design.

Strengths

A strength of the study is that the analyses were based on a very large unselected population, with complete enumeration within each cluster over the study period and very little missing length of stay outcome data (0.05%). Identification and recruitment bias [30] was very unlikely in this trial because although clinicians at each site would have been aware of the control/intervention status within each site, the inclusion criteria was very broad (consisting of all patients presenting at emergency departments with suspected acute coronary syndrome) and the HiSTORIC screening tool was embedded into the patient record [3]. In addition, patients did not individually consent to the study and so there was virtually no prospect of the patients self-selecting which intervention/control group to be recruited into, or indeed patients influencing outcomes due to knowledge of the intervention/control group.

Limitations

The additional analyses presented in this article were all post-hoc and were conducted after the main HiSTORIC trial analyses. They therefore should be considered as purely exploratory and supportive of the findings shown in the main trial report [3]. We also report on a few experimental methods such as “data augmentation” and “adjustment for propensity score”. Although adjustment for propensity score is a widely used method in clinical research, it has not to our knowledge been previously applied in the context of stepped wedge trials. Further research is needed to evaluate the performance of these methods in this context. In addition, the limitations mentioned in the main HiSTORIC trial report also carry forward into this study. In particular, the standard care arm of the HiSTORIC trial used a serial testing strategy based on the time of onset of symptoms, rather than a fixed time point 3 to 6 hours from presentation, which is more commonly used in other countries [3]. Furthermore, the statistical models were fitted using the lme4 package [9] in R software [24], which makes the assumption of a simple correlation structure of compound symmetry for linear mixed effects models (i.e. no autocorrelation in model residuals). Although we attempted to fit models assuming autoregressive or general (unstructured) correlation using the nlme package [31], the models would not fit, presumably for the same reason as for the GEE model: the within-cluster sizes were too large. Nevertheless, we believe that compound symmetry is a reasonable assumption given that independent individuals attending emergency departments within each site may be expected to be approximately equally correlated (or uncorrelated) with each other conditional on the fixed effects including season and time of presentation. Indeed, a few of the models fitted include sophisticated modelling of the time trend (e.g. via splines), which we think makes the compound symmetry assumption even more plausible because residual autocorrelation due to incomplete modelling of the time trend is less likely.