Introduction

Clinical research can only benefit patient and population health if findings are incorporated into routine care. There are delays and inappropriate variations in uptake of effective treatments and withdrawal of less effective or harmful treatments [1]. This translation gap is important to policy makers, healthcare systems, and research funders because it limits the health, social, and economic impacts of clinical research [2].

United Kingdom primary care presents particular implementation challenges: growing demand, increasing complexity of care, and limited workforce capacity, against a background of frequent organisational reconfigurations [3,4]. We identified 107 clinical guidelines produced by the National Institute for Health and Care Excellence (NICE) relevant to UK general practice [5]. Many implementation studies focus on a single clinical condition or problem (e.g., diabetes, antibiotic stewardship), limiting generalisability, as it is uncertain whether an implementation strategy developed for one condition will work for another. It is impracticable and inefficient to devise an implementation strategy for every new guideline. Furthermore, the clinical significance of behaviours often targeted in implementation studies, such as receipt of processes of care or investigations, is doubtful [6]. Implementation strategies are required, capable of integration into primary care systems and adaptable to different clinical priorities.

We earlier derived evidence-based indicators that could be measured using routinely collected data [5]. We found marked variations in indicator achievement and scope for improvement in a sample of 89 general practices [7]. We subsequently focused our efforts on four ‘high impact’ indicators that would benefit population health if implemented more consistently: achievement of recommended treatment targets for all of haemoglobin A1c (HbA1c), blood pressure (BP), and cholesterol in type 2 diabetes [8]; avoidance of risky prescribing of nonsteroidal anti-inflammatory drugs (NSAIDs) and anti-platelet drugs [9]; achievement of recommended BP levels in patients at high risk of cardiovascular events [10]; and anticoagulant prescribing for stroke prevention in atrial fibrillation (AF) [11]. We drew upon systematic reviews of implementation strategies [6,12–14], theory-guided interviews with primary care staff [15], and workshops with health professionals and patients to develop a multifaceted implementation package [16]. We compared the effects of the adaptable implementation package against implementation control on adherence to four different evidence-based high impact indicators.

Methods

Randomisation and masking

The trial statistician performed the two-stage randomisation using a bespoke computer-generated minimisation programme implemented in R, software version 2.15.2 (R Foundation for Statistical Computing, Vienna, Austria) (incorporating a random element). First, practices were stratified by CCG and list size, and randomised to Trial 1 (implementation packages targeting either diabetes control or risky prescribing), Trial 2 (BP control or anticoagulation in AF), or no intervention (80:64:34). As recruitment exceeded expectations, we included a ‘no intervention’ arm to allow us to evaluate any nonspecific effects of research participation (this paper presents comparisons within Trials 1 and 2; analysis of the no intervention group will be reported elsewhere). Secondly, within each trial we stratified practices by CCG, list size, and pre-intervention adherence to the two relevant targeted indicators and randomised them 1:1 to individual trial arms (Fig 1). Within each trial, we assumed that any clinical effects of either implementation package would be independent of one another; each intervention arm therefore acted as the control arm for the other intervention arm in the same trial. For example, practices assigned to the diabetes control package acted as the implementation control comparison for practices assigned to the risky prescribing package. Practices from one CCG (29 practices) involved in a concurrent initiative targeting one of our study priorities, anticoagulation in AF, were excluded from Trial 2. Practices and trial personnel were not blind to allocation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Trial profile.

^aSafe haven practices are for those patients who have demonstrated violent tendencies, who have been removed from usual general practice, or for ex-offenders. ^bThe flow diagram presents the whole randomisation process, but this paper reports the planned comparisons within Trials 1 and 2. The no intervention arm is shaded to indicate that it was not included in the analyses described here; analysis of this group will be reported elsewhere. ^cOne practice in the risky prescribing arm merged with a non-ASPIRE practice in advance of the fourth feedback report. However, as they received the first three feedback reports, some outcome data are available, and they are included in the final analyses. ^dOne practice in the BP control arm closed in advance of the third feedback report. However, as they received the first two feedback reports, some outcome data are available and they are included in the final analyses. As an example, for the diabetes control practices: there were 14,522 patients acting as the intervention group for the diabetes control implementation package and 18,131 patients acting as the control group for the risky prescribing implementation package. A&F follow-up: delivery of audit and feedback reports during the intervention period; AF, atrial fibrillation; BP, blood pressure; RP, risky prescribing.

https://doi.org/10.1371/journal.pmed.1003045.g001

General practice and trial personnel involved in intervention delivery and trial statisticians responsible for analysis were, of necessity, aware of allocation assignment. Data collection for all endpoints was masked to allocation.

Results

We assessed 278 practices for eligibility in January 2015, excluding 35 because of closure or earlier participation in the programme (Fig 1). We invited 243 (87%) practices to participate in February 2015; 56 (23%) opted out, largely because of workload pressures, and nine were excluded for other reasons.

In April 2015, we randomised 178 (73%) practices to Trial 1 (80 practices), Trial 2 (64 practices), or no intervention (34 practices). Analyses of the no intervention arm will be reported elsewhere. Within each trial, practices were randomised 1:1 to intervention arms.

Baseline characteristics for general practices and patients were well balanced by trial and intervention (Tables 1 and 2). The mean practice IMD was 30.22 (SD 13.86) within the top quarter of UK social deprivation [24].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General practice characteristics at baseline by trial arm.

https://doi.org/10.1371/journal.pmed.1003045.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Patient characteristics at baseline by trial arm (all patients from all practices in each intervention arm).

https://doi.org/10.1371/journal.pmed.1003045.t002

No practices actively withdrew from the trial. One practice closed (BP intervention arm), another merged (risky prescribing) with a non-trial practice during the intervention period, and two trial practices merged (diabetes and risky prescribing). As outcome data for these practices were available up to the times of the second, third, and fourth feedback reports, respectively, all were included in the analysis using their most recent data.

Between May 2015 and April 2016, all intervention practices received feedback reports as intended, including an end-of-study report (S6 Table summarises intervention delivery). Educational outreach visits were delivered to 67 (47%) of 144 practices; 52 (68%) of the 77 practices declining visits gave no reason. Sixteen practices (24% of those receiving educational outreach visits) utilised additional pharmacist support. One hundred twenty-six (88%) practices joined the organisational groups, allowing them to access searches and computerised prompts (risky prescribing practices only). Second educational visits were delivered to eight (6%) practices.

Primary analyses demonstrated varying results across the implementation packages at 11 months post-randomisation (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Primary outcome achievement: Baseline rates, outcome rates, and ORs adjusted for baseline achievement and covariates.

https://doi.org/10.1371/journal.pmed.1003045.t003

The diabetes implementation package had no observed effect on diabetes treatment targets. Achievement was 24.2% in intervention practices; 23.7% in control practices (adjusted odds ratio [OR] 1.03; 97.5% confidence interval [CI] 0.89–1.18; p = 0.693; ICC = 0.015).

The risky prescribing implementation package reduced high-risk NSAID and anti-platelet prescribing. In intervention practices, 4.9% of patients had a record of risky prescribing; 6.0% in control practices (adjusted OR 0.82; 97.5% CI 0.67–0.99; p = 0.017; ICC = 0.022).

The BP implementation package had no observed effect on BP control. Achievement was 53.6% in intervention practices; 52.3% in control practices (adjusted OR 1.05, 97.5% CI 0.96–1.16; p = 0.215; ICC = 0.006).

The AF implementation package had no observed effect on anticoagulant prescribing. Achievement was 73.2% in intervention practices; 75.2% in control practices (adjusted OR 0.90; 97.5% CI 0.75–1.09; p = 0.214, ICC = 0.009).

There was little evidence of any intervention effects for any secondary outcome (S3–S6 Tables) except the risky prescribing implementation package, which showed some evidence of improvement against an individual indicator: patients aged 65 years or over prescribed aspirin and clopidogrel without co-prescription of gastro-protection. Adjusted prescribing levels were 25.3% in intervention practices, compared to 35.2% in control practices (adjusted OR 0.62; 97.5% CI 0.39–0.99; p = 0.021). The conclusions for total serum cholesterol and HbA1c were robust to log transformation of the outcomes in a sensitivity analysis conducted following model diagnostic checks. We observed no intervention effects on any QOF indicators, including those not targeted by the implementation packages.

In estimating cost-effectiveness, we assumed equal costs for all four implementation packages. We conservatively doubled the cost per package. The total cost of package delivery was £175,592, and cost per practice was £2,439 (£175,592/144*2). Costs per patient (not only those eligible for indicator criteria) for an average practice list size of 7,130 were negligible (£0.28). No interventions were cost-saving. The risky prescribing package is highly likely to be cost-effective with an ICER of £1,359 and 79% chance of cost-effectiveness (Table 4), with results largely driven by two indicators. While the deterministic ICER for the BP package indicates cost-effectiveness, this is highly uncertain, with the probabilistic results indicating only a 52% chance of cost-effectiveness. We have not presented model results for the diabetes package as incremental costs and benefits were negligible, making model results unreliable. Results were robust to deterministic sensitivity analyses when, for example, we increased and reduced intervention costs. S1–S4 Figs present cost-effectiveness planes and acceptability curves for the risky prescribing and BP control implementation packages.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Cost-effectiveness analysis: Mean probability sensitivity analysis outcomes at the practice level for the primary risky prescribing and BP control outcomes.

https://doi.org/10.1371/journal.pmed.1003045.t004

Discussion

Our pragmatic, randomised trials found that an adaptable, multifaceted implementation package improved clinical care for only one of four high-impact indicators in general practices serving relatively socially deprived populations. The odds of risky prescribing for a patient in an intervention practice were 18.5% lower than for a patient with the same characteristics in a control practice, which could ultimately be associated with changes in mortality, morbidity, and health service use, depending on generalisability to the general population. There was insufficient evidence of an effect upon diabetes control, BP control, and anticoagulation for AF. Our findings suggest that the design and delivery of implementation strategies need to account for differences in the nature of targeted clinical behaviours and go further than the kinds of adaptations in content that we applied.

Commonly used interventions to implement evidence-based practice, such as audit and feedback, educational outreach, and computerised prompts generally have modest if variable effects on clinical performance [12–14]. Tailoring such interventions to identified needs and barriers offers a means to enhance their effects, but how best to do this and improve patient outcomes remains uncertain [25].

Our findings confirm the effectiveness, generalisability, and value of interventions incorporating audit and feedback to improve prescribing safety in UK primary care [26–28]. The PINCER trial compared one-off feedback with pharmacist outreach visits to one-off feedback alone, whilst the D-QIP study provided financial incentives in addition to weekly feedback. Given our pragmatic approach to practice recruitment, our intervention and findings compare most closely to those of the EFIPPS trial, which found that quarterly feedback, with or without embedded behaviour change techniques, reduced risky prescribing by a similar magnitude [28]. Our modelling also indicated relative cost-effectiveness, mainly driven by improved levels of gastro-protection. It is noteworthy that the cost of the risky prescribing package was only £0.28 per patient.

We found no benefit of the implementation package on the management and outcomes of our targeted long-term conditions. Systematic review evidence suggests that interventions targeting systems of patient management along with patient-mediated interventions (which we did not include) are likely to be important components of strategies in this context [29].

The balanced incomplete block designs permitted comparison of intervention effects while mitigating any potential nonspecific performance effects of trial participation [30]. Our trials were pragmatic in three ways. Opt-out recruitment ensured that participating practices were generally representative of the wider population [19]. Data collection was minimally intrusive. For intervention delivery, all practices received but were not obliged to act on feedback reports, whilst outreach visits were optional. Hence, the implementation packages were tested under real-world conditions, increasing confidence in wider applicability to routine general practice settings. We further demonstrated no adverse impacts on incentivised indicators of care not targeted by the implementation package.

Our evaluation had five main limitations. First, the use of routinely collected data may have compromised the precision of trial outcomes and hence ability to demonstrate effects. However, we extracted structured data that are reasonably reliably coded in general practice, partly incentivised by QOF. Second, given the multifaceted nature of the implementation package, we cannot make any inferences about the relative effects of individual intervention components. Third, educational outreach visits were delivered by facilitators not allocated to specific trial arms, thereby risking contamination between arms. We had instructed facilitators to focus only on delivering the implementation package to which each practice was assigned. Fourth, an 11-month follow-up period may have been too short to detect changes in patient outcomes, such as those related to diabetes or BP control, especially if a number of general practices only received educational outreach visits later in this period. This explanation is unlikely, as other trials have demonstrated changed clinical outcomes within similar durations of follow-up [29]; we also did not detect any improvements in processes of care for diabetes and BP control, respectively. Fifth, our composite endpoint for diabetes control requiring achievement of all HbA1c, BP, and cholesterol treatment goals may have been too demanding. Our clinical and patient advisors had considered this endpoint fair, if challenging.

Our work highlights three methodological issues. First, the implementation package effect on risky prescribing was modest but important at a population level. Foregoing a randomised design, as has been suggested for quality improvement research [31], would have reduced confidence in the validity of our findings and risked false positive conclusions. For example, the trial design accounted for temporal changes, including improvements in risky prescribing and anticoagulation achievement and declines in BP and diabetes achievement (Table 3), which would otherwise be difficult to interpret. Second, whilst we see the pragmatic design as a strength, a more explanatory approach could have made full engagement with our implementation package a condition of trial participation. Such mandating is seldom possible or even desirable in quality improvement programmes dependent upon professional engagement, particularly if they encourage ‘gaming’ behaviours to achieve goals whilst circumventing real action. Similarly, our opt-out approach to recruitment may have reduced self-selection of more enthusiastic practices, as well as administrative burden. Those responsible for leading quality improvement initiatives often specifically wish to include less enthusiastic or poorer performing practices. Third, a critical challenge prior to pragmatic evaluations is to develop interventions that are sufficiently durable to withstand the relatively hard-pressed and evolving environments of clinical practice. Whilst we followed the UK Medical Research Council framework for the development and evaluation of complex interventions, practice engagement with our implementation package was highly variable [32]. We would now recommend more intensive field work involving iterative cycles of testing and refining interventions prior to scaling up for definitive evaluation.

We had set out to develop an implementation package that could be adapted for different clinical priorities in primary care. We offer four interrelated explanations for the observed differences in intervention effects. First, the targeted clinical behaviour(s) and associated endpoints varied according to the extent of control held over them by clinical staff. Clinicians could make relatively straightforward changes to reduce risky prescribing, such as adding gastro-protection for prescribed aspirin, with limited input from patients. The observed significant difference in achievement of the associated secondary outcome on gastro-protection provides evidence to support such a mechanism (although we would still advise cautious interpretation, given that these analyses were not formally adjusted as we tested across multiple secondary endpoints). In contrast, improving BP control can require at least two consultations and changes in patient behaviour, as well as finding a pharmacological agent that is acceptable to and effective for an individual patient. This is consistent with evidence that adherence to clinical recommendations that are more complex or disruptive to routine practice is lower compared with simpler recommendations [33]. Interventions involving audit and feedback have also been shown to reduce other discouraged prescribing behaviours, namely of antibiotics [34], and offer a means to address urgent priorities such as rising opioid prescribing [35]. Second, the risky prescribing package included an automated computerised prompt requiring a one-step justification for continued prescribing. Such prompts are generally effective in changing prescribing behaviour [14]. However, a recent UK trial found that on-screen reminders did not improve adherence to the other prescribing behaviour we targeted, anticoagulation prescribing for AF [36]. With the benefit of hindsight, this is unsurprising. Qualitative work we undertook before the trials revealed that anticoagulation prescribing comprises a series of deceptively complex considerations and behaviours, which include balancing benefits and risks with patients [15]. Clinicians sometimes lacked confidence in starting treatment, given that they encountered it relatively infrequently in routine practice and felt frustrated by complicated guidance that made treatment difficult to explain to patients. Third, the number of patients requiring action to achieve indicators varied by trial arm, with far more in the diabetes and BP control arms relative to risky prescribing and anticoagulation for AF (see S5 Table). Hence, the larger numbers of patients requiring action may have undermined clinicians’ perceived feasibility and motivation. Moreover, making changes for even a small number of patients within the risky prescribing indicator set would have had a comparatively larger effect upon achievement relative to other indicators. Fourth, the indicators for three targeted long-term conditions were incentivised in the QOF. As well as aligning the implementation package with existing quality improvement schemes, our intention had been to encourage practices to move beyond the QOF by adopting more challenging, evidence-based goals. Within the present constraints of UK primary care, it was hard for practices to take on additional, unrewarded work. Clinicians may, nevertheless, have been motivated to address risky prescribing because it concerned patient safety.

In conclusion, an adaptable implementation package improved prescribing safety in general practice. However, we observed no benefits of the package within the context of an existing financial incentive system targeting similar aspects of care for three long-term conditions. Improving patient outcomes for long-term conditions requiring relatively complex management may require systemised approaches that target patient as well as professional behaviour.

Supporting information

Acknowledgments

The ASPIRE programme team comprises Susan Clamp, Rebecca Lawton, Rosie McEachan, Martin Rathfelder, Judith Richardson, Tim Stokes, Vicky Ward, Robert West, and Ian Watt, in addition to the named authors. The ASPIRE programme team can be contacted via Robbie Foy (r.foy@leeds.ac.uk). We acknowledge the contributions of Andrew Davies, Peter Heudtlass, Chris Jackson, and John Turgoose and thank the members of the ASPIRE Programme Steering Committee for their advice and expertise. We thank Richard Neal, Suzanne Richards, and Noah Ivers for helpful comments on earlier versions of this manuscript.

Disclaimer: The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.