Whole system transformation in the NHS

The predominant NHS England-led improvement programme between 2015 and 2018 focused on developing and evaluating a series of new ‘whole system’ integrated models of care through pilot sites, known as ‘Vanguards’ [9]. These were created as locally-driven prototypes for integrating health and social care services within a geographically bound region, aiming to transition the NHS to functioning on a ‘place-based’ framework. In this model organisations (including acute, community and mental health trusts and local authorities) within a region have joint responsibility for improving the outcomes of the population within the region, rather than for the care delivered to individual patients within their respective organisations. It was anticipated that approaches developed by successful Vanguards would be disseminated across the NHS in England [1, 3, 10].

Aligned to the Vanguard implementation programme were a range of evaluations, which have provided preliminary indications about what could be useful, both in terms of the models and the approaches to implementation used [11–13]. However, due to a range of design and data collection challenges, it has been difficult to draw firm conclusions from them [12, 14–16]. For example, it has been almost impossible to establish which service components included in the Vanguard programme led to improvement, how these affected each organisation, and the subsequent impact on outcomes [12]. Problems stemmed from the rapidity of implementation and difficulty aligning implementation and evaluation efforts, in particular as evaluation was often reported to be a post hoc activity that was infrequently embedded within the transformation efforts nor prioritised by leadership [11, 12].

Consequently, none of the evaluations were able to measure fidelity to the model, as often there was no clearly ascribed approach at the outset nor any logic models developed to explain the theorised relationship between service changes and outcomes, and there were significant challenges in accessing performance and qualitative data from participating organisations [11, 12]. In contrast, MRC Guidelines highlight that useful evaluation of a complex intervention requires assessment of both the effectiveness of the model being adopted, and the effectiveness of the implementation process itself [17]. Without both, it is difficult to determine if poor outcomes are due to an ineffective intervention, partial implementation, or poor fidelity to the model. Where a model is found to be useful, it is important to understand the factors and processes observed to be associated with improved outcomes. It is also critical that evaluations take place over an appropriate timeframe, as premature evaluation could mislead conclusions about a model’s effectiveness, which may reflect incomplete implementation rather than an ineffective model [18].

Service transformation within CAMHS

During this period of NHS transformation, focus has been placed on improving CAMHS services, supported by the largest injection of spending in its history. Identifying optimal care models and addressing the implementation issues related to whole system transformation for CAMHS has never been more important. Long waiting times have been a major barrier to CYP accessing mental health care in the UK. In 2017 it was estimated that more than 12.5% of CYP in England experience a mental health problem [19]. Prior to the start of national CAMHS transformation, in 2013 the average waiting time to access a routine care appointment in CAMHS was as high as 15 weeks, with only 31% of CYP who required intervention accessing services [20]. For those CYP that were offered support, a lack of flexibility in service models to provide care in line with patient needs and preferences resulted in dissatisfaction in the care received, leading to poor patient engagement, and poor clinical outcomes [5]. Difficulty accessing care was made worse by patients ‘falling through the cracks’ when transferring between services, as a result of poor staff coordination and organisations not being incentivised to provide care along whole pathways.

A growing body of evidence incorporated into the THRIVE framework indicates that these problems within CAMHS could be improved by actively involving CYP in their own mental health care as collaborative participants and decision-makers, rather than passive consumers of treatment. Enhancing these opportunities through shared decision-making (SDM) is associated with better clinical outcomes as well as greater patient satisfaction [21–23]. Reports from CYP, professionals and carers indicate that multi-agency working improves treatment experience, and promotes a more comprehensive delivery of care [24, 25]. Considering this and additional evidence it is recognized that tackling CAMHS waiting times requires three fundamental changes in practice: (1) tailoring interventions and therapies to fit clinical practice better [26, 27], (2) tailoring interventions and therapies to fit CYP’s needs and preferences [27], and (3) building and tailoring interventions and therapies to fit non-clinical (non-NHS) contexts [28]. Building on this evidence, care delivered through goal-focused approaches have been shown to improve patient flow through the service, reducing waiting times, and improving service accessibility [29, 30].

The i-THRIVE implementation programme supports sites to incorporate many of these principles, which are outlined in the THRIVE Framework [4–6], into their delivery of care. However, while elements are evidence informed, there remains no large-scale evaluation of the effectiveness of the approach in terms of the outcomes achieved, the key components of a service that has adopted the THRIVE principles, nor the optimal approach to its implementation. The national CAMHS transformation, which includes all CCGs from 2016–2022, has over seventy sites that chose to use i-THRIVE as the basis of their transformation efforts. This provides a unique opportunity to undertake a national evaluation of i-THRIVE, and if adequate quality data are obtained, a study of the implementation has the potential to extend knowledge of what works in the development of whole system, place-based approaches to delivering care in CAMHS and for the NHS more widely.

Developing the model of implementation

The i-THRIVE model of implementation takes an evidenced-informed approach to whole system service transformation and has been designed to implement THRIVE’s framework of concepts and principles in a CAMHS setting. The THRIVE Framework [4–6] comprises a set of concepts and principles that set out what a ‘THRIVE-like’ system of CAMHS might look like across the macro (systemic: commissioning and provider leadership systems), meso (organisational structures and relationships, and how these work together within clinical pathways) and micro (individual practitioners’ ways of working) dimensions of a mental health care system for CYP (see S1 Table). The i-THRIVE approach to implementation translates these concepts and principles into a series of components developed by drawing on the evidence-base to create a structure and set of tools to guide implementation efforts. The six main components of the i-THRIVE model are described in Table 1, along with the implementation science principles relating to integrated care systems that they draw on. For example, drawing on Normalisation Processing Theory [31, 32], the ‘THRIVE Assessment Tool’ (see S3 Fig) is designed to provide a practical guide for implementation and a means to assess a site’s alignment with THRIVE principles, and the ‘i-THRIVE Academy’ (see S2 Table) is designed to offer training to staff around ways of working that support the delivery of the THRIVE principles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The components of the i-THRIVE programme.

https://doi.org/10.1371/journal.pone.0265782.t001

With the aim to strike a balance between supporting local determinism and retaining fidelity to the THRIVE Framework, the components of the i-THRIVE model are designed to support sites to deliver care that is aligned to the THRIVE principles while also compatible with the local context, consistent with evidence indicating that innovative service models are more easily adopted if they can fit local contexts [17]. Following this approach, the components of the i-THRIVE model do not aim to prescribe a set protocol for how sites should deliver the THRIVE principles of care, but aim to guide each site to develop a holistic approach to their local interpretation of the THRIVE principles–their own ‘THRIVE model of care’–and support them to develop an implementation plan to guide them in how to implement it. For example, SDM is a core THRIVE principle. i-THRIVE does not prescribe how shared decision making should be implemented, but provides a range of options and tools. i-THRIVE Option Grids© have been developed by the implementation team [48], however, the i-THRIVE implementation programme has also collated a range of other approaches to SDM that have been successfully used by i-THRIVE sites. Core to i-THRIVE is to make such support resources readily accessible and the i-THRIVE Community of Practice website signposts to all of these along with a slew of relevant resources. The Community of Practice day focussing on SDM includes presentations from a range of sites on how they have approached SDM. The i-THRIVE academy module on SDM, includes training for Option Grids and other approaches. The i-THRIVE Assessment Tool assesses whether SDM has been successfully implemented, and does not prescribe a particular approach. Together, this enables sites to choose an approach that best fits their local context [49].

The ‘i-THRIVE Approach to Implementation’ is manualized and is structured to be a four-phase process, drawing on the Quality Implementation Framework [50] (see S2 Fig) and Normalisation Process Theory [31, 32], for implementing complex interventions. Each phase is supported by a range of tools, described in detail on the programme website [51]. Phase One includes forming a comprehensive understanding of the local system, including the existing challenges and needs of the local population. Stakeholders and system leaders across health, social care, education and third sector organisations are invited to engage with the programme during this phase, and governance is established. The THRIVE Assessment Tool is used to assess how ‘THRIVE-like’ the CAMHS system is compared to the THRIVE principles. Through multi-agency discussion, areas of strength and weakness are identified, which supports the development of an implementation plan and a measurable approach for monitoring change. Phase Two includes building capacity within the system by appointing ‘THRIVE Champions’, and identifying the training needs of managers leading transformation and staff on the front-line. Front-line training is designed to be conducted through the i-THRIVE Academy, created to provide a set of four modules to support staff development, each of which addresses the core requirement of one of the four THRIVE domains (see S2 Table). A local i-THRIVE Community of Practice is also created to provide a mechanism for shared learning across implementation teams. To ensure that the data required to support improvement will be available and used within the teams, data collection mechanisms are established during this phase. Phase Three focuses on implementation of the new system using a variety of change management and quality improvement methods (e.g. ‘Plan, Do, Study, Act’ cycles), as well as establishing information and quality infrastructures within providers and commissioners. Measurement systems enabling collaborative assessment of progress are set up during this phase to identify potential issues so that these can be tackled across the locality. Phase Four focuses on learning, embedding and sustaining changes to ensure these become ‘business as usual’ once the transformation programme is complete. This involves reflection on what is working better or has been learnt from implementation, and offers an opportunity to provide feedback and share local learning with the broader i-THRIVE Community of Practice.

Taking each of these components into account, a logic model has been created for the programme and used as a framework for evaluation (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. i-THRIVE logic model.

The i-THRIVE logic model shows the inputs, processes, outputs and the expected outcomes and impact of implementation.

https://doi.org/10.1371/journal.pone.0265782.g001

Evaluation objectives

The evaluation has two sets of objectives, the first involves evaluation of the effectiveness of the i-THRIVE model in improving the performance of CAMHS services, and the second relates to evaluating the approach to implementation.

Evaluating the effectiveness of the i-THRIVE model

Evaluation of i-THRIVE model focusses on the relationship between the model of care used and the outcomes achieved within a region. Our research questions are: (1) Does i-THRIVE lead to improvements in patient experience, clinical outcomes and service performance outcomes in CYP’s mental health services? (2) Which elements of the service lead to improvements in outcomes? (3) Who benefits from i-THRIVE, and what are the consequences for equity and diversity?

Evaluating the process of implementation

The evaluation of the implementation process extends from the logic framework in Fig 1 and draws on MRC guidance to understand context, quantify implementation and explore mechanisms of impact (see Fig 2) [17, 18]. The Research Questions focus on: (1) What approaches to implementation have been employed by sites? (2) What is the relationship between implementation (including measures of context, fidelity, dose and clinical pathways) and outcomes (including patient experience, service performance and clinical outcomes)? (3) What are the barriers and facilitators to implementation?

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The conceptual model underpinning the study design (Moore et al., 2015), indicating measurement domains.

https://doi.org/10.1371/journal.pone.0265782.g002

Study setting and design

We plan to use a matched cohort study design. Twenty CAMHS sites across England are to be included in the study, comprising: ten ‘accelerator sites’ have been identified that adopted the i-THRIVE model from the start of the NHS England-funded CAMHS transformation. We identified ten ‘comparator sites’ that chose to use different transformation approaches within the same timeframe which were comparable on broad demographic criteria. All sites are defined as geographically discrete CAMHS commissioning areas within England, aligned to either a single NHS Clinical Commissioning Group (CCG) or a small number of CCGs working in collaboration, with responsibility for commissioning CAMHS for their local population. Since the i-THRIVE model aims to support a whole-system integrated approach to delivering CYP’s mental health care, each site will be described in terms of the CCG(s), the trust(s) providing CAMHS, and the partner Local Authority(ies).

Site recruitment is now complete. Sites were recruited through an email invitation to participate in the study sent to all CCGs in the UK inviting them to participate in an implementation programme to support the adoption of the i-THRIVE model. Sites applied to be an implementation site and were asked to demonstrate commitment to implement the model over four years from both health and social care, and to participate in the evaluation. In return they were offered implementation support via the i-THRIVE Implementation program and the funded evaluation. Ten sites met the criteria and were adopted into the study. It was important to identify matched control as closely as possible by baseline characteristics, commitment to transformation and approach to joint working between health and social care. Our approach was to match for baseline characteristics first (population size and density reported by the Office of National Statistics–Mid 2015 Population Estimates for Clinical Commissioning Groups in England [52], combined CAMHS and CCG funding 2016–2017 and level of deprivation defined by the Department for Communities and Local Government data—English Indices of Deprivation 2015 [53]. An ordered list of preferred sites for each test site was comprised and each was approached in order of preference according to matching criteria. Sites were then interviewed to establish the commitment of health and social care to CAMHS transformation and participation in the evaluation over four years. Sites were asked to agree not to implement i-THRIVE within the four years of the evaluation. If all the criteria were met, controls were included in the study until we had ten matched controls.

Data collection

Data will be collected between 2015 and 2020 for all sites. The measurement plan is embedded within the logic model (see S3 Table for an overview of all measures to be collected). Covid-19 has delayed data collection which has not been completed and data analysis remains to be performed.

Outcome measures

Patient and service level data collection will take place at baseline and then annually over the four-year study period. Outcome measures are to be categorised into three groups:

Clinical outcomes will comprise measures of symptom improvement and patient reported outcomes that are routinely recorded by each site.

Patient experience and engagement with services is to be assessed using routine patient experience measures provided by each site (such as The Friends and Families Test, and Children’s Global Assessment Scale) and engagement through the proportion of patients attending appointments compared to those who do not.

Service performance outcomes are focused on three areas: access, waiting times, and efficiency. Access will be measured by: the number of new referrals received to the service, the number of referrals assessed by triage, and by Tier 3 services and the number of patients receiving treatment. Waiting times will be assessed by calculating three indicators: the average waiting times between referral received to triage, referral received and assessment, referral received and the first episode of care within a specialist service and the maximum waiting time. Efficiency will be measured by the average number of: contacts per patient, face-to-face appointments, non-face-to-face appointments, discharges, and re-referrals. Sub-analyses will be conducted to assess ethnicity, age and diagnoses to assess the impact of on equity and diversity.

Fig 3 and Table 2 show the type of data to be collected, define the measures to be used, and how these measures relate to the CAMHS care pathway.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Quantitative measures collected per patient, defining ‘access’, ‘diversity’, ‘waiting times’, ‘efficiency’ and ‘clinical outcomes’.

The CAMHS pathway is illustrated by the grey boxes. Our base cohort includes all patients referred to a site within the four-year period of the evaluation. For every patient referred we will collect demographic information (age at referral, ethnicity and presence of learning disability). For each subsequent contact point (triage to discharge) the information shown in the relevant box will be collected. This refers to one episode of care. In the case of a patient being re-referred, each subsequent referral will be identified as a separate episode of care. Episodes of care will be recorded accumulatively and given an ID unique to the patient in question. A range of measures have been constructed for the evaluation using data from different parts of the data model. These are numbered 1–12 above and details are provided in Table 2.

https://doi.org/10.1371/journal.pone.0265782.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Description and definition of the outcome measures and the source of data to be used for the calculation.

https://doi.org/10.1371/journal.pone.0265782.t002

Each of the measures is numbered and corresponds to the measure of the same number found in Fig 3.

Process evaluation measures

The approach to data collection for the process evaluation has been designed based on the MRC guidance on how to evaluate the process of implementation for a complex intervention. Fig 2 illustrates the conceptual model, based on Moore et al [18], showing the domains measured within this evaluation. In summary, we measure context (details of the local context plus barriers and facilitators to implementation), fidelity (how close local models align with THRIVE principles), dose (the quantity of what is implemented), reach (the extent that the intervention reaches its target audience), and pathway mapping (details of the structures of child mental health pathways, the services offered and the extent to which these are integrated).

Context will be explored to understand the barriers and facilitators to implementation through a survey based on the inner and outer setting constructs defined by the Consolidated Framework for Implementation Research (CFIR) [54]. This is a validated, mixed-method, multi-level framework that conceptualises implementation according to five areas and 39 constructs. The survey will be completed by the i-THRIVE Implementation Leads for accelerator sites, and the CAMHS Transformation Leads for comparator sites.

Fidelity will be assessed by the alignment of services to the THRIVE principles across the macro (senior system leadership and commissioning), meso (service management) and micro (front-line professionals working with CYP) levels of the system. Data will be collected through semi-structured interviews using the i-THRIVE Assessment Tool, a measure that has been developed for this specific purpose, and which will be validated during the course of the i-THRIVE programme. Purposive sampling will be used to recruit three interviewees per system level at each site, to collect a multi-agency perspective across CAMHS, third sector, clinical commissioning groups, education and local authority services. Interviews will be conducted at two time-points: baseline and follow-up (up to four years after service implementation), and will be digitally recorded and transcribed verbatim. The interview transcripts will be scored by researchers who are blind to data collection in relation to how ‘THRIVE-like’ the service described in the transcript is performing.

Dose and reach will be assessed using a nine-item ‘adoption’ survey that has been developed based on the RE-AIM framework [55] which is to be sent via email to all front-line staff working within the accelerator sites, to measure their understanding of i-THRIVE and the THRIVE principles. The survey is to be sent at two time-points: baseline and follow-up (four years after service implementation). In addition, mechanisms of impact of service transformation will be explored through semi-structured interviews with both accelerator and comparator sites through questions related to barriers and facilitators of the service implementation.

Pathway mapping will be undertaken to compare the structure of CAMHS pathways at baseline and after CAMHS transformation to explore whether transformation led to pathways becoming more consistent with NHS England guidelines and the extent to which services and pathways are integrated. We will record: the services provided in a region, including NHS, local authority, third sector and school based mental health interventions; who provides the service, its modality and its relationship to other services in the pathway; and the number of access and assessment points in each system. Data will be collected by reviewing documents of local transformation plans at baseline and after implementation. These will then be supplemented by semi-structured interviews with the site leads to confirm key details and the accuracy of the maps.

We will gain a clear view of what is implemented in place of THRIVE within control sites by bringing together pathway data to understand the changes to the structure of clinical pathways, through the interviews with sites (approximately 18 per site which provide the detail of how sites approached transformation at macro, meso and micro level), and through a survey of the implementation leads.

Ethics approval and consent to participate

The NHS/University Joint Research Office reviewer the study and determined it to be a service evaluation and therefore does not require ethics or review by the REC (IRAS application number: 250439). For qualitative aspects, participants are implementation leads, managers and front-line staff, and are interviewed about their service transformation project. Verbal informed consent to take part is obtained at the beginning of each interview, including explicit consent for the interview to be recorded, analysed and used as part of the service evaluation. This is recorded and transcripts have been made of each recording to document this. The data used for the quantitative evaluation is retrospective routinely collected service data extracted from electronic records. It is anonymised at source by business intelligence staff at each trust, removing all identifying information, and after review by local information governance leads, is provided to the evaluation team. Consent for the use of retrospective de-identified service data for evaluation purposes is not legally or ethically required and the requirement for ethics was waived by the joint research office.

Data analysis plan

The qualitative data will be coded, sorted and classified following the ‘framework approach’ based on the CFIR framework. General statistical test such as t-tests, ANOVA tests, and chi-square tests will be used to evaluate group differences between accelerator and comparator sites, or between the baseline and follow up data collection time points. Difference-in-difference (DID) or Interrupted Time Series (ITS) models will be adopted to test the causal relationship between the i-THRIVE intervention, and service and patient outcomes. The DID design is recommended for quasi-experimental studies that compare the outcomes of groups exposed to different policies and environmental factors at different times [56]. We will compare trusts over time which could have experienced i-THRIVE with the comparison trusts. Care will be taken that the comparison trusts do not become THRIVE-like using our specially developed metric of intervention fidelity to control for so-called "spillover" effects. In order to strengthen the comparison group we intend to extend the DID design with propensity scores weighting that will enable us to include many more potential control observations from the NHS data set from sites that are more similar to those in the treated group [57]. A key assumptions of DID designs is common trends assumption, that observations should differ by the same amount in every period [56] suggesting that divergences from the common trend after the policy intervention are due to the implementation of i-THRIVE and not to other confounding factors. This assumption will be tested in regression models controlling for other potential confounders such as trust size, the level of economic deprivation, etc. Structural equational modelling will be conducted to test the causal relationship of potential variables to determine which elements of the service lead to improvements in outcomes. Poisson regression with a population offset (to model a rate rather than a count) will be used for variables relating to access to treatment, and where the outcome of interest is a probability (e.g. patterns of service use including ‘did not attends’) logistic or Probit regression will be chosen as model options. To obtain correct statistical inference (standard errors), repeated observations over time will also be taken into account [58]. All qualitative data will be analysed using NVivo (version 12) and all quantitative data will be analysed using R (version 4.0.0).

Data management plan

Data collected will be qualitative and quantitative in nature. Anonymised interviews and service performance data will be stored securely on password protected UCL servers for the duration of the study and then for a further ten years. Anonymised data sets may be transferred to University of Cambridge for analysis. Transfer will be via secure file transfer using the encrypted UCL dropbox account and will be stored on password protected Department of Psychiatry, University of Cambridge servers. Only members of the research team will have access to the data. In line with UCL data management policy, following publication all data will be archived at the university and then destroyed after ten years.