Figures
Abstract
Background
‘Engager’ is an innovative ‘through-the-gate’ complex care intervention for male prison-leavers with common mental health problems. In parallel to the randomised-controlled trial of Engager (Trial registration number: ISRCTN11707331), a set of process evaluation analyses were undertaken. This paper reports on the depth multiple case study analysis part of the process evaluation, exploring how a sub-sample of prison-leavers engaged and responded to the intervention offer of one-to-one support during their re-integration into the community.
Methods
To understand intervention delivery and what response it elicited in individuals, we used a realist-informed qualitative multiple ‘case’ studies approach. We scrutinised how intervention component delivery lead to outcomes by examining underlying causal pathways or ‘mechanisms’ that promoted or hindered progress towards personal outcomes. ‘Cases’ (n = 24) were prison-leavers from the intervention arm of the trial. We collected practitioner activity logs and conducted semi-structured interviews with prison-leavers and Engager/other service practitioners. We mapped data for each case against the intervention logic model and then used Bhaskar’s (2016) ‘DREIC’ analytic process to categorise cases according to extent of intervention delivery, outcomes evidenced, and contributing factors behind engagement or disengagement and progress achieved.
Results
There were variations in the dose and session focus of the intervention delivery, and how different participants responded. Participants sustaining long-term engagement and sustained change reached a state of ‘crises but coping’. We found evidence that several components of the intervention were key to achieving this: trusting relationships, therapeutic work delivered well and over time; and an in-depth shared understanding of needs, concerns, and goals between the practitioner and participants. Those who disengaged were in one of the following states: ‘Crises and chaos’, ‘Resigned acceptance’, ‘Honeymoon’ or ‘Wilful withdrawal’.
Conclusions
We demonstrate that the ‘implementability’ of an intervention can be explained by examining the delivery of core intervention components in relation to the responses elicited in the participants. Core delivery mechanisms often had to be ‘triggered’ numerous times to produce sustained change. The improvements achieved, sustained, and valued by participants were not always reflected in the quantitative measures recorded in the RCT. The compatibility between the practitioner, participant and setting were continually at risk of being undermined by implementation failure as well as changing external circumstances and participants’ own weaknesses.
Trial registration number
ISRCTN11707331, Wales Research Ethics Committee, Registered 02-04-2016—Retrospectively registered https://doi.org/10.1186/ISRCTN11707331.
Citation: Weston L, Rybczynska-Bunt S, Quinn C, Lennox C, Maguire M, Pearson M, et al. (2022) Interrogating intervention delivery and participants’ emotional states to improve engagement and implementation: A realist informed multiple case study evaluation of Engager. PLoS ONE 17(7): e0270691. https://doi.org/10.1371/journal.pone.0270691
Editor: Dylan A. Mordaunt, Flinders University, AUSTRALIA
Received: March 1, 2021; Accepted: June 15, 2022; Published: July 14, 2022
Copyright: © 2022 Weston et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of anonymity and participant privacy concerns. Due to the small population pool from which data were sampled, the longitudinal linked datasets, and the resulting sensitive nature of some of the comments, we believe that anonymity could be breached if the full data set were made available. Study participants consented to interviews with the understanding that their data would remain anonymous and confidential. Participants did not consent to have their full transcripts made publicly available. Ethical approval for the qualitative interviews - including the information sheets and consent forms that outlined the aforementioned data confidentiality statements - was provided by the chair of East of England – Essex Research Ethics Committee (reference number 13/EE/0249). Due to the aforementioned concerns, size of the data set, likelihood of introducing inconsistencies or errors or even reducing the meaning of the data by redacting data to make it available, as well as fears of breaching internal confidentiality (participants being able to identify themselves or others), a minimal data set cannot be made available on request at this time. All data enquiries may be sent to Dr. Fiona Giles at the National Institute for Health Research (contact via Fiona.giles@nihr.ac.uk). Researchers meeting the qualifications needed to access confidential data, could potentially obtain the minimal data set by contacting Dr. Fiona Giles at the National Institute for Health Research.
Funding: The author(s) disclose receipt of the following financial support for the research, authorship, and/ or publication of this article: ENGAGER is funded by the National Institute for Health Research (NIHR) under its Programme Grant for Applied Research Programme (Grant number: RP-PG-1210-12011). This research was also supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula at the Royal Devon and Exeter NHS Foundation Trust. The views expressed are those of the author(s) and not necessarily those of the National Health Service (NHS), NIHR or the Department of Health. The funder had no role in the design of this study and did not have any role during execution, analysis, interpretation of findings or decision to submit results. The views expressed are those of the authors and not necessarily those of NHS, NIHR or the Department of Health.
Competing interests: The authors declare that they have no competing interests.
Abbreviations: CORE-OM, Clinical Outcomes in Routine Evaluation- Outcome Measure; FPE, Formative Process Evaluation; M, Mean; MAG, Met-At-Gate; MBA, Mentalisation-Based Approach; PE, Process Evaluation; RCT, Randomised-Controlled Trial; SD, Standard Deviation; SRQR, Standards for Reporting Qualitative Research
1 Background
The success of an intervention depends on how fully and how well it is put into practice, engages the intended population, and generates changes in keeping with its intrinsic logic. Process evaluations provide essential insight into implementation, including the extent to which interventions were delivered as intended (‘fidelity’), variations in engagement, and specific circumstances under which outcomes were attained [1–4]. The knowledge gained identifies where implementation worked well (and may be translated into future practice) [5], and where there were inadequacies in intervention delivery or theory [6, 7]. Here we report a pragmatic and theory driven approach to complex intervention evaluation, providing an exemplar for testing underlying programme theory about how complex interventions work, taking into account the contributions of intervention participants.
1.1 The Engager intervention
Engager was developed in response to an omission in mental health focused RCTs carried out in prison settings [8]; this despite the high prevalence of mental health problems reported among prison populations [9–11]. For example, compared to the general population, research has found that prisoners are more likely to display: higher rates of drug dependency and hazardous alcohol use; anxiety and depression; and greater instances of self-harm and suicide ideation [12–14]. There is also substantial comorbidity, particularly in regard to substance use; and co-occurring social problems such as unstable housing or homelessness, familial relationship breakdowns; and unemployment or financial concerns [15, 16]. Despite the prevalence of mental health problems among prison populations, they tend to have minimal access to mental health care, and poor continuity of service provision on release from prison [15, 17, 18]. Similarly, although complex relationships between mental health, substance misuse, social exclusion and criminal behaviour are known to exist, they tend to be studied separately and interventions designed to address them developed in isolation from each other [1]. Therefore Engager was a novel intervention designed to bridge this gap in service provision for offenders between prison and the community.
Engager was a parallel two-group randomised controlled trial with 1:1 individual allocation to either: (a) standard care plus the intervention (intervention group) or (b) standard care alone (control group) across two investigation centres (South West and North West of England). Participants were 280 prisoners (140 per group) meeting eligibility criteria: one or more common mental health problem (anxiety, depression, PTSD), serving a sentence of less than 2 years, and due to be released within four months from the date of recruitment to the RCT.
Participants recruited to the intervention arm of the Engager Randomised Control Trial (RCT) worked one-to-one with a practitioner, who supported their often co-morbid, needs [12–14, 19] as they transitioned from prison back into the community. Engager Practitioners were recruited from a range of health and social care backgrounds, including support workers, substance use workers, and third sector providers; they often had limited therapeutic experience but were trained and supervised by clinical leads. Using a person-centred approach, the intervention was designed to be tailored to individuals’ psychosocial needs, embedding multi-agency working by overcoming service barriers and facilitating access to community services [17–19]. Based on established principles of behaviour change [20, 21] and collaborative care [22, 23], Engager was developed using realist methods [23, 24], and therapeutically underpinned by a mentalisation-based approach (MBA) [25]. MBA involves helping people develop the ability to understand actions by others and one’s self in terms of thoughts, feelings, wishes and desires, learning how to respond more adaptively in moments of distress.
The realist formative process evaluation (FPE) of the pilot trial built, tested, and refined the underlying Engager intervention theory, producing an evidence-based intervention logic model [25]. From the pilot trial FPE data a logic model was produced and was then represented diagrammatically (see S1 File). See [25] for the programme theory development process and detailed theory as intended for delivery in the Engager RCT. This included an implementation delivery platform which sought to facilitate standardisation of intervention delivery within and between sites. The components of the implementation delivery platform included: a comprehensive manual describing actions for Engager practitioners and supervisors to guide their delivery of intervention components; a training programme for supervisors and Engager practitioners, including the logic and rationale of the model and mentalisation-based approaches; a programme of supervision for Engager practitioners to receive support while delivering prison and community care, both one-to-one with supervisors and team supervision incorporating informal peer-to-peer discussions and formal monthly meetings attended by members of the research team to provide meta-supervision; and organisational agreements and support for clinical governance needs (record keeping, line management etc) and inter-organisational working (with prison and probation). Some flexibility in intensity of work and number of contacts was permitted as required by individual participant need; with a minimum ‘dose’ of two pre-release contacts and eight post-release contacts over a one-four month pre- and three-five month post- release period (see [26] for the trial protocol).
1.2 The realist-informed process evaluation
Realist evaluation seeks to explain how interventions work ‘for whom, why, and under what circumstances’ [27]. This involves scrutinising how intervention components produce desired outcome patterns by examining mechanisms (underlying causal pathways) that promote or hinder outcome attainment and intervention effectiveness [28]. It is posited that mechanisms (the ways in which people respond to the resource on offer) may only activate in a certain set of circumstances (the ‘context’); and it is a combination of this context and the associated underlying mechanisms that have causal power and lead to intervention outcome attainment. Realist evaluation can contribute to understanding the interacting, constraining and enabling factors that determine how complex interventions take effect [29–31].
Developed using realist informed methods [24, 27], the Engager logic model comprised:
- Core components; the series of actions the Engager Practitioners were asked to deliver;
- Internal responses or ‘mechanisms’ that the components were expected to activate in intervention participants; and
- Tangible ‘outcomes’ that the intervention aspired to produce for prison leavers as a result.
Engager, a novel intervention, was theoretically developed given the lack of previous innovation of mental health interventions for prison-leavers. The exploratory and explanatory depth multiple case study analysis reported in this paper aimed to:
- Determine the degree to which core components were delivered, mechanisms activated, and intended outcomes achieved in line with the theoretical logic model;
- Explore individual participants’ engagement with the intervention over time; and
- Identify aspects of intervention theory/ delivery that could be improved.
This analysis was combined with the other analyses which comprised the parallel process evaluation of the RCT and contributed to the wider interpretation of the both results of the RCT and the refinement of the intervention and its delivery [32].
2 Methods
2.1 Study design and setting
A realist informed, longitudinal, qualitative, depth, multiple case study design was used. Individual Engager participants in receipt of the intervention were ‘cases’ (the unit of delivery for the intervention). The study setting comprised the two intervention delivery regions (North West and South West) of England.
A trial registration application for the Engager randomised-controlled trial (Trial registration number: ISRCTN11707331) was made in December 2015 through the NIHR portfolio registration system (before recruitment) but due to administrative delays the ISRCTN registration date was 4/02/2016 while recruitment started 14/1/2016. The authors confirm that all ongoing and related trials for this intervention are registered.
2.2 Sample
All intervention arm participants (n = 140) were given the opportunity to participate in additional data collection for the process evaluation (PE). We purposively sampled potential PE participants using a framework based on ‘Clinical Outcomes in Routine Evaluation–Outcome Measure’ (CORE-OM) [33] baseline scores (the primary trial outcome measure assessing psychological distress). From this, 24 were opportunity sampled to ensure that we could contact them and collect data within the timeline. Although small sample sizes are often noted in terms of potential limited generalisability of findings; we followed guidance for traditional qualitative research when determining our sample source and size [34]. That is to say, we used a) a relatively sample size in order to support the depth of longitudinal case-oriented analyses fundamental to qualitative research; and b) purposive sampling to ensure cases selected contained a breadth of richly-textured contextual and circumstantial variation, crucial for testing programme theory in realist enquiry.
2.3 Data collection
Sources of data comprised face-to-face semi-structured interviews with participants at three time-points where possible (baseline/ pre-release; follow-up at three-six months post-release; and further follow up 12 months post-release); as well as interviews with Engager practitioners, and practitioners from other services (e.g. probation officers, support housing providers). We also collected practitioner activity logs which included daily timesheet activities (e.g. face-to-face sessions with prison-leavers and community liaison work) and detailed session records including purpose, session content, resources used, goals, events and actions. See Table 1 in S2 File for a full list of data sources collected for each case study participant. Interviews were audio-recorded and transcribed verbatim. The realist informed interview schedules [29, 35] were guided by the realist intervention theories developed in the FPE [24], and tailored to data collection time points.
2.4 Ethics approval and consent to participate
Ethics approval from the relevant ethics committees were obtained, as reported in the trial protocol. Specifically, we obtained ethical approval from East of England–Essex Research Ethics Committee (reference number 13/EE/0249); National Offender Management Service NRC (reference number 2013–187); National Offender Management Service (ref: 2015–283) approval and local governance approvals for each site (Devon Partnership Trust NHS Trust, Dorset Hospital University Foundation NHS Trust and Lancashire Care NHS Foundation Trust). The study was adopted by the National Institute for Health Research (NIHR) Clinical Research Network and the study sponsor is Devon Partnership NHS Trust. All prison-leavers participating in the process evaluation provided written informed consent. All intervention participants approached to take part in the process evaluation were assured that participation was voluntary, they could withdraw from the study at any time without giving any reason, and without their medical care or legal rights being affected. Participation in the process evaluation (or not) had no bearing on intervention delivery.
2.5 Analysis
A qualitative approach was taken to understand the depth of individual participants’ experiences of the intervention over time, as well as practitioners’ experiences of delivery for these people.
2.5.1 Within-case analyses.
To understand the extent of intervention delivery for each case we first extracted the number and location of sessions from practitioner timesheet and case note data. Only sessions recorded as ‘direct participant contact’ (face-to-face, or over the telephone if the length of call exceeded ten minutes) were included. We also extracted detailed session records from the practitioner activity logs. The session descriptions were analysed and the ‘focus’ of each session was coded as one of three categories:
- Therapeutic: Record suggests the primary focus of the session was therapeutic (e.g. reflections on past behaviour);
- Practical: Record suggests the primary focus of the session was practical (e.g. housing);
- Both: Both therapeutic and practical aspects to the session were recorded.
Then we interrogated individual data sources for each case, using a combined deductive (against the coding framework based on the theory logic model) and inductive approach, being alert to unanticipated experiences.
For each case we visually itemised each component, mechanism, and outcome from the intervention theory logic model using cells in a Microsoft Excel spreadsheet. Where there was evidence of something occurring for a case we mapped it against the relevant item, colour-coding differences between where we had substantial evidence something occurred often (e.g. from multiple sources); partial, tentative or conflicting evidence (e.g. differences in accounts); or evidence that something definitely didn’t happen.
Implementation fidelity was explored by ascertaining where there were gaps or ‘silences’ in the data. Consistent silences led us to reflect on whether certain intervention components had not been delivered; and/ or whether our data capture methods had failed to identify when they had occurred. Two researchers (LW and SRB) worked independently on the analysis of each case, double-coding five cases to compare and establish coding reliability and work through uncertainties.
When all of the cases had been depicted in this way we had a visual colour-coded map of each case’s overall ‘experience’ of intervention delivery and their response to the support offer, illustrated by differences in colour-coded Excel cells. Six case study maps were presented to, and interrogated by, a sub-group of the wider project study team which included academic and practitioner representation. The six cases were selected on the basis that their participant data provided the most evidence for when mechanisms led to sustained change and when they did not.
2.5.2 Cross-case analyses.
Once all available evidence for each case had been analysed and coded against the relevant item from the logic model, and key issues resolved, LW and SRB conducted a cross-case comparison. Cases were grouped together, consistent with emerging patterns in outcome and mechanism activations using a ‘pile sort exercise’. We used a realist lens throughout, uncovering contingencies and conditions between component delivery, mechanism activation, outcome attainment, and (if applicable) reasons for disengagement. Both researchers conducted the cross-case comparison independently and then interpretations were brought together and considered reflexively. We discussed each case in-depth to test the rigour of assumptions made and reach consensus when differences were encountered. Concepts were considered and disconfirming ideas and alternative explanations explicitly sought. In this way, theory emerged iteratively concerning the necessary antecedents for sustained engagement and positive outcomes (see [36] for more detail on this process).
For the cross-case comparison we used Bhaskar’s (2016) ‘Description-Retroduction-Elimination-Identification-Correction’ (DREI(C)) procedural analytic method [37]. The procedure examines whether outcomes were achieved and tracks back to consider the multitude of interacting mechanisms and components of the intervention delivered to help us understand how resource offers were responded to (see Box 1 for details on the DREIC procedure).
Box 1. Bhaskar’s (DREIC) analytic procedure
- Description of outcomes - involves noting down observations about a participant’s pathway, tracking the following: a) the resource offers made to participants through the intervention; b) were they able to achieve some of their personalised goals? c) can we see evidence of momentary, short-term effects but this does not result in the achievement of longer-term desired outcomes? d) did participants accept intervention delivery and achieve outcomes but was there little evidence of mechanism activation (e.g. feeling cared-for)?
- Retroduction - assessing the evidence available and moving beyond that which was observable alone, in order to make logical inferences about underlying structures and mechanisms. Retroduction has been defined as a ‘mode of analysis in which events are studied with respect to what may have, must have, or could have caused them… Asking why events have happened in the way they did’. By interrogating the cases using retroductive reasoning we sought the answers to questions such as what led some to sustain engagement and achieve sustained change? Why did those who disengaged do so at the points that they did? What distinguished those that disengaged from those that did not?
- Elimination of competing theories - entails re-examining empirical data to determine which of the competing theories best fits with the available evidence within specific cases and across cases and datasets. Tensions and ambiguities in the data were resolved through iterative discussions, though the evidence in some cases was partial and we were not able to draw a full conclusion.
- Identification of specific mechanisms at work, including negative (or rival logic) effects - involves reviewing the evidence accrued and considering together: a) if sufficient evidence exists in the dataset to draw any conclusions and proceed; b) whether particular mechanisms are required to happen in sequence or parallel with other mechanisms to achieve outcomes; and c) whether there is grading of mechanism offer and emotional response.
- Correction of initial theory and elaboration of refined theory with detailed contextual contingency in relation to operation of mechanisms - the results are compared back to the original hypotheses, the sensitizing concepts from the realist synthesis of the literature, and the descriptive statistics (where available) that detail resource offers made to individuals. We assessed where we had good evidence that particular types of support were likely to activate positive mechanism responses and lead to sustained positive outcomes.
The cross-case comparison culminated in the clustering of the 24 colour-coded case logic models into groups. We clustered cases first on the basis of patterns of intervention delivery and outcomes attained; and secondly based on hypothesised reasoning behind disengagement. Analyses were cumulative and iterative, in line with other qualitative realist evaluations [38] [39], and interpretations were explored with all stakeholders (including practitioners, other researchers, men with lived experiences of leaving prison). Post-analysis, and after the main trial quantitative evaluation, we were un-blinded to the 6-month follow-up CORE-OM scores for the 24 cases (where available). We contrasted these with their baseline CORE-OM scores and the outcomes observed from this qualitative analysis to explore similarities and differences between the quantitative and qualitative assessments of whether the intervention had produced a positive change in individual participants.
3 Results
3.1 Intervention delivery and disengagement
The within-case analyses showed that just over half the cases (13 of 24) received the minimum ‘dose’ of intervention delivery as defined for the statistical per-protocol analysis (i.e. minimum of two pre-release contacts and eight post-release contacts, see Table 1 below). Whilst all 24 cases received the minimum two pre-release contacts (M = 6.9, SD = 3.3) two cases received as little as one post-release contact (M = 12.4, SD = 11.7, range 1–47). This was despite multiple attempts to re-establish contact noted in practitioner logs. By contrast, two cases received as many as 47 post-release contacts (an average of one-two contacts per week over five-six months). Most participants received in the range of five– 15 post-release contacts, with a few dramatic outliers.
Across the 24 case studies, 454 intervention delivery sessions were recorded (166 in prison and 288 in the community). A description about the purpose or content focus of these sessions was available for just 261 of these. Using the coding framework based on the theory of the logic model, the available data evidenced that the majority of sessions were ‘practical’ in nature (total n = 165, 63%) e.g. attempting to source housing, transport to/ from appointments. Only 33 sessions (13%) were coded as being solely ‘therapeutic’ in nature and a further 63 (24%) contained elements of both practical and therapeutic support. There was a general lack of evidence concerning the use of specific therapeutic techniques (such as ‘stop, rewind, explore’ or ‘microslice’) and how participants responded to these.
Once we knew the extent of intervention delivery across the 24 depth case studies, we then used the ‘DREIC’ [37] procedure and pile sort exercise described above to understand what differentiated those who completed the intervention from those who disengaged. This analyses led us to group case studies in 2 ways. First, on the basis of patterns in intervention delivery, mechanism activation, and outcome attainment; and then on the basis of patterns in reasoning underpinning disengagement from the intervention.
3.2 Cross-case patterns in delivery, mechanisms and outcomes
First, we sorted cases by intervention dose and content delivered, mechanisms activated, and outcomes achieved (see Table 2 below).
We found that just five of the 24 cases sustained positive change on the logic model outcomes (pile sort group 5). These cases received the greatest number of intervention sessions post-release (M = 27, SD = 16.9, range 9–46) and the greatest number of therapeutic focused sessions (M = 7.8, SD = 5.1, range = 3–16). By contrast, the other 19 cases disengaged before the end of the intervention and did not sustain positive change. On average they received 7.5 sessions (SD = 5.5, range 1–18) and had fewer therapeutic focused sessions (M = 3.3, SD = 4.6, range 1–16).
A detailed summary of the variations between pile sort groups in terms of intervention delivered, mechanisms activated and outcomes achieved can be found in Table 4 in S3 File. In brief, the evidence suggested that the cases in pile sort group 5 maintained contact with their Engager practitioner and achieved sustained change based on their jointly agreed individualised goals, which included not returning to prison, abstaining from substances, gaining employment, and/ or improving familial relationships. The extent of therapeutic support appeared to differentiate those in this group (who sustained change) from those in groups 1–4 (who did not sustain change). None of the cases who had not engaged appeared to be doing well. These two contrasting experiences are illustrated by the statements below. All participant names have been changed:
“I found it really useful, taking me to probation and helping me to keep to appointments and, you know, being able to be open and honest and talk to him about other things that have gone on and happened and things. So things are really good actually… Staying clean, staying on the right track”
Michael, six-month follow-up
“Didn’t see [the practitioner] many times, like three, four, maybe. Everything just sort of went downhill when I moved in [to the shared house] really because that’s when I started selling drugs”
Anthony, six-month follow-up
3.3 Cross-case patterns in reasoning behind sustained and non-sustained engagement
The content of intervention delivery appeared to differentiate those who sustained change from those who did not, so we then used the DREIC [37] procedure again to look at why cases engaged or disengaged from the intervention, exploring elements of delivery by practitioners, participant response and context. Through the data we interpreted both the level of engagement and the perceived ‘state’ participants were in over the course of the intervention. After doing this, we grouped the 24 cases according to the particular state they were in when they finished the intervention (either through completion or disengagement). Analysis of the groups of cases led to the conceptualisation of five ‘internal states’, representing a data-grounded theoretical explanation of why individuals maintained engagement or disengaged base on how their individual needs were, or were not, adequately met by the intervention. The conceptual states were named: ‘Crises but coping’, ‘Crises and chaos’, ‘Resigned acceptance’, ‘Honeymoon’ and ‘Wilful withdrawal’. They aim to represent the states of thinking, emotion and behaviour each individual was in, in terms of continued engagement or disengagement from the intervention, at the end of intervention delivery. The five states and their distinguishing features are summarised below in Table 3. Worked exemplars illustrating how we arrived at our conclusions can be found in S4 File.
There were no discernible contextual differences between the five ‘crises but coping’ cases who sustained engagement, and the other 19 who did not. The former were no less in need of support in terms of substance use, homelessness and broken relationships and did not necessarily display motivation or confidence in their capacity to change at the outset, but each went on to build trust, maintain engagement with the EP, and achieve positive change. The key determinants of sustained engagement and positive outcomes were identified as:
- The quality of the relationship between practitioner and participant–needed to be based on genuine concern, the building of epistemic trust, unconditional positive regard, and demonstrations of integrity over time;
- The quality of therapeutic work undertaken–needed to address personal goals, and support the participant’s development of confidence and self-belief that their goals were attainable; and
- The presence of a robust shared understanding between practitioner and participant–needed to be genuinely understood by both parties and based on the participant’s own priorities, rather than a generalised assumptions of need.
Importantly, achievement of medium- to long-term outcomes was associated with mechanisms which had been activated more than once or twice, suggesting prison-leavers needed a consistent, tailored intervention, grounded in quality therapeutic content and delivered over time in order to have a sustained effect.
“When you’ve got someone telling you you’re worthless all the time and it’s your fault all the time you start to believe it so you think you are worthless. Until someone says, no Callum*, you’re actually alright… Getting the help from you guys and that, you’re telling me not to give up sort of thing, it does help. Like, I’ve changed the way I’m thinking a little bit. I don’t need to press that f*** it button too quickly because otherwise I’ll just end up back in prison. Housing is probably the main priority at the minute, and the Jobcentre, show someone I’ve got income coming in… I’m more positive about myself [now]” Callum, a ‘crises but coping’ participant, at his 12-month follow-up interview
Lastly, we explored how the qualitative analysis outcomes for these cases compared with the change in their CORE-OM scores from baseline to six-month follow-up (where data was available). We found that our interpretation of what participants and practitioners experienced as a ‘success’ was not always reflected in the change to CORE-OM scores (see additional file 3). Understanding participants’ individual contextual circumstances and ‘internal states’ during the intervention can go some way to explain this discrepancy. As an example illustration, a participant we characterised as being in a ‘honeymoon’ state during early disengagement reported responses to questions in relation to his mental health which meant he had a significantly improved 6-month follow-up CORE-OM score. However, by the time he was interviewed again two months later, he had lost his job, his mental health had regressed, he was back self-medicating, self-harming, and had made a further suicide attempt.
By contrast, a participant we identified as being in a ‘crises but coping’ state at the end of the intervention reported that he was still experiencing a lot of anxiety post-release. This participant reported a non-clinically significant improvement in his six-month follow-up CORE-OM score, but he himself identified that his anxiety was directly related to his ongoing complex circumstances. He was in the process of removing himself from a toxic relationship, continuing to abstain from substances, had not re-offended, and as a result of the therapeutic work with the Engager Practitioner, was managing uncomfortable thoughts and feelings for the first time instead of blocking them out as he had done in the past. While this participant’s overall change in CORE-OM score was negligible, and housing status worse, his personal circumstances from baseline to follow-up were vastly different and he saw himself as having made several sustained positive changes to the trajectory of his life.
3.4 Implementation fidelity
A thorough examination of delivery fidelity to the theoretical model for the whole trial sample will be reported elsewhere. During the analyses reported here, we recorded gaps in intervention delivery for individual cases and then compared patterns of delivery fidelity across cases. We identified evidence of common gaps in certain intervention components being delivered across-cases. These are summarised below.
3.4.1 Overarching site differences.
Team dynamics encountered difficulty where Engager practitioners had different visions of how the intervention should be delivered, for instance, one Engager practitioners’ sense of ‘going the extra mile’ for a prison-leaver, could be interpreted by another as fostering dependency. Risk formulations were fallible as supervisors had different perceptions of risk between the two sites. In one site there was a tendency towards overreliance on historical presentation of risk rather than dynamic present factors. When this translated to there being a requirement for prison-leavers to be seen by two practitioners together, this negatively affected the amount of contact they received. We saw variation between sites in terms of team supervision records and the level of information sharing and planning that took place when an Engager practitioner took a leave of absence. Although they covered for each other at times, in some cases there seemed little information sharing about the particular cases involved meaning that some prison-leavers missed out on valuable sessions because stand-in practitioners weren’t able to progress with any ongoing work. This lack of continuity (in terms of both practitioner and session content) may also have affected prison-leavers’ ongoing engagement because they tended to placed great value on continuity of care. The geographical dispersal of prison-leavers in one site, who were often located in rurally isolated villages, meant that liaison work with other services was more challenging. Similarly, the wider geographical area one team were working in made it more difficult to find a convenient location to meet regularly as a team.
3.4.2 Therapeutic emotional work.
We lacked evidence concerning the use of MBA techniques with prison-leavers. Engager practitioners often weren’t readily able to recall specific examples of times they’d used MBA, and their session note records tended to centre on what the prison-leaver said rather than the therapeutic tools they themselves used to elicit responses. This absence of therapeutic work, and a related lack of shared understanding between practitioner and prison-leaver, appears to underpin disengagement from the intervention for a number of cases. In some instances over the course of the intervention the practitioner appeared to have learnt to rely on generalised assumptions of need pertaining to ‘prison-leavers in general’, such as housing or substance misuse support, rather than ascertaining from the prison-leaver himself what support he might value. Where mutually-agreed upon goals weren’t specified before a prison-leaver’s release, and achievable steps to attain said goals under-developed, the prison-leaver discontinued engagement with the intervention and their capacity to sustain positive change was undermined.
3.4.3. Maintaining community contact.
Sustaining contact with prison-leavers post-release was another area of difficulty for practitioners. When a prison-leaver was released to some form of housing (including private, temporary or supported), practitioners had a direct line of contact to them after their release; and this remained true even when they didn’t have a telephone or weren’t attending probation. However often prison-leavers were released to no fixed abode, with no personal telephone, and no requirement to attend probation regularly. When this happened the practitioner struggled to find a means of contacting the prison-leaver and by the time they did, the prison-leaver was often back in prison after returning to maladaptive patterns of behaviour, and their time on the intervention had elapsed.
3.4.4. Family and friends liaison.
Only in a handful of cases did practitioners have direct contact with family and friends. Only in one or two cases did the prison-leaver’s family feel directly supported by the practitioner. Our implementation data suggests that practitioners saw contact with the family as something which helped offer another perspective on the prison-leaver, and a context for understanding of some of their issues and challenges. Contact with the family meant the practitioner could better assess the support the prison-leaver could draw on, or the dysfunctional and tenuous relationships which might undermine their success in the community. Work around family and friends may be less direct and more about emotional work with the prison-leaver to help them navigate their way through difficult relationships. This component perhaps wasn’t always apparent in our data because the prison-leaver had little or no familial or other support to draw from. Prison-leavers sometimes would talk about regaining contact with family and on other occasions they would explain that there had been irreparable damage and familial ties had been severed.
3.4.5. Handling a good ending to the intervention.
So many of the cases disengaged before they reached the end of the intervention so establishing whether there was consistency in how prison-leavers experienced the ending was difficult. Clear documentation regarding the transfer of support onto another worker was evident in only 3 of the 24 cases we looked at; and in all 3 cases the contact arrangements were often loose and short term (for example support while living in temporary accommodation). In some cases, either there was no ongoing services working with the prison-leaver, or there was no identifiable key worker that the practitioner could assign their case too. For others, they had disengaged earlier on and didn’t have a formal ending. This absence of appropriate hand-over support lends itself to the idea of looking for cost-effective ways to ‘leave the door open’ past the end of the intervention so that prison-leavers aren’t left in a void of support.
The standards for reporting qualitative research (SRQR) was used to guide the reporting of qualitative findings (see S5 File).
4 Discussion
The findings revealed that around half the sampled cases received the minimum ‘dose’ of intervention delivery; and although practical support was delivered on many occasions, this was insufficient to generate sustained positive change for many participants. The most profound changes were observed in participants who sustained engagement and received more therapeutic support; reaching a state of ‘crises but coping’. They had developed their capacity to think, and to understand their circumstances while striving to improve them. By comparison, the other participants, who had less therapeutic support and disengaged before they had developed their capacity to mentalise effectively, did not achieve the same status of ‘coping’.
‘Crises and chaos’ prison-leavers, despite initial positive responses, were unable to self-regulate their emotions and found it difficult to think clearly about how to effect change. Those with ‘resigned acceptance’ may be understood as feeling ‘hopeless’, defined as a system of belief characterised by negative expectations for the future; and perceived helplessness in one’s own capacity to change such anticipated outcomes [40, 41]. Similarly, ‘wilful withdrawal’ prison-leavers felt unable to confront their difficulties, opting to actively ‘escape’ and return to previous coping behaviour patterns. By contrast, ‘honeymooners’ displayed an apparent initial competence to cope with life which masked the true insecure reality of their situations. This was further compounded by an inability to communicate their vulnerability to others, resulting in their needs going unheard and their circumstances unchanged.
4.1 Understanding intervention disengagement
To engage in and benefit from treatment, offenders may need to possess certain cognitive, emotional, volitional, and behavioural properties, and exist in contexts where changes are possible and supported [21, 42–44]. For participants unable to sustain engagement, there were incompatibilities between their internal state, the intervention, and the external context at the point they took part. ‘Crisis’ was not a predictor of disengagement. Indeed, the five ‘crises but coping’ prison-leavers were still experiencing substantial social crises after the conclusion of the intervention. This demonstrates that, when delivered well, it was possible to achieve change in this population within this setting. Ensuring delivery fidelity was, however, an ongoing challenge. As reported above, therapeutic emotional work was often observed to be missing or inconsistently delivered. The psychological aspects of delivery which appeared to be most prone to delivery failure, in what were extremely challenging conditions for practitioners, were developing a true ‘shared understanding’ and routinely applying mentalisation based approaches.
To the best of our knowledge, the ‘crises but coping’ participants had no discernibly distinguishable features at recruitment to the trial. Their social environments, offending history, personal circumstances, and motivation to change, were similar to the other participants. It is plausible that participants who were categorised as displaying chaos, honeymoon, resigned acceptance or wilful withdrawal characteristics might have been able to achieve a ‘crises but coping’ state if they had received more, and better, therapeutic support. We were not able to measure all potential contextual features affecting disengagement. However, while reasons behind participants’ disengagement were highly personal and individual, each disengagement reflects a failing in the intervention theory and/ or delivery to adequately address their needs.
Through understanding the circumstances that led to disengagement, we firstly proposed enhancements to make the intervention more implementable. For example, the therapeutic mentalisation-based approach was intended to underpin all elements of delivery including building trust and rapport, developing a shared understanding, co-creating a shared action plan and mobilising resources to support meeting goals. However, our analysis demonstrated that delivery of ‘therapeutic’ support was often lacking or artificially delineated from the rest of the intervention in ways that were not intended. This highlights the need for clear and accurate operationalisation of an intervention theory, transmitted effectively to those responsible for delivery, in order to ensure that misinterpretations do not lead to inconsistent or flawed delivery. Further enhancements to training and supervision in terms of quality, intensity and frequency may also be required when supporting experienced but clinically unqualified practitioners in complex environments.
Secondly, aspects of the intervention theory may need to be improved. Practitioners need to be equipped with varied skills and methods of approach to re-engage participants from diverse groups whose commitment to an intervention may vary over time. Good therapeutic work is essential for sustained engagement and extended contact time within the protected structure of the prison environment may provide a valuable opportunity for practitioners to build trust and develop a true shared understanding with the participant. For example, multiple prison contacts offer practitioners the opportunity to get beneath the ‘honeymooner’ façade of coping, and time to demonstrate trustworthiness to those at risk of ‘wilfully withdrawing’. Prison contacts may also provide practitioners with an opportunity to develop a sense of control in someone with ‘resigned acceptance’ and can help build the thinking capacity of a ‘crises and chaos’ prison-leaver before they are faced with pressures of living in the community and associated heightened unregulated emotions. Dealing with such a range of psychological challenges was not fully anticipated and we propose that the theory, manual and training should reflect the knowledge and skills needed to support the range of internal states likely to be encountered.
4.2 Challenges of evaluating complex interventions in complex systems
Delivering complex person-centred interventions with fidelity (as intended, with engagement) [45–48] has repeatedly been shown to be hard to achieve [49–52]. Deviations from adherence to the intervention model are particularly likely in cases like Engager, whose model is broad in scope, non-linear in nature, and includes multiple component parts [7]. Interventions situated within wider complex social systems raise further challenges, because of the multitude of interdependent elements which can affect each other and the system as a whole [53]. The criminal justice system itself may be considered one of the most complex systems to conduct research in [54].
Mapping out intervention components is a method of increasing robustness of fidelity testing, a process adopted in this study. However, the intervention’s flexible, person-centred, structure meant that component delivery was not intended to be uniform across the study population and the absence of certain components may have resulted from deliberate omission rather than non-adherence to the model. Our case study approach, with data covering multiple perspectives over time, facilitated interpretation as to whether intervention delivery occurred and was responded to as intended. Limitations included the relatively small number of cases and inevitable attrition of prison-leavers from the research process resulting in some partial case studies and incomplete sources of evidence. This highlights a particular evaluation difficulty in terms of delineating between evidence of limited delivery versus limited evidence of delivery.
This process evaluation needs to be considered alongside the other analyses forming the parallel process evaluation and quantitative evaluation of the RCT. Additionally our depth case study approach raises important questions about the value and reliability of standard outcome measures such as the CORE-OM in the evaluations of complex interventions. It has been argued that such measures may be considered inadequate in complex intervention evaluations because they do not account for individual contexts, subjective perceptions of success, or unpredictable idiosyncratic individualised changes arising from person-centred interventions [55, 56]. For example, in this sample of Engager participants some demonstrated no ‘clinical recovery’ in their psychological distress, according to the Engager primary trial outcome measure [33] and yet appeared to be to some extent on the path to recovery in the process evaluation; for others the reverse was true. ‘Recovery’ defined as ‘a deeply personal, unique process of changing one’s attitudes, values, feelings, goals, skills, and/ or roles’ [57] is highly subjective and variable, so perhaps the discrepancy is not surprising. As such, we suggest that outcome measures such as the CORE-OM may provide a snapshot image of how participants are in a particular moment in time, but they do not always reveal the underlying complexity of an individual’s response to an intervention. The process of change and development of resilience may not necessarily be captured in discrete trial outcome measures, and here the CORE-OM may not have been sensitive enough to detect the early changes in participants’ thought processes and coping strategies during the relatively short time frame of the study. Understanding the multi-faceted nature of individuals’ responses to an intervention is crucial for understanding how to optimise its future implementability; and we would argue also provides an alternative lens on effectiveness.
5 Conclusions
This paper demonstrates that understanding delivery of a complex intervention, and participants’ responses to its component parts, provides useful lessons in how to make person centered (flexible) complex interventions implementable. The qualitative methodology provided deeper insights and understanding than the more binary measures of efficacy recorded in the quantitative evaluation of the RCT. This demonstrates the limitations of relying on, or prioritizing, quantitative outcome measures when assessing the success of intervention implementation. By examining programme theory in relation to delivery we found that rapport, while a necessary precursor for therapeutic work, was not a guarantee of longevity of Engager practitioner and participant relationships post-release. Core delivery mechanisms often had to be ‘triggered’ numerous times to produce sustained change. In addition, implementation failure in the shape of inconsistent or missing therapeutic work, brought with it missed opportunities for change. By contrast, when delivered as intended with a robust shared understanding developed using a mentalisation-based approach, some cases demonstrated positive sustained change while managing ongoing personal and social difficulties. Prior motivation to change was not found to be a pre-requisite for success; individuals’ readiness and commitment to change ebbed and flowed. Thus compatibility between practitioner, participant, and setting are continually at risk of being undermined by implementation failure as well as changing external circumstances and participants’ own weaknesses. Furthermore, implementation of person centred interventions are at risk of failing, if practitioners do not adequately continually adjust to individuals’ needs. To address this, and change from a requirement for standardised delivery, practitioner teams are likely to require enhanced training and regular robust clinical supervision to help them sustain good practice. Implementation of complex care interventions to those with even modest levels of distrust must account for and connect to participants’ priorities, grounded in a relationship underpinned by epistemic trust, in order for them to be more widely implementable.
Supporting information
S2 File. Process evaluation participant data sources.
https://doi.org/10.1371/journal.pone.0270691.s002
(DOCX)
Acknowledgments
We would like to thank the individuals who received the Engager intervention and participated in the research and the practitioners in both sites who delivered the intervention in often challenging situations. We would like to thank Laura Gill for providing management and administrative support to submit the paper. We would also like to thank and acknowledge researchers [Lauren Carroll, Rebecca Greer, Caroline Stephenson and Roxanne Todd] that worked tirelessly in the background supporting us with recruitment and data collection.
References
- 1. Dobson D., Cook T.J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3, 269–276.
- 2. Hasson H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science, 5, 67. pmid:20815872
- 3. Glasziou P., Chalmers I., Altman D.G., Bastian H., Boutron I., Brice A., et al. (2010). Taking healthcare interventions from trial to practice. BMJ, 341, 384–387.
- 4. Campbell M., Fitzpatrick R., Haines A., Kinmonth A.L., Sandercock P., Spiegelhalter D., et al. (2000). Framework for design and evaluation of complex interventions to improve health. BMJ, 321, 694–696. pmid:10987780
- 5. Bradley F., Wiles R., Kinmonth A.L., Mant D., & Gantley M. (1999). Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP). The SHIP Collaborative Group. BMJ, 318, 711–715. pmid:10074018
- 6. Hawe P., Shiell A., Riley T., Gold L. (2004). Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. Journal of epidemiology and community health;58(9):788–93. pmid:15310806
- 7. Carroll C., Patterson M., Wood S., Booth A., Rick J., & Balain S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. pmid:18053122
- 8. Kouyoumdjian F.G., McIsaac K.E., Liauw J., Green S., Karachiwalla F., Siu W., et al. (2015). A systematic review of randomized controlled trials of interventions to improve the health of persons during imprisonment and in the year after release. American Journal of Public Health, 105, 13–33.
- 9.
Brooker C., Repper J., Beverley C., Ferriter M., & Brewer N. (2002). Mental health services and prisoners: a review, school of health and related research ScHARR. Sheffield: University of Sheffield.
- 10. Rotter M., Way B., Steinbacher M., Sawyer D., & Smith H. (2002). Personality disorders in prison: aren’t they all antisocial? Psychiatric Quarterly, 73, 337–49. pmid:12418360
- 11. Fazel S., & Danesh J. (2002). Serious mental disorder in 23000 prisoners: a systematic review of 62 surveys. Lancet, 359, 545–50. pmid:11867106
- 12.
Singleton N., Meltzer H., Gatward R., Coid J., & Deasy D. (1998). Psychiatric morbidity among prisoners in England and Wales. London: The Stationery Office.
- 13.
National Statistics. Safety in custody statistics bulletin, England and Wales, deaths in prison custody to December 2016, assaults and self-harm to September 2016. London: Ministry of Justice.
- 14. Howerton A., Byng R., Campbell J., Hess D., Owens C., & Aitken P. (2007). Understanding help seeking behaviour among male offenders: qualitative interview study. BMJ, 334, 303–268. pmid:17223630
- 15. Byng R., Weyer Brown C., Sheaff R., Samele C., Duggan S., Harrison D., et al. (2012). Care for offenders: continuity of access, NIHR service delivery and organisation programme SDO project, 210.
- 16.
Stewart D. (2008). The problems and needs of newly sentenced prisoners: Results from a national survey. London: Ministry of Justice.
- 17.
Williamson M. (2006). Improving the health and social outcomes of people recently released from prisons in the UK. A perspective from primary care. London: The Sainsbury Centre for Mental Health.
- 18. Thomas E.G., Spittal M.J., Heffernan E.B., Taxman F.S., Alati R., Kinner S.A. (2016). Trajectories of psychological distress after prison release: implications for mental health service need in ex-prisoners. Psychological Medicine, 46, 611–621. pmid:26549475
- 19. Fazel S., Hayes A.J., Bartellas K., Clerici M., & Trestman R. (2016). Mental health of prisoners: prevalence, adverse outcomes, and interventions. Lancet Psychiatry, 3(9):871–881. pmid:27426440
- 20. Michie S., van Stralen M. M., & West R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6, 42. pmid:21513547
- 21.
Michie S., Atkins L., & West R. (2014). The behaviour change wheel—a guide to designing interventions. London: Silverback Publishing.
- 22. Byng R, Quinn C, Sheaff R, Samele C, Duggan S, Harrison D, et al. (2012). COCOA: care for offenders, continuity of access. Final report NIHR Service Delivery and Organisation programme.
- 23. Pearson M, Brand SL, Quinn C, et al. (2015) Using realist review to inform intervention development: Methodological illustration and conceptual platform for collaborative care in offender health. Implementation Science 10: 134. pmid:26415961
- 24. Brand S.L., Quinn C., Pearson M., Lennox C., Owens C., Kirkpatrick T., et al. (2018). Building programme theory to develop more adaptable and scalable complex interventions: Realist formative process evaluation prior to full trial. Evaluation.
- 25. Fuggle P.B., Bevington D., Cracknell L., Hanley J., Hare S., Lincoln J., et al. (2014). The adolescent mentalisation-based integrative treatment (AMBIT) approach to outcome evaluation and manualization: adopting a learning organization approach. Clinical Child Psychology and Psychiatry, 17. pmid:24595808
- 26. Kirkpatrick T., Lennox C., Taylor R., Anderson R., Maguire M., Haddad M., et al. (2017). Evaluation of a complex intervention (Engager) for prisoners with common mental health problems, near to and after release: study protocol for a randomised-controlled trial. BMJ Open, 8.
- 27.
Pawson R., & Tilley N. (1997). Realistic Evaluation. London: Sage.
- 28. Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Social Science & Medicine. 2012;75(12):2299–306. pmid:22989491
- 29. Blackwood B., O’Halloran P., et al. (2010). On the problems of mixing RCTs with qualitative research: the case of the MRC framework for the evaluation of complex healthcare interventions. Journal of Research in Nursing, 15(6), 511–521.
- 30.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions. London: Medical Research Council; 2008.
- 31.
Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: UK Medical Research Council (MRC) guidance. London: MRC; 2014.
- 32. Byng R., Kirkpatrick T., Lennox C., Warren F.C., Quinn C., Anderson R., et al. (submitted to The Lancet). Evaluation of a complex intervention for prisoners with common mental health problems, near to and after release: The Engager randomised controlled trial.
- 33. Evans C., Mellor-Clark J., Margison F., Barkham M., Audin K., Connell J., et al. (2000). Clinical outcomes in routine evaluation: The CORE-OM. Journal of Mental Health, 9, 247–255.
- 34. Vasileiou K., Barnett J., Thorpe S., et al. Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol 18, 148 (2018). pmid:30463515
- 35. Manzano A. (2016) The craft of interviewing in realist evaluation. Evaluation, 22 (3). pp. 342–360.
- 36. Rybczynska-Bunt S, Weston L, Byng R, et al. Clarifying realist analytic and interdisciplinary consensus processes in a complex health intervention: A worked example of Judgemental Rationality in action. Evaluation. 2021;27(4):473–491.
- 37.
Bhaskar R (2016) Enlightened Common Sense: The Philosophy of Critical Realism. London: Routeledge, 30–35.
- 38. Wand T., White K., & Patching J. (2010). Applying a realist(ic) framework to the evaluation of a new model of emergency department based mental health nursing practice. Nursing Inquiry;17:231–9. pmid:20712661
- 39. Wand T., White K., & Patching J. (2011). Realistic evaluation of an emergency department based mental health nurse practitioner outpatient service in Australia. Nursing Health Science,3:199–206. pmid:21595814
- 40. Beck A. T., Steer R. A., Kovacs M., & Garrison B. (1985). Hopelessness and eventual suicide: A 10-year prospective study of patients hospitalized with suicidal ideation. American Journal of Psychiatry, 142, 559–563. pmid:3985195
- 41. Duke N.N., Borowsky I.W., Pettingell S., & McMorris B.J. (2011). Examining Youth Hopelessness as an Independent Risk Correlate for Adolescent Delinquency and Violence. Maternal and Child Health Journal, 15, 87–97. pmid:20012345
- 42. McMurran M., & Ward T. (2004). Motivating offenders to change in therapy: An organizing framework. Legal and Criminological Psychology, 9, 295–311.
- 43. Ward T., & Stewart C. A. (2003). The treatment of sex offenders: Risk management and good lives. Professional Psychology, Research and Practice, 34, 353–360.
- 44. Ward T., & Brown M. (2004). The good lives model and conceptual issues in offender rehabilitation. Psychology, Crime, and Law, 10, 243–257.
- 45. Dusenbury L., & Hansen W. B. (2004). Pursuing the course from research to practice. Prevention Science, 5, 55–59. pmid:15058913
- 46. Greenhalgh T., Robert G., Bate P., Kyriakidou O., Macfarlane F., & Peacock R. (2004). A systematic review of the literature on diffusion, dissemination and sustainability of innovations in health service delivery and organisation. How to spread good ideas, 1–424.
- 47. Dane A.V., & Schneider B.H. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical Psychology Review,18, 23–45. pmid:9455622
- 48. Dusenbury L., Brannigan R., Falco M., & Hansen W.B. (2003). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256. pmid:12729182
- 49. Hardeman W., Michie S., Fanshawe T., Prevost A.T., McLoughlin K., & Kinmonth A.L. (2008). Fidelity of delivery of a physical activity intervention: Predictors and consequences. Psychological Health, 23, 11–24. pmid:25159904
- 50. Michie S., & Abraham C. (2008). Advancing the science of behaviour change: a plea for scientific reporting. Addiction, 103, 1409–10. pmid:18783495
- 51. Lorencatto F., West R., Bruguera C., & Michie S. (2014). A method for assessing fidelity of delivery of telephone behavioral support for smoking cessation. Journal of Consulting and Clinical Psychology, 82, 482–491. pmid:24294836
- 52. Walton H., Spector A., Tombor I. & Michie S. (2017). Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: A systematic review of measure quality. British Journal of Psychological Society, 22, 872–903.
- 53. Lee B.Y., Bartsch S.M., Mui Y., Haidari L.A., Spiker M.L., & Gittelsohn J. (2017). A systems approach to obesity. Nutrition Reviews; 75 (suppl 1): 94–106. pmid:28049754
- 54.
Martin C. (2000) ’Doing research in a prison setting’, in Jupp V., Davies P., & Francis P. (Eds.) Doing criminological research (p. 215–233), London/California/Delhi: SAGE Publications.
- 55. Kaplan R. M. and Irvin V. L. (2015) Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS ONE 10(8): e132382. pmid:26244868
- 56. Greenberg M. T. and Abenavoli R. (2017) Universal interventions: fully exploring their impacts and potential to produce population-level impacts. Journal of Research on Educational Effectiveness 10 (1), 40–67.
- 57. Anthony W.A. (1993). Recovery from mental illness: the guiding vision of the mental health system in the 1990s. Psychosocial Rehabilitation Journal, 16, 11–23.