Objectives

The primary objective of this review was to assess the effects of psychological training and consultation on probation practice and risk management outcomes in relation to both probation staff and the individuals under their supervision. The primary review questions were:

What impact has the OPD Pathway core community delivery had on the:

1. Professional practice approaches and attitudes of probation staff working with individuals screened in to the OPD Pathway?
2. Estimated risk of harm and reoffending or recall rates for those screened into and accessing OPD core community services?
3. Psychological wellbeing and relationships of those screened into and accessing OPD core community services and of the staff managing these individuals?

Protocol registration

The scope of this review was to assess literature that addressed the established outcomes of the OPD Pathway programme [12]. As these outcomes are freely accessible within the OPD Pathway Strategy that was published in 2015, a protocol was not registered for this systematic review. While the absence of a protocol means there is no a-priori public record of the methodological approach, the detail presented in this section provides a transparent account to ensure the ethical integrity of the review in the absence of such a document.

Eligibility criteria

In order to be eligible for inclusion in the review, papers were required to:

• Address the OPD Pathway core community delivery.
• Report impact (positive, neutral or negative) on the outcomes identified in 1-3 above.
• Be published in English.
• Have a UK affiliation (as the OPD Pathway exists only across England & Wales).

Papers were excluded if they:

• Were based within custodial settings, forensic mental health services (e.g., Medium Secure Units) or if they were resettlement OPD Pathway interventions;
• Did not address impact (e.g., implementation, process) or if the outcomes considered were outside the scope of this review (e.g., satisfaction with service delivery or access to additional services);
• Were not primary research (e.g., reviews or book chapters), anecdotal evidence of impact (i.e., opinion pieces or author view rather than impact for which qualitative or quantitative data were reported).

Information sources

The electronic databases Scopus (which included full coverage of MEDLINE, EMBASE and Compendex) and EBSCO were searched for the period 01.01.2012 (national introduction of OPD) to 19.10.2022. Additionally, stakeholders in the OPD Pathway (i.e., central leads, regional commissioners, and thematic leads) were contacted for published material that may not have been identified through the search. Four additional titles were suggested by the stakeholders consulted.

Search strategy

Two separate searches were run to address the two different population groups covered in this review (offenders and probation staff), as detailed in Table 1 with affiliation country limited to UK.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Search terms utilised for systematic review.

https://doi.org/10.1371/journal.pone.0318428.t001

Selection process

A population-exposure-outcome (PEO) algorithm was employed to guide the selection process:

Data collection process

To extract and collate data from published literature, an Excel file was designed to capture all relevant fields and to manage referencing. No additional software or systems were employed due to restrictions imposed on adding software to government computers. A detailed extraction chart (see Supplemental Materials - DOI 10.17605/OSF.IO/RZ8A5) was developed to ensure complete and systematic extraction of all relevant data from all included records, with testing of chart fields carried out on the 10 calibration articles. All data were extracted by the first and second authors and a selection of extract records was reviewed by a further reviewer. Where information was lacking in final articles, and where it was unclear whether a study related to the OPD Pathway, article authors were contacted for further information (N = 1). Articles that included subject matter beyond impact (e.g., processes or implementation of services) or unrelated to the specified outcomes had only review-relevant data extracted.

Data items

The specific outcomes for this review were drawn directly from the overarching OPD Pathway programme policy documentation [12]. These outcomes were:

• Workforce: competence, confidence, psychologically informed practice, improved risk management, improved attitudes towards PD;
• Wellbeing: (specific to service-users) interpersonal skills, thinking skills, coping strategies, prosocial behaviour, happiness, reduced self-harm; (staff-specific) burnout, job satisfaction;
• Risk and reconviction: risk assessments, violent behaviour, recalls, reoffending;
• Relational improvement (in relation to service-users or staff): between staff and service users, peer relations, family relationships.

Changes in recall rates were categorised as a workforce outcome, indicating changes in the use of punitive risk management practices, and as a risk outcome indicating changes in reoffending or offence-paralleling behaviour [15]. Acknowledging that the dynamics of multiple external factors likely affected the interplay between these interpretations of recall rates, findings below were reported in line with their presentation in respective studies.

In addition to outcomes, the contextual variables extracted were: participant demographics (age, gender, offence type {for service users}, time in service {for staff}), participant type (staff or service user), number of participants, group characteristics (where applicable), intervention description, and intervention duration. Research details were also extracted including research design, overall aims, measures used, measurement frequency, analysis used, follow-up duration, alpha values (for psychometrics), themes identified (for qualitative studies), mean difference/change, effect size (for quantitative studies), and direction of effect. Where contextual information was absent or limited, this was acknowledged in the description of findings.

Owing to the highly heterogenous nature of the included studies and the numerous outcomes explored, a formal statistical synthesis of findings was not possible as part of this review. Relatedly, while impact is difficult to definitively evidence through qualitative data, findings relating to relationship outcomes were largely of a qualitative nature. A narrative synthesis approach was thus adopted to summarise the data in relation to each overarching outcome explored.

Study risk of bias assessment

It was noted that the majority of included studies were undertaken and reported by those commissioned to deliver services. Owing to the nature of the OPD Pathway programme, with few areas having access to dedicated research and evaluation resources, it was necessary to deem this as an acceptable risk of bias, as to exclude studies on this basis would result in little or no material to review. To formally assess risk of bias, the Mixed Methods Appraisal Tool (MMAT [16]) was utilised as this offers a method-specific approach to the critical appraisal of study design and methodology. Through the use of screening questions (S1 - Are there clearly stated research questions?; S2 Do the collected data allow to address the research question?), papers were deemed worthy of critical appraisal if the answer to either or both screening question was “yes”. The four studies failing to meet these criteria were not graded but were retained in the review (see below). The individual appraisals for each paper are presented in the Findings section, with individual method-specific question responses providing insight to decisions made.

Effect measures

Of the 12 studies utilising a quantitative or mixed methods approach, six employed a range of measures of effect size, reflecting the heterogeneity in research design and analytic approach. In relation to workforce outcomes (e.g., competence, confidence), effects were indicated through the use of Cohen’s d, Cohen’s f, partial Eta squared, and the coefficient of determination (r). In relation to wellbeing, two quantitative studies addressed burnout but only one stated a measure of effect, utilising Cohen’s d. Risk and reconviction rates were measured in five quantitative studies, three of which indicated measures of effect. Cohen’s d, Cohen’s f, and partial Eta squared were provided in these studies. One study utilised a measure of relational improvement once between an intervention and control group, indicating the degree of effect through the coefficient of determination (r). Of the 12 studies incorporating measures of statistical effect in their analyses, the predominant approaches to analysis were tests of group difference (t-tests and ANOVAs). The analyses and effect measures used in each study are presented in tabular form in the Findings section.

Certainty assessment

In order to evaluate the quality of the evidence contained within each of the included studies, the Mixed Methods Appraisal Tool (MMAT [16]) was applied to each article. For each study type, assessment outcomes were reported as a count of the total of five respective criteria. In relation to screening criterion 1 (Are there clear research questions), articles that reported clear aims or hypotheses in place of research questions were deemed to have met this criterion. The final column in Table 2 indicates the study type (1 – Qualitative; 2 – Quantitative RCT; 3 – Quantitative non-randomised; 4 – Quantitative descriptive; 5 – Mixed Methods) and the total score achieved by those meeting MMAT screening criteria. A score value of NG in this column indicates that the article was not graded due to not meeting initial MMAT screening criteria with further detail of the critical appraisal provided in Table 3. In addition to the four articles that did not meet the MMAT screening criteria required to be graded, seven articles met only two or fewer of the five criteria relevant to their study design. However, all 20 articles were included in this review in order to present as complete a picture as possible of the potential impact of this multifaceted programme of interventions. It should be noted that the data available for analysis in these studies have largely been of a ‘real world’ nature, necessitating convenience sampling of staff and service users operating in dynamic systems where interruptions to organised activity, high turnover and cross-deployment are commonplace. As such, while quality may fall short of the highest scientific standards, the difficulties of conducting research in this field suggest that any insights into the effective functioning of these services may still contribute to the broader evidence base.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Overview of studies included in systematic review including quality appraisal rating.

https://doi.org/10.1371/journal.pone.0318428.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Risk of bias and critical appraisal of studies as demonstrated through MMAT assessment.

https://doi.org/10.1371/journal.pone.0318428.t003

Study selection

As shown in Fig 1, there were 207 published articles initially identified in electronic database searches, with key stakeholders of the OPD Pathway programme suggesting a further four articles that did not appear in the search results. Once duplicate entries were removed, all 174 articles were screened according to the criteria set out above, resulting in a final set of 20 papers for inclusion. The majority (n = 19) of these articles were identified in electronic databases with one being identified through the stakeholder route. There were 43 papers whose full-text content precluded inclusion in this review due to inappropriate study population, setting, or service type.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flow diagram of article search and selection process.

https://doi.org/10.1371/journal.pone.0318428.g001

Study characteristics

Several studies explored the impact of the combination of all components of delivery in tandem, while others attempted to isolate component parts for evaluation. Most frequently isolated components were those of consultation and formulation, however several studies claiming to focus on these also made reference to other components, such as training or group-based supervision. Additionally, as a written formulation is commonly a product of a psychological consultation, many studies treated these concepts as interconnected elements of the service (i.e., a single component). As a result of these anticipated overlaps in component parts of core community delivery, findings from this review were grouped and synthesized according to outcomes rather than to the elements of intervention in order to avoid obvious contamination effects. Consequently, results were grouped first by the outcome explored (workforce (n = 17); wellbeing (n = 4); risk and reconviction (n = 8); and relational improvement (n = 6)), and then by participant group to which they referred (e.g., staff (n = 13), offenders (n = 3), or both (n = 4)). Summary data for each study is presented in Table 2 in alphabetical order by author and MMAT scores are presented in the final column.

Workforce outcomes

Risk and reconviction

Relationships

Certainty of evidence (limitations).

Overall, the quality of evidence presented in the studies included in this review was not of a consistently high standard, as evidenced in the critical appraisal presented in Table 3. Studies failing to meet screening criteria were not scored, as indicated by a ~  in the relevant cells. Scores on the MMAT tended towards low to medium, with only one paper reaching a score of 4/5 and one qualitative paper meeting all five criteria relevant to their study type. For qualitative studies, while approaches used to analyse and gather data were appropriate, several studies failed to meet the criterion of adequately deriving findings from the data. This may have been affected by journal word limits preventing thorough representation of the data, however there were some clearly tenuous interpretations made. The single RCT included in the review failed to meet most criteria, largely due to an absence of reported processes and figures, coupled with a high attrition rate. Of the seven non-randomised quantitative studies assessed, none met the criterion of accounting for confounders in the design and analysis and four failed to demonstrate that the intervention was administered as intended. These studies’ strengths were in relation to appropriateness of data collection and measurement. The two descriptive studies reviewed differed in quality, but both failed to demonstrate a low risk of nonresponse bias and it was not possible to tell if the appropriate analysis methods were used. Finally, one mixed methods study was found to have not met quality standards for each methodological component and the integration of qualitative and quantitative findings were difficult to interpret.

Most quantitative studies appeared to be under-powered (this was not always reported but could be inferred from results) and suffered from high attrition rates. Where significant differences were identified, analyses were often based on small sample sizes or convenience samples (who may not have been representative of the wider population under study, even at a regional level). Additionally, the use of unvalidated psychometrics or bespoke qualitative tools to measure key outcomes warrants caution in interpreting the impact of the intervention under scrutiny, particularly where items were phrased over-simplistically (e.g., “I understand how personality disorder can be linked to offending”). Impact on outcomes reported in many of these studies may be better understood as of a perceived nature rather than as directly evidenced. Additionally, it was possible that post-training responses reflected compliance with training intentions rather than actual learning or development on the part of participants.

The use of historic records to match groups, while necessitated by an absence of contemporary controls, reduced the robustness of the comparisons made. The continuously changing policies governing probation supervision processes and activities suggests historic comparison groups may differ in important ways. Follow-up periods in some studies were quite short or contaminated by exposure to additional elements of service delivery, limiting the scope of impact or strength of comparisons made. Weak design and poorly applied analytical methods meant that methodological rigour was also questionable in several studies. In relation to qualitative findings, all interview and focus group studies relied on self-selected participants, thus the likelihood of positive response biases was high. Those disinclined to engage with the evaluation process may have been those who saw no benefit in the intervention or who did not access interventions offered. Similarly, papers utilising case studies presented somewhat anecdotal evidence, with limited consideration of extraneous factors which may have accounted for the outcomes observed. However, many authors did recognise significant limitations and warned against over-interpretation of findings, instead acknowledging potential directions for further, better designed and more controlled research. While the difficulties of successfully conducting real world research are acknowledged [41], there are a number of useful approaches that practice-based researchers can consider to help mitigate against these pitfalls. For example, embedding well-validated outcome measures into routine practice prior to an intervention can help to ensure measures used are robust and fit for purpose while also enabling a larger and more representative participant sample to be achieved. Maintaining control groups for the duration of an evaluation can ensure useful comparison are made although it is recognised that approaches such as a stepped wedge design may be practically and ethically more appropriate and feasible. Where small-scale designs such as case studies are used, it is important to ensure that rigorous, validated methods are adopted for case study [42] or single case [43] research. Finally, utilising an appropriate research design to fit with the available data and resource can strengthen the overall quality of evaluation undertaken.

In assessing sensitivity of this review to achieved outcomes, a brief sensitivity analysis comparing high- and low-quality studies was conducted. Nine studies achieved medium or high MMAT scores while 11 scored low or could not be graded due to screening criteria. Table 4 presents a comparison of the direction of effect reported in high- vs low-quality studies in relation to each overarching outcome. In relation to workforce outcomes for medium-high scoring studies, some indicated improved competence and a greater understanding of personality, while others noted no or partial effects of training on specific outputs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Summary table of brief sensitivity analysis.

https://doi.org/10.1371/journal.pone.0318428.t004

Gaps

Across the literature reviewed, notable gaps in the outcomes explored were evident. Workforce outcomes were the most frequently explored, however the quality of studies was low to moderate and, as noted above, unvalidated and potentially biased measures were used to identify changes in competency and in risk management. This would suggest more exploration of this key outcome is warranted. It may be particularly useful to move away from simplistic measures such as increases in PD knowledge and instead explore what features of OPD Pathway community delivery made a useful difference to OM practice. For example, introducing standardised measures of psychologically-informed risk management and relational approaches would be helpful. Work would need to be conducted to compile a selection of validated, fit for purpose measures however, given the issues of measurement identified across the studies in this paper, such an endeavour would be worthwhile. In addition, and given the nuances in OPD delivery (i.e., individually focussed and needs-led based on a case-specific consultation and formulation), more rigorous and replicable qualitative approaches using validated and robust case study, single case and ‘small n’ designs need to be utilised.

In terms of risk management, understanding the mechanisms of change in relation to therapeutic risk taking and what effect this may have on probation supervision processes may help inform service developments. Relatedly, Fitzgibbon et al., [45] noted the importance of integrating clinical judgments with actuarial measures on risk within probation settings in order to ensure an individualised assessment is achieved. One key takeaway from studies demonstrating workforce improvements was that greater exposure to OPD Pathway delivery appeared to achieve greatest impact, thus a particularly useful area of further study would be to more closely examine this potential “dose effect”. It is possible that greater exposure has a greater impact through a broader range of practices being shared and developed over time. For example, a wider array of relational skills use amongst probation staff has been found to lower reconviction rates [46]. It would thus be useful, for example, to understand whether the impact of greater OPD Pathway exposure was related to particularly effective components in relational working espoused by this programme or to individual staff availing of multiple skills-based offerings.

In relation to risk and reconviction, direct impact was not rigorously evaluated in the studies included in this review, although it is appreciated that outcomes of this nature require lengthy study periods in order to allow for appropriate time for relevant events to be observed. Intermediate factors for these outcomes may be more informative, such as reductions in severity of incidents and considerations of mechanisms of change, as touched on by Clark and Chuan [27] in exploring changes to the reasons for recall. Without dedicated objective research resources, longitudinal study design is unlikely in the current OPD Pathway provision, certainly at regional levels. It is also relevant to consider the appropriateness in allocation of individuals to specified intervention programmes prior to exploring related risk and reconviction outcomes [47]. Intermediary outcomes, such as changes in factors known to affect risk levels and reconviction rates, were largely absent in the literature identified for this review, while studies of other OPD services have utilised reasons for recall to custody as an indicative measure of reduced harm, noting reduced seriousness of offending in the reconviction group [48]. A large international body of research on risk and reoffending in relation to offender supervision practices may provide a useful resource for potential approaches to evaluating the impact of psychological interventions offered in the community [46,49]. Such large-scale evaluation of these outcomes is a clear priority for OPD stakeholders, however, barriers such as complex data governance, security and access request processes, difficulty in linking sources of data and a lack of dedicated research staff with the technical resources and know-how to manage available data need to be overcome. Consequently, these barriers to accessing routinely collected organisational-level data at a national or local service level often preclude attempting such studies.

Wellbeing, in spite of being a major intended outcome of the OPD Pathway, was scarcely explored in the reviewed literature. This was particularly evident in relation to offender wellbeing, with just one study addressing coping mechanisms and no other studies addressing any other measure of psychological health or functioning such as skills development, substance use, self-harming or prosocial behaviour. While direct psychometric measurement of these factors is achievable in a prison setting, such self-report measures may be more difficult to gather from a probation cohort, although not impossible [50]. Mixed methods approaches would allow quantitative approaches to be complemented by qualitative investigation of the nuances in wellbeing factors on an individual level. It might be particularly useful to explore mechanisms of change as a result of the psychologically-informed and relational approach promoted by the OPDP (e.g., [11]) in order to determine if, and to what extent, improved wellbeing and mental health results. While studies from within custodial settings such as prison treatment units have provided some insights in this vein [51], the complexities of offender rehabilitation in the community present particular challenges and warrant dedicated attention [52]. The linking of health systems’ data to probation data could significantly help address this gap in our understanding of relevant outcomes in the community. Staff wellbeing was only directly addressed in terms of burnout, while brief mentions of job satisfaction and workplace stress occurred somewhat anecdotally in some qualitative studies. As a result, it was not possible to consider a more granular analysis of which specific components of psychological support, training, consultation or supervision were most effective in this sphere. Clearly this is an area requiring extensive further research especially as within the wider literature on antecedents to correctional staff burnout, few studies address potential interventions to tackle this [53]. Relatedly, the demands on probation staff time and resources necessitate succinct approaches to outcome measurement, thus routinely incorporating general measures of wellbeing and job satisfaction factors may help address the shortfall in literature of this nature (e.g., [54]. In addition research on wellbeing should attend to factors such as staff turnover, sickness, presenteeism and compassion fatigue. Given the continuously changing policies directing probation work, it would be useful to gain an insight into whether and how OPD Pathway processes could help in alleviating the inherent stresses of working in such an unpredictable system.

Finally, relational improvements have been noted as a key goal of the OPD Pathway, and while the OM-offender relationship was explored to some extent, no other element of relationship outcomes were evaluated in the reviewed literature, beyond at an anecdotal level. Burnett et al., discussed in detail the changing nature of probation work over the last few decades and noted the significant shift in attitude toward OM-offender relationships throughout these policy changes [55]. As such, it would be particularly useful to explore the impact and role of OPD services in redeveloping this key process of probation management. To this end, relevant tools (e.g., the Relational Security Explorer or the Working Alliance Inventory) ought to be explored for use in self-evaluation of relational security within OPD services [56,57]. Indeed, in the context of the reunification of probation services, it would be particularly useful to explore the differential impacts on staff wellbeing for those coming from an indirect, phone-based supervision paradigm compared to those with in-person supervision experience.

Future research

In keeping with the suggestions for further studies already made, future research should attend to the areas of enquiry under evaluation and the approaches to research design and analysis being adopted. The real-world nature of this area of research often presents insurmountable barriers to following a robust scientific design, particularly at a local level. However, through careful construction, a pragmatic use of routine practices, and perhaps a clearer focus on specific inputs (e.g., theories underpinning delivery or a particular service component) or outcomes, it can be possible to generate high quality research content, particularly through utilising realist approaches to evaluation. Some of the difficulties of conducting high quality research in these settings may be overcome by, for example, having an agreed national approach to outcomes monitoring for people on probation. As already noted, this may involve incorporating well-validated measures of key outcomes into routine processes. This would allow the limited resource available for research and evaluation to be targeted at data analysis rather than data collection.

Aligning research design with the underpinning Theory of Change for each service type within the OPD Pathway could also help guide the production of relevant and useful findings that help identify areas of effectiveness and areas for improvement. Clearly documenting inputs, such as theoretical approaches utilised, might help improve the rigour of impact analyses carried out, leaving less room for ambiguity around specifics of service delivery. It is notable, however, that several studies achieved low MMAT scores due to an absence of reporting of steps taken, such as accounting for confounders in design or analysis, or not demonstrating that interventions were administered as intended. Clearer reporting of intervention and analytical processes may have improved scoring considerably in several studies. It may therefore be helpful for those publishing work in this area to use a framework such as MMAT to self-rate their study at the design stage and their paper(s) prior to submission. Relatedly, external collaboration with academic institutions, research councils, or independent researchers, both at a service-specific and national level, may help counteract potential bias in interpretation of findings and increase rigour. Finally, given the considerable differences in the psychological complexities and offending risks presented by male and female OPD service users, future research designs could consider taking a gendered approach to help ensure clarity of outcomes as they relate to each group. As such, it may be useful to explore any differences in how men and women respond to similar OPD interventions, or to evaluate gender-specific services to help inform practice development and potentially identify gender-specific outcomes to explore in future studies. Taken together these steps may help to maximise the quality and robustness of the resulting publications and increase the impact such research can have on service design and practice.

Limitations of review process

This review was carried out on published literature only and as such, there may be a substantial body of unpublished research which could have contributed to the knowledge base built herein. It is also acknowledged that search terms employed did not clearly map onto all outcomes explored, particularly in relation to offender search terms where specific outcomes of reoffending, risk, wellbeing, etc., were not utilised. Search terms were developed to ensure the correct participant groups were captured and less priority was given to specified outcomes to avoid potentially missing content that used vague or indicative language. Additionally, while the initial screening and selection stages of this review were conducted by two researchers, the majority of the extraction and all of the synthesis process was carried out by the first author alone. As a result, the conclusions drawn and quality assessments made were not subject to any formal external validation processes. Future reviews of this nature would benefit from wider collaboration with independent reviewers to strengthen reliability and avoid potential bias in interpretation of findings. It is further acknowledged that a protocol for this review was not registered in advance of carrying out the study. While protocol registration is increasingly used to provide an assurance that reviewers will not stray from their intended scope, assurance regarding scope was provided by the previously published and publicly available OPD Strategy [12] which describes the nature of Core OM delivery and the intended outcomes of these services. Further, the detailed description of the method presented above is intended to allow replication of the study conducted and to provide assurance as to the methodological integrity of this analysis. It is relevant to note that higher quality study design in the reviewed papers may have allowed for the identification of common themes and contrasting insights derived from different methodological approaches, however such comparison of findings by research design was not possible in this study.

Implications for practice and policy

The reviewed literature presented too little in terms of concrete outcomes directly resulting from OPD Pathway interventions to be informative about clear directions for criminal justice policy and probation practice, however, there were some useful indicative findings. Overall, there was some support for the provision of psychological guidance and consultation processes being helpful to probation staff in terms of their direct working with complex individuals. Gaining insights into the historic factors that influence behavioural responses to perceived triggers and learning to utilise more psychologically informed communication methods with their service users may help probation staff to identify and mitigate against potential risk factors for further offending or harmful behaviours. Two qualitative staff studies made brief reference to the additional pressure felt by OMs as a result of OPDP involvement. This raises the pertinent question of potential unintended or iatrogenic effects on staff (such as compassion fatigue) experienced as a result of developing a psychologically informed approach and understanding of offending behaviour. It may therefore be prudent to explore in more detail the implications of adopting relational ways of working, and the nature of the training, and supervision required for staff to enable and sustain this way of working. Indeed, staff supervision and reflective practice are increasingly being recognised as an essential component for the safe and effective delivery of services [58]. Perhaps the integration of frameworks such as bio-psycho-social theory and trauma-informed practice into the established probation qualification training programme may also support this development.