Addressing social relationships through SEL interventions

Loneliness and childhood victimisation are well-established risk factors associated with problematic mental and physical health and academic functioning [13–15]. Loneliness can occur when there is a perceived difference between the actual and desired quantity and/or quality of one’s relationships [16] and is associated with long-lasting negative health outcomes for youth [e.g., depression, anxiety; 17,18]. With experiences of loneliness found to peak in early adolescence and schools identified as critical environments for targeting and alleviating loneliness [17], the UK Government’s loneliness strategy aims to prevent loneliness in youth by nurturing connected school communities [19].

Indeed, schools have a duty of care to children, to nurture student growth and to prevent bullying behaviours [20]. Bullying is intentional and reoccurring exclusionary or aggressive behaviour over time and in the presence of a power imbalance between perpetrator and victim [21]. It is associated with poorer academic performance and psychological, behavioural, and social maladjustment for both the bully and the victim [13,22,23]. Approximately 35% of 10–15-year-olds in England and Wales were bullied between 2022−23 [24] with a prevalence of 16% reported in Greater Manchester [25].

Peer social support is an important protective factor in child mental health [26], and moderates negative effects of stressful life events (e.g., bullying). It is defined as access to and receipt of emotional or physical support and resources through interactions with those in one’s social network [27,28]. Peer relationships become increasingly important during early adolescence, and young people turn primarily to peers for advice or support when experiencing distress [29]. The UK Government thus aim to support the development of quality peer relationships in schools to improve children’s mental health [30].

Children who have poor peer relationships (e.g., experience victimisation, social exclusion or lack close friends) are most likely to experience loneliness [17,31–33]. The ‘friendship protection hypothesis’ suggests that peer relationships protect against negative experiences and their negative outcomes [34] including loneliness, bullying perpetration and victimisation [35–37]. The importance of fostering caring school environments that promote positive peer relationships is therefore critical [17].

Although meta-analytic evidence demonstrates that SEL interventions improve social-emotional skills (effect size [ES] = 0.21–0.57), peer relationships (ES = 0.22) and reduce bullying behaviours (ES = 0.16–0.22) and victimisation (ES = 0.14–0.24) [2,3], the influence of implementation variability on loneliness and peer support remain underexplored [38,39]. Null intent-to-treat (ITT) effects were found when examining the impact of the PATHS SEL programme (Promoting Alternative THinking Strategies) on social support [40], however moderate intervention effects were revealed in moderate-high compliance classrooms (ES = 0.63) [41]. There is also emerging empirical support that school-based interventions targeting social-emotional competencies can alleviate loneliness [42] and the PATHS programme reduced experiences of loneliness among children in the intervention group relative to the control (ES = 1.35–1.68) [39]. The current study aims to contribute to this building evidence base by examining the impact of Passport on these relational outcomes. While a focus on SEL and relational outcomes in this study is therefore advantageous from a developmental perspective, this decision is also pragmatic (this is a secondary data analysis, as detailed in the Methods).

Underpinned by coping theory, Passport primarily aims to help children identify and use positive coping strategies and to offer, seek and accept help [11]. Lessons on ‘Relationships’ and ‘Dealing with Conflict’ aim to support children in making friends, dealing with bullying, and use adaptive coping skills to negate feelings of loneliness or rejection that can occur when a friendship ends. Passport is currently implemented in c.115 schools across the UK; evaluating its efficacy in building relation skills in this context is therefore timely [43]. No research to date has examined the effectiveness of SEL interventions on loneliness and peer victimisation after accounting for implementation variability in a causal framework [9]. The current study aims to address this research gap.

Implementation matters in school-based trials

The standard approach to evaluating intervention effects is through applying an ITT framework which upholds the bias-protective principle of randomisation (“once randomised, always analysed”), yet accurate causal inference is limited by the assumption of full compliance (i.e., that the intervention group delivered the intervention in its entirety without deviating from the manual/protocol). Unlike pharmaceutical and medical trials where treatment compliance is closely monitored and/or controlled, the assumption does not hold in school-based trials amidst theoretical support for, and empirical evidence of, variation in intervention delivery [2]. Per-protocol and as-treated are often used alongside ITT to address implementation variability, yet these traditional methods produce biased effect estimates (e.g., by excluding data of non-compliers following randomisation and disrupting group equivalence).

A simplistic solution is to focus solely on the ITT effect [44], yet this can lead to biased estimates of true intervention effects [45], which can manifest in two ways. If participants experience adverse intervention effects and consequently discontinue treatment, but participant adherence is not addressed in the analysis, positive intervention effects are overestimated. Conversely, if participants fail to adhere to the protocol despite positive outcomes, intervention effects are underestimated. Thus, considering the average intervention effect in the context of compliance alongside the ITT effect (i.e., the average intervention effect across all participants offered the intervention) provides valuable insight into the true intervention impact among those who received the intervention.

There is a call for educational research to move "beyond ITT" and use instrumental variable approaches [46] such as propensity score analysis, treatment effect bounding, and CACE [45]. These approaches use a causal framework that specifically models participant compliance and are popular in the medical, pharmaceutical and psychological literature. Though increasingly employed, they are uncommon in school-based trials [47]. This study focuses on the use of CACE [48] which estimates the impact of the intervention among the subpopulation of compliers [45]. It does so by using random assignment as an instrumental variable and by applying assumptions, such as monotonicity and exclusion restrictions, to infer the proportion of compliers. Although CACE is a robust analytic choice [47], its application to education is not without limitations. Identifying and overcoming these limitations is an issue we now address.

Overcoming limitations to CACE using a multiverse framework

There are a number of limitations to CACE. First, school-based trials typically model compliance using dosage [49,50], as quantity of lessons delivered is considered an appropriate proxy for estimating the impact of both offering the intervention (i.e., ITT effect) and intervention receipt (i.e., CACE effect) when school attendance is mandatory. However, dose offered and dose received are conceptually distinct [51]. One could therefore argue that delivering an intervention session (i.e., dosage) does not guarantee intervention receipt if the programme objectives or core components are not delivered (i.e., poor fidelity); if lessons are delivered with poor quality; if students are not engaged or actively participating during lessons (i.e., low responsiveness), or if students are absent (i.e., low reach). The reliance on dosage as the sole indicator of intervention receipt in CACE models is criticised as arbitrary [52], arguably neglecting empirical evidence that interventions can yield positive outcomes even at low dosage levels [53,54]. Our recent systematic review revealed that multiple SEL implementation dimensions (fidelity, dosage, reach, responsiveness, quality) were associated with students outcomes and could therefore be considered as potential indicators of intervention receipt [9].

Second, compliance is modelled as a binary variable in a CACE model, yet defining ‘full’ compliance of a prevention intervention is significantly challenging. CACE is dependent on the exclusion restriction assumption (non-compliers derive no benefit from the intervention; see Data analysis) which is difficult to validate in the absence of theory-informed compliance thresholds [55] which are largely untested and unknown in psychological and educational interventions [56]. According to the developer of Passport, there is no theoretically informed threshold for optimal programme delivery and empirical research has not established a definition of compliance in the context of this intervention [e.g., 11]. An inductive approach to defining compliance is an alternative solution, but selecting the cut-off point for defining a complier and non-complier is “somewhat arbitrary along the percent compliance continuum” [46, p.22]. Researchers are encouraged to use multiple sensitivity analyses to observe any change across compliance thresholds [46]. Using the 50^th and 75^th percentile to define compliance is established practice in educational research [41,49,50,57,58].

The selection of a compliance definition and threshold can impact outcomes of the CACE model. Typically, methodological choices available to researchers range in quality, with strong theoretical and/or empirical support guiding selection of a superior approach. However, when there are numerous sound and theoretically justifiable decisions, selecting one over another is arbitrary [12]. Arbitrary data processing decisions can constrain a dataset and subsequently limit the validity of the conclusions drawn from it [59]. To increase transparency of these limitations, we constructed a multiverse of possible modelling choices for our CACE model.

Multiverse analysis systematically explores how different, equally defensible analytic choices impact research findings, to evaluate the robustness of results and transparently convey uncertainty [12]. A multiverse approach will reveal the instability or robustness of results that hinge on these data processing decisions and help to determine the moderating role of each implementation dimension on outcomes, and the level of implementation that yields desired intervention effects, thus addressing multiple and pertinent research priorities [7,8]. Crenshaw and colleagues [60] demonstrate the strength of this framework in a clinical trial for treatment of post-traumatic stress disorder amidst challenges with participant recruitment, differential attrition between conditions, and missing follow-up data. A multiverse approach facilitated examination of the robustness of results that were based upon pre-registered analytic decisions (e.g., ITT) versus alternative analytic choices that addressed unexpected challenges (e.g., as-treated), demonstrating the treatments’ efficacy across all analytic universes and thus ensuring confidence in conclusions drawn. Despite its strength in transparently revealing the fragility or robustness of conclusion, multiverse analysis remains underutilised in this field [61]. Our study extends the application of this framework to school-based intervention research.

The current study

This paper is a response to the call for increased methodological rigor when assessing implementation variability in education research [9]. We employ CACE to model implementation variability within a causal framework [45], a rigorous analytic method, however unreported researcher degrees of freedom can limit replicability and credibility of conclusions [62]. We therefore apply a multiverse framework to transparently report uncertainty [12].

This research thus aims to address the following research questions: (1) What is the impact of Passport on primary school-aged children’s relational outcomes (i.e., loneliness, peer social support, and victimisation)? (2) What is the impact of Passport on primary school-aged children’s relational outcomes when accounting for intervention compliance? (3) How does the complier average causal effect of Passport on pupil outcomes vary by compliance definition?

Design

A two-arm (intervention vs usual practice) parallel cluster randomised controlled trial design was used. The Passport to Success trial took place in mainstream primary schools across Greater Manchester between September 2022–2025. Passport was implemented in Year 5 classrooms of participating schools in the intervention arm during the 2023/2024 academic year. Schools allocated to the control arm continued delivering the usual school provision. Ethical approval for the trial was obtained from the University of Manchester Research Ethics Committee [Ref: 2022-14050-24401]. The trial was prospectively registered with ISRCTN [ISRCTN12875599] and detailed methodology is available in the published protocol [43]. This paper reports on methods pertaining to the current study only, which was preregistered on the OSF: www.osf.io/s5pmw.

Sample

Sixty-two schools participated in the current study. Eligible schools were mainstream, non-independent primary schools situated in Greater Manchester, delivering education to Year 4 – Year 6 students and had not previously delivered Passport. The final student sample consisted of 2425 pupils (50.1% male) in 79 classrooms. Students were eligible to take part in the trial if they were in a Year 4 classroom in a participating school in the 2022/2023 academic year, they had not been opted out of the study and they assented to participate. At baseline, the proportion of participating students that were eligible for free school meals (FSM; 34%) and had special educational needs (SEN; 18%) were greater than the national average [63]. Participant demographics are reported in Table 1. All variables, bar FSM, were balanced between trial arms [see 64].

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Participant demographic information and descriptive statistics for study variables.

https://doi.org/10.1371/journal.pone.0349949.t001

Procedure

School recruitment commenced 20.09.2022 and concluded 21.03.2023. The trial manager (JS) contacted Local Authorities in Greater Manchester to nominate local schools and received signed Memorandum of Agreement and a Data Sharing Agreement from sixty-two schools.

Parents/carers of Year 4 children received an information sheet and the opportunity to opt their child out of participating in data collection. In line with schools’ in loco parentis responsibilities, children opted out of the study in intervention schools still received the Passport programme, which was delivered as part of the school curriculum. Participating children assented prior to completing baseline outcome surveys (T0; April – July 2023), administered via Qualtrics.

Randomisation took place following baseline survey data collection by an independent statistician with schools as the unit of randomisation. The allocation sequence was generating using MinimPy software [65]. Minimisation was used to ensure balance across trial arms for school size and proportion of students eligible for FSM.

Year 5 teachers in intervention schools received mandatory 2-hour online training in delivering Passport from Partnership for Children in September 2023; a 1-hour optional booster training session was offered in February 2024 and aimed to address any challenges that arose during the initial months of programme delivery. Post-intervention data collection took place in April – July 2024 (T1).

Intervention

The reporting of the intervention is guided by the TIDieR checklist [66]. Passport follows the adventures of two children as they navigate a fantasy world with dragons and mythical creatures. There are 18 sequential lessons (approx. 55 minutes) delivered once weekly on topics of emotions, relationships and helping others, dealing with difficult situations (e.g., bullying), fairness and justice, and coping with change and loss. Passport’s theory of change posits that fostering coping flexibility will help children self-regulate, inhibiting the onset of mental distress [11]. The programme’s logic model is illustrated in Humphrey et al. [64].

Lessons are student-led and delivered during class time by Year 5 teachers using physical and digital resources [67], including comic strips, board games and role play. Teachers must adhere to the structure, sequence and content of the programme manual however they are encouraged to augment lessons, personalising them as necessary to meet the individual needs of their class.

Usual practice

Schools allocated to the control arm of the trial continued with usual practice. A usual practice survey administered to all participating classroom teachers revealed that approximately two thirds of the topics covered by Passport were taught across participating schools (e.g., identifying and managing emotions) and two thirds of classrooms delivered propriety SEL curricula (e.g., Jigsaw).

Outcome measures

Data analysis

Selecting a compliance definition and threshold.

Construction of the multiverse data set is detailed in S2 File and visually illustrated in Fig 1. Ten theoretically plausible specifications were identified and pre-registered (https://osf.io/z62wa). Skewed dosage data resulted in 9 specifications (see Compliance Thresholds) and thus 27 models were analysed. The first author was responsible for data analysis and accessed the data following publication of the pre-registration.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Decision tree for multiverse specifications.

https://doi.org/10.1371/journal.pone.0349949.g001

ITT analyses

After controlling for student and classroom-level covariates, null ITT effects were found for peer support, victimisation and loneliness (see Table 2).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Standardised ITT effect for relational outcomes (N = 2,425).

https://doi.org/10.1371/journal.pone.0349949.t002

CACE analyses

The CACE effects for each analytic universe are reported in Table 3 (for individual model results, including entropy values, class count, and strength of predictors, see S4 File). The entropy value (E; range 0–1) indicates how well a model distinguishes between latent classes, with high values indicating stronger class separation between compliers and non-compliers [81]. Entropy values varied across models and only four met the pre-registered threshold of .76 [82]. The four optimal-entropy models examined intervention effects on victimisation with compliance modelled using moderate reach, engagement, adherence, and dosage. Compliance predictors were weak across multiple CACE models (see S4 File).

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Standardised CACE effect for relational outcomes (N = 2425).

https://doi.org/10.1371/journal.pone.0349949.t003

The multiverse analyses revealed a statistically significant and negative intervention effect for peer support when compliance was defined by high reach (β = −.38, 95% CI (−.68, −.08), E = .71, d = −.21). The remaining 26 CACE models were not statistically significant, indicating that the intervention did not improve student self-report victimisation, loneliness or peer support relative to the control, irrespective of implementation variability. Additionally, several models revealed substantial effect sizes with corresponding unstandardised models considered statistically significant, explained by differences in the sampling distribution between standardised and non-standardised coefficients. Our primary interpretation and reporting are based on standardised results as per our pre-registration, and standardised estimates are reported in S4 File. The results of the multiverse are illustrated by Figs 2–4 for victimisation, loneliness and peer support, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. CACE estimates (x-axis) across five compliance definitions (y-axis) for victimisation outcome.

Compliance definition is along the y-axis and point estimates are plotted as standardised beta coefficients on the x-axis, with pink circles representing the 50^th percentile (moderate compliance) and blue triangles representing the 75^th percentile (high compliance) across model specifications. Horizontal lines indicate 95% confidence intervals. An asterisk (*) indicates a confidence interval that extends beyond the x-axis limits shown and is truncated for visualisation.

https://doi.org/10.1371/journal.pone.0349949.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. CACE estimates (x-axis) across five compliance definitions (y-axis) for loneliness outcome.

Compliance definition is along the y-axis and point estimates are plotted as standardised beta coefficients on the x-axis, with pink circles representing the 50^th percentile (moderate compliance) and blue triangles representing the 75^th percentile (high compliance) across model specifications. Horizontal lines indicate 95% confidence intervals. An asterisk (*) indicates a confidence interval that extends beyond the x-axis limits shown and is truncated for visualisation.

https://doi.org/10.1371/journal.pone.0349949.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. CACE estimates (x-axis) across five compliance definitions (y-axis) for peer social support outcome.

Compliance definition is along the y-axis and point estimates are plotted as standardised beta coefficients on the x-axis, with pink circles representing the 50^th percentile (moderate compliance) and blue triangles representing the 75^th percentile (high compliance) across model specifications. Horizontal lines indicate 95% confidence intervals. An asterisk (*) indicates a confidence interval that extends beyond the x-axis limits shown and is truncated for visualisation.

https://doi.org/10.1371/journal.pone.0349949.g004

Evaluating entropy values

Entropy values ranged between .59–.78 across models with most (85%) below the recommended threshold of .76 [82]. We initially hypothesised that suboptimal entropy values could be explained by negatively skewed dosage data resulting in low score variance across compliance definitions which were weighted by dosage. Transforming continuous implementation dimensions into a binary variable weighted by dosage may result in equifinality [83], whereby different combinations (e.g., low attendance and high dosage vs high attendance and low dosage) produce the same overall compliance score. This is in contrast to a ‘clean’ binary variable for dosage. However, the multiverse revealed variation across all compliance definitions and thresholds bar dosage, making this theory less viable. A clear pattern emerged when looking at the outcome variables, with the strongest entropy values observed for victimisation models and the weakest for loneliness. Although this hypothesis is speculative, this trend may be explained by the compliance predictors in the CACE models, which included the baseline score of the outcome under investigation. It is possible that bullying victimisation (and by proxy, perpetration) is a stronger predictor of teacher compliance as an externalising behaviour relative to feelings of loneliness which are internalising [84].

Evaluating precision and confidence intervals (CI)

The precision of point estimates varied across implementation dimensions and outcomes, as reflected in the differing widths of CI. Specifically, wide CI were observed when modelling dosage (victimisation), high quality (loneliness) and moderate engagement (peer support), indicating greater uncertainty around the estimated effects in these models. To a lesser extent, this was also observed for high adherence in relation to peer support.

These patterns may result from a number of potential factors. First, weak predictor variables may contribute to this imprecision. Indeed, some model results should be interpreted with a degree of caution, as the multiverse revealed differences in the strength and specificity of the compliance predictors across CACE models, which also likely impacted the specificity of class separation [85]. Strong compliance predictors are essential to enhancing the reliability and robustness of CACE models, as they directly influence class assignment precision and, consequently, the validity of estimated intervention effects [44]. Model results with weak compliance predictors potentially violate the exclusion restriction assumption and should therefore be interpreted with caution [86].

Despite theory and evidence informing covariate selection, this secondary data analysis was limited to available predictors. This issue of low predictive power of carefully chosen compliance predictors is not unique to the current study [87,88]. Predictors were consistent between models to limit inflating the number of models in the multiverse, nonetheless reviews suggest that contextual and individual factors may differentially influence implementation behaviours in school-based social-emotional and mental health interventions [89,90].

Furthermore, three models with low entropy values had comparatively larger CI: loneliness moderated by high quality (E = .65) and social support moderated by moderate engagement (E = .6) and high adherence (E = .67). However, wide CI for a model with strong entropy (victimisation moderated by dosage; E = .78) requires consideration of an alternative explanation. Ninety-two percent of all lessons were delivered on average resulting in negatively skewed dosage data and a small number of non-compliers (38%, n = 921), reducing power to detect moderation effects. Wide CI therefore reflect limited variability in the moderator, rather than outcome variability. Finally, implementation data may have been subject to measurement error, which is explored further below. Taken together, caution is warranted in interpreting these specific findings as the low precision limits confidence in the estimated effects, discussion of which we now turn.

Evaluating the direction and size of effects

The CACE effects were largely consistent with the ITT effect, indicating that there was no statistically significant difference between Passport and usual provision for student’s relational outcomes for all but one model. Additionally, the multiverse revealed differences in the direction of point estimates between compliance thresholds. When compliance was modelled using the 50^th percentile as the cut-off point between compliers and non-compliers, the direction of effects was mainly positive, but direction of effects was mainly negative for high compliance. An interesting trend was also observed for the size of effects: negligible effects were observed for loneliness across all models, except for high reach which had the largest effect size of all 27 models (d = .32). The second largest effect for high reach and victimisation, but effects were negative (d = −.3). Similar effects were observed for peer social support in classrooms with high adherence (d = −.26), high quality (d = −.23) and high reach (d = −.21, p < .05).

Opposing directional effects between high and moderate compliers indicates that there may be an implementation ‘sweet spot’ such that more is not always better [7]. For instance, high compliance to activity structure and timing (i.e., adherence) may require teachers to go against their professional judgement to adapt lesson delivery to their unique class context [91]. Teachers who prioritised adherence to activity timing over student-led discussion thought that Passport lessons were less effective [92]. Indeed, such a top-down prescriptive approach to teaching is criticised as a ‘democratic deficit’ in education [93]. Moderate compliers that adhere comparatively less to a top-down delivery model, and in turn remain responsive to student-led discussion, may more successfully foster an environment conducive to SEL [91].

The finding that high quality delivery is associated with negative effects for peer social support is harder to explain; Passport may inadvertently nurture emotional and social awareness without providing students with skills to seek support or to managing complex social situations. Findings from the process evaluation support this hypothesis, as some teachers described adapting programme content to their class (an indicator of quality) which resulted in inadvertently removing essential activities that award help-seeking (e.g., the Help Thermometer) and the identification of coping skills (the Dragon’s Path); indeed, some students struggled to apply strategies in practice, particularly in the context of persistent bullying [94]. It is possible that greater exposure to lesson content, without sufficient structural support to practice skills, prevented development of targeted skills required to navigate social challenges.

This theory can be further interrogated in the context of an unexpected trend towards poorer scores for high attending classrooms (i.e., reach). The effectiveness of Passport may be compromised in large classroom settings if students have fewer opportunities to engage meaningfully in discussions or practice-based activities. Teacher interviews suggest that managing activities became increasingly challenging with over 30 students, particularly as Passport is student-led and centres around peer group discussions [92]. Passport was initially designed and piloted in schools with an average class size of 20 pupils [11] yet the programme is now delivered in English classrooms [67] which have on average 26 pupils [95], as reflected in the current trial’s sample. This 30% increase in class size may disrupt the programme’s core components and prevent students from deriving the intended benefits from guided practice elements (e.g., role play, games) that are essential to building confidence and competency to navigate such social challenges successfully.

The unexpected finding that high reach is paradoxically related to poorer outcomes (d = .21 −.32), compared to moderate reach, could be further explained by drawing upon the economic theory of opportunity cost, or the value of the next-best alternative that must be forgone when making a choice [96]. Delivering a universal programme like Passport requires schools to make resource allocation decisions within finite timetables and staffing constraints [97] – in other words, Passport must replace another lesson [98]. Teachers who report high attendance may choose to deliver a universal programme like Passport as an alternative to targeted support, which is more costly [99]; in contrast, moderate attendance may reflect a strategic approach, whereby an intervention like Passport is delivered when a portion of students who require targeted support leave the classroom to engage in those activities. Although critical to informing economic evaluations of school-based interventions [100], it’s unclear how opportunity cost plays out in practice [101]. Further qualitative interviews were conducted with school staff to understand how perceptions of an intervention’s value and cost informs decisions to adopt and implement a universal intervention like Passport [92].

Finally, this trend towards poorer outcomes among high compliers may be explained by a complementary phenomenon of ‘response shift’ [102], whereby an intervention that aims to improve student outcomes (e.g., emotional literacy) improves measurement accuracy post-intervention. We recommend that future studies consider using differential item functioning analyses to examine whether control and intervention participants interpret measure items differently [103]. While previous research found no difference in teachers’ interpretation of bullying behaviours following delivery of the PATHS programme [104], to the author’s knowledge, the possibility of response shift has not yet been explored among children participating in school-based SEL interventions.

Considering measurement error

As per our trial protocol, implementation data was collected using a cross-sectional teacher-report survey to minimise burden and mitigate potential attrition compared to, for example, teacher observations [43]. This approach required teachers to provide a global rating across the 18-week intervention period, rather than session-specific assessments. This methodological constraint, though necessary, may have resulted in several limitations.

First, this approach limits identification and examination of within-programme fluctuation across implementation dimensions [7]. Second, the retrospective design introduces potential for recall bias, particularly recency effects, whereby scores may have been disproportionally influenced by the final ‘Celebration’ session (which was thoroughly enjoyed by staff and students) [94]. Third, self-report implementation is subject to social desirability bias, whereby responses are inflated to align with perceived expectations or professional standards [105,106]. Accordingly, the consistency of results across conceptually distinct implementation dimensions may result from reliance on single-reporter implementation data, inflating intercorrelations among dimensions.

The optimal approach to assessing implementation in school-based interventions is third-party observations [9,107] with an increasing number of tools available to researchers [108,109]. Observations offer value in terms of objectivity and rigour, but can be resource intensive and burdensome on school staff [7] and are not immune to bias (e.g., observer effect) [107]. Additionally, the inter-rater reliability between teacher and observer fidelity measures tends to be low [105,110]. It is therefore essential to consider approaches to optimising self-report measures, such as assessing student responsiveness using student-report surveys (rather than teacher-report) [9] and involving participants in survey design and development [111]. One such approach is ecological momentary assessment (EMA) to capture participant implementation behaviours (e.g., student responsiveness, teacher quality) during programme delivery. EMA capture variation in behaviours and attitudes across time and contexts, providing insight into whether and how responses vary between and across participants groups, contexts and programme sessions [112].

These suggestions align with calls to embed implementation considerations during programme design rather than an “added and separate effort” [113], and are important given evidence that researchers and practitioners may conceptualise implementation constructs differently [e.g., fidelity; 114]. This was observed in the current trial when teachers simultaneously reported high fidelity (i.e., adherence to lesson learning objectives, activity sequence and structure, and programme manual for core activities), while detailing lesson adaptations, modifications, and omissions [94]. This conceptual distinction could be explained by whether teachers are considering and reporting pedagogical adherence (how it is taught, e.g., role play) or practice-based adherence (what is taught, e.g., identifying emotions) [115]. Teachers draw upon their pedagogical knowledge to adapt an SEL intervention [116] and may therefore still report high fidelity if they believe they are adhering to the programme ethos despite pedagogical deviations [117]. The construct validity of implementation measures is rarely assessed or reported in school-based SEL research, raising concerns about the accuracy with which current tools capture the intended construct [9]. For example, the current measure of quality asked teachers if they felt able to engage pupils and well-prepared to teach lessons; one could argue that these items are indicators of programme design rather than quality of teacher delivery.

We urge researchers to carefully consider best practice when assessing implementation variability in school-based trials [106]. Ideally, implementation data is collected from multiple informants (e.g., student responsiveness self-reported by students) and at multiple timepoints. Moving forwards, researchers using multi-method multi-informant approaches (e.g., event-contingent EMA) would benefit from extending the multiverse framework to explore whether seemingly homogenous results in a dataset are the result of measurement issues outlined above or a well-delivered intervention [118]. Using a study within a trial (SWAT) design can provide valuable insights for advancing school-based process evaluations [119], such as the validity of approaches to collecting implementation data (e.g., weekly, cross-sectional). Our Implementation Quality Appraisal Checklist provides tiered guidance to improve research practice, which can be useful when managing resource constraints [9].

Moving from individual parts to a whole picture

The multiverse reveals variability in results depending on how we define and model intervention compliance, which manifested in entropy values, precision of confidence intervals and the strength and direction of intervention effects. Despite this variability, clear patterns emerged across the multiverse: certain outcomes (e.g., victimisation) produced stronger entropy values than others (e.g., loneliness) and moderate compliance was more often associated with positive outcomes compared to high compliance.

When considering the results of the multiverse as a whole, it instils confidence in findings that were not variable across analytic universes: all but one of the 27 CACE models were not statistically significant, indicating that the CACE effects are largely consistent with the ITT results and Passport was no better than usual practice in improving children’s relational outcomes, however it is delivered. This further supports the hypothesis that null intervention effects are not a result of implementation failure but can be explained by low programme differentiation [64], as approximately 60% of teachers reported implementing similar universal SEL programmes that overlapped substantially with Passport content. This finding is not unexpected given that schools are encouraged to prioritise SEL due to robust evidence of its efficacy [120].

Our findings reinforce the importance of considering implementation behaviours as distinct constructs and thus shaped by different determinants [6,89]. Future research would benefit from further empirical work examining student, teacher and classroom characteristics that predict implementation behaviours [121]. This would not only result in more robust modelling but have important practical implications; identifying individuals or contexts at greater risk of non-compliance can inform targeted allocation of resources to promote equitable implementation. For researchers utilising the current paper to guide implementation and process evaluation, we recommend using theory and evidence to prospectively identify appropriate predictors to enhance model robustness, for example, using qualitative and quantitative evidence of factors influencing implementation behaviours from systematic reviews. Additionally, involving teachers at participating schools via participatory co-design could inform the final selection of compliance predictors ensuring contextual relevancy. These recommendations have value beyond CACE; while its critics propose using a propensity score approach as a more robust alternative [52], this approach similarly relies on good covariates to predict compliance [45].

Furthermore, the findings from the multiverse challenge the assumption that more is always better when delivering a universal SEL programme like Passport. Considering that time is a major barrier to programme delivery in schools [90], our results support the continued calls to empirically establish intervention core components. This could be achieved using a factorial trial design [115,122] or bottom-up network approaches, whereby variation in intervention outcomes are mapped onto the associated active ingredients delivered that week [123]. This suggestion may draw scepticisms due to the subsequent burden on intervention participants, particularly amidst continued challenges with recruiting schools to trials [124,125]. Subsequently, secondary data analysis may be an appropriate solution [e.g., 126]. Such insights could refine intervention models, removing redundant activities to produce a streamlined intervention (i.e., how much is good enough) [7,127].

Finally, we are encouraged to practice caution when null and/or negative intervention effects juxtapose participants’ experiences of a programme’s effectiveness [128]. The qualitative strand of the Passport trial found that students thoroughly enjoyed Passport and many teachers expressed an interest in continuing programme delivery for future year groups [94]. The programme manual provided teachers with clear structure and guidance for teaching SEL, and supported strong implementation, as indicated by the negatively skewed implementation data. This aligns with teacher preferences for clear, structured and time-efficient SEL curricula [129]. It is therefore possible that a manualised SEL programme like Passport, with minimal planning and preparation time, provides additional benefits for teachers, rather than students, when compared to the usual curriculum, given that educators’ time is one of the most valuable resources available to schools [99]. However, our findings highlight a critical gap in our understanding of how schools navigate the opportunity cost, or trade-off, when deciding whether to adopt and deliver a universal intervention like Passport. These decision-making processes that guide resource allocation are poorly understood, particularly in the context of competing demands and resource constraints [97,98,129]. Further research will explore what drives school’s programme preferences, and how this underpins resource allocation, which is essential to understanding and predicting programme adoption, implementation and sustainability patterns [92].