https://orcid.org/0000-0001-5109-4893
Figures
Abstract
Youth driving under the influence of cannabis (DUIC) is a growing public health concern. While brief smartphone interventions have shown promise in reducing substance use and alcohol-impaired driving among youth, their efficacy for DUIC remains limited. Using the six-step Intervention Mapping framework, we developed and tested High Alert, a digital smartphone intervention designed to reduce DUIC among high-risk Canadian youth. The intervention was previously tested in a pilot randomized controlled trial comparing High Alert to an active control (exposure to six DUIC infographics) and a passive control (no contact). This study presents a comprehensive evaluation of High Alert using Step 6 of the Intervention Mapping framework. Reporting on this evaluation serves as a practical guide for researchers utilizing Intervention Mapping, offering valuable insights into High Alert’s formative, process, outcome, and acceptability evaluations to enhance DUIC prevention efforts. Formative and acceptability evaluations revealed High Alert’s positive reception among youth, with most participants willing to engage with it and recommend it to their peers. The program received high ratings for content and delivery, surpassing the static infographics used in the active control. Outcome evaluations demonstrated preliminary efficacy in reducing DUIC behaviour, particularly driving after cannabis co-use, compared to the no-contact group. Process evaluations highlighted implementation challenges, including online study bot activity, recruitment barriers (e.g., participant skepticism, limited ad targeting options), high attrition rates, and low adherence. Findings highlight the importance of Step 6 in Intervention Mapping, emphasizing the need for transparent and rigorous evaluation to inform future interventions. Addressing recruitment and implementation challenges is essential for improving the scalability and effectiveness of interventions targeting high-risk behaviours such as DUIC and will inform High Alert’s future testing.
Citation: Colonna R, Tucker P, Mandich A, Alvarez L (2025) Using intervention mapping to evaluate ‘High-Alert,’ a brief smartphone intervention to reduce youth cannabis-impaired driving. PLoS One 20(8): e0329383. https://doi.org/10.1371/journal.pone.0329383
Editor: Souparno Mitra, NYU Grossman School of Medicine: New York University School of Medicine, UNITED STATES OF AMERICA
Received: January 18, 2025; Accepted: July 15, 2025; Published: August 22, 2025
Copyright: © 2025 Colonna et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant High Alert data files are available from the OSF database (https://doi.org/10.17605/OSF.IO/YT37Z).
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Background
With the legalization of recreational cannabis in Canada and across several U.S. states, there are growing concerns regarding driving under the influence of cannabis (DUIC), particularly among youth [1,2]. Recent evidence suggests that young drivers (ages 16–24 years) engage in the highest rates of cannabis use and DUIC [3,4], and express the least concern towards this risky behaviour [5]. National fatality data reveals that 38% of drivers aged 16–19 and 41% of those aged 20–24 killed in crashes tested positive for cannabis [6]. These rates surpass the 28.3% of all drivers testing positive for cannabis post-legalization (2018–2021) and the 23.3% pre-legalization (2015–2018), suggesting an upward trend [7]. Given that cannabis impairs youth driving performance [8] and increases crash risk [9,10], increasing attention is being paid to strategies aimed at reducing DUIC among young drivers. This includes the growing use of brief digital and smartphone-based approaches [11–16], especially as many DUIC educational programs have shifted online due to the COVID-19 pandemic [17].
For decades, brief educational interventions delivered face-to-face or via computers have shown efficacy in reducing youth substance use and alcohol-impaired driving [18–22]. These brief interventions draw on behaviour change principles to increase perception of risk, enhance self-efficacy, correct misperceptions of descriptive norms, provide personalized feedback, offer normative comparisons, challenge positive alcohol expectancies, and deliver tailored education [19,22,23]. Smartphone-delivered brief interventions applying these principles are gaining popularity for their wider reach, easier access, and enhanced availability and replicability [23,24]. Preliminary evidence from the United States suggests that brief smartphone interventions can reduce alcohol-impaired driving and DUIC among youth [11,20,25,26]. However, additional evidence is needed to develop and test these initiatives for young Canadians [17].
In response to this need, the High Alert intervention was developed using the 6-step Intervention Mapping framework [27] to reduce DUIC among Canadian youth. The Intervention Mapping framework is common in the substance use literature [28,29] and includes the following steps: (1) conducting a needs assessment, (2) identifying intervention objectives, (3) selecting intervention theory-based methods and practical strategies, (4) developing intervention components and materials, (5) planning and executing the initial implementation, and (6) evaluating the intervention [27]. A detailed description of High Alert’s development (Steps 1–4) is reported elsewhere [30], with findings of the initial implementation (Step 5) providing initial support for High Alert’s potential to reduce DUIC behaviours compared to a no-contact control group [31]. The current paper describes Step 6, detailing the findings from the intervention’s systematic evaluation.
Extensive literature underscores the value of Intervention Mapping Steps 1–5, providing insights into program development, theoretical foundations, implementation, and primary outcomes of health promotion programs [28–30,32–43]. However, fewer studies rigorously document Step 6, a necessary evaluation process that reflects the intervention’s fidelity, outcomes, and acceptability, among other components [28–30,32–43]. In fact, the product of Step 6 is a comprehensive evaluation that informs program improvements before implementation (formative evaluation), monitors the intervention’s implementation and fidelity (process evaluation), determines if the final program met its goals and objectives (outcome evaluation), and assesses program acceptability, facilitating future implementations [27,44].
Reporting on this final step is particularly important for interventions addressing high-risk behaviours such as cannabis use and DUIC, like High Alert. While challenges such as participant engagement, attrition, and fidelity are common to many health promotion interventions, digital interventions often face additional complexities, including technical issues, digital literacy barriers, and risks of contamination in virtual settings [24,45,46]. Moreover, digital interventions must comply with several regulatory, ethical, and data protection standards [24], especially when participants disclose criminal behaviour such as DUIC (e.g., as in [31]). Evidence demonstrates that interventions with fidelity assessments are more effective in reducing youth cannabis use [47], yet few smartphone-based programs targeting substance use and impaired driving have been rigorously evaluated, with some exacerbating harmful use [15,16,48]. In this context, evaluation is essential for the High Alert program to support ongoing program improvement, assess initial implementation and fidelity, identify mechanisms of change, and ensure participant acceptability, all of which can inform future testing and implementations.
Thus, this study aimed to conduct a comprehensive evaluation of High Alert using Step 6 of the Intervention Mapping framework. Reporting on this evaluation acts as a practical guide for researchers utilizing Intervention Mapping and offers valuable insights into High Alert’s formative, process, outcome, and acceptability evaluations to enhance DUIC prevention efforts.
Methods
Study design
This study followed Step 6 of the Intervention Mapping framework to conduct a comprehensive evaluation of High Alert’s implementation. The High Alert program was evaluated in a three-arm pilot-randomized controlled trial (ClinicalTrials.gov; NCT06098573) comparing High Alert to an active control (six infographics on cannabis and DUIC) and a passive control (no contact) [31]. Participants were recruited via social media and mass-email recruitment from November 2023 to June 2024 and included Ontario drivers aged 18–24 who owned a smartphone, held a valid driver’s license, had access to a vehicle, and reported DUIC three or more times in the past three months [11,49]. Ethics approval was obtained from Western University’s Research Ethics Board (ID #123368), and all participants provided written informed consent prior to participating.
High Alert program
High Alert consists of four interactive e-learning modules that educate youth on: (1) the effects of cannabis on driving, (2) the risks of DUIC, (3) strategies to avoid DUIC, and (4) risky cannabis use. Module content was tailored to young Ontario drivers and informed by previous studies exploring youth substance use, DUIC, and related interventions [12,13,50–52]. The program is delivered over two weekly sessions (10 minutes each) using the Computerized Intervention Authoring System (CIAS, [53]). A comprehensive overview of each module’s content, theoretical foundations, and design is reported in [30], and full details on the pilot trial (e.g., recruitment, randomization, trial arms) and the primary and secondary outcome measures are reported in [31].
Data collection
Data were collected online through Qualtrics Survey Software and the CIAS platform (i.e., usage data) [53]. Measurements were assessed during the pre-testing phase and in the pilot trial at baseline, post-intervention, and 3-month follow-up. A 3-month timeframe was chosen to align with previous research and enable outcome comparison [11,20,25,26,49]. Table 1 provides an overview of measurement administration.
Evaluation components
The comprehensive evaluation covered formative, process, outcome, and acceptability components. Fig 1 displays the program logic model, providing a visual representation of the High Alert program and where each evaluation component fits within the overall framework. While the formative evaluation and the specific primary and secondary behavioural outcomes were previously reported [30,31], summaries are included here to provide a comprehensive overview of the evaluation plan and integration of findings. All other evaluation components are novel to this paper. Table 2 outlines the comprehensive evaluation plan.
DUIC = driving under the influence of cannabis.
Formative evaluation.
The formative evaluation assessed the intervention’s components prior to implementation [27]. A pretest was conducted with five youths in the target age range and five adult cannabis educators (i.e., professionals with experience delivering Canadian cannabis educational programming) [30] to assess each module’s concepts, content (i.e., believable, convincing, interesting, informative, easy to understand, and personally relevant), and general program feedback [27,54]. Additionally, readability scores were calculated for the overall program. Methods and findings of this formative evaluation are presented in [30].
Process evaluation.
The process evaluation assessed High Alert’s implementation during the pilot trial, following the framework by Linnan and Steckler [55]. Key areas included context (the social and environmental conditions impacting implementation), reach (the percentage of the target audience that received High Alert), dose delivery (the amount of the program delivered), dose received (the amount of the program received and engaged with), fidelity (how closely High Alert was delivered as intended), and recruitment (how participants were recruited). This framework shares similarities with the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance) [56] in its focus on key components such as reach, fidelity, and dose. However, it places more emphasis on process-oriented aspects like context, recruitment, dose delivered, and dose received, while the RE-AIM framework emphasizes broader components like effectiveness, adoption, and maintenance [57]. Data on context and some reach aspects were collected via baseline and 3-month follow-up questionnaires. CIAS usage data provided insights into other reach aspects, dose delivered, and dose received. Fidelity was evaluated with CIAS usage data, while recruitment data came from social media analytics (i.e., number of likes, shares, comments, post removals) and the screening survey.
Outcome evaluation.
The outcome evaluation assessed differences in behavioural outcomes, intentions, and effects on determinants following participation in the High Alert program [27]. Measures were adapted from the validated Youth Cannabis and Diving Survey [50] or created for this study.
Behavioural outcomes
The primary outcome was self-reported DUIC behaviour over the past 3 months, assessed at baseline and the 3-month follow-up. Two variables were analyzed (i) DUIC involving cannabis only, and (ii) total DUIC incidents, including cannabis co-use with alcohol/other drugs. Each was analyzed separately to avoid inflating DUIC counts from question misinterpretation (i.e., reporting the same instance across multiple categories; see [31]). The secondary outcome was risky cannabis use measured using the Cannabis Use Disorder Identification Test-Revised (CUDIT-R [58]). The scale was adapted to assess behaviour over the past 3 months (instead of 6), detecting hazardous use (score 8–11) and potential cannabis use disorder (score 12 + , maximum 32) [58].
Behavioural intentions
DUIC intentions were assessed using two questions: a post-intervention measure evaluating participants’ expected change in DUIC after program completion (rated on a 3-point scale from ‘less than before’ to ‘more than before’), and a change score that measures differences in future DUIC intent between baseline and 3-month follow-up (rated on a 5-point Likert scale from ‘definitely not’ to ‘definitely yes’) [50].
Behavioural determinants
Determinants of DUIC identified in Step 2 of Intervention Mapping included knowledge, attitudes, risk perception, perceived norms, and self-efficacy [30]. These determinants were identified from prior literature that explored DUIC behaviour using various theoretical frameworks, such as social learning theory [59], the theoretical framework of general deterrence and general prevention [50], the theory of planned behaviour [60], and the prototype willingness model [61]. Knowledge was assessed immediately post-intervention with a 27-item knowledge score. Attitudes refer to a person’s feelings toward an object, behaviour, or concept [62] and were measured with two questions gauging participants’ moral stance on whether DUIC is wrong and their beliefs about its acceptability [50]. Risk Perceptions relate to an individual’s subjective judgement on the perceived risks associated with a behaviour [63] and were evaluated with two questions that assess DUIC’s dangerousness [50,59,64,65] and likelihood of experiencing legal repercussions [50]. Perceived norms focused on the perceived value from peers on planning a safe ride home instead of DUIC, while self-efficacy assessed confidence in avoiding DUIC in the coming months and resistance to peer influence [50].
Acceptability evaluation.
The last evaluation assessed participants’ acceptability and reactions to High Alert post-intervention [27,54]. Participants rated the overall program content (i.e., believable, convincing, interesting, informative, easy to understand, and personally relevant) and delivery (i.e., ease of use, engagement) using a 4-point Likert scale from not at all to very. General feedback was solicited with open-ended questions asking participants about the program concepts (i.e., what they liked and disliked) and a question about program recommendations to peers.
Data analysis
Table 2 outlines the data analysis methods for each evaluation variable. Analyses were conducted using IBM SPSS version 26 and R version 4.4.1, incorporating both descriptive (e.g., frequencies, means) and inferential statistics. Generalized Linear Mixed Modelling was used to assess group differences over time for the primary outcome variable (i.e., past DUIC), and a two-way mixed Analysis of Variance (ANOVA) was used to assess the secondary outcome (i.e., risky cannabis use). The Kruskal-Wallis H test was used for assessing change scores across three groups with ordinal (i.e., Likert scale) or non-normal data, while the Mann-Whitney U test was applied for comparisons between two groups under similar conditions. Participants’ written responses to optional open-ended questions were reviewed and presented descriptively rather than formally coded due to the limited number of responses and their supplementary purpose to expand on closed-ended answers (e.g., selecting ‘Yes’ or ‘No’ and providing reasoning) [66].
Results
Formative evaluation
Five adult cannabis educators and five youth participated in the High Alert intervention pre-test, where they offered feedback on the concepts and content of each module and the overall program [30]. In summary, all modules received higher ratings (out of 4) for being informative (3.88), easy to understand (3.85), believable (3.69), and interesting (3.62), compared to personally relevant (3.58) and convincing (3.47) [30]. Module 2 (risks of DUIC) had the highest overall score (3.77) across these categories, followed by Module 1 (effects of cannabis on driving; 3.70), Module 3 (strategies to avoid DUIC; 3.67), and then Module 4 (risky cannabis use; 3.59). Participants’ feedback led to several enhancements in High Alert prior to its implementation: graphics were incorporated for most pages, completion badges were introduced for each module, references were updated, terminology was modified, and technological issues were resolved. The readability scores for the final program corresponded to a 7th-grade reading level (i.e., Flesch Reading Ease Score of 63.7 and a Kincaid Grade Level of 7.2) [30].
Process evaluation
Context.
The pilot randomized controlled trial ran from November 2023 to October 2024 [31]. Two contextual factors identified a priori that may influence DUIC behaviours were assessed and included (i) seeking information on cannabis and/or DUIC and (ii) experiencing any other external factors (i.e., personal, social or environmental). At the 3-month follow-up, 13.5% (n = 7) reported seeking information and provided various comments: “Google impacts,” “I looked into the laws online…”, “I’ve googled the negative side effects…” and “Online training and research...” Likewise, 19.2% (n = 10) reported various external factors influencing their DUIC behaviour, including “home situation,” “new job,” “higher risk insurance rate, so I don’t want to risk an accident,” “due to examination period and new gym schedule,” and “living in the remote north with no taxis.” No statistical differences existed in the proportions of participants across groups reporting information seeking (χ2 (2) = 3.607, p = .165) or external factors (χ2 (2) = 2.588, p = .274). Unfortunately, not all responses indicated whether participants engaged in more or less DUIC as a result of either factor. Another potential contextual factor identified post-hoc [31] was seasonal effects, as cannabis use and impaired driving infractions peak later in the year and decline early into the new year [67,68]. Since most participants were enrolled in late 2023 and completed follow-ups in early 2024, and three mentioned school or icy weather conditions as contextual factors, we suspect timing may have had an effect. However, the impact of these contextual factors remains speculative and warrants further investigation in future studies.
Reach.
There were 102 young drivers (55.4% female, mean age = 20.57 years, SD = 1.93) in the pilot trial, with half (n = 52) completing the 3-month follow-up. After the baseline assessment, 37 were computer-randomized to High Alert, 34 to the active control, and 31 to the passive control. A participant flow diagram is presented in [31]. No significant baseline differences were found between groups. Notably, 13 participants in High Alert and 10 in the active control did not start their program despite being automatically directed to it after completing the baseline assessment. We suspect some participants closed their web browser before submitting the baseline survey (i.e., did not click submit), resulting in Qualtrics initially deleting their response or not sending the program links. A brief analysis in [31] comparing 3-month follow-up completers with non-completers shows that dropouts had higher baseline rates of cannabis use and DUIC. A deeper investigation for this paper comparing baseline rates in the High Alert group shows that DUIC instances were highest among the 13 participants who did not start the program (mean = 25.4, median = 8) and the eight who began but did not finish it (mean = 27, median = 6.5), compared to the 16 who completed it (mean = 12.4, median = 5). Furthermore, 77% of non-starters (10/13) and 62.5% of non-completers (5/8) indicated using cannabis at least four times per week at baseline, whereas only 31.3% (5/16) of completers reported the same.
Dose delivered/Dose received.
Among the 24 participants who started the High Alert program, one did not finish Session 1 (Modules 1 and 2), five did not start Session 2 (Modules 3 and 4), and two started but did not finish Session 2. Two reminder emails containing program links were sent, confirming that access issues were not the reason for non-completion. Session 1 took just under 10 minutes to complete (mean 9.4, median 8.2), and Session 2 took around 12 minutes (mean 12.2, median 9.5). The minimum program completion time was 9 minutes, and the maximum was 51 minutes. As such, 66.7% (n = 16) of participants received the full intervention.
Fidelity.
While all modules were completed in the order specified in the trial protocol, the 1-week timing between sessions varied, demonstrating poor fidelity in delivery. Of the 16 participants who completed both High Alert sessions, just 31% (n = 5) adhered to the intended 7-day spacing between them. Two participants completed the sessions on the same day, three completed them 13–18 days apart, five completed them between 20 and 31 days apart, and one completed them 75 days apart. This variability in session timing may have influenced the overall program effect, highlighting potential challenges in adhering to the prescribed schedule.
Engagement demonstrated good fidelity, with participants in Module 1 (n = 24) responding to 100% of the closed-ended questions (e.g., accept/ challenge) and 92% of the free-text questions. Similar engagement rates were observed in Module 2 (91% closed, 89% free-text; n = 23) and Module 3 (100% closed, 93% free-text; n = 18). Module 4 had slightly lower participants (n = 16) and engagement (85% closed, 89% free-text). Free-text responses for reflections and activities were also engaged with as intended. For instance, when participants were requested to provide “a sentence or two” (Module 1), the average response length was 17.6 words, compared to an average response length of 9.1 words when instructed to “jot down your thoughts” (Module 4). One participant reported issues with embedded video links, and another was suspected of speeding through modules. Aside from these, no other issues were identified, demonstrating good fidelity in engagement.
Recruitment.
Despite various fraud prevention strategies [69–74], initial public social media posts received numerous fake bot responses, leading us to cease open public recruitment. This hindered our ability to monitor various metrics (e.g., social media shares/likes) and accurately track the number of eligible humans screened. Attempts to use paid advertisements on Facebook and Instagram proved costly and inefficient due to restrictions in the audience targeting options. We could only filter by location (Ontario) and ages (18–24), with no specific interest-based targeting options (e.g., cannabis-related groups) due to Meta’s advertising restrictions. This limited our reach to young cannabis users, resulting in low engagement and very high click costs. Some sites (e.g., Reddit and Snapchat) rejected paid advertisements since the study pertained to cannabis. Private social media posts (e.g., Reddit, Discord) yielded some success and attracted little bot attention. However, skepticism was prevalent across these private forums, with many expressing that the study aimed to identify and punish youth admitting to illegal behaviour. We encouraged eligible participants to privately share the recruitment poster with peers (i.e., passive snowball sampling). However, we could not assess this strategy’s effects due to ethical limits on tracking participant actions. Finally, we used Western University’s mass email recruitment system to invite registered students to participate in November 2023, January 2024, and February 2024. Although this approach proved to be the most effective, it also limited the sample’s diversity by focusing solely on students from a single university. Several other Ontario universities and colleges were invited to support recruitment but declined. We estimate that 83.3% (n = 85) of participants were recruited via Western’s mass email system (38 provided a verified Western email address), while the remaining 16.7% (n = 17) came from social media. In the last two months of recruitment, we set values in the survey URL to identify which social media site/recruitment source respondents came from. This strategy proved useful in identifying the recruitment sources that brought in the most participants while attracting minimal bot attention and is recommended for future implementations.
Outcome evaluation
Behavioural outcomes.
Findings from the primary and secondary behavioural outcomes are reported elsewhere [31]. In summary, High Alert showed the greatest mean reductions in DUIC behaviour over the past 3 months compared to the active and passive controls. These reductions were statistically significant when comparing High Alert with the passive control for total DUIC instances (cannabis and co-use), but not the active control or when examining DUIC from cannabis alone. There were no significant differences in risky cannabis use between groups at the 3-month follow-up [31].
Behavioural intentions.
Following the completion of High Alert (n = 16) and the active control (n = 22), significant differences were noted in participants’ expectations of reducing DUIC. In the High Alert group, 94% (15/16) expected a reduction in DUIC, compared to 50% (11/22) in the active control (U = 248.00, z = 2.617, p = .033). Change scores for future DUIC intent showed reductions across all three groups over time. The High Alert group (n = 16) had the largest decrease (mean: −1.31, median: −1.5), followed by the passive control (mean: −0.45, median: −1, n = 16) and then active control (mean: −0.38, median: −0.5, n = 20), although these differences were not statistically significant (H(2)= 4.387, p = .112).
Behavioural determinants.
Table 3 outlines the differences in change scores across groups and the proportion of participants showing more favourable scores. Participants’ knowledge scores rose from baseline to post-intervention and were significantly higher among High Alert than the active control (U = 106.50, z = −2.077, p = .039). High Alert participants (n = 16) knowledge increased by 4.56 points (SD = 3.44) on the 27-point scale, while the active control group (n = 22) had a smaller increase of 2.18 points (SD = 2.82). Notably, High Alert participants demonstrated the greatest improvements across other determinants, except for normative beliefs, where the passive control group showed the largest change (−0.40). High Alert participants also had the highest proportion of individuals with improved scores (compared to no change or worsening attitudes) across all variables, except for normative beliefs and self-efficacy to avoid DUIC. However, none of these changes were statistically significant between groups.
Acceptability
There were 16 participants in High Alert and 22 in the active control that completed the post-test immediately after completing their randomly assigned program. Table 4 displays differences in program content and delivery scores, program recommendations, and perceived impact on DUIC behaviour (results presented above). The High Alert program received a significantly higher rating across all content-related categories except for believability (average 3.63) than the active control (average 2.93), with ratings similar to youth from the pre-test (average 3.66). There wasn’t much difference between the two programs’ averaged delivery scores (3.41 vs 3.29). Notably, 81% (13/16) of people in the High Alert group would recommend this program to their peers, which is significantly higher than the 40% (9/22) who would recommend the active control (U = 94.00, z = −2.568, p = .015). Feedback about High Alert was mostly positive, with 50% (n = 8) of participants providing optional feedback on specific aspects of the program they liked, and 12.5% (n = 2) providing comments on parts they disliked. Positive written examples included “Other participant’s answers,” “knowledge quizzes,” "the phrases to use against peer pressure,” and “the prompts and reflections throughout the module.” The two negative comments were: “Most of the embedded video links didn’t work” and “I felt like the ‘see what other youth answered’ were not real answers, more like model answers.”
Discussion
This study presents the evaluation for High Alert using Step 6 of the Intervention Mapping framework and reports findings from the formative, process, outcome, and acceptability evaluations. Given that Step 6 is often underreported compared to earlier steps in this framework, this study extends the current DUIC prevention literature and serves as a practical guide for researchers employing Intervention Mapping for evaluation.
The formative and acceptability evaluations underscore High Alert’s positive reception among youth, with most participants willing to engage with it and recommend it to peers. The program achieved high content ratings that were significantly greater than the various static infographics displayed to the active control. Examining the findings from each evaluation together provides additional insights. For instance, Module 4 on risky cannabis use received the lowest pre-test ratings and participant engagement scores, which suggests a need for potential refinements before future testing and implementations.
The outcome evaluation provided initial support for High Alert’s potential to reduce DUIC behaviours compared to no contact, particularly when considering driving after cannabis co-use. However, no differences were found when examining the DUIC (cannabis-only) variable, prompting a need for additional testing and better outcome variable measurement. High Alert participants demonstrated significantly greater knowledge gains when compared to the active control group and were more likely to report expectations of reducing DUIC afterwards. This suggests that High Alert’s contents and interactive elements (e.g., reflective exercises, real-life cases, multimedia) might have been more effective at reinforcing learning compared to static materials in the active control. However, knowledge gains did not translate into significant differences across other behavioural outcomes, highlighting a common gap between knowledge acquisition and engaging in risky behaviour [50,75,76].
Aside from knowledge, no significant changes were observed in other behavioural DUIC determinants. One explanation is that the theory-based methods and practical strategies used to deliver content (as described in [30]) may not have effectively influenced the targeted determinants. However, High Alert participants demonstrated the greatest improvements in mean change scores and had the highest proportion of individuals with improved attitudes across most determinants, even though they were not statistically significant. Alternatively, the statistical methods used to assess changes in determinants across groups (i.e., analyzing change scores with the Kruskal-Wallis H test or Mann-Whitney U tests) might have been insufficient to detect subtle changes over time. Although these non-parametric tests were chosen a priori due to the large number of Likert-type variables, violations of normality, and their use in similar studies [77–80], future research could benefit from more robust analytical techniques, such as repeated measures mixed modelling, which Teeters and colleagues [11] used to identify changes in perceptions of DUIC dangers over time.
The process evaluation recognized several recruitment and implementation challenges, offering insights for future improvement. Recruitment was hindered by skepticism about the study’s intentions, limited targeting options in paid advertisements, and the prevalence of bot responses on public social media. Bots are increasingly problematic in online research, and many published fraud prevention methods were ineffective in this trial [69–74]. While mass email recruitment was effective and attracted the least bot attention, it limited the sample diversity. Future research should focus on private recruitment strategies and establish early partnerships (e.g., academic institutions, high schools, driving schools) for mass outreach to verified participants. Public social media could be valuable for future widespread implementation but should be avoided in the interim efficacy testing to mitigate bot exposure. Retention was another challenge, with pilot trial attrition (57%) exceeding that of comparable studies [26,81,82] and highest among those reporting frequent cannabis use and DUIC [31]. This echoes findings from previous studies [83] and highlights the need for tailored strategies to improve retention among higher-risk participants. Another issue was adherence to the 7-day spacing between Sessions 1 and 2, a decision informed by expert and youth recommendations [12,13,30]. This split may have led to greater attrition, as 20% (n = 5) of those who started High Alert did not start Session 2. Unfortunately, we were unable to identify why participants did not adhere to or start the second session. No technical issues were reported, and emails may have been flagged as junk or ignored until the reminder was sent. Future research could also send session reminders via text messaging [84] and explore whether redesigning High Alert into a single session leads to better adherence and fewer dropouts than two separate sessions [85].
Integrating findings from all four evaluations offers a holistic view of High Alert’s initial implementation, preliminary efficacy, and areas for improvement in program delivery and testing. Reporting these findings from Step 6 provides valuable guidance for researchers conducting similar studies, enabling them to develop comprehensive evaluations and proactively address many of the implementation challenges we faced. Step 6 evaluations are often underreported [28–30,32–43], yet they are essential for understanding the complexities of program delivery. Without this data, there is a risk of misinterpreting the effect of interventions, as it overlooks critical elements such as fidelity, participant engagement, and contextual influences. By underscoring the importance of these evaluations, we advocate for more detailed and transparent reporting in intervention research, especially for studies targeting high-risk behaviours like youth cannabis use and DUIC.
Limitations
Despite the strengths of this study, which include using a comprehensive, evidence-based evaluation framework, gathering acceptability findings that can be compared across pre-test and pilot trial participants, and the ability to identify implementation challenges and refine the program before a larger randomized controlled trial, several limitations must also be considered. First, technical issues arose with the Qualtrics surveys and randomization, which may have resulted in a smaller sample size and potential group crossover. Some participants likely closed their browsers before submitting the baseline survey, leading to incomplete responses being deleted and reducing the potential sample size. Although settings were modified soon after to prevent future deletion, some responses continued to be flagged as incomplete and prevented the automatic sending of program links. Efforts were made to identify these participants and manually resend the links, but some cases may have been missed, further reducing the sample. Additionally, during the first week of recruitment, the Qualtrics randomizer failed to record assignments for about 15 participants, requiring manual identification based on program engagement. This increased administrative work and the risk of group crossover (e.g., incorrectly labelling participants as passive control if they didn’t engage). The small sample resulting from these technical issues limits the generalizability of the findings and may have affected the statistical power to detect significant differences in some outcomes. Additionally, the recruitment challenges, including participant skepticism, limited paid advertising options, and bot interference, likely introduced bias and limited the representativeness of the sample.
Second, while Qualtrics surveys were password-protected with participants’ unique codes, the CIAS platform for High Alert lacked this feature. This led to some mismatched codes across platforms, likely from participant typos or bot entries. Since CIAS data were anonymous and only linked via the unique code (with no email addresses stored), we were unable to verify participation and differentiate between legitimate and bot responses for mismatched codes, leading to the exclusion of some data.
Third, participants received immediate access to the first High Alert session after randomization, but the link to the second session was automatically sent one week later, regardless of engagement. While this approach was initially chosen to reduce administrative burden and human errors in sending links, it likely contributed to poor program delivery fidelity. Some participants, having received the link to the second session early, may have completed both sessions in one day, while others experienced large gaps between sessions due to delays in starting the second session.
Fourth, participants’ written responses were reviewed and presented descriptively due to the limited number of responses and their supplementary purpose [66]. However, future trials with larger samples and additional written responses should consider formal qualitative analysis, such as formative or summative content analysis [86], to provide additional insights that may have been missed. Lastly, although the acceptability findings indicate a positive reception of the program, these results may not fully reflect the views of participants who dropped out before completing the program. This is especially relevant for outcome evaluations, as most dropouts reported higher cannabis use and DUIC at baseline.
Future directions
The comprehensive evaluation of the High Alert intervention provided valuable insights into the program’s preliminary efficacy, pilot implementation, and areas for refinement. Key identified improvements include the need for a more rigorous assessment of contextual factors (e.g., clarifying how these factors influence DUIC engagement), enhancing recruitment to mitigate bots and increase the appeal to participate, addressing technical issues, improving retention strategies for high-risk participants, monitoring fidelity rates more effectively, enhancing DUIC outcome measures, and employing more robust analytical techniques to evaluate changes in behavioural determinants. These insights are essential for optimizing the program before further testing. According to the Intervention Mapping framework, Step 5 can be revisited if the program demonstrates efficacy, facilitating additional planning in program adoption, implementation, and sustainability [27]. This proactive evaluation approach establishes a strong foundation for future studies to evaluate digital smartphone interventions (including High Alert) in larger randomized controlled trials and, if found efficacious, to evaluate their effectiveness in real-world settings. Such information is crucial, especially given the common limitations associated with smartphone interventions, including the rapid pace of technological development, challenges with user engagement, technical issues, and the need to comply with regulatory, ethical, and security requirements [23,24,45,46]. Addressing these limitations proactively can enhance the effectiveness of interventions and ensure that digital programs like High Alert can be successfully implemented and sustained at scale. While this pilot study intentionally recruited youth who recently engaged in DUIC to allow for immediate assessment of DUIC behaviour change, High Alert has the potential for broader applications. Expanding to all youth, regardless of current cannabis or DUIC behaviours, could foster a widespread shift in attitudes and norms around the risks of DUIC and better position peers to be environmental agents of change [87–89]. This is advantageous given youth’s lack of awareness of the risks of DUIC [50,51,59,90,91] and can be explored in future research.
Conclusions
Comprehensive evaluations are crucial for programs like High Alert to support ongoing improvements, assess implementation, evaluate program objectives, and ensure participant acceptability for broader applications. Findings from this evaluation offer valuable insights for refining the program and support the design of a larger randomized controlled trial. If proven effective, High Alert may serve as a practical tool for reducing DUIC among Canadian youth and help address this public health issue.
References
- 1.
Woods-Fry H, Robertson RD, Vanlaar WGM. Road safety monitor 2020: trends in marijuana use among Canadian drivers. Ottawa, ON: Traffic Injury Research Foundation; 2020.
- 2.
EKOS Research Associates. Public Opinion Research on Drug Impaired Driving. Public Safety and Emergency Preparedness Canada; 2017.
- 3.
MADD Canada. 2018-2019 Annual Report. Oakville, ON; 2019.
- 4.
Brown S, Vanlaar WG, Robertson RD. Marijuana Use Among Drivers in Canada, 2000-2017: TIRF’s National Fatality Database. 2020. Report No.: 1989766374.
- 5.
Koundakjian K, Maslov A, Ellingwood H. Behaviors and beliefs related to cannabis before legalization: a public safety perspective. Public Safety Canada; 2018.
- 6.
Traffic Injury Research Foundation. Alcohol, Other Drugs & Driving: Know the Facts. Ottawa, ON: Impaired Driving Coalition of Canada; 2023.
- 7.
Brown S, Vanlaar W, Robertson R. Drug Use in Fatal Collisions in Canada | 2000-2021. Traffic Injury Research Foundation; 2024.
- 8. Alvarez L, Colonna R, Kim S, Chen C, Chippure K, Grewal J, et al. Young and under the influence: A systematic literature review of the impact of cannabis on the driving performance of youth. Accid Anal Prev. 2021;151:105961. pmid:33421731
- 9. Rogeberg O, Elvik R. The effects of cannabis intoxication on motor vehicle collision revisited and revised. Addiction. 2016;111(8):1348–59. pmid:26878835
- 10. Rogeberg O. A meta-analysis of the crash risk of cannabis-positive drivers in culpability studies-Avoiding interpretational bias. Accid Anal Prev. 2019;123:69–78. pmid:30468948
- 11. Teeters JB, King SA, Hubbard SM. A mobile phone-based brief intervention with personalized feedback and interactive text messaging is associated with changes in driving after cannabis use cognitions in a proof-of-concept pilot trial. Exp Clin Psychopharmacol. 2021;29(2):203–9. pmid:34043401
- 12. Colonna R, Alvarez L. Characteristics of mobile-based brief interventions targeting substance use among youth: a rapid review. J Substance Use. 2022;28(3):293–304.
- 13. Colonna R, Tucker P, Holmes J, Wilson J, Alvarez L. Mobile-based brief interventions targeting cannabis-impaired driving among youth: A Delphi Study. J Subst Abuse Treat. 2022.
- 14. Sedrati H, Belrhiti Z, Nejjari C, Ghazal H. Evaluation of Mobile Health apps for Non-Medical Cannabis Use: A Scoping Review. Procedia Comput Sci. 2022;196:581–9.
- 15. Tofighi B, Chemi C, Ruiz-Valcarcel J, Hein P, Hu L. Smartphone Apps Targeting Alcohol and Illicit Substance Use: Systematic Search in Commercial App Stores and Critical Content Analysis. JMIR Mhealth Uhealth. 2019;7(4):e11831. pmid:31008713
- 16. Wilson H, Stoyanov SR, Gandabhai S, Baldwin A. The Quality and Accuracy of Mobile Apps to Prevent Driving After Drinking Alcohol. JMIR Mhealth Uhealth. 2016;4(3):e98. pmid:27502956
- 17. Colonna R, Pathan Z, Sultania A, Alvarez L. “Stoned on the road”: A systematic review of cannabis-impaired driving educational initiatives targeting young drivers in Canada. Int J Drug Policy. 2025;142:104835. pmid:40383076
- 18. D’Amico EJ, Parast L, Shadel WG, Meredith LS, Seelam R, Stein BD. Brief motivational interviewing intervention to reduce alcohol and marijuana use for at-risk adolescents in primary care. J Consult Clin Psychol. 2018;86(9):775–86. pmid:30138016
- 19. Scott-Sheldon LAJ, Carey KB, Elliott JC, Garey L, Carey MP. Efficacy of alcohol interventions for first-year college students: a meta-analytic review of randomized controlled trials. J Consult Clin Psychol. 2014;82(2):177–88. pmid:24447002
- 20. Teeters JB, Borsari B, Martens MP, Murphy JG. Brief Motivational Interventions Are Associated With Reductions in Alcohol-Impaired Driving Among College Drinkers. J Stud Alcohol Drugs. 2015;76(5):700–9. pmid:26402350
- 21. Mun E-Y, Li X, Lineberry S, Tan Z, Huh D, Walters ST, et al. Do Brief Alcohol Interventions Reduce Driving After Drinking Among College Students? A Two-step Meta-analysis of Individual Participant Data. Alcohol Alcohol. 2022;57(1):125–35. pmid:33592624
- 22. Steinka-Fry KT, Tanner-Smith EE, Hennessy EA. Effects of Brief Alcohol Interventions on Drinking and Driving among Youth: A Systematic Review and Meta-analysis. J Addict Prev. 2015;3(1):11. pmid:26221619
- 23. Carey KB, Scott-Sheldon LAJ, Elliott JC, Garey L, Carey MP. Face-to-face versus computer-delivered alcohol interventions for college drinkers: a meta-analytic review, 1998 to 2010. Clin Psychol Rev. 2012;32(8):690–703. pmid:23022767
- 24. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. J Med Internet Res. 2017;19(6):e232. pmid:28663162
- 25. Teeters JB, Soltis KE, Murphy JG. A Mobile Phone-Based Brief Intervention With Personalized Feedback and Text Messaging Is Associated With Reductions in Driving After Drinking Among College Drinkers. J Stud Alcohol Drugs. 2018;79(5):710–9. pmid:30422784
- 26. Teeters JB, Armstrong NM, King SA, Hubbard SM. A randomized pilot trial of a mobile phone-based brief intervention with personalized feedback and interactive text messaging to reduce driving after cannabis use and riding with a cannabis impaired driver. J Subst Abuse Treat. 2022;142:108867. pmid:36007434
- 27.
Bartholomew LK, Parcel GS, Kok G, Gottlieb NH. Planning health promotion programs: an intervention mapping approach. 2nd ed. Jossey-Bass/Wiley; 2006.
- 28. Brendryen H, Johansen A, Nesvåg S, Kok G, Duckert F. Constructing a Theory- and Evidence-Based Treatment Rationale for Complex eHealth Interventions: Development of an Online Alcohol Intervention Using an Intervention Mapping Approach. JMIR Res Protoc. 2013;2(1):e6. pmid:23612478
- 29. Voogt CV, Poelen EAP, Kleinjan M, Lemmers LACJ, Engels RCME. The development of a web-based brief alcohol intervention in reducing heavy drinking among college students: an Intervention Mapping approach. Health Promot Int. 2014;29(4):669–79. pmid:23525645
- 30. Colonna R, Tucker P, Mandich A, Alvarez L. Developing a mobile-based brief intervention to reduce cannabis-impaired driving among youth: An intervention mapping approach. Int J Drug Policy. 2024;134:104626. pmid:39476788
- 31.
Colonna R, Tucker P, Mandich A, Alvarez L. Preliminary efficacy of ‘high-alert’ a brief smartphone intervention to reduce cannabis-impaired driving among youth: A pilot randomized controlled trial. 2025 (Forthcoming).
- 32. Douet L, Milne R, Anstee S, Habens F, Young A, Wright D. The completeness of intervention descriptions in published National Institute of Health Research HTA-funded trials: a cross-sectional study. BMJ Open. 2014;4(1):e003713. pmid:24384896
- 33. Schroter S, Glasziou P, Heneghan C. Quality of descriptions of treatments: a review of published randomised controlled trials. BMJ Open. 2012;2(6):e001978. pmid:23180392
- 34. Conn VS, Cooper PS, Ruppar TM, Russell CL. Searching for the intervention in intervention research reports. J Nurs Scholarsh. 2008;40(1):52–9. pmid:18302592
- 35. Gosens LCF, Otten R, de Jonge JM, Schellekens AFA, VanDerNagel JEL, Didden R, et al. Development of a personalised substance use disorder treatment for people with mild intellectual disabilities or borderline intellectual functioning: An intervention mapping approach. J Intellect Dev Disabil. 2022;47(2):131–40. pmid:39818578
- 36. Hadjiconstantinou M, Schreder S, Brough C, Northern A, Stribling B, Khunti K, et al. Using Intervention Mapping to Develop a Digital Self-Management Program for People With Type 2 Diabetes: Tutorial on MyDESMOND. J Med Internet Res. 2020;22(5):e17316. pmid:32391797
- 37. Potzel AL, Gar C, Seissler J, Lechner A. A Smartphone App (TRIANGLE) to Change Cardiometabolic Risk Behaviors in Women Following Gestational Diabetes Mellitus: Intervention Mapping Approach. JMIR Mhealth Uhealth. 2021;9(5):e26163. pmid:33973864
- 38. Schönbach DM, Chillón P, Marques A, Peralta M, Demetriou Y. Study protocol of a school-based randomized controlled trial to promote cycling to school among students in Germany using intervention mapping: the ACTS Project. Front Public Health. 2021:1050.
- 39. Svendsen MJ, Sandal LF, Kjær P, Nicholl BI, Cooper K, Mair F, et al. Using Intervention Mapping to Develop a Decision Support System-Based Smartphone App (selfBACK) to Support Self-management of Nonspecific Low Back Pain: Development and Usability Study. J Med Internet Res. 2022;24(1):e26555. pmid:35072645
- 40. van Beurden SB, Greaves CJ, Abraham C, Lawrence NS, Smith JR. ImpulsePal: The systematic development of a smartphone app to manage food temptations using intervention mapping. Digit Health. 2021;7:20552076211057667. pmid:34868619
- 41. Bakhuys Roozeboom MC, Wiezer NM, Boot CRL, Bongers PM, Schelvis RMC. Use of Intervention Mapping for Occupational Risk Prevention and Health Promotion: A Systematic Review of Literature. Int J Environ Res Public Health. 2021;18(4):1775. pmid:33670376
- 42. Garba RM, Gadanya MA. The role of intervention mapping in designing disease prevention interventions: A systematic review of the literature. PLoS One. 2017;12(3):e0174438. pmid:28358821
- 43. Fassier JB, Sarnin P, Rouat S, Péron J, Kok G, Letrilliart L, et al. Interventions Developed with the Intervention Mapping Protocol in Work Disability Prevention: A Systematic Review of the Literature. J Occup Rehabil. 2019;29(1):11–24. pmid:29948471
- 44.
Rossi PH, Lipsey MW, Freeman HE. Evaluation: A systematic approach. Sage Publications; 2003.
- 45. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res. 2011;13(4):e126. pmid:22209829
- 46. Eysenbach G. Issues in evaluating health websites in an Internet-based randomized controlled trial. J Med Internet Res. 2002;4(3):E17. pmid:12554548
- 47. O’Hara SE, Paschall MJ, Grube JW. A Review of Meta-Analyses of Prevention Strategies for Problematic Cannabis Use. Curr Addict Rep. 2024;11(5):886–93. pmid:39328973
- 48. Crane D, Garnett C, Brown J, West R, Michie S. Behavior change techniques in popular alcohol reduction apps: content analysis. J Med Internet Res. 2015;17(5):e118. pmid:25977135
- 49.
Statistics Canada. National cannabis survey: Detailed information for 2020. Ottawa, ON: Statistics Canada; 2021.
- 50. Colonna R, Hand CL, Holmes JD, Alvarez L. Young drivers’ determinants of driving under the influence of cannabis: Findings from the Youth Cannabis and Driving Survey (YouCanDS). J Safety Res. 2021;78:229–41. pmid:34399919
- 51. Colonna R, Hand CL, Holmes JD, Alvarez L. Exploring youths’ beliefs towards cannabis and driving: A mixed method study. Transp Res F Traffic Psychol Behav. 2021;82:429–39.
- 52. Alvarez L, Colonna R. Characteristics of a technology-based intervention for young drivers’ hazard perception: A Delphi study. J Transp Health. 2020;19:100951.
- 53. Ondersma SJ, Chase SK, Svikis DS, Schuster CR. Computer-based brief motivational intervention for perinatal drug use. J Subst Abuse Treat. 2005;28(4):305–12. pmid:15925264
- 54.
National Cancer Institute. Making health communication programs work: A planner’s guide. 2002.
- 55.
Linnan L, Steckler A. Process evaluation for public health interventions and research. 2002.
- 56. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7. pmid:10474547
- 57.
Al Daccache M, Bardus M. Process Evaluation. In: The Palgrave Encyclopedia of Social Marketing. Cham: Springer International Publishing; 2020. p. 1–4.
- 58. Adamson SJ, Kay-Lambkin FJ, Baker AL, Lewin TJ, Thornton L, Kelly BJ, et al. An improved brief measure of cannabis misuse: the Cannabis Use Disorders Identification Test-Revised (CUDIT-R). Drug Alcohol Depend. 2010;110(1–2):137–43. pmid:20347232
- 59. Aston ER, Merrill JE, McCarthy DM, Metrik J. Risk Factors for Driving After and During Marijuana Use. J Stud Alcohol Drugs. 2016;77(2):309–16. pmid:26997189
- 60. Earle AM, Napper LE, LaBrie JW, Brooks-Russell A, Smith DJ, de Rutte J. Examining interactions within the theory of planned behavior in the prediction of intentions to engage in cannabis-related driving behaviors. J Am Coll Health. 2020;68(4):374–80. pmid:30681931
- 61. Scott B, Ward N, Otto J, Finley K. Modeling the system of beliefs that influence driving under the influence of cannabis (DUIC) in Washington State. Accid Anal Prev. 2021;151:105988. pmid:33484972
- 62. Ajzen I, Fishbein M. Attitude-behavior relations: A theoretical analysis and review of empirical research. Psychol Bulletin. 1977;84(5):888–918.
- 63. Rosenstock IM, Strecher VJ, Becker MH. Social learning theory and the Health Belief Model. Health Educ Q. 1988;15(2):175–83. pmid:3378902
- 64. Arterberry BJ, Treloar H, McCarthy DM. Empirical Profiles of Alcohol and Marijuana Use, Drugged Driving, and Risk Perceptions. J Stud Alcohol Drugs. 2017;78(6):889–98. pmid:29087824
- 65. McCarthy DM, Lynch AM, Pederson SL. Driving after use of alcohol and marijuana in college students. Psychol Addict Behav. 2007;21(3):425–30. pmid:17874895
- 66. O’Cathain A, Thomas KJ. Any other comments? Open questions on questionnaires–a bane or a bonus to research? BMC Med Res Methodol. 2004;4:1–7.
- 67. Palamar JJ, Le A, Han BH. Quarterly trends in past-month cannabis use in the United States, 2015-2019. Drug Alcohol Depend. 2021;219:108494.
- 68. Juth V, Smyth JM, Thompson K, Nodes J. Social and Environmental Predictors of Alcohol-Related Legal Infractions in College Students. J College Stud Develop. 2010;51(4):373–84.
- 69.
Wang H, Luo X, Wang W, Yan X. Bot or human? detecting chatgpt imposters with a single question. arXiv preprint arXiv:230506424. 2023.
- 70.
Zhang Z, Zhu S, Mink J, Xiong A, Song L, Wang G, editors. Beyond bot detection: combating fraudulent online survey takers. In: Proceedings of the ACM Web Conference 2022; 2022.
- 71. Xu Y, Pace S, Kim J, Iachini A, King LB, Harrison T, et al. Threats to Online Surveys: Recognizing, Detecting, and Preventing Survey Bots. Social Work Res. 2022;46(4):343–50.
- 72. Bybee S, Cloyes K, Ellington L, Baucom B, Supiano K, Mooney K. Bots and nots: Safeguarding online survey research with underrepresented and diverse populations. Psychol Sex. 2022;13(4):901–11. pmid:36439051
- 73.
Brainard J, Lane K, Watts L, Bunn D. The wasps are clever: Keeping out and finding bot answers in internet surveys used for health research. 2022.
- 74. Storozuk A, Ashley M, Delage V, Maloney EA. Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks. Quantit Method Psychol. 2020;16(5):472–81.
- 75. Ajzen I, Joyce N, Sheikh S, Cote NG. Knowledge and the Prediction of Behavior: The Role of Information Accuracy in the Theory of Planned Behavior. Basic Appl Soc Psychol. 2011;33(2):101–17.
- 76. Gibbons FX, Gerrard M. Predicting young adults’ health risk behavior. J Pers Soc Psychol. 1995;69(3):505–17. pmid:7562392
- 77. Nordholt PU, Christalle E, Zill JM, Dirmaier J. Engagement With a Web-Based Intervention to Reduce Harmful Drinking: Secondary Analysis of a Randomized Controlled Trial. J Med Internet Res. 2020;22(11):e18826. pmid:33216008
- 78. Burner E, Zhang M, Terp S, Ford Bench K, Lee J, Lam CN, et al. Feasibility and Acceptability of a Text Message-Based Intervention to Reduce Overuse of Alcohol in Emergency Department Patients: Controlled Proof-of-Concept Trial. JMIR Mhealth Uhealth. 2020;8(6):e17557. pmid:32496203
- 79. Chan Y, Walmsley RP. Learning and understanding the Kruskal-Wallis one-way analysis-of-variance-by-ranks test for differences among three or more independent groups. Phys Ther. 1997;77(12):1755–62. pmid:9413454
- 80. Thompson RG, Aivadyan C, Stohl M, Aharonovich E, Hasin DS. Smartphone application plus brief motivational intervention reduces substance use and sexual risk behaviors among homeless young adults: Results from a randomized controlled trial. Psychol Addict Behav. 2020;34(6):641–9. pmid:32175752
- 81. Linardon J, Fuller-Tyszkiewicz M. Attrition and adherence in smartphone-delivered interventions for mental health problems: A systematic and meta-analytic review. J Consult Clin Psychol. 2020;88(1):1–13. pmid:31697093
- 82. Prior E, Dorstyn D, Taylor A, Rose A. Attrition in Psychological mHealth Interventions for Young People: A Meta-Analysis. J Technol Behav Sci. 2023;9(4):639–51.
- 83. Radtke T, Ostergaard M, Cooke R, Scholz U. Web-Based Alcohol Intervention: Study of Systematic Attrition of Heavy Drinkers. J Med Internet Res. 2017;19(6):e217. pmid:28659251
- 84. Schwebel FJ, Larimer ME. Text message reminders as an adjunct to a substance use intervention for adolescents and young adults: Pilot feasibility and acceptability findings. Digit Health. 2020;6:2055207620965052. pmid:33110614
- 85. Duan N, Braslow JT, Weisz JR, Wells KB. Fidelity, adherence, and robustness of interventions. Psychiatr Serv. 2001;52(4):413. pmid:11274482
- 86. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. pmid:16204405
- 87. Brechwald WA, Prinstein MJ. Beyond Homophily: A Decade of Advances in Understanding Peer Influence Processes. J Res Adolesc. 2011;21(1):166–79. pmid:23730122
- 88. Caouette JD, Feldstein Ewing SW. Four mechanistic models of peer influence on adolescent cannabis use. Current Addict Rep. 2017;4:90–9.
- 89. Leadbeater BJ, Foran K, Grove-White A. How much can you drink before driving? The influence of riding with impaired adults and peers on the driving behaviors of urban and rural youth. Addiction. 2008;103(4):629–37. pmid:18339107
- 90. McDonald AJ, Hamilton HA, Wickens CM, Watson TM, Elton-Marshall T, Wardell JD, et al. Driving under the influence of cannabis risk perceptions and behaviour: A population-based study in Ontario, Canada. Prev Med. 2021;153:106793. pmid:34517043
- 91.
McKiernan A, Fleming K. Canadian Youth Perceptions on Cannabis. 2017. Report No.: 9781771783712.