Ethics

This study received approval from the organization’s Research Ethics Board (REB #2024/138) before initiation and prior to participation, all individuals provided written informed consent.

Study design

This study follows the Guidance for Publishing Qualitative Research in Informatics checklist to ensure transparency and comprehensive reporting [24]. S1 Table in the Supporting Information Files provides the checklist along with the location of each element within the manuscript.

This study adopted a mixed methods approach, primarily qualitative in nature, with quantitative data used to assess feasibility. The study was guided by the RE-AIM framework, which includes five categories: Reach, Effectiveness, Adoption, Implementation, and Maintenance [19]. Reach was assessed quantitatively by measuring the proportion of TAY participants who consented to share their data with clinicians, serving as a proxy for clinician access to the dashboards. This analysis utilized existing routinely collected data from the TAY Cohort Study to confirm that clinicians had sufficient access to the dashboards, thereby establishing the feasibility of conducting the qualitative component of the study.

The remaining RE-AIM categories were evaluated using qualitative data from clinician interviews. Effectiveness was assessed at the clinician level to explore the dashboards’ perceived impact on decision-making and care processes. Adoption explored the extent to which clinicians integrated dashboards into their workflows and the factors influencing uptake. Implementation considered the fidelity and consistency of dashboard delivery as intended, based on clinicians’ experiences. Maintenance was assessed through clinicians’ perceptions of whether dashboards could be sustained and incorporated into routine practice over time. Due to limitations in system analytics and the focus on clinician experiences rather than patients, a gap currently identified in the literature, objective, patient-level, and indicator-based metrics were not included in this evaluation [25].

Setting and timeline

The present evaluation was conducted within the Child, Youth, and Family (CYF) services program, which encompasses inpatient and outpatient mental health care for children, youth, and their families. While individual clinics may serve different populations, the program overall treats individuals aged five to the mid- to late twenties. Care ranges from consultation to short- and long-term follow-up, without standardized criteria for ongoing care. Patients and/or their caregivers within the CYF services program may be invited to participate in the TAY Cohort Study (https://www.taycohort.ca/), and those who consent to share their results from the study via dashboards allow clinicians to access this information through the EMR system. Patients may be referred to the study at any point in their care timeline, for example during their initial intake visit or later while receiving ongoing care. The timeline from consent to dashboard availability involves several steps. After consent, participants complete diagnostic assessments and self-report measures, which take approximately three to four hours in total. Research staff then present findings at consensus meetings with senior clinicians, held to review assessment results and reach agreement on diagnoses. Once consensus is reached, dashboards are populated with diagnostic assessment findings and self-report measures and uploaded to the general documentation section of the EMR system for each patient. For an example of the TAY Cohort Study dashboard populated with mock data, please contact the corresponding author. The patient’s primary clinicians (i.e., psychologist or psychiatrist) are notified via email to their organizational account when a dashboard has been completed for their patient; however, all clinicians who are part of the patient’s care team can access the TAY Cohort Study dashboard through the EMR system. Although the TAY Cohort Study is longitudinal, dashboards are currently developed only for baseline assessments. For consistency, within the present manuscript, patients who chose to participate in the TAY Cohort Study will hereafter be referred to only as participants, with the intention of capturing their dual role as both patients in the CYF services program and TAY Cohort Study participants.

Participants and procedures

Routinely collected data were first requested from the TAY Cohort Study team to confirm clinician access to the dashboards by determining the proportion of TAY participants who consented to share their data with clinicians, thereby establishing feasibility for the qualitative component. Eligible participants were clinicians in the CYF services program who had access to, or prior experience using the dashboards. Recruitment occurred via email and during clinical team meetings, where a research team member introduced the study. Each clinician was contacted three times over a period of three to four weeks to accommodate busy schedules and increase the likelihood of participation. Interested clinicians scheduled a session with a research analyst to provide informed consent and complete a 30-minute semi-structured qualitative interview, conducted in person or via videoconference. Clinicians were recruited and interviewed between April 2, 2025, and May 26, 2025. As the TAY Cohort Study is ongoing and continues to actively recruit, this evaluation includes data from participants enrolled between May 4, 2021, and March 12, 2025 (1,053 out of a target of 1,500). At the time of this data cut, approximately 63 clinicians including psychologist, psychiatrists, nurses, and social workers, had access to the dashboards.

Measures

Quantitative data routinely collected through the TAY Cohort Study, specifically participant consent and dashboard availability, were extracted to assess dashboard reach. These data included: (1) the date of the participant’s consent discussion; (2) whether consent was provided to share dashboards with clinicians; and (3) the date of dashboard completion. A semi-structured interview guide, informed by four of the five RE-AIM framework categories, was developed to explore clinicians’ perceptions of the dashboards, with example questions presented in Table 1. The interview guide was co-developed with the TAY Cohort Study’s youth advisory group to ensure their perspectives on the use of their clinical data were fully incorporated. Youth input focused on understanding what they want to know about how clinicians interact with their research data and how this may affect their care. Their feedback was actively integrated to address the questions and concerns most important to them. In addition to the interview guide, clinician participants provided basic information about their professional designation, years of experience, and the populations with whom they typically work.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. RE-AIM framework definitions, data sources, example questions, and subcategories.

https://doi.org/10.1371/journal.pmen.0000605.t001

Data analytic strategy

Descriptive statistics summarized dashboard reach, using three metrics: (1) the proportion of participants who consented to share their results with clinicians; (2) among those who consented, the proportion of dashboards completed between May 4, 2021, and March 12, 2025; and (3) the average time from participant consent to dashboard completion. To assess potential effects of time on completion duration, reflecting early implementation challenges, average completion times were compared between the first and second halves of participants who had their dashboards completed. Quantitative data were analyzed using SPSS version 29.0 [26].

Semi-structured qualitative interviews were audio-recorded and transcribed verbatim. Data were analyzed concurrently with collection using directed content analysis, a commonly used method in healthcare research that facilitates examination of data through predefined concepts, in this case the RE-AIM framework [27]. Conducting analysis alongside data collection allowed early insights to inform subsequent interviews, enabling exploration of emerging subcategories and refinement of the inquiry based on observed patterns. Data collection ceased once insights from clinician interviews began to sound redundant and no new subcategories were emerging [28]. NVivo software version 14.0 was used to manage and organize the qualitative data [29].

Each transcript was coded according to the RE-AIM framework categories, and illustrative quotations were selected to highlight key subcategories. Subcategories within each RE-AIM category were developed deductively from the interview data and refined through ongoing discussions among the researchers, with overlapping or redundant categories merged or eliminated to create a clear coding structure. To enhance the trustworthiness of the findings, the first author (SK) led the coding, analysis, and interpretation, regularly reviewing decisions with the third (JD) and last (KC) authors [30]. These researchers collaboratively discussed interpretations, resolved discrepancies, and reached consensus on the definitions of each subcategory within the RE-AIM framework. Representative quotations are provided alongside each subcategory to support the validity of the findings [31].

Sample description

Seven clinicians participated in this study, resulting in a response rate of just over 11%. Clinicians represented disciplines such as medicine (psychiatry), psychology, and nursing, with experience ranging from a couple of months to over 20 years. The characteristics of the recruited sample largely resemble those of the broader clinician population who had access to and/or used the dashboards. The populations with whom they worked included adolescents and youth aged 11–24 with autism, mood and anxiety disorders, substance use disorders, and concurrent disorders, across inpatient and outpatient settings.

Primary findings

Quantitative data informing the Reach category, along with subcategories derived from interview transcripts informing the other RE-AIM categories, are presented below. Subcategories are organized by RE-AIM category and illustrated with representative quotes. Table 1 summarizes the subcategories, their data sources, and example interview questions, and Fig 1 provides a visual representation of the subcategories.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Visual representation of the subcategories by each RE-AIM category.

https://doi.org/10.1371/journal.pmen.0000605.g001

Reach.

Out of a final enrollment target of 1,500 participants for the TAY Cohort Study, 1,053 had been recruited as of March 12, 2025. Descriptive analyses indicated that 1,024 participants (97.2%) consented to share their study data with clinicians. Among these, 709 participants (69.2%) had their dashboards completed and uploaded into the EMR system, by the same date. The mean time from consent to dashboard completion was 251 days (SD = 151.92), or approximately 8.2 months, with a range of 37–794 days. Dashboards that were not completed or uploaded resulted from delays in scheduling diagnostic assessment interviews, delays in holding consensus meetings, or delays in populating dashboards after these steps were completed.

To account for potential delays during the initial implementation phase, the time from consent to dashboard completion was compared between the first half of participants (n = 354 of 709) and the subsequent half (n = 355 of 709) whose dashboards had already been completed and uploaded into the EMR system. Among the first half, the mean time from consent to completion was 320 days (SD = 170.88), approximately 10.5 months, with a range of 47–794 days. In the subsequent half, the mean time decreased to 182 days (SD = 86.35), approximately 6 months, with a range of 37–526 days. A breakdown of dashboard completion per 100-day interval across the TAY Cohort Study period is presented in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Dashboard completion per 100-day interval.

https://doi.org/10.1371/journal.pmen.0000605.g002

Implications

This study underscores the importance of involving clinicians not only in the co-design and implementation of dashboards but also in their ongoing evaluation to maintain clinical relevance [46,47]. Equally important is the engagement of operational staff responsible for deployment and maintenance to ensure effective functioning. For example, this study identified delays between participant consent and dashboard completion. Although specific segments of the process could not be isolated, efforts are underway to reduce delays by streamlining workflows in collaboration with research staff. A key contributor to these delays is the need to confirm diagnostic assessment findings in consensus meetings, which typically occur one to two months after assessments. The organization is working with research staff to increase the frequency of presenting diagnostic findings at these meetings, with the goal of ensuring they occur within days of the assessment. This approach may reduce the timeline from participant consent to dashboard completion by one to two months. Alternatively, as suggested in this study, removing assessments that require confirmation through consensus meetings may also help reduce delays, since self-report assessments are immediately available for use in clinical contexts. These findings highlight that successful dashboard integration depends on both clinician engagement and coordinated staff contributions, emphasizing the importance of their involvement in evaluation initiatives.

Emerging implications of this study may also suggest the potential value of leadership roles such as Chief Medical Informatics Officers, who serve as liaisons between clinical practice and information technology within academic healthcare organizations [48], as these individuals and their teams can help prioritize and support dashboard implementation. Areas for improvement identified in this study, including enhancing dashboard visibility and developing clinician-requested alerts, may particularly benefit from their expertise and oversight. The organization in which this study was conducted has established a Digital Innovation Hub (https://www.camh.ca/en/science-and-research/digital-innovation-hub) to accelerate the design, development, implementation, and scaling of digital health innovations to support mental health and substance use care by fostering partnerships among researchers, clinicians, and industry partners. Although the present study did not leverage this resource, as it was established after the initiation of the TAY Cohort Study, future work could benefit from such collaboration. Healthcare organizations considering dashboard implementation may consider assessing whether comparable roles or resources exist, as they may be helpful in supporting successful implementation, enhancing effectiveness, and facilitating the integration of research data into routine clinical care.

The findings of this study may also emphasize the importance of pilot testing dashboard tools with representative clinicians within their workflows, rather than solely relying on co-design, before broader implementation. While co-design can ensure dashboards are visually appropriate, usability testing provides critical insights into practical challenges, such as difficulties locating dashboards or inefficient notification processes, that may impede adoption or reduce effectiveness. Prioritizing usability testing within real world clinical contexts enables early identification of workflow mismatches and ensures tools are both relevant and usable. Engaging clinicians early in this process may further enhance engagement, increases perceived utility, and supports long-term adoption [11,49,50]. Despite its importance, formative usability testing is reported even less frequently, with only 22% of studies incorporating this process [42]. Future dashboard development and evaluation should consider prioritizing usability testing to enhance dashboard quality, facilitate integration into clinical workflows, and support the translation of research data into routine clinical care, ultimately maximizing their impact on mental health care [51].

Strengths and limitations

The present study has several strengths, including a novel contribution to an underexplored area, a theory-informed evaluation using the RE-AIM framework, a mixed methods design that integrated both qualitative and quantitative data, and incorporation of end-user experiences to address the study objectives. However, several important limitations should be considered when interpreting these findings. First, the sample included only seven clinicians, primarily psychologists and psychiatrists, with limited representation from nursing, resulting in a response rate of approximately 11%. Although multiple disciplines were represented, the small and self-selected sample may not fully capture the range of experiences with the dashboards within the CYF services program. In particular, perspectives from social workers, nurses, and other clinicians were underrepresented, which may limit the validity and transferability of these findings. Furthermore, while data collection ceased once interviews with participating clinicians began to show redundancy and no new subcategories emerged, additional perspectives from underrepresented professions may have yielded new or divergent insights or further corroborated the present findings. Second, due to data limitations, differences between specific segments of the dashboard completion timeline, including consent to diagnostic assessment, assessment to consensus meeting, and consensus meeting to dashboard availability, could not be examined. Additionally, due to data limitations within the EMR system, we were unable to determine the total number of clinicians who have used the TAY Cohort Study dashboards. Third, the effectiveness of the clinician-facing dashboards was evaluated solely at the clinician level using qualitative interviews. No objective or indicator-based data were incorporated to assess effectiveness, such as changes in clinical decision-making, participant outcomes, or service efficiency. Fourth, self-selection bias is a potential concern, as clinicians with stronger opinions, either positive or negative, may have been more likely to participate, whereas less engaged or neutral clinicians may have opted out, limiting perspective diversity [52]. Some participants also reported implementation challenges that affected dashboard use, limiting the ability to fully assess categories such as adoption, effectiveness, and sustainability due to their limited hands-on experience.

Future directions

Further research is needed to explore these implementation issues in greater depth, ideally through the integration of multiple data sources (e.g., objective, indicator-based metrics) and methodological approaches. Research in this area should also seek to identify additional contextual factors that may hinder the uptake of clinician-facing research dashboards in clinical care, particularly given the limited evaluation of such dashboards in mental health settings [13]. It will also be important to examine the experiences of mental health professionals from other disciplines in their use of clinician-facing dashboards to identify potential differences in their experiences. Finally, subsequent evaluations may benefit from reassessing RE-AIM categories after initial implementation challenges have been addressed, as perceptions of reach, effectiveness, adoption, implementation, and maintenance may evolve over time. Attention should also be given to how barriers encountered within one category may influence or shape perceptions in other domains, given the interrelated nature of the RE-AIM framework.

Conclusion

Guided by the RE-AIM framework, this study used a mixed-methods design that was primarily qualitative, supplemented by quantitative data, to evaluate the reach, implementation, adoption, effectiveness, and maintenance of clinician-facing dashboards developed from a large adolescent and youth mental health and functioning cohort study at a research and teaching hospital in Canada. Findings indicate that clinicians viewed the dashboards as a valuable supplementary source of information that may aid in supporting clinical decision-making and the integration of participant-reported data into care. Challenges such as delays in dashboard completion, difficulty locating dashboards, and misaligned notification processes limited adoption and may have hindered a full assessment of effectiveness, implementation, and sustainability. Clinicians recommended improving access, timely data availability, and communication strategies to enhance usability and uptake. These findings may suggest the importance of engaging end users, operational staff, and subject matter experts collaboratively, alongside usability testing and continuous evaluation, to support the effective integration of research data into clinical workflows and improve mental health care. Future research should explore factors affecting dashboard use in various mental health care settings and how challenges in one RE-AIM category may influence others.