https://orcid.org/0000-0001-6189-6857
Figures
Abstract
In Canada, public health measures necessitated by the COVID-19 pandemic resulted in a rapid onset and prolonged, widespread increase in the use of virtual primary care services, including for mental health conditions. Our aim was to develop standards on virtual delivery of mental health services in primary care in Canada using information obtained from an earlier rapid review as well as participant feedback obtained through interviews and a focus group. We developed standards using three interlinked processes. First, we completed a rapid review of guidelines regarding virtual primary mental health care services. We then invited health care workers and people with lived experience of mental health concerns to participate in a focus group and interviews. Finally, members of the study team drafted standards and shared them with an advisory group, who reviewed their feasibility, phrasing, and acceptability through a modified Delphi process. Standards ranked as having less than 100% feasibility and acceptability were brought to a virtual discussion of the advisory group to finalize the list. Seven participants were recruited into the focus group and interviews. We identified three themes: (i) patients’ and providers’ agreement about expectations regarding virtual care, (ii) accessibility and equity, and (iii) safety planning in the delivery of virtual care. We drafted 18 standards on virtual primary mental health care delivery that were reviewed by an advisory group of identified experts. Thirteen standards were included in the final list. The standards bring attention to continuity of care, and resources and information that should be given to patients to further health equity. These standards provide guidance for the organization and delivery of virtual mental health services in Canadian and international primary care, particularly within the context of single payer health systems.
Citation: Yusuf A, Salvalaggio G, Melamed O, Montesanti S, Atherton H, Langford L, et al. (2024) Developing standards for virtual delivery of mental health services in Canadian primary care: A qualitative study and modified Delphi process. PLOS Ment Health 1(5): e0000071. https://doi.org/10.1371/journal.pmen.0000071
Editor: Ariel Soares Teles, Instituto Federal do Maranhão: Instituto Federal de Educacao Ciencia e Tecnologia do Maranhão, BRAZIL
Received: March 18, 2024; Accepted: September 20, 2024; Published: October 17, 2024
Copyright: © 2024 Yusuf et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data related to this project are stored on a secure server at Unity Health Toronto and are not publicly available because of limitations required by the research ethics approval for the project. Interested parties can contact the Research Ethics Board of Unity Health Toronto (researchethics@unityhealth.to) for further details.
Funding: This work was supported by the Canadian Institutes of Health Research (QA4 181019 to BO). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Background
As a result of the COVID-19 pandemic and related public health restrictions, virtual delivery of health care became far more common in high income nations and has persisted since [1]. Virtual care is "any interaction between patients and/or members of their circle of care, occurring remotely, using any forms of communication or information technologies…" [2]. In the first months of the COVID-19 pandemic in Canada, the usage of virtual care increased substantially; for example, there was a 5,600% increase in virtual visits in Ontario; virtual care made up more than 70% of primary care visits [3]. Between 2019 and 2020, only 4% of all physician-provided mental health services were delivered virtually; one year later, this percentage increased to 57% of all physician-provided mental health services [4, 5].
Approximately 20% of people in Canada will experience a mental health issue each year [6]. Mental health services in Canada are often first accessed through primary care [7], with the most common reasons for primary care visits being anxiety and depression [8, 9]. Yet, approximately 17% of Canadians–at least 6 million people–do not have access to primary care providers [10]. Given that virtual care services are proposed to improve access to primary health care for many people while decreasing the cost of delivering care when performed well [11, 12] and that psychiatry-based mental health care has been reported as particularly amenable to virtual care delivery compared to other forms of physical ailments [13], it is evident that virtual delivery of mental health services will be an important ongoing pillar of primary health care delivery. Thus, improving access to virtual care will be an important component of improving health equity within the Canadian health care system [12].
Substantial increases in virtual care use have resulted in an abundance of literature on virtual care service delivery [12–32]. Our team completed a rapid review of guidance and standards for virtual delivery of mental health services from high-income country settings and identified 40 resources from 9 countries [33]. Themes from content analysis of those resources included: screen patients for appropriateness of virtual care; obtain emergency contact details; communicate transparently with patients; improve marginalised patients’ access to care; support health equity for all patients; determine the cost-effectiveness of virtual care; inform patients of insurance coverage for virtual care services; increase provider training for virtual care and set professional boundaries between providers and patients [33]. Despite these resources being available in other countries, efforts to organise virtual care guidance and recommendations have been limited in Canada until recently [33]. Existing frameworks and guidelines for virtual care in Canada, such as the Canadian Medical Association’s 2021 ‘Virtual Care Playbook’ [34] and the Foundation of Medical Regulatory Associations of Canada’s 2021 guidance [35] are largely focused on operational considerations such as what size screen to use, or noting that the limitations of virtual care must be disclosed to patients. Existing guidance is therefore inadequate to support primary care providers to be able to provide safe, effective virtual care in general; and this guidance is not focused on mental health in particular, which has its own set of challenges including patient safety considerations that need to be specifically addressed. Various publications in the literature have already noted how poorly planned or delivered virtual health care appointments can increase the possibility of patients experiencing negative consequences. For example, one article found that patients who report to ‘walk-in’ virtual health visits (that is, episodic care settings where there is no expectation of longitudinal follow up) were more likely to visit an emergency room later [36]. Another article noted that the privacy of patient health information may be compromised if adequate protections are not put in place [28]. Some authors caution that virtual health care could worsen health inequities and disparities [12]; for example, in the United States, virtual health care services tend to be used more by people of higher income, while people of lower income use in-person visits more frequently [5].
Thus, it is necessary to develop standards for the delivery of virtual mental health services. Standards provide a benchmark for high quality care, and they provide guidance for practitioners and organizations enabling comparison with each other. According to Health Quality Ontario, “Quality standards outline for clinicians and patients what quality care looks like” [37]. Implementing standards can also ensure that quality care is delivered in a consistent manner for all patients by minimising variation [38].
Many research publications and provincial documents were quickly developed over the course of the COVID-19 pandemic to guide and support health care providers as they sought to adjust to the rapidly-changing health care landscape. Yet, despite the quickly growing popularity and usage of virtual care, there is a paucity of recommendations that emphasize mental health services in primary care, health equity and improvement of quality of care within a Canadian context. The objectives of this study were to develop and refine a list of standards on virtual delivery of mental health services in primary care in Canada using information from an earlier rapid review [33] as well as the participant feedback obtained in this study through interviews and a focus group.
Methods
Study design
We conducted a qualitative study using a modified Delphi process which included findings from a rapid review [33], consultation with people with lived experience and health workers, expert review from an external advisory group, and development of consensus among the study team. (Fig 1) The study was mostly conducted virtually; all meetings were conducted over Zoom. This manuscript has been reported using the Consensus Reporting Items for Studies in Primary Care (CRISP) checklist [39], located in S1 Table.
Ethics statement.
This project was approved by the Unity Health Toronto Research Ethics Committee (approval #22–262). Informed consent was obtained verbally prior to the focus group and interviews.
Focus group and interviews.
We used the findings of an earlier rapid review of guidance from high-income country settings [33] to develop questions for a focus group and a series of interviews with healthcare providers and people with lived experience. (S1 Text). We recruited seven participants into one focus group and two interviews (assignment based on a participant’s request; they could choose to be questioned individually in an interview or as part of a group).These lasted about one hour each, and were held between May and July, 2023 on Zoom; we used the internal features of Zoom to record each interview and to generate draft transcripts, which a research staff member reviewed and edited in detail for accuracy for use in analysis. Data were stored on a secure server at Unity Health Toronto.
Analysis of focus group and interview data.
Three team members (BO, NE, AY) used thematic analysis to code the focus group and interview transcripts [40]. This involved three phases: (1) reading of the data, (2) coding of the data, which allowed for early data management and analysis, and (3) theming, where the team members shared their perspectives, deliberated and grouped codes into themes (42). In the first phase, reading, all team members familiarised themselves with the transcripts by reading through them individually. In the second phase, coding, the team members created “codes” for each transcript, defined by Saunders et al. as “labels for concepts in the data that are directly relevant in the study objective”, and recorded these codes and their definitions. After individual coding was completed for the first transcript in Microsoft Word, BO, NE and AY met to negotiate consensus and develop these codes into a shared codebook to use for the two subsequent transcripts [13]. Lastly, in the third phase, theming, the three team members individually grouped together repeating codes and constructs into overarching themes, using Microsoft Excel. They then met together to deliberate and decide on a final set [40].
Sample quotes were identified to exemplify key themes during this phase. Following a virtual discussion, all members of the study team had the opportunity to review the themes and offer feedback on the data interpretation. Data analysis was completed using Microsoft Word and Excel.
Delphi consensus process.
Three study team members (AY, NE, BO) drafted an initial list of standards (containing 18 standards in total) using information from the rapid review (specifically the themes described in the manuscript) [33] and the results of the analysis of the focus group and interviews. We then conducted a modified Delphi consensus process by sending these to the 14 members of the advisory group [41].
In a typical Delphi process, a group of expert panel members is asked to repeatedly, independently, and anonymously rank and evaluate statements on a given topic [42, 43]. In subsequent rounds, the anonymized responses from the previous rounds are shared with panelists, to give panelists the opportunity to read other opinions and decide whether to alter their own (43). The format of the Delphi process is designed to attain a group’s consensus on a set of drafted statements while minimising bias that may originate from group conformity [42].
In this study, we implemented a modified two-step Delphi consensus process, drawing inspiration from the method used by Eubank et al [41]. Using this methodology, we first prepared a Google Forms-based survey asking for independent feedback on the agreeability, feasibility, and phrasing of the initial draft of standards. We sent this to the 14 members of our advisory group. Seven people from the advisory group responded; this group comprised researchers, physicians, health care providers and people with lived experience who did not participate in drafting the original standards. None of the authors who originally drafted the standards took part in this Delphi process.
We modified the Delphi approach by having the advisory group conduct the process to combine and condense the initial list, rather than sending out a survey more broadly for prioritization. We did this because of resource and time limitations within our one-year study period. Once feedback from this round was put together, the study team met in an online video meeting to go over the feedback and make final revisions to the list of proposed standards. Any standard that had a score of less than 100% for both feasibility and agreeability was brought to that meeting. Five members attended.
Study team. The study team was based at a research hospital based in Toronto, Ontario, Canada. The authors GS, OM, SM, HA, LL and BN were involved in the conception of the study and methodology design. The focus group and interviews were conducted by a research coordinator with experience in public health (NE) and the principal investigator (BO) who is a family physician and health services researcher. We were cognizant of power differentials particularly within interactions between us as research professionals and people with lived experience of mental illness. Although we sought to limit the effect of this on people’s responses by recruiting participants for the focus group and interviews with whom we did not have any pre-existing relationship, it remains possible that people may have been inhibited in sharing criticism of their own virtual care experiences.
External advisory group. We established an external advisory group comprising 14 primary care providers, health care and mental health services researchers and professionals and people with lived experience of mental illness. This group provided guidance over the course of the project and acted as external consultants to the authors.
Participants
We invited individuals through our personal networks and connections from various backgrounds across Canada, including people with lived experience, health care providers, policy makers, administrators, and clinicians, to participate in interviews, focus group and the advisory group (who completed the Delphi process) to discuss their views on virtual primary mental health care services and review the standards. We also conducted a Google search to identify at least one mental health organisation in every province and territory and contacted these organisations (19 in total) through email to recruit participants with experience in delivering or receiving mental health care. Participants in the focus group and interviews were provided with a $50 Canadian gift card as reimbursement for their time and participation in the study.
Results
Focus group
Participants included: one psychotherapist, four people with lived experience of mental illness, one social worker and one psychiatrist. Two participants were from Alberta, two from British Columbia and three from Ontario.
Four people with lived experience and one health care provider met in the focus group. Each of the two interviews included one health care provider. All of the participants were adults (18 years or older) who self-identified as people with lived or living experience of mental illness, mental health clinicians, policy makers or health care administrators.
Codes assigned from analysis were developed into three themes: (1) patients and providers agreeing about expectations regarding virtual care, (2) accessibility and equity, and (3) safety planning in virtual care.
Patients and providers agreeing about expectations regarding virtual care
Several topics were covered during the focus group and interviews, including modality preferences (e.g., videocall versus telephone call) and the usage of asynchronous care, primarily for communication and appointment planning. There was substantial diversity of opinion on the optimal modality for virtual care. Some participants reported preferring telephone to video, expressing that telephone calls aided them by allowing:
“…[them] to tune into the voice that much easier […without…] think[ing] about [their] own pair of verbals or body language…”. [Health care provider, Focus group 1]
Other participants expressed that maintaining communication through body language is a vital part of care, stating they:
“… read people’s body language, their tone of voice…” [Person with lived experience, Focus group 1],
during appointments. One participant noted that the modality used ultimately depends on the patient and their needs and preferences:
“But in the end, it’s also the person’s choice and there are a few people who after the initial assessment, don’t want to come back in [person] if they can avoid it.” [Health care provider, Interview 2].
Some providers expressed concerns about how much time they may be expected to invest into communicating asynchronously with their patients, noting that asynchronous care is better for quick communication but not for serious, complex conversations. One participant expressed:
“If it’s a question about something, we’ve got no problem like about a resource or something fair. But if I have to spend more than 10 minutes engaged in responding to or looking something up for you, then there will be a charge associated with that.” [Health care provider, Interview 1].
Accessibility and equity
Although participants commented on the benefits of virtual care, they also discussed gaps in the accessibility and equity of these services. For instance, one participant described how virtual appointments could improve privacy in their own life:
“… I felt like there was less shame too, because I didn’t have to tell my workplace where I was going, I could sort of schedule it on my own time. And it felt more personalized to me doing it from my own home.” [Person with lived experience, Focus group 1].
Another participant pointed out that virtual care is still inaccessible for some:
“… a lot of people […] wouldn’t necessarily have access to data in their phones …” [Person with lived experience, Focus group 1].
Another participant brought attention to the fact that virtual primary care appointments are impossible to set up without first having access to a primary care physician. They wondered:
“…about the situation where people ‘don’t have a primary care physician, or they ‘don’t have a primary care clinic necessarily. So, they ‘don’t have access, in a sense …” [Health care provider, Focus group 1].
Safety planning in virtual care
The final theme was safety planning in virtual care. Conversations on this theme described situations in which virtual care may be riskier to a patient compared to in-person appointments. For example, one participant suggested it may be easier on a patient to have certain conversations in-person rather than virtually, stating that they would advise their patients to meet in-person for follow-up if something traumatic came up during a video call:
“…, there are things that you discover, during the phone visit, that you may not be aware of at the beginning, or you may not have considered at the beginning…” [Health care provider, Interview 2].
Participants also discussed emergency situations, where a patient is experiencing a crisis at a different location from their provider. One participant described:
“…if someone’s at such a high crisis point, and they cannot be grounded, virtually, it puts you in a really kind of sticky spot, because although we may have their emergency contact information, you know, to call someone in the event of- I find not being able to kind of make that eye contact, not being able to see that they’re doing the exercise properly… anxiety provoking for the clinician as well.” [Health care provider, Focus group 1].
Participants also discussed potential strategies to minimise these risks. For example, participants believed it was important to develop and implement a plan for emergencies:
“…make the emergency plan explicit…have an explicit conversation about what’s going to happen if you’re at risk, and you’re in crisis, and what are we going to do about it?” [Health care provider, Focus group 1].
Delphi consensus process and standards development
After the focus group, interviews and rapid review (34) were completed, three study team members (NE, AY, BO) developed a draft list of 18 standards. Seven advisory group members (who had not taken part in drafting the standards initially) took part in the modified Delphi consensus process to rate the agreeability and feasibility of the 18 draft standards. The number of participants between the first and second rounds of the Delphi process dropped from seven to five. These advisory group members self-identified as four researchers, two health care administrators, two healthcare providers and one person with lived experience. (People could choose more than one role.) The initial draft list of standards and the feedback provided on them are presented in Table 1.
Out of the 18 draft standards, only four (22.2%) were unanimously rated both agreeable and feasible (Standards 2,5,6 and 8). 10 standards (55.6%) were unanimously rated agreeable (Standards 1–8, 14, 17), and four (22.2%) standards were unanimously rated feasible (Standards 2, 5, 6, 8).
The study team then made final revisions to the draft standards; these revisions included modifying phrasing of statements, deleting statements entirely and combining statements together. Five draft standards were left unmodified. Three draft standards were combined into one. Wording was revised for seven draft standards. Three draft standards were deleted. After these revisions were made, the standards were brought back to the study team, who made minor changes to their wording.
The final list of standards developed from this process are presented in Table 2.
Discussion
We developed a list of standards that can be used nationally and serve as a blueprint for delivering virtual mental health services in primary care. This process was comprehensive and drew on previous research, feedback from interviews and a focus group with community members and health care workers and input from a diverse group of panelists. Although this project was focused on Canadian primary care, we believe these standards are applicable to other single-payer healthcare settings.
Data from the interview and focus group led to the development of three themes: (1) patients and providers agreeing about expectations regarding virtual care, (2) accessibility and equity, and (3) safety planning in virtual care. These themes were used in the development of the drafted standards 4–12, 14, and 15; findings from the rapid review [33] were used to draft the other standards. There were some similarities between these standards and guidance that has been developed in other settings. For example, Ontario Health (which oversees the administration of the health system in Ontario, Canada’s most populous province) published a guidance document emphasizing the need for health equity, assessing population health needs and determining the best modalities of virtual care for patients [31]. Other literature exploring how to improve delivery of virtual mental health care in Canada also described the importance of considering the impact of health care disparities, including those experienced by populations experiencing inequitable access to quality care, during guideline development [44]. This is represented in standard 11 (“Patients from equity-seeking groups should have opportunities for prioritized access to virtual mental health services, if that is their preference and it is feasible”) (Table 2).
As part of this overall research project, we reviewed Canadian provincial and territorial policies on delivering virtual health care [45] and found some other similarities and differences to the standards we created. For example, standards of practice published online by the College of Physicians and Surgeons of Alberta stated those providing virtual care must assess the suitability of their patient’s physical environment, obtain consent, develop plans for adverse or emergency situations and confirm the location of their patient prior to initiating each virtual appointments [46], consistent with standards 4 and 5 in our study (Table 2). The College of Physicians and Surgeons of Saskatchewan provided similar guidance through their virtual care policy, and reported that physicians providing virtual care should, among other things, strive to ensure confidentiality and verify the appropriateness of virtual care with each patient before commencing a virtual appointment [47].
Some of the standards were developed from the policy and literature reviewed in our earlier work [33, 45], instead of originating from one of the interviews or focus group. In this project, our team performed a rapid review on virtual health care guidance published in the literature, through the lens of the Quadruple Aim Framework. The Quadruple Aim framework guided this search through its focus on four key principles when designing and delivering health care services: improving patient experience, improving population health, reducing health care costs and improving provider experience [48]. For example, Standard 1, which calls for health care providers to perform standardized symptom assessments for their patients, originated from the research found through the rapid review [33] as well as policy mandating that physicians continue to monitor their patients’ symptoms despite being based virtually [49, 50]. This type of assessment is based on the concept of measurement-based care. Measurement-based care can be described as “…routinely administered outcome measures with practitioner and patient review to inform clinical decision-making” [51]. It has been reported to be viewed favourably among Canadian clinicians delivering psychiatric care, so long as it can be implemented through methods considering ease-of-use and feasibility [52]. Although measurement-based care has been found to be effective to improve symptoms of anxiety and depression [51], it also provides a method to collect and retain a longitudinal, comprehensive record of patients’ health and symptoms in the absence of in-person tests–a useful tool to use when providing health care virtually.
However, there were also key areas in which our findings differed from other work in this area. These differences are valuable as they inform further research. For example, one report described the importance of coordinating virtual care services from local and regional perspectives to ensure that the unique circumstances and needs of different regions are taken into account [44]. Surprisingly, this did not come up during the focus group, interview or team meetings. Nevertheless, this is important to meet the needs of patients and health care providers across Canada. Another report encouraged providers to self-assess their capacity and aptitude to deliver virtual primary care services [31]. Capacity and aptitude can be built through workplace training initiatives around virtual health care. This is an important aspect of virtual health care delivery, as providers must have the tools and learning they require to successfully conduct virtual mental health appointments.
Another area for further research may explore how to ensure that episodic appointments (e.g., ‘walk-in’ services) ensure patient continuity of care. This is an important step for improving virtual care services, especially given that reports have critiqued walk-in virtual primary care services for putting patients in more danger of experiencing harm [5, 28]. For example, physicians delivering care virtually may be unable to link patients to critical community resources because the physician is not that patient’s regular healthcare practitioner and may be unaware of the patient’s background or neighbourhood [5].
Strengths and limitations
The overarching goal of this project was the development of virtual primary mental health care standards based on the opinions of health care practitioners, people with lived experience and other community stakeholders in Canada. We created a list of standards that reflected a wide range of opinions. Consensus-driven approaches, like the one we employed, have also been used frequently across different studies discussing standard development for primary mental health care, particularly among younger populations [53–56]. A second major strength of this project was the emphasis on assessing the feasibility and agreeability of each standard with the advisory group. This allowed for standards to be debated, revised and changed to better meet the needs of patients in Canada.
Limitations included the fact that the focus group and interviews recruited a small number of participants (n = 7) and the Delphi process also only involved seven external advisory group members (there was no overlap between those two groups), which limited the range of discussions and opinions. Although the opinions and feedback expressed within this manuscript represent diversity in terms of roles within primary mental health care, respondents from non-urban environments were unrepresented. Additionally, in our original plan, we had intended to execute a formal Delphi panel using panelists that were entirely external to our team. However, due to time and resource constraints, we instead used the same anonymous Delphi process with members of the advisory group. We feel that this was an appropriate use of the process, because the members of our team were located in different regions (e.g. provinces, countries, etc.) and worked in somewhat different fields (e.g. academia, government, etc.), allowing us to maximise feedback received from this group while minimising group think. Moreover, to minimise the bias that would have come from authors reviewing standards that they wrote, none of the authors who took part in drafting the initial standards participated in any of the rounds of the Delphi consensus. Additionally, because of the afore-mentioned constraints, we were only able to implement one formal round of critiques assessing the feasibility and acceptability through the Delphi process, instead of mirroring the two rounds of critiques completed by Eubank et al. [41].
Conclusions
The standards developed in this project can be used by primary care providers, health care organizations, and policymakers as a benchmark for what constitutes high quality virtual delivery of mental health services in Canadian primary care. Given that mental health issues are the most common reason for presentation to primary care providers, reducing unwarranted variation in care for these important issues is an important aim for health care in Canada. Future work should explore implementation of these standards into diverse primary care settings.
Supporting information
S1 Table. Consensus reporting items for studies in primary care- the CRISP statement.
https://doi.org/10.1371/journal.pmen.0000071.s001
(DOCX)
Acknowledgments
We would like to thank study participants for their valuable contributions. We also thank the entire ‘Standards for Virtual Delivery of Mental Health Services in Canadian Primary Care’ study team and advisory group for their involvement in this project. We are very thankful to Ngozi Ekeleme and Karen Waite for their contributions. We are grateful to the Canadian Institutes of Health Research for their funding of this work. Braden O’Neill acknowledges the support of Clinician-Scientist awards from the Departments of Family and Community Medicine at St. Michael’s Hospital and the University of Toronto, and the Li Ka Shing Knowledge Institute, St. Michael’s Hospital.
References
- 1. Neves AL, Li E, Gupta PP, Fontana G, Darzi A. Virtual primary care in high-income countries during the COVID-19 pandemic: Policy responses and lessons for the future. Eur J Gen Pract. 2021 Dec;27(1):241–7. pmid:34431426
- 2. Shaw J, Jamieson T, Agarwal P, Griffin B, Wong I, Bhatia RS. Virtual care policy recommendations for patient-centred primary care: findings of a consensus policy dialogue using a nominal group technique. J Telemed Telecare. 2018 Oct;24(9):608–15. pmid:28945161
- 3. Glazier RH, Green ME, Wu FC, Frymire E, Kopp A, Kiran T. Shifts in office and virtual primary care during the early COVID-19 pandemic in Ontario, Canada. CMAJ. 2021 Feb 8;193(6):E200–10. pmid:33558406
- 4. CIHI. Use of physician mental health services in Canada. 2023.
- 5. Eastburn J, Harris A, Nagarajan N, Rost J. McKinsey & Company. 2023. Is virtual care delivering on its promise of improving access? Available from: https://www.mckinsey.com/industries/healthcare/our-insights/is-virtual-care-delivering-on-its-promise-of-improving-access
- 6. Canadian Mental Health Association. Fast Facts about Mental Illness. https://cmha.ca/brochure/fast-facts-about-mental-illness/; 2021.
- 7. Moroz N, Moroz I, D’Angelo MS. Mental health services in Canada: Barriers and cost-effective solutions to increase access. Healthc Manage Forum. 2020 Jul 2;33(6):282–7. pmid:32613867
- 8. Stephenson E, Butt DA, Gronsbell J, Ji C, O’Neill B, Crampton N, et al. Changes in the top 25 reasons for primary care visits during the COVID-19 pandemic in a high-COVID region of Canada. PLoS One. 2021;16(8):e0255992. pmid:34383844
- 9. Stephenson E, O’Neill B, Gronsbell J, Butt DA, Crampton N, Ji C, et al. Changes in family medicine visits across sociodemographic groups after the onset of the COVID-19 pandemic in Ontario: a retrospective cohort study. CMAJ Open. 20210615th ed. 2021;9(2):E651–8. pmid:34131028
- 10. Flood CM, Thomas B, McGibbon E. Canada’s primary care crisis: Federal government response. Healthc Manage Forum. 2023 Sep;36(5):327–32. pmid:37424188
- 11.
Canadian Institute for Health Information [Internet]. 2023. Virtual care in Canada. Available from: https://www.cihi.ca/en/virtual-care-in-canada
- 12. Affleck E, Bourdouleix S, MacLaren G, Adams O, Meyer C, Larsen D, et al. Enhancing Equitable Access to Virtual Care in Canada: Principle-based Recommendations for Equity [Internet]. 2021. Available from: https://www.canada.ca/content/dam/hc-sc/documents/corporate/transparency/health-agreements/bilateral-agreement-pan-canadian-virtual-care-priorities-covid-19/enhancing-access-principle-based-recommendations-equity/based-recommendations-equity-en.pdf
- 13. Scott J, Yellowlees P, Becker DF, Chen C. Virtual Care and Mental Health: Dismantling Silos to Strengthen Care Delivery. Telemed Rep. 2023;4(1):174–9. pmid:37484480
- 14. Barnett JE, Kolmes K. The practice of tele-mental health: Ethical, legal, and clinical issues for practitioners. Practice Innovations. 2016;1(1):53.
- 15. Shore JH. Best practices in tele-teaming: Managing virtual teams in the delivery of care in telepsychiatry. Curr Psychiatry Rep. 2019;21:1–8. pmid:30637488
- 16. Cordina J, Fowkes J, Malani R, Medford-Davis L. McKinsey & Company. 2022 [cited 2022 Jul 17]. Patients love telehealth—physicians are not so sure | McKinsey. Available from: https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/patients-love-telehealth-physicians-are-not-so-sure
- 17. Hilty DM, Serhal E, Crawford A. A Telehealth and Telepsychiatry Economic Cost Analysis Framework: Scoping Review. Telemed J E Health. 2023 Jan;29(1):23–37. pmid:35639444
- 18. Summer G, Adelman DS, Fant C. COVID-19 and telehealth: How to complete a successful telehealth visit. Nurse Pract. 2021 Jun;46(6):43–7. pmid:34004641
- 19. Miu AS, Vo HT, Palka JM, Glowacki CR, Robinson RJ. Teletherapy with serious mental illness populations during COVID-19: telehealth conversion and engagement. Couns Psychol Q [Internet]. 2021 Oct 2;34(3–4):704–21. Available from: https://doi.org/10.1080/09515070.2020.1791800
- 20. Sachs JW, Graven P, Gold JA, Kassakian SZ. Disparities in telephone and video telehealth engagement during the COVID-19 pandemic. JAMIA Open [Internet]. 2021 Jul 1;4(3):ooab056. Available from: pmid:34632322
- 21. McCord C, Bernhard P, Walsh M, Rosner C, Console K. A consolidated model for telepsychology practice. J Clin Psychol. 2020 Jun;76(6):1060–82. pmid:32285940
- 22. Guidelines for the practice of telepsychology. Am Psychol. 2013 Dec;68(9):791–800. pmid:24341643
- 23. Drum KB, Littleton HL. Therapeutic boundaries in telepsychology: Unique issues and best practice recommendations. Prof Psychol Res Pr. 2014 Oct;45(5):309–15. pmid:25414540
- 24.
National Institute of Mental Health. What is Telemental Health? [Internet]. Available from: https://www.nimh.nih.gov/health/publications/what-is-telemental-health
- 25. Johnson GR. Toward uniform competency standards in telepsychology: A proposed framework for Canadian psychologists. Canadian Psychology [Internet]. 2014;55:291–302. Available from: https://api.semanticscholar.org/CorpusID:146423117
- 26. Sabin JE, Skimming K. A framework of ethics for telepsychiatry practice. Int Rev Psychiatry. 2015;27(6):490–5. pmid:26493214
- 27. Budhwani S, Fujioka JK, Chu C, Baranek H, Pus L, Wasserman L, et al. Delivering Mental Health Care Virtually During the COVID-19 Pandemic: Qualitative Evaluation of Provider Experiences in a Scaled Context. JMIR Form Res. 2021 Sep;5(9):e30280. pmid:34406967
- 28. Hardcastle L, Ogbogu U. Virtual care: Enhancing access or harming care? Healthc Manage Forum [Internet]. 2020 Jul 20;33(6):288–92. Available from: pmid:32686506
- 29. Wong A, Bhyat R, Srivastava S, Boissé Lomax L, Appireddy R. Patient Care During the COVID-19 Pandemic: Use of Virtual Care. J Med Internet Res. 2021 Jan;23(1):e20621. pmid:33326410
- 30. Palesy D, Forrest G, Crowley ME. Education guidelines, frameworks and resources for building virtual care capacity: An integrative review. J Telemed Telecare. 2023 Apr;29(3):222–43. pmid:36628539
- 31. Ontario Health. Clinically Appropriate Use of Virtual Care in Primary Care: Target Audience, Concept and Statements. 2022.
- 32. Dorn SD. Backslide or forward progress? Virtual care at US healthcare systems beyond the COVID-19 pandemic. NPJ Digit Med. 2021;4(1):1–3. pmid:33398041
- 33. Ekeleme N, Yusuf A, Kastner M, Waite K, Montesanti S, Atherton H, et al. Guidelines and recommendations about virtual mental health services from high-income countries: a rapid review. BMJ Open [Internet]. 2024 Feb 1;14(2):e079244. Available from: http://bmjopen.bmj.com/content/14/2/e079244.abstract pmid:38388504
- 34.
Canadian Medical Association Virtual Care Playbook. 2021. https://digitallibrary.cma.ca/link/digitallibrary52 [Accessed 29 May 2024]
- 35. Framework on Virtual Care. Foundation of Medical Regulatory Colleges of Canada. 2021. https://fmrac.ca/wp-content/uploads/2022/07/FMRAC-Framework-on-Virtual-Care.pdf [Accessed 29 May 2024]
- 36. Lapointe-Shaw L, Salahub C, Bird C, Bhatia RS, Desveaux L, Glazier RH, et al. Characteristics and Health Care Use of Patients Attending Virtual Walk-in Clinics in Ontario, Canada: Cross-sectional Analysis. J Med Internet Res. 2023 Jan;25:e40267. pmid:36633894
- 37. Health Quality Ontario. Evidence to Improve Care [Internet]. Available from: https://www.hqontario.ca/Evidence-to-Improve-Care/Quality-Standards [Accessed 29 May 2024]
- 38. Dhalla IA, Tepper J. Improving the quality of health care in Canada. Can Med Assoc J. 2018 Oct 1;190(39):E1162 LP–E1167. pmid:30274993
- 39. Phillips WR, Sturgiss E, Glasziou P, Hartman TC olde, Orkin AM, Prathivadi P, et al. Improving the Reporting of Primary Care Research: Consensus Reporting Items for Studies in Primary Care—the CRISP Statement. The Annals of Family Medicine [Internet]. 2023 Oct 3;3029. Available from: http://www.annfammed.org/content/early/2023/09/27/afm.3029.abstract pmid:37788942
- 40. Saunders CH, Sierpe A, Plessen C von, Kennedy AM, Leviton LC, Bernstein SL, et al. Practical thematic analysis: a guide for multidisciplinary health services research teams engaging in qualitative analysis. BMJ. 2023 Jun 8;381:e074256. pmid:37290778
- 41. Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol [Internet]. 2016;16(1):56. Available from: pmid:27206853
- 42. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021 Jul;11(4):116–29. pmid:34322364
- 43. Barrett D, Heale R. What are Delphi studies? Evid Based Nurs. 2020;23(3):68–9. pmid:32430290
- 44. Lo B, Strudwick G, Mah L, Mushquash CJ, McKenzie K, Khenti A, et al. Effective Modalities of Virtual Care to Deliver Mental Health and Addictions Services in Canada. 2022 Sep 19;
- 45. Ekeleme N, Yusuf A, O’Neill B. Canadian Provincial and Territorial Medical Regulatory College Guidance for Delivery of Virtual Mental Health Services in Primary Care: Policy Overview [Internet]. 2023 [cited 2024 Feb 15]. Available from: https://pcmhstandards.ca/policy-overview/
- 46. College of Physicians and Surgeons of Alberta. Virtual Care | Standard of Practice [Internet]. 2014. Available from: https://cpsa.ca/physicians/standards-of-practice/virtual-care/
- 47. College of Physicians and Surgeons of Saskatchewan. Practical Information for Virtual Visits. 2020.
- 48. Bodenheimer T, Sinsky C. From triple to Quadruple Aim: Care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6. pmid:25384822
- 49.
College of Physicians and Surgeons of Alberta. Virtual Care | Standard of Practice [Internet]. 2022 [cited 2024 Feb 15]. Available from: https://cpsa.ca/wp-content/uploads/2021/12/Virtual-Care.pdf
- 50.
College of Physicians and Surgeons of Saskatchewan. Policy-Virtual Care [Internet]. 2023 [cited 2024 Feb 15]. Available from: https://www.cps.sk.ca/imis/CPSS/CPSS/Legislation__ByLaws__Policies_and_Guidelines/Legislation_Content/Policies_and_Guidelines_Content/Virtual_Care.aspx
- 51. Zhu M, Hong RH, Yang T, Yang X, Wang X, Liu J, et al. The efficacy of measurement-based care for depressive disorders: systematic review and meta-analysis of randomized controlled trials. J Clin Psychiatry. 2021;82(5):37090. pmid:34587377
- 52. Cheung BS, Murphy JK, Michalak EE, Liu J, Yang X, Wang X, et al. Barriers and facilitators to technology-enhanced measurement based care for depression among Canadian clinicians and patients: Results of an online survey. J Affect Disord. 2023 Jan;320:1–6. pmid:36162664
- 53. Sayal K, Amarasinghe M, Robotham S, Coope C, Ashworth M, Day C, et al. Quality standards for child and adolescent mental health in primary care. BMC Fam Pract. 2012;13(1):51. pmid:22672193
- 54. Gill PJ, O’Neill B, Rose P, Mant D, Harnden A. Primary care quality indicators for children: measuring quality in UK general practice. British Journal of General Practice. 2014;64(629):e752–7. pmid:25452539
- 55. Graham T, Rose D, Murray J, Ashworth M, Tylee A. User-generated quality standards for youth mental health in primary care: a participatory research design using mixed methods. BMJ Qual Safety. 2014 Oct 1;23(10):857 LP– 866. pmid:24920648
- 56. Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002 Dec 1;11(4):358 LP– 364. pmid:12468698