The COVID-19 pandemic has had a devastating impact on healthcare systems and care delivery, changing the context for patient and family engagement activities. Given the critical contribution of such activities in achieving health system quality goals, we undertook to address the question: What is known about work that has been done on patient engagement activities during the pandemic?
To examine peer-reviewed and grey literature to identify the range of patient engagement activities, broadly defined (inclusive of engagement to support clinical care to partnerships in decision-making), occurring within health systems internationally during the first six months of the COVID-19 pandemic, as well as key barriers and facilitators for sustaining patient engagement activities during the pandemic.
The following databases were searched: Medline, Embase and LitCOVID; a search for grey literature focused on the websites of professional organizations. Articles were required to be specific to COVID-19, describe patient engagement activities, involve a healthcare organization and be published from March 2020 to September 2020. Data were extracted and managed using Microsoft Excel. A content analysis of findings was conducted.
Twenty-nine articles were included. Few examples of more genuine partnership with patients were identified (such as co-design and organizational level decision making); most activities related to clinical level interactions (e.g. virtual consultations, remote appointments, family visits using technology and community outreach). Technology was leveraged in almost all reported studies to interact or connect with patients and families. Five main descriptive categories were identified: (1) Engagement through Virtual Care; (2) Engagement through Other Technology; (3) Engagement for Service Improvements/ Recommendations; (4) Factors Impacting Patient Engagement; and (5) Lessons Learned though Patient Engagement.
Evidence of how healthcare systems and organizations stayed connected to patients and families during the pandemic was identified; the majority of activities involved direct care consultations via technology. Since this review was conducted over the first six months of the pandemic, more work is needed to unpack the spectrum of patient engagement activities, including how they may evolve over time and to explore the barriers and facilitators for sustaining activities during major disruptions like pandemics.
Citation: Cadel L, Marcinow M, Sandercock J, Dowedoff P, Guilcher SJT, Maybee A, et al. (2021) A scoping review of patient engagement activities during COVID-19: More consultation, less partnership. PLoS ONE 16(9): e0257880. https://doi.org/10.1371/journal.pone.0257880
Editor: Mariusz Duplaga, Uniwersytet Jagiellonski w Krakowie Biblioteka Jagiellonska, POLAND
Received: December 7, 2020; Accepted: September 14, 2021; Published: September 29, 2021
Copyright: © 2021 Cadel et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: This paper was funded by the Canadian Foundation for Health Care Improvement. Dr. Kerry Kuluski holds the Dr. Mathias Gysler Research Chair in Patient and Family Centred Care at the Institute for Better Health, Trillium Health Partners. The funding for this Chair, supported by the Trillium Health Partners Foundation, was used to support Dr. Kuluski’s time in leading the study reported in this paper.
Competing interests: The authors have declared that no competing interests exist.
The World Health Organization declared the COVID-19 outbreak, originating in Wuhan, China, as a pandemic on March 11, 2020 . As of June 2021, over 174 million cases had been confirmed worldwide, with more than 3.6 million deaths . Two hundred and nineteen countries/regions have been affected by this pandemic . COVID-19 has not only affected healthcare systems, it has had a major impact on everyday life involving local and national economies , education systems , travel industries , immigration  and the environment . In relation to healthcare systems, continuing to deliver high quality care amidst a pandemic has been challenging.
How healthcare systems interacted with patients and caregivers (forms of patient engagement) has changed over the course of the COVID-19 pandemic. The pandemic caused a major shift in health system activities with implications on how care was delivered and how patients and families were engaged as partners. For example, visitor restrictions were implemented in hospitals and long-term care facilities around the world [8–10], non-COVID-19 related research in some contexts was suspended  and many patient and family advisory committees, at least in the North American context, were put on hold . An editorial by Richards and Scowcroft (2020) from the United Kingdom also described the abandonment of patient and public involvement in health system decisions and the omission of patient feedback on policies and guidelines .
Patient and family engagement in care and program planning is fundamental to person-centered care and quality of care [14, 15]. Over the past several decades, health systems have been working towards greater involvement of patients and families through engagement activities, such as the development of patient and caregiver advisory committees, consultations on guidelines or protocols, receiving feedback on experiences to promote change in care delivery and the healthcare system and working in partnership using co-design methods to design new programs and service delivery models [16–19]. Through patient engagement, care can be transformed by individuals who are directly impacted by healthcare decisions . While these examples refer to more ‘active’ forms of engagement, activities can be more broad in nature, ranging—from clinical consultations to partnership activities (advisory groups and participation in policy decision making) . Engagement, in these various forms, is intended to optimize care to meet the health and social care needs of patients and caregivers [22, 23].
Patient engagement can be defined as the partnership between patients, families and health professionals across different levels of the healthcare system with the goal of improving health and healthcare . Reflecting on the broad range of engagement activities, Carman and colleagues’ (2013) well cited, multidimensional framework for patient and family engagement outlines three core levels of engagement: 1) direct care, 2) organizational design and 3) governance and policy making. At each level, there is a continuum of engagement from consultation, to involvement, to partnership and shared leadership  reflecting both active and passive forms of engagement. Each stage of the continuum involves increased participation and collaboration from those being engaged, as they progress from information sharing to active partnership in the development and evaluation of healthcare programs and policies.
Patient engagement has been noted as an integral component of health care delivery, and is critical for obtaining feedback for improving processes, safety and experiences . The pandemic halted many non-essential services; however, continuing to engage with patients and families during a pandemic is important to ensure quality of care is continually provided to those who are using and interacting with the healthcare system [14, 15]. With a rapidly changing environment as a result of COVID-19, understanding the initial impact of the pandemic on patient engagement activities requires our attention. The purpose of this scoping review was to examine peer reviewed and grey literature to identify the broad range of patient engagement activities that were happening within health systems during the first six months of the COVID-19 pandemic (March 2020 to September 2020), as well as key barriers and facilitators for patient engagement activities. We acknowledge that a scoping review is only one way to identify the type and extent of evidence available relating to patient engagement activities during the initial stages of COVID-19. Given that COVID-19 related papers were being published rapidly, our aim was to scope and synthesize the literature and to highlight knowledge gaps to help focus areas of improvement and future work .
Materials and methods
This scoping review followed the methodological framework for scoping reviews outlined by Levac and colleagues , as well as the PRISMA-ScR reporting guidelines by Tricco and colleagues (see S2 Table) . A protocol was developed and registered on Open Science Framework (https://osf.io/khuap).
Identifying the research question
The following research question was formulated to guide this scoping review: What is known in the literature about work that has been done internationally within healthcare organizations on patient engagement during the first six months of the COVID-19 pandemic? Based on this question, the two main objectives were: (1) to identify strategies that have been implemented or sustained to engage with patients during the COVID-19 pandemic and (2) to identify the key barriers and facilitators to engaging with patients during the COVID-19 pandemic.
Identifying relevant studies
A search strategy was developed with guidance from the Medical Library Association’s resource on COVID-19 Literature Searches , as well as in consultation with a librarian at Trillium Health Partners (see S1 Table). The following are examples of terms searched using Boolean operators, truncations, wild cards and proximity operators: coronavirus, COVID-19, Sars-CoV-2, patient, family and caregiver engagement, participation, design and partner. Three databases were searched on May 25, 2020: Medline (Ovid Interface), Embase (Ovid Interface) and LitCOVID . A search for relevant grey literature was conducted using Google (Advanced Google searches), and the websites of key health system and guidance producing organizations and associations (e.g. World Health Organization, United Kingdom National Health Service). Due to the high volume of publications related to COVID-19, grey literature was searched regularly throughout this study until September 1, 2020 (when analysis and manuscript writing began).
For inclusion, articles were required to meet the following criteria: (1) specific to the context of COVID-19; (2) included a description of patient engagement activities; (3) involved a healthcare organization; and (4) published from March 11, 2020 to September 1, 2020. To be considered a patient engagement activity, we followed Carman’s framework, which includes engaging with patients, families, their representatives, and health professionals in direct care (e.g. participating in clinical consultations, participating in support groups), organizational design and governance (e.g. serving on advisory councils, assisting with hiring committees) and/or policy making (e.g. collaborating with policy-makers, setting priorities for healthcare resources) . The patient engagement activity did not have to be specific to COVID-19, just occurring during the pandemic. Based on the rapidly changing environment, this review was bounded within the first six months after the pandemic was declared in order to gain an understanding of the initial impact of COVID-19 on patient engagement activities. Articles were excluded if they were: (1) conference abstracts and articles in which the full-text was not available or accessible; or (2) articles published prior to 2019. Articles were excluded if the full-text was not accessible because information would not be able to be extracted from it. Articles published prior to 2019 were excluded because they would not be related to the COVID-19 pandemic.
The database searches identified 762 articles and the grey literature searches identified 11 articles (reports, commentaries, etc.; see Fig 1). Following deduplication using EndNote X9 , 473 articles remained for screening. Covidence, a software platform for systematic and scoping reviews , was used to facilitate the screening processes. The research team (KK, LC, MM) independently screened the titles and abstracts of 20 articles to test their agreement; with an agreement of 90%, the remaining articles were divided amongst the three reviewers and double screened (two individuals screened each article for inclusion). All disagreements were discussed in virtual meetings until consensus among the team was achieved. Articles that were included were then reviewed at the full-text level. A subset of full-text articles (n = 10) were independently screened by the research team (KK, LC, MM) to test their agreement (90%). All full-text articles were double screened and disagreements were resolved through virtual meetings.
Charting the data
A data extraction form was developed in Microsoft Excel by the research team (KK, LC, MM). Data were extracted from the full-text articles by a single person (LC) to ensure consistency. A spot check of extracted data was performed (MM) for 10% of included articles to ensure the extracted data was complete and accurate. Extracted data contained the following information: study characteristics, population characteristics, patient engagement characteristics (description, target population, challenges, lessons learned, level of engagement ), intervention characteristics (if applicable), study outcomes and conclusions.
Collating, summarizing and reporting the results
The extracted data were analyzed using descriptive quantitative and qualitative approaches. Descriptive quantitative analyses included summarizing the number of articles based on the method of data collection, year of publication, country in which it was conducted and the type of patent engagement activity. Carman and colleagues’ framework was initially used to categorize each of the activities based on their level of engagement ; however, in doing so, it was identified that the majority of activities were categorized as direct care. A more in-depth analysis, guided by Hsieh and Shannon’s conventional approach to content analysis , was then conducted by two authors (KK, LC) to identify the nuances and details of each activity. Both authors read the extracted descriptions and purpose of the engagement activities, referring back to the full-text as needed, and applied a one or two word description to the activity. The authors then worked together (through virtual meetings and in-depth conversations) to review the simplified descriptions of the engagement activities and identified broader commonalities, which allowed for all articles to be classified into five categories (described in detail in the Results). This analysis provided a more nuanced set of categories within Carmen’s categories of engagement.
A total of 473 unique articles were screened at the title and abstract level and 83 were screened at the full-text level. Twenty-nine articles met the eligibility criteria and were included in this scoping review (see Fig 1). The majority of included articles were not original research articles (e.g. editorials, reviews, commentaries; n = 15). The most common study design of the empirical articles was quantitative (n = 12), with one mixed method study and one qualitative study. The work was carried out across 14 different countries, but the United States was the most common (n = 17). All of the included articles were published in 2020, with the majority published in May (n = 15). Table 1 outlines the characteristics of included articles.
During the analysis, five main descriptive categories were identified related to the literature on engagement: (1) Engagement through Virtual Care; (2) Engagement through Other Technology; (3) Engagement for Service Improvements/ Recommendations; (4) Factors Impacting Patient Engagement; and (5) Lessons Learned through Patient Engagement (see Fig 2). These categories are not mutually exclusive as some activities fell into more than one category. Across all categories, the majority of activities involved individual level patient engagement (i.e. information sharing and dissemination, feedback on attitudes and experiences), with limited activities empowering patients as leaders, integrating patient and caregiver partners as team members or co-designing interventions, programs or initiatives with patients and caregivers. Despite not limiting the topic of patient engagement activities to COVID-19, we identified that most of the articles included activities specific to the pandemic. Table 2 describes the patient engagement characteristics.
Engagement through virtual care
The primary type of patient engagement was through virtual care, with two sub-categories of engagement activities: (a) virtual care for health and/or social services (n = 13) [35, 37, 41, 43, 45, 46, 48, 51–54, 56, 58] and (b) virtual care for connecting with families and caregivers, including their involvement in care team discussions (n = 3) [35, 48, 49].
Virtual care for health and social services.
Virtual care was primarily used for clinical services, inclusive of both health and social care (e.g. consultations, care conferences, home monitoring, remote appointments/check-ups) [35, 37, 41, 43, 45, 46, 48, 51–54, 56, 58]. Virtual visits and consultations extended beyond medical care to include social services [35, 37, 43, 48]. For example, healthcare providers and care coordinators at a HIV clinic in the United States used telemedicine to consult with patients, check on their medication supply, help set up technology (answer questions) and ensure food and housing security during COVID-19 .
In light of the COVID-19 pandemic, many healthcare organizations and institutions transitioned from in-person appointments to online platforms (i.e. telemedicine/ telehealth). This transition and increased use of telemedicine to encourage virtual interactions between healthcare providers and patients was described in several included articles [41, 46, 56, 58]. For example, from March 2020 to April 2020, telemedicine visits (telephone and video) at one institution increased from fewer than 100 visits per day to greater than 2,200 visits per day . A number of platforms were used for providing virtual clinical services, with one article describing appropriate platforms on which to engage with patients (e.g. Skype for Business/ Microsoft Teams, Updox, Zoom for Healthcare, Doxy.me, Google G Suite Hangouts Meet, Cisco Webex Meetings/Webex Teams, Amazon Chime, GoToMeeting, Spruce Health Care Messenger) . Further, authors reflected on the potential benefits of telemedicine for patients, including: the ability to reach more patients (i.e. rural, international, low-socioeconomic status) and improved access (eliminating parking, public transit, etc.) [37, 51, 56, 58].
Feedback on virtual care services was also obtained from patients and healthcare providers (examples combining virtual care for clinical services and technology for gathering information [41, 54]. For example, a quality improvement initiative received feedback from patients (n = 119) and physicians (n = 14) on the rapid implementation of telemedicine (via InTouch or Zoom) on a physical medicine and rehabilitation department in the United States . Patients were engaged through telemedicine and follow-up surveys to identify their demographic characteristics (age, gender, insurance status, travel time saved), telemedicine details and characteristics (accompanied by family/friend, type, reason and time of visit) overall experiences (developing a treatment plan, communication, convenience, discussing concerns/questions, satisfaction) and future value. The majority of patients (>90%) rated their telemedicine experience as good or very good across the experience measures and most believed it would be valuable to complete a future telemedicine visit.
Virtual care for connecting with families and caregivers.
Virtual care was also used to connect healthcare providers or patients with family members and caregivers during healthcare interactions [35, 48, 49]. WhatsApp, Zoom, FaceTime and MyChart (online tool combining Epic and Zoom) were the main platforms for connecting. Despite visitor restrictions, these platforms gave family members the opportunity to be involved in clinical rounds [35, 49] and remain socially connected with patients while they were in hospital . One study explored family members’ of palliative patients (n = 16) thoughts and experiences with the use of WhatsApp for sharing information on the patients’ progress . Overall, family members were happy to virtually attend clinical rounds, receive and exchange information during the call and see their loved one. However, participants also noted that the virtual connection did not replace their physical presence. Technology-based solutions allowed for improved communication and connection between healthcare providers and patients or family members, but the importance of providing clear communication about COVID-19 and self-isolation, addressing patients concerns’ and demonstrating compassion throughout the pandemic were also highlighted .
Engagement through other technology
Patient engagement activities through the use of other technology (e.g. social media, online-based interventions) were identified as a way of sharing information (n = 4) [36, 42, 59, 60], creating connections (n = 4) [33, 44, 55, 59] and gathering information (n = 6) [34, 38, 39, 41, 54, 57].
Technology for sharing information.
The use of technology for engaging patients and the community through information sharing was identified in four articles [36, 42, 59, 60]. Information was shared through online platforms, including social media (e.g. Twitter, Sina Weibo). Online-based channels were used in China to share information about the pandemic, improve risk communication and community engagement practices, and respond to the publics’ concerns . The importance of sharing information through the communities’ trusted sources, combatting rumors early by disclosing up-to-date information and ensuring information was accessible and comprehensible to the public was noted by the authors. Twitter was also used as a communication tool for healthcare providers and the general public to share information about infectious diseases . Twitter chats and hashtags were discussed as valuable tools for both engaging with, and disseminating information to, other professionals and the public. During COVID-19, the authors explained the importance of Twitter for remaining up-to-date on the state of the pandemic and relevant literature being published, while ensuring credible resources were followed (@WHO, @CDCgov, @IDSAInfo). A combination of social media platforms (e.g. Facebook, WhatsApp) was also used by a volunteer campaign in Syria, Volunteers Against Corona, to effectively communicate, stay connected and share information about COVID-19 .
Technology for creating connections.
Patient engagement activities using technology (digital health solutions, email, teleconferencing) for creating or maintaining social connections were identified in four articles [33, 44, 55, 59]. The importance of maintaining person-centred care, which included promoting communication between patients, families and healthcare providers and limiting relationship and social restrictions during the pandemic was discussed in two articles [44, 55]. Residents at a rural healthcare facility in Australia remained socially engaged with their family and friends through the use of email and social media .
Another paper described a future mixed methods case study which aims to develop an app for older adults, their families and peers to enhance their overall health and well-being during social distancing measures due to COVID-19 . This digital health solution, Activating Digital to Support Social Distancing COVID-19 Aware Family Engagement (ADAPT-CAFÉ), will allow families and peers to remain in contact with older adults (in the community or in hospital) through virtual interactions. The app will also integrate goal setting, promote good nutrition and physical activity and track symptoms. During beta phase testing of the app, qualitative interviews will be conducted with participants and their feedback will be integrated in future iterations of the app, reflecting an example of co-design.
Technology for gathering information.
Patient engagement activities using technology for gathering information (experiences, satisfaction, knowledge, attitudes, concerns, health outcomes) were identified in six articles [34, 38, 39, 41, 54, 57]. These activities included survey development and/or completion, self-screening and a self-help intervention.
The use of surveys to gain information and insights about participants’ knowledge, attitudes, concerns, health outcomes and overall experiences (with their condition, care or engagement in activities) was identified in a number of articles [34, 38, 41, 54, 57]. These articles involved patients as partners in research activities (responding to surveys and participating in randomized controlled trials). There were few examples of activities that involved patients in multiple aspects of a research project; however an example of greater involvement was a global patient experience survey for individuals with rheumatic disease that was launched to identify patients’ concerns with the disease and treatment and the impact of COVID-19 on their physical and mental health . This survey was developed through rapid engagement of patients with rheumatic disease, patient organizations and rheumatologists. Patients were involved in all stages of the research, from study design to dissemination of the survey (i.e. development of research questions, study design, measuring instruments; participating in recruitment and the steering committee; and writing of the manuscript). Their main responsibilities included providing input on the content of the survey questions, reviewing the survey questions for culturally appropriate language and translating the survey into different languages. Involvement was facilitated through the use of an online messaging and collaboration tool. Sirotich and colleagues described a process for, and the benefits of, the rapid engagement of multiple stakeholders in order to address a complex problem.
Engagement for service improvements/recommendations
Few articles (n = 4) involved patients as partners as part of the development of guidelines or service improvements [40, 47, 50, 61]. Two of the included articles conducted surveys to inform the design of services [47, 50]. A cross-sectional study was conducted by Yassa and colleagues (2020) in Turkey to understand pregnant women’s knowledge, attitudes and concerns towards COVID-19, with the goal of developing targeted counselling and messages during the pandemic . Based on the findings from the survey, the authors noted the importance of providing education, mental health support and counseling to pregnant women during the COVID-19 pandemic; however, no specific programs or services had been designed at the time of our scoping review.
Additionally, two articles developed frameworks to ensure family support/ family-centred care during the pandemic [40, 61]. The “Person-Centred Guidelines for Preserving Family Presence in Challenging Times” (pg 1)  was created in virtual consultation with over 60 organizations, inclusive of patients, caregivers, advocates, clinicians and policy-makers. The guidelines were developed for healthcare leaders and health authorities and are intended to be applied across different contexts, including resource-challenged settings and among vulnerable populations. The goal of the guidelines is to balance individual needs with safety and community needs, support the principles of person-centred care across the continuum, keep patients and families connected through continual assessments, minimize risks and isolation, communicate expectations and policies clearly, support social connections, educate patients and families on discharge processes and partner with families. Some specific examples include keeping a digital diary of the patients’ ICU experience (to help fill in the gaps for patients and families about what happened during hospital stay and to minimize stress when discharged home) and redeploying staff, volunteers or medical students to act as “connectors” between patients and families, who might be separated (due to hospital stay), particularly if there is no technology available.
Factors impacting patient engagement
Just under half of the included articles (n = 13) described technological barriers or individual patient characteristics that impacted organizations’ ability to engage with patients and families [37, 40, 41, 44, 47–49, 51, 53, 54, 56–58]. The most common technological barriers to engaging with patients were: technology that did not support video conferencing, limited technology literacy among those engaging with it (usually patients and families), lack of comfort using technology for medical visits and slow or poor internet connection [37, 40, 41, 49, 51, 56, 58]. Individuals with cognitive, vision or hearing impairments, limited decision-making capacity or who required an interpreter to speak with their care provider created challenges for virtual engagement [40, 41, 47, 53]. The lack of patient access to a smart phone or internet, as well as the providers’ inability to read nonverbal cues and body language represented additional barriers to virtual care [40, 41, 48, 49, 51].
Lessons learned through patient engagement
Despite many challenges to engaging with patients and families during COVID-19, several articles also reflected on lessons learned [33, 37, 39, 41–43, 49, 53–56, 58]. There was an identified need for more team-based models and approaches to virtual care and telehealth [37, 41, 43]. Authors explained that virtual care should reflect the care an individual would receive in person, so if a patient would be supported by a multidisciplinary team at an in-person clinic, they should receive the same level of support virtually . The need for training prior to engaging in virtual care and telehealth was identified to ensure providers, patients and families were comfortable using the platform before participating in live sessions [41, 56]. Lastly, the responses to providing care differently (i.e. virtually) during COVID-19 should be used as an example for future disasters, that limit in-person care contact [53, 58]. Successful response strategies to maintain patient engagement during COVID-19 should be referred to and used as guidance if future disruptions were to occur .
To our knowledge, this scoping review was the first to examine what is known in the literature about patient engagement activities during the COVID-19 pandemic. Our findings showed a delineation between tools to support patients and caregivers in receipt of care interactions (most articles) and partnership activities in the design of care (few articles). Partnership activities like co-designing a new care delivery pathway or participating in decision making tables is how engagement is more traditionally defined. However, we took at broad view on engagement, in line with Carman’s framework, to include direct care consultations, allowing us to capture the various ways in which health systems stayed connected to patients and families during the pandemic.
Based on Carman’s continuum, the majority of published literature at the time of our scoping review was centered on activities at the level of direct care (e.g. consultations, care conferences, home monitoring, remote appointments/check-ups) . In some cases, the focus of these consultations was not just on medical care needs, but on social determinants of health and ability to self-manage (perhaps due to heightened acknowledgement of social needs amidst the pandemic). Many organizations were incorporating virtual visits into their practice prior to the pandemic. Virtual care and telehealth were emerging in certain fields to target specific population groups, but these technologies were often resisted by organizations because of worries around privacy and security , with concerns about ownership of data, authorization of unspecified use and data security noted as reasons for the lack of large-scale uptake . In spite of these concerns, it is possible that COVID-19 presented a window of opportunity for this work to be pushed forward and adopted at a rapid pace, as many Health Insurance Portability and Accountability Act (HIPAA) private health information confidentiality violations were waived during the pandemic to allow for the use of non-encrypted technologies for virtual care .
We also saw some examples of broader community engagement, also through use of technology, to not only share information, but to create a space for the community to connect, ask questions or share concerns [42, 55, 59]. Given the short time window of our scoping review, it is not surprising that individual engagement such as virtual consultations and home monitoring were implemented and published more quickly compared to other levels of engagement along Carman’s continuum (e.g. partners in organizational governance and policy change) . Changes to organization governance and policies often require more time, effort and resources. Overall, it is possible that the pandemic created the space, context and opportunity to quickly push forward and transition to a more virtual environment, which is reflected in the literature. A virtual environment is important for facilitating both individual interactions and patient partnership.
While it may seem that virtual care and the use of technology for providing health and social services do not reflect active patient engagement, such activities allow for capacity building and can incorporate models of shared decision-making, goal setting and patient autonomy in directing their own care. As such, these activities demonstrate a more active form of engagement and highlight how partnership may occur during virtual and direct care activities. However, it was not always clear in the included articles if these more active forms of direct care engagement were used.
Some initiatives did signify a deeper form of engagement by way of bi-directional information sharing (providing feedback to improve services and programs), as well as virtually connecting families to be part of clinical consultations due to visitor restrictions, similar to what we would see in activities such as bedside rounding . Furthermore, and perhaps most promising, was the involvement of multiple stakeholders (patients, clients and patient organizations) in a virtual environment in the design of a global patient experience survey for individuals with rheumatic disease, showing the potential of engagement during a crisis across all stages of the research cycle .
This scoping review highlighted the rapid transition to a virtual environment, through the use of online platforms, social media and telehealth. Almost all of the included articles used technology, in one form or another, to engage with patients and families during COVID-19, with virtual care being the most common (n = 13). Virtual care and the use of technology for patient engagement activities can open doors and increase opportunities for involvement [66, 67]. For example, accessibility, continuity of care, cost effectiveness, health outcomes, satisfaction and attention to equity (in terms of access and patients’ social, cultural and health needs being addressed) were highlighted as potential benefits of virtual care in a rapid review conducted by Li and colleagues (2020) . Similarly, a literature review of telehealth in rural Australia identified several benefits, including: lower costs, improved convenience accessing services and specialists and improved quality of services .
On the other hand, virtual care and technology use may also close doors and limit involvement from certain individuals and population groups in both clinical care interactions and, by extension, from patients and families involved in partnership activities (e.g., decision making tables, co-design activities). As identified in this review, engagement was impacted by poor internet connection, lack of technology or technology that did not support video conferencing and an individual’s lack of knowledge or comfort using technology [37, 40, 41, 49, 51, 56, 58]. Many of these challenges with telehealth and virtual care have been echoed in the literature [66, 68–70]. Bandwidth issues and access to devices have been noted to negatively impact the ability to engage virtually, as well as an individual’s overall experience [70, 71]. In addition to technology-related barriers to engagement, this review highlighted that individual level factors (i.e. health literacy, socioeconomic status) can also negatively impact virtual interactions, and as such, an individual’s opportunity to engage in activities when they are technology-based [40, 41, 47, 53]. Social and economic factors have been noted to create challenges for virtual visits, as mental health issues and low levels of health literacy can impede conversations between healthcare professionals and patients . Persons who are older, non-English speaking, unemployed, low income, live in rural areas or are a racial/ethnic minority (African American, Latino, Japanese, Chinese, Filipino, South Asian) can also experience a divide from virtual care . Furthermore, the use of virtual care and technology has the potential to increase opportunities for involvement (for patients both in clinical care interactions and partnership activities), but it may also hinder them. It is critical to better understand how to ensure involvement is not limited or restricted when transitioning to online platforms. The “Person-Centred Guidelines for Preserving Family Presence in Challenging Times” report recognized that technology is not always an option for people and included examples of non-technology based ways to connect (through volunteers, medical students, etc.) who act as connectors between patients and families. This connector role could also serve as the interface between patients, caregivers and other stakeholders.
The challenges related to virtual care engagement (language, health literacy, socio-economic status) parallel the kinds of barriers noted in patient engagement activities generally [72, 73]. For example, Chegini and colleagues conducted a qualitative study in which low levels of health literacy, cultural barriers, ineffective patient education and patient unwillingness were identified as patient-related barriers impacting engagement in the safe delivery of care in hospital . Since this scoping review pointed to equity challenges in engaging in virtual care it will be important to place emphasis on addressing these barriers as we move into the COVID-19 recovery period. Further, equity challenges in participating in patient engagement activities extend beyond virtual care and can occur along the continuum of engagement; thus highlighting the importance of applying a health equity lens to engagement, which is reflexive, intersectional and trauma-informed .
In using Carman’s framework as a guide, we identified several limitations that could be addressed in future work. Despite having three levels of engagement across a continuum of three categories, we identified challenges categorizing direct care level engagement activities. This was based on a limited ability to highlight differences in these activities. For example, the majority of patient engagement activities were categorized at the direct care level, as consultation or involvement. However, we identified a number of differences in these engagement activities that were lost with this categorization and we were able to create a more nuanced set of categories, many of which fit under the umbrella of direct are activities. Based on the purposes of this review, it was important to highlight these nuances and differences in the engagement activities, which is why we conducted a more in-depth analysis following the categorization according to the framework. Additionally, Carman et al.’s definition of patient engagement involves an active partnership between patients, families, representatives and health professionals; however, the levels of engagement along the framework continuum do not all include active partnership . It is important for the definition and framework to present consistent information to eliminate confusion around the classification of activities as patient engagement. Overall, this framework served as a fundamental starting point from which we adapted to better reflect the nuances discovered in our scoping review.
Based on the findings from this scoping review, several areas of future work have been identified. There is a need for more original research on all patient engagement activities (clinical care interactions and partnership activities) occurring during the pandemic to better describe engagement work from different perspectives, as well as exposure factors related to success. For example, research evaluating the use of technology for other types of patient engagement (i.e., advisory committees, planning and decision-making activities), exploring the perspectives of patients, families, providers and researchers regarding engagement activities during COVID-19 and gaining a better understanding of how organizations transitioned to maintain patient engagement. It is critical to understand strategies that have worked, or not worked, to allow organizations to continue patient engagement activities. The findings of this scoping review may imply that other types of engagement activities (e.g., partnering in organizational design and governance, policy decisions) were not occurring as much as individual level engagement, but it is highly likely that our results were more a reflection that this work, and the learning that can be gained from it, may not yet be published. However, it is important to explore this area further to identify if (and why) partnership activities at the organizational level came to a halt during the initial stages of the pandemic and beyond. Understanding the structures and processes that allow for engagement activities to continue during times of major disruption is critical for improving patient engagement activities and overall experiences. Environmental scans, original qualitative research and case studies, for example, are recommended methods for capturing these other, organizational level forms of engagement and is the next phase of work being conducted by our team. Additionally, as the focus of this scoping review was on patient engagement activities occurring during the first six months of the pandemic, it is important to explore if, and how, engagement approaches have evolved over the course of the pandemic. Lastly, there is a need to explore if, and how, patients, caregivers and families can remain involved in engagement activities without relying solely on technology.
There are a few limitations to be noted. First, based on the rapid publication of COVID-related research and topics, it is possible that relevant articles were missed because of the state of indexing. Second, our search was conducted in English, so it is possible that articles published in other languages were missed. Third, based on the current state of the literature, with many of the included articles being editorials and commentaries that lacked contextual information, we did not conduct a critical appraisal of the quality of the articles. However, this is not a requirement for scoping reviews . Lastly, we acknowledge that this review is not reflective of all the patient engagement work that is being done during COVID-19, as there is the potential that engagement is occurring on different platforms or other levels of engagement that take longer to implement have not yet been written about or published. For example, while we included articles that used social media for engagement, we did not specifically search social media (Twitter, Facebook, Sina Weibo, etc.) for posts related to engagement. Despite these limitations, this review presents a summary of the work that was implemented and published during the first six months of the pandemic.
This scoping review identified a number of examples of how healthcare systems stayed connected to patients and families during the pandemic. Though we had exclusively looked for examples of patient engagement activities, we found few examples of patient partnership and more examples of direct care consultations via technology as well as broader community engagement for purposes of sharing and receiving information related to the pandemic. Other research methods to explore specific contexts and initiatives (e.g., qualitative investigations or case studies) may better unpack the full spectrum of patient engagement activities that occurred during the pandemic and give greater insight into the barriers and facilitators of sustaining these activities.
The authors would like to thank the librarian at Trillium Health Partners (Antonia Giannarakos) for her guidance on the search strategy. We would also like to thank Dr. Alison Freeland, Sandy Dayes, Shiza Sheikh and Aditi Desai from Trillium Health Partners and Dr. Carol Fancott, Julie Drury and Jessie Checkley from Healthcare Excellence Canada for their ongoing support throughout the project. The findings and reflections in this paper do not necessarily reflect the views of our Collaborators and Funders.
- 1. World Health Organization. Rolling updates on coronavirus disease (COVID-19): World Health Organization; 2020. https://www.who.int/emergencies/diseases/novel-coronavirus-2019/events-as-they-happen.
- 2. World Health Organization. Coronavirus disease (COVID-19) pandemic: World Health Organization; 2020. https://www.who.int/emergencies/diseases/novel-coronavirus-2019.
- 3. Jones L, Palumbo D, Brown D. Coronavirus: A visual guide to the economic impact United Kingdom: BBC News; 2020. https://www.bbc.com/news/business-51706225.
- 4. Azevedo JoP, Hasan A, Goldemberg D, Iqbal SA, Geven K. Simulating the Potential Impacts of COVID-19 School Closures on Schooling and Learning Outcomes: A Set of Global Estimates: The World Bank; 2020. 61 p.
- 5. Becker E. How hard will the coronavirus hit the travel industry? United States: National Geographic; 2020. https://www.nationalgeographic.com/travel/2020/04/how-coronavirus-is-impacting-the-travel-industry/.
- 6. Government of Canada. How the coronavirus disease (COVID-19) is affecting immigration, refugees, citizenship and passport services Canada: Government of Canada; 2020. https://www.canada.ca/en/immigration-refugees-citizenship/services/coronavirus-covid19.html.
- 7. Zambrano-Monserrate MA, Ruano MA, Sanchez-Alcalde L. Indirect effects of COVID-19 on the environment. Science of The Total Environment. 2020;728:138813. pmid:32334159
- 8. Rocca R. Coronavirus: Toronto-area hospitals implement no visitor policies Toronto, Canada: Global News; 2020. https://globalnews.ca/news/6712594/coronavirus-toronto-hospitals-no-visitors/.
- 9. Coronavirus: Hospitals across southern England impose visitor restrictions England: BBC News; 2020. https://www.bbc.com/news/uk-england-51962793.
- 10. Hafner K. ‘A Heart-Wrenching Thing’: Hospital Bans on Visits Devastate Families United States: The New York Times; 2020. https://www.nytimes.com/2020/03/29/health/coronavirus-hospital-visit-ban.html.
- 11. MacCharles T. All non-COVID19 research and clinical trials at hospitals suspended or cancelled due to funding crunch Ontario, Canada: The Star; 2020. https://www.thestar.com/news/canada/2020/04/10/all-non-covid19-research-and-clinical-trials-at-hospitals-suspended-or-cancelled-due-to-funding-crunch.html.
- 12. Institute for Patient-and-Family-Centered Care. PFCC and COVID-19 Virginia, USA: Institute for Patient- and Family-Centered Care; 2020 [May 18, 2021]. https://www.ipfcc.org/bestpractices/covid-19/index.html.
- 13. Richards T, Scowcroft H. Patient and public involvement in COVID-19 policy making. BMJ. 2020;370:m2575. pmid:32611571
- 14. Berwick DM, Feeley D, Loehrer S. Change from the inside out: health care leaders taking the helm. JAMA. 2015;313(17):1707–8. Epub 2015/03/27. pmid:25812026
- 15. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759–69. Epub 2008/05/14.
- 16. Baker GR, Judd M, Fancott C. Creating ‘Engagement-Capable Environments’ in Healthcare. In: Baker MJaCM G.R., editor. Patient Engagement: Catalyzing Improvement and Innovation in Healthcare Toronto, Ontario: Longwoods Publishing Company; 2016. p. 11–34.
- 17. Estabrooks CA. Engagement-Capable Environments—No Less Challenging than other Large System Changes. Healthc Pap. 2017;17(2):40–5. Epub 2018/03/30. pmid:29595443.
- 18. Fancott C, Baker GR, Judd M, Humphrey A, Morin A. Supporting Patient and Family Engagement for Healthcare Improvement: Reflections on "Engagement-Capable Environments" in Pan-Canadian Learning Collaboratives. Healthcare quarterly (Toronto, Ont). 2018;21(SP):12–30. Epub 2018/12/20. pmid:30566400.
- 19. Rowland P, Brosseau M, Houle C. Patient Roles in Engagement-Capable Environments: Multiple Perspectives. Healthcare quarterly (Toronto, Ont). 2018;21(SP):45–9. Epub 2018/12/20. pmid:30566403.
- 20. Health Quality Ontario. Ontario’s Patient Engagement Framework: Creating a strong culture of patient engagement to support high quality health care. Toronto, Canada: Health Quality Ontario, 2017.
- 21. Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient And Family Engagement: A Framework For Understanding The Elements And Developing Interventions And Policies. Health Affairs. 2013;32(2):223–31. pmid:23381514
- 22. World Health Organization. What are integrated people-centred health services? Geneva, Switzerland: World Health Organization; nd [cited 2020 April 13]. https://www.who.int/servicedeliverysafety/areas/people-centred-care/ipchs-what/en/.
- 23. Kuluski K, Nelson MLA, Tracy CS, Alloway CA, Shorrock C, Shearkhani S, et al. Experience of Care as a Critical Component of Health System Performance Measurement: Recommendations for Moving Forward. Healthc Pap. 2017;17(2):8–20. pmid:29595439.
- 24. World Health Organization. Patient Engagement: Technical Series on Safer Primary Care. Geneva: Department of Service Delivery and Safety, World Health Organization, 2016.
- 25. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143. pmid:30453902
- 26. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implementation science: IS. 2010;5:69. pmid:20854677.
- 27. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and ExplanationThe PRISMA-ScR Statement. Ann Intern Med. 2018;169(7):467–73. pmid:30178033
- 28. Medical Library Association. COVID-19 Literature Searches: Medical Library Association; 2020. https://www.mlanet.org/p/cm/ld/fid=1713.
- 29. Chen Q, Allot A, Lu Z. Keep up with the latest coronavirus research. Nature. 2020;579(7798):193. Epub 2020/03/12. pmid:32157233.
- 30. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 2016;104(3):240–3. pmid:27366130.
- 31. Babineau J. Product review: Covidence (systematic review software). JCHLA / JABSC. 2014;35(2):68–71.
- 32. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qualitative health research. 2005;15(9):1277–88. Epub 2005/10/06. pmid:16204405.
- 33. Meinert E, Milne-Ives M, Surodina S, Lam C. Agile Requirements Engineering and Software Planning for a Digital Health Platform to Engage the Effects of Isolation Caused by Social Distancing: Case Study. JMIR Public Health Surveill. 2020. pmid:32348293.
- 34. Sirotich E, Dillingham S, Grainger R, Hausmann JS. Capturing patient-reported outcomes during the COVID-19 pandemic: Development of the COVID-19 Global Rheumatology Alliance Patient Experience Survey. Arthritis care & research. 2020. http://dx.doi.org/10.1002/acr.24257. pmid:32386125
- 35. Overton J, Denton K, Frumovitz M, Lewis C, Christensen S, Bezerra J, et al. Coronavirus (COVID-19): Patient experience-Administrative services on the frontline during crisis. Head & neck. 2020. https://dx.doi.org/10.1002/hed.26259. pmid:32415893
- 36. Ekzayez A, Al-Khalil M, Jasiem M, Al Saleh R, Alzoubi Z, Meagher K, et al. COVID-19 response in northwest Syria: innovation and community engagement in a complex conflict. J Public Health (Oxf). 2020. pmid:32436578.
- 37. Mgbako O, Miller EH, Santoro AF, Remien RH, Shalev N, Olender S, et al. COVID-19, Telemedicine, and Patient Empowerment in HIV Care and Research. AIDS and behavior. 2020. https://dx.doi.org/10.1007/s10461-020-02926-x.
- 38. Wei N, Huang BC, Lu SJ, Hu JB, Zhou XY, Hu CC, et al. Efficacy of internet-based integrated intervention on depression and anxiety symptoms in patients with COVID-19. Journal of Zhejiang University Science B. 2020;21(5):400–4. http://dx.doi.org/10.1631/jzus.B2010013. pmid:32425006
- 39. Amir-Behghadami M, Gholizadeh M. Electronic Screening System through Community Engagement: A National Strategic Plan to Find COVID-19 Patients and Reduce Clinical Intervention Delays. Infect Control Hosp Epidemiol. 2020. pmid:32362298.
- 40. Hart JL, Turnbull AE, Oppenheim IM, Courtright KR. Family-Centered Care During the COVID-19 Era. Journal of pain and symptom management. 2020. http://dx.doi.org/10.1016/j.jpainsymman.2020.04.017. pmid:32333961
- 41. Tenforde AS, Iaccarino MA, Borgstrom H, Hefner JE, Silver J, Ahmed M, et al. Feasibility and High Quality Measured in the Rapid Expansion of Telemedicine During COVID-19 for Sports and Musculoskeletal Medicine Practice. PM & R: the journal of injury, function, and rehabilitation. 2020. http://dx.doi.org/10.1002/pmrj.12422.
- 42. Hu G, Qiu W. From guidance to practice: Promoting risk communication and community engagement for prevention and control of coronavirus disease (COVID-19) outbreak in China. J Evid Based Med. 2020. pmid:32445287.
- 43. Medina M, Babiuch C, Card M, Gavrilescu R, Zafirau W, Boose E, et al. Home monitoring for COVID-19. Cleveland Clinic journal of medicine. 2020. http://dx.doi.org/10.3949/ccjm.87a.ccc028. pmid:32409432
- 44. Griffin KM, Karas MG, Ivascu NS, Lief L. Hospital Preparedness for COVID-19: A Practical Guide from a Critical Care Perspective. American journal of respiratory and critical care medicine. 2020. https://dx.doi.org/10.1164/rccm.202004-1037CP. pmid:32298146
- 45. Japan ECMOnet for COVID-19. Japan ECMOnet for COVID-19: Telephone consultations for cases with severe respiratory failure caused by COVID-19. Journal of intensive care. 2020;8(1):24. http://dx.doi.org/10.1186/s40560-020-00440-9. pmid:32288941
- 46. Murphy H. Managing Diabetes in Pregnancy before, during and after COVID-19. Diabetes Technology & Therapeutics. 2020. http://dx.doi.org/10.1089/dia.2020.0223. pmid:32396397
- 47. Yassa M, Birol P, Yirmibes C, Usta C, Haydar A, Yassa A, et al. Near-term pregnant women’s attitude toward, concern about and knowledge of the COVID-19 pandemic. The Journal of Maternal-Fetal & Neonatal Medicine. 2020:1–8. https://dx.doi.org/10.1080/14767058.2020.1763947.
- 48. Kreimer S. Neuropalliative Care During COVID-19—Addressing Isolation, Fear, and Terminal Illness. Neurology Today. 2020;20(11):1–22.
- 49. Mercadante S, Adile C, Ferrera P, Giuliana F, Terruso L, Piccione T. Palliative Care in the Time of COVID-19. Journal of pain and symptom management. 2020. http://dx.doi.org/10.1016/j.jpainsymman.2020.04.025. pmid:32376263
- 50. Padala PR, Jendro AM, Gauss CH, Orr LC, Dean KT, Wilson KB, et al. Participant and Caregiver Perspectives on Clinical Research During Covid-19 Pandemic. Journal of the American Geriatrics Society. 2020. http://dx.doi.org/10.1111/jgs.16500.
- 51. Peahl AF, Smith RD, Moniz MH. Prenatal Care Redesign: Creating Flexible Maternity Care Models Through Virtual Care. American journal of obstetrics and gynecology. 2020. http://dx.doi.org/10.1016/j.ajog.2020.05.029. pmid:32425200
- 52. Fortune C, Gietzen A, Guillot G, Lewis N, Nielsen K, Richard M, et al. Protocol for a partially nested randomised controlled trial to evaluate the effectiveness of the scleroderma patient-centered intervention network COVID-19 home-isolation activities together (SPIN-CHAT) program to reduce anxiety among at-risk scleroderma patients. J Psychosom Res. 2020. pmid:32521358.
- 53. Patel PD, Cobb J, Wright D, Turer R, Jordan T, Humphrey A, et al. Rapid Development of Telehealth Capabilities within Pediatric Patient Portal Infrastructure for COVID-19 Care: Barriers, Solutions, Results. Journal of the American Medical Informatics Association: JAMIA. 2020. http://dx.doi.org/10.1093/jamia/ocaa065. pmid:32302395
- 54. Annis T, Pleasants S, Hultman G, Lindemann E, Thompson JA, Billecke S, et al. Rapid Implementation of a COVID-19 Remote Patient Monitoring Program. Journal of the American Medical Informatics Association: JAMIA. 2020. http://dx.doi.org/10.1093/jamia/ocaa097. pmid:32392280
- 55. Edvardsson D, Baxter R, Birkett P, Dunlop A. Supporting lives lived whilst protecting lives saved during COVID-19. Nordic Journal of Nursing Research. 2020;40(2):61–3.
- 56. Contreras CM, Metzger GA, Beane JD, Dedhia PH, Ejaz A, Pawlik TM. Telemedicine: Patient-Provider Clinical Engagement During the COVID-19 Pandemic and Beyond. Journal of gastrointestinal surgery: official journal of the Society for Surgery of the Alimentary Tract. 2020. http://dx.doi.org/10.1007/s11605-020-04623-5. pmid:32385614
- 57. Brown TS, Bedard NA, Rojas EO, Anthony CA, Schwarzkopf R, Barnes CL, et al. The Effect of the COVID-19 Pandemic on Electively Scheduled Hip and Knee Arthroplasty Patients in the United States. Journal of Arthroplasty. 2020. http://dx.doi.org/10.1016/j.arth.2020.04.052.
- 58. Peters AL, Garg S. The Silver Lining to COVID-19: Avoiding Diabetic Ketoacidosis Admissions with Telehealth. Diabetes Technology & Therapeutics. 2020. http://dx.doi.org/10.1089/dia.2020.0187. pmid:32383989
- 59. Kullar R, Goff DA, Gauthier TP, Smith TC. To Tweet or Not to Tweet-a Review of the Viral Power of Twitter for Infectious Diseases. Current Infectious Disease Reports. 2020;22(6):14. http://dx.doi.org/10.1007/s11908-020-00723-0.
- 60. Chen Q, Min C, Zhang W, Wang G, Ma X, Evans R. Unpacking the black box: How to promote citizen engagement through government social media during the COVID-19 crisis. Comput Human Behav. 2020. pmid:32292239.
- 61. Dokken DAE. Family Presence During Challenging Times. Pediatric nursing. 2020;46(4):161–2.
- 62. Hall JL, McGraw D. For Telehealth To Succeed, Privacy And Security Risks Must Be Identified And Addressed. Health Affairs. 2014;33(2):216–21. pmid:24493763
- 63. Hardcastle L, Ogbogu U. Virtual care: Enhancing access or harming care? Healthcare Management Forum. 2020;33(6):288–92. pmid:32686506
- 64. Centers for Medicare and Medicaid Services. Medicare telemedicine healthcare provider fact sheet: Medicare coverage and payment of virtual services Maryland, United States: US Centers for Medicare & Medicaid Services; 2020. https://www.cms.gov/newsroom/fact-sheets/medicare-telemedicine-health-care-provider-fact-sheet.
- 65. Fowler B, Johns J, Tanniru M, Ballijepally V, Roumani Y, Bobryk D, et al. Engaging patients through Multi-Disciplinary Rounding–The case study at a Michigan hospital. Journal of Hospital Administration. 2018;7:17.
- 66. Li J, Roerig M, Saragosa M, Carbone S, Abdelhalim R, Jamieson M, et al. Virtual Primary Care in Northern, Rural and Remote Canada. Toronto, Canada: North American Observatory on Health Systems and Policies, 2020 26.
- 67. Moffatt JJ, Eley DS. The reported benefits of telehealth for rural Australians %J Australian Health Review. 2010;34(3):276–81.
- 68. Glauser W. Virtual care is here to stay, but major challenges remain. Canadian Medical Association Journal. 2020;192(30):E868. pmid:32719027
- 69. Nguyen A, Mosadeghi S, Almario CV. Persistent digital divide in access to and use of the Internet as a resource for health information: Results from a California population-based study. International Journal of Medical Informatics. 2017;103:49–54. pmid:28551001
- 70. Horner L. The Challenges and Benefits of Implementing Virtual Care New Jersey, United States: Inside Digital Health; 2018. https://www.idigitalhealth.com/news/the-challenges-and-benefits-of-implementing-virtual-care.
- 71. Ipsos MORI, York Health Economics Consortium, Salisbury C. Evaluation of Babylon GP at hand: Final evaluation report. London, United Kingdom: Ipsos MORI Social Research Institute, 2019.
- 72. Chegini Z, Janati A, Babaie J, Pouraghaei M. Exploring the barriers to patient engagement in the delivery of safe care in Iranian hospitals: A qualitative study. Nurs Open. 2019;7(1):457–65. pmid:31871731.
- 73. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Services Research. 2014;14(1):89. pmid:24568690
- 74. Roche P, Shimmin C, Hickes S, Khan M, Sherzoi O, Wicklund E, et al. Valuing All Voices: refining a trauma-informed, intersectional and critical reflexive framework for patient engagement in health research using a qualitative descriptive approach. Research involvement and engagement. 2020;6:42. Epub 2020/07/24. pmid:32699647.