Setting

This study was conducted virtually within Canada, a high-income country with provincially and territorial-based public health insurance that covers all necessary hospital and physician services [15]. Both within and across the provinces and territories of Canada, primary healthcare is delivered through different models. Largely, it is funded following a fee-for-service (FFS) model, but has adopted new funding models such as the enhanced fee-for-service model and capitation model. For example, in Ontario, the enhanced FFS model is similar to the traditional FFS model, except that it includes additional targeted fee increases, extended hours, performance-based initiatives, and patient enrollment. In contrast, the capitation model reimburses primary care physicians for each person attributed to them, based on age and sex adjusted factors [16]. More recently, Ontario has been forming Ontario Health Teams including family physicians, nurse practitioners, social workers, dietitians, and other healthcare professionals to increase interdisciplinary care for a given community [17].

Design

This emergent qualitative study used virtual deliberative dialogues. Deliberative dialogue is a participatory method in which people affected by an issue are gathered to provide advice to decisionmakers [18–20]. With facilitator support, participants work together to interpret existing evidence and identify approaches to the issue that reflect their—sometimes conflicting—values and tacit knowledge [21, 22]. We selected this method because we believe that the values and wisdom of primary care stakeholders can guide health AI in directions that support equity and advance the Quadruple Aim (optimizing health system performance through improved patient experience, improved clinical experience, better outcomes, and lower costs [21–23]).

To adhere to COVID-19 public health restrictions, we held twelve 90-minute dialogues by video conference between September 8^th and October 15^th, 2020. Dialogues were divided into three rounds and all sessions were facilitated by one trained Master’s-level graduate student (TLU). Before taking part, participants reviewed an information package that provided an overview of health AI, key definitions, examples of AI in medicine, possible uses in primary care, and ethical challenges. The study was approved by the research ethics boards of Unity Health Toronto and the University of Toronto. Verbal informed consent was obtained from all participants and documented prior to scheduling their first dialogue session.

Participants

Patients and primary care providers were recruited under distinct purposive variation sampling frames from various online advertisement platforms and email distribution lists. This recruitment method is outlined further by Patton et al. [24], and was chosen in order to optimize variation in age, gender, race and ethnicity, education level, income, province of residence for patients and gender, race and ethnicity, provider type, country of health professions education, years in practice, practice size, and province of practice for primary care providers. Specifically, patients were recruited using Kijiji and social media channels across 10 provinces as well as patient advisor distribution lists (patient advisory network in British Columbia, patient-family advisory council of downtown Toronto hospitals). Primary care physicians were recruited through social media channels, primary care research network news channels, and through email invitations from AP to practice colleagues and those in his network interested in digital health and health informatics. To be eligible, patients must have seen their primary care provider once within the last year and providers had to work at least one day in the clinic per week. To promote variation, we collected sociodemographic information during consent interviews and then iteratively adjusted recruitment strategies. We used a critical case frame for system leaders, seeking those involved in digital health, health informatics, or primary care governance [24]. Patients and providers were invited to take part in up to three rounds, while system leaders were invited to join one final-round session or to provide feedback on the dialogue from the first two sessions by survey. The sessions were designed this way to allow for interactive dialogue between the different perspectives of patients, providers, and health system leaders in real-time. An honorarium of $32.50 USD was offered to participants for every session attended. TLU was acquainted with two patient participants. AP was known to most providers in the study and observed three sessions.

Data collection & analysis

Audiovisual recordings and fieldnotes were generated for each session by one of the observers (ACN, JM). Following a session, TLU and the designated observer met after each session to analyze fieldnotes using interpretive description methods, coding for concerns, perceived barriers, and facilitators of AI implementation in primary care [25–28]. This analysis guided the emergent nature of the study (e.g., participant-facing summaries, use cases, etc.) and generated an initial code list. Early codes were later confirmed and expanded upon through dual-coder thematic analysis of verbatim transcripts (TLU, KD). Transcript analysis occurred in two passes, the first being entirely inductive and the second using agreed upon codes. TLU, KD, and AP then met to review coded transcripts and agree on final interpretations. The final interpretations were compiled into four major themes and then compared to Sittig and Singh’s sociotechnical model for studying health information technology (HIT) in complex adaptive health systems [12]. Data was managed using Microsoft Excel and Word. Saturation was reached for most concepts after nine sessions.

Rigor

Dialogue guides for rounds 2 and 3 were not pilot tested because of the study’s emergent nature. These guides were iterated upon and further developed as the study progressed. Investigator triangulation was achieved through the involvement of multiple researchers in data collection and analysis [29]. Data triangulation was achieved through careful comparison of field notes, primary transcripts, researcher interpretive reflections, and participant activities throughout the study. We created multiple opportunities for participants to member-check researcher interpretations. Eighteen participants provided reflections on preliminary findings at the study’s conclusion.

Participants

Three rounds of deliberative dialogue sessions were completed with 48 participants from 8 Canadian provinces. This included 22 patients, 21 primary care providers, and 5 health system leaders; participants’ characteristics can be found in S1 Table and have been published elsewhere [30].

Findings

From these dialogue sessions, a series of barriers and strategies to implement AI in primary care emerged, which were grouped into four major themes: (1) system and data readiness, (2) the potential for bias and inequity, (3) the regulation of AI and big data, and (4) the importance of people as technology enablers. Within each of these 4 themes, participants spoke about trust, either in reference to trust in the technology or in the provider using it, so trust will be discussed within each theme. When asked what they thought about algorithm explainability, all but one of the patient participants agreed that they would trust AI algorithms used in their care by proxy of the trust they placed in their provider. Patients in one session suggested that health AI-enabled harms could damage public trust in health institutions. Patients and providers across sessions identified regulatory oversight, scientific evidence, and participatory co-development processes as builders of trust.

System and data readiness

Significant concern around adoption and the readiness of Canada’s health system came up in many sessions and was particularly focused on both the readiness of our current technology/data, as well as the willingness for clinics to engage with and adopt new AI technologies. Some primary care clinics still operate using paper and fax machines, and therefore implementing AI may be too advanced. Additionally, primary care clinics operate independently and are fragmented from one another. Each clinic "enters things in all different ways. We have to make sure that the tool is good enough to manage the many different ways that people get information and enter information" (PR-7905). Creating an adoption strategy and designing tools to accommodate each clinic may be challenging.

Participants acknowledged the potential for AI to improve health and healthcare. However, a few patients and most providers recognized that the optimal performance of health AI tools depends upon a level of HIT interoperability that does not exist in many jurisdictions. For AI tools to reliably inform clinical decision-making and support integrated, equitable care, providers believed that algorithms must synthesize data from multiple sources, including administrative databases and EHRs across levels of care and care settings. They also believed that certain advanced AI tools (e.g., rare disease diagnosis) would have to synthesize an exponentially increasing body of evidence and ever-changing guidelines to be clinically useful.

Furthermore, when discussing the willingness to adopt AI technology, it was highlighted that gaining buy-in from clinic owners and operators may be challenging since many of them were promised a series of quality improvements during the EHR rollout that were not met. As one participant suggested, “the quality improvements that were expected didn’t happen” (PR-4404). They may be hesitant to invest and place trust in an AI system unless they can clearly and quickly realize the benefit of doing so.

Potential for bias and inequity

With varying levels of technological readiness and challenges with clinic uptake, the potential for biased algorithms and inequitable access to AI tools becomes increasingly concerning. Several providers worried that lagging digitization in resource-constrained settings (e.g., rural communities), poor interoperability of established HIT systems, or data monopolies held by leading EHR vendors could negatively impact health equity by driving the uneven application of AI tools across settings, or limiting datasets available for developing AI tools to smaller, non-representative populations, thereby increasing the risk of bias and inequity.

The potential for bias was a concern that came up in every dialogue session. Some participants brought up past studies where AI tools were proven to further perpetuate the biases of their developers and of society. As one patient participant mentioned, “there could be bias in your software. Because these factors are not all equal, they have different weights of importance. And so how that’s applied will affect how good the results are going to be” (PT-5981). There was also concern for the bias that could be present in the guidelines and criteria that feed the AI algorithms. Participants were worried that if pharmaceutical companies or other private enterprises sponsored the tool, they may bias the algorithms to produce outcomes that favor their product. Participants linked this theme to notions of trust. As evidenced by the following quotation, “we want to make sure that the tools we use don’t create new problems… they are an opportunity to address some of the biases that already exist in our system. . .and that these tools are probably only as good as the data we provide them” (PR-7905).

Equity and access to AI technology were discussed from many different perspectives throughout the dialogue sessions. From a patient perspective, there was expressed concern that some populations may not want to or be able to interact with AI tools, which as a result, may further deter them from the health system as a whole and widen the care gap that already exists. From a primary care clinic and provider level, the clinics that are frequently last to procure EHRs and new technology will likely also be the last to adopt a new AI tool; thus, we may further widen the divide between the advantaged and disadvantaged primary care practices and the individuals they treat. Overall, while participants are hopeful for the potential that AI holds, it will likely not serve populations that most frequently “fall through the cracks” (PT-4653), and therefore it is important to consider these populations and their ability to access and trust the tool before it is implemented widely in primary care.

Regulation of AI and big data

Participants from all three groups believed that inadequate regulatory oversight of health AI tools places the public at risk of harm to physical health or wellbeing. This was a perceived barrier to health AI adoption because it decreased patient and provider trust in available tools. To feel comfortable using an AI tool in clinical decision-making, providers wanted assurance that it had been approved by a regulatory authority after meeting rigorous standards of performance, in terms of accuracy, risk of bias, and patient safety. Participants from all three groups, but especially patients, were concerned that existing privacy laws do not adequately prevent AI developers from monetizing PHI or allowing PHI to be used in ways that impact patient wellbeing (e.g., insurance or loan decisions). One provider stated that they would like to be aware of “how this data is being used and whether there were any concerns about this data being used by a [private] company” (PR-7905) and a patient mentioned, “I think it’s really important to clearly set out what the data be used for and ensure that the patient understands how the confidential confidentiality and privacy still applies in those situations" (PT-4653). Evidently, all participants stressed the importance of regulatory oversight to ensure the tools are frequently monitored for safety and that there is insight into the ownership, control, and access of health data to ensure it is not being monetized or used inappropriately.

Importance of people as technology enablers

People play a vital role in the success of any given technology; therefore, the barriers that impact the people using and affected by the technology must be properly mitigated. From the dialogue sessions, these barriers included the need for provider training and skill preservation, the fear of changing the provider-patient relationship, and the challenge of designing tools to properly suit the user’s needs. Firstly, many providers were unsure about the training that would be needed to successfully adopt a given AI tool and were worried that it may impede their clinic workflow. As stated by one provider, “oftentimes, you have to adapt your workflow and sometimes your clinical processes to accommodate the tool where I really strongly feel it needs to be the reverse” (PR-4404). After being trained on the tool, some were concerned that the clinical skills and reasoning of primary care providers could be at risk due to dependence on the tool. As one provider said, “we don’t want to lose that ‘art of medicine’ that we have learned over the years, because it’s a very challenging profession where you see people with undifferentiated complaints, so you have to know something about everything. And with AI, if we rely too much on it, we might lose that clinical judgment." (PR-1783).

Secondly, some participants feared that these technologies could harm patient-provider relationships, while others thought it may enhance them. For example, one patient mentioned, “I want my doctor to be present with me in a conversation and not staring at the computer and looking at all the tools or boxes that they need to check off” (PT-4159). Many agreed with this notion, but some did refute and stated that if the AI tools are designed correctly, they may provide more space and time for relationship building and trust between the patient and the provider. Lastly, all participants agreed that the tools must be designed deliberately, to beneficially serve the needs of the users and not contribute to patient, provider, or administrative burnout.

Strategies to promote adoption

To promote the adoption of health AI in primary care and beyond, participants offered a host of strategies to build patient and provider trust in the technology based on each theme (summarized in Table 1). Of these, the most robustly supported was a participatory AI co-development model that meaningfully involves patients, interprofessional primary care providers, clinical administrative staff, HIT specialists, and health policymakers. Participants believed that stakeholder engagement must occur upstream of piloting, ideally at the conception phase, and remain ongoing. They also specified that stakeholders must be diverse in terms of sociodemographic characteristics (e.g., race or ethnicity, gender, income), geographic location, and healthcare needs, as well as professional settings for providers (e.g., telehealth, walk-in, community health centers). Through this model, described by a patient as “designing with, rather than for,” stakeholders could co-construct AI tools that address shared application priorities and promote patient-centered practices. Within the model, participants viewed processes that facilitate iterative stakeholder input (e.g., a participatory piloting program) as particularly useful for ensuring usability and workflow integration, correcting for bias, and improving algorithm performance through feedback. To specifically help overcome the barriers associated with system and data readiness, investments are required to allow for different technological systems to “talk to one another”. This would thereby enable integration between primary care clinics using different EHR technologies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Strategies to promote adoption based on each theme.

https://doi.org/10.1371/journal.pone.0281733.t001

The second theme, the potential for bias and inequity, may begin to be mitigated by promoting the use of unbiased, representative, good quality data to feed algorithms. It was recommended by one participant that to ensure the AI tool remains free of bias overtime, it should be re-assessed and analyzed routinely to check for bias. There was an expressed desire for ethics training courses for those involved at many stages of the design, implementation, and evaluation life cycle of the technology.

Strategies to assist with the regulation of AI and big data will require strong leadership commitment by health leaders and government players. Providers felt that this commitment to health AI would act as an example for primary care clinics to follow. Additionally, to increase trust in the technology, national AI rules and regulations for healthcare must be designed and implemented. There was an expressed need for content standards and protocols to further enable interoperability, and a need for regulatory standards to ensure representation and bias mitigation within the algorithms and training data sets.

Building trust, preventing burnout, and preserving patient-provider relationships begins with outlining a clear value proposition for each user and individual affected by the technology. It was highlighted that the benefits, limitations, and uses of the technology must be transparently communicated. Following this, simple AI technology that can complete basic operational tasks should be implemented first, before transitioning into the use of complex AI systems to encourage uptake and trust in the technology.