To solicit leading health informaticians’ predictions about the impact of AI/ML on primary care in the US in 2029.
In September 2019, health informatics experts were selected by the research team, and invited to participate the Delphi poll. Participation in each round was anonymous, and panelists were given between 4–8 weeks to respond to each round. In Round 1 open-ended questions solicited forecasts on the impact of AI/ML on: (1) patient care, (2) access to care, (3) the primary care workforce, (4) technological breakthroughs, and (5) the long-future for primary care physicians. Responses were coded to produce itemized statements. In Round 2, participants were invited to rate their agreement with each item along 7-point Likert scales. Responses were analyzed for consensus which was set at a predetermined interquartile range of ≤ 1. In Round 3 items that did not reach consensus were redistributed.
A total of 16 experts participated in Round 1 (16/29, 55%). Of these experts 13/16 (response rate, 81%), and 13/13 (response rate, 100%), responded to Rounds 2 and 3, respectively. As a result of developments in AI/ML by 2029 experts anticipated workplace changes including incursions into the disintermediation of physician expertise, and increased AI/ML training requirements for medical students. Informaticians also forecast that by 2029 AI/ML will increase diagnostic accuracy especially among those with limited access to experts, minorities and those with rare diseases. Expert panelists also predicted that AI/ML-tools would improve access to expert doctor knowledge.
Citation: Blease C, Kharko A, Locher C, DesRoches CM, Mandl KD (2020) US primary care in 2029: A Delphi survey on the impact of machine learning. PLoS ONE 15(10): e0239947. https://doi.org/10.1371/journal.pone.0239947
Editor: Claudia Marotta, IRCCS Neuromed, ITALY
Received: May 3, 2020; Accepted: September 16, 2020; Published: October 8, 2020
Copyright: © 2020 Blease et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Individual participant data for Rounds 1 and 3 of the Delphi Poll cannot be shared publicly because of the need to ensure participant anonymity (per Beth Israel Deaconess Medical Center Institutional Review Board). De-identified raw data from Round 2 is available as S3 Appendix. All relevant aggregated data are within the manuscript and its Supporting Information files.
Funding: CB was supported by an Irish Research Council-Marie Skłodowska-Curie Fellowship, and a Keane Scholar Award. AK was funded by the School of Psychology, University of Plymouth. CL was funded by a Swiss National Science Foundation grant (P400PS_180730). The study funders played no role in the study design; writing of the report; or the decision to submit the manuscript for publication. Researchers were independent of influence from study funders.
Competing interests: The authors have declared that no competing interests exist.
Attention in medicine and related fields has increasingly focused on the potential of big data, artificial intelligence (AI), and machine learning (ML) to change the delivery of healthcare [1–4]. Much of this debate has focused on the promise of AI/ML to augment or even disintermediate the clinical roles of physicians in gathering and monitoring patient health information, and to undertake core tasks such as diagnostics, prognostics, and the formulation of personalized patient healthcare plans [1, 4–9]. Differentiating the hype from hope in the discourse about AI/ML in medicine is crucial to better understand the scope for the computerization of medicine. Although broad predictions of the impact of AI/ML on healthcare are ubiquitous, credible short-term predictions are necessary to address questions about resource allocation, and the adequacy of medical education and training.
Recently a number of surveys have explored medical students’, and physicians’ views about the impact of AI/ML on the future of medical practice [10–16]. Currently, there is scarce exploration of consensus views among informaticians ; in particular, on how AI/ML might meaningfully influence medical care in the short-term . To address this research gap, we designed a Delphi survey to explore leading health informaticians’ predictions about the impact of machine learning on primary care in the US in 2029. To our knowledge, this is the first investigation of experts’ opinions about the impact of AI/ML on the future of the near future of general medical practice.
The Delphi method
The Delphi Method, developed by the Rand Corporation in the 1950s [19, 20] is designed to pool the opinions of a purposive sample of identified experts in a given field to establish consensus predictions [21, 22]. Delphi polls rely on non-probability sampling techniques to identify a panel of experts: since participants are not randomly selected, representativeness is neither intended nor assured . The selected panel of experts is invited to answer a series of questions anonymously . Participants are next asked to reassess their initial judgments in light of group trends until consensus is obtained . This anonymous, iterative technique carries distinctive advantages over focus groups by avoiding the influences of individual dominant personalities, group-think, and helps to keep participants ‘on topic’ [19, 23].
Delphi surveys are particularly well suited to exploring consensus views related to new lines of inquiry, and for establishing goal-setting, and needs assessments in policy-making [19, 23]. Since Delphi polls provide more accurate predictions than other forecasting methodologies, the approach is often used as a policy and practice heuristic for health care management, and resource allocation [25–27].
We used a modified Delphi technique which is structured into three discrete rounds [19, 23, 28, 29]. In Round One, questions are open-ended, requiring free-text answers. Responses are aggregated and coded into a series of statements. In Round Two, experts are provided with this list of statements, and requested to provide their level of agreement with each item. Depending on the survey items, Round 2 and 3 questionnaires requested ‘yes’ or ‘no’ responses, or participants’ level of agreement with statements on 7-point Likert scales: 1 = greatly decrease, 2 = moderately decrease, 3 = slightly decrease, 4 = remain the same, 5 = slightly increase, 6 = moderately increase, 7 = greatly increase; 1 = very unlikely, 2 = moderately unlikely, 3 = slightly unlikely, 4 = uncertain, 5 = slightly likely, 6 = moderately likely, 7 = very likely; or 1 = strongly disagree, 2 = moderately disagree, 3 = slightly disagree, 4 = neutral, 5 = slightly agree, 6 = moderately agree, 7 = strongly increase. Those statements that reach a predefined level of agreement are omitted to reduce participant survey fatigue, and items that lack consensus are re-circulated via a final anonymous poll. In the third and final round, panelists are reminded of their own response to the remaining statements as well as the median response of other experts, and are invited to preserve or revise their answer. A key aim of Delphi methodology is to maintain as high a response rate as possible rounds [23, 30, 31], and the accuracy of forecasts has been demonstrated to improve between each round .
The expert panel
Although there is no universally agreed sample size for Delphi polls , our aim was to balance the size of the panel with a high response rate between the three rounds. We therefore aimed to achieve a panel of around 12–15 individuals who would agree to share their expertise, and be committed to giving their time to respond to each round. Using purposive sampling methodology, the research team compiled a list of 27 highly trained and knowledgeable individuals with context-specific knowledge about health informatics and primary care in the US. Addressing the question about how to identify domain-specific ‘experts’, our goal was to prioritize panelists for their recognized competence in the field of health informatics. We defined expertise to mean a person who had published significant contributions within the field of health informatics, and/or individuals who were currently appointed as research leaders, or as health information officers. Acknowledging that heterogeneous panels have been shown to result in more accurate estimates , and that what counts as an expert can be influenced by goals, values, and the manner in which knowledge is generated, we aimed to recruit diverse participants from across academia, healthcare, non-profit organizations, and industry; and to strive for panelists with a varied complementarity of interests within health informatics. Measures were also taken to ensure demographic diversity among invited participants along the lines of gender, age, nationality, and race/ethnicity. This study was deemed exempt research by Beth Israel Deaconess Medical Center Institutional Review Board and granted ethical approval by the University of Plymouth, UK.
Prospective panelists were contacted via email in September 2019, with an invitation and internet link to the survey. Individuals were informed that we desired a commitment on the part of experts to respond to all three rounds, that adequate response time would be given to answer each round of the survey, participation was voluntary and unpaid, and that participants could withdraw at any time. Prospective participants were also informed that they would remain anonymous to other participants, their individual responses would not be shared with other panelists, and their contribution would be confidential. Respondents’ names were also replaced with a study ID number by AK in order to preserve participant anonymity among other team members in data analysis.
We created an electronic questionnaire on JISC Online Surveys hosted at the University of Plymouth, UK (https://www.onlinesurveys.ac.uk/). The poll incorporated a three-step modified Delphi method which took place between September 2019 and January 2020. Participants were sent 3 reminders after each round of the survey, and given 4–6 weeks to respond to Rounds 1 and 2, and 8 weeks to respond to Round 3 which fell over the New Year period.
In the first round, the Delphi survey requested demographic information; this was followed by 5 sections, with 7 open-ended questions, on the impact of machine learning on primary care by 2029 (see S1 Appendix; Table 1). The sections comprised: (1) Patient care (3 open-ended questions); (2) Access to care (1 open-ended question); (3) Primary care workforce (1 open-ended question); (4) Technological breakthroughs (1 open-ended question); and (5) The future of the primary care physicians (1 closed ended question; and 1 open-ended question). We also included a final comment-box for feedback on the survey.
Responses to Round 1 were collated and coded into lists of statements. Coding was conducted by CB and independently reviewed by CL and AK, and subsequent revisions were made. Comments that were unrelated to the themes, or were deemed redundant were eliminated. Similar statements were grouped together and translated into concise items; whenever possible, replication of exact phrasing by participants was employed. These items were circulated in Round 2, and an online survey was sent to each individual member of the panel. Participants were requested to respond to categorical variables by selecting a ‘yes’ or ‘no’ response, and to questions with continuous variables by using predefined 7-point Likert scales (see S2 Appendix).
Prior to consensus analysis of responses in Rounds 2 and 3, for categorical variables consensus was set at ≥ 75%, and for continuous variables consensus along 7-point semantic differential scales was set at an interquartile range of ≤ 1 [19, 34]. After analysis of Round 2 results, items that did not reach consensus were redistributed for Round 3. In Round 3, each participant received a personalized survey link. Panelists were reminded of their response to items in Round 2, and provided with the median collated response of the other participants.
We obtained 17/29 response for Round 1 (see Table 2). One invited expert circulated the online survey to another respondent who was later excluded bringing the total to 16/29 (55%) (see Fig 1). Round 1 comprised of 4 (25%) female and 12 (75%) male participants. Respondents differed from invited non-respondents in terms of gender: initial invitations were extended to 11 (38%) females and 18 (62%) male experts. All 16 panelists in Round 1 held an MD (n = 11, 69%), PhD (n = 9, 56%), or both (n = 1, 6%) (also see: Acknowledgments).
Responses to Round 1 were translated into itemized lists of statements. As a result of this process, the was survey was expanded into 57 items arranged into 7 sections: (i) diagnostic accuracy (10 items), (ii) healthcare disparities (5 items), (iii) empathic care of patients (8 items), (iv) access to care (9 items), (v) primary care workforce (7 items), (vi) technological advancements in primary care (10 items), and (vii) the long-term future of the profession (8 items) (see S2 Appendix). The panel was repeatedly prompted to forecast changes to primary care by 2029, and questions emphasized that predictions should be restricted to the US context. Throughout the survey experts were reminded to “predict what you believe will happen and not what you personally would like to see happen”. After completing each section, participants were also invited to provide free text comments, and following completion of the survey, offered to provide any additional feedback.
Rounds 2 and 3
In Round 2, 13/16 experts participated in the online survey (response rate of 81%) [see S3 Appendix, for Round Two raw data]. In Round 3, 13/13 experts responded (response rate of 100%). In Rounds 2 and 3, participants included 4 (31%) females, and 9 (69%) male participants (see Table 2 for demographic information). Table 3 presents the item means and standard deviations for item responses, and also indicates the items that reached consensus in Round 2, those that obtained consensus in Round 3, and items that failed to secure expert consensus. As described in the Methods, and as indicated in Table 3, items reflect three different 7-point Likert Scales. To undertake interpretation of panelists’ predictions, these items were divided into three rational, a priori categories. For the scale identified as ‘I’, responses were bounded into items that experts expected to increase (item mean of 4.5 and greater), remain about the same (item mean of 3.5–4.4), and to decrease (item mean of 3.4 and less). For the scale identified as ‘L’, responses were differentiated into items that the experts predicted to be likely (item mean of 4.5 and greater), items that they were uncertain about (item mean of 3.5–4.4), and those they predicted to be unlikely (item mean of 3.4 and less). Finally, for the scale identified as ‘A’, responses were bounded into items about which the panel agreed (item mean of 4.5 and greater), those that they were neutral on (item mean of 3.5–4.4), and those items about which they disagreed (item mean of 3.4 and less).
Primary care workforce.
Informaticians disagreed with the prediction that in the US “there is a 90% chance primary care doctors will be obsolete 100 years from now”. Panelists also agreed that primary care in the US would be one of the last specialties to be replaced by AI/ML. In the short-term, by 2029 in the US, experts forecast that advancements in AI/ML will incur a number of workforce changes in primary care (see Fig 2).
Note. For items 33, 34, 35, 36 the scale used was: Greatly decrease = 1, Remain the same = 4, Greatly increase = 7. For items 37 and 39: Very unlikely = 1, Uncertain = 4, Very likely = 7.
Dependent on educational background some contrasting predictions emerged (see Table 4). Beyond 2029, experts without a medical degree (MD) considered it likely that primary care doctors would always be needed to deliver empathic aspects of care–a prediction that did not engender consensus among panelists with a medical degree. Similarly, panelists without a medical education strongly agreed that the adoption of AI/ML tools in US healthcare will be slow, by 2029, due to the culture of medicine while those with a medical education did not reach consensus on this item. Conversely, experts with a medical degree forecast that by 2029 US doctors will transition from the role of dispensers of knowledge to managing teams and information systems; however, there was no consensus on this item among participants without an MD.
Overall, our experts forecast that by 2029 AI/ML will increase rates of diagnostic accuracy especially for conditions where the markers of illness are relatively homogenous (see Fig 3). Among their predictions, panelists envisaged that by AI/ML tools will improve diagnostic accuracy among persons with limited access to human experts, individuals identifying as from minority groups, or for those with rare conditions.
Access to care.
As a result of the disintermediation of physicians’ expertise, our experts predicted that by 2029, AI/ML will increase access to primary care in the US (see Fig 4).
Note. For items 24, 25, 26, 27, 28 the scale used was: Greatly decrease = 1, Remain the same = 4, Greatly increase = 7. For items 31 and 32 the scale used was: Very unlikely = 1, Uncertain = 4, Very likely = 7.
Comparisons of ratings between participants with and without a medical education resulted in some divergence. Respondents without an MD predicted that, by 2029 as a result of AI/ML, patient access to medical care in the US will lag behind other developed countries; participants with an MD did not reach consensus on this item (see Table 4).
Empathic care of patients.
Experts envisaged that by 2029 in the US, the availability of AI/ML tools will help to augment levels of empathic care (see Fig 5).
Note. For items 16, 17, 18 the scale used was: Greatly decrease = 1, Remain the same = 4, Greatly increase = 7. For items 20, 21, 22, 23: Very unlikely = 1, Uncertain = 4, Very likely = 7.
Panelists were divided on whether, by 2029, AI will offer direct resources for delivering empathic care to patients (see Table 4). Participants with an MD considered this unlikely, while others failed to reach consensus on this item.
Summary of major findings
The collective forecasts of medical informaticians have been missing from discussions about how AL/ML will influence the short-term future of primary care (see Box 1). In this Delphi poll there was consensus that in the next decade in the US, AI/ML will engender training and primary care work forces changes, improve rates of diagnostic accuracy, and increase access to primary care.
Economists forecast that in the coming decades, AI/ML will revolutionize the workplace [35, 36]. Taking a long view, informaticians in this Delphi poll predicted that 100 years from now it is unlikely that primary care doctors will be obsolete. Panelists further envisaged that primary care will be one of the last medical specialties to be displaced by technology. However, in the short term, by 2029, our experts did foresee workforce and training changes in US primary care as a result of AI/ML. Experts were collectively uncertain about whether AI/ML tools would enable lower-level clinicians to do higher level jobs, though it was not clear whether this prediction was driven by technological or regulatory considerations. Panelists anticipated a shift towards computing and engineering in the educational background of students entering medical school in 2029, and increased training demands on medical students to work with AI/ML in healthcare. However, the survey did not reveal whether experts perceived there to be risks of physicians using ML/AL tools without a computing and engineering background, or indeed, an ethics or evidence-based perspective, on these techniques. The panel’s predictions on education trends should also be observed against the currently limited debate about the need for curricular changes in medical education [37–39].
For many reasons, including financial, social, and geographical, timely access to primary care in the US remains a considerable problem. Compounding matters, with fewer medical students entering primary care, inefficiencies, and demographic changes–an ageing population, and more people suffering from chronic conditions for longer–it is widely envisaged that ambulatory medicine will become increasingly strained [40, 41]. The results of this Delphi poll suggest that AI/ML tools may help to address some of these challenges. Experts envisaged that by 2029 there would be increased access to care via AI/ML-enabled tools for medical triage and routine patient self-diagnosis, and with the growth of telemedicine.
The panel also predicted increasing medical precision. By 2029, experts envisaged that the use of AI/ML-enabled tools among patients will help to reduce diagnostic errors both for diseases with homogenous symptoms, and for more difficult medical cases. Perhaps contributing to these reductions, experts anticipated that advancements in AI/ML will engender revisions in disease classifications. These positive predictions should be viewed against current evidence that diagnostic error is both common and harmful. In the US, recent estimates suggest a diagnostic error rate of 13–15% affecting the lives of around 12 million Americans annually, contributing to 10% of all deaths, and the highest proportion of medical malpractice claims [42–44]. Patients from racial and ethnic minorities, and those on low-incomes, are at higher risk of diagnostic error . Our experts predicted that diagnostic accuracy will increase for individuals with limited access to care, minorities, or patients with rare conditions. While there are currently considerable concerns about the potential for algorithmic biases to be baked into AI/ML tools, driven in part by the underrepresentation of underprivileged demographic groups in training phases of machine learning , there was consensus among our Delphi panel that data collection in 2029 will be more representative of minority populations. This prediction may help to explain why the panel anticipated improved diagnostic accuracy for minorities.
Nonetheless, experts were less optimistic that AI/ML will narrow health disparities in the US by 2029. Current findings point to a “digital divide” in healthcare. Many factors drive current differential usage of digital health innovations including costs, lack of broadband access, and lower levels of digital and health literacy among underprivileged populations [47, 48]. Research also suggests that in the US, health app usage is more common among people who are younger, better educated, on a higher income, or in better health . Our panel predicted that US healthcare will become increasingly productized. Although the poll provided no causal explanations for this prediction, in a growing health app economy, experts may have anticipated that disadvantaged patients will continue to be less likely to adopt eHealth tools. In addition, there was consensus that private hospitals will have greater access to AI/ML-enabled resources to improve diagnostic accuracy than public hospitals. Existing structural disparities in care may also have been perceived to be a factor that will perpetuate inequities in eHealth.
Our Delphi poll provided nuanced forecasts on the theme of physician empathy. There was collective consensus that, by 2029, AI/ML would not free up more time with patients in US primary care; however, experts did forecast that levels of empathy in primary care would increase in this time period. The survey did not fully illuminate the reasons for this but there was consensus that AI/ML-enabled tools will assist physicians in shared decision-making, and help provide information on patients’ lifestyles and the social determinants of individuals’ health. Conceivably, the panel may have envisaged that such data might enhance physicians’ personal knowledge about patients thereby fostering more empathic care. Again, these views appear to differ subtly from those of physicians. In qualitative research a common prediction among physicians is that, by liberating health professionals from administrative tasks, AI/ML will indirectly facilitate more time with patients thereby enhancing levels of empathy [11, 15] Survey research also indicates skepticism among physicians that AI/ML will be able to directly substitute for, or augment clinicians, in the provision of empathic care [10–12, 15].
In terms of physicians’ responsibilities, experts did not envisage that AI/ML will help to reduce documentation burdens by 2029 [10, 12]. This prediction contrasts with the more optimistic opinions of surveyed physicians. For example, in 2019, a global survey of psychiatrists found that the majority (83%, 657/791) judged it likely that future technology will fully replace physicians in the task of documentation with 84% (552/657) of these respondents predicting that this will happen in the next 10 years . Similarly, in 2018, survey research conducted among primary care physicians in the UK revealed comparable results: most UK general practitioners (80%, 578/720) anticipated that future technology will fully replace humans in the task of documentation with 79% (458/578) of these respondents believing that this will happen in the next decade .
Finally, experts in this Delphi poll did not weigh in on specific policy, legal, or ethical issues in relation to the impact of AI/ML on primary care. However, there was consensus that by 2029 regulatory issues will pose greater challenges than technical problems.
Strengths and limitations
To our knowledge this is the first Delphi poll to explore experts’ predictions about the short-term effects of AI/ML on a medical specialism. Major strengths of the survey were the high response rates between rounds, and the diversity of participants. Although only around one third of Round 3 panelists were female (31%), currently around 25% of health IT leaders in the US are women . The expert panel comprised leading health informaticians around half of whom also had a medical background. Panelists were drawn from diverse backgrounds, nationalities, and ethnicities including 3 participants in Round 1 who do not reside in the US but who are knowledgeable about the US healthcare system. We also note that the majority of experts primarily held allegiances to academia, and medicine, rather than industry; nonetheless, this may have been a strength rather than a limitation, resulting in more modest predictions.
This survey has several limitations. As with all Delphi polls, there is no guarantee of accuracy in forecasts. No standardized guidelines exist for identifying, excluding, or selecting suitable experts from the field of interest [23, 27]. Reliability of predictions is dependent on the specialist knowledge of the participants which can be influenced by norms and values, motivational biases, and stakeholder interests [51, 52]. Although there was strong consensus among our panel of experts, we noted some divergence in opinions between participants with and without a medical degree. Conceivably, professional medical allegiances may have affected predictions; overall, however, we cannot speculate on how the composition of our panel strengthened or diminished the quality of predictions. Whilst participant retention rates between rounds were high, the number of panelists was limited, and more participants in the first round may have resulted in different consensus opinions [53, 54].
Importantly, two events arising in the immediate period after data collection–one global and one in the US context–may affect the reliability of the Delphi poll. The coronavirus pandemic has (and currently is) exerting a significant impact on the delivery of primary care in the US. Driven by this crisis, current evidence shows a substantial uptick in demand for telemedicine consultations, and in the use of AI/ML-driven triage tools [55, 56]. Although it is too early to predict with certainty whether increase in these applications will persist after the pressure on frontline medicine has abated, it seems possible that our experts’ forecasts on the influence of AI/ML on access to care may be especially well supported.
Second, the survey was administered prior to the finalized ruling, in March 2020, by the National Coordinator of Health Information Technology (ONC) on the 21st Century Cures Act . Designed to maximize innovation in healthcare by creating a competitive health app economy, this federal ruling sets out technical standards about how data must be shared, mandating patients’ right to access their digital medical records. While the final ruling may have been anticipated by some of our experts in the months preceding the announcement, we cannot be certain about whether or how its publication might otherwise have influenced consensus predictions of our participants. However, we suggest that uncertainty prior to the ruling may have fostered more cautious predictions about the impact of AI/ML on primary care among our experts.
A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be.
This Delphi poll provides the consensus predictions of leading health informaticians on the impact of AI/ML on primary care in the US. The panel forecast that, in the long-term (100 years from now) primary care doctors will not be obsolete, and furthermore, that general medicine will be one of the last medical specialties to be displaced by technology. By 2029 in the US, however, experts did forecast that AI/ML will exert an impact on the delivery and quality of primary care. Specifically, the panel predicted increased rates of diagnostic accuracy including for the most disadvantaged patient populations, greater access to primary care, and enhanced levels of empathic patient care. Against the panel’s forecast that healthcare in the US would be increasingly productized, there was consensus that regulatory issues will pose greater challenges than technical ones in improving diagnostic accuracy. Experts were also less optimistic about the prospects of AI/ML to precipitate other desirable short-term changes in medicine. By 2029 in the US, the panel predicted that AI/ML would not narrow healthcare disparities, reduce documentation burdens on primary care physicians, or increase the total time spent with patients. In the next decade, experts forecast increased AI/ML training requirements for medical students.
The central goal of Delphi polls is expert prediction. However, forecasts can also help us to exert control over the future by facilitating forward planning, and focusing attention on where, and how, relevant actors might intervene to create preferable outcomes. Innovations in digital care pose myriad practical, ethical, and regulatory issues including (but by no means limited to): the creations of standards for assessing the reliability and approval of medical algorithms and apps, questions about patient privacy, and the security of patients’ online health information [58, 59]. In reviewing these findings we are struck by the contrastive predictions of our experts with those of surveyed physicians [10, 11]. As others have noted, medical schools have been slow to adapt curricula and offer courses aimed at promoting AI/ML literacy among students [37–39]. We conclude that to empower both physicians and patients, and to rise to the challenges of the next decade, it is incumbent on the medical community, health and medical educators, and policy-makers to take action to improve digital literacy both among patients and our current and future health professionals .
It is standard practice in Delphi polls to acknowledge the panel of experts who shared their valuable insights. We are indebted to them for giving up their time. The following participants (12 out of 16) granted us permission to report their names: David Bates, MD; Andrew Beam, PhD; Gabriel Brat, MD; Wendy Chapman, PhD; Enrico Coiera, PhD FACMI; Peter Embi, MD, MS, FACP, FACMI, FAMIA; John Halamka, MD; Arjun Manrai, PhD; JP Onella, PhD; Neil Sebire, MBBS, BClinSCi, MD, FRCOG, FRCPath, FFCI; Jessie Tenenbaum, PhD; Kun-Hsing Yu, MD PhD.
- 1. Coiera E. The fate of medicine in the time of AI. The Lancet. 2018;
- 2. Kohane IS, Drazen JM, Campion EW. A glimpse of the next 100 years in medicine. Mass Medical Soc; 2012.
- 3. Steinhubl SR, Kim K, Ajayi T, Topol EJ. Virtual care for improved global health. The Lancet. 2018;391(10119):419.
- 4. Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? Jama. 2013;310(22):2395–2396. pmid:24158428
- 5. Hinton G. Deep learning—a technology with the potential to transform health care. JAMA. 2018;320(11):1101–1102. pmid:30178065
- 6. Darcy AM, Louie AK, Roberts LW. Machine learning and the profession of medicine. Jama. 2016;315(6):551–552. pmid:26864406
- 7. Mandl KD, Bourgeois FT. The Evolution of patient diagnosis: From art to digital data-driven science. Jama. 2017;318(19):1859–1860. pmid:29075757
- 8. Goldhahn J, Rampton V, Spinas GA. Could artificial intelligence make doctors obsolete? Bmj. 2018;363:k4563. pmid:30404897
- 9. Lin SY, Mahoney MR, Sinsky CA. Ten ways artificial intelligence will transform primary care. Journal of general internal medicine. 2019;34(8):1626–1630. pmid:31090027
- 10. Blease C, Bernstein MH, Gaab J, Kaptchuk TJ, Kossowsky J, Mandl KD, et al. Computerization and the future of primary care: A survey of general practitioners in the UK. PloS one. 2018;13(12):e0207418. pmid:30540791
- 11. Blease C, Kaptchuk TJ, Bernstein MH, Mandl KD, Halamka JD, DesRoches CM. Artificial Intelligence and the Future of Primary Care: Exploratory Qualitative Study of UK General Practitioners’ Views. Journal of medical Internet research. 2019;21(3):e12802. pmid:30892270
- 12. Doraiswamy PM, Blease C, Bodner K. Artificial intelligence and the future of psychiatry: Insights from a global physician survey. Artificial Intelligence in Medicine. 2020;102:101753. pmid:31980092
- 13. Dos Santos DP, Giese D, Brodehl S, Chon SH, Staab W, Kleinert R, et al. Medical students’ attitude towards artificial intelligence: a multicentre survey. European radiology. 2018;1–7.
- 14. Sit C, Srinivasan R, Amlani A, Muthuswamy K, Azam A, Monzon L, et al. Attitudes and perceptions of UK medical students towards artificial intelligence and radiology: a multicentre survey. Insights into Imaging. 2020;11(1):14.
- 15. Blease C, Locher C, Leon-Carlyle M, Doraiswamy PM. Artificial Intelligence and the Future of Psychiatry: Qualitative Findings from a Global Physician Survey. arXiv preprint arXiv:191009956. 2019;
- 16. Boeldt DL, Wineinger NE, Waalen J, Gollamudi S, Grossberg A, Steinhubl SR, et al. How consumers and physicians view new medical technology: comparative survey. Journal of medical Internet research. 2015;17(9).
- 17. Liyanage H, Liaw S-T, Jonnagaddala J, Schreiber R, Kuziemsky C, Terry AL, et al. Artificial Intelligence in Primary Health Care: Perceptions, Issues, and Challenges: Primary Health Care Informatics Working Group Contribution to the Yearbook of Medical Informatics 2019. Yearbook of medical informatics. 2019;28(1):41. pmid:31022751
- 18. Ückert F, Ammenwerth E, Dujat C, Grant A, Haux R, Hein A, et al. Past and next 10 years of medical informatics. Journal of medical systems. 2014;38(7):74. pmid:24952607
- 19. Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Practical assessment, research & evaluation. 2007;12(10):1–8.
- 20. Dalkey N. An experimental study of group opinion: the Delphi method. Futures. 1969;1(5):408–426.
- 21. Jiang R, Kleer R, Piller FT. Predicting the future of additive manufacturing: A Delphi study on economic and societal implications of 3D printing for 2030. Technological Forecasting and Social Change. 2017;117:84–97.
- 22. Linstone HA, Turoff M. The Delphi Method: Techniques and Applications, AddisonWesley. Reading, MA. 1975;
- 23. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing. 2000;32(4):1008–1015. pmid:11095242
- 24. Rubinson L, Neutens JJ. Research techniques for the health sciences. MacMillan New York; 1987.
- 25. Parente R, Anderson-Parente J. A case study of long-term Delphi accuracy. Technological Forecasting and Social Change. 2011;78(9):1705–1711.
- 26. Norcross JC, Pfund RA, Prochaska JO. Psychotherapy in 2022: A Delphi poll on its future. Professional Psychology: Research and Practice. 2013;44(5):363.
- 27. Donohoe H, Stellefson M, Tennant B. Advantages and limitations of the e-Delphi technique: Implications for health education researchers. American Journal of Health Education. 2012;43(1):38–46.
- 28. Custer RL, Scarcella JA, Stewart BR. The Modified Delphi Technique–A Rotational Modification. Journal of vocational and technical education. 1999;15(2):50–58.
- 29. Powell C. The Delphi technique: myths and realities. Journal of advanced nursing. 2003;41(4):376–382. pmid:12581103
- 30. Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning: A guide to nominal group and Delphi processes. Scott Foresman; 1975.
- 31. Buck AJ, Gross M, Hakim S, Weinblatt J. Using the Delphi process to analyze social policy implementation: A post hoc case from vocational rehabilitation. Policy Sciences. 1993;26(4):271–288.
- 32. Rowe G, Wright G, McColl A. Judgment change during Delphi-like procedures: The role of majority influence, expertise, and confidence. Technological Forecasting and Social Change. 2005;72(4):377–399.
- 33. Yaniv I. Group diversity and decision quality: amplification and attenuation of the framing effect. International Journal of Forecasting. 2011;27(1):41–49.
- 34. Ludwig B. Predicting the future: Have you considered using the Delphi methodology. Journal of extension. 1997;35(5):1–4.
- 35. Susskind R, Susskind D. Technology will replace many doctors, lawyers, and other professionals. Harvard Business Review. 2016;11.
- 36. Frey C, Osborne M. The future of employment: how suspectible are jobs to computerization? Technological Forecasting and Social Change. 2017 Jan;(114):254–80.
- 37. Kolachalama VB, Garg PS. Machine learning and medical education. NPJ digital medicine. 2018;1(1):1–3.
- 38. Giunti G, Guisado-Fernandez E, Belani H, Lacalle-Remigio JR. Mapping the Access of Future Doctors to Health Information Technologies Training in the European Union: Cross-Sectional Descriptive Study. Journal of medical Internet research. 2019;21(8):e14086. pmid:31407668
- 39. Paranjape K, Schinkel M, Panday RN, Car J, Nanayakkara P. Introducing artificial intelligence training in medical education. JMIR Medical Education. 2019;5(2):e16048. pmid:31793895
- 40. Davis K, Stremikis K, Squires D, Schoen C. Mirror, Mirror on the Wall, 2014 Update: How the U.S. Health Care System Compares Internationally [Internet]. The Commonwealth Fund. 2014 [cited 2020 Jan 23]. Available from: https://www.commonwealthfund.org/publications/fund-reports/2014/jun/mirror-mirror-wall-2014-update-how-us-health-care-system
- 41. Bodenheimer T, Bauer L. Rethinking the primary care workforce—an expanded role for nurses. N Engl J Med. 2016;375(11):1015–7. pmid:27626516
- 42. Tehrani ASS, Lee H, Mathews SC, Shore A, Makary MA, Pronovost PJ, et al. 25-Year summary of US malpractice claims for diagnostic errors 1986–2010: an analysis from the National Practitioner Data Bank. BMJ quality & safety. 2013;22(8):672–680.
- 43. Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf. 2013;22(Suppl 2):ii21–ii27.
- 44. Newman-Toker DE, Schaffer AC, Yu-Moe CW, Nassery N, Tehrani ASS, Clemens GD, et al. Serious misdiagnosis-related harms in malpractice claims: The “Big Three”–vascular events, infections, and cancers. Diagnosis. 2019;6(3):227–240. pmid:31535832
- 45. Staff I of M. Unequal treatment: Confronting racial and ethnic disparities in healthcare. National Academies Press; 2004.
- 46. Manrai AK, Funke BH, Rehm HL, Olesen MS, Maron BA, Szolovits P, et al. Genetic misdiagnoses and the potential for health disparities. New England Journal of Medicine. 2016;375(7):655–665. pmid:27532831
- 47. Rodriguez JA, Lipsitz SR, Lyles CR, Samal L. Association Between Patient Portal Use and Broadband Access: a National Evaluation. Journal of General Internal Medicine. 2020;1–2.
- 48. Lin SC, Lyles CR, Sarkar U, Adler-Milstein J. Are Patients Electronically Accessing Their Medical Records? Evidence From National Hospital Data. Health Affairs. 2019;38(11):1850–1857. pmid:31682494
- 49. Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals and other eHealth technologies based on race, ethnicity, and age: a database and survey study of seniors in a large health plan. Journal of medical Internet research. 2016;18(3):e50. pmid:26944212
- 50. Morse S. What 3 charts say about women in health IT leadership roles, salary and the rise of chief nursing officers [Internet]. Healthcare IT News. 2018 [cited 2020 Mar 8]. Available from: https://www.healthcareitnews.com/news/what-3-charts-say-about-women-health-it-leadership-roles-salary-and-rise-chief-nursing-officers
- 51. Burgman MA. Trusting judgements: how to get the best out of experts. Cambridge University Press; 2016.
- 52. Collins H, Evans R. Rethinking expertise. University of Chicago Press; 2008.
- 53. Stirling A. A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface. 2007;4(15):707–719.
- 54. Woolley AW, Chabris CF, Pentland A, Hashmi N, Malone TW. Evidence for a collective intelligence factor in the performance of human groups. science. 2010;330(6004):686–688. pmid:20929725
- 55. Armour S, McKinnon JD. Telemedicine Gets a Boost From Coronavirus Pandemic: Medicare patients get more flexibility in seeking remote treatment. The Wall Street Journal [Internet]. 2020 Mar 17 [cited 2020 Mar 20]; Available from: https://www.wsj.com/articles/trump-administration-expands-medicare-telehealth-program-11584461973
- 56. Mann DM, Chen J, Chunara R, Testa PA, Nov O. COVID-19 transforms health care through telemedicine: evidence from the field. Journal of the American Medical Informatics Association. 2020;
- 57. Office of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services (HHS). 21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program: Final Rule [Internet]. 2020 [cited 2020 Mar 15]. Available from: https://www.healthit.gov/cerus/sites/cerus/files/2020-03/ONC_Cures_Act_Final_Rule_03092020.pdf
- 58. Vayena E, Blasimme A, Cohen IG. Machine learning in medicine: Addressing ethical challenges. PLoS medicine. 2018;15(11):e1002689. pmid:30399149
- 59. Zuboff S. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power: Barack Obama’s Books of 2019. Profile Books; 2019.