Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Artificial intelligence-assisted clinical decision support for childhood asthma management: A randomized clinical trial

  • Hee Yun Seol ,

    Contributed equally to this work with: Hee Yun Seol, Pragya Shrestha

    Roles Conceptualization, Data curation, Formal analysis, Writing – original draft, Writing – review & editing

    Affiliations Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America, Department of Internal Medicine, Pusan National University Yangsan Hospital, Yangsan, Korea

  • Pragya Shrestha ,

    Contributed equally to this work with: Hee Yun Seol, Pragya Shrestha

    Roles Data curation, Writing – review & editing

    Affiliations Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America, Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  • Joy Fladager Muth,

    Roles Conceptualization, Data curation, Formal analysis, Writing – original draft, Writing – review & editing

    Affiliation Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  • Chung-Il Wi,

    Roles Conceptualization, Data curation, Formal analysis, Writing – original draft, Writing – review & editing

    Affiliations Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America, Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  • Sunghwan Sohn,

    Roles Data curation, Formal analysis, Writing – review & editing

    Affiliation Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, United States of America

  • Euijung Ryu,

    Roles Conceptualization, Data curation, Supervision, Writing – review & editing

    Affiliation Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota, United States of America

  • Miguel Park,

    Roles Conceptualization, Writing – review & editing

    Affiliation Division of Allergic Diseases, Mayo Clinic, Rochester, Minnesota, United States of America

  • Kathy Ihrke,

    Roles Data curation, Writing – original draft, Writing – review & editing

    Affiliations Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America, Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  • Sungrim Moon,

    Roles Conceptualization, Data curation, Formal analysis, Writing – review & editing

    Affiliation Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, United States of America

  • Katherine King,

    Roles Formal analysis, Writing – review & editing

    Affiliation Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota, United States of America

  • Philip Wheeler,

    Roles Conceptualization, Data curation, Formal analysis, Writing – review & editing

    Affiliation Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America

  • Bijan Borah,

    Roles Formal analysis, Writing – review & editing

    Affiliation Department of Health Service Research, Mayo Clinic, Rochester, Minnesota, United States of America

  • James Moriarty,

    Roles Data curation, Formal analysis, Writing – review & editing

    Affiliation Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, United States of America

  • Jordan Rosedahl,

    Roles Formal analysis, Writing – review & editing

    Affiliation Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, United States of America

  • Hongfang Liu,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, United States of America

  • Deborah B. McWilliams,

    Roles Conceptualization, Data curation, Writing – review & editing

    Affiliation Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  •  [ ... ],
  • Young J. Juhn

    Roles Conceptualization, Data curation, Writing – review & editing

    juhn.young@mayo.edu

    Affiliations Precision Population Science Lab, Mayo Clinic, Rochester, Minnesota, United States of America, Department of Pediatric and Adolescent Medicine, Mayo Clinic, Rochester, Minnesota, United States of America

  • [ view all ]
  • [ view less ]

Abstract

Rationale

Clinical decision support (CDS) tools leveraging electronic health records (EHRs) have been an approach for addressing challenges in asthma care but remain under-studied through clinical trials.

Objectives

To assess the effectiveness and efficiency of Asthma-Guidance and Prediction System (A-GPS), an Artificial Intelligence (AI)-assisted CDS tool, in optimizing asthma management through a randomized clinical trial (RCT).

Methods

This was a single-center pragmatic RCT with a stratified randomization design conducted for one year in the primary care pediatric practice of the Mayo Clinic, MN. Children (<18 years) diagnosed with asthma receiving care at the study site were enrolled along with their 42 primary care providers. Study subjects were stratified into three strata (based on asthma severity, asthma care status, and asthma diagnosis) and were blinded to the assigned groups.

Measurements

Intervention was a quarterly A-GPS report to clinicians including relevant clinical information for asthma management from EHRs and machine learning-based prediction for risk of asthma exacerbation (AE). Primary endpoint was the occurrence of AE within 1 year and secondary outcomes included time required for clinicians to review EHRs for asthma management.

Main results

Out of 555 participants invited to the study, 184 consented for the study and were randomized (90 in intervention and 94 in control group). Median age of 184 participants was 8.5 years. While the proportion of children with AE in both groups decreased from the baseline (P = 0.042), there was no difference in AE frequency between the two groups (12% for the intervention group vs. 15% for the control group, Odds Ratio: 0.82; 95%CI 0.374–1.96; P = 0.626) during the study period. For the secondary end points, A-GPS intervention, however, significantly reduced time for reviewing EHRs for asthma management of each participant (median: 3.5 min, IQR: 2–5), compared to usual care without A-GPS (median: 11.3 min, IQR: 6.3–15); p<0.001). Mean health care costs with 95%CI of children during the trial (compared to before the trial) in the intervention group were lower than those in the control group (-$1,036 [-$2177, $44] for the intervention group vs. +$80 [-$841, $1000] for the control group), though there was no significant difference (p = 0.12). Among those who experienced the first AE during the study period (n = 25), those in the intervention group had timelier follow up by the clinical care team compared to those in the control group but no significant difference was found (HR = 1.93; 95% CI: 0.82–1.45, P = 0.10). There was no difference in the proportion of duration when patients had well-controlled asthma during the study period between the intervention and the control groups.

Conclusions

While A-GPS-based intervention showed similar reduction in AE events to usual care, it might reduce clinicians’ burden for EHRs review resulting in efficient asthma management. A larger RCT is needed for further studying the findings.

Trial registration

ClinicalTrials.gov Identifier: NCT02865967.

Introduction

Growing deployments of Electronic Health Records (EHRs) systems have established large practice-based longitudinal patient records causing an increase in the volume of unstructured data (80%) in the currently available health care records [1]. Inefficient and ineffective use of EHRs due to overwhelming volume has led to physician burnout (70% of clinicians reported health information technology [HIT]-related stress) due to increased workload during their limited (e.g.,15–20 min) clinic visit [25]. The application of Artificial Intelligence (AI) to health care may potentially address these challenges through AI-assisted clinical decision support (CDS) tools. The National Academy of Medicine suggested delivering high-value care in the personal and social context through science and technology as one of the key directions for the US health care system [6].

The major challenges for better asthma care during the EHRs era is the lack of efficient and effective CDS meaningfully supporting clinicians and their care teams leading to high-value asthma care improving care quality and outcomes while reducing the costs [79]. Not surprisingly, some CDS even increased the costs and time for asthma care at the clinic due to providing more services (eg, almost twice as many lasted >1 hour after implementation of CDS) [8]. At present, no augmented AI-assisted CDS tools for streamlining childhood asthma management are available which fully leverage technologies harnessing EHRs. While many AI algorithms have been developed [1012] and even approved for Software as Medical Device (SaMD) by the FDA [12, 13], few AI algorithms have been tested and shown to have an actual improvement in health outcomes in a randomized clinical trial (RCT) [14, 15].

We developed the Asthma-Guidance and Prediction System (A-GPS), an AI-assisted CDS tool providing 1) a high-level summary of relevant clinical information (eg, asthma care quality, risk factors and outcomes) for each asthmatic patient, 2) machine-learning-based predictive analytics for future asthma exacerbation (AE) and 3) asthma management options to help clinicians make efficient and effective clinical decision-making for optimal asthma management. Here we report results of an exploratory pragmatic RCT that assessed the effectiveness and efficiency of intervention via A-GPS on pertinent asthma outcomes in a real-world primary care setting.

Methods

Study design and subjects

The study was designed as a single-center pragmatic RCT with a stratified randomization design for one year (Dec 13, 2016, to Dec 12, 2017) which assessed the effectiveness of A-GPS on asthma outcomes (Figs 1 and 2). This study was registered at the Clinical Trial Registration (NCT02865967). This study (IRB number:15–004435) was approved by the Mayo Clinic Institutional Review Board (IRB) on July 8, 2016.

thumbnail
Fig 1. Study design for assessing the comparative effectiveness of A-GPS intervention via an RCT with stratified randomization (see S1 File. The Design and Implementation of Asthma-Guidance and Prediction System (A-GPS) for the definition for each stratum for details).

https://doi.org/10.1371/journal.pone.0255261.g001

thumbnail
Fig 2. A-GPS report for individual patients provided to Primary Care Providers (PCPs) quarterly.

https://doi.org/10.1371/journal.pone.0255261.g002

Study subjects were stratified into three strata based on asthma severity, asthma care status, and asthma diagnosis.

Briefly, Stratum 1 is comprised of children with persistent asthma (clinically defined by the Healthcare Effectiveness Data and Information Set (HEDIS) [16] and/or the National Asthma Education and Prevention Program (NAEPP) guideline [17] [see S1 File for details]), enrolled in an Asthma Management Program (AMP), a care coordination program for asthma. Stratum 2 was a group of children with persistent asthma defined but not enrolled in AMP. Lastly, given the significant number (24%) of children without a diagnosis of asthma despite the recurrent asthma-like symptoms fulfilling asthma criteria (a delayed diagnosis of asthma) [1821], Stratum 3 included children with recurrent asthma-like symptoms who met Predetermined Asthma Criteria (PAC) in this pragmatic clinical trial which is summarized in S1 File. Predetermined Asthma Criteria developed by Drs. John Yunginger and Charles Reed, renown asthma researchers at Mayo Clinic is conceptually similar to the 2015 Canadian Thoracic and Canadian Pediatric Society asthma criteria (1. documented airflow obstruction, 2. Documented reversibility of airflow obstruction and 3. no evidence of an alternative diagnosis) based on patients’ EHRs but did not have a diagnosis of asthma yet [22, 23]. To our knowledge, PAC was the only existing predetermined criteria for asthma that determines asthma status and the index date of incident asthma retrospectively based on medical records using AI algorithm at the time of our study [2426]. PAC was found to have high reliability, and extensive epidemiologic work for asthma has used PAC showing the excellent construct validity in identifying known risk factors for asthma and asthma-related adverse outcomes (e.g., serious and common infections) [2739].

Study population and setting

Olmsted County, southeastern Minnesota, is a virtually self-contained health care environment (only two health care systems provide clinical care to nearly all Olmsted County, MN residents), and 98% of residents authorize their medical records to be used for research [40]. According to U.S. census data in 2010, the age, sex, and ethnic characteristics of Olmsted County residents were similar to those of the state of Minnesota and the Upper Midwest [41]. However, Olmsted County has been becoming more diverse as indicated by the racial and ethnic characteristics of children enrolled in public schools (In 2019, 35.2% reported to be a non-white). The prevalence of asthma in a population of school-age children in our study setting, Olmsted County, MN, has been reported to be 17.6% [42]. Asthma is the most prevalent chronic illness with the third highest health care expenditures in children and adolescents in Olmsted County, MN [43]. Mayo Clinic Primary Care Pediatric Practices offer primary care service at four locations within Olmsted County, and this study was conducted in the primary care practice site (i.e., including teaching pediatric faculty, residents, and nurse practitioners).

Study intervention

Subjects in each stratum were randomized to either A-GPS plus usual asthma care (intervention group) or usual asthma care only (control group) using computer-generated randomization and 1:1 assignment. The detailed development and implementation of A-GPS as intervention are described in S1 File The Design and Implementation of Asthma-Guidance and Prediction System (A-GPS). Also, we report all relevant data collected, and their definitions and data source in S1 File. Briefly, A- GPS report for each patient was provided to primary care providers (PCP) including pediatric residents and nurse practitioners at pre-determined dates every three months for PCP to review updated asthma status of each patient of theirs and determine whether the asthma management plan needed to be revised. Thus, PCP were not blinded to the intervention. A-GPS report included 1) summary of relevant information (quality of care, risk factors and outcomes) for asthma management (see 32 variables and their definitions and data sources listed in S1 File), 2) a machine learning (ML) algorithm (Bayesian classifier) forecasting the risk of AE in one year based on EHR data in the past 3 years (see S1 File for developing our ML algorithm for details), and 3) asthma management plans as shown in Fig 2. A-GPS report was generated by data mining tools including language processing (NLP) algorithms and ML algorithm based on preceding clinical encounters, patient-reported outcomes (eg, ACT score), and non-clinical data (eg, traffic volume and socioeconomic status) (see S1 File The Design and Implementation of Asthma-Guidance and Prediction System (A-GPS)).

After PCP reviewed the one-page updated A-GPS report every three months, they were offered multiple options to revise the current asthma management plans and informed the management plans (if they decided to change) to asthma care manager who carried out the revised management plans for each patient. The control group received usual asthma care without A-GPS report. As part of usual asthma care, we allowed parents to opt in and out of AMP, and AMP coordinators reached out and encouraged parents of persistent asthmatic children (e.g., Stratum 2) to be enrolled in AMP to offer care coordination for asthma to patients and their parents. Children and families were blinded to the study groups (i.e., intervention vs. control), but there was no deviation from usual asthma care described above. Inclusion criteria were as follows: children 1) under age 18 living in Olmsted County, 2) who had received medical care from pediatric primary care practice, Mayo Clinic, Rochester, between 2013 and 2016 and 3) research authorization using medical record for research. We enrolled the eligible children and invited their 42 PCP (teaching faculty and residents of pediatrics as well as nurse practitioners) to the study. All the PCP for eligible children participated in the study. Our study coordinators obtained and recorded the signed Health Insurance Portability and Accountability Act (HIPAA) form and the Mayo Clinic Institutional Review Board-approved verbal consent (See S1 File) from a parent or a guardian for study participation for their children given the nature of this minimal risk study.

Endpoints

1. Primary endpoint (effectiveness): Occurrence of AE within 1 year since the initiation of the study, defined by emergency department visits/hospitalization for asthma or unscheduled visits for asthma requiring oral corticosteroid [44].

2. Secondary endpoints:

  1. Clinicians’ burden for reviewing and collecting clinical data from EHRs for making a clinical decision (efficiency): We surveyed 42 participating PCP asking how many minutes per each patient were spent reviewing A-GPS report for decision making (intervention group) and how many minutes per each patient they estimated would be needed to collect and review data listed in A-GPS without A-GPS report for their clinical decision making for asthma management (control group). We surveyed the participated PCP for the burden of EHRs review at the end of the study due to two main reasons: it was difficult for clinicians to accurately time for the duration of sporadic EHRs review for asthma care for patients in the control group (usual care) as they often review EHRs at multiple time points in a day or even longer whereas clinicians could report time spent for reviewing A-GPS report in the intervention group at the quarterly report time resulting in performance bias in estimating the burden for chart review; studies assessing clinician’s burden for reviewing EHRs for asthma management were often based on survey derived from their perceived burden instead of objective measurement of burden itself [2, 45]. We also asked two open-ended questions including “What parts of the current asthma care will be improved by the asthma-GPS report?” and “Which components of the asthma-GPS report can be improved upon?” (see their comments in S1 File).
  2. Health care cost (efficiency): We defined health care cost in terms of direct medical expenses for each patient but did not include indirect costs such as patient’s out-of-pocket costs. Cost data were obtained from the Mayo Clinic Rochester Cost Data Warehouse (see S1 File for details). Detailed descriptions of this methodology have been published elsewhere [46].
  3. Asthma control status (effectiveness): As an additional asthma outcome measure, quarterly asthma control status was measured by administering Asthma Control Test (ACT), Childhood ACT [1, 2, 47, 48], or Test for Respiratory and Asthma Control in Kids (TRACK) over the phone or via online [49, 50] depending on age (as ACT and Childhood ACT were validated for only children aged 4 years or above). The proportion of time (i.e., 4-time points for one year) when patients had well-controlled asthma (ACT or Childhood ACT>19 or TRACK>80) during the study period was determined.
  4. Timeliness of asthma follow-up care after AE (efficiency): Any documented care for asthma either via clinic visit or by asthma care coordinator’s contact after AEs and time gap (in days) were retrieved as a measurement of asthma care quality.
  5. Enrollment in an AMP (effectiveness): Stratum 2 was a group of children with persistent asthma (i.e., eligible for AMP as long as the age was 5 years or above) but not enrolled in AMP. We assessed how many children of the intervention vs. control groups participated in AMP during the study period because enrollment in AMP represented care quality improvement effort in addition to timeliness of asthma follow-up care after AE.

Covariates.

We also collected pertinent variables on asthma outcomes, asthma care quality, and risk factors from EHRs which are described in S1 File.

Sample size estimation.

Our hypothesis was that A-GPS intervention reduces the frequency of asthma exacerbation as the primary end point greater than usual care for asthma. According to the literature, average asthma intervention (e.g. care coordination) reduced asthma exacerbation as the primary end point defined by hospitalization and ED visits related to asthma by 59.2% and by 50.6%, respectively in a one-year time frame [51]. If we assume that the baseline prevalence rates of hospitalization and ED visits were 30.4% and 63.6% respectively and that the rate of asthma exacerbation in the control group rate decreased by 10% (relative reduction), with 80% power and a two-sided alpha level of .05, we would need approximately 100 subjects each for the intervention group and control group to detect the difference in reduction of the frequency of asthma exacerbation between intervention and control group (usual care).

Randomization and blinding.

Once study subjects for each stratum were identified, children were assigned to intervention or comparison groups by computer (Excel, Microsoft) -generated sequential random numbers for all children in each stratum consecutively. Given the different nature of A-GPS and usual care and infeasibility of blinding, clinicians were not blinded to the assignment to intervention and control group (i.e., usual care).

Statistical analysis

All analyses were performed using the intention to treat methods. For our primary outcome, we used a stratified logistic model to assess the association of A-GPS intervention with exacerbation during the study period. Additionally, to determine if the intervention reduced the proportion of subjects with AE during the trial (12 months) compared to the pre-trial time (12 months) more so than the control group, a logistic regression model predicting AE using the sandwich variance estimator was fit including the two times frames, group, and the interaction of group and time was examined. The median time to care team’s follow-up after AE was estimated using Kaplan Meier method and the estimated effect was given from Cox proportional hazards model with subjects with no follow up censored at the study end date. We utilized ANCOVA to compare the average proportion of time (ie, continuous variable) a subject was considered to have well-controlled asthma over their total follow-up period between two groups (intervention vs. control) while adjusting for age. A paired Wilcoxon signed-rank test (was used to assess clinicians’ time to review EHR with or without A-GPS to collect data listed in A-GPS. Difference-in-difference (DID) analysis (difference in incremental health care cost [from baseline to the end of study] between intervention and control group) was carried out to assess the different costs between intervention and control patients using a generalized linear modeling framework with cost having gamma distribution and logarithmic link. DID analysis was conducted through a linear regression of costs on time indicator (pre vs. post intervention), the intervention indicator and their interaction, so that the estimated coefficient interaction term provides treatment effect. The underlying standard error is generated through bootstrap sampling with 100 repetitions.

Results

Basic characteristics of study subjects

Out of 555 who were invited to the study, 184 consented to participate in the study and were randomized (90 were assigned to the intervention group and 94 into the control group as shown in Fig 3).

thumbnail
Fig 3. Consort flow diagram for subject enrollment and random allocation of subjects in each stratum to intervention vs control group (see Fig 1 for randomization, intervention and follow up).

https://doi.org/10.1371/journal.pone.0255261.g003

The mean age at the time of enrollment was 9.0 years with 57% males and 72% Whites (Table 1). Overall, characteristics at baseline between the intervention group and the control group were not significantly different (Table 1).

thumbnail
Table 1. Baseline characteristics of study subjects between the intervention and the control.

https://doi.org/10.1371/journal.pone.0255261.t001

Primary endpoint

Because of the guidance that A-GPS provided, PCPs took a significantly higher number of actions (interventions) for intervention group (eg, 18 referrals to AMP (vs. 7 in control group), 5 referrals to community health workers (vs. 0), requests for 8 skin tests (vs. 5), 23 spirometry (vs, 12), 33 asthma-specific regular visits (vs. 4), and 28 ACT update (vs. 7)), of which were executed by asthma care team, compared to control group. The main results on the comparison in primary outcomes between the intervention and control groups are summarized in Fig 4 and Table 2.

thumbnail
Fig 4. Comparisons of the frequency of asthma exacerbation (the primary endpoint) between the intervention and control group (Fig 4A for overall intervention and control group and Fig 4B for individual strata) during the 12 months prior to the trial and the 12-month trial.

A. Comparison between overall intervention and control group. B. Comparison between intervention and control group at pre-trial and trial by stratum.

https://doi.org/10.1371/journal.pone.0255261.g004

thumbnail
Table 2. Primary and secondary outcomes in intervention and control groups.

https://doi.org/10.1371/journal.pone.0255261.t002

During the study period, overall, 14% (n = 25) of the subjects had at least one AE whereas 21% (n = 38) developed AE during one year prior to the trial. The proportion of children with AE in both groups decreased from the baseline (P = 0.042), specifically 17% → 12% in the intervention vs. 25% → 15% in the control group (Fig 4A). However, there was no difference in AE frequency between the two groups (12% for the intervention group vs. 15% for the control group, block stratified Odds Ratio: 0.82; 95%CI 0.34–1.96; P = 0.66) during the study period. Outcome comparisons among three strata as a post-hoc analysis done showed no statistical differences in the primary endpoints by each stratum (Fig 4B): control groups had a reduction in AE in strata 1 and 2 regardless of the availability of AMP and while there was no AE event in the intervention group, 4% of the control group had AE events in stratum 3.

Secondary endpoints

The results on secondary outcomes are summarized in Table 2. Forty-two primary care providers participated in this trial and 28 (67%) completed surveys. A-GPS significantly reduced clinicians’ burden for chart review to make clinical decision for asthma management by 67%, with an estimated median time to review patient’s medical records of 3.5 min (IQR: 2–5) with A-GPS intervention vs. 11.3 min (IQR: 6.3–15) without A-GPS (P<0.001). Average decrease within a person with A-GPS (vs. without A-GPS) was 7.3 min. This pattern was similarly found in residents (an average decrease of 7 min) and other PCP (8 min). The median overall self-reported satisfaction analog score for A-GPS was 7 (1: lowest, 10: highest) after the study was completed. Other comments are described in S1 File.

Mean health care costs with 95%CI of children during the trial (compared to before the trial) in the intervention group were lower than those in the control group (-$1,036 [-$2177, $44] for the intervention group vs. +$80 [-$841, $1000] for the control group; Table 2), though there was no significant difference (p = 0.12). These heath care costs did not take into account clinician’s time to review EHRs.

There was no difference in the proportion of duration when patients had well-controlled asthma during the study period between the intervention and the control groups as shown in Table 2. However, asthma control status at baseline was strongly associated with the asthma control status over follow up even after adjusting for age (age-adjusted difference = 0.26; P<0.001).

Among those who experienced AE during the study period (n = 25), those in the intervention group had timelier follow up by the clinical care team compared to those in the control group but no significant difference was found (HR = 1.93; 95% CI: 0.82–1.45, P = 0.10). Among children in Stratum 2, a total of 8 children in the intervention group were enrolled in AMP whereas none of the control group participated in AMP during the study period.

Adverse events

There were 0 serious adverse events resulting in death or breach of confidentiality in both intervention and control groups.

Discussion

To our knowledge, this exploratory pragmatic trial is the first RCT that assessed the comparative effectiveness of an AI-assisted CDS leveraging EHRs for asthma management in reducing AE. While A-GPS showed similar reduction in AE events to usual care, A-GPS reduced clinicians’ self-reported burden for reviewing EHRs by 67% compared to the control group and provided more actionable guidance for asthma management to PCPs in a way better addressing the needs of patients, compared to control group (usual care).

As both intervention and control groups showed a reduction of AE from the baseline, it reduced the comparative effectiveness of the intervention in our study. It is, however, difficult to disentangle the effect of A-GPS from other effects (e.g., ripple effect of intervention, effect of AMP or secular trend). We do not think these observations were due to the effect of AMP, a care coordination program for children with persistent asthma which was available only to stratum 1 because our study results showed a similar reduction of AE in the control group in stratum 2 (children with persistent asthma who were not enrolled in AMP). Also, secular trend of improvement of AE is unlikely as we observed reduction of AE in the intervention group but increase of AE in the control group in stratum 3, although it was not statistically significant. Thus, we postulate that as clinicians were not blinded to the study assignment, the intervention with A-GPS might reduce AE in both intervention and control groups as a ripple effect (beneficial effects of interventions in untreated control groups). This effect has been widely recognized in observational studies and clinical trials [5254]. However, given the limited RCT for AI or HIT-assisted interventions, the degree and nature of a ripple effect in adoption and implementation of HIT are poorly understood in the AI or HIT literature, although empirical observations for a ripple effect in HIT have been reported [55]. A few noteworthy aspects need to be considered for interpreting our study results. In stratum 2, there were only 3 patients out of 23 children (13%) in the intervention group who had a history of AE in the prior year before the study, compared to 11 of 30 children (37%) in the control group, suggesting a potential imbalance of covariates despite randomization which might impact the results. In addition, under usual asthma care practice, our study permitted study participants to opt in and out of AMP (depending on asthma control status assessed by clinicians and parental choice as long as the age was≥ 5 years) during the study period. As all 8 children, who were enrolled in AMP during the study period, were the intervention group and they were more likely to be children with poorly controlled asthma, this might be one of reasons which could reduce the effectiveness of the intervention in stratum 2.

A-GPS intervention might significantly reduce clinicians’ burden (estimated time for chart review and clinical decision making) by 67%, compared to the control group (3.5 minutes vs. 11.3 minutes, respectively). This benefit remained similar across PCP including pediatric residents. While the average reduced time for EHRs review (7.3 min) seems to be small, given the reported average duration for clinical review for EHRs (14.8 to 17 min) in the US and elsewhere [5], this effect size for reducing clinician’s burden for EHRs review for asthma management might represent a major benefit of A-GPS intervention achieving efficient and effective asthma management. Importantly, while A-GPS reduced clinicians’ burden for reviewing EHRs, it provided significantly more actionable guidance for asthma management to PCPs in a way better addressing the needs of patients, compared to control group (usual care). We believe this is a key benefit of A-GPS in asthma care.

In the context of health care costs, A-GPS intervention reduced direct health care costs from those for the prior year, compared to the control group but no significant difference was found (-$1,036 for the intervention group vs. +$80 for the control group, p = 0.12). However, we did not include the reduction of clinician’s time and effort in the cost-benefit analysis because it is not captured in billing data, which was used for estimating costs. Also, although statistically not significant, the intervention group had a timelier follow up after AE, compared to the control group (7 vs. 28 days, P = 0.10), given the importance of follow-up care for AE on asthma management, this timelier follow up in the intervention group in this study could represent an important benefit of A-GPS. Also, as discussed above, children with persistent asthma in the intervention group were significantly more likely to participate in AMP, compared to the control group. This care quality improvement might reflect that A-GPS helps PCP and care team better capture the unmet needs of families and improve access to AMP. Given our study as an under-powered exploratory study and the potential benefits, our study findings need to be assessed with studies with larger sample size.

Our study has several limitations. First, this study was a single-center single-blinded pragmatic trial, which limits its generalizability to other study settings and makes it difficult to minimize performance bias, which potentially resulted in a ripple effect as discussed above. Second, our study as an exploratory pragmatic trial fully integrated in a real-world primary care practice setting, was limited by a small sample size to be handled by Asthma Management Program (AMP) care coordinators of clinical practice and relatively lower AE incidence both in previous year and during the clinical trial despite our effort to enroll as many children with a history of AE as possible. Third, our study did not include lung function measures or medications in defining persistent asthma and study outcomes but included only clinical outcomes as a pragmatic trial. Especially, identification of Stratum 3 was based on PAC which has its own limitations in ascertaining asthma status in children. Fourth, the intervention was not synchronized with a clinical visit for asthma but prescheduled quarterly report for intervention which might have reduced the effectiveness of the intervention. Lastly, the population of Olmsted County, Minnesota is predominantly white (90%) and Scandinavian in ancestry, which may limit the generalizability of study results to other racial/ethnic groups. Also, our study has a few important strengths. First, our study included the existing AI tools in A-GPS report such as NLP algorithms for asthma ascertainment (NLP algorithms for PAC [2426] and Asthma Predictive Index [56]) who showed promising performance in ascertaining asthma status using EHRs on a large scale. Also, we incorporated clinically unavailable data such as individual-level SES as measured by validated HOUSES and traffic volume data know to trigger asthma exacerbation [57, 58] which provide clinicians with contextual information in asthma management. None of participating PCP and only 7% of subjects were censored (loss of follow up) before the completion of the study. Our study setting has an epidemiological advantage of being self-contained health care environment which is likely to capture all asthma-related events during the study period.

In conclusion, while A-GPS-based intervention showed similar reduction in AE events to usual care, it might have a potential to reduce clinicians’ burden for EHRs review resulting in more efficient asthma management while providing more actional guidance for asthma management. Cluster randomized trials with a larger sample size are needed to further assess the effectiveness and external validity of A-GPS.

Acknowledgments

We thank Mrs. Kelly Okeson for her administrative assistance. We would like to deeply thank all clinician participants, Asthma Management Program staff, and clinical care team for their time and efforts as their support and valuable insights were instrumental for the development, implementation, and execution of A-GPS in this study. Also, we would like to express our sincere gratitude to all parents and their children with asthma who participated in this study.

References

  1. 1. Martin-Sanchez F, Verspoor K: Big data in medicine is driving big changes. Yearb Med Inform 2014, 9(1):14–20.
  2. 2. Gardner RL, Cooper E, Haskell J, Harris DA, Poplau S, Kroth PJ, et al: Physician stress and burnout: the impact of health information technology. Journal of the American Medical Informatics Association: JAMIA 2019, 26(2):106–114. pmid:30517663
  3. 3. EHRs shoudl be a tool not a task [https://www.medicaleconomics.com/article/ehrs-should-be-tool-not-task]
  4. 4. Linzer M, Bitton A, Tu SP, Plews-Ogan M, Horowitz KR, Schwartz MD, et al: The End of the 15–20 Minute Primary Care Visit. Journal of general internal medicine 2015, 30(11):1584–1586. pmid:25900539
  5. 5. Holmgren AJ, Downing NL, Bates DW, Shanafelt TD, Milstein A, Sharp CD, et al: Assessment of Electronic Health Record Use Between US and Non-US Health Systems. JAMA Internal Medicine 2020.
  6. 6. Dzau VJ, McClellan MB, McGinnis JM, Burke SP, Coye MJ, Diaz A, et al: Vital Directions for Health and Health Care: Priorities From a National Academy of Medicine Initiative. JAMA 2017, 317(14):1461–1470. pmid:28324029
  7. 7. Bell LM, Grundmeier R, Localio R, Zorc J, Fiks AG, Zhang X, et al: Electronic health record-based decision support to improve asthma care: a cluster-randomized trial. Pediatrics 2010, 125(4):e770–777. pmid:20231191
  8. 8. Shiffman RN, Freudigman M, Brandt CA, Liaw Y, Navedo DD: A guideline implementation system using handheld computers for office management of asthma: effects on adherence and patient outcomes. Pediatrics 2000, 105(4 Pt 1):767–773. pmid:10742318
  9. 9. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE: Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Medical informatics and the Internet in medicine 2001, 26(3):191–201. pmid:11706929
  10. 10. Li A, Walling J, Ahn S, Kotliarov Y, Su Q, Quezado M, et al: Unsupervised analysis of transcriptomic profiles reveals six glioma subtypes. Cancer Res 2009, 69(5):2091–2099. pmid:19244127
  11. 11. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al: Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Jama 2016, 316(22):2402–2410. pmid:27898976
  12. 12. Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA: The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med 2018, 24(11):1716–1720. pmid:30349085
  13. 13. Center for Devices and Radiological Health FaDAF: Software as a Medical Device (SAMD): Clinical Evaluation. In. Edited by (FDA) FaDA, vol. FDA-2016-D-2483; 2020.
  14. 14. Angus DC: Randomized Clinical Trials of Artificial Intelligence. JAMA 2020, 323(11):1043–1045. pmid:32065828
  15. 15. Wijnberge M, Geerts BF, Hol L, Lemmers N, Mulder MP, Berge P, et al: Effect of a Machine Learning–Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial. JAMA 2020, 323(11):1052–1060. pmid:32065827
  16. 16. Assurance NCfQ: Technical Specifications for Health Plans. The Healthcare Effectiveness Data and Information Set (HEDIS), vol. 2; 2016.
  17. 17. National Asthma E, Prevention P: Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Management of Asthma-Summary Report 2007. J Allergy Clin Immunol 2007, 120(5 Suppl):S94–138.
  18. 18. Bisgaard H, Szefler S: Prevalence of asthma-like symptoms in young children. Pediatr Pulmonol 2007, 42(8):723–728. pmid:17598172
  19. 19. Molis WE, Bagniewski S, Weaver AL, Jacobson RM, Juhn YJ: Timeliness of diagnosis of asthma in children and its predictors. Allergy 2008, 63(11):1529–1535. pmid:18925889
  20. 20. Yoo KH, Johnson SK, Voigt RG, Campeau LJ, Yawn BP, Juhn YJ: Characterization of asthma status by parent report and medical record review. The Journal of allergy and clinical immunology 2007, 120(6):1468–1469. pmid:17981319
  21. 21. Juhn Y, Kung A, Voigt R, Johnson S: Characterisation of children’s asthma status by ICD-9 code and criteria-based medical record review. Primary care respiratory journal: journal of the General Practice Airways Group 2011, 20(1):79–83. pmid:21063669
  22. 22. Yunginger JW, Reed CE, O’Connell EJ, Melton L, O’Fallon WM, Silverstein MD: A community-based study of the epidemiology of asthma. Am Rev Respir Dis 1992, 146(4):888–894. pmid:1416415
  23. 23. Ducharme FM, Dell SD, Radhakrishnan D, Grad RM, Watson WTA, Yang CL, et al: Diagnosis and management of asthma in preschoolers: A Canadian Thoracic Society and Canadian Paediatric Society position paper. Canadian Respiratory Journal: Journal of the Canadian Thoracic Society 2015, 22(3):135–143.
  24. 24. Wi CI, Sohn S, Rolfes MC, Seabright A, Ryu E, Voge G, et al: Application of a Natural Language Processing Algorithm to Asthma Ascertainment. An Automated Chart Review. American journal of respiratory and critical care medicine 2017, 196(4):430–437. pmid:28375665
  25. 25. Wu ST, Juhn YJ, Sohn S, Liu H: Patient-level temporal aggregation for text-based asthma status ascertainment. Journal of the American Medical Informatics Association: JAMIA 2014, 21(5):876–884. pmid:24833775
  26. 26. Wu ST, Sohn S, Ravikumar KE, Wagholikar K, Jonnalagadda SR, Liu H, et al: Automated chart review for asthma cohort identification using natural language processing: an exploratory study. Annals of allergy, asthma & immunology: official publication of the American College of Allergy, Asthma, & Immunology 2013, 111(5):364–369.
  27. 27. Yunginger JW, Reed CE, O’Connell EJ, Melton LJ, 3rd, O’Fallon WM, Silverstein MD: A community-based study of the epidemiology of asthma. Incidence rates, 1964–1983. Am Rev Respir Dis 1992, 146(4):888–894. pmid:1416415
  28. 28. Beard CM, Yunginger JW, Reed CE, O’Connell EJ, Silverstein MD: Interobserver variability in medical record review: an epidemiological study of asthma. J Clin Epidemiol 1992, 45(9):1013–1020. pmid:1432015
  29. 29. Hunt LW Jr., Silverstein MD, Reed CE, O’Connell EJ, O’Fallon WM, Yunginger JW: Accuracy of the death certificate in a population-based study of asthmatic patients. Jama 1993, 269(15):1947–1952. pmid:8464126
  30. 30. Silverstein MD, Reed CE, O’Connell EJ, Melton LJ, 3rd, O’Fallon WM, Yunginger JW: Long-term survival of a cohort of community residents with asthma. N Engl J Med 1994, 331(23):1537–1541. pmid:7969322
  31. 31. Bauer BA, Reed CE, Yunginger JW, Wollan PC, Silverstein MD: Incidence and outcomes of asthma in the elderly. A population-based study in Rochester, Minnesota. Chest 1997, 111(2):303–310. pmid:9041973
  32. 32. Silverstein MD, Yunginger JW, Reed CE, Petterson T, Zimmerman D, Li JT, et al: Attained adult height after childhood asthma: effect of glucocorticoid therapy. J Allergy Clin Immunol 1997, 99(4):466–474. pmid:9111490
  33. 33. Juhn YJ, Qin R, Urm S, Katusic S, Vargas-Chanes D: The influence of neighborhood environment on the incidence of childhood asthma: a propensity score approach. J Allergy Clin Immunol 2010, 125(4):838–843 e832. pmid:20236695
  34. 34. Juhn YJ, Sauver JS, Katusic S, Vargas D, Weaver A, Yunginger J: The influence of neighborhood environment on the incidence of childhood asthma: a multilevel approach. Soc Sci Med 2005, 60(11):2453–2464. pmid:15814171
  35. 35. Juhn YJ, Weaver A, Katusic S, Yunginger J: Mode of delivery at birth and development of asthma: A population-based cohort study. J Allergy Clin Immun 2005, 116(3):510–516. pmid:16159617
  36. 36. Yawn BP, Yunginger JW, Wollan PC, Reed CE, Silverstein MD, Harris AG: Allergic rhinitis in Rochester, Minnesota residents with asthma: frequency and impact on health care charges. J Allergy Clin Immunol 1999, 103(1 Pt 1):54–59. pmid:9893185
  37. 37. Seol HY, Rolfes MC, Chung W, Sohn S, Ryu E, Park MA, et al: Expert artificial intelligence-based natural language processing characterises childhood asthma. BMJ Open Respiratory Research 2020, 7(1). pmid:33371009
  38. 38. Seol HY, Sohn S, Liu H, Wi CI, Ryu E, Park MA, et al: Early Identification of Childhood Asthma: The Role of Informatics in an Era of Electronic Health Records. Frontiers in pediatrics 2019, 7:113. pmid:31001500
  39. 39. Juhn YJ: Risks for infection in patients with asthma (or other atopic conditions): Is asthma more than a chronic airway disease? The Journal of allergy and clinical immunology 2014, 134(2):247–257 e243. pmid:25087224
  40. 40. St Sauver JL, Grossardt BR, Yawn BP, Melton LJ, 3rd, Pankratz JJ, Brue SM, et al: Data resource profile: the Rochester Epidemiology Project (REP) medical records-linkage system. Int J Epidemiol 2012, 41(6):1614–1624. pmid:23159830
  41. 41. St. Sauver JL, Grossardt BR, Leibson CL, Yawn BP, Melton Iii LJ, Rocca WA: Generalizability of Epidemiological Findings and Public Health Decisions: An Illustration From the Rochester Epidemiology Project. Mayo Clinic Proceedings 2012, 87(2):151–160. pmid:22305027
  42. 42. Yawn BP, Wollan P, Kurland M, Scanlon P: A longitudinal study of the prevalence of asthma in a community population of school-age children. Journal of Pediatrics 2002, 140(5):576–581.
  43. 43. Zhong W, Finnie DM, Shah ND, Wagie AE, St. Sauver JL, Jacobson DJ, et al: Effect of Multiple Chronic Diseases on Health Care Expenditures in Childhood. J Prim Care Community Health 2015, 6(1):2–9. pmid:25001922
  44. 44. Busse WW, Morgan WJ, Taggart V, Togias A: Asthma outcomes workshop: Overview. The Journal of allergy and clinical immunology 2012, 129(3):S1–S8. pmid:22386504
  45. 45. Lee MS, Ray KN, Mehrotra A, Giboney P, Yee HF Jr, Barnett ML: Primary care practitioners’ perceptions of electronic consult systems: A qualitative analysis. JAMA Internal Medicine 2018. pmid:29801079
  46. 46. Visscher SL, Naessens JM, Yawn BP, Reinalda MS, Anderson SS, Borah BJ: Developing a standardized healthcare cost data warehouse. BMC Health Serv Res 2017, 17(1):396. pmid:28606088
  47. 47. Schatz M, Sorkness CA, Li JT, Marcus P, Murray JJ, Nathan RA, et al: Asthma Control Test: reliability, validity, and responsiveness in patients not previously followed by asthma specialists. J Allergy Clin Immunol 2006, 117(3):549–556. pmid:16522452
  48. 48. Cajigal S, Wells KE, Peterson EL, Ahmedani BK, Yang JJ, Kumar R, et al: Predictive Properties of the Asthma Control Test and Its Component Questions for Severe Asthma Exacerbations. The journal of allergy and clinical immunology In practice 2017, 5(1):121–127.e122. pmid:27544712
  49. 49. Murphy KR, Zeiger RS, Kosinski M, Chipps B, Mellon M, Schatz M, et al: Test for respiratory and asthma control in kids (TRACK): a caregiver-completed questionnaire for preschool-aged children. J Allergy Clin Immunol 2009, 123(4):833–839.e839. pmid:19348922
  50. 50. Zeiger RS, Mellon M, Chipps B, Murphy KR, Schatz M, Kosinski M, et al: Test for Respiratory and Asthma Control in Kids (TRACK): Clinically meaningful changes in score. The Journal of allergy and clinical immunology 2011, 128(5):983–988. pmid:21906790
  51. 51. Findley S, Rosenthal M, Bryant-Stephens T, Damitz M, Lara M, Mansfield C, et al: Community-based care coordination: practical applications for childhood asthma. Health Promot Pract 2011, 12(6 Suppl 1):52s–62s. pmid:22068360
  52. 52. Gorin AA, Lenz EM, Cornelius T, Huedo-Medina T, Wojtanowski AC, Foster GD: Randomized Controlled Trial Examining the Ripple Effect of a Nationally Available Weight Management Program on Untreated Spouses. Obesity (Silver Spring, Md) 2018, 26(3):499–504. pmid:29388385
  53. 53. Jackson SE, Steptoe A, Wardle J: The influence of partner’s behavior on health behavior change: the English Longitudinal Study of Ageing. JAMA Intern Med 2015, 175(3):385–392. pmid:25599511
  54. 54. White E, Hurlich M, Thompson RS, Woods MN, Henderson MM, Urban N, et al: Dietary Changes Among Husbands of Participants in a Low-Fat Dietary Intervention. American Journal of Preventive Medicine 1991, 7(5):319–325. pmid:1790038
  55. 55. Katheryn Courville PD: How a smartphone application changes the behavior, thinking and attitudes of interprofessional team members: finding, mechanisms using realist evaluaiton. In: 2018 Clinical Informatics Conference: 2018; Scottsdale, AZ; 2018.
  56. 56. Kaur H, Sohn S, Wi CI, Ryu E, Park MA, Bachman K, et al: Automated chart review utilizing natural language processing algorithm for asthma predictive index. BMC pulmonary medicine 2018, 18(1):34. pmid:29439692
  57. 57. Lindgren P, Johnson J, Williams A, Yawn B, Pratt GC: Asthma exacerbations and traffic: examining relationships using link-based traffic metrics and a comprehensive patient database. Environ Health 2016, 15(1):102. pmid:27809853
  58. 58. Sbihi H, Koehoorn M, Tamburic L, Brauer M: Asthma Trajectories in a Population-based Birth Cohort. Impacts of Air Pollution and Greenness. American journal of respiratory and critical care medicine 2017, 195(5):607–613. pmid:27606967