Amid growing rates of burnout, physicians report increasing electronic health record (EHR) usage alongside decreasing clinical facetime with patients. There exists a pressing need to improve physician-computer-patient interactions by streamlining EHR workflow. To identify interventions to improve EHR design and usage, we systematically characterize EHR activity among internal medicine residents at a tertiary academic hospital across various inpatient rotations and roles from June 2013 to November 2016. Logged EHR timestamps were extracted from Stanford Hospital’s EHR system (Epic) and cross-referenced against resident rotation schedules. We tracked the quantity of EHR logs across 24-hour cycles to reveal daily usage patterns. In addition, we decomposed daily EHR time into time spent on specific EHR actions (e.g. chart review, note entry and review, results review).In examining 24-hour usage cycles from general medicine day and night team rotations, we identified a prominent trend in which night team activity promptly ceased at the shift’s end, while day team activity tended to linger post-shift. Across all rotations and roles, residents spent on average 5.38 hours (standard deviation = 2.07) using the EHR. PGY1 (post-graduate year one) interns and PGY2+ residents spent on average 2.4 and 4.1 times the number of EHR hours on information review (chart, note, and results review) as information entry (note and order entry).Analysis of EHR event log data can enable medical educators and programs to develop more targeted interventions to improve physician-computer-patient interactions, centered on specific EHR actions.
Citation: Wang JK, Ouyang D, Hom J, Chi J, Chen JH (2019) Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS ONE 14(2): e0205379. https://doi.org/10.1371/journal.pone.0205379
Editor: Brenessa Lindeman, University of Alabama at Birmingham, UNITED STATES
Received: September 13, 2018; Accepted: January 19, 2019; Published: February 6, 2019
Copyright: © 2019 Wang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors have provided aggregated data and code in the following public repository: https://github.com/HealthRex/CDSS/tree/master/scripts/ResidentRotationEHRUsage. The repository includes electronic health record (EHR) usage patterns aggregated at a user-day level (the activity of a given resident on a given work day), including total time spent per action category (e.g. chart review, note entry), number of distinct patient records accessed, number of total EHR actions, etc. The repository also includes Python code necessary to compute summary statistics and 24-hour EHR usage cycles, used to generate Table 1 and Fig 1.
Funding: This project was supported by the Stanford Cyber Initiative. Jonathan H. Chen was supported in part by the NIH Big Data 2 Knowledge Award Number K01ES026837 through the National Institute of Environmental Health Sciences. Patient data was extracted and de-identified by Gomathi Krishnan of the STRIDE (Stanford Translational Research Integrated Database Environment) project, a research and development project at Stanford University to create a standards-based informatics platform supporting clinical and translational research. The STRIDE project described was supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1 RR025744. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH, or Stanford Healthcare.
Competing interests: The authors have declared that no competing interests exist.
As medical educators, we hope our resident trainees value direct patient care and contact. Instead, we may progressively find their attention dominated by electronic health records (EHR) that mediate their work and proxy for their patients. Observational studies confirm an increasing shift from direct patient care to computer use in the wake of duty hour restrictions,[2,3,4] informing ongoing reforms into the structure of medical training. Physicians report increasing time spent on paperwork and the computer, alongside less time available for clinical interactions with patients. A growing burden of redundant clinical notes, alert fatigue, and an overflowing inbox has led to a systemic “4000 clicks a day” problem that has contributed to physician job dissatisfaction and burnout rates. Indeed, correlations between an increasing EHR task load and physician burnout have been demonstrated.[6,9] There exists a pressing need to understand clinical EHR usage to inform opportunities to improve effective patient care processes. Previous studies have quantified EHR usage intensity through direct observation and self-reported diaries.[2, 10] Ironically, with the amount of time trainees spend on the EHR, the computer also provides precise, reproducible, and scalable quantification of their electronic activities. Event log data, for instance, can capture a provider’s digital timeline of actions by tracking his or her clicks within an EHR interface. Published comparisons between manual observation of clinician activity and automatic analysis of logged EHR timestamps confirm that the two approaches yield similar results for workflow analysis.[11, 12] Systematic evaluation of event log data can thus elucidate provider EHR usage patterns and inform the development more targeted interventions to improve physician-computer-patient interactions.
Using event log timestamps, systematically characterize the intensity of EHR usage across different inpatient medicine rotations and roles.
Logged timestamps from the Epic EHR system for inpatient rotations of internal medicine residents at an academic tertiary care hospital were extracted from the STRIDE project from June 2013 to November 2016. We tracked the quantity of logged EHR actions accessed over 24-hour cycles per user binning by half-hour intervals. Logged EHR actions correspond to behaviors performed on the EHR as clinicians navigate components (e.g. notes, orders, results) of a patient’s electronic chart. Epic-coded EHR actions were binned into broad behavioral categories by a board-certified internal medicine physician. The most common action categories included chart review (provider review of patient medical history, diagnoses, symptoms, demographics, etc.), note review, results review (provider review of lab and imaging results), note entry, order entry, and navigator use. The EHR (Epic) navigator consists of pre-curated sequences of modules to facilitate common actions such as admission, rounding, and discharge. To estimate time spent on a given action, we considered time intervals between event logs separated by five minutes or less of inactivity. For example, if a resident accessed a chart review action at time A and a results review action at time B, and the time interval between A and B was less than five minutes, the interval would be attributed to action A (chart review). If the time interval surpassed five minutes, idleness would be assumed. Sensitivity of results to integer idleness thresholds varying between 5–10 minutes were evaluated. Daily EHR usage was estimated as the sum of all active inter-access time intervals per day. User timestamps were cross-referenced against resident year and rotation schedules to account for each user’s progression through the internal medicine residency program over the three-year window. User-days with less than one hour of activity were excluded from analysis to account for remote access during vacation days. P-values were computed using two-sample t-tests allowing unequal variances (Welch’s t-test). Analyses were performed with Python 2.7 and R 2.13.
This study received written approval from the Stanford Institutional Review Board (Protocol Number 35059). The IRB waived the requirement for explicit informed consent, designating the study a “Chart Review” based on data already collected for other (clinical) purposes.
During the three-year window, 15,909,629 unique actions were logged by 101 unique residents covering 99 PGY1 (post-graduate year one) intern-years and 61 PGY2+ resident-years. Fig 1 illustrates the intensity of EHR interactions per resident day during a 24-hour cycle for PGY1 interns versus PGY2+ residents across different inpatient rotations.
Mean number of EHR actions logged per user day in half hour increments over a 24-hour cycle for different inpatient rotations, split by PGY1 interns (A) and PGY2+ residents (B). General Medicine Day and Night Teams transition care responsibility at 7 AM and 7 PM. Daytime patient care is regularly interrupted by 10 AM morning report and 12 PM noon teaching conferences. PGY1 interns do not rotate through the ICU. ICU housestaff have a separate noon teaching conference and daily 3 PM care conference. Only the ICU rotation integrates computers-on-wheels (COWs) for physician use in patient care and rounding practices, while other rotations segregate physician (computer) workrooms from patient care areas.
Table 1 and Table 2 highlights mean daily EHR usage times, decomposed by mean time spent on common EHR action categories across each inpatient rotation and role. Across all rotations, trainees spent a mean of 5.38 hours (PGY1 = 5.47, PGY2+ = 5.21) with standard deviation = 2.07 (PGY1 = 2.10, PGY2+ = 2.02) and median of 5.33 hours (PGY1 = 5.47, PGY2+ = 5.09) with interquartile range = 3.85–6.85 (PGY1 = 3.90–6.97, PGY2+ = 3.80–6.65) per work day using the EHR. Medical chart reviews account for the largest portion of activity (>40% on average) followed primarily by note entry and review. PGY1 interns spent disproportionately more time on note entry compared to PGY2+ residents during general medicine (15.2% vs. 4.0% of daily EHR usage, P<0.001) and emergency medicine rotations (18.4% vs. 13.9%, P<0.001), but similar times during the night team rotation (11.9% vs. 11.3%, P = 0.28). Conversely, across all three rotations, PGY2+ residents spent disproportionately more time on note review compared to PGY1 interns (general medicine = 20.7% vs. 13.0%, P<0.001; night team = 21.8% vs. 11.6%, P<0.001; emergency medicine = 11.6% vs. 8.1%, P<0.001). Trainees spent roughly 7.6 minutes (PGY2+ only), 9.7 minutes (PGY1 = 12.3, PGY2+ = 6.6), 3.7 minutes (PGY1 = 3.5, PGY2+ = 4.2), and 7.2 minutes (PGY1 = 8.0, PGY2+ = 5.5) on chart review actions per patient across intensive care unit, general medicine, night team, and emergency department rotations, respectively.
EHR usage summary statistics and mean time spent on common EHR action categories per user-day across different inpatient rotations for PGY1 interns. Time spent on category-specific actions are also shown as a percentage of mean daily EHR usage time.
EHR usage summary statistics and mean time spent on common EHR action categories per user-day across different inpatient rotations for PGY2+ residents. Time spent on category-specific actions are also shown as a percentage of mean daily EHR usage time.
Daily EHR usage estimations prove robust to the choice of inactivity cutoff; the standard deviation of mean daily EHR usages computed using inactivity cutoffs from 5–10 minutes is 28.8 minutes. The choice of a 5-minute cutoff captures the vast majority of active computer sessions (98.4% of inter-access time intervals), while excluding lengthy idle sessions. Excluding user-days based on an activity threshold of one hour removes 11.6% of all user-days (435/3756), corresponding to roughly one day off per week.
Discussion and conclusion
With increasing reliance on EHRs to mediate patient care, direct analysis of EHR audit logs provides a granular and objective way to characterize physician EHR usage. A key limitation of this approach is the estimation of idle time between access logs; overestimating idle times could overestimate EHR usage times. However, our sensitivity analysis showed that estimations were robust to integer idle time cutoffs between 5–10 minutes. Additionally, although results are aggregated across multiple years, the reported EHR usage statistics are derived from a single academic center, and EHR interfaces are often modified according the needs of each provider system. Nonetheless, we observe that key statistics including mean daily EHR usage time are similar to those found in prior studies conducted at other institutions.[10, 12] The pattern of EHR activity over 24-hour cycles provides qualitative insights into resident behavior (Fig 1). The 2011 ACGME duty hour restrictions prompted the separation of the hospital’s General Medicine rotation into Day and Night Teams. These 24-hour cycles may suggest that night teams treat patient care as discrete shift work, with clinical activities promptly ceasing at 7 AM as noted by the steep drop-off, while general medicine (day team) EHR activity often lingers well beyond duty hour recommendations (9 PM onwards) and restrictions (11 PM onwards), defined as ten and eight hours before the subsequent 7 AM shift.Perhaps more concerning is the large burden of physician-computer interaction across all four rotations, a phenomena that almost certainly restricts direct physician-patient interaction. Assuming a 12-hour work day3, PGY1 interns and PGY2+ residents spent nearly half (PGY1 = 46%, PGY2+ = 43%) of work time on the EHR, respectively. Sinsky et al. confirmed a similar statistic through manual observation of provider workflow, noting that between 37–49% of attending work hours were dedicated to EHR or desk tasks.  This trend is especially notable considering the sharp peak in EHR activity upon day team arrival (Fig 1), suggesting the traditional model of pre-rounding at the patient bedside has been replaced by the workroom computer as the trusted source for patient information. Nonetheless, time spent on EHR tasks is still highly individual among providers as we observe substantial variability in mean daily EHR usage statistics, with standard deviations ranging between 1.56–2.58 hours across the four medicine rotations.Comparing between roles (Table 1), we see that PGY2+ residents execute a greater number of EHR actions with disproportionately more time spent on note review, yet less daily EHR time, compared to PGY1 interns. Indeed, in the clinic, PGY2+ residents embrace a supervisory role and often oversee twice the number of patients (note the nearly 2:1 median patient ratio during general medicine rotations) compared to PGY1 interns. Conversely, PGY1 interns are delegated time-intensive note entry duties albeit for a smaller population of patients, potentially accounting for greater daily EHR time. Across diverse rotations, we see significant time spent on the same pattern of EHR activities. PGY1 interns and PGY2+ residents spent on average 2.4 and 4.1 times the number of EHR hours on information review (chart, note, and results review) as information entry (note and order entry) (Table 1), respectively. Improvements in EHR design could target the burden of data retrieval by encouraging more concise documentation and redesigned or automated content organization. By identifying specific EHR activities that consistently dominate resident computer usage across multiple inpatient rotations and roles, we hope to facilitate a more targeted, data-driven approach to improving physician-computer-patient interactions.
Patient data was extracted and de-identified by Gomathi Krishnan of the STRIDE (Stanford Translational Research Integrated Database Environment) project, a research and development project at Stanford University to create a standards-based informatics platform supporting clinical and translational research.
- 1. Chi J & Verghese A Clinical education and the electronic health record: the flipped patient. JAMA 312, 2331–2 (2014). pmid:25490318
- 2. Block L, Habicht R, Wu AW, Desai SV, Wang K, Silva KN, Niessen T, Oliver N, Feldman L. In the wake of the 2003 and 2011 duty hours regulations, how do internal medicine interns spend their time? J. Gen. Intern. Med. 28, 1042–1047 (2013). pmid:23595927
- 3. Ouyang D, Chen J H, Hom J, & Chi J Internal medicine resident computer usage: an electronic audit of an inpatient service. JAMA Internal Medicine. 176, 252–54 (2016). pmid:26642261
- 4. Desai S V, Asch D A, Bellini L M,Chaiyachati K, Liu M, Sternberg A, et al. Education Outcomes in a Duty-Hour Flexibility Trial in Internal Medicine. The New England Journal of Medicine. 10, 1056 (2018).
- 5. Asch D A, Bilimoria K Y & Desai S V. Resident Duty Hours and Medical Education Policy—Raising the Evidence Bar. N. Engl. J. Med. 376, 1704–1706 (2017). pmid:28402246
- 6. Babbott S, Manwell LB, Brown R, Montague E, Williams E, Schwartz M, Hess E, Linzer M.Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc. 21, 1006 (2014).
- 7. Friedberg M, Chen PG, Van Busum KR, Aunon FM, Pham C, Caloyeras J.P., et al. Factors Affecting Physician Professional Satisfaction and Their Implications for Patient Care, Health Systems and Health Policy. RAND Corporation, 2013.
- 8. Hill RG Jr, Sears LM, & Melanson SW 4000 clicks: a productivity analysis of electronic medical records in a community hospital ED. Am J Emerg Med. 31, 1591–1594 (2014).
- 9. Shanafelt TD, Dyrbye LN, Sinsky C, Hasan O, Satele D, Sloan J, West CP. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin Proc. 91, 836–848 (2016). pmid:27313121
- 10. Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L,et al. Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Annals of Internal Medicine 165, 753–60 (2016). pmid:27595430
- 11. Hribar MR, Read-Brown S, Goldstein IH, Reznick LG, Lombardi L, Parikh M, Chamberlain W, Chiang MF.Secondary Use of Electronic Health Record Data for Clinical Workflow Analysis. Journal of the American Medical Informatics Association. 25, 40–46 (2018). pmid:29036581
- 12. Arndt BG, Beasley JW, Watkinson MD, Ternte JL, Tuan WJ, Sinsky C, et al. Tethered to the EHR: Primary Care Physician Workload Assessment Using EHR Event Log Data and Time-Motion Observations. Annals of Family Medicine. 15, 419–26 (2017). pmid:28893811
- 13. Lowe H J, Ferris TA, Hernandez M & Weber S C STRIDE—An integrated standards-based translational research informatics platform. AMIA Annu. Symp. Proc. 2009, 391–5 (2009). pmid:20351886