https://orcid.org/0000-0003-4973-1050
Figures
Abstract
Study objective
The purpose of this feasibility study is to develop and validate a new assessment tool and scoring system for multitasking competency for physicians in-training in a timed simulated setting. The multitasking competency includes ability to appropriately prioritize and implement tasks for different patients who present simultaneously.
Methods
We designed three single task stations with different levels of difficulty and priority. These skill stations were then combined to create a multitasking simulation scenario. Skill checklists and the global rating scale were utilized to assess the participants’ performance. A multitasking score, multitasking index, and priority score were developed to measure the multitasking ability of participants.
Results
Thirty-three first-year postgraduate physicians were recruited for this prospective study. The total performance scores were significantly higher for the single-tasking stations than for the multitasking scenario. In terms of the time needed to complete the tasks, the participants spent more time on the multitasking scenario than on the single-tasking scenario. There were significant correlations between the global rating scale and the multitasking score (rho = 0.693, p < 0.001) and between the global rating scale and the multitasking index (rho = 0.515, p < 0.001). The multitasking score, multitasking index, and priority score did not have any significant correlations with the total single-tasking score.
Citation: Huang W-C, Hsu S-C, Yang C-H, Lin C-W, Suk F-M, Hu K-C, et al. (2021) A novel approach: Simulating multiple simultaneous encounters to assess multitasking ability in emergency medicine. PLoS ONE 16(9): e0257887. https://doi.org/10.1371/journal.pone.0257887
Editor: James Andrew Rowley, Harper University Hospital, UNITED STATES
Received: February 24, 2021; Accepted: September 13, 2021; Published: September 28, 2021
Copyright: © 2021 Huang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This research was supported by Wan-Fang Hospital, Taipei Medical University, Grant number- 110-wf-eva-23. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In the current healthcare system, physicians’ abilities to manage frequent interruptions and prioritize essential tasks in a timely manner have become increasingly valuable [1]. Especially in an emergency department setting, interruptions are part of the working conditions. The high frequency of interruptions experienced during a shift by emergency medicine (EM) physicians has been documented [2, 3]. Previous studies demonstrated that interruptions divert physicians’ attention away from their primary tasks. Interruptions that lead to breaks in a task could cause medical errors and increase the potential for harming patients [4, 5]. In an ideal environment, distractions would be minimized or eliminated; however, distractions are inevitable in the emergency room setting. Therefore, the EM physician must learn how to cope in a multitasking environment.
The ability to multitask has been recognized as an essential skill in emergency medicine [6]. In EM, multitasking is defined as the ability to prioritize and implement the evaluation and management of multiple patients in the emergency department, including handling interruptions and task switching to provide optimal care [6–8]. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Emergency Medicine developed “The Emergency Medicine Milestone Project,” which provides a framework for the assessment of the development of the resident physician in key dimensions of the elements of physician competency in the specialty [9]. Multitasking is one of the sub competencies in patient care. The current method of evaluating multitasking ability is mainly dependent on direct observation. The mini-clinical examination (mini-CEX) and the standardized direct observation tool for EM (SDOT) are the two most commonly used direct observation tools [6, 10, 11]. Although simulation-based assessment is one of the evaluation methods suggested by The Emergency Medicine Milestone Project, such a method’s validity has not yet been documented.
While medical simulation has become a standard educational modality within graduate and undergraduate medical education, there are few publications regarding the use of simulation as an evaluation tool in patient care [12–16]. A few studies have demonstrated using simulations of multiple encounters to teach multitasking [17–19]. However, utilizing multiple encounter simulation-based methods to assess multitasking skills has not been fully investigated. The purpose of this feasibility study is to develop and validate a new assessment tool and scoring system for multitasking competency for physicians in-training in a timed simulated setting.
Methods
In this study, we designed a simulated multitasking scenario which included three skills (intubation, bedside ultrasonography and suturing) stations with different levels of priority and difficulty. In the scenario, participants performed these tasks simultaneously, and assign priorities according to their perception. The main objective of this study design is to validate the feasibility of this tool to measure the ability to multitask. We further observed whether the novel method could assess learners’ performance and provide educators additional information to provide feedback.
Study population
This study enrolled first-year postgraduate physicians who had 2 years of clinical experience and completed one-month emergency and simulation training at an academic center in Taiwan. Participation in this study was voluntary and no remuneration was provided. A total of 33 first year post-graduates were recruited via email and social media.
Study design
We designed three short single task stations with different levels of difficulty and priority (low, moderate, high). Intubation tasks were considered as high priority, bedside ultrasonography was moderate and wound suturing was low priority.
These three short single-task stations were combined to create a multitask scenario.
Participants completed three independent single task scenarios and one combined multitasking scenario, and were evaluated using a standardized checklist for each task. The checklist for the multitasking scenario was a combination of the checklists of the single task scenarios. We compared the score of the 3 single task stations against the score of each station in the multitasking scenario to determine the impact of multitasking on skill performance. To reduce the testing effect bias, approximately half of the participants (n = 16) were assessed on the multitasking scenario first. The remaining participants completed the single-tasking stations first. Participants were randomized to either the multitasking simulation first or skill stations in series first. A computer-generated list of random numbers was used for the allocation.
Ethical approval
This observational and prospective study was approved by the Association of Taipei Medical University Joint Institutional Review Board (TMU-JIRB NO: N201808021).
We explained the purpose and methods (The participants were videotaped and the videos were reviewed by raters) of the research in written documents and regarded the return of completed survey forms as consent to participate in the research.
Scenario design
The three skill stations and their priority and difficulty levels are summarized in Table 1. The completion time for each single task station was recorded. In the multitasking scenario, we individually recorded the total amount of time spent at each of the 3 tasks. In the multitasking evaluation, all participants started at the suture station (low priority). During the scenario, two standardized nurses delivered updates about the other stations at preset times (e.g., intubation station: poor oxygen saturation with respiratory failure; ultrasound station: severe abdominal pain with hypotension). The participants had to make decisions about which task to handle first. While working on any given station, they faced two interruptions with status updates from the other stations (Fig 1, S1 File).
Scoring method and measurement
Each scenario was assessed by two experienced (over ten years of practice) emergency physicians. Each single task scenario was evaluated using a standardized checklist previously validated by the clinical and educational experts’ consensus and analysis in Taiwan. The multitasking scenario was evaluated using the same checklists (S1 Table) and an additional Global Rating Scale (GRS) based on the 14th sub competency (Multitasking) of the Emergency Medicine Milestones (S2 File) [23]. A priority score was also determined according to the order in which the participant completed the tasks. To measure multitasking ability in the combined simulation scenario, we used a multitasking score (MS), multitasking index (MTI) and a priority score (PS). The MS is the sum of the individual task checklist scores to assess the ability of task completion in the multitasking scenario. In addition, the MTI was defined as the multitasking simulation score divided by the sum of the single-tasking simulation scores. The MTI is designed to quantify the ratio of task completion and build a model to directly observe multitasking performance. To correctly assess the multitasking ability of the participants, they must have an acceptable level of competency in each of the skills included. As a result, the MTI scores were calculated only for the participants who scored greater than 50% on all three single-tasking stations. Multitasking ability includes the performance of each independent skill and the ability to prioritize the performance of each of the scenario tasks. Consequently, we developed the algorithm shown in Fig 2 to calculate the PS. The participants’ performances were scored by raters using the above tools as well as the GRS for the single-tasking stations and the multitasking scenario.
(I: intubation; F: FAST; S: suture).
Scenario rating
All raters had completed standardized training and held a valid certification as OSCE raters by the Taiwan Medical Education Association. To assess inter-rater reliability, 12 additional participants were recruited and assessed in 3 single-task and a multitasking scenario. All scenarios were assessed by a single assessor and video recorded. The video recording was assessed by a second assessor with no previous knowledge of the initial assessment score. (The inter-rater reliability for all encounters is provided in S3 File)
Statistical analysis
The statistical analysis of the data obtained in the study was performed with R 3.2.4 software (R Foundation for Statistical Computing, Vienna, Austria). Continuous variables are expressed as the means and standard deviations (SDs) or medians and interquartile ranges. Categorical variables are expressed as counts or percentages. The Mann-Whitney U-test was used for continuous variables that did not adhere to a parametric distribution. The degree of association between variables was measured by the Spearman rank correlation test. All statistical tests were two-sided and paired. A value of p < 0.05 was considered statistically significant.
Results
The baseline characteristics of the participants are described in Table 2. In total, 33 first-year medical graduates were included in this study, of whom 20 (60.61%) were male. The median age of the medical graduates was 25 years (IQR: 25 years to 25 years). The total performance scores were significantly higher for the single-tasking scenarios than for the multitasking scenario. After stratifying the scores by different priorities, the scores at each station were still significantly higher for the single-tasking scenarios than for the multitasking scenario (Fig 3). Additionally, the order in which the scenarios were evaluated did not influence the MS (Fig 4). In terms of the time needed to complete the tasks, the participants spent more time on the multitasking scenario than on the single-tasking scenarios. However, the subgroup data suggested that the participants used a significant amount of time on intubation in the multitasking scenario (Fig 5). To assess the prioritization ability of the participants, the PS was used. The median of the PS was 3 (IQR = 3–2). There were significant correlations between the GRS and the MS (rho = 0.693, p < 0.001). In our data of MTI, 6 participants who do not satisfy the criteria were excluded from the calculations (the participants need to obtain greater than 50% of score on each single-tasking station). There were also significant correlations between the GRS and the MTI (rho = 0515, p < 0.001). However, the GRS was not significantly correlated with the PS. The MS, MTI, and PS were not significantly correlated with the total single-tasking score (Table 3).
Discussion
The ability to multitask demonstrates two core competencies, namely, prioritization when faced with multiple patients and implementation to maintain focus on individual patients [20–22]. In the past, the methods used to evaluate multitasking ability were based on direct observation by clinical preceptors [10, 23]. However, these methods require a relatively long period for observation. Additionally, multitasking situations in clinical practice are often unpredictable and uncontrollable.
The accurate measurement of multitasking ability requires specific clinical challenges to be present simultaneously. In this study, we have demonstrated the use of a simulated multitasking scenario with tasks with different priority levels to assess physicians’ multitasking abilities.
In this study, we observed that the clinical performance of the participant was affected by multi patient encounters. In single-tasking scenarios, participants can concentrate on a single task; there is no prioritization needed, and less psychological stress is imposed. When faced with the need for multitasking, participants experience interruptions. In addition, they are required to prioritize patients, which imposes more psychological stress. Our results indicate that the overall performance scores for the single-tasking stations were higher than those for the multitasking scenario.
The results correspond with the findings of A Parush et al. which proposed that the multitasking environment affects EM physicians’ concentration and performance [24].
When comparing the participants’ performance on the multitasking scenario and each single-tasking station, Our results showed that in the multitasking scenario, participants allocated significantly more time on the intubation station than on the lower priority stations (suture/FAST). This result can be explained by the findings of a previous publication, which suggests that participants prioritize the more critical patients and spend more time completing the tasks associated with those patients [25]. Conversely, they spend less time on the other stations with lower priority levels, and these results are also consistent with real clinical practice [26]. During multitasking situations, in high-priority tasks the performance of the clinicians was relatively unaffected but they required additional time to complete these tasks. Low-priority tasks were easily neglected in the multitasking setting. This finding reveals that it is important to provide additional training or clinical assistance to address this issue.
Our study showed a significant positive correlation between the MTI and the traditional GRS score of direct observation in the multitasking scenario.
If MTI is used for the assessment of multitasking ability, the efficacy will be similar to direct observation, indicating that we could use the multitasking scenario to assess participants’ multitasking ability under safe and controllable simulated conditions. The MS and GRS scores were also positively correlated. However, we believed that the Multitasking Index could better represent the participant’s implementation ability in a multitasking setting since it considers the participant’s single-tasking score (Their baseline ability). The MTI could be used to establish a standard tool to assess the competencies of different levels of physicians.
In addition, we also used the PS to evaluate the decision-making abilities of the participants during the multitasking evaluation. Interestingly, the PS was not correlated with the GRS score. This may be because the GRS focuses primarily on the completion of the clinical tasks and does not emphasize decision making. Prioritization is a crucial skill in clinical practice, especially during multiple patient encounters. In the past, we emphasized the completion of clinical tasks when evaluating multitasking, but prioritization is an important core competency and component of multitasking [27]. Therefore, we believe that the PS should remain a reference parameter when evaluating a participant’s multitasking ability.
In previous studies assessing multitasking, direct observation was the major technique used [10, 23]. For EM residents, multitasking ability is commonly assessed with the Mini-CEX and SDOT in clinical settings. However, evaluating multitasking in an emergency room can cause clinical interference and assess performance less objectively [28]. It is impossible to have enough suitable patients for clinical preceptors to be able to evaluate multitasking at any time in clinical practice. Therefore, the well-designed multiple encounter simulation developed for this study can be used to observe targeted clinical competencies. Moreover, this method of the assessment does not interfere with clinical work and preserves patient safety. Additionally, multitasking simulation can provide comparable assessment accuracy to the traditional forms of assessment by direct observation.
Limitations
Some limitations of this study should be noted. As this was a pilot study, only 33 participants, carried out at a single institution. The sample size and the single institution may limit result generalizability. Because we only enrolled first-year medical graduates, gold standards for multitasking could not be established. Future studies, should include larger sample sizes and participants at different levels of professional development, such as EM residents and EM attending physicians. In addition, we hope to identify specific factors that negatively impact performance during multi patient encounters. A better understanding of these factors and their impact on task performance will allow the development of better training methods or courses. Finally, the checklists that were utilized in our study only assessed basic competency, but provide limited information about task performance quality.
Conclusion
Using simulated multi-patient encounters, this study shows initial validity that the multitasking core competency can be assessed using three newly developed assessment tools: the multitasking score, the multitasking index, and the priority score. This setting may allow educators to offer concrete and timely feedback to trainees without interference to emergency patient care.
Supporting information
S1 File. The multitasking scenario assessment demonstration.
https://doi.org/10.1371/journal.pone.0257887.s001
(DOCX)
S2 File. Global Rating Scales for multitasking scenario.
https://doi.org/10.1371/journal.pone.0257887.s002
(DOCX)
S3 File. The inter-rater reliability for all encounters assessed.
https://doi.org/10.1371/journal.pone.0257887.s003
(DOCX)
S1 Table. Checklists for stations and multitasking scenario.
https://doi.org/10.1371/journal.pone.0257887.s004
(DOCX)
Acknowledgments
We thank our colleagues Hao-Yu Chen and Hung-Chen Chen in the Center for Education in Medical Simulation (CEMS) who provided insight and expertise that greatly assisted the research.
References
- 1. Adams TN, Rho JC. Multitasking simulation: Present application and future directions. Med Teach. 2017;39: 120–122. pmid:27633071
- 2. Chisholm CD, Collison EK, Nelson DR, Cordell WH. Emergency department workplace interruptions: Are emergency physicians “interrupt-driven” and “multitasking”? Acad Emerg Med. 2000;7: 1239–1243. pmid:11073472
- 3. Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking clinician: Decision-making and cognitive demand during and after team handoffs in emergency care. Int J Med Inform. 2007;76: 801–811. pmid:17059892
- 4.
Institute of Medicine. To Err Is Human. Building a Safer Health System. National Academy Press. Washington, D.C.: National Academies Press; 2000. https://doi.org/10.17226/9728 pmid:25077248
- 5. Grundgeiger T, Sanderson P. Interruptions in healthcare: theoretical views. Int J Med Inform. 2009;78: 293–307. pmid:19081295
- 6. Heng KWJ. Teaching and evaluating multitasking ability in emergency medicine residents—what is the best practice? Int J Emerg Med. 2014;7: 1–5. pmid:24386894
- 7. Beeson MS, Carter WA, Christopher TA, Heidt JW, Jones JH, Meyer LE, et al. The development of the emergency medicine milestones. Burton J, editor. Acad Emerg Med. 2013;20: 724–729. pmid:23782404
- 8. Counselman FL, Babu K, Edens MA, Gorgas DL, Hobgood C, Marco CA, et al. The 2016 Model of the Clinical Practice of Emergency Medicine. J Emerg Med. 2017;52: 846–849. pmid:28351510
- 9.
The Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine. The Emergency Medicine Milestone Project. 2012; 22. Available: http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf
- 10. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003;138: 476–81. Available: http://www.ncbi.nlm.nih.gov/pubmed/12639081 pmid:12639081
- 11. Wang EE, Dyne PL, Du H. Systems-based practice: Summary of the 2010 council of emergency medicine residency directors academic assembly consensus workgroup-teaching and evaluating the difficult-to-teach competencies. Acad Emerg Med. 2011;18: S110–S120. pmid:21999553
- 12. Girzadas DV., Clay L, Caris J, Rzechula K, Harwood R. High fidelity simulation can discriminate between novice and experienced residents when assessing competency in patient care. Med Teach. 2007;29: 472–476. pmid:17885976
- 13. Okuda Y, Quinones J. The use of simulation in the education of emergency care providers for cardiac emergencies. Int J Emerg Med. 2008;1: 73–77. pmid:19384655
- 14. Reid J, Stone K, Brown J, Caglar D, Kobayashi A, Lewis-Newby M, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation. 2012;83: 879–886. pmid:22198422
- 15. Beeson MS, Vozenilek JA. Specialty milestones and the next accreditation system: an opportunity for the simulation community. Simul Healthc. 2014;9: 184–91. pmid:24614793
- 16. Hart D, Bond W, Siegelman JN, Miller D, Cassara M, Barker L, et al. Simulation for Assessment of Milestones in Emergency Medicine Residents. Acad Emerg Med. 2018;25: 205–220. pmid:28833892
- 17. Kobayashi L, Shapiro MJ, Gutman DC, Jay G. Multiple Encounter Simulation for High-acuity Multi patient Environment Training. Acad Emerg Med. 2007;14: 1141–1148. pmid:18045888
- 18. Fraser TN, Genuardi M, McCullough SA, Peloquin G, Currier PF, Miloslavsky EM. A Pilot Study of Multiple Simultaneous Critical Care Simulations. J Grad Med Educ. 2015;7: 303. pmid:26221469
- 19. Poolton JM, Zhu FF, Malhotra N, Leung GKK, Fan JKM, Masters RSW. Multitask training promotes automaticity of a fundamental laparoscopic skill without compromising the rate of skill learning. Surg Endosc. 2016;30: 4011–4018. pmid:26743112
- 20. France DJ, Levin S, Hemphill R, Chen K, Rickard D, Makowski R, et al. Emergency physicians’ behaviors and workload in the presence of an electronic whiteboard. Int J Med Inform. 2005;74: 827–37. pmid:16043391
- 21.
Salvucci DD, Taatgen NA, Borst JP. Toward a unified theory of the multitasking continuum. Proceedings of the 27th international conference on Human factors in computing systems—CHI 09. New York, New York, USA: ACM Press; 2009. p. 1819. https://doi.org/10.1145/1518701.1518981
- 22. Skaugset LM, Farrell S, Carney M, Wolff M, Santen SA, Perry M, et al. Can You Multitask? Evidence and Limitations of Task Switching and Multitasking in Emergency Medicine. Ann Emerg Med. 2016;68: 189–195. pmid:26585046
- 23.
Council of Emergency Medicine Residency Directors. Standardized Direct Observation Tool. [cited 12 Mar 2019]. Available: https://www.cordem.org/resources/residency-management/cord-standardized-assessment-methods/
- 24. Parush A, Worthington J, Abbott C, Frank JR, Ed MA. Situational Awareness and Patient Safety. AORN J. 2017;106: 433–468. pmid:29107261
- 25. Ratwani RM, Fong A, Puthumana JS, Hettinger AZ. Emergency Physician Use of Cognitive Strategies to Manage Interruptions. Ann Emerg Med. 2017;70: 683–687. pmid:28601266
- 26. Blocker RC, Heaton HA, Forsyth KL, Hawthorne HJ, El-Sherif N, Bellolio MF, et al. Physician, Interrupted: Workflow Interruptions and Patient Care in the Emergency Department. J Emerg Med. 2017;53: 798–804. pmid:29079489
- 27. Weigl M, Müller A, Sevdalis N, Angerer P. Relationships of Multitasking, Physicians’ Strain, and Performance. J Patient Saf. 2012; 1. pmid:23007246
- 28. Barrett A, Galvin R, Steinert Y, Scherpbier A, O’Shaughnessy A, Horgan M, et al. A BEME (Best Evidence in Medical Education) review of the use of workplace-based assessment in identifying and remediating underperformance among postgraduate medical trainees: BEME Guide No. 43. Med Teach. 2016;38: 1188–1198. pmid:27626710